What is telemetry?

I’ve heard the word telemetry a few times in my career but mainly when going to the hospital where they have a department named as such. The department’s job is to monitor the data or metrics that are being obtained via connected sensors. The other situation I’ve run across the word telemetry is on software product installs that wanted to send product data back to the manufacturer for data collection purposes.

In this case telemetry refers the collection of insightful metrics from a HashiCorp Vault deployment which will be shipped to a centralized platform such as Splunk.

What is StatsD?

StatsD is a protocol sending application metrics to a centralized platform such as Splunk for ingestion. In this example we’re going to use Splunk as our StatsD server for collecting information about our Vault deployment. HashiCorp Vault has a number of different options for a StatsD server.

Vault Configuration

Add the following block to the Vault configuration file to send telemetry data to our Splunk server.

telemetry {   
    statsd_address = "splunk01.grt.local:8125" 
}

Splunk Configuration

Enable a StatsD Input If you don’t have a StatsD input enabled:

• Go to Settings > Data Inputs in the Splunk bar
• Under Local inputs, click Add new next to UDP
• Click Select Source Type, then select Metrics > statsd
• Select an existing metrics Index or create a new one

Enter the number of the port you are using for StatsD 8125 is the standard port for StatsD. Click Next to proceed to the next menu.

Click Select Source Type, then select Metrics > statsd

Create a new metrics index for Vault telemetry data ingestion. Click “Save” and then “Review” to proceed to the confirmation dialog.

Click “Submit” to create the input.

The data input has been configured and we should now be receiving metrics from Vault.

The search command below can be used to view the metric names that are being ingested.

| mcatalog values(metric_name) where index="vault_metrics"

Now that we have the names of metrics that are available to us we can access the data and create a nice graph similar to the one below by running the following search command. The search below fetches the allocated memory bytes for the Vault process.

| mstats avg(_value) WHERE index="vault_metrics" AND metric_name=vault.vault.runtime.alloc_bytes span=30s

References

HashiCorp Vault Telemetry Overview https://www.vaultproject.io/docs/internals/telemetry.html

HashiCorp Vault Telemetry Configuration https://www.vaultproject.io/docs/configuration/telemetry.html

Splunk StatsD Configuration http://docs.splunk.com/Documentation/Splunk/7.1.1/Metrics/GetMetricsInStatsd

HashiCorp Vault Metrics https://github.com/hashicorp/vault-consul-monitoring/wiki/Vault-server-metrics

Splunk Metrics http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/Search

What Is Telemetry? How Telemetry Works, Benefits of Telemetry, Challenges, Tutorial, and More
https://stackify.com/telemetry-tutorial/