HashiCorp Vault Metrics with Splunk
What is telemetry?
I’ve heard the word telemetry a few times in my career but mainly when going to the hospital where they have a department named as such. The department’s job is to monitor the data or metrics that are being obtained via connected sensors. The other situation I’ve run across the word telemetry is on software product installs that wanted to send product data back to the manufacturer for data collection purposes.
In this case telemetry refers the collection of insightful metrics from a HashiCorp Vault deployment which will be shipped to a centralized platform such as Splunk.
What is StatsD?
StatsD is a protocol sending application metrics to a centralized platform such as Splunk for ingestion. In this example we’re going to use Splunk as our StatsD server for collecting information about our Vault deployment. HashiCorp Vault has a number of different options for a StatsD server.
Vault Configuration
Add the following block to the Vault configuration file to send telemetry data to our Splunk server.
telemetry {
statsd_address = "splunk01.grt.local:8125"
}
Splunk Configuration
Enable a StatsD Input If you don’t have a StatsD input enabled:
- Go to Settings > Data Inputs in the Splunk bar
- Under Local inputs, click Add new next to UDP
- Click Select Source Type, then select Metrics > statsd
- Select an existing metrics Index or create a new one
Enter the number of the port you are using for StatsD 8125 is the standard port for StatsD. Click Next to proceed to the next menu.
Click Select Source Type, then select Metrics > statsd
Create a new metrics index for Vault telemetry data ingestion. Click “Save” and then “Review” to proceed to the confirmation dialog.
Click “Submit” to create the input.
The data input has been configured and we should now be receiving metrics from Vault.
The search command below can be used to view the metric names that are being ingested.
| mcatalog values(metric_name) where index="vault_metrics"
Now that we have the names of metrics that are available to us we can access the data and create a nice graph similar to the one below by running the following search command. The search below fetches the allocated memory bytes for the Vault process.
| mstats avg(_value) WHERE index="vault_metrics" AND metric_name=vault.vault.runtime.alloc_bytes span=30s
References
HashiCorp Vault Telemetry Overview https://www.vaultproject.io/docs/internals/telemetry.html
HashiCorp Vault Telemetry Configuration https://www.vaultproject.io/docs/configuration/telemetry.html
Splunk StatsD Configuration http://docs.splunk.com/Documentation/Splunk/7.1.1/Metrics/GetMetricsInStatsd
HashiCorp Vault Metrics https://github.com/hashicorp/vault-consul-monitoring/wiki/Vault-server-metrics
Splunk Metrics http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/Search
What Is Telemetry? How Telemetry Works, Benefits of Telemetry, Challenges, Tutorial, and More
https://stackify.com/telemetry-tutorial/