Docker Security with Splunk

Containers are wildly popular in the IT space due to their portability, speed, and flexibility. One of the current concerns with utilizing containers in a production environment is security.

Docker Inc. (https://www.docker.com/) is one of the most notable companies in the container space and the Center for Internet Security has published a security benchmark for the Docker platform that aims to provide guidance as it relates to Docker security best practices.
https://benchmarks.cisecurity.org/downloads/show-single/?file=docker16.100

Docker has provided the Docker Bench for Security utility (https://github.com/docker/docker-bench-security) that checks for dozens of common best-practices around deploying Docker containers in production. The utility provides excellent feedback about the benchmark compliance of the Docker host but running the utility manually across dozens or hundreds of host is unmanageable and doesn’t provide centralized reporting.

The existing test scripts can output the results of the tests in a human readable format to the console as well as a log file but the format isn’t really great for loggin. We need to manipulate the output of the script to generate a simplified format for logging the results. The script below is used to output the results of the tests to a json file for ingestion into splunk or other logging applications.

#!/bin/bash

# Create timestamp in the format of mm/dd/yyy hh:mm:ss
timestamp=$(date +"%m/%d/%Y %H:%M:%S")
# Set host variable as the system hostname
host=$(hostname)
# Log file location
logfile='dockerbenchsecurity.json'
# Set check variable as the first command line argument
check=$1
# Set check variable as the second command line argument
check_description=$2
# Set check variable as the third command line argument
status=$3
# Set check variable as the fourth command line argument
status_details=$4

# Check if no arguments are supplied
if [ $# -eq 0 ]
  then
    echo "usage: docker-bench-security-logging [docker check] [docker check description] [docker check status] [docker check status description]"
fi

# Create json file
cat < < EOF >> $logfile
{"timestamp" : "$timestamp","host" : "$host","check" : "$check","check_description" : "$check_description","status" : "$status","status_details" : "$status_details" }
EOF

Docker-Bench-Security Tests

The docker-bench-security repository that contains the Docker bench utility contains a test directory that contains all of the tests or checks that are run against the system. The code snippet below is a portion of the modified tests to handle dumping the results to a json file. The example shows that the script for generating the json entries is being called with arguments for the name of the check, the description of the check, the status or result of the check and the details of the status.

# 1.1
check_1_1="1.1  - Create a separate partition for containers"
check=$(echo $check_1_1 | cut -d "-" -f 1)
check_description=$(echo $check_1_1 | cut -d "-" -f 2,3 | sed -e 's/^[ t]*//')
grep /var/lib/docker /etc/fstab >/dev/null 2>&1
if [ $? -eq 0 ]; then
  ./docker-bench-security-logging.sh "$check" "$check_description" "pass"
else
  ./docker-bench-security-logging.sh "$check" "$check_description" "warn"
fi

Json file

The json format was chosen because of it’s ability to be ingested easily into the various different logging tools as well as it’s ease of readability by humans. The default location of the json file is in the directory that the scripts are ran from with the dockerbenchsecurity.json name but it can be changed with the “logfile” variable at the top of the docker-bench-security-logging.sh script.

timestamp: Generated timestamp in the mm/dd/yyyy hh:mm:ss format to support compliance with standard logging formats.

host: The hostname of the Docker host.

check: The number of the CIS benchmark check/test.

check_description: A description of the CIS benchmark check/test.

status: The result of the CIS benchmark check/test.

status_details: The details of the check/test result. This is typically only populates with the “info” status.

{"timestamp" : "03/04/2016 14:08:32","host" : "kgilabdocker02.kgilab.local","check" : "1.17","check_description" : "Audit Docker files and directories - /etc/sysconfig/docker-storage","status" : "info","status_details" : "/etc/sysconfig/docker-storage not found" }
{"timestamp" : "03/04/2016 14:08:33","host" : "kgilabdocker02.kgilab.local","check" : "1.18","check_description" : "Audit Docker files and directories - /etc/default/docker","status" : "info","status_details" : "/etc/default/docker not found" }

Using docker-bench-security-logging

All of the code can be downloaded from github (https://github.com/martezr/docker-bench-security-logging) to integrate into an existing environment.

git clone https://github.com/martezr/docker-bench-security-logging

The script is ran by calling the docker-bench-security-logger.sh script.

sh docker-bench-security-logger.sh

Splunk Integration

Splunk (http://www.splunk.com/) is one of the most popular applications utilized for aggregating and correlating machine data. Splunk is an extremely robust tool providing various features for ingesting the data as well as displaying the data in a meaningful manner.

Splunk Docker

A typical installation of Splunk would be performed on a full Linux machine (physical or virtual) and generally includes several components such as the search head, indexer and forwarder spread across several machines for performance reasons; but in this scenario we just need it to test out manipulating the data from our logs. To quickly and easily provision a Splunk server Docker was chosen as the tool or platform of choice. An excellent image is provided by outcoldman (https://github.com/outcoldman/docker-splunk) and the docker image is hosted on Docker hub for easy access to the image. It is important to note that precaution should always be taken when utilizing an unknown image from Docker hub for security reasons.

The following command can be run to provision the Splunk container for ingesting our data.

docker run --hostname splunk --name splunk -p 8000:8000 -p 9997:9997 --env SPLUNK_ENABLE_LISTEN=9997 -d outcoldman/splunk:6.3.3

Log into the Splunk web interface (http://docker_host_ipaddress:8000) with the username “admin” and the password “changeme”.



The first time splunk is launched a prompt to change the default password is presented. Change the password or select “Skip” to login.



Select “Add Data” to begin the ingestion process.



Since this is simply an example of to integrate Docker with Splunk we’re going to simply upload a copy of the log file for Splunk to ingest. Typically a Splunk forwarder would be installed on the Docker host to forward the logs to the Splunk indexer. Select “upload” to continue.



Click “Select File” to select a file to upload or drag and drop the file onto the provided box.



Once the file has been uploaded click “Next” to continue.



By default Splunk will recognize that the file is a json file and will automatically choose the _json source type. For this example the _json sourcetype is all we need. Click “Next” to continue.

Configure “Host” information, the host property will be automatically set to the hostname of the Docker host in a typical forwarding environment where the logs are being forwardered from the Docker host to the Splunk indexer. Select the Index used for storing the event data and click “Review” to continue.

Click “Submit” to accept the input settings.

Click “Start Searching” to view the indexed data from the log file.

The Search app provides an interface to manipulate the data as desired.

Splunk Dashboard

One of the most useful features of a logging tool like splunk is the ability to create dashboards to aggregate correlated data into a single pane of information. In the screenshot below we’ve utilized some different visualizations such as a pie chart, a table, and single value metrics to display the benchmark test results in an a simple, easy to digest manner. The dashboard displays all the results of the tests as well as the individual test statuses to quickly see how compliant the machine is with the CIS benchmark.

While the dashboard shown displays just a few metrics, there are numerous different metrics that can be extracted from the data being ingested. This post covered integration with Splunk but the log file could easily be ingested into several other tools.

References:

http://stackoverflow.com/questions/12524437/output-json-from-bash-script
https://github.com/outcoldman/docker-splunk
https://github.com/docker/docker-bench-security
http://docs.splunk.com/Documentation/Splunk/6.1.3/Viz/ChartConfigurationReference#Pie_charts