AWS Compliance as Code with Chef InSpec using AWS Lambda

InSpec 2.0 added builtin support for scanning public cloud resources in AWS and Azure (https://blog.chef.io/2018/02/20/announcing-inspec-2-0/). With the addition of these new features it only made sense to be able to perform scanning of the cloud environment from within the environment and a serverless option like lambda made a lot of sense.

Advantages

  • Simplified IAM permission model for multiple AWS accounts
  • All the benefits of serverless architecture
  • Credential/Access key management is non-existant since an IAM role is used to access AWS

Problem

InSpec is written in Ruby which created an interesting problem given that AWS has not added official support for Ruby as a language that AWS Lambda can utilize.

Solution

There are a number of solutions such as using JRuby, Traveling Ruby and others but the most effective solution was covered in the post below.

https://pprakash.me/tech/2015/11/29/executing-ruby-code-in-aws-lambda/

The solution details compiling a version of Ruby from source and installing the desired gems in a Amazon Linux instance then bundling it up to run in the Lambda function. While this solution would work in most situations I found that the version of openssl in the Amazon Linux instance is not the same that is available on the Lambda functions.

To overcome this challenge an environment that matched that of AWS lambda was required to compile ruby with the correct openssl version. The LambCI Docker container (https://github.com/lambci/docker-lambda) was exactly the environment that was needed to properly compile ruby with the correct version of openssl.

Below is the Dockerfile used to compile ruby 2.5.1, install InSpec and package up the deployment files needed to run it on AWS lambda.

All the code in this post can be found in a github repo. https://github.com/martezr/serverless-inspec

FROM lambci/lambda:build-python3.6  
ENV RUBY_VERSION 2.5.1  
ENV GEM_INSTALLED inspec

RUN yum -y install zlib-devel gcc zip

RUN curl -sL https://cache.ruby-lang.org/pub/ruby/2.5/ruby-$RUBY_VERSION.tar.gz | tar -zxv

WORKDIR ruby-$RUBY_VERSION

RUN ./configure --prefix=/var/task/customruby --disable-werror --disable-largefile --disable-install-doc --disable-install-rdoc --disable-install-capi --without-gmp --without-valgrind  
RUN make  
RUN make install  
RUN /var/task/customruby/bin/gem install $GEM_INSTALLED

COPY lambda.py /var/task  
WORKDIR /var/task

# Create a lambda deployment package
RUN zip -r lambda.zip customruby/ lambda.py  

In addition to the Dockerfile a build script is used to copy the deployment package from the docker container to the local host to allow uploading to the S3 bucket for running from Lambda.

#!/bin/bash

# Build Docker Image
docker build -t rubylambda .

# Run Docker Images
docker run --name rubylambda rubylambda bash

# Copy Lambda zip file
rm -Rf lambda.zip  
docker cp rubylambda:/var/task/lambda.zip .

# Cleanup Docker Containers
docker rm rubylambda  

Lambda Configuration

Since AWS Lambda doesn’t support natively running Ruby we need to use a supported language to call our ruby code and in this case Python is the language of choice. Below is the snippet of code that executes InSpec when the Lambda function is triggered.

import time  
import os  
from subprocess import Popen, PIPE, STDOUT

def lambda_handler(event, context):  
    github_repo = os.environ['GITHUB_REPO']

    # Specify region to scan
    if os.environ['INSPEC_AWS_REGION']:
        aws_region = os.environ['INSPEC_AWS_REGION']
    else:
        aws_region = os.environ['AWS_REGION']

    cmd = '/var/task/customruby/bin/inspec exec --no-color ' + github_repo + ' -t aws://' + aws_region
    p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
    output = p.stdout.read()
    print(output.decode('utf-8'))
    return

if __name__ == '__main__':  
    lambda_handler('event', 'handler')

IAM Permissions

The Lambda function requires permissions to scan/inspect the AWS environment as well as write logs of executions to CloudWatch. The two AWS managed policies provide the necessary IAM permissions to run InSpec against an AWS environment.

  • ReadOnlyAccess (AWS Managed Policy)
  • AWSLambdaBasicExecutionRole (AWS Managed Policy)
    • logs:CreateLogGroup
    • logs:CreateLogStream
    • logs:PutLogEvents

Environment Variables

The Lambda function requires a number of environment variables to support manipulating the code on the fly without making changes to the underlying code.

  • HOME (Required) – The working directory for the Lambda function. This must be set to /tmp to allow InSpec to write to the directory.
  • GITHUB_REPO – The Github repository of the InSpec profile to execute
  • INSPEC_AWS_REGION – The AWS region that InSpec will scan

CloudFormation Template

The CloudFormation template below deploys the following components to bootstrap an AWS account with the InSpec lambda function.

  • IAM Role: IAM role for running the lambda function
  • Lambda Function: Lambda function for running InSpec against the AWS environment
  • S3 Bucket: An S3 bucket for storing the Lambda code

    AWSTemplateFormatVersion: 2010-09-09
    Description: ‘CloudFormation Stack to deploy serverless InSpec’
    Parameters:
    GithubRepo:
    Type: String
    Default: https://github.com/martezr/serverless-inspec-profile.git
    Description: Enter the Github repository that contains the InSpec AWS profile.
    InSpecAWSRegion:
    Type: String
    Default: us-east-1
    Description: Enter the AWS region for InSpec to scan.
    S3BucketName:
    Type: String
    Description: Enter the name of the S3 bucket where the lambda deployment package is stored.
    Resources:
    InSpecLambdaRole:
    Type: ‘AWS::IAM::Role’
    Properties:
    RoleName: ‘InSpecLambdaRole’
    AssumeRolePolicyDocument:
    Version: 2012-10-17
    Statement:
    – Effect: Allow
    Principal:
    Service:
    – lambda.amazonaws.com
    Action:
    – ‘sts:AssumeRole’
    ManagedPolicyArns:
    – ‘arn:aws:iam::aws:policy/ReadOnlyAccess’
    – ‘arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole’
    InSpecLambda:
    Type: ‘AWS::Lambda::Function’
    Properties:
    FunctionName: InSpecLambda
    Description: InSpec Compliance Lambda
    Timeout: ’45’
    Environment:
    Variables:
    HOME: /tmp
    GITHUB_REPO: !Ref GithubRepo
    INSPEC_AWS_REGION: !Ref InSpecAWSRegion
    Handler: lambda.lambda_handler
    Runtime: python3.6
    Role:
    ‘Fn::GetAtt’:
    – InSpecLambdaRole
    – Arn
    Code:
    S3Bucket: !Ref S3BucketName
    S3Key: lambda.zip
    InSpecLambdaS3Bucket:
    Type: AWS::S3::Bucket
    BucketName: !Ref S3BucketName

References

GitHub Serverless-InSpec
https://github.com/martezr/serverless-inspec

GitHub Serverless-InSpec Example Profile
https://github.com/martezr/serverless-inspec-profile

InSpec Tutorial Part #5
http://www.anniehedgie.com/inspec-basics-5