Serverless InSpec using Native Ruby on AWS Lambda

InSpec is a compliance as code tool that helps organizations perform compliance checks in an automated fashion. Last year I detailed creating an AWS Lambda function that ran an InSpec scan by compiling a version of Ruby and then making that available to the Lambda function to run InSpec since it is written in Ruby. Toward the end of last year I was excited to hear that AWS was officially supporting Ruby as a native programming language for Lambda functions (https://aws.amazon.com/blogs/compute/announcing-ruby-support-for-aws-lambda/).

All the code referenced in this post can be found in this github repository (https://github.com/martezr/serverless-inspec).

Serverless Framework

With the previous iteration of serverless InSpec I created a cloudformation template that handled a lot of the heavy lifting to create the lambda function but even then there a number of tasks that weren’t automated. As I was learning how AWS Layers worked with Ruby I came across a blog post that covers how to use the ServerLess Framework (https://serverless.com/) to handle pretty much all of the dirty work for me (https://blog.francium.tech/deploying-ruby-gems-with-native-extensions-on-aws-lambda-using-the-serverless-toolkit-9079e34db2ab).

The result is a single yaml file that declares the Lambda function deployment and all of it’s associated pieces (IAM role, Lambda Layer, Log Group, etc.). The ServerLess Framework uses a Cloud Formation stack to manage and update the deployment.

service: inspec-serverless

provider:
  name: aws
  runtime: ruby2.5
  region: us-east-1
  iamManagedPolicies:
    - 'arn:aws:iam::aws:policy/ReadOnlyAccess'
  environment:
    HOME: /tmp
    INSPEC_PROFILE: "https://github.com/martezr/serverless-inspec-profile"

functions:
  inspec_scan:
    handler: handler.inspec_scan
    layers:
      - {Ref: InspecLambdaLayer }

layers:
  inspec:
    path: layer

The ServerLess Framework makes the deployment extremely simple but we still need to get everything setup like compiling the gems for the Lambda layer and installing the Serverless Framework. To simplify the build process I’ve created a “builder” AWS EC2 instance to do all the buidling and deployment.

Builder

The builder is an AWS EC2 instance that is used for the two primary tasks needed to be accomplished to deploy the Lambda function. The first one is installing the serverless framework and the second is building all of the gems required for InSpec. A docker container is used to build all of the gems.

AWS EC2 Instance Terraform

The github repository contains Terraform code for provisioning an EC2 instance for the builder. The Terraform code creates an IAM role with the following permisions to allow the the serverless framework running on the EC2 instance to provision the Lambda function.

data "aws_iam_policy_document" "inspec-serverless" {

  statement {
    sid = "LambdaAccess"
    effect = "Allow"
    actions = [
      "lambda:*"
    ],
    resources = ["*"]
  }

  statement {
    sid = "IAMAccess"
    effect = "Allow"
    actions = [
      "iam:AttachRolePolicy",
      "iam:CreateRole",
      "iam:DeleteRole",
      "iam:DeleteRolePolicy",
      "iam:DetachRolePolicy",
      "iam:GetRole",
      "iam:PassRole",
      "iam:PutRolePolicy"
    ],
    resources = ["*"]
  }

  statement {
    sid = "S3Access"
    effect = "Allow"
    actions = [
      "s3:CreateBucket",
      "s3:DeleteBucket",
      "s3:DeleteBucketPolicy",
      "s3:DeleteObject",
      "s3:DeleteObjectVersion",
      "s3:GetObject",
      "s3:GetObjectVersion",
      "s3:ListAllMyBuckets",
      "s3:ListBucket",
      "s3:PutBucketNotification",
      "s3:PutBucketPolicy",
      "s3:PutBucketTagging",
      "s3:PutBucketWebsite",
      "s3:PutEncryptionConfiguration",
      "s3:PutObject"
    ],
    resources = ["*"]
  }

  statement {
    sid = "CloudFormationAccess"
    effect = "Allow"
    actions = [
      "cloudformation:CancelUpdateStack",
      "cloudformation:ContinueUpdateRollback",
      "cloudformation:CreateChangeSet",
      "cloudformation:CreateStack",
      "cloudformation:CreateUploadBucket",
      "cloudformation:DeleteStack",
      "cloudformation:Describe*",
      "cloudformation:EstimateTemplateCost",
      "cloudformation:ExecuteChangeSet",
      "cloudformation:Get*",
      "cloudformation:List*",
      "cloudformation:PreviewStackUpdate",
      "cloudformation:UpdateStack",
      "cloudformation:UpdateTerminationProtection",
      "cloudformation:ValidateTemplate"
    ],
    resources = ["*"]
  }

  statement {
    sid = "LogGroupAccess"
    effect = "Allow"
    actions = [
      "logs:CreateLogGroup",
      "logs:DeleteLogGroup",
      "logs:DescribeLogGroups",
      "logs:DescribeLogStreams",
      "logs:FilterLogEvents",
      "logs:GetLogEvents"
    ]
    resources = ["*"]
  }
}

Post provisioning steps

The following steps are automatically performed if the Terraform in the github repo is used to standup the builder instance.

Install the Serverless Framework

Once the EC2 instance is provisioned the installation of the Serverless framework is extremely simple and just required installing node.js.

# Setup node.js repo
curl -sL https://rpm.nodesource.com/setup_10.x | sudo bash -

# Install node.js
sudo yum -y install nodejs

# Installing the serverless cli
npm install -g serverless

Build the ruby gems

The next step is to build the ruby gems and we’ll use a docker container to simplify this process.

# Install docker
sudo yum -y install docker

# Start docker
systemctl start docker && systemctl enable docker

Once Docker has been installed and started on the EC2 instance we need to go through the process of building the InSpec gems in a folder structure that works with what the AWS Lambda function is expecting. We need to create a Gemfile that will be used by bundler to build the gems.

source 'https://rubygems.org'

gem 'inspec'

Once the Gemfile has been created we just need to run the lambci/lambda docker container to build the gems. The code below is based upon the folder structure in the github repo (https://github.com/martezr/serverless-inspec) of the example.

#!/bin/bash

# Cleanup ruby directory
rm -Rf ruby

# Build ruby gems
docker run --rm -it -v $PWD:/var/gem_build -w /var/gem_build lambci/lambda:build-ruby2.5 bundle install --path=.

# Copy gems
mkdir -p code/layer/ruby/gems/2.5.0/
cp -Rf ruby/2.5.0/* code/layer/ruby/gems/2.5.0/

Lambda Deployment

With everything setup and ready to go we just need to deploy the Lambda function using the serverless command sls deploy which will package our code along with layer and upload them to an S3 bucket. Once they’ve been uploaded the lambda function and layer will be created.

Change directory to /serverless-inspec/code and modify the serverless.yml file by changing the INSPEC_PROFILE to the desired InSpec profile to run and the S3_DATA_BUCKET to the S3 bucket where the JSON output will be stored. The ruby code currently only supports scanning AWS and doesn’t not support scanning instances at this point in time.

service: inspec-serverless

provider:
  name: aws
  runtime: ruby2.5
  region: us-east-1
  stage: prod
  iamManagedPolicies:
    - 'arn:aws:iam::aws:policy/ReadOnlyAccess'
  environment:
    HOME: /tmp
    INSPEC_PROFILE: "https://github.com/martezr/serverless-inspec-profile"
    S3_DATA_BUCKET: mreed-bucket

functions:
  inspec_scan:
    handler: handler.inspec_scan
    layers:
      - {Ref: InspecLambdaLayer }

layers:
  inspec:
    path: layer

Once the serverless.yml file has been updated, run the sls deploy command to deploy the lambda function.

[[email protected] code]# sls deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
.....
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service inspec-serverless.zip file to S3 (372 B)...
Serverless: Uploading service inspec.zip file to S3 (37.66 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
...................
Serverless: Stack update finished...
Service Information
service: inspec-serverless
stage: dev
region: us-east-1
stack: inspec-serverless-dev
resources: 6
api keys:
  None
endpoints:
  None
functions:
  inspec_scan: inspec-serverless-dev-inspec_scan
layers:
  inspec: arn:aws:lambda:us-east-1:684882843674:layer:inspec:5

The Lambda function should be deployed into the AWS account and can be invoked via the GUI as well as the AWS CLI. The screenshot below shows what the lambda function execution should look like.

References

Serverless Framework (https://serverless.com/)

Serverless Framework IAM Permissions (https://gist.github.com/ServerlessBot/7618156b8671840a539f405dea2704c8)