Advanced VMware vSphere Template Orchestration

Template management is a critical facet of infrastructure management and traditionally one of the more challenging operations there is. The advent of tools like HashiCorp Packer have provided administrators with the ability declaratively automate the template creation process.

Automation is great but in most organizations a single piece of automation is part of a larger process that includes tickets and potentially other hand offs.

In this example we’ll walk through creating a process for creating VMware vSphere machine templates in a fully automated fashion with deep integrations with other tools for documenting and testing the created template.

The code for this example can be found in the following GitHub repositories.

The StackStorm pack that contains the orchestration workflow: https://github.com/martezr/stackstorm-grtlab

The Terraform and Packer code: https://github.com/martezr/orchestration-demos

Tools

The orchestration detailed in this example contains a number of tools that are covered below.

StackStorm

StackStorm is an open source orchestration tool with numerous integrations with third-party tools for simplifying the creation of complex orchestration workflows. StackStorm will drive the orchestration in this example and handle all of the hand offs.

Github

Github is used for storing all of the code used in this example and is where StackStorm will fetch code from.

ServiceNow

ServiceNow is an ITSM or IT Service Management platform that we’ll use for incident management and as a CMDB in our example.

Jenkins

Jenkins is a continuous integration (CI) server typically used for testing and building code. In our example Jenkins will be used for validating our deployed code. This shows how the workflow could incorporate tests that may already exist that are created by the applications developers or QA team. This helps to reduce potential issues created by base template updates

ServerSpec

ServerSpec is a Ruby based testing framework that I being used for acceptance testing in our example. A few simple tests are being run but it shows how the integration can provide tremendous value.

HashiCorp Packer

Packer is a tool for defining a template in a declarative manner via a JSON file.

HashiCorp Terraform

Terraform is used to provision the virtual machine that is used for validating that the new template doesn’t break any downstream applications.

Stages

The high level stages of the orchestration displayed in the image below are detailed below along with the StackStorm code for each stage.

Create Incident

This first stage of the workflow is to create an incident in ServiceNow for the new template. The goal of this stage is integrating with ServiceNow to provide a record of what is being done as part of the automated template creation process. The StackStorm code below shows that we’re creating a new record in the incident table and providing some metadata about the incident such as the description and the submitter.

Once the record has been created in the incident table we are storing the sysid of the ServiceNow incident in the incident_id variable for use later when we resolve the incident. Additionally we’re specifying the next stage of the workflow which cleaning up any existing clones of our code git repo.

create_incident:
  action: servicenow.create_record
  input:
    table: "incident"
    payload: {"short_description":"CentOS 7 Monthly Image Update","category":"software","description":"Updating CentOS 7 template","caller_id":"stackstorm"}
  next:
    - when: <% succeeded() %>
      publish: incident_id=<% result().result.sys_id %>
      do: cleanup_git_repo

The screenshot below displays a new incident created by the orchestration to track the work performed during this process and align with change management processes.

Clone Github Repo

The second stage of the workflow cleans up any existing copy of the Github repository and clones the Github repository that contains the Packer template code as well as the Terraform code used for testing the newly created template.

cleanup_git_repo:
  action: linux.rm
  input:
    target: /stackstorm/orchestration-demos
    recursive: True
    sudo: True
    hosts: localhost
  next:
    - when: <% succeeded() %>
      do: clone_build_repo

Once the local copy of the repo is removed then a clone operation is performed.

clone_build_repo:
  action: git.clone
  input:
    source: https://github.com/martezr/orchestration-demos.git
    destination: /stackstorm/orchestration-demos
    hosts: localhost
  next:
    - when: <% succeeded() %>
      do: create_template

Build Template

The third stage of the workflow builds the VMware vSphere template from an ISO image using HashiCorp Packer and the Jetbrains vSphere Packer plugin.

The Packer JSON file is hosted in the cloned GitHub repository and the variable file that contains sensitive information is stored on the filesystem of the StackStorm host. This data could be retrieved in a more secure fashion such as retrieved from HashiCorp Vault or StackStorm’s encrypted data storage.

create_template:
  action: packer.build
  input:
    packerfile: /stackstorm/orchestration-demos/packer/centos7base.json
    variables_file: /stackstorm/packer-vars.json
  next:
    - when: <% succeeded() %>
      do: provision_test_stack

Provision Test Stack

The fourth stage of the workflow provisions a VMware vSphere virtual machine based on the previously created template and configures it using Puppet via the Terraform Puppet Provisioner.

In a real world example this might be an entire application stack owned by another team that consumes the template. This provides tighter integration to help prevent breaking changes from being propogated.

provision_test_stack:
  action: terraform.pipeline
  input:
    variable_files: ["/stackstorm/vsphere.auto.tfvars"]
    plan_path: /stackstorm/orchestration-demos/terraform/teststack
    backend: {"path":"/stackstorm/terraform.tfstate"}
  next:
    - when: <% succeeded() %>
      do: output_test_stack_info

Validate Test Stack

The fifth stage of the workflow is run some validation tests against the stack. These tests might be tests provided by teams that own applications that are built on the base template. In this example StackStorm executes a defined job in Jenkins that runs some ServerSpec tests.

validate_test_stack:
  action: jenkins.build_job_wait
  input:
    project: "validate_teststack"
    parameters: {"target":"<% ctx().ip_address %>"}
  next:
    - when: <% succeeded() %>
      do: destroy_test_stack
    - when: <% failed() %>
      do: destroy_test_stack

The screenshot below displays the console output of the Jenkins job that was run to perform the validation of the test virtual machine provisioned by Terraform.

Destroy Test Stack

The sixth stage of the workflow is to tear down the virtual machine used for testing the updates to the base template. Since Terraform was used to provision the virtual machine we’ll just run a Terraform Destroy to tear it down.

destroy_test_stack:
  action: terraform.destroy
  input:
    variable_files: ["/stackstorm/vsphere.auto.tfvars"]
    plan_path: /stackstorm/orchestration-demos/terraform/teststack
    state_file_path: "/stackstorm/terraform.tfstate"
  next:
    - when: <% succeeded() %>
      do: create_cmdb_record

Fetch Template Metadata

The seventh stage of the workflow is to fetch the metadata for the template such as the hardware specifications to input them into the CMDB in ServiceNow. The Packer JSON is just being outputted via the cat command but a custom script for parsing file or fetching the data directly from vCenter would provide a more robust solution. The output is being stored in the template_data variable.

get_template_data:
  action: core.local
  input:
    cmd: cat /stackstorm/orchestration-demos/packer/centos7base.json
  next:
    - when: <% succeeded() %>
      publish: template_data=<% result().stdout.builders[0] %>
      do: create_cmdb_record

Create CMDB Entry

The eighth stage of the workflow is creating a record in the CMDB for the new vSphere template. The payload contains metadata about the template that we defined in the previous stage and made available to this stage with the publish declaration.

create_cmdb_record:
  action: servicenow.create_record
  input:
    payload: {"name":"<% ctx().template_name %>","description":"CentOS 7 Demo Template","guest_os":"<% ctx().template_data.guest_os_type %>","cpus":"<% ctx().template_data.CPUs %>","memory":"<% ctx().template_data.RAM %>","disk_size":"<% ctx().template_data.disk_size %>","object_id":"1"}
    table: "cmdb_ci_vmware_template"
  next:
    - when: <% succeeded() %>
      do: close_incident

The screenshot below displays the CMDB record for the template created by the orchestration.

Close Incident

The nineth and final stage of the workflow is to close or resolve the incident in ServiceNow. This stage closes the loop on the change in ServiceNow with the template successfully being created. In the example we’re passing the incident_id variable that was returned from the very first stage in the workflow along with providing the state of “6” which is resolved and the required fields of code and notes.

close_incident:
  action: servicenow.update
  input:
    table: "incident"
    sysid: <% ctx().incident_id %>
    payload: {"state":"6", "close_code": "Solved (Permanently)", "close_notes": "Created  <% ctx().template_name %> template"}

The screenshot below displays the incident created by the first stage of the orchestration now in a “resolved” state due to closing out the incident as part of the orchestration.

This example could be extended much further to include additional systems and stages.

This example shows just some of the possibilities of creating advanced orchestration with a tool like StackStorm.