Puppet DevOpsSec with VMware vRealize Orchestrator

A common concern among all system administrators is how to secure their network systems and ensure they are compliant with the company’s security policies. Configuration management (CM) tools like Puppet or Chef are now being leveraged heavily to ensure nodes are compliant with the security configuration baseline that has been defined by the company. Those tools are excellent in ensuring that nodes are in a desired state as it relates to their security configuration. But what happens if a node is provisioned and for some reason it doesn’t have a Puppet or Chef agent installed on it? How do we ensure that node has the correct SSH server configuration or appropriate openssl patches? This non-compliant and essentially non-existent node severely compromises the entire security posture of the entire network.

How do you secure what you can’t see?

Nodes that are “invisible” to our configuration management tool are likely not rogue nodes that some hacker stood up but VMs that were created by an administrator that sort of just slipped into production.

In this post we’re going to walk through how to utilize VMware vRealize Orchestrator to scan our VMware infrastructure to check if our VMs have a Puppet agent installed on them.

Workflow

Now that we’ve got an idea of what we want to accomplish let’s walk through how we’re going to do that at a high level.

  • Check all VMs in the environment for the ‘No-Puppet’ vSphere tag
  • Scan all VMs in the environment without the ‘No-Puppet’ vSphere tag
  • Send an email to administrators with the list of VMs that Puppet is not installed on

Query all VMs without the ‘No-Puppet’ vSphere tag

The first step in the workflow is to compile a list of VMs in the vSphere environment that don’t have the ‘No-Puppet’ vSphere tag. We’re utilizing vSphere to denote virtual machines that we know a Puppet agent will never be installed on so there is not point scanning it. This step required quite a bit of logic as I was unable to find a built-in method or action to do what I needed done.

// Define No Puppet vSphere Tag
var noPuppetTag = 'No-Puppet'

// Get all VMs in cluster
vms = System.getModule("com.vmware.library.vc.cluster").getAllVMs  
OfCluster(cluster)

// Set the VAPI endpoint to the first endpoint returned
var endpoints = VAPIManager.getAllEndpoints();  
var endpoint = endpoints[0]

if (endpoint == null) {  
  throw "'endpoint' parameter should not be null";
}

// Fetch tags
var client = endpoint.client();

var tagging = new com_vmware_cis_tagging_tag(client);  
var tags = tagging.list();


var attachedTags = new com_vmware_cis_tagging_tag__association(client) ;

for each (tag in tags){  
  if (tagging.get(tag).name == noPuppetTag){
    appliedVMs = attachedTags.list_attached_objects(tag)
  }
}

// Fetch the VM ID of tagged VMs
taggedVMs = [];

for each (vm in appliedVMs){  
  taggedVMs.push(vm.id);
  System.log(vm.id);
}

// Fetch tagged VMs in the selected cluster
checkVMs = [];

for each (vm in vms){  
  var tagStatus = false;
  for each (vmTag in taggedVMs){
    if (vm.id.toString() == vmTag){
      tagStatus = true;
      System.log("Tagged VMs " + vm.name);
    }
  }

  if (tagStatus != true){
    checkVMs.push(vm);
    System.log("Non-Tagged VMs " + vm.name);
  }
}

propsArray = new Array();

for each (vm in checkVMs){  
 var prop = new Properties();
 prop.put("vm", vm);
 propsArray.push(prop);
}
Puppet Compliance Scan

The next step in our workflow is to scan or check all the VMs in our vSphere environment without the ‘No-Puppet’ tag for the installation of the Puppet agent.

The dilemma that I faced when implementing the scanning was that in a vSphere environment with 100s or 1000s of VMs how do I run these scans and it not take forever. This forced me to learn about parallel workflow execution which triggers the specified workflow for all of the VMs in the array in parallel.

Now with parallel execution figured out I needed to build a workflow to actually determine whether Puppet is installed on the guest os.

  • Check if the VM is Windows or Linux:
    Since our environment may contain both Linux and Windows virtual machines we need to branch appropriately.
  • Check if the Puppet agent is installed:
    While not always effective this workflow checks for the existence of the Puppet folder structure.
  • Output the status to a variable:
    Take the status from the agent check workflow and outputs a status boolean

Generate Compliance Report

The final step in the workflow is to take the output from all of the individual compliance scan workflows that were ran and compile it all into a report.

var content = '';

// Create content header
content += "VMs missing Puppet agent" + "n"  
content += "------------------------" + "n"

for each (token in worfklowTokens){  
  var inputProps = token.getInputParameters();
  var props = token.getOutputParameters();
  var machine;

  for each (var key in inputProps.keys) {
    machine = inputProps.get(key);
    System.log("Key: "+key+" Value: "+ inputProps.get(key));
  } 

  for each (var key in props.keys) {
    System.log("Key: "+key+" Value: "+props.get(key));
    if (props.get(key) == false){
        content += machine.name + "n";
    }
  }
}
System.log(content);  

Additional Concepts

There are a number of additional steps that we could add to our automations to really push things toward auto-remediation to ensure that all VMware VMs are compliant.

  • Assign a ‘Puppet Quarantine’ NSX security tag to the VM to restrict inbound and outbound access to services that allow the installation of the Puppet agent.
  • Automatically install a Puppet agent with a ‘secure baseline’ role that ensure that the node complies with the corporate security policies.
  • Email an administrator and require them to properly classify the node before the ‘Puppet Quarantine’ NSX security tag is removed from the VM.

References

vRO Parallel Workflows
http://pubs.vmware.com/orchestrator-72/index.jsp#com.vmware.vrealize.orchestrator-dev.doc/GUID-6FBD1D08-252A-4A43-A6D0-0938783E3D44.html?resultof=%2522%2570%2561%2572%2561%256c%256c%2565%256c%2522%2520