In the first post (https://www.greenreedtech.com/aws-compliance-as-code-with-chef-inspec-using-aws-lambda/) we looked at getting Chef InSpec to run in an AWS Lambda function and in this post we extended that functionality by writing the the output to a JSON file that gets stored in S3.
The code used in this post can be found on Github - https://github.com/martezr/serverless-inspec
Generate JSON File
The standard output generated by InSpec is in a human readable format that is absolutely perfect for humans, but we want the output to be in a machine readable format like JSON.
Support Human and Machine readable output
The first thing we need to do is update the InSpec command to output to JSON but we also want to be able to look at the Lambda Function output and see the human readable format.
To allow us to enable both use cases we just need to leverage two reporters for the InSpec command output. The first reporter is the cli which is the human readable format and then we'll write the JSON format to a JSON file to be used later in the Lambda function.
cmd = '/var/task/customruby/bin/inspec exec --reporter cli json:/tmp/inspec_output.json --no-color ' + github_repo + ' -t aws://' + aws_region
The updated command outputs the human readable format to the Lambda console as well as the cloudwatch logs associated with the Lambda function.
Generate a timestamped JSON file
Now that our InSpec command is generating JSON output we need to add it to a timestamped file for uniqueness when it's uploaded to S3. Our first step is to load the JSON from the temporary inspec_output file created by the InSpec command and then use that data as the input for our final JSON file that will be upload to S3.
""" Create JSON File """ data = json.load(open('/tmp/inspec_output.json')) filename,outputfile = generate_json('aws',data)
The code above references the function below which handles the timestamp generation and writing the JSON file to disk in the /tmp directory.
def generate_json(service_type,output): """ Generate the JSON filename """ filename = 'inspec-' + service_type + '-' + time.strftime("%Y%m%d-%H%M%S") + '.json' outputfile = '/tmp/' + filename f = open(outputfile,"wb") f.write(output) return filename,outputfile
Upload to S3
With the JSON file created we can now upload it to S3 to be ingested by our logging solution. The destination S3 bucket for log storage is an environment variable for the Lambda function. The IAM role associated with the Lambda function must have permission to write/put the JSON file to the S3 bucket specified.
""" Upload InSpec output to S3 Bucket """ s3 = boto3.client('s3') s3.upload_file(outputfile, s3_bucket, filename)
Once the Lambda function completes we should see a new JSON file added to the S3 bucket specified in our environment variables.