Mitigating Terraform Secrets Exposure

Mitigating Terraform Secrets Exposure

Properly managing secrets (passwords, access keys, etc.) is a critical aspect of any development pipeline given the consequences if those secrets were to be exposed. This is especially necessary from a tool like Terraform that integrates with numerous systems to build out the infrastructure.

Terraform (https://www.terraform.io/) is a very popular Infrastructure as Code (IaC) tool created by HashiCorp. There is a nagging problem with Terraform and it's how it manages secrets both from within Terraform itself and secrets it receives from secret management systems like HashiCorp Vault. Secrets are exposed in plaintext when utilizing Terraform, a practice that most would not expect given the sensitive nature of this information.

There is an open issue on Github that has been open for nearly three years with no progress on a solution.

https://github.com/hashicorp/terraform/issues/516

Hashicorp has provided warnings in their documentation about the current state of secrets management in Terraform and a few proposed mitigation techniques.

https://www.terraform.io/docs/state/sensitive-data.html

With this information we're going to walk through the possible exposure scenarios with a real example using Terraform and a secret stored in HashiCorp Vault.

Vault Integration

Hashicorp Vault is a very popular secrets management product which happens to be made by Hashicorp who also makes Terraform.

https://www.terraform.io/docs/providers/vault/index.html

Below is the warning that is provided about Terraform/Vault integration in which the secrets are displayed in plaintext.

Important Interacting with Vault from Terraform causes any secrets that you read and write to be persisted in both Terraform's state file and in any generated plan files. For any Terraform module that reads or writes Vault secrets, these files should be treated as sensitive and protected accordingly.

Exposure Scenarios

To test the secrets exposure we'll use the simple Terraform configuration below in which we retrieve the secret from Vault and write it to a file on the file system which of course is not secure but an easy way to test.

The secret that we will use is an example password that we might store in Vault to access a system that Terraform needs to interact with.

Password: VAULTSECRETPASSWORD

provider "vault" {}

data "vault_generic_secret" "super_password" {  
  path = "secret/password"
}

resource "local_file" "foo" {  
    content     = "${data.vault_generic_secret.super_password.data["value"]}"
    filename = "${path.module}/tfsecrettest.txt"
}

Terraform Plan: When we run the Terraform Plan command, the credentials are output to the system console in plaintext only when the associated resource needs to be created or the value of secret changes.

Refreshing Terraform state in-memory prior to plan...  
The refreshed state will be used to calculate this plan, but will not be  
persisted to local or remote state storage.

data.vault_generic_secret.super_password: Refreshing state...  
The Terraform execution plan has been generated and is shown below.  
Resources are shown in alphabetical order for quick scanning. Green resources  
will be created (or destroyed and then created if an existing resource  
exists), yellow resources are being changed in-place, and red resources  
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when  
"apply" is called, Terraform can't guarantee this is what will execute.

  + local_file.foo
      content:  "VAULTSECUREPASSWORD"
      filename: "/root/terraformsecret.txt"


Plan: 1 to add, 0 to change, 0 to destroy.  

Terraform Show: When we run the Terraform Show command, the credentials are output to the system console every time the command is run following a successful apply.

data.vault_generic_secret.super_password:  
  id = eac523f6-87c1-996d-8d2b-13f5a1cea757
  data.% = 1
  data.value = VAULTSECUREPASSWORD
  data_json = {"value":"VAULTSECUREPASSWORD"}
  lease_duration = 2764800
  lease_id = 
  lease_renewable = false
  lease_start_time = RFC6669
  path = secret/password
local_file.foo:  
  id = a221e15cdc86c058dd4f51f8573826ca7e33bbcb
  content = VAULTSECUREPASSWORD
  filename = /root/terraformsecret.txt

Terraform Apply: When we run the Terraform Apply command, the credentials are output to the system console in plaintext initially when the associated resource is created and on any changes.

data.vault_generic_secret.super_password: Refreshing state...  
local_file.foo: Creating...  
  content:  "" => "VAULTSECUREPASSWORD"
  filename: "" => "/root/terraformsecret.txt"
local_file.foo: Creation complete (ID: a221e15cdc86c058dd4f51f8573826ca7e33bbcb)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.  

Terraform State file: The secret is stored in plaintext in the Terraform state files.

{
    "version": 3,
    "terraform_version": "0.10.2",
    "serial": 10,
    "lineage": "0aac32a1-aff7-45e2-a031-eea2c0a4aa4c",
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {
                "data.vault_generic_secret.super_password": {
                    "type": "vault_generic_secret",
                    "depends_on": [],
                    "primary": {
                        "id": "eac523f6-87c1-996d-8d2b-13f5a1cea757",
                        "attributes": {
                            "data.%": "1",
                            "data.value": "VAULTSECUREPASSWORD",
                            "data_json": "{\"value\":\"VAULTSECUREPASSWORD\"}",
                            "id": "eac523f6-87c1-996d-8d2b-13f5a1cea757",
                            "lease_duration": "2764800",
                            "lease_id": "",
                            "lease_renewable": "false",
                            "lease_start_time": "RFC6669",
                            "path": "secret/password"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": ""
                },
                "local_file.foo": {
                    "type": "local_file",
                    "depends_on": [
                        "data.vault_generic_secret.super_password"
                    ],
                    "primary": {
                        "id": "a221e15cdc86c058dd4f51f8573826ca7e33bbcb",
                        "attributes": {
                            "content": "VAULTSECUREPASSWORD",
                            "filename": "/root/terraformsecret.txt",
                            "id": "a221e15cdc86c058dd4f51f8573826ca7e33bbcb"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": ""
                }
            },
            "depends_on": []
        }
    ]
}

Now that we've looked the possible exposure scenarios we'll walk through ways that we can mitigate them.

Mitigation Techniques

The following are techniques that can be used to mitigate the possible exposure of secrets to unintended parties.

General Mitigation Techniques

  • All state files should be stored in a secure and ideally encrypted manner such as remotely in an encrypted S3 bucket.
  • Ensure that communication from Terraform to all resource providers is encrypted.

AWS S3 Bucket for State Management

  • The S3 bucket should be encrypted.
  • Access to the bucket should be restricted to only those that need access.
  • Access logging should be configured to provide an audit trail of access to the bucket.

CI/CD Pipeline Integration

  • Access to Terraform related Jenkins jobs should be restricted to only those that need access with some form of role based access control.
  • Access to the Jenkins job log files should be restricted to only those that need access or possibly purged from the Jenkins master if not needed.

Developer Terminal Screen

  • Developers should close any terminal sessions in which Terraform was run as soon as possible since the password will be in the buffer.

There are other tools and scripts that have been created by members of the community primarily focused on redacting secrets from the state files. The effectiveness of these solutions has not been validated and they are being posted simply to provide additional information about the way in which others have attempted to solve this problem.

  • Terrahelp: A tool written in GO that integrates with HashiCorp Vault to provide both file and inline encryption of Terraform state files.
  • Terraformectomy: A shell script for redacting aws secrets from Terraform state files.
  • Terraform IAM Redact Script: Another shell script for redacting AWS secrets from Terraform state files.

References

Terrahelp Blog Post
https://opencredo.com/securing-terraform-state-with-vault/

Terrahelp Github Page
https://github.com/opencredo/terrahelp

Terraformectomy Github Page
https://github.com/tgjamin/terraformectomy

Terraform IAM Redact Script Github Page https://github.com/swoodford/aws/blob/master/terraform-redact-iam-secrets.sh

Subscribe to