Terraform Secret Management

Where to store secrets

If you store secrets in code, anyone/anything with access to the code can access the system protected by the secret. Think about the fact that your Terraform code is often stored in multiple places at once, such as, for example, developer workstations, CI/CD servers, and the VCS. There is no control over the secret or audit data to back up who or what has access.

You can secure secrets using encryption and save the secret as files in the VCS. But you still need to consider how to manage the encryption, store the private key, and manage to grant access to users, which can easily lead to the perception of security where in fact, your data is at risk as the private key is stored insecurely for ease of access, for example. Security can defeat itself if it’s too cumbersome to use in the real world.

The alternative to storing secrets as files and managing access controls is to offload the key encryption management to a provider and use a managed service instead. The secrets are stored in an encrypted system in transit and at rest, and it’s far easier to control who has access to the audit trail and change the password on a rotation. Different methods can also be used to access systems, so they can be used by developers and automation as needed.

There are multiple solutions, some are listed below, which support UI and API access making them versatile options to safely store ands retrieve secrets.

• HashiCorp Vault
• AWS Secrets Manager
• Azure Key Vault
• Google Secrets Manager
• Confidant
• Keywhiz

Terraform Authentication using Azure SPN

Provide Terraform credentials to manage resources

As described in my document about using an Azure SPN as an environment variable, Azure SPN, the secret “could” be used in the provider section of a Terraform configuration, but it is highly recommended that you never do this as per the first rule of fight club, that is to never store secrets in plan text.

Workstation environments (humans)

One of the most used methods for users authenticating their workstation environment with their cloud provider is to use environment variables. In essence, the secrets are saved as variables to the local workstation with pre-fixes that Terraform looks for when it requires and loads if they are present, if they are not then Terraform moves onto alternate authentication methods. This is detailed in the Azure SPN link from the previous paragraph.

Automation solutions (machines)

In most companies a CI/CD platform will likely be used to automate your Terraform workflows. This requires the “machine” to have access to your target cloud platform with credentials that can CRUD infrastructure. In these situations you still need a set of credentials such as an Azure SPN or AWS Credentials with the limited permissions required for your environment. The difference in most automation systems is that the credentials are saved as secrets, variables or contexts which the automation system is configured to call when required. This means your user or developer does not need to know the credentials or even have access to the target Cloud platform.

Terraform Cloud is a HashiCorp platform that runs a Terraform pipeline and stores Terraform state files. I have written a document detailing how Terraform Cloud Authentication works.

GitHub Actions is a widely used version control system that works with Git. It has a CI/CD pipeline engine called GitHub Actions that can be configured to connect with various platforms such as Azure as part of the pipeline. The authentication can be handled via either an SPN (username/password) or preferably using OpenID Connect (OIDC). I have written up details about how to authenticate to Azure from GitHub Actions.

Providing credentials for resources, such as databases

Aside from configuring Terraform with credentials it can use to access the target platform to manage resources, you may also need to provide resource specific secrets such as credentials to access a database.

Manually using temporary environment variables

When Terraform executes it supports reading data via environment variables. So if you have a database and you need to provide the credentials you can declare the variables values.

In the variables.tf file you create the variables.

variable "db_username" {
  description = "The database username"
  type        = string
  sensitive   = true
}

variable "db_password" {
  description = "The database password"
  type        = string
  sensitive   = true
}

In the resource block in the main.tf file you define the inputs as the var.db_usernameand var.db_password values.

On the system running the Terraform code, you can define the variable values themselves with a specific pre-fix of TF_VAR_*.

So to declare the username and password values you would run the following commands:

export TF_VAR_db_username=(value)
export TF_VAR_db_password=(value)

This is a pragmatic approach and just requires the operator to access the secrets from their secure storage first then manually set them as environment variables and then run the code. At scale this is inefficient and also means your operator still needs to access the secrets and somehow ensure they are secure whilst being used. Plus this approach is not going to work for CI/CD pipelines as the automation advantage is lost if a human needs to be involved. There are better but more complicated methods such as taking advantage of API access to the secure secret management tools listed at the start of this document.

Automatically using a managed service and an API call

Instead of providing the database secrets manually as temporary environment variables you can store them in a managed service like AWS Secrets Manager and then call them from there when needed from within the Terraform configuration.

As an example, let’s say we create the database credentials as secrets in our AWS Secrets Manager. The name of the secret is database00-creds and the credentials are saved as a basic key:value pair.

{"username":"admin","password":"secret"}

In the Terraform data sources section we first retrieve the secret from the AWS Secrets Manager via an API call.

data "aws_secretsmanager_secret_version" "credentials" {
  secret_id = "database00-creds"
}

The secrets are stored in JSON format so you can decode the retrieved data value using the jsondecode function and set then a local values.

locals {
  database00-creds = jsondecode(
    data.aws_secretsManager_secret_version.credentials.secret_string
  )
}

Finally, in the resource configuration itself you can refer to the database credentials from the values set in locals.

username = local.database00-creds.username
password = local.database00-creds.password

Be aware of secrets in Terraform State and Plan files

It’s very important to understand that regardless of which method you use to provide a resource with credentials they are still written to the Terraform State and Plan files as plain text. This is not the case for credentials used by Terraform itself to access the target platform as part of the provider configuration.

This is a security concern because Terraform stores the state as plain text. The sensitive = true flag only redacts the output from the CLI, but it doesn’t secure the data in the state file.

You need to take extra steps to secure your state. Here are some best practices:

Secure your backend: Use a secure and encrypted backend to store your state file. For instance, if you’re using AWS, you can store the state file in an S3 bucket with server-side encryption enabled. You can also enable versioning for your S3 bucket so you can track and revert changes, if necessary.

Use remote state with strict access controls: Only allow access to the state files to the necessary entities, whether they are users or machines. Use proper IAM roles and policies to control access.

Never print secrets as output: Although the sensitive flag will redact the output from the CLI, don’t create an output value for sensitive data to avoid any potential mishap.

Consider using a tool like Vault: You can use HashiCorp’s Vault service to fetch secrets at runtime. This can be integrated with Terraform via the Vault provider. In this way, the secret data doesn’t need to be stored in the state file.

However, there’s no full-proof method to completely avoid storing sensitive data in the Terraform state when it’s required by resources. These steps are just precautions to mitigate risks.