terraform-secret-placement

Terraform Secret Placement

Terraform has its strengths and its challenges. Of it’s strengths, Terraform gets you referential integrity, reproducability, and a rich set of clients with great documentation.
One of its challenges includes keeping sensitive secrets out of State Files.

So just get rid of the state.

1
terraform --state-out /dev/null -lock=false
  • Secrets will be created, and won’t hit a disk until its at its final resting place.
  • A lock won’t be required if you create new resources every time.

Implementation

Consider creating a DataDog Token to be stored in a DataBricks Secret.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
provider "datadog" {}    # Reference docs for ENV Variable settings.
provider "databricks" {} # OAuth and other identity protocols may be available.

# Create a Datadog API key
resource "datadog_api_key" "my_datadog_key" {
  name = "terraform-databricks-token"
}

# Create a Databricks secret scope (if not already existing)
resource "databricks_secret_scope" "monitoring" {
  name                     = "monitoring"
  initial_manage_principal = "users"
}

# Store Datadog API key inside Databricks secret scope
resource "databricks_secret" "datadog_api_key" {
  key          = "datadog_api_key"
  string_value = datadog_api_key.my_datadog_key.key
  scope        = databricks_secret_scope.monitoring.name
}

Fire-and-Forget

The resources you create do not get added to state. You won’t be able to destroy or reference the resources after they’re created. This is the design.

This practice works best on resources which will not be further referenced in terraform, like credentials.

If you’d like to work with the resources after, you can consider:

  • Printing out non-sensitive identifiers allows you to work with these specific resources (e.g. “Token Names.”)
  • Noting the resources you’re creating, as secret keys are uniquely identifying and should be easy to find again.
  • Token resources should be temporary. Run this script frequently. Expire tokens quickly. Delete expired things.

Alternatives

This approach comes off as “hacky” to some critics at first blush. Compare this process against operational work to install application credentials.

Manual Alternative

Every organization will go through a phase of manually passing credentials. These procedures will be inconsistent, staff intensive, potentially involve more than one person/team, require exposing the secret (if only to yourself), and sharing the secret to another third-party service (including secrets ending up in Slack.)

Imperative Scripting

You can achieve similar results using a non-declarative programming language, like python. This will be more work. It will require integrating with multiple APIs, adopting new under-reviewed packages, and will need to marshal values without Terraform’s type assurance. Each API will have its nuances, and getting up to speed on how they work.

The same credentials you’ll need for these processes, can be used to create Terraform Providers.

Additional Considerations

Expect manual clean-ups if you want to “undo.” You can figure out roll-backs or terraform destroys with these resources, but that will likely take some effort. Manual clean-ups may be best, and are reasonable.

Some resources are globally unique. Creating these resource types with this fire-and-forget technique AND in tracked state, could lead to confusion (much like doing manual work in a Terraform-controlled-environment.)

Development of these scripts benefits from having a development environment. If you don’t get your script right the first time, you can expect wanting to clean up half-runs as you go. Fortunately, Terraform will allow you to validate, test, and see the plan before anything is executed on the environment. Without any review, your first run will be ensured by allof Terraform’s reference and type integrity checks, before it’s executed.

Additional Thoughts

You can Terraform values directly into Password Managers such as 1Password, Bitwarden, and LastPass.

Most modern build tools, like Github Actions and Terraform Cloud have providers to receive secrets directly from scripts such as this.

Using terraform_remote_state data resources, you can reference resources in your deployed environment. This provides READ-ONLY access to state, and allows you to easily for_each over many resources, like you’ll want to do for key rotations.

Conclusion

Using Terraform to define credential transfers to their final destination with its declarative syntax provides an effective, more secure, way to support operational work, likely using the same tools which you’re using in production.