Environments: Dev, Test and Production

One configuration, several environments

Most organizations run their systems more than once: a development environment to try things, a test environment that looks like production, and production itself. The idea of a deployment environment comes from the Git & CI/CD topic; Kubernetes teams solve the same problem with Helm values and Kustomize overlays. This page is about the infrastructure underneath.

At Kestrel Energy, test runs fewer portal web servers than production; everything else — the nginx configuration, the firewall rules, the database settings — should be identical, because a test that does not look like production proves little. Kestrel's release process also has an acceptance environment between test and production (the A in DTAP); the examples on this topic use dev, test and prod to stay short, and acceptance would be one more environment built the same way. Three questions decide every design:

• Where do the differences live?
• How does each environment get its own state?
• How do you stop a change meant for test from reaching production — by accident or by a compromised laptop?

Why each environment needs its own state

The tool compares the code with the state. If two environments share one state, the tool believes there is only one set of objects, and every switch between environments looks like a large change. A test run shows it: a configuration was applied for dev with its default values (two web servers), and then planned with the production values against that same state.

Variable definitions files

A variable definitions file (.tfvars) sets values for a configuration's input variables, one name = value per line in HCL syntax. Passing a different file per environment is the simplest way to keep the differences in one place. envs/dev.tfvars:

environment = "dev"
web_count   = 1

envs/prod.tfvars:

environment = "prod"
web_count   = 4
services = {
  portal    = 8080
  meter-api = 8443
  billing   = 9100
}

The variables are declared in the configuration as described on Variables, Outputs and Expressions. Planning with the production file against the dev state:

$ cd /home/amara/kestrel-infra/vars
$ tofu plan -var-file=envs/prod.tfvars

…
OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement

OpenTofu will perform the following actions:

  # local_file.service["billing"] must be replaced
…
      ~ filename             = "./out/kestrel-dev-billing.conf" -> "./out/kestrel-prod-billing.conf" # forces replacement
…
  # local_file.web[2] will be created
…
  # local_file.web[3] will be created
…
  # local_sensitive_file.db_admin must be replaced
…
Plan: 8 to add, 0 to change, 6 to destroy.
…

Six existing dev objects would be destroyed and replaced by production ones — exactly what you would expect from the tool's point of view, and exactly what you never want. With real servers, this plan deletes the dev environment and builds production in its place, using the dev credentials. Each environment needs its own state: a separate backend key, a separate workspace, or a separate directory. The rest of this page is about those choices.

CLI workspaces

A workspace is a named state for the same configuration in the same backend. Every working directory starts with one workspace, default; tofu workspace new creates more, and tofu workspace select switches between them. The current name is available in the code as terraform.workspace, so the configuration can adapt:

locals {
  web_count = {
    default = 1
    dev     = 1
    test    = 2
    prod    = 4
  }
}

resource "local_file" "inventory" {
  filename = "${path.module}/out/${terraform.workspace}/web-hosts.txt"
  content = join("\n", [
    for i in range(local.web_count[terraform.workspace]) :
    format("web%02d.%s.kestrel.example", i + 1, terraform.workspace)
  ])
}

output "workspace" {
  value = terraform.workspace
}

$ cd /home/amara/kestrel-infra/ws
$ tofu workspace new test
$ tofu workspace new prod
$ tofu workspace list

Created and switched to workspace "test"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "tofu plan" OpenTofu will not see any existing state
for this configuration.
Created and switched to workspace "prod"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "tofu plan" OpenTofu will not see any existing state
for this configuration.
  default
* prod
  test

The asterisk marks the selected workspace. Applying in prod builds four hosts, in test two, and each workspace keeps its own state file:

$ tofu apply -auto-approve | tail -6
$ cat out/prod/web-hosts.txt

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

workspace = "prod"
web01.prod.kestrel.example
web02.prod.kestrel.example
web03.prod.kestrel.example
web04.prod.kestrel.example

$ tofu workspace select test
$ tofu apply -auto-approve | tail -6
$ find . -name "terraform.tfstate*" | sort

Switched to workspace "test".

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

workspace = "test"
./terraform.tfstate.d
./terraform.tfstate.d/prod/terraform.tfstate
./terraform.tfstate.d/test/terraform.tfstate

With the local backend, non-default workspaces live in terraform.tfstate.d/<name>/; the default workspace uses terraform.tfstate in the folder itself (absent here, because nothing was applied in it). Remote backends store each workspace under its own key or prefix. Two safety checks are built in:

$ tofu workspace select staging
$ tofu workspace delete prod

╷
│ Error: Workspace "staging" doesn't exist
│
│ You can create this workspace with the "new" subcommand
│ or include the "-or-create" flag with the "select" subcommand.
╵

╷
│ Error: Workspace is not empty
│
│ Workspace "prod" is currently tracking the following resource instances:
│   - local_file.inventory
│
│ Deleting this workspace would cause OpenTofu to lose track of any
│ associated remote objects, which would then require you to delete them
│ manually outside of OpenTofu. You should destroy these objects with
│ OpenTofu before deleting the workspace.
│
│ If you want to delete this workspace anyway, and have OpenTofu forget about
│ these managed objects, use the -force option to disable this safety check.
╵

What workspaces are not

Workspaces are convenient for short-lived copies: a test environment per feature branch, a quick experiment next to the real thing. They are a weak boundary between test and production, and the OpenTofu documentation says so directly: workspaces are not appropriate for deployments that require separate credentials and access controls, because all workspaces of a working directory use the same backend.

• Whoever can apply to test can apply to production: same credentials, same state bucket.
• The selected workspace is invisible in the code. A colleague who forgets that the last select was prod applies a test change to production. Run tofu workspace show before every apply, and show the workspace in your shell prompt.
• Differences hide in lookups such as local.web_count[terraform.workspace], scattered through the code.

A directory per environment

The directory-per-environment layout gives each environment its own root module folder, with its own backend configuration and variable values, while the actual resources live in shared modules that every environment calls. It is more files, and that is the point: what production runs is written down in production's own folder.

kestrel-infra/
├── modules/
│   └── web-host/            shared module, released as v1.2.0
└── envs/
    ├── test/
    │   ├── backend.tf
    │   ├── main.tf
    │   └── terraform.tfvars
    └── prod/
        ├── backend.tf
        ├── main.tf
        └── terraform.tfvars

envs/prod/main.tf calls the released module, not the folder next to it, so production only changes when someone raises the version (see version constraints). Illustrative:

module "portal" {
  source = "git::https://git.example.com/kestrel/tofu-modules.git//web-host?ref=v1.2.0"

  environment = "prod"
  web_count   = var.web_count
}

Here the terraform.tfvars files are fine, because each directory belongs to exactly one environment. You run the tool in the environment's folder (cd envs/prod), and the pipeline job for production runs only there. The cost is some repetition between the folders; keep it small by putting everything real in modules, so the environment folders contain little more than module calls, values and a backend block.

Partial backend configuration

Backend blocks are read before anything else, so traditionally they could not use input variables. A partial backend configuration leaves some or all backend settings out of the code and supplies them at init time, from a file or from -backend-config options. The same code can then be pointed at a different state per environment. Illustrative:

backend.tf

terraform {
  backend "s3" {}
}

backends/prod.s3.tfbackend

bucket       = "kestrel-tofu-state-prod"
key          = "portal/terraform.tfstate"
region       = "eu-central-1"
encrypt      = true
use_lockfile = true

$ tofu init -reconfigure -backend-config=backends/prod.s3.tfbackend
$ tofu plan -var-file=envs/prod.tfvars -out=prod.tfplan

-reconfigure tells init to use the new backend settings without offering to copy the state of the previous one — what you want when switching between environments, and dangerous if you mistype the file name, so scripts and pipelines should pass both files from one place. Note the separate bucket per environment: that makes it possible to give the test pipeline no access at all to production state. OpenTofu (1.8 and later) can also use variables and locals in the backend block itself, because it evaluates them early; that is an OpenTofu feature, so do not rely on it in code that must also run with Terraform.

Separate credentials and access

Separate state is about correctness. Separate credentials are about safety: a mistake, a bad module or a stolen laptop with test credentials must not be able to touch production.

• Separate cloud accounts or subscriptions for production and non-production, where the platform allows it. That is the strongest boundary the cloud offers.
• One pipeline identity per environment, with an IAM role that can change only that environment, obtained through workload identity federation (short-lived OpenID Connect tokens) instead of stored keys. The plan job gets a read-only role, plus just enough to take the state lock (with use_lockfile, writing and deleting the lock file).
• Production state readable and writable only by the production pipeline role and a small break-glass group (see sensitive values).
• An approval gate before every production apply: at Kestrel, the production job waits for Yara Haddad's approval in a protected environment (see deployment approval).
• Humans read-only in production, with a documented, logged way to get write access in an emergency.

Infrastructure as Code in Pipelines shows how such a pipeline is built.

Environment parity and promotion

Environment parity means keeping environments as alike as possible — same code, same module versions, same settings — so that what worked in test predicts what will happen in production. Differences should be deliberate, few and visible: size and count, names, network ranges, credentials.

This is environment promotion applied to infrastructure: a new module version is used first in dev, then in test, then in production, each step a small pull request that changes one ref or version. If production runs v1.2.0 and test runs v1.4.0, the test results say little about the next production change. Parity erodes in a few familiar ways:

• Copy-pasted environment folders that are edited separately and slowly diverge.
• "Temporary" manual fixes in one environment (drift) that are never put into code.
• Conditionals such as var.environment == "prod" ? … : … spread through modules, so test literally runs different code.
• Test environments that are cut down so far (no load balancer, one database instead of a cluster) that whole classes of problems cannot appear there.

Terragrunt and other wrappers

With many environments and many configurations, the directory-per-environment layout repeats the same backend block and the same inputs over and over. Terragrunt is an open-source (MIT license) wrapper around OpenTofu and Terraform that generates the backend configuration, passes inputs to modules and runs commands across many folders, keeping such setups DRY ("don't repeat yourself"). As of October 2026 the current release is v1.1.6; version 1.0 appeared in March 2026. An illustrative layout: a root.hcl at the top defines the state location once,

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket       = "kestrel-tofu-state"
    key          = "${path_relative_to_include()}/terraform.tfstate"
    region       = "eu-central-1"
    encrypt      = true
    use_lockfile = true
  }
}

and each environment folder, such as envs/prod/portal/terragrunt.hcl, contains only what is specific to it:

include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "git::https://git.example.com/kestrel/tofu-modules.git//web-host?ref=v1.2.0"
}

inputs = {
  environment = "prod"
  web_count   = 4
}

The state key is derived from the folder path, so every folder gets its own state automatically. Terragrunt is not installed on the machine used for this topic's recordings, so no output is shown. Alternatives are HCP Terraform workspaces and Terraform Stacks (hosted services from HashiCorp), or a few lines of pipeline script that pass the right -backend-config and -var-file per environment. Add a wrapper when the repetition really hurts, not before: it is one more tool everybody has to learn.

Choosing an approach

Whichever layout you choose, check it against the good habits from the Cloud topic: small changes, reviewed plans, pinned versions and no secrets in code.