Troubleshooting Infrastructure as Code

A method for infrastructure failures

The general structured troubleshooting method applies unchanged. What is special about infrastructure tools is that a run passes through clear stages, and each stage fails in its own way. Knowing the stage halves the search.

Then the usual loop: read the first error (later ones are often consequences), reproduce it with more detail, make the case smaller (one resource, one host with --limit), fix the cause, and plan again. Write what you find in the incident or change ticket as you go; see Reading Errors, Logs and Documentation.

Errors in the configuration

tofu validate catches most mistakes in the files themselves, without touching the state or any cloud. Every error box shows the file, the line and the offending code, and for typos OpenTofu suggests the name it expected. A misspelled argument:

╷
│ Error: Unsupported argument
│
│   on main.tf line 17, in resource "local_file" "upstream":
│   17:   file_permisson = "0644"
│
│ An argument named "file_permisson" is not expected here. Did you mean
│ "file_permission"?
╵

A misspelled variable is reported once for every place it is used — here twice. Fix the name once and both boxes disappear:

╷
│ Error: Reference to undeclared input variable
│
│   on main.tf line 13, in resource "local_file" "upstream":
│   13:     environment = var.enviroment
│
│ An input variable with the name "enviroment" has not been declared. Did you
│ mean "environment"?
╵
╷
│ Error: Reference to undeclared input variable
│
│   on main.tf line 23, in resource "terraform_data" "release":
│   23:     environment = var.enviroment
│
│ An input variable with the name "enviroment" has not been declared. Did you
│ mean "environment"?
╵

An attribute that the resource does not export (the random_id resource calls it hex, not hexadecimal) and a value of the wrong type:

╷
│ Error: Unsupported attribute
│
│   on main.tf line 15, in resource "local_file" "upstream":
│   15:     deployment  = random_id.deployment.hexadecimal
│
│ This object has no argument, nested block, or exported attribute named
│ "hexadecimal".
╵

╷
│ Error: Incorrect attribute value type
│
│   on main.tf line 2, in resource "random_id" "deployment":
│    2:   byte_length = "four"
│
│ Inappropriate value for attribute "byte_length": a number is required.
╵

Some errors only appear at plan time, because validate does not read variable values. A terraform.tfvars with web_servers = "192.0.2.11" (a string instead of a list) passes validate; the plan then prints what it had worked out so far and stops with:

…
╷
│ Error: Invalid value for input variable
│
│   on variables.tf line 7:
│    7: variable "web_servers" {
│
│ The given value is not suitable for var.web_servers declared at
│ variables.tf:7,1-23: list of string required, but have string.
╵

In a pipeline, where -input=false forbids prompting, a variable without a value fails like this:

╷
│ Error: No value for required variable
│
│   on variables.tf line 1:
│    1: variable "environment" {
│
│ The root module input variable "environment" is not set, and has no default
│ value. Use a -var or -var-file command line argument to provide a value for
│ this variable.
╵

All of these exit with code 1. The rules behind them (types, validation, where values come from) are on Variables, Outputs and Expressions.

Version and provider errors

tofu init installs providers that satisfy the constraints in required_providers and records the exact versions in the dependency lock file .terraform.lock.hcl. Most init errors come from a disagreement between those two.

The lock file no longer matches

Ravi Kumar raised the constraint for the local provider to ~> 3.0, while the lock file still pinned 2.9.1:

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
╷
│ Error: Failed to resolve provider packages
│
│ Could not resolve provider hashicorp/local: locked provider
│ registry.opentofu.org/hashicorp/local 2.9.1 does not match configured
│ version constraint ~> 3.0; must use tofu init -upgrade to allow selection
│ of new versions
╵

The message names the fix: tofu init -upgrade allows OpenTofu to pick a newer version. Here that failed as well, for a different reason — no 3.x release of the provider existed:

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/local versions matching "~> 3.0"...
╷
│ Error: Failed to resolve provider packages
│
│ Could not resolve provider hashicorp/local: no available releases match the
│ given constraints ~> 3.0
╵

After a deliberate -upgrade, read the provider's changelog, plan, and commit the updated lock file together with the constraint change. A provider upgrade is a change like any other: it can rename attributes or turn an update into a replacement.

A provider that does not exist

A typo in the source address (locals instead of local) asks the registry for a provider that is not there:

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/locals versions matching "~> 2.5"...
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/locals: provider registry registry.opentofu.org does not have a
│ provider named registry.opentofu.org/hashicorp/locals
│
…
╵

The same error appears for a provider that exists in the Terraform Registry but has not been published to the OpenTofu Registry, and when a module uses a provider without declaring it in its own required_providers, so that OpenTofu guesses hashicorp/<name>. tofu providers shows which module requires what. Old provider versions can also fail on newer platforms: on a 64-bit ARM machine, hashicorp/local ~> 1.4 failed with "does not have a package available for your current platform, linux_arm64".

The OpenTofu or Terraform version

A module can say which versions of the tool it needs. Here OpenTofu and Terraform have drifted apart, and a test run showed a surprise. In OpenTofu 1.13.1, a required_version = ">= 1.14.0" inside a terraform { } block in a .tf file was not enforced: tofu validate succeeded. The same block in a versions.tofu file was enforced, and so was OpenTofu's own language block:

compat.tf

language {
  compatible_with {
    opentofu = ">= 1.14.0"
  }
}

╷
│ Error: Incompatible module
│
│   on compat.tf line 3, in language:
│    3:     opentofu = ">= 1.14.0"
│
│ This module is not compatible with OpenTofu v1.13.1.
│
│ To proceed, either choose another supported OpenTofu version or update this
│ version constraint. Version constraints are normally set for good reason,
│ so updating the constraint may lead to other errors or unexpected behavior.
╵

The language block is OpenTofu-only syntax, available since OpenTofu 1.12; Terraform does not know it, so in a module that both tools use, put it in a .tofu file, which Terraform does not read. The OpenTofu documentation explains the background: older OpenTofu versions treated required_version as their own constraint in all cases, which got in the way of modules written for Terraform, and it recommends a versions.tofu with a language block (or required_version) for OpenTofu next to a versions.tf for Terraform. When both files exist, OpenTofu ignores versions.tf. Whatever the exact rule in your version, test it: run tofu validate with a constraint your tool does not meet, and check that it fails. With Terraform, required_version in a .tf file is enforced as before.

A stuck state lock

Every command that may write the state first takes a lock on it, so two runs cannot change the same infrastructure at once (see State). When the lock is taken, the second run stops with a box that tells you who holds it:

╷
│ Error: Error acquiring the state lock
│
│ Error message: resource temporarily unavailable
│ Lock Info:
│   ID:        59a466c2-1a7b-c2e1-5aa0-b69f4af768c3
│   Path:      terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       amara@control01
│   Version:   1.13.1
│   Created:   2026-10-06 16:00:45.221332488 +0000 UTC
│   Info:
│
│
│ OpenTofu acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

Who is the user and host of the run that holds the lock, Operation what it was doing and Created when it started (in UTC). Most of the time the right response is to wait: another apply is running. Add -lock-timeout=5m to let the command retry instead of failing at once.

Stale locks and force-unlock

With a local state file the lock is held by the operating system and dies with the process. With a remote backend (an S3 bucket, a PostgreSQL database, HCP Terraform) the lock is a record in the backend, and a run that crashes, is killed or loses its network connection can leave it behind. Such a stale state lock blocks every later run until someone removes it.

force-unlock is the command that removes it: tofu force-unlock LOCK_ID (or terraform force-unlock). It changes no infrastructure, only the lock. Kestrel's portal pipeline was stuck on Monday morning (INC-32114) after a runner was restarted during an apply. Tomasz Nowak first checked that no pipeline job and no colleague was running OpenTofu against production, then copied the ID from the lock box:

tofu force-unlock 7c41d0e2-93ab-4f6e-b2d8-51e0a6c3f915

Do you really want to force-unlock?
  OpenTofu will remove the lock on the remote state.
  This will allow local OpenTofu commands to modify this state, even though it
  may still be in use. Only 'yes' will be accepted to confirm.

  Enter a value: yes

OpenTofu state has been successfully unlocked!

The state has been unlocked, and OpenTofu commands should now be able to
obtain a new lock on the remote state.

Then he ran a plan before anything else: the interrupted apply may have changed resources that the state does not know about yet.

On a local state file there is nothing to force: force-unlock refuses, as the documentation says ("Local state files cannot be unlocked by another process"):

╷
│ Error: State locked by another local process
│
│ Local state cannot be unlocked by another process.
╵

How old is too old? A small script can turn the lock information into a decision, using the longest time a normal run takes as the yardstick. It only gives a hint; checking that the run is really gone stays a human job.

import json
from datetime import datetime, timezone

# The lock description OpenTofu wrote next to the state (shortened)
lock = json.loads(
    '{"ID": "59a466c2-1a7b-c2e1-5aa0-b69f4af768c3",'
    ' "Operation": "OperationTypeApply", "Who": "amara@control01",'
    ' "Version": "1.13.1", "Created": "2026-10-06T16:00:45Z"}'
)

# Pipeline jobs that apply production are known to take at most 30 minutes
LONGEST_EXPECTED_RUN = 30 * 60

now = datetime(2026, 10, 6, 17, 12, 0, tzinfo=timezone.utc)
created = datetime.fromisoformat(lock["Created"].replace("Z", "+00:00"))
age = int((now - created).total_seconds())

print(f"Lock {lock['ID'][:8]} held by {lock['Who']} for {lock['Operation']}")
print(f"Age: {age // 60} min {age % 60} s")
if age > LONGEST_EXPECTED_RUN:
    print("Older than any normal run: check that the run is really gone,")
    print(f"then: tofu force-unlock {lock['ID']}")
else:
    print("A run may still be busy: wait, do not unlock.")

Lock 59a466c2 held by amara@control01 for OperationTypeApply
Age: 71 min 15 s
Older than any normal run: check that the run is really gone,
then: tofu force-unlock 59a466c2-1a7b-c2e1-5aa0-b69f4af768c3

Lock errors that are not about locks

The lock box also wraps errors that happen while the state is being read for locking. With OpenTofu's state encryption, both of these were reported as "Error acquiring the state lock" in a test run — the real cause is in the Error message line. Enabling encryption on an existing plaintext state without a migration step:

╷
│ Error: Error acquiring the state lock
│
│ Error message: failed to write backup file: encountered unencrypted payload
│ without unencrypted method configured
│
…

And a wrong passphrase:

Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: failed to write backup file: decryption failed for all
│ provided methods
│ attempted decryption failed for state: decryption failed: cipher: message
│ authentication failed
│
…

Neither is fixed with force-unlock. The first needs the migration with an unencrypted fallback method, the second the right key; both are on State.

After a killed apply

In a test run an apply was killed with kill -9 while it was replacing a resource. The local lock died with the process (the .terraform.tfstate.lock.info file stayed behind but no longer blocked anything). The state still described the old object, although the apply had already destroyed it; the next plan noticed and proposed to create it again. That is the general recovery pattern: after any interrupted apply, do not edit the state — run a plan, read it carefully, and let the next apply bring state and reality back together. OpenTofu 1.13 also writes a partial errored.tfstate file if OpenTofu itself crashes, to help recovery; keep it until the situation is understood.

An unexpected replacement

The most dangerous plan is the one that looks like a small change but says must be replaced. Replacement means the old object is destroyed and a new one created — for a database volume, the data goes with it. The plan always says why: the attribute that causes it carries the comment # forces replacement.

  # time_sleep.db_migration must be replaced
-/+ resource "time_sleep" "db_migration" {
      ~ id              = "2026-10-06T16:00:39Z" -> (known after apply)
      ~ triggers        = { # forces replacement
          ~ "release" = "2026.10.1" -> "2026.10.2"
        }
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Here the replacement was intended (changing triggers is how this resource is meant to be rerun). When it is not, the usual causes are:

• An attribute that cannot change in place, such as the name of a database or the availability zone of a disk. The provider documentation lists which arguments force replacement.
• A renamed resource label or a resource moved into a module: to OpenTofu that is one resource removed and another added. A moved block fixes it (see Changing Infrastructure Safely).
• A count index shift after removing an item from the middle of a list; for_each with stable keys avoids it.
• A provider upgrade that changed a default or how an attribute is compared.
• A tainted resource (below).

At Kestrel, a provider upgrade once produced "Plan: 1 to add, 0 to change, 1 to destroy." for db01's data volume. Hamid Rahimi stopped the change, found the attribute marked # forces replacement, and pinned the provider back until the module was adjusted. The volume also had prevent_destroy, which would have stopped the apply anyway — a test run shows what that looks like:

╷
│ Error: Resource instance cannot be destroyed
│
│   on main.tf line 5:
│    5: resource "random_password" "db" {
│
│ Resource instance random_password.db has prevent_destroy set, but the plan
│ calls for it to be destroyed.
│
│ To proceed, either disable prevent_destroy for this resource or exclude
│ instances of this resource from this round using:
│     -exclude="random_password.db"
╵

Treat this error as a fire alarm, not an obstacle: something wants to destroy a resource that someone marked as precious. The -exclude suggestion is OpenTofu-specific (the option was added in OpenTofu 1.9).

Tainted resources

When a create fails halfway — the cloud created the virtual machine but a later step timed out — OpenTofu cannot be sure the object works. It keeps it in the state as a tainted resource, and the next plan replaces it:

  # aws_instance.web03 is tainted, so it must be replaced
-/+ resource "aws_instance" "web03" {
…
Plan: 1 to add, 0 to change, 1 to destroy.

If you have checked that the object is fine after all, tofu untaint aws_instance.web03 removes the mark (only in the state; it prints "Resource instance aws_instance.web03 has been successfully untainted."). The opposite, tofu taint, is deprecated: to force a replacement, use tofu apply -replace=ADDRESS, which shows the replacement in a plan before anything happens.

Debug logging

When the error message is not enough, OpenTofu and Terraform can log what happens inside: the configuration loading, the provider plugins starting, every call to a provider. The environment variable TF_LOG switches this on, with a level from most to least detailed: TRACE, DEBUG, INFO, WARN, ERROR (JSON gives trace-level logs as JSON). The log goes to standard error, so redirect it when you pipe:

TF_LOG=DEBUG tofu plan -var environment=test 2>&1 | head -40

2026-10-06T18:03:03.591+0200 [INFO]  OpenTofu version: 1.13.1
2026-10-06T18:03:03.591+0200 [DEBUG] This is an official build of OpenTofu
…
2026-10-06T18:03:03.591+0200 [INFO]  Go runtime version: go1.27.1
2026-10-06T18:03:03.591+0200 [INFO]  CLI args: []string{"tofu", "plan", "-var", "environment=test"}
…
2026-10-06T18:03:03.593+0200 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2026-10-06T18:03:03.593+0200 [DEBUG] ignoring non-existing provider search directory /home/amara/.terraform.d/plugins
…
2026-10-06T18:03:03.597+0200 [INFO]  backend/local: starting Plan operation
…

A plain plan of four resources produced 193 lines at DEBUG. To keep the normal output readable, send the log to a file with TF_LOG_PATH (which only works when TF_LOG is set as well):

TF_LOG=INFO TF_LOG_PATH=tofu-debug.log tofu plan -var environment=test -concise
wc -l tofu-debug.log

No changes. Your infrastructure matches the configuration.
31 tofu-debug.log

TF_LOG_CORE and TF_LOG_PROVIDER take the same levels but limit the log to OpenTofu itself or to the providers. Provider-level logs are where you find the actual API request that the cloud rejected.

Ansible: unreachable hosts

Ansible reports two different kinds of trouble. A task can fail on a host it reached, or Ansible cannot reach or log in to the host at all. The second is UNREACHABLE!: the host is counted under unreachable in the PLAY RECAP, skipped for the rest of the play, and the command exits with code 4. One real attempt against a documentation address that is never routed:

ansible all -i '192.0.2.10,' -m ping -u amara \
    -e "ansible_ssh_common_args='-o ConnectTimeout=3 -o UserKnownHostsFile=/home/amara/kestrel-infra/known_hosts -o StrictHostKeyChecking=no'"

[ERROR]: Task failed: Failed to connect to the host via ssh: ssh: connect to host 192.0.2.10 port 22: Connection timed out
Origin: <adhoc 'ping' task>

{'action': 'ping', 'args': {}, 'timeout': 0, 'async_val': 0, 'poll': 15}

192.0.2.10 | UNREACHABLE! => {
    "changed": false,
    "msg": "Task failed: Failed to connect to the host via ssh: ssh: connect to host 192.0.2.10 port 22: Connection timed out",
    "unreachable": true
}

(The StrictHostKeyChecking=no was for this one-off test against a throwaway known_hosts file; see the warning below.) The text after "Failed to connect to the host via ssh:" is OpenSSH's own message, and it tells you which layer failed:

Emeka Obi met the first one on his second day: web02 was unreachable after it had moved to a new address, 192.0.2.14. DNS had been updated, the inventory had not. ansible-inventory --host shows the variables Ansible will really use for a host — in the test inventory, web02 still pointed at its old address:

ansible-inventory --host web02

{
    "ansible_connection": "local",
    "ansible_host": "192.0.2.12",
    "kestrel_env": "test",
    "kestrel_root": "{{ playbook_dir }}/hosts/{{ inventory_hostname }}",
    "nginx_worker_connections": 2048,
    "portal_port": 8080
}

(In the test setup every host uses the local connection, which is why ansible_connection is local; on a real network it would be ssh.) If the inventory looks right, try the same connection by hand with ssh -v amara@192.0.2.14, and run Ansible with -vvvv, which prints the full SSH command and its debug output.

Privilege escalation errors

Tasks that need root use become: true, which runs them through sudo by default. If sudo wants a password and Ansible has none, the task fails:

[ERROR]: Task failed: Missing sudo password
…
fatal: [db01]: FAILED! => {"msg": "Task failed: Missing sudo password"}

Either run with --ask-become-pass (-K), or — for automation — give the deploy user a narrow passwordless rule in sudoers. "Incorrect sudo password" means the password was given but wrong. Since ansible-core 2.19, a timeout while waiting for the privilege escalation prompt counts as unreachable rather than as a failed task, so ignore_errors does not catch it (ignore_unreachable does). This page was recorded with ansible-core 2.19; the current release as of October 2026 is 2.21.

Ansible: failed tasks

A task that runs but fails prints, since ansible-core 2.19, an [ERROR]: line with the reason, an Origin: line with file, line and column, a short code excerpt, and then the classic fatal: … FAILED! line with the module's full result. A command task reading a file that does not exist:

TASK [Read a file that does not exist] *****************************************
skipping: [web02]
[ERROR]: Task failed: Module failed: non-zero return code
Origin: /home/amara/kestrel-infra/ansible/checks.yml:26:7

24       ignore_errors: true
25
26     - name: Read a file that does not exist
         ^ column 7

fatal: [web01]: FAILED! => {"changed": false, "cmd": ["cat", "/home/amara/kestrel-infra/ansible/hosts/web01/etc/kestrel/feature-flags.yml"], "delta": "0:00:00.002898", "end": "2026-10-06 18:05:17.458042", "msg": "non-zero return code", "rc": 1, "start": "2026-10-06 18:05:17.455144", "stderr": "cat: /home/amara/kestrel-infra/ansible/hosts/web01/etc/kestrel/feature-flags.yml: No such file or directory", "stderr_lines": ["cat: /home/amara/kestrel-infra/ansible/hosts/web01/etc/kestrel/feature-flags.yml: No such file or directory"], "stdout": "", "stdout_lines": []}

In the result, read msg first, then rc (the command's exit status) and stderr: here cat said "No such file or directory". A failed host is dropped from the rest of the play, and ansible-playbook exits with code 2. Older material shows errors as ERROR! …; that is the format before 2.19.

Undefined variables

A variable that is never defined fails when the task uses it. In 2.19 the message is a chain of caused by sections, from the task down to the exact expression:

TASK [Use a variable that was never defined] ***********************************
[ERROR]: Task failed: Finalization of task args for 'ansible.builtin.debug' failed: Error while resolving value for 'msg': 'backup_target' is undefined

Task failed.
Origin: /home/amara/kestrel-infra/ansible/filters.yml:27:7
…
<<< caused by >>>

Error while resolving value for 'msg': 'backup_target' is undefined
Origin: /home/amara/kestrel-infra/ansible/filters.yml:29:14

27     - name: Use a variable that was never defined
28       ansible.builtin.debug:
29         msg: "Backups go to {{ backup_target }}"
                ^ column 14
…

Read the last caused by section: it points at the column. If the variable should exist, the question is where its value comes from — a group or host variable file that does not apply to this host, a typo, or a task that registers it but was skipped (--start-at-task is a common way to skip the task that sets a variable). Precedence and ansible-inventory --host are on Variables, Facts and Templates.

A module that cannot be found

A module from a collection that is not installed stops the whole playbook before anything runs (exit code 4):

[WARNING]: Error loading plugin 'community.general.ini_file': No module named 'ansible_collections.community'
[ERROR]: couldn't resolve module/action 'community.general.ini_file'. This often indicates a misspelling, missing collection, or incorrect module path.
Origin: /home/amara/kestrel-infra/ansible/billing-ini.yml:6:7
…

Install the collection from the project's requirements.yml (see Roles and Collections), and check that the pipeline installs it too.

More detail: -v to -vvvv

Each -v adds detail. -v shows which configuration file is used and the full result of every task; -vv adds the version header; -vvv adds how the inventory was parsed, the connection and every command Ansible runs on the host; -vvvv adds connection debugging (for SSH, the output of ssh -vvv). A ping with -vvv, shortened:

ansible web01 -m ping -vvv

ansible [core 2.19.14]
  config file = /home/amara/kestrel-infra/ansible/ansible.cfg
…
Using /home/amara/kestrel-infra/ansible/ansible.cfg as config file
…
Parsed /home/amara/kestrel-infra/ansible/inventory/hosts.ini inventory source with ini plugin
…
<192.0.2.11> ESTABLISH LOCAL CONNECTION FOR USER: amara
…
Using module file /home/amara/.venv/ansible/lib/python3.11/site-packages/ansible/modules/ping.py
…
web01 | SUCCESS => {
    "changed": false,
    "invocation": {
        "module_args": {
            "data": "pong"
        }
    },
    "ping": "pong"
}

Over SSH the connection line reads ESTABLISH SSH CONNECTION and is followed by the SSH command line. The config file = line answers a common question: which ansible.cfg is in effect? Ansible ignores an ansible.cfg in a world-writable directory (with a warning), and ansible-config dump --only-changed lists every setting that differs from the default and where it came from. For the test project's ansible.cfg:

…
CONFIG_FILE() = /home/amara/kestrel-infra/ansible/ansible.cfg
DEFAULT_FORKS(/home/amara/kestrel-infra/ansible/ansible.cfg) = 10
DEFAULT_HOST_LIST(/home/amara/kestrel-infra/ansible/ansible.cfg) = ['/home/amara/kestrel-infra/ansible/inventory/hosts.ini']
…
DEFAULT_VAULT_PASSWORD_FILE(/home/amara/kestrel-infra/ansible/ansible.cfg) = /home/amara/kestrel-infra/ansible/.vault-pass
…

Recovering safely

Under pressure, the quickest-looking fix is often the one that makes things worse. A few rules keep a bad hour from becoming a bad week:

• Back up the state before you touch it. tofu state pull > state-backup-INC-32114.tfstate saves the current state (store it as carefully as the state itself: it contains the same secrets). Remote backends with versioning keep older versions too.
• Never edit state JSON by hand. Use moved, removed and import blocks, or as a last resort tofu state mv and tofu state rm, and plan after each step.
• Prefer a plan over a habit. No -auto-approve, -lock=false or -target "to get it through"; each one removes a safety check exactly when you need it.
• Fix the cause in code. If an emergency needed a manual change, put it into the configuration afterward, or the next apply will undo it.
• Write it down. Record what you saw, what you ran and why in the incident or change ticket, so the next person — or the postmortem — does not have to guess.

The same layered approach for containers is on kubectl and Troubleshooting, and for plain SSH loops on Remote Execution.