Secrets in Automation

Where secrets leak in automation

Configuring db01 needs the PostgreSQL admin password. Deploying the portal needs a database password, an API token for the meter-api and cloud credentials for the pipeline. Automation cannot avoid secrets; it can only control where they appear. These are the places where they end up by accident:

The rest of this page is about three ways out: keep secrets encrypted in the repository (Ansible Vault, SOPS), keep them outside the repository in a secrets manager, or avoid long-lived secrets altogether with short-lived credentials. Real setups combine all three.

Ansible Vault

Ansible Vault is the encryption feature built into ansible-core: it encrypts whole variable files, or single values, with a password, so they can be committed next to the playbooks. Ansible decrypts them in memory when a playbook runs and is given the vault password. The ansible-vault command has the subcommands create, edit, view, encrypt, decrypt, encrypt_string and rekey.

The output on this page was recorded with ansible-core 2.19 (the current release line is 2.21, as of October 2026); the vault commands behave the same. The only "secrets" are the obviously fake values example-only and example-only-too.

The vars and vault pattern

Hamid Rahimi, Kestrel's database administrator, keeps the db01 secrets in two files per group. The readable file says which variables exist; the encrypted file holds the values, each with a vault_ prefix:

ansible/
├── .vault-pass            vault password file (mode 600, in .gitignore)
├── group_vars/
│   └── db/
│       ├── vars.yml       readable: names point to vault_ variables
│       └── vault.yml      encrypted with Ansible Vault
└── db-secrets.yml         playbook

group_vars/db/vars.yml:

postgres_admin_password: "{{ vault_postgres_admin_password }}"

Because vars.yml stays readable, a reviewer can still search for where postgres_admin_password comes from, and a pull request diff shows that a secret was added, even though the value itself is unreadable. The same pattern appears on Network Automation with Git. ansible-vault create opens an editor (so it needs a terminal) and encrypts the file when you save it. Hamid typed these two lines:

vault_postgres_admin_password: example-only
vault_billing_sftp_password: example-only-too

$ ls -l .vault-pass
$ ansible-vault create --vault-password-file .vault-pass group_vars/db/vault.yml
$ cat group_vars/db/vault.yml
$ ansible-vault view --vault-password-file .vault-pass group_vars/db/vault.yml

-rw------- 1 amara amara 13 Oct  6 18:07 .vault-pass
$ANSIBLE_VAULT;1.1;AES256
63316433626265316134383735363435333036646137366362323530636230393366636465616132
3939663362353434613833366562376235393239356335370a626431613132366464663033306662
35623963653232333337333538353933626637656565613431336530653763643963626561383430
6364636136646266610a363533373361313366623136336639303862626465623866383431326362
33343731353339356337353163326536663334623665373831656664613332363363353237333736
35343466366133626638623061313434383635363332653561326237623564313962633466393335
63623662633166383035386436303565623961366665323138356664663332633933333332356539
39396461623831653236323838373239323461633831626332366232333739376562636232623965
3730
vault_postgres_admin_password: example-only
vault_billing_sftp_password: example-only-too

create prints nothing when it succeeds. The encrypted file starts with the header $ANSIBLE_VAULT;1.1;AES256 (format 1.1, cipher AES256), followed by hexadecimal text. Encrypt the same content again and the hex is completely different, because every encryption uses a new random salt. The password file is only readable by its owner and is listed in .gitignore: it is the one thing that must never be committed.

Encrypting an existing file or a single value

encrypt turns an existing plain file into a vault file in place. A wrong password fails; without --vault-password-file, Ansible asks for the password (the input is not echoed):

$ cat secrets/meter-api.yml
$ ansible-vault encrypt --vault-password-file .vault-pass secrets/meter-api.yml
$ head -3 secrets/meter-api.yml
$ ansible-vault view --vault-password-file wrong-pass secrets/meter-api.yml
$ ansible-vault view secrets/meter-api.yml

meter_api_token: example-only-token
Encryption successful
$ANSIBLE_VAULT;1.1;AES256
37353763396637373765656265396238353664653063303732343337623639646638363133643438
6635663061613434623565396366313932646461616465340a353461653236303136393433376464
[ERROR]: Decryption failed (no vault secrets were found that could decrypt). for secrets/meter-api.yml
Vault password:
meter_api_token: example-only-token

"Encryption successful" goes to standard error and only appears in an interactive terminal, so do not expect it in a pipeline log. encrypt_string encrypts one value and prints it as a YAML variable with the !vault tag, ready to paste into an otherwise readable file:

$ ansible-vault encrypt_string --vault-password-file .vault-pass "example-only" --name "billing_api_key"

billing_api_key: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          33353238613965666463346234663535663831643939613039303664346138386265336436326637
          6335343531373633356638626232336661313564666362380a383734316538373265306238303463
          38363839343730303831393235653565373638313135623438396236333863616362663862396534
          3938353536393332300a646335353563306262326264326635646231646233646261373536636434
          3861

Using the vault in a playbook

A playbook that reads vaulted variables needs the password. Without it, the run stops before the first task:

$ ansible-playbook db-secrets.yml

PLAY [Configure PostgreSQL credentials on db01] ********************************
[ERROR]: Attempting to decrypt but no vault secrets found.

The exit status is 4. Give the password with --vault-password-file (or --ask-vault-pass to type it). In a pipeline, the password comes from the CI system's secret store, for example through a small script that prints an environment variable. You can also set vault_password_file in ansible.cfg, but then everyone who runs Ansible from that folder silently uses it, so be deliberate about that.

Vault IDs: one password per environment

One vault password for everything means that anyone who may run the test playbooks can also read the production passwords. A vault ID is a label attached to a vault password, so one run can hold several passwords and each encrypted file or value says which one it needs:

$ ansible-vault encrypt --vault-id prod@prompt group_vars/prod/vault.yml
$ ansible-playbook site.yml --vault-id test@.vault-pass-test --vault-id prod@prompt

prod@prompt means "the password labeled prod, asked at the prompt"; after the @ you can also give a file or an executable script that prints the password. Files encrypted with a label get a version 1.2 header that names it: $ANSIBLE_VAULT;1.2;AES256;prod. The label is a hint for choosing the right password, not access control: what protects production is that only the production pipeline knows the production password.

How vault encryption works

Format 1.1 uses AES-256 in counter mode for the data and an HMAC-SHA256 to detect a wrong password or a damaged file (see symmetric encryption and HMAC). The keys are derived from your password with PBKDF2-HMAC-SHA256 and 10,000 iterations. Under the header, the hex text decodes to three lines: the salt, the HMAC and the ciphertext. This program takes the billing_api_key value recorded above and checks candidate passwords exactly the way Ansible does before it decrypts anything (the decryption itself needs AES, which is not in Python's standard library):

import hashlib
import hmac

vaulted = """$ANSIBLE_VAULT;1.1;AES256
33353238613965666463346234663535663831643939613039303664346138386265336436326637
6335343531373633356638626232336661313564666362380a383734316538373265306238303463
38363839343730303831393235653565373638313135623438396236333863616362663862396534
3938353536393332300a646335353563306262326264326635646231646233646261373536636434
3861"""

header, *hex_lines = vaulted.splitlines()
print("Header:", header.split(";"))

body = bytes.fromhex("".join(hex_lines))  # first layer of hex
salt_hex, hmac_hex, cipher_hex = body.split(b"\n")
salt = bytes.fromhex(salt_hex.decode())
stored_hmac = bytes.fromhex(hmac_hex.decode())
ciphertext = bytes.fromhex(cipher_hex.decode())
print(f"Salt: {len(salt)} bytes, HMAC: {len(stored_hmac)} bytes, "
      f"ciphertext: {len(ciphertext)} bytes")


def password_matches(password):
    # 80 bytes: 32-byte AES key + 32-byte HMAC key + 16-byte counter start
    derived = hashlib.pbkdf2_hmac("sha256", password.encode(), salt, 10_000, 80)
    hmac_key = derived[32:64]
    computed = hmac.new(hmac_key, ciphertext, hashlib.sha256).digest()
    return hmac.compare_digest(computed, stored_hmac)


for guess in ["example-only", "Example-only"]:
    print(f"{guess!r:16} ->", "HMAC matches" if password_matches(guess) else "rejected")

Header: ['$ANSIBLE_VAULT', '1.1', 'AES256']
Salt: 32 bytes, HMAC: 32 bytes, ciphertext: 16 bytes
'example-only'   -> HMAC matches
'Example-only'   -> rejected

Two practical consequences follow:

• Use a long random vault password, not a memorable one. Anyone with a copy of the repository can try passwords offline, as this program does. 10,000 PBKDF2 iterations make each guess only slightly expensive (current guidance for password storage uses hundreds of thousands; OpenTofu's state encryption defaults to 600,000). A 32-character random password makes guessing hopeless anyway; generate one with openssl rand -base64 32 and keep it in a password manager or secret store.
• Length is not hidden. The 12-character token became 16 bytes of ciphertext (padded to the AES block size). An attacker learns roughly how long a secret is, never what it is.

When someone who knew the vault password leaves, or the password file leaks, run ansible-vault rekey on every vault file to re-encrypt it with a new password. Rekeying does not change the secrets inside, and old copies in Git history can still be opened with the old password, so also rotate the secrets themselves (see when a secret leaks).

Keeping secrets out of the output: no_log

Encryption at rest does nothing once a task prints the decrypted value. no_log is a task keyword: no_log: true tells Ansible to replace the task's result with a fixed "censored" message in all output and logs, whatever the verbosity. The playbook below writes the admin password to a file, then shows the classic leak: a debug task that prints a connection string. (kestrel_root points to the folder that stands in for db01's file system in the recording; on a real server it would be empty.)

db-secrets.yml:

- name: Configure PostgreSQL credentials on db01
  hosts: db
  gather_facts: false

  tasks:
    - name: Write the admin password file
      ansible.builtin.copy:
        content: "{{ postgres_admin_password }}\n"
        dest: "{{ kestrel_root }}/etc/postgresql/16/main/admin.pw"
        mode: "0600"
      no_log: true

    - name: Debug a connection string (leaks the password)
      ansible.builtin.debug:
        msg: "postgresql://postgres:{{ postgres_admin_password }}@db01:5432/portal"

    - name: Same debug with no_log
      ansible.builtin.debug:
        msg: "postgresql://postgres:{{ postgres_admin_password }}@db01:5432/portal"
      no_log: true

$ ansible-playbook db-secrets.yml --vault-password-file .vault-pass -v

Using /home/amara/kestrel-infra/ansible/ansible.cfg as config file

PLAY [Configure PostgreSQL credentials on db01] ********************************

TASK [Write the admin password file] *******************************************
ok: [db01] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

TASK [Debug a connection string (leaks the password)] **************************
ok: [db01] => {
    "msg": "postgresql://postgres:example-only@db01:5432/portal"
}

TASK [Same debug with no_log] **************************************************
ok: [db01] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

PLAY RECAP *********************************************************************
db01                       : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

This was the second run, so the password file already existed and the first task reports ok. Even at -v, which normally prints the whole result, the no_log tasks show only the censored message, while the second task prints the password for everyone who can read the pipeline log. The written file has mode 0600 (-rw-------), readable only by its owner.

SOPS and age

Ansible Vault is tied to Ansible and encrypts whole files, so a diff of a vault file is unreadable. SOPS takes a different approach. SOPS (Secrets OPerationS) is an editor for encrypted files: it encrypts only the values in a YAML, JSON, ENV or INI file and leaves the keys readable, so a reviewer can see that meter_api_token changed without seeing its value. It started at Mozilla in 2015, has been a CNCF (Cloud Native Computing Foundation) Sandbox project since May 2023, and the current release is 3.13.3 (as of October 2026). SOPS was not installed on the test machine, so its output below is illustrative.

SOPS can encrypt for cloud key management services (AWS KMS, Google Cloud KMS, Azure Key Vault and others), for PGP keys, or for age; the SOPS documentation recommends age over PGP. age is a small, modern file-encryption tool with short keys: a public key ("recipient") starting with age1 that anyone may know, and a private key starting with AGE-SECRET-KEY-1 that only the people and systems that must decrypt hold.

A .sops.yaml file at the top of the repository tells SOPS which keys to use for which files, so nobody has to remember:

creation_rules:
  - path_regex: secrets/prod/.*\.yaml$
    age: age1prodexampleonlyrecipientkeynotarealkey000000000000000000000
  - path_regex: secrets/test/.*\.yaml$
    age: age1testexampleonlyrecipientkeynotarealkey000000000000000000000

$ sops edit secrets/prod/meter-api.yaml
$ cat secrets/prod/meter-api.yaml

meter_api_url: ENC[AES256_GCM,data:Qm9ndXMt…,iv:…,tag:…,type:str]
meter_api_token: ENC[AES256_GCM,data:aW52YWxp…,iv:…,tag:…,type:str]
sops:
    age:
        - recipient: age1prodexampleonlyrecipientkeynotarealkey000000000000000000000
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            …
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-10-06T09:12:44Z"
    mac: ENC[AES256_GCM,data:…,iv:…,tag:…,type:str]
    version: 3.13.3

sops edit decrypts into an editor and re-encrypts on save; sops decrypt prints the plain file; sops encrypt encrypts an existing one. To decrypt, SOPS looks for the age private key in ~/.config/sops/age/keys.txt on Linux, or in the file named by the SOPS_AGE_KEY_FILE environment variable. The sops: block lists who can decrypt and a message authentication code that detects tampering. Ansible reads SOPS files through the community.sops collection; in Kubernetes, the GitOps tools decrypt them (see secrets in GitOps).

Secret managers: HashiCorp Vault and OpenBao

Encrypted files still put secrets in Git: everyone who ever held the key can read every old version. A secret manager keeps the secrets in a separate service. Automation authenticates to it at run time, reads what it needs, and nothing secret is ever committed. The cloud services (AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager) are described on Cloud Security and Compliance; two self-hostable products are common in infrastructure work.

HashiCorp Vault stores secrets, issues credentials and certificates, and decides who may read what through policies. Clients log in with an auth method (a Kubernetes service account, a cloud identity, a JSON Web Token from a CI system) and receive a token with a limited lifetime. Its license changed from MPL 2.0 (Mozilla Public License) to the Business Source License 1.1 from version 1.15.0 in 2023; the current release is 2.1.1 (as of October 2026), and IBM now owns HashiCorp. OpenBao is the community fork of Vault from before that license change, under MPL 2.0 and governed under the OpenSSF (Open Source Security Foundation, part of the Linux Foundation); the current release is 2.7.1. Its command is bao where Vault's is vault, and for everyday use the two work the same way.

Dynamic secrets

The strongest feature of these tools is a dynamic secret: a credential that does not exist until someone asks for it. Instead of storing one shared database password, Vault or OpenBao holds an admin connection to PostgreSQL on db01 and creates a new database user for each request, with a lease (lifetime). When the lease expires, the service deletes the user again. Every run gets its own credential, and a leaked one stops working within the hour.

$ bao read database/creds/portal-readonly

Key                Value
---                -----
lease_id           database/creds/portal-readonly/Yq3NnB2tPkW0cS1vJm8aR4dE
lease_duration     1h
lease_renewable    true
password           example-only-generated
username           v-token-portal-r-7Kd2xQ9mLp4vR1sT8wZb-1791277200

Ansible reads from Vault and OpenBao through the community.hashi_vault collection (a lookup plugin); Terraform and OpenTofu use the Vault provider. Either way, the playbook or configuration contains only the path of the secret, which is safe to commit.

Secrets in Terraform and OpenTofu

Terraform and OpenTofu have their own pitfall: the state file records the attributes of every resource, and a plan file records the values it will set. Marking a variable or output sensitive = true only hides the value in plan and apply output: it is still stored in plain text in state, as the State page shows with a recorded random_password. Three tools address this, from best to last resort:

1. Let the platform keep the secret. Many cloud resources can generate and store their own password. For an AWS RDS database, manage_master_user_password = true makes RDS create the password in AWS Secrets Manager and rotate it; Terraform never sees it.
2. Ephemeral values and write-only arguments. An ephemeral resource produces a value that exists only during the run; a write-only argument (names ending in _wo) accepts a value that the provider sends on but never stores. Together they keep a secret out of both state and plan. Terraform added them in 1.10 and 1.11, OpenTofu in 1.11; they only work where the provider implements them.
3. Protect the state itself: a private backend with strict access, and in OpenTofu, client-side state and plan encryption (see State).

Ravi Kumar uses the second option for the portal's database password: an ephemeral random password goes straight into AWS Secrets Manager, where the portal reads it at start-up.

ephemeral "random_password" "portal_db" {
  length  = 24
  special = false
}

resource "aws_secretsmanager_secret" "portal_db" {
  name = "kestrel/prod/portal-db"
}

resource "aws_secretsmanager_secret_version" "portal_db" {
  secret_id                = aws_secretsmanager_secret.portal_db.id
  secret_string_wo         = ephemeral.random_password.portal_db.result
  secret_string_wo_version = 1
}

Because the value is not stored, the provider cannot notice when it changes. That is what secret_string_wo_version is for: raise it to 2 and the next apply generates and writes a new password. The example is illustrative (it needs an AWS account); the random_password ephemeral resource exists in the hashicorp/random provider, and the _wo arguments in recent hashicorp/aws versions.

Short-lived credentials with OIDC

The most dangerous secret in an infrastructure repository is the one the pipeline uses to change the cloud: it can usually create and delete almost anything. A long-lived access key stored as a CI secret is valid everywhere, for years, until someone rotates it. OpenID Connect (OIDC) removes it: the pipeline proves which repository, branch or environment it is running for with a signed token, and the cloud exchanges that token for credentials that expire within the hour. The general mechanism is workload identity federation.

The job needs permissions: id-token: write, which only allows it to request the token. Everything depends on the trust policy of the cloud role, and in particular on the condition on the token's subject (sub) claim, which says where the run comes from. This is the AWS trust policy for Kestrel's production apply role, written in HCL:

data "aws_iam_policy_document" "tofu_apply_prod_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:kestrel-energy/kestrel-infra:environment:production"]
    }
  }
}

Only jobs in the production environment of the kestrel-infra repository can take this role, and that environment requires Yara Haddad's approval (see deployment approvals). The plan job on pull requests uses a different, read-only role. Common subject formats in GitHub Actions:

Azure (federated credentials on a managed identity or app registration) and Google Cloud (workload identity pools) follow the same pattern; how the pipeline uses these roles is covered on Infrastructure as Code in Pipelines.

When a secret leaks anyway

Sooner or later a password ends up in a commit, a log or a ticket. The order of steps matters:

1. Rotate first. Change the password or revoke the key at its source, then update it in the vault or secret manager. Removing it from Git is not enough: clones, forks, CI caches and log archives already have it (see when a secret leaks and secret rotation).
2. Check whether it was used: cloud audit logs, database logins, API access logs since the leak.
3. Clean up the log, artifact or history where possible, and record the incident.
4. Prevent the repeat: add the file to .gitignore, add no_log, turn on secret scanning and push protection.

A short checklist for any automation repository:

• No plain-text secrets in Git: vault files, SOPS files or secret manager paths only. Vault password files and age private keys in .gitignore.
• A separate vault password or key per environment; production keys only in the production pipeline.
• no_log: true on every task that handles a secret; no secrets in debug output.
• State and plan files treated as secrets: restricted backend, short-lived artifacts, state encryption where available.
• Pipelines use OIDC with narrow trust policies; a read-only role for plans, a scoped role per environment for applies, following least privilege.
• Nothing secret in images or cloud-init user data; instances fetch secrets at boot with their own identity.