Images and Bootstrapping

From a blank machine to a working server

Terraform or OpenTofu can create a virtual machine in seconds, but a machine that has just been created is only an operating system. It does not yet have nginx, the monitoring agent, the Kestrel portal or the right host name. Bootstrapping is the process that takes a new machine from that first start to the point where it does its job, or where a configuration management tool can take over. There are three moments to do the work, and each suits different steps:

1. When the image is built ("baking"): before any machine exists. Everything that is slow and the same for every server goes here.
2. At first boot ("frying"): cloud-init applies the per-instance details passed as user data.
3. While it runs: the application fetches secrets with the machine's own identity, and configuration management (Ansible, Puppet) keeps the server up to date, if it is not replaced instead.

The Cloud topic introduces machine images and a first cloud-init example; this page goes into how to build the images, how cloud-init really behaves, and how to debug it. Packer and cloud-init need a cloud or a hypervisor, so all their output here is illustrative, written from their documentation.

Golden images

A golden image is a machine image that an organization builds, tests and approves as the standard starting point for its servers: patched, hardened, with the agents and software they all need. Every web server starts from the same tested bits, instead of from a public image plus twenty minutes of installing at boot, each time with whatever the package mirrors offer that day.

Good golden images share a few habits:

• Built by code, never by hand. A template in Git describes the image; nobody logs in to a builder machine to "quickly install one more thing".
• Versioned and immutable. Kestrel names its web image kestrel-web-2026.10.1: year, month and a build number. A published version is never changed; a fix is a new version.
• Rebuilt on a schedule, at least monthly and usually weekly, so new servers start with current patches (see remediation). An image that is never rebuilt quietly becomes the oldest, least patched thing in the estate.
• Promoted like an application. The same image version goes to test first and then to production, as in build once, promote.
• Free of secrets and identity. No passwords, no private keys, no SSH host keys or machine ID that every copy would share (see cleaning before capture).

The same idea exists for containers, where the image is built from a Dockerfile (see Building Good Images), and on hypervisors as VM templates.

Building images with Packer

Packer is HashiCorp's tool for building machine images from a template. It starts a temporary machine from a base image, runs provisioners on it (shell commands, file uploads, an Ansible playbook), then saves the result as a new image and deletes the temporary machine. One template can produce images for several platforms at once: AWS, Azure, Google Cloud, VMware or QEMU for local virtual machines. As of October 2026 the current version is 1.16.1; since 1.10.0 (December 2023) Packer is under the Business Source License, like Terraform, and the builders are separate plugins instead of being bundled.

Templates are written in HCL, in files ending in .pkr.hcl. Older tutorials use JSON templates and expect the plugins to be built in; current Packer wants a packer { required_plugins { … } } block and packer init. Ravi Kumar's template for the web image, kestrel-web.pkr.hcl:

packer {
  required_plugins {
    amazon = {
      source  = "github.com/hashicorp/amazon"
      version = "~> 1.8"
    }
    ansible = {
      source  = "github.com/hashicorp/ansible"
      version = "~> 1.1"
    }
  }
}

variable "image_version" {
  type    = string
  default = "2026.10.1"
}

source "amazon-ebs" "kestrel_web" {
  region        = "eu-west-1"
  instance_type = "t3.small"
  ssh_username  = "admin"
  ami_name      = "kestrel-web-${var.image_version}"

  # Debian's official AWS account publishes the base images
  source_ami_filter {
    owners      = ["136693071363"]
    most_recent = true
    filters = {
      name                = "debian-13-amd64-*"
      virtualization-type = "hvm"
    }
  }

  tags = {
    Name    = "kestrel-web"
    Version = var.image_version
    Owner   = "ravi.kumar"
  }
}

build {
  sources = ["source.amazon-ebs.kestrel_web"]

  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get -y upgrade",
    ]
  }

  provisioner "ansible" {
    playbook_file = "../ansible/web-image.yml"
  }

  provisioner "shell" {
    inline = ["sudo cloud-init clean --logs --machine-id"]
  }
}

• source describes where to build: the base image (the newest Debian 13 image from Debian's own account), the machine size and the name of the result.
• build lists the steps, in order: patch the operating system, run the playbook web-image.yml, which applies the same Ansible roles that would otherwise configure a running web server (see Roles and Collections), and finally clean cloud-init's traces so every copy boots as a new machine.
• The plugin versions are pinned with ~> constraints, and the base image comes from a known owner: a public image from an unknown account is a supply chain risk.

The workflow mirrors Terraform's:

$ packer init .
$ packer fmt .
$ packer validate .
$ packer build .

amazon-ebs.kestrel_web: output will be in this color.

==> amazon-ebs.kestrel_web: Prevalidating AMI Name: kestrel-web-2026.10.1
==> amazon-ebs.kestrel_web: Found Image ID: ami-0a1b2c3d4e5f60718
==> amazon-ebs.kestrel_web: Launching a source AWS instance...
…
==> amazon-ebs.kestrel_web: Provisioning with shell script: /tmp/packer-shell1879330511
==> amazon-ebs.kestrel_web: Provisioning with Ansible...
…
==> amazon-ebs.kestrel_web: Stopping the source instance...
==> amazon-ebs.kestrel_web: Creating AMI kestrel-web-2026.10.1 from instance i-0c4d5e6f7a8b9c0d1
==> amazon-ebs.kestrel_web: Terminating the source AWS instance...
Build 'amazon-ebs.kestrel_web' finished after 9 minutes 41 seconds.

==> Wait completed after 9 minutes 41 seconds

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs.kestrel_web: AMIs were created:
eu-west-1: ami-0f9e8d7c6b5a43210

Ten minutes is normal, which is exactly why this work belongs in the image: it happens once a week in a pipeline, not every time the portal needs one more web server during a busy evening.

cloud-init and user data

cloud-init is the standard program that configures a Linux machine during its first boot in a cloud or on a hypervisor. It is preinstalled in most cloud images. At boot it asks the platform for the instance's metadata and user data (on a cloud, through the instance metadata service; on a hypervisor, often from a small attached disk) and then sets the host name, creates users, writes files, installs packages and runs commands. The current release is 26.2 (as of October 2026).

User data is the free-form configuration you hand to a new instance when you create it, for cloud-init (or, on Windows, the provider's agent) to act on. It can be a shell script, but most often it is cloud-config: cloud-init's YAML format, which must start with the line #cloud-config and describes what to do with keys such as hostname, users, write_files, packages and runcmd. Because the image already contains everything heavy, Kestrel's user data is small:

cloud-init.yaml.tftpl (a template OpenTofu fills in):

#cloud-config
hostname: ${hostname}
fqdn: ${hostname}.kestrel.example.com
write_files:
  - path: /etc/kestrel/environment
    permissions: "0644"
    content: |
      KESTREL_ENV=${environment}
bootcmd:
  - [sh, -c, "echo boot $(date -u +%FT%TZ) >> /var/log/kestrel-boot.log"]
runcmd:
  - [systemctl, enable, --now, kestrel-portal-config.service]

The kestrel-portal-config service is baked into the image; at start-up it fetches the portal's settings and database password from the secret manager, using the instance's own cloud identity. The user data itself contains nothing secret. Note the quoted "0644": unquoted, YAML 1.1 parsers read 0644 as the octal number 420 (see Configuration Languages).

Passing user data from OpenTofu

The servers themselves are created with OpenTofu, which renders the template once per host with templatefile() and passes the result as user data (illustrative: it needs an AWS account):

data "aws_ami" "kestrel_web" {
  owners = ["self"]

  filter {
    name   = "name"
    values = ["kestrel-web-2026.10.1"]
  }
}

resource "aws_instance" "web" {
  for_each = toset(["web01", "web02"])

  ami                         = data.aws_ami.kestrel_web.id
  instance_type               = "t3.small"
  iam_instance_profile        = "kestrel-web-prod"
  user_data_replace_on_change = true

  user_data = templatefile("${path.module}/cloud-init.yaml.tftpl", {
    hostname    = each.key
    environment = "prod"
  })

  metadata_options {
    http_tokens = "required"
  }

  tags = {
    Name = each.key
  }
}

• The image is chosen by its exact version name, not by "the most recent", so a new image reaches production only when someone changes this line in a reviewed pull request.
• iam_instance_profile gives each server a role, so it can fetch its secrets without any password in the user data.
• http_tokens = "required" enforces version 2 of the AWS instance metadata service, which makes it much harder for a vulnerable application to be tricked into revealing the user data and the role's credentials.
• user_data_replace_on_change = true makes a change to the user data replace the instance. Without it, the AWS provider stops and starts the instance with the new user data, and, as the next section shows, cloud-init then ignores most of it.

Per-instance and per-boot: why a reboot changes nothing

cloud-init keeps a cache in /var/lib/cloud, including the ID of the instance it last configured. At every boot it compares that ID with the one the platform reports. If they differ, this is the first boot of a new instance and it runs everything, including the per-instance modules (host name, SSH host keys, write_files, runcmd). If they match, it is just a reboot, and only the per-boot modules run, such as bootcmd. This small model follows that rule for a few modules:

# A much simplified model of how cloud-init decides what to run at boot.
FREQUENCY = {  # module -> how often it runs (as in cloud-init's own modules)
    "bootcmd": "always",
    "set_hostname": "per-instance",
    "ssh (new host keys)": "per-instance",
    "write_files": "per-instance",
    "runcmd": "per-instance",
}


def boot(disk, instance_id, user_data):
    """Boot a machine whose disk holds cloud-init's cache (a dict)."""
    first_boot = disk.get("instance_id") != instance_id
    if first_boot:
        disk["instance_id"] = instance_id
    ran = [
        module
        for module, how_often in FREQUENCY.items()
        if how_often == "always" or first_boot
    ]
    print(f"{instance_id} ({user_data}): {', '.join(ran)}")


web01_disk = {}
boot(web01_disk, "i-0a11", "user data v1")  # first boot
boot(web01_disk, "i-0a11", "user data v2")  # edited user data, then reboot

# An image captured from web01 without cleaning: the cache travels along,
# but a cloud gives the new instance a new ID, so it is detected as new.
web02_disk = dict(web01_disk)
boot(web02_disk, "i-0b22", "user data v2")

# A VM cloned on a hypervisor whose seed reuses the same instance ID.
clone_disk = dict(web01_disk)
boot(clone_disk, "i-0a11", "user data v2")

clone_disk.clear()  # what `cloud-init clean` does to the cache
boot(clone_disk, "i-0a11", "user data v2")

i-0a11 (user data v1): bootcmd, set_hostname, ssh (new host keys), write_files, runcmd
i-0a11 (user data v2): bootcmd
i-0b22 (user data v2): bootcmd, set_hostname, ssh (new host keys), write_files, runcmd
i-0a11 (user data v2): bootcmd
i-0a11 (user data v2): bootcmd, set_hostname, ssh (new host keys), write_files, runcmd

Three lessons follow from the five lines:

• Line 2: editing user data and rebooting runs only bootcmd. The new write_files and runcmd are ignored. To apply new user data, create a new instance (or use configuration management for the change).
• Line 4: a clone that reports the same instance ID is treated as a reboot of the original. It keeps the original's host name and, worse, its SSH host keys, so every clone has the same identity.
• Line 5: after the cache is removed, cloud-init treats the next boot as a first boot again.

Cleaning before capture

That last step is why the Packer template ends with cloud-init clean --logs --machine-id. The command removes cloud-init's cache and logs, and resets /etc/machine-id so that every server started from the image generates its own. On first boot, cloud-init's SSH module then deletes the host keys and generates new ones (the default). Skip the clean step and clones can share a machine ID and SSH host keys, which confuses monitoring and logging and breaks the trust that known_hosts depends on.

When first boot goes wrong

A new web server that never joins the load balancer usually means cloud-init did not finish, or did something other than expected. Log in (through the provider's serial console if SSH is not up yet) and ask cloud-init first:

$ cloud-init status --wait --long

status: done
extended_status: done
boot_status_code: enabled-by-generator
last_update: Tue, 06 Oct 2026 08:14:31 +0000
detail: DataSourceEc2Local
errors: []
recoverable_errors: {}

--wait blocks until cloud-init is finished, which also makes it useful in scripts and Packer builds that must wait for first boot. The exit status is 0 when everything ran without error, 2 when it finished with recoverable errors, and 1 when it crashed; with errors, the errors and recoverable_errors fields say what went wrong. Then:

• /var/log/cloud-init-output.log holds the output of your own commands (runcmd, scripts): read it first.
• /var/log/cloud-init.log is cloud-init's detailed log: which data source it found, which modules ran and how long each took.
• sudo cloud-init schema --system --annotate checks the user data the instance received against the cloud-config schema and marks the errors in place. The two most common problems, according to the cloud-init documentation, are invalid YAML and a first line that is not exactly #cloud-config; in both cases cloud-init does not apply the configuration as you meant.

$ cloud-init schema --config-file cloud-init.yaml --annotate

#cloud-config
write_files:
  - path: /etc/kestrel/environment        # E1
    permisions: "0644"
    content: |
      KESTREL_ENV=prod

# Errors: -------------
# E1: Additional properties are not allowed ('permisions' was unexpected)

Run the same check on your rendered template before it ever reaches a cloud: in the pipeline, against the file that OpenTofu would pass as user data. Ravi's pipeline does this with a test copy of the template, so a typo fails the pull request instead of the first boot of web03.

What to bake, what to configure at boot

Immutable servers

With a good image pipeline, a new possibility appears: never change a running web server at all. To patch, build image kestrel-web-2026.10.2, start new servers from it, move the traffic, and remove the old ones. This is the server form of immutable infrastructure (see What Is Infrastructure as Code?), and containers apply the same idea to applications.

• No drift: nobody can have changed a server that lives for two weeks and was never logged into.
• Easy rollback: the previous image still exists; point the configuration back at it.
• Tested as a whole: the exact image that ran in test runs in production.

It only works for stateless servers, whose data lives elsewhere. web01 and web02 qualify: the portal keeps its data in PostgreSQL on db01 and its files in object storage, and the load balancer or an autoscaling group replaces instances one at a time. db01 does not: replacing the database server means moving its data. Hamid Rahimi therefore keeps db01 as a long-lived server, patched in place with Ansible, with backups and a protected data volume. Most estates mix both models, and that is fine as long as everyone knows which servers are which.