Working Safely on Production Systems

What makes production different

The production environment is the live environment that real users, customers and business processes depend on, with real data. Everything else (development, test, acceptance) exists so that mistakes happen there instead; the release and deployment management page describes those environments, often called DTAP (in Dutch: OTAP).

In production, mistakes are visible to customers, cost money, can expose personal data and are sometimes impossible to undo: a deleted table without a backup stays deleted, an email sent to 80,000 customers stays sent. And production is where most outages start. The Google site reliability engineering (SRE) book reports that, in Google's own experience, "roughly 70% of outages are due to changes in a live system". That is one company's figure, not an industry statistic, but the pattern is familiar to anyone who has worked in operations: things usually break when someone touches them.

That is good news, because changes are something you control. The habits on this page are not about being afraid of production; they make it possible to change it often and calmly.

Know which system you are on

A surprising number of serious incidents start with a correct command typed into the wrong window. You have a terminal open to acceptance and one to production, both with the same prompt, and after a phone call you continue in the wrong one. The defense is to make the difference impossible to miss:

• Make production look different. Put the environment name and the full hostname in the shell prompt; many teams also use a different terminal color or profile for production sessions.
• Read the prompt before a risky command, every time. Run hostname if in doubt.
• Close production sessions when you are done. A forgotten SSH session to a production server is a mistake waiting to happen.
• Check the context in cloud and database tools too: the selected subscription, account or project in a cloud console, and the connected server and database in a SQL client.

ravi@lgn01:~$
ravi@lgn01:~$

[ACC]  ravi@lgn-acc-01.example.com:/etc/login-service $
[PROD] ravi@lgn-01.example.com:/etc/login-service $

Use your privileges sparingly

The less power a session has, the less damage a mistake can do. This is the principle of least privilege (see authorization models) applied to your own daily work:

• Use a normal account for email and browsing, and a separate admin account only for admin tasks; see privileged access management.
• On Linux, run single commands with sudo instead of opening a root shell that stays open all afternoon.
• Connect read-only when you only need to look, for example with a read-only database account for investigating data.
• Where your organization offers just-in-time access, request admin rights for the task and let them expire.

Check before you press Enter

Before any command or click that changes production, take ten seconds and answer five questions. Experienced engineers do this so routinely that it looks like hesitation; it is not.

1. WHERE  Am I on the host, cluster, database or account I mean?
2. WHAT   Exactly which objects does this touch? Read the wildcards,
          the WHERE clause, the selection, the list of servers.
3. HOW MANY  Do I expect 3 rows, 1 server, 12 files? Check the
          count first.
4. UNDO   If this is wrong, how do I get back? Is there a backup
          or snapshot from today?
5. WHO    Does anyone need to know before I do this (service desk,
          the change record, the incident channel)?

Questions 2 and 3 catch the most expensive mistakes. A DELETE without a WHERE clause, a wildcard that matches more than you thought, an Ansible inventory group that contains every server instead of two: all of them run perfectly, which is exactly the problem. Count first, then act.

Dry runs and previews

A dry run is a way of running a command or tool so that it reports what it would do without actually doing it. It turns question 2 ("exactly which objects?") from a guess into a list you can read. Many tools have one built in:

With Terraform or OpenTofu, save the plan with -out and apply that file, so exactly the reviewed actions run; Ansible's --check skips command and shell tasks, so it is a forecast, not a guarantee (see saved plans and check mode).

When a tool has no dry-run option, build one: run the selecting half of the command on its own and read the result. Here Joost cleans up old log files. He first lists and counts what the command selects, and only then runs the same command with -delete added:

# Set up: two old log files and one from today
mkdir -p logs
touch -d 2026-01-05 logs/billing-2026-01-05.log
touch -d 2026-01-06 logs/billing-2026-01-06.log
touch logs/billing-today.log

# Step 1, the dry run: which files are older than 30 days?
find logs -name "*.log" -mtime +30 | sort
find logs -name "*.log" -mtime +30 | wc -l

# Step 2: the same selection, now with -delete
find logs -name "*.log" -mtime +30 -delete
ls logs

logs/billing-2026-01-05.log
logs/billing-2026-01-06.log
2
billing-today.log

The important part is that step 2 is step 1 with one word added, not a new command typed from memory. Scripts can protect themselves in a similar way: in Bash, ${DIR:?} stops the script with an error if the variable is empty, instead of silently turning a path such as "$DIR/old" into "/old". See safer scripts for more of these guards.

# Stops with "DIR: parameter null or not set" if DIR is empty
rm -r -- "${DIR:?}/old"

The four-eyes principle

The four-eyes principle is the rule that a significant action needs two people: one who prepares or carries it out and one who checks or approves it. It is common in banking and in IT change management, and in daily practice it takes several forms:

• Peer review of the change before it is made: a colleague reads the plan, the configuration diff or the script. In code and infrastructure-as-code teams this is the pull request review; see pull requests and code review.
• Two people at the keyboard for risky manual steps: one types, the other reads each command before Enter and watches the output.
• Approval by someone with the authority to accept the risk, for example a change authority for a normal change.

The point is not distrust. The author of a change reads what they meant to write; a reviewer reads what is actually there. That is why even very experienced engineers ask for review, and why a reviewer who only clicks "approve" without reading adds no safety at all. For critical tasks such as payments or production access, organizations formalize the same idea as separation of duties (see access control).

Small changes and blast radius

The blast radius of a change or a failure is how much it can affect if it goes wrong: how many servers, services, users or customers. You cannot make every change safe, but you can almost always make its blast radius smaller.

• One first, then the rest. Apply the change to one server, one site or one group of users, verify, and only then continue. A canary release does this for software (see DevOps and CI/CD).
• Small batches. Ten small changes on different days are easier to verify and to undo than one big change that touches everything. When something breaks, the cause is obvious.
• Separate risky parts. Do not combine a firmware update, a configuration change and a migration in one window. If something fails, you want to know which part.
• Switch, don't rebuild. Feature flags and blue-green deployments make it possible to switch new behavior off without a new deployment.

The SRE book recommends automation for exactly these steps: "Implementing progressive rollouts", "Quickly and accurately detecting problems" and "Rolling back changes safely when problems arise". Even without automation, you can work the same way by hand.

Back-out plans and backups

Every change needs an answer to "how do we go back?" before it starts. A back-out plan, described on the release and deployment management page, gives the steps to return to the previous working state and the criteria for using them. For your own work, three habits make the difference between a plan and a hope:

• Test it. Run the back-out steps in acceptance, with a timer. An untested back-out often fails at the worst moment.
• Set the trigger in advance: which measurement, after how many minutes, and who decides. That is a stop rule; see working under pressure.
• Save the current state first: a copy of the configuration file, a database backup, a VM snapshot. A snapshot is useful for a quick return but is not a backup; see backup and recovery.

For infrastructure as code the back-out is usually "revert the commit and apply again", which does not bring back a destroyed database; see a safe change, step by step.

Compare two change records for the same work:

CHG-7752  Increase DB connections on login servers.
Low risk. Will do tonight.

CHG-7752  Normal change · risk: medium
Purpose:  Raise login-service DB pool 50 → 80 as capacity headroom
          (the incident-time raise in INC-31902 was reverted 2 Oct)
Scope:    login-service config on lgn-01 and lgn-02 only
Window:   Sun 11 Oct 2026 00:30–01:30 (clear of the 02:30 billing
          export)
Tested:   Acceptance, Tue 6 Oct; load test passed, back-out tested
          (6 min)
Review:   Hamid Rahimi (DBA): db01 limit 200 is shared with meter-api
          and the billing export; per-role limit for the portal
          raised to 160 under CHG-7751 first
Approved: Yara Haddad (release manager), 7 Oct
Steps:    1. Save current config of lgn-01 and lgn-02
          2. Apply to lgn-01 only, restart, smoke test login
          3. Watch login error rate for 15 min
          4. Apply to lgn-02, restart, smoke test login
Verify:   Login error rate below 0.5%; DB connections below 70%
Back-out: Restore saved config and restart, about 6 min per node.
          Trigger: smoke test fails, or error rate above 1% for
          5 min. Ravi (implementer) backs out without asking.
Informed: Service desk (Mei Chen); status page notice Fri 9 Oct

The clear record answers every question a reviewer, the service desk or a colleague at 03:00 would ask. It also limits the blast radius (one node first) and contains its own stop rule.

Maintenance windows, freezes and Fridays

Timing is a safety measure too. A maintenance window puts disruptive changes at a quiet, agreed moment, and a change freeze blocks non-urgent changes during periods when the business cannot afford trouble, such as a billing run or a forecast storm. Both are explained under the change schedule. For the person doing the work, the practical rules are:

• Plan the change so that it, its verification and a possible back-out fit in the window. If the back-out does not fit, the window is too short or the change too big.
• Respect a freeze even when "it's only a small change". Small changes cause outages too, and the freeze exists because an outage now would be especially expensive.
• Be well rested. A window at 02:00 means sleeping beforehand, not working a full day and then staying up.

A recurring debate is whether to avoid changes on Friday afternoons, sometimes called "read-only Friday". Supporters point out that fewer people are around over the weekend if something goes wrong. Critics, often from the continuous-delivery side, argue that if a team is afraid to deploy on Friday, its tests, monitoring and rollback are not good enough, and that is the real problem to fix. Both have a point: a team with small, automated, easily reversible deployments can safely deploy on Friday, while a large manual change late on a Friday deserves a second thought.

Site reliability teams also freeze releases when a service has used up its error budget; see error budget policy.