Agile in IT Operations

The examples on this page follow Kestrel Energy's network operations team: four engineers, among them Tomasz Nowak and the newer Emeka Obi, who keep the network running for the offices, the data center and the customer portal, and who are also migrating the company's firewalls this year.

Why operations is different

Agile frameworks were mostly designed for teams that build a product: work can be ordered in advance, and a plan for the next two weeks has a fair chance of surviving. Operations teams live in a different rhythm:

• Interrupts. Incidents, urgent requests and escalations arrive at any moment and often cannot wait.
• On-call. Someone is responsible outside office hours, and a bad night affects the next day's capacity (see ITSM in Practice).
• Keeping the lights on. Patching, certificate renewals, backups, capacity checks and audits: routine work that never appears in a product roadmap but cannot be skipped.
• Many customers. The network team serves every other team, each of which thinks its request is the most important.

None of this makes agile unsuitable. It means the agile ideas that matter most for operations are the ones about flow, visibility and limiting work in progress, more than the ones about fixed iterations.

The four types of work

The novel The Phoenix Project (Gene Kim, Kevin Behr and George Spafford, 2013), widely read in IT operations, popularized a simple classification of IT work into four types:

1. Business projects: work requested by the business, such as the network for a new office.
2. Internal IT projects: improvements IT makes for itself, such as the firewall migration or new monitoring.
3. Changes: routine updates and adjustments that come out of the first two and out of daily operations, such as a new firewall rule or a firmware update.
4. Unplanned work: incidents and urgent problems that interrupt everything else.

Unplanned work is work that nobody scheduled and that displaces scheduled work when it arrives: restoring a failed link, handling a security alert, an urgent request from management. The book's point, which matches most ops teams' experience, is that unplanned work is the most dangerous type: it is invisible in plans, it crowds out the projects and changes that would prevent it, and so it tends to grow. Making it visible (on the board and in the numbers) is the first step to reducing it.

Reserving capacity for interrupts

If a team plans 100% of its capacity and then loses a third of it to incidents every week, every plan fails and every commitment becomes a negotiation. The fix is an interrupt buffer: a share of the team's capacity that is deliberately left unplanned and reserved for unplanned work. Planned work is sized to fit what remains.

The size of the buffer should come from data, not from hope. Using the average would mean the buffer is too small about half of the time; a higher percentile gives a buffer that holds in most weeks. The example uses eight weeks of hours logged on unplanned tickets by the network operations team.

import math

# Network operations team: 4 engineers x 36 focus hours = 144 hours a week.
capacity = 144

# Hours spent on unplanned work (incidents, urgent requests,
# escalations) in the last 8 weeks, from the ticket system.
unplanned = [38, 52, 41, 47, 70, 44, 39, 49]

def percentile(values, pct):
    # Nearest-rank method, as for a service level expectation.
    ordered = sorted(values)
    rank = math.ceil(pct / 100 * len(ordered))
    return ordered[rank - 1]

average = sum(unplanned) / len(unplanned)
print(f"Average unplanned work: {average:.1f} hours ({average / capacity:.0%})")

for pct in (50, 75, 90):
    hours = percentile(unplanned, pct)
    print(f"{pct}th percentile:       {hours} hours ({hours / capacity:.0%})")

buffer = percentile(unplanned, 75)
over = [week for week, hours in enumerate(unplanned, start=1) if hours > buffer]
print(f"\nReserve {buffer} hours a week for interrupts; "
      f"{capacity - buffer} hours remain for planned work.")
print(f"Weeks that would have overrun the buffer: {over}")

Average unplanned work: 47.5 hours (33%)
50th percentile:       44 hours (31%)
75th percentile:       49 hours (34%)
90th percentile:       70 hours (49%)

Reserve 49 hours a week for interrupts; 95 hours remain for planned work.
Weeks that would have overrun the buffer: [2, 5]

With a buffer of 49 hours, the team plans about 95 hours of project work and changes each week and expects to overrun the buffer in roughly one week out of four. In week 5, a failing core switch took 70 hours; no buffer should be sized for that. In such a week the team accepts that planned work slips, says so early, and looks at what caused the spike. When the buffer is not used up, the spare hours go to the next planned items or to removing the causes of unplanned work.

Kanban for operations

Ops Kanban is the use of Kanban by an operations team: all work, planned and unplanned, on one board, with WIP limits and explicit rules about what goes first. It suits operations because it does not need a fixed plan: work is pulled when there is capacity, and urgent work has a defined path instead of an argument. The general mechanics are on Kanban.

Classes of service

Classes of service are explicit policies that decide how different kinds of work are treated, based on the cost of delaying them. The idea comes from the Kanban Method; the four classes below are the ones most often used:

Intangible work is the class that quietly disappears in busy teams, and it is exactly the work (automation, runbooks, upgrades) that reduces future unplanned work. Many teams protect it with a minimum share of capacity, for example one item in progress at all times.

The expedite lane

The expedite lane is a separate swimlane at the top of the board for items so urgent that they may bypass the queue and the normal WIP limits. To keep it meaningful, it has its own strict limit, usually one item, and an explicit policy for what qualifies (for the NOC: a priority 1 or 2 incident, or a security alert from the security team). If everything is urgent, nothing is: an expedite lane that is always full means the team's priorities or capacity need a conversation with management.

Scrum for operations teams

Scrum can work for an operations team, but it fits some teams much better than others.

• It works when most of the work can be planned: a platform or automation team, a team building infrastructure as code, or an ops team whose unplanned work is a modest and stable share. A Sprint Goal ("all branch offices monitored by the new system") gives focus that a stream of tickets does not.
• It struggles when unplanned work dominates. If half of the capacity disappears into incidents, the Sprint Goal is missed almost every Sprint and the events start to feel like overhead. Kanban, or Scrum with Kanban practices (see Kanban with Scrum), is usually a better fit then.

Teams that do use Scrum often adapt the way they handle interrupts:

• Shorter Sprints, such as one week, so that new urgent work rarely has to wait long for the next Sprint Planning.
• An interrupt buffer left out of Sprint Planning, as described above.
• An interrupt rotation (sometimes called the "shield" or "goalkeeper"): each Sprint, one Developer handles incoming tickets and questions so the others can work toward the Sprint Goal undisturbed. The role rotates so everyone shares the load and keeps their knowledge of daily operations.

Combining project work and daily operations

Tomasz leads the firewall migration: 24 tasks to replace the old firewalls at the data center and five offices before the support contract ends. At the same time, he and Emeka handle the team's share of tickets. Without an agreement, the pattern is familiar: the project slips every week because "tickets came first", and the tickets wait whenever a migration weekend approaches.

The team made both kinds of work visible on the same board and agreed rules:

1. All work is on the board: tickets, changes and migration tasks (FWM-nn).
2. In progress: at most 4 items for the team; expedite does not count, max 1.
3. Every day at least one migration task is in progress.
4. Interrupt rotation: one engineer per week takes new tickets first;
   this week Emeka, with Tomasz as backup for firewall questions.
5. The project plan assumes 95 hours a week of planned work, not 144.
6. Ten-minute stand-up at 09:15: what is blocked, what is aging, what is next.

Rule 3 protects the project from being starved; rule 4 protects Tomasz's focus without leaving the queue unattended; rule 5 makes the project plan honest. The migration milestones and the dependencies on suppliers can still be tracked with predictive tools (see Predictive Planning); agile and predictive planning combine well here. Pairing Emeka with Tomasz on migration tasks also spreads firewall knowledge, so the team does not depend on one person for every firewall question.

Agile and ITIL

Agile and ITIL are sometimes presented as opposites: one fast and informal, the other slow and bureaucratic. In practice they answer different questions. ITIL describes what an organization needs to manage to deliver IT services well: incidents, problems, changes, service levels, knowledge. Agile describes how teams organize their work: in short cycles, with feedback, limited work in progress and continuous improvement. ITIL's own guiding principles include progressing iteratively with feedback (see ITSM in Practice).

• Incident management stays the same process; incidents simply appear as expedite or standard items on the team's board (see Incident Management).
• Change enablement is where the two meet most visibly. Frequent, low-risk changes, such as a routine firewall rule or a regular release of a team's application, can become pre-authorized standard changes that a team carries out without a change advisory board (see Change Enablement). That keeps control where risk is real and removes waiting where it is not.
• Continual improvement and the Sprint Retrospective or Kanban's improvement practice are the same idea at different scales (see Continual Improvement).

Agile, DevOps and SRE

DevOps extends agile ideas beyond the development team: developers and operations share responsibility for running what is built, and automation (continuous integration and delivery, infrastructure as code) replaces manual handoffs (see DevOps and CI/CD). Site reliability engineering (SRE) adds numbers for balancing speed and stability (see ITSM in Practice):

• Toil, the manual, repetitive work that grows with the service, is measured and capped so that time remains for engineering it away (see Toil). On an ops board, toil reduction is typical intangible work.
• Error budgets turn reliability targets into a planning rule: while the budget lasts, the team can ship features; when it is spent, reliability work goes first (see SLOs and error budgets). For a Product Owner, that is an input for ordering the backlog. The Observability & SRE topic covers SRE in depth, including how an error budget policy is written down.

Metrics for operations teams

Ops teams can combine two sets of measures. The flow metrics from Kanban show how the team's work moves; the incident metrics from service management show how the service behaves:

How to calculate the flow metrics is shown on Kanban, and mean time to restore on Incident Management. As with any metric, use them to improve the system, not to rank engineers: a team that is judged on ticket counts soon learns to split tickets.