Three Ways to Run an Outage: IncidentOps Styles Compared

Three Ways to Run an Outage: IncidentOps Styles Compared

Ask a room of DevOps learners how their last workplace handled an outage and you will get three kinds of stories. Some describe a heroic engineer who saved the day through sheer stubbornness. Some describe a tidy process with a rota and a message template. A few describe something closer to an emergency-services crew, with a commander, a scribe and a spokesperson. These are three different styles of IncidentOps, which is what teams do, deliberately or not, whenever a live service fails without warning. Setting them next to each other helps you spot each one in the wild and decide where to aim, whether you are job-hunting, studying or leading a team.

Style one: heroics

The Ops Atlas starts its maturity ladder with a level called Heroic. Nobody has written down severity levels or on-call rules, and success depends on whoever is awake and brave. The upside is speed in small matters: one capable person can fix a minor issue before a meeting would even finish. The downside is that it does not scale, burns out the people who are good at it, and teaches the organisation nothing. When the hero is on holiday, the next incident takes twice as long.

Where it shines: tiny teams with simple systems and a low rate of trouble.

Where it fails: anything that crosses team boundaries, anything that lasts more than an hour, and anything that needs customers to be told what is happening.

Style two: the defined process

The next step up is a written severity policy, a named rota and a standard message. Teams at this stage are what the Atlas labels Defined. A single on-call schedule exists, there is a template for updates, and people know who to call. In the Atlas’s tier framework, this corresponds to a team with one rota and a template, and it is a perfectly sensible place to stop for a while. Many organisations with a handful of services never need more.

Where it shines: teams that see occasional incidents and need consistency without much overhead.

Where it fails: serious events with many responders, where nobody has been told who coordinates, who touches production and who talks to customers.

Style three: a coordinated command structure

The third style brings in the roles borrowed from emergency management. A commander coordinates but does not fix. An operations group is the only set of people changing production. A communications lead posts on a schedule. A deputy stands by, a scribe records, and liaisons speak for other teams. Sub-teams form once the headcount passes about five. The Atlas suggests this fits organisations whose incidents cross team lines or that see two or more serious incidents per month.

Where it shines: larger systems, regulated sectors and any setting where a long outage must be coordinated across shifts.

Where it fails: when adopted too early. A five-person startup does not need a deputy and three liaisons. Ceremony without need breeds resentment.

Head-to-head comparison

  • Declaration: heroics rely on feeling; the defined process uses written triggers; the command structure adds an explicit declared state that opens a channel and assigns roles.
  • Roles: heroics have none; the defined process has an on-call person; the command structure uses distinct hats with a commander, an operations lead and a communicator.
  • Communication: heroics are silent or ad hoc; the defined process posts a template; the command structure keeps a cadence, with an early first note, a scoping post soon after and refreshed updates through the first two hours.
  • Handover: heroics hope the next person guesses; the defined process hands over with notes; the command structure uses a spoken confirmation.
  • Review: heroics skip it; the defined process holds a meeting; the command structure names an owner before the call ends and tracks follow-up actions against deadlines.

The two higher levels on the ladder

The Atlas continues the ladder beyond the command style. A level it calls Coordinated is reached when the update cadence is kept in ninety percent of the most severe incidents. The next level, Learning, is reached once four in five priority follow-ups have been finished by their deadlines across two quarters in a row. A fifth level, Adaptive, describes teams that adjust their own process from evidence. These are proposals from the Atlas, not industry standards, but they give you concrete targets rather than a vague aspiration to “get better”.

What this means for your career

If you are studying, learn to recognise the styles in job descriptions. A posting that mentions on-call rotas and runbooks hints at the defined style. One that talks about incident commanders, severity policies and post-incident reviews points higher up the ladder. Prepare to talk about both. Hiring panels reward candidates who can weigh pros and cons rather than recite a fashionable process.

Building your own judgement matters here. Our article on sharpening the critical thinking that DevOps problem-solving demands is a good complement, because choosing the right amount of process is itself a problem-solving exercise. If you are filling in the background first, the knowledge gaps students tend to leave open will help you decide what to study before you step into an on-call rota.

Choosing the right style for a team

A quick guide: if you rarely have incidents and everyone can fit in a single chat thread, heroics with a short written policy may do. If incidents arrive monthly and cross a boundary or two, aim for the defined style with a template and a status page. If they are frequent, serious, or involve many people at once, move to the command structure, but introduce one role at a time. Start with the communicator, since silence is usually the first complaint. Add a commander next. Bring in the rest as headcount requires.

Whatever you choose, keep the paging path and the status page independent from the systems they report on. That is true at every level and costs little.

A short scenario to test yourself

Suppose a login service starts failing at four in the afternoon and customers are reporting it. In the heroic style, one engineer notices, starts digging and tells the team after forty minutes. In the defined style, the on-call person is paged, posts the template and escalates when stuck. In the command style, somebody declares, a commander is named within minutes, the operations lead proposes a rollback while the communicator posts the first note, and a scribe starts the timeline. Which would you rather be on at midnight? Which would you rather explain to a customer afterwards? Answering honestly tells you what to practise next.

One more comparison: how reviews differ

Heroic teams tend to hold no review, or they hold one that ends with a single person being blamed. Defined teams hold a meeting but often count tickets rather than checking outcomes. Coordinated teams pick a facilitator from outside the thick of it, limit the list to a few actions with one owner and one date each, and track closure. The difference in learning is large, even though the meeting length is similar.

Read the source

The full ladder, the tiering rules and the 30-60-90 adoption plan that goes with them all appear in the IncidentOps entry of the Atlas. Read it with a pencil, place your current or most recent team on the ladder, and note one practice that would move you up a step. That small exercise is more useful than any argument about which style is best in the abstract.