The Friday Restore Drill: A Platform Team’s BackupOps Story

The Friday Restore Drill: A Platform Team’s BackupOps Story

Nobody in the following story exists; it is a sketch of how a study habit can harden into a working habit. Imagine a platform group of six people inside an online-courses company of moderate size. They have a dashboard full of green ticks, a tidy wiki and, as far as anyone knows, a working backup system for their course database. Then a new trainee asks a question nobody can answer: “When did we last get anything back from it?”

The question that made the room go quiet

The trainee had just finished a module on operational reliability and was curious in the way new people are. The team lead looked at the others. The senior engineer said it had been fine the last time. The person who set the jobs up two years earlier had moved to another team. Nobody had an actual date.

The lead did something sensible and slightly uncomfortable: she wrote “last restore: unknown” on the whiteboard, and asked the trainee to help find out. That small act, putting an unknown in plain sight, is where the story really begins. It is also close to the stance taken in the Ops Atlas’s BackupOps entry, which suggests that any asset without recent restore evidence should be treated as unprotected until shown otherwise.

The first drill goes badly

On a Friday afternoon, the pair restored the newest copy of the course database to a scratch server. It took longer than expected, because the copy was large and the restore ran as a single stream. When it finished, a table of lesson progress was empty. Digging in, they found that a scheduled job had been writing to a path that had changed after a storage migration. The backup tool had been reporting success for weeks, since it dutifully backed up an empty folder.

The group did not panic. They did what good teams do with an uncomfortable discovery: they made it boring. The trainee wrote a short note on what had been found, the lead logged it as an incident against the most important tier, and the senior engineer pulled the previous month’s copy to retrieve what they needed. They lost a few days of progress data, which was irritating but survivable, and they decided not to call it a disaster. They called it a lesson that arrived cheaply.

What the team changed in the first month

Their plan followed a pattern the Atlas lays out for a thirty-day start, adapted to their size.

  1. Inventory and owners. They listed every dataset, from the course database to the certificate generator’s templates, and put one name beside each. The trainee was named owner of a low-risk internal wiki, partly as a learning exercise.
  2. Tiers. They ranked the twenty most important items and marked three as tier one.
  3. Alert on silence. Instead of waiting for error emails, they configured an alert that fires when a success message fails to arrive on schedule, routed to a chat channel hosted outside their main platform.
  4. A hand restore. They repeated Friday’s drill for another tier-one asset, this time timing each phase.
  5. A first number. They published the share of tier-one assets with a successful recent restore. It was one in three. Nobody enjoyed saying so, and everybody respected it.

Learning, not just operating

This is where the story connects to career growth. The trainee’s curiosity was the trigger, but the team’s habit of writing everything down turned it into a skill. Readers who are building their own foundations may enjoy our piece on the knowledge gaps that students often need to close, since recovery work tends to sit in exactly the space between textbooks and production. The post on developing sharper problem-solving instincts covers the thinking style the drill rewarded: ask what evidence exists, then go and get it.

Month two: separating the copies

The next gap was independence. Their copies lived in the same cloud account as the production database, reachable by the same deployment credentials. In the Atlas’s four-question model, the first question is whether a copy is independent of what it protects, and theirs was not. They created a separate account, gave the backup tool write-only access, and restricted delete rights to two people who did not deploy code. Recovery keys went into an escrow arrangement outside the production environment, with a printed procedure in a sealed envelope. Later the lead added a locked retention vault in the reversible mode, deciding to postpone the irreversible option until the policy settled.

They also timed a second full restore and compared it to what they had promised the business. With four parallel streams the transfer dropped by about three quarters, though adding the application checks made the total less impressive than the first estimate. That gap was useful, because it stopped the team from promising a number they had not measured.

Where they ended up

Three months on, the team ran a monthly sample restore, a quarterly full restore of one tier-one asset and an alert for missing success. Their proven-recoverable figure rose from one third to every tier-one asset. The trainee, now a regular, wrote the runbook in the wiki and ran the next drill without supervision. At the Atlas’s maturity ladder, they had moved from hope toward owned and isolated copies, with the top rungs still ahead.

The takeaways, in the order they happened

  • A newcomer’s honest question was worth more than a year of dashboards.
  • Making the unknown visible is the first piece of engineering.
  • A drill that fails on Friday is cheaper than an outage that fails on Monday.
  • Number first, polish later: publish the first measurement even when it looks bad.

If you are studying operations, borrow the whiteboard trick. Write “last restore: unknown” next to a system you care about, then be the person who replaces the unknown with a date.