{"id":2087,"date":"2026-11-16T09:50:00","date_gmt":"2026-11-16T09:50:00","guid":{"rendered":"https:\/\/devopsschool.org\/blog\/?p=2087"},"modified":"2026-11-18T19:50:00","modified_gmt":"2026-11-18T19:50:00","slug":"drill-keeps-missing-rto","status":"publish","type":"post","link":"https:\/\/devopsschool.org\/blog\/drill-keeps-missing-rto\/","title":{"rendered":"When Your Recovery Drill Keeps Missing Its Target: A Diagnosis"},"content":{"rendered":"<p>The symptom is familiar to every learner who has moved from tutorials to a real environment: the failover drill ran, everybody worked hard, and the stopwatch still says the recovery took longer than the target. Here we handle it as a diagnosis exercise. We state the symptom, test four likely causes in order, and match each with a fix, using the DisasterRecoveryOps guidance in the Ops Atlas as our checklist.<\/p>\n<h2>The symptom, stated precisely<\/h2>\n<p>&#8220;Our recovery is too slow&#8221; is not a symptom you can debug. Write it as numbers. Total elapsed time from first failure to confirmed working service; the target; the gap. If your target is sixty minutes and the drill took sixty-four, you are four minutes over, which is a very different problem from being three hours over. The Atlas uses exactly that example, with a drill that missed by four.<\/p>\n<p>Now break the total into segments: noticing, deciding, executing, verifying. Most teams only time the third. The worked example in the Atlas lists six minutes of detection, twenty-five of deciding and thirty-three of technical work, so the slow part was not the machinery.<\/p>\n<h2>Cause one: the decision took too long<\/h2>\n<p><strong>How to confirm:<\/strong> look at the gap between the first alert and the moment someone with authority said go. If it exceeds the allowance computed from your objective, this is your main culprit. In the sample, deciding took twenty-five minutes against sixteen allowed, an overrun of nine.<\/p>\n<p><strong>Fix:<\/strong> publish a latest-start time, equal to the objective less your slowest recent technical drill less a margin, and name who may invoke the plan. Remove the approval chain where possible, or pre-authorise spend up to a limit. A bit of the same analytical discipline is taught in our article on <a href=\"https:\/\/devopsschool.org\/blog\/devops-problem-solving-how-to-build-critical-thinking-skills\/\">sharpening the reasoning skills behind DevOps troubleshooting<\/a>; apply it here by asking &#8220;what evidence would change our decision?&#8221; before the drill, not during.<\/p>\n<h2>Cause two: the technical sequence is not what you think it is<\/h2>\n<p><strong>How to confirm:<\/strong> have an observer log every step as performed, including improvisation, and compare with the written runbook. Watch for console clicking from memory, which the Atlas lists as a recovery anti-pattern since it is slow, unreviewable and fragile under stress.<\/p>\n<p><strong>Fix:<\/strong> script the sequence in the safe order: stop the old side first, make the data copy writable next, steer users last. Leaving out the first move invites split-brain, and the Atlas cautions that failover on health checks alone should be used carefully for that reason.<\/p>\n<h2>Cause three: hidden dependencies stall the move<\/h2>\n<p><strong>How to confirm:<\/strong> list the waits. Did launching capacity in the recovery region take longer than expected? Was an image or quota out of date? Did a sign-in step fail because it relied on the very platform that was down? The Atlas recounts a 2025 regional event where authentication services were among the casualties, and a 2023 event in which a regional identity service was offline for about thirteen hours.<\/p>\n<p><strong>Fix:<\/strong> maintain a dependency inventory and refresh it as part of the plan check. Hold break-glass logins for the recovery side in a secrets store hosted away from the platform that is in trouble. Pre-provision or pre-warm the capacity whose creation depends on a control plane you might not be able to reach. The Atlas notes that stale quotas and images are common culprits, so treat them as a recurring audit item.<\/p>\n<h2>Cause four: the standby style does not fit the target<\/h2>\n<p><strong>How to confirm:<\/strong> compare the objective with what the chosen style can plausibly achieve. Rebuilding from backups will rarely deliver a one-hour outcome for a large estate.<\/p>\n<p><strong>Fix:<\/strong> move the service up one level, or relax the objective with the business owner&#8217;s agreement. In the Atlas sample tier matrix, a one-hour objective goes with a warm or hot standby plus switchovers every quarter, while eight hours fits a pilot light and seventy-two hours fits backup and restore.<\/p>\n<p>One more point on verification, which teams often forget to time. After traffic moves, someone has to confirm a real customer journey works end to end: log in, place a test order, read it back. If that confirmation is vague, the clock keeps ticking in everyone&#8217;s head even though the systems are fine. Agree in advance on the single check that counts as &#8220;back&#8221;.<\/p>\n<h2>A fifth suspect: scope creep in your objectives<\/h2>\n<p>Sometimes the target is the problem. Asking for no downtime and no data loss across every system appears in the Atlas anti-pattern list, because nobody can fund it, and the plan quietly fails the first time it meets budget. Tier the services and set honest numbers for each.<\/p>\n<h2>A worked pass through the checklist<\/h2>\n<p>Suppose a lab drill takes seventy minutes against a sixty-minute target. The timeline shows ten minutes to notice, twenty to decide, thirty-five to execute and five to verify. The biggest single block is execution, but the decision block alone exceeds a sensible allowance. Fix the decision first, because it is cheap; then examine why execution needed thirty-five minutes. In many labs the answer is a couple of manual steps and one slow capacity launch, both of which respond to scripting and pre-provisioning. These figures are made up for illustration; swap in your own.<\/p>\n<h2>The checklist in one place<\/h2>\n<ol>\n<li>Express the miss in minutes and split it into four segments.<\/li>\n<li>Check decision latency against its allowance.<\/li>\n<li>Compare the observed steps with the runbook.<\/li>\n<li>List every wait caused by a dependency.<\/li>\n<li>Confirm the standby style fits the tier.<\/li>\n<li>Re-run the drill and log the change.<\/li>\n<\/ol>\n<h2>What this means for your career<\/h2>\n<p>Engineers who can walk into a failed drill, split the timeline and name the real bottleneck are in short supply. The skill transfers to every kind of incident and sits comfortably within the wider capabilities discussed in <a href=\"https:\/\/devopsschool.org\/blog\/devops-for-it-transformation-practical-skills-roadmap-and-strategy\/\">a practical skills roadmap for DevOps-led IT change<\/a>. If you are studying, practise on a lab: build a tiny service, create a replica, time a switch, and write up the result.<\/p>\n<p>For the complete discipline and its supporting standards, read <a href=\"https:\/\/www.xopsschool.com\/disasterrecoveryops\/\">the Ops Atlas page on DisasterRecoveryOps<\/a>, then take its worked scorecard and fill it in with your own lab figures. Re-run the drill after each fix, because the second run often reveals that the bottleneck has moved elsewhere. That is not a failure; it is how a slow recovery becomes a predictable one.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Four likely reasons a failover drill overruns its target, how to confirm each from the timeline, and the fix that matches, in the order worth testing.<\/p>\n","protected":false},"author":5,"featured_media":2088,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[489],"tags":[],"class_list":["post-2087","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops-careers"],"_links":{"self":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts\/2087","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/comments?post=2087"}],"version-history":[{"count":1,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts\/2087\/revisions"}],"predecessor-version":[{"id":2211,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts\/2087\/revisions\/2211"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/media\/2088"}],"wp:attachment":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/media?parent=2087"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/categories?post=2087"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/tags?post=2087"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}