{"id":1902,"date":"2026-11-06T19:50:00","date_gmt":"2026-11-06T19:50:00","guid":{"rendered":"https:\/\/devopsschool.org\/blog\/?p=1902"},"modified":"2026-11-06T19:50:00","modified_gmt":"2026-11-06T19:50:00","slug":"alert-hygiene-audit-checklist","status":"publish","type":"post","link":"https:\/\/devopsschool.org\/blog\/alert-hygiene-audit-checklist\/","title":{"rendered":"Alert Hygiene Audit: A Half-Day Checklist for DevOps Teams"},"content":{"rendered":"<p>An alert hygiene audit is the most underrated maintenance task in a DevOps team&#8217;s calendar. Rules accumulate with every new service, every postmortem and every &#8220;let&#8217;s make sure this never happens again&#8221; conversation, but they are almost never removed. The result, a year or two later, is a paging setup that fires so often that responders acknowledge by reflex and real incidents are noticed late. This checklist is designed to be worked through in a single half-day session with the people who actually carry the pager. Each item has a pass condition, so the outcome is a concrete list of changes rather than a general agreement that things are noisy.<\/p>\n<h2>Before the alert hygiene audit: gather the inputs<\/h2>\n<p>Do not start with opinions. Start with data. Export at least thirty days of alert history from your monitoring and paging tools, including, for each firing: rule name, service, severity, start and end time, who acknowledged it, and whether a follow-up action (a ticket, a commit, a config change, a rollback) was recorded.<\/p>\n<ul>\n<li>[ ] Alert history exported for the last 30\u201390 days<\/li>\n<li>[ ] Current rule definitions exported, with owners where known<\/li>\n<li>[ ] On-call rota for the same period, so pages can be mapped to people<\/li>\n<li>[ ] List of incidents declared in the period, to check what was caught and what was missed<\/li>\n<\/ul>\n<p>If the team is unsure where the line sits between monitoring and alerting, a short refresher on <a href=\"https:\/\/devopsschool.org\/blog\/alerting\/\">what alerting means and how it is typically used<\/a> helps align vocabulary before the session.<\/p>\n<h2>Section 1: Volume and actionability<\/h2>\n<ul>\n<li>[ ] <strong>Pages per on-call shift calculated.<\/strong> Pass: the number is low enough that each page can realistically be investigated, not just acknowledged. Many teams aim for a small handful per shift at most.<\/li>\n<li>[ ] <strong>Action rate calculated per rule.<\/strong> For each rule, divide firings that led to a recorded action by total firings. Pass: every paging rule has a non-trivial action rate.<\/li>\n<li>[ ] <strong>Top ten noisiest rules listed.<\/strong> These usually account for a disproportionate share of volume. Pass: each has a decision recorded (keep, tune, downgrade, delete).<\/li>\n<li>[ ] <strong>Self-resolving alerts identified.<\/strong> Rules that fire and clear within a few minutes without intervention. Pass: each has a persistence condition added or is downgraded.<\/li>\n<\/ul>\n<h2>Section 2: Duplicates and grouping<\/h2>\n<ul>\n<li>[ ] <strong>Repeat notifications for one ongoing condition checked.<\/strong> Pass: an active condition produces one notification with updates, not a new page every evaluation cycle.<\/li>\n<li>[ ] <strong>Grouping keys reviewed.<\/strong> If you use Prometheus with Alertmanager, check the group_by labels and group timing settings. Pass: alerts sharing a cluster, service and cause arrive as a single grouped notification.<\/li>\n<li>[ ] <strong>Upstream\/downstream relationships recorded.<\/strong> Pass: when a shared dependency fails, downstream symptom alerts are inhibited or folded into the upstream incident.<\/li>\n<li>[ ] <strong>Multiple tools alerting on the same signal identified.<\/strong> Pass: each condition pages from exactly one source.<\/li>\n<\/ul>\n<h2>Section 3: Severity and routing<\/h2>\n<ul>\n<li>[ ] <strong>Written definition of &#8220;page-worthy&#8221; exists.<\/strong> Pass: a short statement agreed by the team, typically limited to active user impact, imminent data loss, or a hard limit being reached before the next business day.<\/li>\n<li>[ ] <strong>Every paging rule mapped against that definition.<\/strong> Pass: rules that do not meet it are moved to a ticket queue or daily summary.<\/li>\n<li>[ ] <strong>Separate channels for separate urgencies.<\/strong> Pass: interrupting pages, working-hours tickets and informational notices do not share a channel or a notification sound.<\/li>\n<li>[ ] <strong>Out-of-hours paging reviewed separately.<\/strong> Pass: nothing pages at night unless it meets the definition and cannot wait until morning.<\/li>\n<\/ul>\n<p>Rota design is closely tied to this section, and this overview of <a href=\"https:\/\/devopsschool.org\/blog\/on-call\/\">how on-call responsibility is organised in practice<\/a> is useful background for anyone new to carrying the pager.<\/p>\n<h2>Section 4: Quality of each remaining alert<\/h2>\n<ul>\n<li>[ ] <strong>Symptom-based where possible.<\/strong> Pass: paging rules describe what users experience (errors, latency, failed jobs) rather than raw resource levels alone.<\/li>\n<li>[ ] <strong>Runbook link present.<\/strong> Pass: every paging rule links to a page stating what the alert means and the first two or three checks.<\/li>\n<li>[ ] <strong>Named owner assigned.<\/strong> Pass: each rule has a team or individual responsible for tuning it.<\/li>\n<li>[ ] <strong>Message text readable at 3 a.m.<\/strong> Pass: the alert title states the affected service and the impact in plain words, not just a metric name.<\/li>\n<\/ul>\n<h2>Section 5: Coverage check<\/h2>\n<p>Cutting noise must not create blind spots. Cross-reference the declared incidents from the input stage.<\/p>\n<ul>\n<li>[ ] <strong>Each real incident was detected by an alert.<\/strong> Pass: or, where it was reported by a user first, a gap is logged with a proposed rule.<\/li>\n<li>[ ] <strong>No retired rule would have caught a past incident.<\/strong> Pass: confirmed for every rule marked for deletion.<\/li>\n<li>[ ] <strong>Maintenance windows handled.<\/strong> Pass: planned work uses silences or suppression so it does not generate pages.<\/li>\n<\/ul>\n<p>A clean alert set also makes the wider response process smoother, since responders start each incident with fewer distractions; this summary of <a href=\"https:\/\/devopsschool.org\/blog\/incident-management\/\">how teams structure incident handling from detection to resolution<\/a> shows where alerting sits in that flow.<\/p>\n<h2>Section 6: Keep it from drifting back<\/h2>\n<ul>\n<li>[ ] <strong>Review cadence scheduled.<\/strong> Pass: a recurring audit, monthly or quarterly, is on the calendar.<\/li>\n<li>[ ] <strong>New rule checklist adopted.<\/strong> Pass: any new paging rule must state its owner, runbook, and why it meets the page-worthy definition before merge.<\/li>\n<li>[ ] <strong>On-call handover notes include noise.<\/strong> Pass: outgoing responders flag any rule that fired without needing action.<\/li>\n<\/ul>\n<h2>Further reading<\/h2>\n<p>For teams whose estates are large enough that manual grouping rules become hard to maintain, automated correlation is the next step. Engineers weighing that route will find write-ups on grouping and filtering alerts across big estates at <a href=\"https:\/\/www.theaiops.com\/\">TheAIOps.com, which focuses on AIOps<\/a>. Whatever tooling you use, the principles on this checklist remain the foundation: fewer pages, each one meaningful, owned and actionable.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Paging rules pile up and rarely get removed. A half-day audit checklist with pass conditions to cut duplicate pages, retire dead rules and protect on-call.<\/p>\n","protected":false},"author":5,"featured_media":1903,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[486],"tags":[],"class_list":["post-1902","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts\/1902","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/comments?post=1902"}],"version-history":[{"count":1,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts\/1902\/revisions"}],"predecessor-version":[{"id":1924,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/posts\/1902\/revisions\/1924"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/media\/1903"}],"wp:attachment":[{"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/media?parent=1902"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/categories?post=1902"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devopsschool.org\/blog\/wp-json\/wp\/v2\/tags?post=1902"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}