Alert Hygiene Audit: A Half-Day Checklist for DevOps Teams

Parth Gill November 6, 2026

An alert hygiene audit is the most underrated maintenance task in a DevOps team’s calendar. Rules accumulate with every new service, every postmortem and every “let’s make sure this never happens again” conversation, but they are almost never removed. The result, a year or two later, is a paging setup that fires so often that responders acknowledge by reflex and real incidents are noticed late. This checklist is designed to be worked through in a single half-day session with the people who actually carry the pager. Each item has a pass condition, so the outcome is a concrete list of changes rather than a general agreement that things are noisy.

Before the alert hygiene audit: gather the inputs

Do not start with opinions. Start with data. Export at least thirty days of alert history from your monitoring and paging tools, including, for each firing: rule name, service, severity, start and end time, who acknowledged it, and whether a follow-up action (a ticket, a commit, a config change, a rollback) was recorded.

[ ] Alert history exported for the last 30–90 days
[ ] Current rule definitions exported, with owners where known
[ ] On-call rota for the same period, so pages can be mapped to people
[ ] List of incidents declared in the period, to check what was caught and what was missed

If the team is unsure where the line sits between monitoring and alerting, a short refresher on what alerting means and how it is typically used helps align vocabulary before the session.

Section 1: Volume and actionability

[ ] Pages per on-call shift calculated. Pass: the number is low enough that each page can realistically be investigated, not just acknowledged. Many teams aim for a small handful per shift at most.
[ ] Action rate calculated per rule. For each rule, divide firings that led to a recorded action by total firings. Pass: every paging rule has a non-trivial action rate.
[ ] Top ten noisiest rules listed. These usually account for a disproportionate share of volume. Pass: each has a decision recorded (keep, tune, downgrade, delete).
[ ] Self-resolving alerts identified. Rules that fire and clear within a few minutes without intervention. Pass: each has a persistence condition added or is downgraded.

Section 2: Duplicates and grouping

[ ] Repeat notifications for one ongoing condition checked. Pass: an active condition produces one notification with updates, not a new page every evaluation cycle.
[ ] Grouping keys reviewed. If you use Prometheus with Alertmanager, check the group_by labels and group timing settings. Pass: alerts sharing a cluster, service and cause arrive as a single grouped notification.
[ ] Upstream/downstream relationships recorded. Pass: when a shared dependency fails, downstream symptom alerts are inhibited or folded into the upstream incident.
[ ] Multiple tools alerting on the same signal identified. Pass: each condition pages from exactly one source.

Section 3: Severity and routing

[ ] Written definition of “page-worthy” exists. Pass: a short statement agreed by the team, typically limited to active user impact, imminent data loss, or a hard limit being reached before the next business day.
[ ] Every paging rule mapped against that definition. Pass: rules that do not meet it are moved to a ticket queue or daily summary.
[ ] Separate channels for separate urgencies. Pass: interrupting pages, working-hours tickets and informational notices do not share a channel or a notification sound.
[ ] Out-of-hours paging reviewed separately. Pass: nothing pages at night unless it meets the definition and cannot wait until morning.

Rota design is closely tied to this section, and this overview of how on-call responsibility is organised in practice is useful background for anyone new to carrying the pager.

Section 4: Quality of each remaining alert

[ ] Symptom-based where possible. Pass: paging rules describe what users experience (errors, latency, failed jobs) rather than raw resource levels alone.
[ ] Runbook link present. Pass: every paging rule links to a page stating what the alert means and the first two or three checks.
[ ] Named owner assigned. Pass: each rule has a team or individual responsible for tuning it.
[ ] Message text readable at 3 a.m. Pass: the alert title states the affected service and the impact in plain words, not just a metric name.

Section 5: Coverage check

Cutting noise must not create blind spots. Cross-reference the declared incidents from the input stage.

[ ] Each real incident was detected by an alert. Pass: or, where it was reported by a user first, a gap is logged with a proposed rule.
[ ] No retired rule would have caught a past incident. Pass: confirmed for every rule marked for deletion.
[ ] Maintenance windows handled. Pass: planned work uses silences or suppression so it does not generate pages.

A clean alert set also makes the wider response process smoother, since responders start each incident with fewer distractions; this summary of how teams structure incident handling from detection to resolution shows where alerting sits in that flow.

Section 6: Keep it from drifting back

[ ] Review cadence scheduled. Pass: a recurring audit, monthly or quarterly, is on the calendar.
[ ] New rule checklist adopted. Pass: any new paging rule must state its owner, runbook, and why it meets the page-worthy definition before merge.
[ ] On-call handover notes include noise. Pass: outgoing responders flag any rule that fired without needing action.