Downtime Alerting Setup: Thresholds, Routing and Testing

A good downtime alerting setup does four things. It checks your site often enough to catch an outage, confirms the failure before it speaks, sends the message to someone who can act, and stays quiet the rest of the time. Most bad setups fail at the second and fourth steps: they page on a single failed request, and within a month everyone has muted the channel.

This guide walks through the setup in the order you will build it: what to monitor, how to set thresholds, where alerts should go, how to handle planned maintenance, and how to test the whole chain before a real incident does it for you.

TL;DR

• Monitoring finds the problem and alerting decides who hears about it. Tune both, because a perfect check with a bad alert rule still wakes the wrong person.
• Require a failure to persist across at least two consecutive checks before you notify anyone, so a single dropped request never pages you.
• Page a human only for urgent, actionable problems that affect users. Everything else belongs in a dashboard or a low-noise channel.
• Use two or three escalation tiers by duration, for example chat first, then a phone alert if the outage continues.
• Test the setup with a deliberate failure, and silence alerts during planned maintenance instead of ignoring them.

Table of Contents

• How monitoring and alerting work together
• Decide what a failure means before you set an alert
• Set incident alert thresholds that hold up
• Route alerts to the right people
• Handle planned maintenance without noise
• Perspective: what we have learned about alerting
• Test your downtime alerting setup
• Set it up with Pingfloat
• A starting configuration you can copy
• FAQ
• Sources

How monitoring and alerting work together

Monitoring and alerting are two separate jobs that people often lump together. Monitoring is the probe: a request goes to your URL on a schedule and the result is recorded. Alerting is the decision layer: it looks at those results, applies rules, and decides whether anyone needs to be told.

That split matters because you tune them differently. Monitoring settings control what you can see: the URL, the interval, the expected status code, the timeout. Alerting settings control what you are told: how many failures count as an outage, who is notified, through which channel, and what happens if nobody responds.

A typical flow looks like this:

1. A probe requests your page and gets a timeout or an error status.
2. The system retries, often from the same or another location, to rule out a network blip.
3. If the failure persists past your rule, an incident opens and the first notification goes out.
4. If the incident is not acknowledged or resolved within a set time, a second, louder notification goes to a wider group.
5. When checks pass again, a recovery message closes the loop.

If you have not yet set up the monitor itself, start with our walkthrough on website uptime monitoring in 15 minutes, then come back here to tune the alerts. For a shorter reference, our website downtime alerts checklist covers the basics in list form.

Decide what a failure means before you set an alert

An alert is only as good as the definition of "down" behind it. A server that returns something is not necessarily a server that works.

Pick the right check target

Monitor the page or endpoint that proves the system works, not just the one that is easiest to hit. A static homepage served from a cache can return a healthy response while your login or checkout is broken. Where you can, add a lightweight health endpoint that touches the database and any critical dependency, and monitor that too.

Define the expected response

Most checks expect a 200 OK status. Decide up front what else counts as failure:

• 5xx responses. A 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable or 504 Gateway Timeout is a server-side failure and should count as down.
• Timeouts. No response within your limit is a failure even if the server eventually answers.
• Wrong content. If your monitor supports keyword matching, check for a string that only appears when the page renders correctly.
• Redirect loops and 4xx errors on pages that should be public. A 401 or 400 on a public URL usually means a configuration change broke something.

The HTTP status codes reference helps when you need to decide which codes belong on the failure list for a given endpoint.

Choose a sensible check interval

Faster is not always better. Google's SRE book notes that for a web service targeting 99.9% annual uptime, probing for a 200 more than once or twice a minute is probably unnecessarily frequent. For most sites, a check every one to five minutes is a reasonable balance between detection speed and noise. Tighten it only for endpoints where a minute of downtime has a real cost.

Set incident alert thresholds that hold up

Thresholds are where most downtime alerting setups succeed or fail. There are three levers: how many failures, how long, and how severe.

Require persistence, not a single failure

The simplest improvement you can make is to refuse to alert on one failed check. Google's SRE guidance on practical alerting recommends that an alert condition persist for a minimum duration, typically at least two evaluation cycles, which cuts false alarms from flapping or a missed collection.

In practice, if you check every minute, require two or three consecutive failures. If you check every five minutes, two consecutive failures means up to ten minutes before you hear about it, so consider shortening the interval for critical endpoints rather than lowering the failure count to one.

Tier alerts by how long the outage lasts

Duration is the cleanest way to separate a blip from an incident. A workable starting pattern:

These numbers are a starting point, not a rule. A hobby project can tolerate longer gaps. A payment endpoint should escalate sooner. The point is that the alert gets louder the longer the problem lasts.

Alert on user impact, not on every symptom

If you track a service level objective, alert on how fast you are burning through your error budget rather than on every failed probe. Google's SRE workbook describes alerting on SLOs so teams are notified about significant events that consume a large fraction of the error budget. It also frames alert quality in terms of precision and recall: precision is the share of alerts that were real problems, and recall is the share of real problems that triggered an alert. A setup with low precision cries wolf. One with low recall misses outages. You want both high.

A quick self-audit: look at the last month of alerts and count how many needed human action. If fewer than most did, your thresholds are too tight.

Use more than one location

A single probe location cannot tell the difference between "your site is down" and "the path between the probe and your site is broken." Where your monitor allows it, confirm a failure from a second location before alerting. This single step removes a large share of false positives caused by regional network trouble. If you suspect a routing problem rather than a server problem, a traceroute shows where packets stop, and our guide to reading traceroute output hop by hop explains what each line means.

Route alerts to the right people

Thresholds decide when you speak. Routing decides who hears it. A flawless threshold sent to a shared inbox nobody reads is still a missed outage.

Match the channel to the urgency

Use a small set of channels and give each a clear purpose:

• Email: a record and a low-urgency heads-up. Never the only channel for a critical service.
• Team chat: early warnings and shared visibility. Good for "someone should look at this."
• Phone, SMS or push: reserved for incidents that need someone now.
• Webhook or API: feeds a ticketing system or an on-call tool, and keeps a log.

If you want to feed alerts into your own tooling, the Pingfloat Public API is the place to start.

Page for the right reasons

Google's SRE book advises paging only for urgent, actionable, user-relevant problems. A human interrupt is expensive, so it should be spent on conditions that are new, urgent and something a person can actually fix. If the responder cannot do anything meaningful, the event belongs in a dashboard or a quieter channel. For a practical take on keeping urgent notifications for novel problems that affect users, this guide to alert mailboxes shows one way to separate signal from routine messages.

A useful test before you create any page-level alert: "If this fires at 3 a.m., what exactly will the person do?" If you cannot answer in one sentence, downgrade it.

Group, deduplicate and name owners

When one root cause breaks five endpoints, five separate pages feel like five problems. Google's practical-alerting chapter describes routing notifications to the correct destination and deduplicating, inhibiting or grouping related alerts. Even without a dedicated tool, you can approximate this:

• Put checks for one service in one group, with one notification target.
• Designate a primary and a backup for each service, by name, not by team alias alone.
• Include in every alert the URL, the failing status or error, the time it started and a link to a runbook or checklist.

Set up an on-call rotation, even a small one

For a two-person team, a weekly rotation with a documented handover is enough. The aim is that one person is clearly responsible at any given hour, so that nobody assumes someone else saw it.

Handle planned maintenance without noise

Planned work is the most common source of alerts that nobody should receive. Deployments, database migrations and certificate swaps all cause brief failures you already know about.

• Silence, do not delete. Pause or schedule a maintenance window for the affected checks rather than removing them, so monitoring resumes automatically.
• Schedule it in advance. A window with a fixed start and end is safer than manually muting, because people forget to unmute.
• Keep the window tight. Cover the expected work plus a short buffer. A window that runs all night hides real problems.
• Check after the window closes. If the site still fails when the window ends, you want the alert to fire at once, and it should.
• Tell the team. A short note in the team channel avoids confusion if someone sees a failing check mid-deploy.

Certificate expiry is a special case, because it is predictable and entirely preventable. Run a regular SSL certificate check so an expiring certificate becomes a calendar task and never a surprise outage.

Perspective: what we have learned about alerting

We build uptime and diagnostic tools, and the pattern we see is consistent: people over-invest in detection and under-invest in deciding what an alert is for. A monitor that checks every thirty seconds and pages three people on the first failure feels thorough. In practice it trains the team to ignore it.

Our view is to start quiet and get louder only where an outage truly hurts. Begin with one check on the page that matters most, require consecutive failures, and send the first notification to a channel people already read. Add a phone-level alert only after you have watched the quiet version for a couple of weeks and know which failures were real. That order costs you a little speed at the start and saves you a lot of trust later.

We also think the check should be free and instant to try. If setting up a first monitor takes an afternoon of account forms, people skip it and learn about outages from customers. That is why our core tools work without installation or an account.

— Andreas

Test your downtime alerting setup

An alert you have never seen fire is an assumption. Test it the same way you would test a backup: on purpose, on a schedule.

Run a controlled failure

1. Create a monitor against a throwaway URL or a staging page you control.
2. Make it fail on purpose: take the page down, return a 503, or point the check at a path that does not exist.
3. Time how long it takes from the first failed request to the first notification. Compare that with the threshold you set.
4. Confirm that every intended recipient received it, on the channel you expected, with enough detail to act.
5. Let the escalation tier fire too, and check that the backup responder is reached.
6. Restore the page and confirm the recovery message arrives.

Check the details people forget

• Does the alert land in a spam folder or a muted chat thread?
• Does the phone notification break through do-not-disturb settings for the on-call person?
• Does the message contain the URL and the error, or only the word "down"?
• If your monitor is the thing that fails, who notices? Consider an independent second check on your most important endpoint.

Review on a schedule

Once a month, or after every real incident, look at the alert log and ask three questions: which alerts needed no action, which outages were noticed late, and which alerts reached the wrong person. Adjust one setting at a time so you can see what changed. Write the outcome in a short postmortem note, even a few lines, so the next change builds on the last.

Set it up with Pingfloat

If you want a starting point that costs nothing, our Website Uptime Monitor lets you check a URL without installing anything or creating an account. Signing up adds options such as scheduled monitoring and saved results, which is where the threshold and routing ideas above come into play.

Pair it with the SSL Checker for certificate expiry and the traceroute tool for diagnosing where a connection breaks. The rest of the suite is on the Pingfloat home page.

A starting configuration you can copy

If you want a concrete baseline, begin here and adjust after two weeks of real data:

Treat the numbers as defaults to challenge. A service with a strict uptime target should shorten them. An internal tool used during business hours can lengthen them and skip the phone tier entirely.

For extra reading on spotting and shortening outages in a business setting, this IT support checklist for reducing downtime covers the operational side that sits around the alerts themselves.

FAQ

How many failed checks should trigger a downtime alert?

Two or three consecutive failures is a good default. Google's SRE guidance recommends requiring a condition to persist for at least two evaluation cycles before alerting. If your interval is long, shorten the interval rather than dropping to a single failure, which invites false alarms.

What are the three types of alerts I should set up?

A practical split is a warning, an incident and an escalation. The warning goes to a team channel after a short failure. The incident goes to the on-call person when the outage persists. The escalation reaches a backup responder if nobody acknowledges it.

How do I avoid alert fatigue?

Page a person only for urgent, actionable problems that affect users, and send everything else to a dashboard or a quiet channel. Group related checks so one root cause produces one notification. Review the alert log monthly and remove or downgrade any alert that rarely needed action.

Should I mute alerts during maintenance?

Yes, but use a scheduled maintenance window with a clear start and end instead of deleting or manually pausing checks. That way monitoring resumes on its own, and if the site is still failing when the window closes, you are alerted immediately.

Sources

• Google SRE Book: Monitoring Distributed Systems
• Google SRE Book: Practical Alerting from Time-Series Data
• Google SRE Workbook: Alerting on SLOs

Recommended

• Get Notified in Minutes: Website Downtime Alerts Checklist for Devs
• 15 Minutes to Website Uptime Monitoring for Devs with Pingfloat
• Website Uptime Monitor
• 99.9% Uptime Meaning: What Three Nines Allow