Reducing Alert Noise

An Alert that fires too often is almost as bad as no Alert at all. People start ignoring the notifications, and the one that matters gets lost. Here are the most useful ways to cut down the noise while still hearing about real problems.

1. Preview before you save

Run the Alert Preview on All Selected Targets over the last few days. If the Trigger Actions list is long, or the preview stops early because the Alert is too noisy, try the ideas below and refresh the preview to see the difference.

2. Ignore short blips

Most networks have the occasional bad sample. Set Trigger the alert after to a few minutes, so a target has to stay triggered before the Alert counts it. Or add a delay to just the actions that notify people.

3. Stop flapping

A target that bounces between good and bad can trigger and recover over and over. Set Clear the alert after to a few minutes, so the target has to stay healthy for a while before the Alert considers it recovered.

4. Use Combined actions for widespread problems

When something affects lots of targets at once, like an ISP outage, a Per target action sends a notification for each one. A Combined action sends one notification that lists them all.

A common setup is to send a Per target action to a low-priority place, like a chat channel or a Summary, and a Combined action to the people who need to act.

5. Escalate with tiers

Use tiers so the right people hear about a problem at the right size. Higher tiers suppress the tiers below them, so nobody is notified twice about the same problem.

6. Link related Alerts

If several Alerts tend to fire for the same problem, put them in a linked group. They'll share one count and one set of notifications.

7. Only notify during working hours

If nobody will act on a problem at 2 AM, there's no reason to send it then. Assign a Schedule, and problems that are still going when the schedule starts are sent then.

8. Decide what offline Agents mean

Laptops and home computers go offline all the time. Use the offline setting to count their targets as passing once they've been offline for a while, so they don't hold an Alert in the triggered state.

9. Narrow down the Agents and Targets

An Alert on every target is easy to set up, but it may watch targets nobody cares about. Use a filter to apply the Alert only to the Agents and Targets that matter, like a specific department or your VoIP servers.

10. Remind, don't repeat

Instead of separate Alerts for "it's still broken," use Retrigger while active on a Combined action to send a reminder every 30 minutes or every few hours while a problem lasts.