Alerting Beats Dashboards for Agent Reliability

There's a quiet assumption baked into most observability tools: that a human is watching. Charts of latency, run volume, and token spend are useful — when someone has them open. The problem is that agent failures don't schedule themselves for business hours. They happen on weekends, during deploys, in the edge cases your tests never reached. By the time you next open the dashboard, the damage is done.

The fix isn't a prettier dashboard. It's inverting the model: instead of you polling the system, the system pushes to you the moment something crosses a line.

The four failure modes worth alerting on

Agents fail in a handful of recognizable ways, and each maps cleanly to a metric you can threshold:

• Correctness collapse — a deploy or upstream change makes runs start failing. Watch error_rate or success_rate over a rolling window.
• Cost runaway — a retry loop or a prompt that ballooned. Watch cost_usd per hour against a budget.
• Latency creep — a slow tool or a degraded model provider. Watch avg_latency_ms for a sustained increase.
• Volume of failures — even a low error rate is a problem at high traffic. Watch failed_runs as an absolute count.

Thresholds, windows, and not crying wolf

The hard part of alerting isn't firing — it's firing only when it matters. A single failed run is noise; a 5% error rate sustained over fifteen minutes is a signal. That's why AgentPulse alert rules combine a metric, a gt/lt operator, a threshold, and a rolling window. The window smooths out the one-off blips so the alert that reaches your phone is one you should act on.

Equally important: an alert that fires every minute until you fix it is an alert you'll mute. AgentPulse deduplicates open alerts — one notification per breach — and only re-arms the rule once the condition resolves. You get told once when it breaks, and once when it's healthy again.

Route it to where you already are

An alert is only useful if it reaches you. AgentPulse delivers to Slack, a generic HTTP webhook, or email — so the signal lands in the channel your team already watches rather than in yet another tool you have to remember to open. Wire the webhook into PagerDuty or Opsgenie if a breach should page someone.

Keep the dashboard — for after

None of this means dashboards are useless. They're exactly the right tool for the second step: once an alert tells you that something is wrong, you open the dashboard and the execution traces to find out why. The mistake is using the dashboard as your detection layer. Detection should be automatic. Investigation is where the human comes in.

Build alerting first. Then let the charts do what they're actually good at.