Alerting

An alert rule evaluates a condition on a schedule and notifies a channel when it holds.

How evaluation works

Rules are evaluated on a fixed 60-second loop. Each rule also carries its own evaluation window and a cooldown, both defaulting to 5 minutes.

The cooldown is what stops a flapping condition from paging you repeatedly: once a rule fires, it will not fire again until the cooldown elapses.

Rule types

These types are evaluated by the alerting loop:

• Threshold — a value crosses a bound. The common case.
• Absence — expected data stopped arriving. This is the one that catches a crashed exporter, which no threshold on the data itself can detect.
• SLO burn rate — an error budget is being consumed too fast.
• Composite — combines other rules.
• Governance — policy conditions on AgentSight data.

Two further types exist but are not driven by this loop:

• Anomaly rules are fired by the AI service, not by the alert evaluator.
• Synthetic rules are fired by the synthetics scheduler. See Synthetic monitoring.

Creating one of those two as an ordinary rule and expecting the evaluator to run it will not work — nothing evaluates it and it silently never fires.

Conditions

A threshold rule compares an aggregate against a value. Three comparators are implemented:

• GT — greater than
• LT — less than
• EQ — equal to

Severity uses the vocabulary LOW, MEDIUM, HIGH, CRITICAL, and drives how each channel renders the alert — PagerDuty urgency, OpsGenie priority, Slack formatting. This field is also not strictly validated on the normal create path, and a severity outside that set falls back to the equivalent of a warning rather than being rejected.

Notification channels

Five channels work with no additional configuration beyond their own credentials:

• Slack — a formatted message to an incoming webhook. Needs webhookUrl.
• PagerDuty — Events API v2. Needs routingKey. Severity maps to critical / error / warning / info.
• Microsoft Teams — an Adaptive Card to an incoming webhook. Needs webhookUrl.
• OpsGenie — creates an alert. Needs apiKey, optionally teams.
• Email — sent through your SMTP server. Needs to, smtpHost, smtpPort, smtpUser, smtpPass, and from. There is no platform-operated mail relay; if you do not have an SMTP server to point at, use another channel.

Slack, PagerDuty, OpsGenie, and Teams deliveries retry up to four times with 1s / 5s / 30s backoff and a 10-second timeout. 4xx responses are not retried, since a rejected payload will be rejected again. Email is not retried — a failed send is lost, so do not make email your only channel for anything you must not miss.

Testing a rule

Every rule has a Test action, and rules can be tested before you trust them. Test notifications are prefixed [TEST] and use a separate deduplication key, so firing one never merges with a live incident or resolves one.

Given that an unimplemented comparator produces a rule that looks fine and never fires, testing after saving is not optional — it is the only thing that distinguishes a working rule from a decorative one.

Runbooks

A rule can carry a runbook URL, which is rendered as a link in every channel. The person woken at 3am by the alert gets a link to what to do about it, which is most of the value of the alert.