Alerting

An alert rule in LeanSignal is a PromQL query plus a condition and an evaluation interval, routed to your channels. Rules live under Alerts in the LeanSignal app, belong to a demand, and — like dashboards — shape what your agents collect: activating a rule adds its metrics to the demand set, so you can alert on a metric that isn’t being forwarded yet and collection starts automatically.

Create an alert

Go to Alerts → Add alert (editor or admin role). The form:

Field	What it does
Name	Shown in the list and in notifications.
Demand	The demand this rule belongs to (required).
Severity	critical, warning, or info — a label carried into notifications; it does not change evaluation.
PromQL query	The expression to evaluate, for example histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).
Condition + Threshold	The breach test: >, ≥, <, ≤, =, or ≠ against a number. If any series returned by the query breaches, the rule breaches.
Evaluate every	How often to run the query. Whole minutes or coarser (1m, 5m, 1h) — the minimum is 1m.
For	How long the breach must hold before the rule fires. 0 fires immediately; otherwise whole minutes or coarser.
Message	Optional text included in the notification — supports {{placeholders}}, see Message templates.
Notify on resolve	Also send a notification when the rule returns to normal.
Channels	The channels to deliver to.

Before saving, use the Preview section: pick a range (last 5m to last 24h) and press Evaluate to chart the query. The preview runs against the agent’s local full-fidelity store, so you can validate an expression even for metrics that aren’t demanded yet — real data, before any collection change.

No channel means paused

A rule with no channel has nowhere to deliver, so it is always saved paused and cannot be resumed until you attach a channel — the form enforces this, and imported rules from the integrations catalog arrive in exactly this state. Rules without a channel carry an amber warning icon in the list.

Message templates

The Message field is a template: write plain text and drop in {{placeholders}} that are filled with the alert’s live values at delivery time. The editor lists every available placeholder (click one to insert it at the caret) and renders a live preview against example values as you type — unknown or malformed tokens are flagged before you save, not when the alert fires.

Placeholder	Fills with
{{alert.name}}, {{alert.severity}}, {{alert.status}}	The rule’s name, its severity, and firing or resolved.
{{alert.query}}, {{alert.condition}}, {{alert.threshold}}	The PromQL, the breach test (e.g. > 80), and the threshold value.
{{value}}	The worst breaching series’ value.
{{series_count}}, {{series_summary}}	How many series are breaching, and a one-line-per-series summary (labels + value).
{{labels.<name>}}	Any series label of the worst breaching series — e.g. {{labels.instance}}, {{labels.job}}.
{{tenant.name}}, {{time.triggered}}, {{link}}	The tenant, the transition time (UTC), and a link back to the alert in LeanSignal.

Example:

{{alert.name}} is {{alert.status}} on {{labels.instance}}:
value {{value}} breached {{alert.condition}}.
{{series_summary}}

The same mechanism (with check-specific placeholders) backs the Message field on synthetic checks.

How evaluation works

The evaluator runs each active rule on its interval against your central dataplane and derives a state:

Status	Meaning
OK	The query returned data and nothing breaches.
Pending	The condition is breaching but hasn’t held for For yet.
Firing	The breach held long enough. A notification went out on the transition.
No data	The query returned nothing (or failed) and the rule wasn’t firing.
Paused	Evaluation is switched off — manually, or because the rule has no channel.
Muted	Evaluating normally, but notifications are suppressed until the mute expires.

Details worth knowing:

• Notifications fire on transitions only — once when the rule enters Firing, and once on Firing → OK if Notify on resolve is on. A rule that stays firing does not re-notify.
• Missing data never auto-resolves an alert. If a firing rule’s query starts returning nothing, it stays firing — silence is not health.
• Muting suppresses delivery, not evaluation. State transitions still happen and are recorded; they do not notify while the mute lasts.
• Pausing stops evaluation entirely and clears the rule’s state; on resume the For timer starts fresh.

Alerts and the demand set

Creating, updating, pausing, resuming, or deleting a rule recomputes the demand set and pushes it to every connected agent:

• Activating a rule turns its metrics on. If the query needs timeseries no dashboard uses, agents start forwarding them within seconds.
• Pausing a rule releases its metrics (unless a dashboard or another rule still demands them) — pausing genuinely stops the cost.
• Muting changes nothing about collection — the rule keeps evaluating, so its metrics stay demanded.

Test, mute, pause

• Test now (in the edit dialog) runs the query live and reports the result without changing state or notifying — for example would fire — condition breaching (0.83, …) or OK — not breaching.
• Mute (bell icon in the list) suppresses notifications for a preset window — 15 minutes, 1 hour, 4 hours, or 24 hours — with Unmute to end it early. Use it during planned maintenance instead of pausing, so state stays honest.
• Pause / Resume (in the list) stops and restarts evaluation. Remember that resume is refused while the rule has no channel.

Choosing thresholds

Alert on meaningful, sustained signals, not transient spikes: smoothed rates over windows, ratios against a capacity or limit, and slow-moving gauges — with a For long enough that a single blip cannot fire. The integration alert sets follow this philosophy (for example, host saturation alerts use load average per core rather than the volatile CPU gauge) and make good templates for your own rules.

Next steps

• Channels — where alerts deliver.
• Work with demands — how rules fit the demand model.
• Troubleshooting — an alert that never fires, and other puzzles.