PromQL cookbook

Worked PromQL, phrased for LeanSignal: every query below runs unchanged on the Metrics page, in a dashboard panel, or in an alert rule. Metric and label names match what the integrations catalog collects — substitute your own where they differ.

Where a query runs

The source switch on the Metrics page decides what a query can see:

• Stored — the central dataplane: only demanded timeseries, kept 30 days. This is what dashboards and alerts read.
• Available — the agent’s edge buffer: everything collected, for the last ~1 day. Requires the agent to be Connected.

Exploring never creates demand — iterate freely against either source; only saved dashboards, active alert rules, and enabled ingestion rules change what is stored. And mind the naming: OTLP → Prometheus normalization turns dots into underscores, appends _total to counters, and appends _ratio to unit-1 gauges — system.disk.io arrives as system_disk_io_bytes_total.

Rates from counters

Counters only go up; you almost always want their rate, not their value.

# Bytes written and read per second, per disk
sum by (device, direction) (rate(system_disk_io_bytes_total[5m]))

# Network throughput per interface
sum by (device, direction) (rate(system_network_io_bytes_total[5m]))

# Requests per second, from the OTel HTTP duration histogram's count series
sum(rate(http_server_request_duration_seconds_count[5m]))

Use increase for “how many over a window” instead of “per second”:

# Network errors in the last hour, per host
sum by (host_name) (increase(system_network_errors_total[1h]))

Percentiles and averages from histograms

# p99 request latency
histogram_quantile(0.99,
  sum by (le) (rate(http_server_request_duration_seconds_bucket[5m])))

# Average request latency (sum over count — no quantile estimation error)
sum(rate(http_server_request_duration_seconds_sum[5m]))
/
sum(rate(http_server_request_duration_seconds_count[5m]))

Split any of these by a grouping label (route, method, host_name) to scope a regression — the high-latency playbook walks the full investigation.

Saturation — gauges and derived ratios

Gauges are read directly. The host examples below use the Host Metrics names; the node_exporter equivalents are noted where they differ.

# Load average vs. cores: above 1.0 the host is queueing
max by (host_name) (system_cpu_load_average_5m)
/
count by (host_name) (count by (host_name, cpu) (system_cpu_time_seconds_total{state="idle"}))

# Memory used, as a fraction of the total
sum by (host_name) (system_memory_usage_bytes{state="used"})
/
sum by (host_name) (system_memory_usage_bytes)

# Filesystem fill fraction, per mountpoint
sum by (host_name, mountpoint) (system_filesystem_usage_bytes{state="used"})
/
sum by (host_name, mountpoint) (system_filesystem_usage_bytes)

(node_exporter: node_load5 over count(node_cpu_seconds_total{mode="idle"}), and 1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes.)

Some receivers also emit ready-made *_utilization_ratio gauges — they are opt-in in the collector config, so prefer the derived ratios above unless you have enabled them.

Aggregation patterns

# The five busiest hosts by CPU-seconds spent non-idle
topk(5, sum by (host_name) (rate(system_cpu_time_seconds_total{state!="idle"}[5m])))

# One value per host — the multi-instance pattern every catalog dashboard uses
max by (host_name) (system_cpu_load_average_5m{host_name=~"$instance"})

Catalog dashboards ship an Instance variable (host_name on OTel dashboards, instance on node_exporter ones) so one dashboard serves the whole fleet or a single host — keep that shape in your own panels. See multi-instance dashboards.

Alert-shaped queries

An alert rule fires while its expression returns any series, so write the threshold into the query. Remember that active alert rules create demand — alerting on a not-yet-collected metric turns its collection on.

# Scrape target down (node_exporter / any scraped exporter)
min by (instance) (up{job="node"}) == 0

# Sustained load: 5-minute load average above 1.5× the core count
max by (host_name) (system_cpu_load_average_5m)
  > 1.5 * count by (host_name) (count by (host_name, cpu) (system_cpu_time_seconds_total{state="idle"}))

# Disk will be full within four hours at the current trend
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0

Pair thresholds with a For duration in the rule so blips don’t page — see Alerting.

Size a demand before you commit to it

The cost of demanding a selector is the number of timeseries it matches. Count that on Available — against everything the agent collects — before a panel or ingestion rule makes it permanent:

# How many series would this selector store?
count({__name__=~"system_.*"})

# Which names dominate that count?
sort_desc(count by (__name__) ({__name__=~"system_.*"}))

Each stored series costs roughly 175 thousand samples per month at the default 15-second cadence — the arithmetic behind the estimate columns on the Demands page. See the demand model.

When a query returns nothing

On Stored, an empty result usually means not demanded, not broken — central storage only holds what the demand set covers. Check the Missing data playbook before concluding anything: is it demanded, does Available have it, is the agent Connected, is the metric name normalized the way you expect.

Next steps

• Metrics — the signal’s full lifecycle, edge to central.
• LogQL cookbook — the same treatment for logs.
• Build dashboards — a saved panel turns a query into demand.