LogQL cookbook

Worked LogQL, phrased for LeanSignal: every query below runs unchanged on the Logs page or in a dashboard’s log panel. Stream labels come from OpenTelemetry resource attributes — service_name, service_namespace, k8s_namespace_name, k8s_pod_name, and friends — so a meaningful service.name on every workload is what makes these queries work.

Where a query runs

The source switch on the Logs page decides what a query can see:

• Stored — the per-tenant Loki: only demanded streams, kept 30 days. This is what dashboards read.
• Available — the agent’s edge Loki: every stream collected, for the last ~1 hour. Requires the agent to be Connected.

Exploring never creates demand — only saved dashboards and enabled ingestion rules change what is stored.

Selectors match streams — and streams are the demand unit

A LogQL query starts with a stream selector; everything after it is a filter stage:

{service_name="checkout"}

# Regex label matching — every service in the prod namespaces
{k8s_namespace_name=~"prod-.*"}

# Multiple labels narrow the stream set
{service_name="checkout", k8s_namespace_name="prod-eu"}

The distinction matters more here than anywhere else: a selector demands the whole stream. Filter stages (|=, | json, …) refine what you see on a query, not what is shipped — saving a panel with {service_name="checkout"} |= "error" stores every checkout line, not just the errors. Keep selectors as narrow as the label set allows.

Line filters

Chain substring and regex filters after the selector; they run in order, so put the most selective first:

# Errors, minus the health-check noise
{service_name="checkout"} |= "error" != "healthz"

# Case-insensitive regex — timeouts however they are spelled
{service_name="checkout"} |~ "(?i)time[ -]?out"

# Find one request across a stream by trace ID
{service_name="checkout"} |= "4bf92f3577b34da6"

Parse structured logs, then filter on fields

# JSON logs: only records whose parsed level is error
{service_name="checkout"} | json | level="error"

# logfmt logs: slow requests by parsed field
{service_name="checkout"} | logfmt | duration > 500ms

# Reshape the output line to just what you need
{service_name="checkout"} | json | line_format "{{.level}} {{.msg}}"

Parsed fields exist only in the query — they don’t change the stream’s labels, and they don’t change what is demanded.

Metric queries over logs

LogQL turns log lines into timeseries, which is also how a log panel gets a histogram — and how you alert on logs from a dashboard-shaped query:

# Error rate per service, lines per second
sum by (service_name) (rate({service_name=~".+"} |= "error" [5m]))

# The five chattiest demanded streams in the last hour
topk(5, sum by (service_name) (count_over_time({service_name=~".+"}[1h])))

# The five heaviest demanded streams by volume — where log cost comes from
topk(5, sum by (service_name) (bytes_over_time({service_name=~".+"}[1h])))

Run the last two on Stored and they audit exactly what you are paying to keep — the cost–value link, queryable.

Find a stream before you demand it

The lean workflow: look first, demand second. On Available, the edge buffer answers “what is there?” before anything is stored centrally:

# Is the new service's log stream reaching the agent at all?
{service_name="payments-v2"}

Found it? Save a dashboard panel with the selector, or add it on the Logs page’s Ingestion rules tab, and the stream starts forwarding within seconds. Storage begins at the moment the demand is saved — the ~1-hour edge window is not backfilled, so demand a stream as soon as you know you want it.

When a search returns nothing

On Stored, an empty result usually means not demanded — the filter at the edge is fail-closed, so an undemanded stream never reaches central storage, by design. On Available, remember the window is only ~1 hour and the agent must be Connected. Work through the Missing data playbook: is the stream demanded, is it on Available, is the agent up, are the labels what you assumed (service_name, not service.name).

Next steps

• Logs — the signal’s full lifecycle, edge to central.
• PromQL cookbook — the same treatment for metrics.
• Build dashboards — a saved log panel turns a selector into demand.