Traces

LeanSignal applies the same demand-driven model to traces that it applies to metrics and logs. This page follows a span through the whole pipeline: how it gets into the agent, how the agent buffers it, how demand decides which services reach central storage, and how you explore them in the app.

Trace demand is service-scoped — that is the one way traces differ from the other signals, and it matters for how complete a stored distributed trace is.

The lifecycle at a glance

   your services              the agent gateway                      central
  ─────────────────    ────────────────────────────────      ──────────────────
   OTLP (spans)     →   local Tempo (all spans, ~1 hour)   →   demanded services'
                        spans queryable live from the app;         spans
                        demand filter forwards only spans          (tenant Tempo,
                        from services a dashboard / rule            30 days)
                        demands

Everything is collected at full fidelity at the edge. Only spans from demanded services are ever forwarded to central storage — so central cost tracks the services you actually inspect. See Demand-driven observability for the why.

How traces are collected

Traces travel OTLP only — the same 4317 (gRPC) / 4318 (HTTP) endpoint on the agent gateway that your app uses for metrics:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://<gateway-host>:4317"
export OTEL_SERVICE_NAME="checkout"
export OTEL_TRACES_EXPORTER="otlp"   # the default in most OpenTelemetry SDKs

Most OpenTelemetry language SDKs and auto-instrumentation agents export traces by default the moment an OTLP endpoint is configured — if your app already sends metrics to the gateway, traces usually need no extra wiring. For collector-mediated topologies, add a traces pipeline with an otlp exporter pointed at the gateway (<gateway-host>:4317; on Kubernetes, use the agent’s Service name).

service.name is the unit of trace collection

Trace demand is scoped by service, so service.name isn’t just a label here — it is the thing you demand. Set it on every instrumented workload.

How traces are buffered at the edge

Inside the gateway, every span flows through a traces/all pipeline before any filtering:

• Local store. Every span is written to a co-located, single-node Tempo — a short ~1-hour full-fidelity window. This is the edge buffer: every span the gateway receives, whether or not anything demands it.
• Queryable live. That buffer is searchable from the Traces page in the app (source: Available) over the agent’s gRPC control channel — so you can inspect traces nobody is storing centrally yet. At the edge, traces are always complete for whatever the gateway received.

Kubernetes

The Helm chart bundles the local trace store: a co-located Tempo runs as its own pod by default. To use a Tempo you operate yourself, set localTempo.deploy=false and point localTempo.writeEndpoint and localTempo.queryEndpoint at it. The Linux and macOS installers set the local Tempo up for you.

How traces are filtered — the demand set

The traces/all pipeline fans into a traces/filtered pipeline. The filter is the set of demanded services. A demanded service has all of its spans forwarded to the per-tenant Tempo; spans from every other service stay at the edge and age out within the hour.

Two things create trace demand:

• Dashboards. Saving a dashboard whose panels query traces demands the services those queries name. A TraceQL query that matches a resource attribute — for example {resource.service.name="checkout"} — demands the checkout service. Span-level conditions (span.http.status_code = 500) filter at query time and don’t narrow what is shipped. See Build dashboards.
• Ingestion rules. A service selector you add directly on the Traces page’s Ingestion rules tab, attached to a demand — forward a service’s spans without building a dashboard for it.

The demanded set is recomputed and pushed to every connected agent in near real time, and central storage keeps demanded services’ spans for 30 days — the same retention as metrics and logs.

Stored spans belong to the rule that demanded them

Traces go one step further than the other signals: every distinct trace selector — whether it comes from a dashboard panel or an ingestion rule — has its spans kept separately, per rule in central storage. That makes rule changes immediate and precise:

• Deleting a rule deletes its stored spans. They leave the Stored view within seconds and are permanently removed after the purge grace window (see Purge). Spans another live rule also matched are unaffected — that rule keeps its own copy.
• Editing a rule’s query is a delete plus a recreate. The reworded rule starts storing fresh from that moment, and spans stored under the old expression are purged. Renaming a rule or moving it between demands doesn’t count — only the TraceQL expression is the rule’s identity.
• Disabling a rule hides its spans. They stay in storage but leave the Stored view until you re-enable the rule — unlike metrics and logs, where data stored by an inactive selector stays queryable.
• Overlapping rules store duplicate copies. A service matched by three rules has its spans stored three times — the price of deleting each rule’s data independently. Keep trace rules disjoint where you can; the duplication counts toward your stored-traces figure.

Note

Per-rule deletion needs a current agent (≥ 0.6.7). Spans stored before the upgrade — or forwarded by older agents — sit in the tenant-wide store: they stay queryable and age out with the 30-day retention, but deleting a rule cannot remove them early.

Cross-service traces are complete only when every service is demanded

A distributed trace is whole in central storage only when every participating service is demanded. Demand checkout but not payments, and stored checkout traces show gaps where payments spans would be. At the edge (Available) traces are always complete for whatever the gateway received — so if a stored trace looks truncated, demand the missing service.

Ready-made demands are metrics-only

The demands in the integrations catalog are metrics-only. Trace demand comes from the dashboards and ingestion rules you create — there is nothing to import to “enable” traces. Collect them, then demand the services you inspect.

How traces show up in the app

The Traces page in the LeanSignal app is where you explore traces. It has two tabs and a source switch:

• Explore tab — a TraceQL search that shows matching traces as a duration-over-time chart and a results table; enter a trace ID to open that trace’s span timeline.
• Ingestion rules tab — create and manage the services this signal forwards on demand.

Search with TraceQL:

{resource.service.name="checkout" && span.http.status_code >= 500}

The source switch selects where the query runs:

Source	What it reads	What’s there
Stored	The per-tenant Tempo	Only demanded services’ spans, kept 30 days, from the moment each demand was saved
Available	The agent’s edge Tempo, live over the agent’s gRPC control channel	Every span collected, for the last ~1 hour, always complete

Available requires the agent to be Connected — the query is tunnelled down the agent’s own control stream, so the edge store is never exposed to the network. If no agent is connected, Available returns an error.

Exploring never creates demand

Searching on the Traces page — against either source — never changes what is stored centrally. Demand comes only from saved dashboards and enabled ingestion rules. Iterate freely before committing to a service’s cost.

Verify

1. Send some traffic through an instrumented app.
2. In the app, open Traces, switch the source to Available, and query {resource.service.name="<your service>"}. This reads the edge store live, so it works before any demand exists.
3. Save a dashboard panel with that query (or add an Ingestion rule), then switch back to Stored: new spans appear from the moment the demand was saved. Nothing is backfilled.

Next steps

• Metrics and Logs — the same lifecycle for the other two signals.
• Build dashboards — a saved trace panel is what turns a service into demand.
• Work with demands — the unit that ties dashboards and rules to cost.
• Agent configuration — gateway ports, endpoints, and retention.