High latency

Latency investigations move signal to signal: metrics say how bad and where, traces say which operation, logs say why. Each step works on Stored data when it is demanded, and on Available (the agent’s edge buffer — 1 day of metrics, ~1 hour of traces and logs, agent must be Connected) when it is not.

Quantify the problem with metrics. Chart the p99 from the service’s duration histogram on the Metrics page or a dashboard — for OpenTelemetry HTTP semconv:
```
histogram_quantile(0.99,
  sum by (le) (rate(http_server_request_duration_seconds_bucket[5m])))
```
Establish when it started and whether it is a step change or a slow drift.
Scope it. Split the same query by route, method, and instance (the $instance variable on catalog dashboards). One bad instance points at a host problem; all instances at once points at a shared dependency or a deploy.
Rule out saturation. Check the host golden signals for the affected instances — load average against the cores line, memory utilization, disk busy time, network errors (Host Metrics) — and the runtime’s GC/thread panels if a runtime integration is imported.
Localize with traces. On the Traces page, search the slow window with TraceQL:
```
{resource.service.name="checkout" && duration > 1s}
```
Use Stored if the service is demanded (30 days of history), otherwise Available for the last ~hour at full fidelity. Open a slow trace’s span timeline and find where the time actually goes.
Mind trace completeness. A stored distributed trace is complete only when every participating service is demanded — a gap where a downstream service’s spans should be means that service isn’t demanded, not that it wasn’t called. At the edge, Available traces are always complete for what the gateway received.
Check the downstream dependency. If slow spans wrap database calls, pivot to the MySQL or PostgreSQL dashboards — slow queries, connection saturation, lock waits.
Explain with logs. On the Logs page, query the service in the same time window as the slow exemplar traces — e.g. {service_name="checkout"} plus a line filter for the request path or trace ID. Undemanded streams are still on Available for ~an hour.
Persist what proved useful. Exploring never creates demand — save the queries that mattered as a dashboard (its selectors enter the demand set from that moment) and add an alert rule on the p99 threshold with a sensible For window, so the next regression pages you instead of waiting to be found.

Next steps

Missing data — when a step above shows nothing at all.
Traces — the trace lifecycle and TraceQL sources.
The demand model — what saving your findings changes.

Was this page helpful?