Skip to content

Troubleshooting

Agent shows Disconnected (or never connects)

Section titled “Agent shows Disconnected (or never connects)”

Work outward from the agent host:

  1. Credentials — on Linux, check /etc/leansignal-agent/agent.env; on macOS, the EnvironmentVariables in /Library/LaunchDaemons/com.leansignal.agent.plist (launchd does not read agent.env). The agent key must match the one in the app (Agents, admin reveal), and LEANSIGNAL_TENANT must be your tenant name — the agent derives every endpoint from it at startup. (LEANSIGNAL_ENDPOINT is only present if you pinned the control host explicitly; if set, it must be <tenant>-grpc.<region>:443.)
  2. Egress — the agent dials out only, gRPC over TLS to port 443. No inbound rules are needed, but the host must reach that endpoint.
  3. Healthcurl -sf http://127.0.0.1:13133/ should return 200. If not, read the logs: journalctl -u leansignal-agent -f (Linux) or the container logs.
  4. Key already in use — an agent key supports one connection at a time. A second agent dialing in with the same key is rejected (agent already connected with this key). Give each agent its own key.

View mode reads your central dataplane, which holds only demanded metrics:

  • Saved the dashboard a moment ago? The demand set recomputes and broadcasts within seconds, but history starts accumulating from that moment — a panel shows data going forward, not backfilled.
  • Works in edit mode but not view mode? Edit mode reads the agent’s local full-fidelity store, so this is exactly the demanded-vs-available gap. Make sure the dashboard was saved (saving is what updates the demand set) and give it a minute.
  • Opt-in metrics — several integrations query metrics their receiver doesn’t emit by default (host utilization ratios, MySQL slow queries, PostgreSQL deadlocks). Each integration page lists what to enable; until then those panels stay empty by design.
  • Metric name mismatch — remember the OTLP → Prometheus normalization: dots become underscores, counters gain _total, and unit-1 gauges gain _ratio. Check the real names on an agent host: curl -s http://127.0.0.1:8428/api/v1/label/__name__/values.

Edit-mode queries tunnel through the connected agent. If the agent is Disconnected, previews fail until it reconnects. Also note the edge buffer is short — about a day of metrics, roughly an hour of logs and traces — so ranges beyond that window are empty locally even for series that exist.

Available reads the agent’s local store, so an empty view means telemetry never reached the agent. Check that your application can reach the agent’s OTLP port — 4317 (gRPC) or 4318 (HTTP) — from where it runs. If the app runs in Docker, localhost points at the app’s own container, not the agent host — see the Docker networking note for the fix. And remember the edge window is short: roughly an hour for logs and traces, so query a recent range.

Available has data, but Stored stays empty

Section titled “Available has data, but Stored stays empty”

Stored holds only the demanded subset, so nothing is forwarded until a demand covers the stream or service:

  • No demand yet? Save a dashboard whose panel queries the stream — saving is what adds its selectors to the demand set — or add an Ingestion rule on the signal’s page. See Dashboards.
  • Agent running with Docker Compose? The compose file defaults the logs and traces endpoints to the agent’s local stores, so demanded data loops back locally instead of reaching central storage. Export LEANSIGNAL_LOKI_ENDPOINT / LEANSIGNAL_TEMPO_ENDPOINT to your ingest host — see Agent configuration.

Stored data stopped growing — is ingestion paused?

Section titled “Stored data stopped growing — is ingestion paused?”

If a signal’s stored data flatlines while the agent stays Connected, check the usage meters on Home: reaching a storage ceiling or a monthly ingest budget pauses that signal’s ingestion, and the meter says so explicitly. The agent backs off and probes once a minute, so ingestion resumes on its own once stored data drops back below the ceiling — purge something, or let the 30-day retention do it — or the month rolls over. See Limits. Data that ages out of the edge buffer while paused is not backfilled.

0/N in Stored Timeseries means the demand set doesn’t cover anything this agent collects — usually because no demand exists yet. Import an integration or save a dashboard/alert that queries its metrics.

Check, in order:

  1. Paused? Rules with no notification channel are always paused and cannot be resumed until you attach one (the amber icon in the alerts list). Imported rules start this way.
  2. No data? Status No data means the query returns nothing from the dataplane. Activating the rule adds its metrics to the demand set, but only from that moment — and only if agents actually collect them (opt-in receiver metrics again). Use Test now in the rule dialog to see the live values.
  3. The For window — the breach must hold for the configured duration on every evaluation. A condition that blips in and out never promotes from Pending to Firing (that’s deliberate — see Alerting).
  • Muted? Muting suppresses delivery while the state still changes.
  • Channel disabled, or the rule’s channel list is empty.
  • Email recipients not eligible — a recipient still Awaiting verification or Blocked by user receives nothing. Check the chips in the channel dialog and use Send test to verify the path end to end.
  • Notifications fire on transitions only — a rule that was already firing before you attached the channel won’t notify until it resolves and fires again.

If Import from catalog… can’t load the catalog (this docs site is the registry), use the Demand slug fallback field with the slug from the integration page, or download the bundle JSON from the docs and use Import from file….

The agent repository is open source — issues and questions are welcome there for anything agent-side.

Was this page helpful?