Troubleshooting
Agent shows Disconnected (or never connects)
Section titled “Agent shows Disconnected (or never connects)”Work outward from the agent host:
- Credentials — on Linux, check
/etc/leansignal-agent/agent.env; on macOS, theEnvironmentVariablesin/Library/LaunchDaemons/com.leansignal.agent.plist(launchd does not readagent.env). The agent key must match the one in the app (Agents, admin reveal), andLEANSIGNAL_TENANTmust be your tenant name — the agent derives every endpoint from it at startup. (LEANSIGNAL_ENDPOINTis only present if you pinned the control host explicitly; if set, it must be<tenant>-grpc.<region>:443.) - Egress — the agent dials out only, gRPC over TLS to port
443. No inbound rules are needed, but the host must reach that endpoint. - Health —
curl -sf http://127.0.0.1:13133/should return 200. If not, read the logs:journalctl -u leansignal-agent -f(Linux) or the container logs. - Key already in use — an agent key supports one connection at a time.
A second agent dialing in with the same key is rejected
(
agent already connected with this key). Give each agent its own key.
Dashboard panels are empty in view mode
Section titled “Dashboard panels are empty in view mode”View mode reads your central dataplane, which holds only demanded metrics:
- Saved the dashboard a moment ago? The demand set recomputes and broadcasts within seconds, but history starts accumulating from that moment — a panel shows data going forward, not backfilled.
- Works in edit mode but not view mode? Edit mode reads the agent’s local full-fidelity store, so this is exactly the demanded-vs-available gap. Make sure the dashboard was saved (saving is what updates the demand set) and give it a minute.
- Opt-in metrics — several integrations query metrics their receiver doesn’t emit by default (host utilization ratios, MySQL slow queries, PostgreSQL deadlocks). Each integration page lists what to enable; until then those panels stay empty by design.
- Metric name mismatch — remember the OTLP → Prometheus normalization:
dots become underscores, counters gain
_total, and unit-1gauges gain_ratio. Check the real names on an agent host:curl -s http://127.0.0.1:8428/api/v1/label/__name__/values.
Edit-mode preview returns nothing
Section titled “Edit-mode preview returns nothing”Edit-mode queries tunnel through the connected agent. If the agent is Disconnected, previews fail until it reconnects. Also note the edge buffer is short — about a day of metrics, roughly an hour of logs and traces — so ranges beyond that window are empty locally even for series that exist.
Logs/Traces: Available is empty
Section titled “Logs/Traces: Available is empty”Available reads the agent’s local store, so an empty view means telemetry
never reached the agent. Check that your application can reach the agent’s
OTLP port — 4317 (gRPC) or 4318 (HTTP) — from where it runs. If the app
runs in Docker, localhost points at the app’s own container, not the agent
host — see the
Docker networking note
for the fix. And remember the edge window is short: roughly an hour for logs
and traces, so query a recent range.
Available has data, but Stored stays empty
Section titled “Available has data, but Stored stays empty”Stored holds only the demanded subset, so nothing is forwarded until a demand covers the stream or service:
- No demand yet? Save a dashboard whose panel queries the stream — saving is what adds its selectors to the demand set — or add an Ingestion rule on the signal’s page. See Dashboards.
- Agent running with Docker Compose? The compose file defaults the logs
and traces endpoints to the agent’s local stores, so demanded data loops
back locally instead of reaching central storage. Export
LEANSIGNAL_LOKI_ENDPOINT/LEANSIGNAL_TEMPO_ENDPOINTto your ingest host — see Agent configuration.
Stored data stopped growing — is ingestion paused?
Section titled “Stored data stopped growing — is ingestion paused?”If a signal’s stored data flatlines while the agent stays Connected, check the usage meters on Home: reaching a storage ceiling or a monthly ingest budget pauses that signal’s ingestion, and the meter says so explicitly. The agent backs off and probes once a minute, so ingestion resumes on its own once stored data drops back below the ceiling — purge something, or let the 30-day retention do it — or the month rolls over. See Limits. Data that ages out of the edge buffer while paused is not backfilled.
An agent stores 0 of N timeseries
Section titled “An agent stores 0 of N timeseries”0/N in Stored Timeseries means the demand set doesn’t cover anything this
agent collects — usually because no demand exists yet. Import an
integration or save a dashboard/alert that queries its
metrics.
An alert never fires
Section titled “An alert never fires”Check, in order:
- Paused? Rules with no notification channel are always paused and cannot be resumed until you attach one (the amber icon in the alerts list). Imported rules start this way.
- No data? Status No data means the query returns nothing from the dataplane. Activating the rule adds its metrics to the demand set, but only from that moment — and only if agents actually collect them (opt-in receiver metrics again). Use Test now in the rule dialog to see the live values.
- The For window — the breach must hold for the configured duration on every evaluation. A condition that blips in and out never promotes from Pending to Firing (that’s deliberate — see Alerting).
An alert fired but nobody was notified
Section titled “An alert fired but nobody was notified”- Muted? Muting suppresses delivery while the state still changes.
- Channel disabled, or the rule’s channel list is empty.
- Email recipients not eligible — a recipient still Awaiting verification or Blocked by user receives nothing. Check the chips in the channel dialog and use Send test to verify the path end to end.
- Notifications fire on transitions only — a rule that was already firing before you attached the channel won’t notify until it resolves and fires again.
Catalog import fails
Section titled “Catalog import fails”If Import from catalog… can’t load the catalog (this docs site is the registry), use the Demand slug fallback field with the slug from the integration page, or download the bundle JSON from the docs and use Import from file….
Still stuck?
Section titled “Still stuck?”The agent repository is open source — issues and questions are welcome there for anything agent-side.
Thanks for your feedback!
Report sent — thank you!