Skip to content
LeanSignal Docs

Agent troubleshooting

Two different failures look similar from the app: an agent that is Disconnected (steps 2–6) and an agent that is Connected but not forwarding (steps 7–10). Read the status first, then take the matching branch. Commands below are for the Linux install; the platform pages have the macOS, Windows, Kubernetes, and Docker equivalents.

  1. Read the status on the Agents page. Connected/Healthy — the control stream is up, skip to step 7. Disconnected/Unknown — work outward from the agent host, starting at step 2.

  2. Check the services on the host. The agent runs as systemd units beside its co-located local stores:

    Terminal window
    systemctl status leansignal-agent leansignal-victoria-metrics leansignal-loki leansignal-tempo
    journalctl -u leansignal-agent -p err -b

  3. Probe the health endpoint. curl -sf http://127.0.0.1:13133/ must return 200. If it doesn’t, the collector itself isn’t up — the journal from step 2 says why (bad config, bad credentials, port conflict).

  4. Verify credentials. In /etc/leansignal-agent/agent.env (macOS: the EnvironmentVariables in the launchd plist), the agent key must match the one on the Agents page (admin reveal) and LEANSIGNAL_TENANT must be your tenant name. An agent key supports one connection at a time — a second agent dialing in with the same key is rejected; give each agent its own key.

  5. Verify egress. The agent dials out only: gRPC over TLS to port 443. No inbound rules are needed, but the host must reach the tenant endpoint — check proxies and egress firewalls.

  6. Confirm the control stream from the edge. The agent self-monitors into its own local store:

    Terminal window
    curl -s --get 'http://127.0.0.1:8428/api/v1/query' \
      --data-urlencode 'query=leansignal_edgecontroller_connection_up'

    1 means connected; the app should agree within seconds.

  7. Connected but storing 0/N timeseries? That is usually correct: the demand set doesn’t cover anything this agent collects. Import an integration or save a dashboard that queries its metrics, and remember paused alert rules demand nothing.

  8. Check demand-set freshness. The agent detail view shows Demand Last Update — when this agent last received a demand-set push. Saving any dashboard should refresh it within seconds; if it stays stale while the agent is Connected, restart the agent (sudo systemctl restart leansignal-agent) and watch it re-sync.

  9. Read the demand filter’s own verdicts. Each filter logs per-batch received / allowed / dropped counts:

    Terminal window
    journalctl -u leansignal-agent | grep 'demand filter' | tail -20

    allowed=0 with no demand is normal; allowed>0 while Stored stays empty points past the agent — check the usage meters on Home for a paused signal (Limits), and on Docker Compose check that LEANSIGNAL_LOKI_ENDPOINT / LEANSIGNAL_TEMPO_ENDPOINT point at your ingest host, not the local stores.

  10. Sanity-check the local stores. All loopback-only; the agent’s self-telemetry is there even before you send anything:

    Terminal window
    curl -s 'http://127.0.0.1:8428/api/v1/label/__name__/values' | head -c 500
    curl -s --get 'http://127.0.0.1:3100/loki/api/v1/query_range' \
      --data-urlencode 'query={service_name="leansignal-agent"}' --data-urlencode 'limit=5'
    curl -s --get 'http://127.0.0.1:3200/api/search' --data-urlencode 'q={}' --data-urlencode 'limit=5'

    Mind the windows: 1 day of metrics, ~1 hour of logs and traces — an empty result over a longer range is expected, not broken.

Next steps

Section titled “Next steps”
Was this page helpful?

www.leansignal.io ·