Skip to content
LeanSignal Docs

VictoriaMetrics

Monitor your VictoriaMetrics instances — ingestion rate, active time series, disk usage, churn, query errors, and process health — with curated dashboards and alerts you import in one step.

VictoriaMetrics exposes its own metrics in Prometheus format at /metrics (port 8428 for single-node). An OpenTelemetry Collector scrapes that endpoint with the Prometheus receiver and forwards OTLP to your LeanSignal agent gateway (one per cluster or environment). The gateway keeps full fidelity locally and forwards only the demanded subset to your central dataplane.

The panels and alert thresholds below are taken from VictoriaMetrics’ own official rules (alerts-health.yml and alerts-single-node.yml), so you’re watching what the VictoriaMetrics team recommends watching — disk saturation, memory headroom, timeseries churn, slow inserts, and rejected rows.

Prerequisites

Section titled “Prerequisites”
  • A LeanSignal agent gateway deployed and connected — one per cluster or environment. If you haven’t deployed it yet, follow Install the agent (source: LeanSignal/leansignal-agent). It should show as Connected under Agents in the LeanSignal app.
  • Network reachability to the gateway’s OTLP port — 4317 (gRPC) or 4318 (HTTP). See Agent configuration.
  • One or more VictoriaMetrics instances — they expose Prometheus metrics at /metrics by default — plus an OpenTelemetry Collector that scrapes them and forwards OTLP to the gateway, configured in Setup.
  • Editor or admin role in the LeanSignal app (importing a demand creates dashboards and alert rules).

Setup

Section titled “Setup”

VictoriaMetrics already exposes everything needed at /metrics — there is nothing to enable on it. Point an OpenTelemetry Collector at that endpoint and forward OTLP to your LeanSignal agent gateway (not a SaaS endpoint):

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: victoriametrics    # keep this name — the "instance down" alert matches on it
          scrape_interval: 30s
          metrics_path: /metrics
          static_configs:
            - targets: [victoriametrics:8428]   # host:port of each vmsingle instance


processors:
  batch: {}


exporters:
  otlp:
    endpoint: leansignal-agent:4317   # per-cluster gateway (use :4318 for HTTP)
    tls:
      insecure: true                  # in-cluster plaintext; provide ca_file for TLS


service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp]

List every instance under targets (or use kubernetes_sd_configs in Kubernetes); each scrape target becomes one value in the dashboard’s Instance filter.

To confirm the endpoint is serving metrics:

Terminal window
curl -s http://victoriametrics:8428/metrics | grep -E '^vm_(rows_inserted_total|free_disk_space_bytes|cache_entries)'

Logs and traces

Section titled “Logs and traces”

VictoriaMetrics logs to stderr/journald — collect those like any host logs: a collector filelog/journald receiver (or your existing shipper pointed at the gateway’s Loki push endpoint, :3500). VictoriaMetrics does not emit traces; the query paths that call it show up in your instrumented applications’ traces instead.

Demands

Section titled “Demands”

VictoriaMetrics ships as a ready-made demand in three variants. You don’t copy any JSON — in the LeanSignal app go to Demands, press the ˅ arrow on the Add Demand button (the arrow opens the import menu), choose Import from catalog…, and pick the variant. Each variant below lists the slug that identifies its published bundle. See Integrations for the full import flow.

The variants are nested — Standard is a superset of Essential, Extended a superset of Standard. Pick whichever is closest to what you need and treat it as a starting point: after import everything is a normal, editable copy, so retune thresholds, add or drop panels, and adjust queries for your environment — your edits reshape the demand automatically. Start small and re-import a larger variant later if you outgrow it.

The golden signals. 1 dashboard · 4 panels · 3 alerts — ingestion, active series, disk, and memory at a glance.

Import this variant from the catalog. Its demand slug:

Demand slug
victoriametrics-otel-demand-essential

Dashboard — Ingestion rate, Active time series, Disk space used, and Resident memory.

Alerts (3)

AlertSeverityFires when
VictoriaMetrics instance downcriticalscrape up = 0 for 2m
VictoriaMetrics disk running out of spacecriticaldisk > 80% used for 30m
VictoriaMetrics memory usage highcriticalRSS > 80% of available for 5m

Review the bundle JSON →

Dashboards

Section titled “Dashboards”

Each variant imports one VictoriaMetrics dashboard, its panels grouped into collapsible sections. Every dashboard carries an Instance filter (on the instance label), so one import covers the whole fleet — view every instance at once, or focus on one. Legends are sortable tables showing last/mean/max per series, so the outlier instance stands out. Dashboards are demand-driven: importing one tells your agents to forward exactly the timeseries its panels query, and nothing else. Panels are ordinary Perses panels — edit queries, add panels, or retune them after import, and your changes reshape the demand automatically.

Alerts

Section titled “Alerts”

Thresholds are conservative starting points — tune them to your workload.

Next steps

Section titled “Next steps”
Was this page helpful?

www.leansignal.io ·