Metrics, dashboards, and logs

Every novad exposes one ops listener carrying Prometheus metrics, the health probes, pprof, and the admin dashboard. This page names the metrics worth a panel, the dashboards that already have them, the one alerting principle, and what the log says at each level.

The metrics listener

--metrics-listen <addr> (NOVAD_METRICS_LISTEN) serves /metrics; empty disables it and with it /healthz, /readyz, /debug/pprof/, and /admin/. The compose stack publishes it as localhost:9191 through 9193; the Fly and Kubernetes deployments scrape port 9090. novagateway has the same flag, defaulting to :9090.

Instruments are declared in OpenTelemetry dotted form (nova.append.bytes) and exported to Prometheus with unit suffixes: . becomes _, an s-unit histogram becomes _seconds (_bucket, _sum, _count), a By unit becomes _bytes, and counters gain _total. So nova.auditor.due.age is nova_auditor_due_age_seconds and nova.write.bytes is nova_write_bytes_total. The per-node dimension is Prometheus's own instance label. The gRPC RED metrics (rpc_server_call_duration_seconds) come from the otelgrpc stats handler.

The metric families

Names below are the OpenTelemetry form; apply the export rule.

Append and write path

Chain, backlog, and absorb

Read and subscribe

Object storage and caches

Auditor

Fleet and node

The Gateway exports nova.gateway.vended.bytes and .proxied.bytes (the offload ratio), .fetch.wait, .fetch.inflight.bytes, .replans, .seam.handoffs, and the footer cache's hits and evictions.

The Grafana dashboards

dashboards/ holds the JSON; the compose stack provisions it, the Kubernetes controller ships it as sidecar ConfigMaps when observability.dashboards is on, and it imports into any Grafana by hand.

Nova — Streaming Log Store (nova-overview) is the single pane: Overview (streams led, stars saturated, append p99, the oldest reclamation backlog, traffic distribution, utilization by star, sheds), Write path, Read path (latency and rate by source), Tiering — object storage (op latency and rate, bytes, average bytes per PUT), Chain — backlog, absorb & turnover (backlog age marked alert on this, resident flights and bytes, stalls by cause, admission wait, slot commit latency, epoch turnovers), Metadata — oxia, Auditor — background reclamation (backlog age by queue marked alert on this, step latency and rate, jobs armed and resolved, garbage reaped, probe verdicts, slots trimmed), Auditor — settle fetch cache, RPC — gRPC, Reliability — recovery & sheds, and Runtime.

Nova — Oxia Metastore (nova-oxia) follows a ref commit through the metastore — client latency as Nova sees it, leader and follower write latency, batch commit, WAL sync, pebble compaction — and graphs each server resource against its provisioned ceiling, so "which layer is the bottleneck" is a glance.

Nova — Object Cache (nova-objectcache) shows the cache tier's effectiveness (hit ratio, S3 traffic saved), residency, and health.

The one alert to wire first

nova_auditor_due_age_seconds is the age of the oldest past-due reclamation item. A throughput graph looks healthy while a queue silently grows; a rising age does not. Alert when it climbs past your grace windows (minutes to warn, hours to page).

The principle generalizes: alert on backlog age, never rate. A full backlog at any fill turns over within seconds and is the normal full-rate condition; minutes of standing age mean the chain is wedged or chronically overdriven. The same rule applies to nova_chain_backlog_age_seconds, nova_writer_limbo_age_seconds, and nova_gateway_fetch_wait_seconds. The Kubernetes controller ships rules for exactly these (fleet and node saturation, stranded, per-class write SLOs, writer-limbo age, rewrite-backlog age).

Beside age, three fleet signals deserve rules: nova_node_stranded at 1 (capacity, now), a sustained rate of nova_dataplane_shed_total{resource="capacity"} (producers being refused), and a sustained rate of nova_node_handoffs_total (the fleet rebalancing constantly instead of settling).

Logging

novad writes structured logs through slog. Three levels, one rule each:

Keys are snake_case, the error key is err, durations render as strings, and every message opens with its package prefix (novad:, writer:, auditor:). --log-level (NOVAD_LOG_LEVEL) takes debug, info, warn, or error; default info.

The star summary

One Info line per --log-summary-interval (default 1m0s; 0 disables), the log's proof of life and load between edges, written even when the star is idle:

novad: star summary leaders=… sessions=… subscriptions=… append_mibps=… offered_mibps=… read_mibps=…
  backlog_standard_age=… backlog_express_age=… retained_mib=… utilization=… axis=…
  draining=… drained=… saturated=… stranded=… auditor_jobs=… auditor_failed=…

Rates are MiB/s over the interval (offered is what producers submitted; append is what the chain acked; read is hot-tail bytes served); backlog_<class>_age is per configured class; retained_mib sums the admission budgets, the tail window cache, and session transport; axis names the resource behind utilization. The auditor fields appear only when one runs in the process.

pprof and health

/debug/pprof/ rides the same listener. A star that stalls without erroring is usually backpressure; the goroutine dump names the blocked stage:

curl -s localhost:9191/debug/pprof/goroutine?debug=1 | less
go tool pprof "localhost:9191/debug/pprof/profile?seconds=10"

/healthz is liveness (process alive, no dependency checks); /readyz is readiness (registered, not draining). Node lifecycle has the exact answers; the admin dashboard is at /admin/ on the same port.

Source documents

• dashboards/README.md, dashboards/nova-overview.json, dashboards/nova-oxia.json, dashboards/nova-objectcache.json
• internal/daemon/summary.go, internal/daemon/stats.go, internal/daemon/observability.go
• docs/design/000-decisions.md (lifecycle logging entry)