The auditor
Observability
Watching the auditor — backlog age, the metric names, the admin UI's audit views, log lines, and how to read the signals
A throughput graph looks healthy while a queue silently grows; a rising age does not. Everything on this page is in service of one rule (alert on backlog age, never rate) and of knowing which dial to reach for when the age climbs.
Alert on backlog age
nova.auditor.due.age (Prometheus: nova_auditor_due_age_seconds{queue}) is the age of the oldest pending marker past its ready time, per time-ordered queue, read as one bounded peek per owned shard. Alert when it climbs past your grace windows (minutes to warn, hours to page) and read it per queue, because each queue's age names a different failure (the table on the due queues page).
Two properties shape the rule:
- An empty queue reports nothing, and so does a dead auditor. Wire an absent-signal companion: no due-age series while
nova_append_bytes_totalclimbs is itself the alert. - An unripe marker is not backlog. Under an age cap a standing batch marker is the policy working; the
denormreading is how far past due the oldest entry sits, never how old it is. The settle contract is lateness at linger scale: alert ondenormage above twice the linger sustained for ten minutes.
Rates (armed and resolved per second) are sizing signals: read them against each other for the flow balance and against append bytes for capacity versus demand.
The metric names
Instruments are declared in dotted form and exported to Prometheus with .→_, _seconds on second-unit histograms, _bytes on byte units, and _total on counters.
| Instrument | Kind | Labels | Meaning |
|---|---|---|---|
nova.auditor.due.age | gauge (s) | queue | oldest past-due marker's age: the alert |
nova.auditor.step.duration | histogram (s) | step | one leased job's handle latency |
nova.audit.jobs.armed | counter | queue | markers armed (re-arms count) |
nova.audit.jobs.resolved | counter | queue | markers resolved, the orphan intent's commit-time relief included |
nova.auditor.lease.skips | counter | queue | claims lost to a peer; climbing against a standing marker means a wedged holder |
nova.auditor.teardown.gated | counter | — | first walks re-armed behind the settle frontier |
nova.auditor.probe.verdicts | counter | verdict = stand, observe, confirm-reap, gc, gone | reclamation probe judgments |
nova.auditor.slots.trimmed | counter | — | chain slots deleted, across every bucket of the set |
nova.auditor.garbage.reaped | counter | — | zombie slot copies settlement deleted above the cut |
nova.auditor.settle.spans | counter | outcome = swapped, covered | batch settlements: bytes moved, or resolved by coverage alone |
nova.auditor.settle.bytes | counter (By) | — | bytes the settle relocated: the rewrite budget's demand signal |
nova.auditor.group.members | histogram | — | markers settled per era job; below the configured width means a thin queue, not a fault |
nova.auditor.member.latency | histogram (s) | — | one member settle's wall clock; with the member parallelism, the walk's throughput denominator |
nova.auditor.member.phase | histogram (s) | phase = window, tailfill, publish, commit, resolve | where a member's latency lives |
nova.fetchcache.hits, .downloads, .download.bytes | counters | — | reads served from scratch versus whole-object downloads: the dedup factor |
nova.fetchcache.evictions | counter | — | files evicted for admission; sustained growth under an active batch means the budget is too small |
nova.fetchcache.fallbacks | counter | — | reads degraded to ranged GETs because the budget could not admit the object |
nova.fetchcache.prefetch.dropped | counter | — | warming hints dropped at the queue; harmless, but a sustained rate means warming lags the settle |
nova.fetchcache.resident.bytes | gauge (By) | — | scratch in use, in-flight downloads included |
Budget utilization has no gauge of its own: read settle.bytes per second against --auditor-rewrite-mib, and the object-store request rate on the audit pods against --auditor-s3-budget. Dead bytes awaiting GC are visible as the gc queue's depth in the admin UI; unresolved-intent age as the orphan queue's peek.
The admin UI's audit views
The admin dashboard rides the ops listener behind --admin-ui and mirrors read-only admin-plane RPCs; see admin UI. Three views cover the auditor:
- Audits:
ListAuditQueuessummarises six queues (retention,reap,gc,denorm,teardown,orphan) with a bounded depth (counted to a cap and rendered asNorcap+; an exact depth would be a full scan) and, for time-ordered ones, the oldest ready time.PeekAuditQueueshows the first entries of one queue across its shards, each with its raw key, ready time, subject, and attributed streams: a bounded peek, never a drain. - Stream detail:
GetStreamDetaillists the stream's pending background jobs from the by-stream index: job type, the raw due key (the operator's handle for correlating logs), ready time, the object for object-scoped jobs, and whether that object is shared with other streams. - Overview: the auditor health strip beside the fleet table.
Pending state only: a marker vanishes when drained, so "when was this stream settled" is history the surface does not keep.
What the daemon logs
The auditor follows the fleet's logging discipline: Info for lifecycle edges, Debug per job, Warn for failures the queue retries. Every message opens with its package prefix.
Lifecycle edges at Info:
| Line | Edge |
|---|---|
auditor: running | the instance announced; carries workers and interval |
auditor: stopped processing jobs; the fleet owns the queues | a drain's quiesce took effect |
auditor left the ring; the seat is vacated | the exit withdrew; peers re-deal from the next pass |
novad: auditor quiesced; no new duty claims | the daemon's drain, with the in-flight grace |
auditor: settling a superseded epoch; its successor's gate waits on it | a restart left residue and the fleet is recovering it |
settle: epoch settled / settle: settlement reaped garbage slots above the cut | a chain reached settled; zombie copies deleted |
auditor: retention trimmed | the watermark advanced for a stream |
auditor: chain trimmed | slots deleted; rewrite_owed is what still stands above the rewritten cursor |
auditor: stream teardown walked; quiesce re-walk armed / auditor: stream torn down | the two teardown walks |
auditor: teardown first walk gated behind the settle frontier / auditor: teardown gated: ripened the chain's owed markers | a gate and its expedite |
reclaim: probe armed gc / reclaim: objects reaped by probe / reclaim: orphan reclaimed | probe outcomes that changed storage |
reclaim: decommission force-drain ripened the star's denorm schedule | a decommission's residue walk |
rewrite: settle converged short of the walk / rewrite: tail-fill converged on the standing revision / rewrite: tail-fill abandoned | a lost create-once race, resolved |
Per job at Debug: auditor: job done with the step, the marker key, the record count, and the duration. At Warn: auditor: job failed; the queue retries next pass, auditor: queue walk failed; the shard retries next pass, auditor: marker is malformed; resolving it away, auditor: job lease contended past the grace; the holder may be wedged, auditor: reap flush failed; the markers re-probe next pass, and rewrite: era warm aborted; settles fall to demand reads. A whole pass failing is the one Error.
The star summary (the periodic Info line with a star's aggregates) carries auditor_jobs and auditor_failed (cumulative finished and failed jobs) whenever an auditor runs in the process, so the log's proof of life covers the background role too. An audit-only pod publishes the same shape into the node stats record every 5 seconds: utilization on the cpu axis, version, start time, never a routing record.
Dashboard rows
dashboards/nova-overview.json (see operations observability) covers the auditor in three places:
| Row | Panels |
|---|---|
| Overview | Oldest reclamation backlog: the single reliability number |
| Auditor — background reclamation | Reclamation backlog age by queue (ALERT ON THIS), Audit jobs armed/s by queue, Audit jobs resolved/s by queue, Auditor step latency by step (p99), Auditor step rate by step, Reclamation probe verdicts, Settlement garbage reaped |
| Auditor — settle fetch cache | Reads: served from disk vs downloaded, Degradations: fallbacks and dropped hints, Scratch resident bytes by pod, Download throughput, Evictions (empty under the per-range strategy) |
| Chain — backlog, absorb & turnover | Chain slots trimmed |
On the compose stack the audit-split container's metrics land on localhost:9194 and under the same panels; Prometheus already lists it as a target.
Reading the signals
| Signal | Meaning | Action |
|---|---|---|
denorm age above twice the linger, sustained | settle capacity below the write rate | raise the rewrite and request budgets or add auditors; check fetch-cache fallbacks first |
denorm age zero with markers standing | markers unripe: the age cap or linger is doing its job | none |
settle age climbing | a dead or superseded chain is not being settled: its records are invisible and its slots pinned | check lease skips and the settler's error lines; a chain-store resolution failure names the bucket set |
teardown age climbing, teardown.gated flat | teardown itself is slow or failing | read the job-failed lines for the stream |
teardown age climbing, teardown.gated climbing | a chain's settle backlog holds the gate | fix the settle; teardown follows within one pass of the frontier passing |
retention age climbing | expired data still readable and stored | the trim write routes to the lease holder; check holder reachability, then the request budget |
reap or trimcheck age climbing | dead objects and slots past their grace | the request budget, then object-store health |
lease.skips climbing against one marker, with the wedged-holder Warn | one member is stuck on a job | the 10-minute job deadline frees it; find the backend it is stuck on |
probe.verdicts{gc} high | many partly-dead objects: a delete- or trim-heavy workload | expected; watch the gc queue's depth for the reclaim awaiting the rewrite |
probe.verdicts{stand} dominating after teardowns | premature markers from arm-before-kill | expected, one GET each |
fetchcache.fallbacks rising | the scratch budget cannot admit the era's warm set | grow --auditor-fetch-cache-mib or the slice's scratch |
settle.spans{covered} far above {swapped} | eras are resolving markers by coverage: the first marker of a run did the moving | healthy |
| audit-only pod not-ready | its ring announcement is not live | the metastore's ephemeral namespace, or the pod is exiting |
| no due-age series while append bytes flow | no auditor is announced, or none owns shards | check the ring: --no-audit everywhere with no audit pool leaves every seat unserved |
Source documents
dashboards/README.md,dashboards/nova-overview.jsondocs/design/008-admin-ui.md,proto/nova/v1/adminplane.protointernal/auditor/auditor.go,internal/storage/rewrite/mover.go,internal/storage/reclaim/prober.go,internal/storage/objectstore/fetchcache/metrics.gointernal/daemon/summary.go,internal/daemon/auditstats.go