The auditor

The auditor is the half of novad that never touches an append or a read. It settles freshly committed transit data into per-stream objects, reclaims dead bytes, enforces the physical half of retention, tears down deleted streams, and settles the chains of stars that will never return. This page says what the auditor is and is not, how it is deployed, and how its runtime is put together; the other five pages in this section cover each duty, the due queues, the rewrite protocol, sizing, and observability.

The background role, never the hot path

A running novad (a star) has two roles. The serving role owns the data plane: appends land in chain slots, the absorb takes released flights into per-stream metadata, reads come from the tail window or from settled objects. The auditor role runs beside it, in the same process by default, and does everything asynchronous. Nothing a producer or consumer waits on ever passes through it.

The split of ownership is strict:

Only the auditor deletes a slot, a settled object, or a transit bundle. That single ownership is what makes the crash-safety arguments on the reclamation page tractable.

Deployment shapes

The same image runs in three shapes:

Startup refuses --audit-only with --no-audit: it would leave nothing to run.

The production shape when background I/O must not compete with serving is the audit-split topology: serving stars on --no-audit, and an identity-less pool of --audit-only pods. Duty routing is rendezvous over the ring, so with the stars out of the ring every duty lands on the pool. On the compose stack one command flips both halves:

NOVA_NO_AUDIT=true docker compose --profile audit-split \
  -f deploy/docker/docker-compose.yml up -d --build

An audit-only pod's readiness probe is its ring membership: it reads ready once its announcement is live (the same record duty rendezvous reads) and not-ready from the moment it withdraws. It publishes a node stats record (utilization, version, start time) so the overview and a controller sizing the pool can see how busy reclamation runs, but never a routing record: an auditor is not a star. See Kubernetes for the pool's slice and autoscaling, and tuning for when to split.

The ring deals seats

Every running auditor announces a session-ephemeral record under cluster/auditors/{star}. The roster those records form is the ring, and the ring deals seats:

• The due keyspace is frozen at 64 shards. Shard s belongs to the live member with the highest rendezvous score for it. A member not yet announced owns nothing and reads nothing.
• Ownership is for discovery only. Discovery is a stateless range read, so a membership change moves which member reads which shards and transfers no state. An overlap during a transition costs duplicate work, never correctness: every mutation is CAS-guarded and idempotent regardless of who runs it.
• The settle rewrite is placed per chain, not per shard: registered stars are dealt one per seat in sorted order, so one auditor owns all of a star's chains and can settle them as contiguous eras. Its markers are visible in every shard, and a member skips the ones placed elsewhere.
• A quiesced auditor withdraws from the ring. Run withdraws on every exit (a drain's stop, a cancelled context, a fatal pass) after its job pool has drained, so "stopped processing" always means "not a member". Without that, peers would keep dealing shards and chains to a seat nobody serves, and the work on it would wait until the process's session died.

A seat nobody serves is work nobody does. That is why a drained star's embedded auditor leaves the ring before the drain reports complete, and why an audit-only pod that stops announcing reads not-ready.

Runtime anatomy

One auditor is a loop over discovery passes, every 30 seconds while idle and immediately again while ready work remains:

1. Discovery readers. For each registered duty and each owned shard, one pull-iterator over the queue's key-ordered scan. The metastore's paging is the only prefetch; no marker is buffered beyond a page.
2. The dispatch pass. The pass rotates over the sources in priority order and offers each one leased job per round to the worker pool. It ends when every source is spent or its 30-second budget expires; ownership and leaked leases refresh at the boundary. How a pass forms eras and groups is on the due queues page.
3. Bounded workers. --auditor-workers (default 2) jobs run at once. No duty may hold more than half the pool, so a fat queue only delays its own completion. A job has a 10-minute deadline: every duty converges from a cut anywhere, so the deadline turns a wedged job into a released lease and a logged retry.
4. Budgets. Every job pays the object-store request budget and the rewrite-byte budget before it touches storage: background work must never starve foreground I/O. The tuning page sizes them.
5. Leases as an optimization. A job claims a session-ephemeral lease under auditor/lease/{jobKey} before it runs. Losing the claim means a peer is on it: skip, never wait. Correctness never rides a lease; the holder's session death simply re-surfaces the job.

The duty harness

Every background job family is a duty with exactly two methods. Judge triages one raw marker before any lease is taken (pure key work against the clock, no I/O) and returns one of four verdicts: run the job, stop the walk (the queue is ordered and this marker is not yet due, so everything after it is younger), discard the marker (malformed beyond use; resolved away without a lease, since nobody could ever run it), or skip it (leave it standing and keep walking, for queues whose order is not an age order). Handle runs one leased job to convergence.

A registration binds the duty to its queue and its scheduling facts: whether the queue is time-ordered (feeds the backlog-age gauge), whether it scans every shard or only owned ones, whether jobs are grouped, and whether markers carry their own lease. Registration order is the dispatch priority, offered a worker first each round, never allowed to monopolize the pool:

Everything each duty does not share (its knobs, dependencies, and instruments) lives on its own struct; the lease machinery, the deadline, the discard path, and the step-latency instrument are the harness's. The duties page catalogues all eight.

Invariants

• All metadata mutations are CAS. Cursors refuse regressions, catalog swaps validate or abandon, the trim watermark only advances.
• Idempotent and crash-convergent. A re-run after a crash at any step converges; every duty is cut-anywhere convergent, which is what lets a job deadline lose nothing.
• Never a scan. Discovery is a bounded, ordered range read over owned shards; liveness is a point lookup; the only bucket listing anywhere is the decommission sweep of a retired star's dead chain prefix.
• Every object has a standing watcher. A ref that names it, its orphan intent, or a due marker: at least one, at every instant. Killers arm the watcher before the kill; an intent resolves only once a ref names its object.
• Work self-announces. The foreground arms the due marker in the same commit that creates the condition. Nothing sweeps for work.
• Bounded everywhere. Workers, per-duty in-flight caps, fan-outs, the fetch cache's scratch budget, the era's warm set and member plan: every population has a limit and backpressures at it.

Source documents

• docs/design/003-auditor.md
• docs/adr/0001-due-queue-auditor-discovery.md
• docs/adr/0022-auditor-duty-harness.md
• docs/adr/0028-auditor-dispatch-pass.md
• internal/auditor/auditor.go, internal/auditor/harness.go, internal/auditor/ring.go