The auditor
Tuning
Sizing the auditor to ingest — budgets, workers, the fetch cache, grace windows, denorm policy, and the audit pool
The auditor's settle rewrite moves every byte the write path commits, so its capacity is part of a fleet's ingest rating: settle throughput must sustain the write rate, or the backlog ages. This page names every dial, what each protects, and how to size an audit pool.
The budgets
Two token buckets pace every job before it touches storage. Both burst one second's worth.
| Flag | Default | What it meters | What it protects |
|---|---|---|---|
--auditor-s3-budget | 50 req/s | Background object-store requests: probe GETs, destination writes, deletes, and source reads that go direct (a read served by the fetch cache paid at download time) | Foreground I/O's share of the object store's request capacity |
--auditor-rewrite-mib | 64 MiB/s | Bytes a rewrite moves through this process: ranged GETs re-encoded and re-PUT, and the fetch cache's downloads. Server-side copies (chunks of 5 MiB and up) are free of it | The host's network and the ingest path's bandwidth |
Both are per auditor process; the fleet's settle capacity is the sum across members. The --help text says it plainly: size them to sustain ingest or the rewrite backlog ages. The design also calls for a third budget on metastore operations per second; today no such flag exists (see the last section).
Workers and dispatch
| Flag | Default | Effect |
|---|---|---|
--auditor-workers | 2 | Jobs in flight per process: the throughput dial for metastore-latency-bound work (teardown walks, GC, probes). No duty may hold more than half; with one worker there is no fairness reserve |
--auditor-markers-per-era | 0 (one marker per job) | Ripe batch markers one settle job gathers as an era. Higher amortizes the per-job cost and the per-chain judge across a contiguous slot run; bounded also by 64 MiB of marker payload per gather round |
--auditor-member-parallel | 8 | Concurrent member settles inside one batch walk. A chain's era runs as one job, so serial member latency would cap its settle rate below the chain's production rate |
Throughput is gated by the budgets, not worker count: a huge pool adds held leases and memory, but no drain rate. Raise workers when the step-latency histogram shows jobs waiting on the metastore rather than on a budget.
The fetch cache
The settle reads each transit bundle once per stream it carries; at high stream density the per-extent ranged GETs amplify across the many streams sharing one bundle. The fetch cache is the alternative strategy: download each bundle whole onto local scratch, once, and let every consumer (the judge's footers, the window walks, the assembly's sub-cutoff extents) read the file.
| Flag | Default | Notes |
|---|---|---|
--auditor-fetch-cache-dir | empty (per-range reads) | Enables the cache. Startup wipes the directory's contents: point it at dedicated scratch (a Kubernetes emptyDir), never at data |
--auditor-fetch-cache-mib | 40960 | Scratch the cache may keep resident. A read the budget cannot admit falls back to a ranged GET: the job never blocks on disk and never fails because of the cache |
--auditor-fetch-parallel | 64 | Concurrent background downloads warming the cache ahead of an era's walk |
The cache is never authoritative; losing any file at any instant costs a re-download. Downloads pay both budgets. On Kubernetes the pool's slice declares scratch, rendered as an ephemeral-storage request plus the volume's size limit, with the in-process budget derived at 90% of it so admission backpressure, never kubelet eviction, is the pressure valve. An explicit scratch of zero renders the per-range posture: no disk, no fetch cache, every extent a ranged GET. That is cheap when the object cache tier's page widening absorbs the amplification, and the pool no longer needs machines with local disks.
Grace windows
| Flag | Default | Rule |
|---|---|---|
--auditor-read-grace | 10m | The window between an object's last death and its physical delete. Keep it above twice any reader's cached-footer lifetime |
--readplan-ttl | 5m | Read-plan validity: presigned URL expiry, and the advisory fetch bound for raw consumers. Must be at most half the read grace; startup refuses otherwise |
--auditor-orphan-grace | 1h | The age at which an unresolved write intent is judged. Keep it above the slowest legitimate publish-to-commit delay |
Shortening the read grace makes reclamation prompter and shrinks the window a vended reader has to finish a fetch; the compose stack runs 2m with a 1m read-plan TTL for demos.
Denorm policy
Two timers per class govern when a batch marker ripens; the size trigger itself is a fixed 8 MiB of accumulated un-settled bytes per stream.
| Flag | Default | Meaning |
|---|---|---|
--denorm-linger | 5m | Delay from a Standard stream crossing the size trigger to its batch settling; it batches crossings into rounds so outputs grow past the trigger |
--express-denorm-linger | 5m | The same for Express streams |
--denorm-age-cap | 24h | Hard bound on a Standard byte's transit residency before its batch settles |
--express-denorm-age-cap | 1h | Hard bound for Express: express storage bills for the transit window across every bucket of the set, so keep it short |
--denorm-output-cap-mib | 256 | Cap on one settle output; an over-cap window splits at row boundaries |
The age cap is what makes a decommission bounded: a retiring star's chains owe at most one cap's worth of ingest. The size trigger with its linger is the operating point; the age cap is the backstop for trickle streams, never the operating point: a fleet whose settle floats at the cap is holding cap-scale untrimmable transit and cap-scale span rows.
The audit-only pool
Split reclamation onto its own pool (--audit-only pods, serving stars on --no-audit) when audit traffic (bundle downloads, rewrites, retention deletes) must not contend with ingest on the serving stars' network. The settle tier receives every ingested byte and transmits it again, so at production ingest it is its own bandwidth line item; embedded in a serving star it competes with appends for the same interface.
The reference pool slice is 3.5 Gbps of wire, 4 cpu, 8Gi/12Gi memory, and 300Gi of scratch, two auditors per NVMe-backed machine. The controller's audit spec carries:
| Field | Default | Meaning |
|---|---|---|
replicas | — | The pool's size; auditors carry no identity and no ladder, quiesce on SIGTERM, and shrink by plain deletion |
s3Budget, rewriteMiB | daemon defaults | First-class dials: reclamation-throughput sizing every real deployment must set |
slice.scratch | reference | The fetch-cache disk; zero renders the per-range posture |
autoscale.min, autoscale.max | — | Bounds on the sizing loop |
autoscale.ageThreshold | 5m | Due age past it means reclamation is behind |
autoscale.busyPercent | 50 | A pool at or above it while behind is starved: one worker joins. Below it the pool is wedged: more workers fix nothing, the pool freezes, and the AuditWedged condition surfaces |
autoscale.lowWaterPercent, autoscale.scaleDownAfter | 30, 30m | Fresh queues and a cold pool for the whole window release one worker; ring churn re-deals shards and re-warms caches, so the pool never flaps |
Sizing by hand: the settle's cost is the bundle bytes it downloads (≈ the pool's ingest share, once eras make each download serve every stream it carries) plus the outputs it writes. Provision rewriteMiB × replicas above the sustained write rate with margin, s3Budget to match the request rate that implies, and scratch large enough to hold an era's warm set (2 GiB per sub-era per job) many times over.
Symptoms
| Symptom | Likely dial |
|---|---|
denorm due age climbing while append bytes flow | settle capacity below ingest: rewriteMiB, s3Budget, replicas; check the fetch cache's fallbacks first |
nova.fetchcache.fallbacks or prefetch.dropped rising under an active era | scratch budget too small for the warm set, or downloads lagging: --auditor-fetch-cache-mib, --auditor-fetch-parallel |
nova.fetchcache.evictions sustained during a batch | the same; the budget cannot hold one era |
| Jobs slow in the step-latency histogram, budgets idle, metastore p99 high | metastore-bound: --auditor-workers, or the metastore plane itself |
teardown age climbing with teardown.gated climbing beside it | a chain's settle backlog holds the gate; fix the settle, not teardown |
reap or trimcheck age climbing | read grace and the request budget: the probe pays one GET per object |
orphan depth high after an incident | expected: intents from cut jobs are judged after the grace; lower --auditor-orphan-grace only if publishes never take that long |
| Lease skips climbing with a Warn naming a wedged holder | a stuck backend on one member; the job deadline (10m) frees it |
| Express transit bytes standing longer than an hour | --express-denorm-age-cap is the bound; if age stays at zero the settle is keeping up and residency is the cap, by design |
What not to tune blindly
The auditor shares the metastore with the append path: a chain commit and a settle's catalog commit ride the same oxia round trips. When 200,000 streams born in one minute reached their settle time together, the synchronized dispatch collapsed metastore write p99 from milliseconds to over a second; chain commits stalled, admission budgets filled, and appends queued for tens of seconds: the serving fleet was the victim of background work that looked, on its own dashboards, perfectly healthy. The design ruled two remedies: a deterministic per-stream jitter on the linger so a cohort ripens spread over half a linger, and a budget on the auditor's metastore operations per second (2000 by default) whose breach defers work into due-queue age. Neither exists in the code today; the flags on this page are the whole surface. So: raise --auditor-workers, the budgets, and --auditor-markers-per-era in steps, and watch the metastore's write p99 and the append path's backlog stalls alongside the auditor's own due age. A budget that lets the auditor drain faster than the metastore can absorb moves the outage from the background to the front door.
Source documents
go run ./cmd/novad --helpdocs/design/018-settle-at-density.md,docs/design/023-constellation-controller.mddocs/perf/2026-08-18-settle-storm-forensics.mdoperator/README.md,internal/auditor/budgets.go,internal/auditor/fetchcache.go