Duties

The auditor runs eight duties. Each drains one due queue, and each is defined by the same six facts: what arms its marker, what it must make true, the steps it takes, what it arms for the next duty, why a re-run after a crash converges, and the dials that shape it. This page is the catalogue; the mechanisms the duties share (the rewrite commit protocol, the probe, the two-phase reap) are on the reclamation page.

Summary

How the duties feed each other:

Teardown

Trigger. due/teardown/{shard}/{readyAtMs}~{stream}, armed by stream deletion at the deletion instant, with the deletion time carried as the marker's value across every re-arm. The first walk arms a second form, …~{stream}~final.

Responsibility. Reclaim everything a deleted stream left (its refs and catalog rows, the objects only it referenced, the rows that served it live) and finally release its ID, without ever racing the settle machinery over the same rows.

What it does. The first walk:

1. Gathers the transit coverage the stream still has on chain slots and judges the settle frontier of every chain it touches: if a batch marker still stands at or below the stream's last slot, the walk is gated. A gated walk ripens that chain's owed markers to now (so the next settle pass resolves them instead of waiting for the age cap), counts the gate, re-arms itself 5 minutes out with the original deletion time, and resolves the current marker. Nothing was armed and nothing killed.
2. Past the frontier, retires every ref below the stream's end. Each settled object a ref names gets its reap-check armed before the ref goes (deferred to one bounded barrier, deduplicated per object); refs into chain slots just delete: the slot's own batch marker is its watcher and the trim-check its executioner.
3. Purges the rows that served the live stream: the trim watermark, the producer handle, the last-writer record.
4. Arms the final marker 3 minutes out (a quiesce measured from this walk, not from the deletion) and resolves.

The final walk asks the keyspace whether a late deposit raced the first walk (a commit whose guards read before the walk and whose write landed just after); if so it reclaims exactly as the first walk did. Either way it purges the tombstone row, which releases the ID.

Output. Reap-checks for every settled object the stream referenced; a ~final marker; eventually a free address.

Converges after a crash because every step is idempotent: re-arms are deterministic keys, the ref walk re-derives from whatever stands, arming a reap-check twice collapses onto one marker, and a crash between arming the final marker and resolving the first simply re-runs the walk and arms a second final marker, each of which resolves its own.

Dials. Teardown jobs dispatch in groups of 16 streams, 8 walking concurrently, so a delete storm amortizes the per-job dispatch cost. Gate retry (5 min) and the quiesce (3 min) are fixed. The gate count is nova.auditor.teardown.gated.

Settle

Trigger. due/settle/{shard}/{readyAtMs}~{star}~{epoch}, armed at every chain mint (which re-announces the star's unsettled epochs with an immediate ready time), at a star's close (the durable announce that backs the close-time nudge to its peers), by decommission, and by the duty itself when it defers.

Responsibility. Drive a chain epoch nothing demands to settled: refs over every committed slot, garbage above the cut reaped, the chain marked. The commit gate settles an epoch a live open is waiting on; this duty is the janitor for the rest: a dead star's un-absorbed acked records (invisible until refs exist), the slots an unsettled epoch pins, and the decommission gate that requires every epoch settled.

What it does.

1. Judges the epoch. A missing chain record (a burned mint, a reaped star) resolves the marker. An epoch below its class's mint head is superseded (terminally halted whatever the star's liveness) and settles at once. The class head is the one epoch a live writer could still be appending: if the star is alive in the registry the marker defers 5 minutes and resolves; if it is dead, the epoch settles. This is the dead-star judgment: a corpse's chains are settled here.
2. Settles: take the settlement claim (another settler holding it leaves the marker standing), adopt the recorded cut or fence the chain from its checkpoint and publish the cut (losing the publish race adopts the winner's), absorb [checkpoint, cut) into refs in 32-slot batches with the checkpoint CASed after each, probe and delete the garbage window above the cut, mark the chain settled.
3. Resolves the marker.

When the daemon provides its settle group, the duty joins the same one-walk-per-epoch lifecycle the open gate and the close-time nudge use, so concurrent demand for one epoch shares one walk.

Output. Refs into the chain's slots, which arm batch markers for the settle rewrite exactly as a live absorb would, and a settled chain the decommission gate accepts.

Converges after a crash because the claim serializes settlers, the cut is published once by CAS and adopted by everyone after, the checkpoint advances only by CAS, and ref commits are idempotent. A settler cut anywhere is succeeded by any other, converging on the same cut.

Dials. None on the command line. The live-head probe interval is 5 minutes. The decommission verb (nova node decommission --star <id>) settles a drained star's open epochs ahead of its gate without waiting for this duty; --force skips the drained-target guard, and then the chains settle only through this duty's own liveness judgment.

Retention

Trigger. due/retention/{shard}/{expiresAtMs}~{stream}, armed once at the stream's birth commit when its configuration carries an age policy (expiry = the first ref's newest record time plus the policy), then re-armed by the duty itself after every pass. The chain sustains itself; a full trim never ends it.

Responsibility. Enforce age retention: advance the trim watermark past every record older than the policy (logical deletion; reads below it fail at once), and reclaim the refs and objects now wholly below it (the physical half).

What it does.

1. Reads the descriptor: a torn-down stream or a lifted policy resolves the marker.
2. Resolves now − max age to a sequence number over the ref-covered range: record time is the anchor.
3. If that point is above the watermark, trims. The duty routes the write to the stream's lease holder when one exists (this star if it leads the stream, otherwise the holder over the data plane) so the leader's in-memory watermark stays exact; a vacant stream trims straight to the metastore, the case an audit-only pod, which leads nothing, almost always sees.
4. Runs the below-point walk every pass, not only advancing ones: retire every ref wholly below the point (the straddling catalog row sheds only its covered entries), arming each settled object's reap-check before its ref goes. Refs into chain slots just delete.
5. Re-arms for the next expiry: the surviving boundary ref's newest record time plus the policy, or now + max age when nothing ref-covered is left.

An explicit Trim from a client is the serving side's verb and only moves the watermark; it arms nothing. Physical front-deletion follows the age point this duty computes; a stream without an age policy keeps its trimmed bytes until a rewrite drops them (the GC rewrite never relocates a chunk below the watermark) or the stream is torn down.

Output. An advanced watermark; reap-checks for the objects below it; the next retention marker.

Converges after a crash because the watermark is monotonic, the walk re-derives from whatever refs stand, arming is idempotent, and a pass that died between the trim and the walk is repaired by the next pass's unconditional walk.

Dials. The policy is per stream (from the bucket's defaults). Read grace (--auditor-read-grace, default 10m) is the reap-check delay the walk arms.

Reap-check

Trigger. due/reap/{shard}/{graceEndMs}~{objectKey} with a payload of the object's storage class, footer window, and a confirmed flag, armed by every killer of a span's referencedness: the retention and teardown walks (before the kill), the completion of a GC rewrite, and the tail-fill that supersedes a standing tail object. The grace end rounds up to a 30-second bucket so the concurrent killers of one shared object collapse onto one marker.

Responsibility. Judge one settled object and act on the verdict: delete it once every span is dead and the read grace has provably passed, hand it to GC when enough of it is dead, or stand it down.

What it does. The probe: one ranged footer GET, then one ref lookup per chunk (128 abreast), with deleted streams' chunks judged dead without a read. The verdict:

Confirmed reaps leave in batches of up to 1000 keys per class under one request token, flushed when a batch fills or at the end of the dispatch pass. Deletes route through the object cache when one is configured, so every cached copy dies with the object.

Output. A deleted object, a GC marker, or nothing.

Converges after a crash because a marker resolves only after its delete landed; a re-probe finds the object gone or dead again and converges. A premature marker (the worst outcome of arm-before-kill) finds a live span and stands down at the cost of one GET.

Dials. --auditor-read-grace (default 10m); the GC thresholds (dead fraction 0.5, minimum 64 MiB of dead bytes) are fixed. Verdicts are counted in nova.auditor.probe.verdicts.

Trim-check

Trigger. due/trimcheck/{shard}/{graceEndMs}~{star}~{epoch}, armed one read grace out when a settle-rewrite batch completes, and when a GC rewrite's source turns out to be a chain slot. One marker per touched epoch per pass.

Responsibility. Delete chain slots (as the only owner of slot deletion, for live and settled epochs alike) without ever probing one.

What it does. Slots are never probed: a standing batch marker doubles as the not-yet-rewritten flag, so slot death is cursor arithmetic.

1. Reads the chain's constants (a reaped epoch resolves the marker) and resolves its bucket set.
2. Advances the rewritten cursor to the chain's earliest standing batch marker, or to the checkpoint when none stands, one bounded prefix read in the chain's single shard. A racing advance loses the CAS and re-reads.
3. If trimmed < rewritten, deletes [trimmed, rewritten) from every bucket of the epoch's set and CASes the trimmed cursor forward.
4. Resolves.

Output. Deleted slots; nova.auditor.slots.trimmed counts them across the set.

Converges after a crash because slot deletion tolerates absence and both cursor CASes refuse regressions: a cut anywhere re-runs and converges.

Dials. None; the grace is the read grace.

Settle rewrite

Trigger. due/denorm/{shard}/{star}~{epoch}~{firstSlot}: one batch marker per span commit, armed by the absorb, the release flush, and settlement alike, carrying the batch's slot inventory (sequence and footer window per slot), storage class, birth, ready time, and whether any of its span commits observed a stream crossing the 8 MiB size trigger. All of one chain's markers land in one shard, ordered by first slot; a separate ready-ordered index is the schedule the drain walks. Ready time is birth + age cap (24h Standard, 1h Express), or birth + linger (5m) when a crossing rode in the batch.

Responsibility. Settle transit bundles into per-stream objects in standard storage (the read layout long-term fetches want, and what lets the bundle's slots be trimmed) within the class's age cap, and at size-trigger cadence for streams that earn it.

What it does. A dispatched job settles an era: a contiguous run of one chain's ripe markers, gathered up to --auditor-markers-per-era and extended with the chain's markers that ripen while the job runs (for at most half the job deadline). Only one era per chain runs fleet-wide, under a chain lease; a marker whose chain is placed on another auditor or already running here is skipped, left standing.

For each marker, in slot order:

1. Judge before any download: fetch the slots' footers (ranged GETs on the plain store), and for each stream they name decide whether it still has standing bytes on the chain: a tombstoned stream is out at once (its spans belong to teardown, and the marker may resolve over them), a stream the era already settled past this marker is covered without a read, otherwise a ref lookup finds its first standing position. An empty plan resolves the marker with no slot body staged.
2. Warm: walk each standing member's transit window, union the slots they reference, and stage them into the fetch cache asynchronously, racing the walk.
3. Settle members, --auditor-member-parallel (default 8) at a time. Each member settles its whole transit window (spans of every batch in its standing run, not just this one), so the first marker of a run does the moving and the rest resolve by coverage. The skip rule: a linger-ripened marker settles only members whose window meets the 8 MiB trigger; a cap-ripened one takes everything. The window first folds into the stream's standing tail object up to the output cap (the tail-fill, a revision of that object), then splits at row boundaries into outputs of at most --denorm-output-cap-mib; each output is published create-once at its derived key under an orphan intent, one catalog commit swaps the whole window visible, then each intent resolves. The full sequence is on the reclamation page.
4. Close the marker: a marker that settled everything arms the chain's trim-check one read grace out and resolves. A marker with residue (members skipped by the trigger rule, or a plan cut at the member cap) is re-stamped to its age cap with the crossing flag cleared, never resolved early and never left permanently ripe.

Output. Settled objects and catalog rows; a trim-check per completed chain; nova.auditor.settle.bytes and nova.auditor.settle.spans count the work.

Converges after a crash because every step re-derives from durable state: output keys derive from what they cover, publishes are create-once (a lost race converges on the standing object and commits what it durably covers), the catalog commit is one CAS, completed members judge as covered on the re-run, and outputs a crashed run published but never referenced stay under their intents for the orphan duty.

Dials. --auditor-markers-per-era (default: one marker per job), --auditor-member-parallel (default 8), --denorm-output-cap-mib (default 256), --denorm-linger / --express-denorm-linger (default 5m), --denorm-age-cap (default 24h), --express-denorm-age-cap (default 1h), the fetch cache flags. The warm set per sub-era (2 GiB) and the member plan per pass (2048) are fixed bounds. Backlog age for this queue measures how far past due the oldest marker sits: a standing unripe marker is the policy working, not a backlog.

GC rewrite

Trigger. due/gc/{shard}/{inverted dead bytes}~{objectKey} with the same probe payload a reap-check carries, armed by the reap probe when an object's dead bytes reach the thresholds. Keyed biggest reclaim first; every marker is ready.

Responsibility. Reclaim the dead share of a partly-dead settled object by moving its live chunks out, so the reap can finish the husk.

What it does. A settled object is named by exactly one catalog entry, and its only death is the trim's:

1. Locates the entry naming the object (the stream's descriptor and one ref lookup). A deleted stream, a missing ref, or a ref naming another object means the object is only reap fodder.
2. Otherwise fetches the footer, reads the trim watermark, and lists the chunks that outlive it.
3. Publishes them at the next revision of the same key under an orphan intent, charged to the rewrite budget, and CASes the catalog entry from the old revision to the new, validate-or-abandon: no matching entry means another job got there first and the output belongs to its intent.
4. Arms the source's watcher (a trim-check if it was a chain slot, otherwise a reap-check one read grace out) and resolves the marker, in that order, so the object is never watcherless.

Output. A new revision; a reap-check on the source; the revision's own intent, which the orphan duty relieves once it finds a ref naming it.

Converges after a crash because the revision key derives from inputs that do not change until the commit CAS, the CAS validates or abandons, and the marker stands as the watcher for the whole pass.

Dials. Dead fraction 0.5 and minimum reclaim 64 MiB are fixed; the rewrite and request budgets pace it.

Orphan

Trigger. due/orphan/{shard}/{objectKey}: a write intent every auditor publish arms before its PUT, carrying the object's storage class and birth time. Resolved by the publisher after its commit. Chain slots publish no intents: their commit is the PUT itself.

Responsibility. Reclaim objects that never became visible (a crash between the PUT and the commit, or a publish that lost its race), so no object can leak that the system did not pre-announce.

What it does. The queue orders by object key, which is no age order, so ripeness is judged per intent against --auditor-orphan-grace (default 1h) and an unripe intent is skipped, not a stop. For an aged intent:

Every outcome relieves the intent; the marker resolves regardless. A transient read fault propagates and the judge never deletes on a read it could not complete.

Output. Deleted orphans; reclaim: orphan reclaimed in the log.

Converges after a crash because the intent stands until judged, the stat and footer read are pure, and the delete is idempotent.

Dials. --auditor-orphan-grace (default 1h); keep it above the slowest legitimate publish-to-commit delay.

What is not a duty

• Decommission residue is an admin verb, not a queue: nova node decommission settles a drained star's chains, gates on every epoch settled, writes the irrevocable record, ripens the star's whole denorm schedule (the default force-drain; --cool-down lets the armed clocks drain instead), reaps its chain records, and sweeps its chain prefix: the one sanctioned bucket listing, over a finite dead keyspace, refused while any slot still owes its rewrite.
• Coalescing small settled objects is not a separate duty. The tail-fill inside the settle rewrite folds new rows into the standing tail object, and revisions never overwrite in place.
• There is no periodic verification sweep and no seal backstop over replicas; every leak is caught by an intent, a marker, or a ref.

Source documents

• docs/design/003-auditor.md, docs/design/012-footer-reclamation.md
• docs/design/016-denorm-policy.md, docs/design/018-settle-at-density.md
• docs/adr/0024-settle-superseded-epochs.md, docs/adr/0026-demand-driven-settlement.md
• internal/auditor/ (one file per duty), internal/storage/rewrite/, internal/storage/reclaim/, internal/storage/chain/settle/