The auditor
Reclamation
The rewrite commit protocol, the probe, the two-phase reap, trim-checks, orphans, and why every crash point converges
Reclamation is how dead data disappears safely: the rewrite that moves live bytes out of standing objects, the probe that judges whether an object may go, the reap that deletes it, and the walks that drive them. This page is the mechanism (the steps, the liveness rule, and the crash enumeration) shared by the duties catalogued on the duties page.
One mover, three policies
The rewrite is the one byte-moving mechanism: read spans of standing objects, publish replacement settled objects, commit the swap. Three policies drive it: settling transit bundles into per-stream objects, GC of dead-heavy objects, and the tail-fill that folds new rows into a stream's standing tail object. They differ in what they select; the commit discipline below is the same for all of them.
The liveness rule under everything: a span is live iff a ref names its object. The object's own footer is the manifest of what it holds (stream, first sequence, count, bytes, timestamps per chunk), and the refs keyspace is the only authority on whether any of it is still wanted. No other record is consulted.
The rewrite commit protocol
Announce the output
Write the orphan intent for the new key under due/orphan/, before anything exists. The key derives from what the output will cover: {bucket}/{stream}/{epoch}/{firstSeq} for a settle output, the same with ~{revision} for a rewrite of a settled object, so a retry after a crash derives the same key.
Read the live spans
Fetch the source footers (one ranged GET per distinct source) and decide what moves: for a settle, the stream's whole standing transit window; for GC, the catalog entry's chunks that outlive the trim watermark. Nothing below the watermark is ever copied.
Assemble the output
Relocate the chunks into one multipart upload. A chunk of at least 5 MiB relocates server-side (a range copy that moves nothing through the auditor's process); smaller chunks are fetched by ranged GET (through the fetch cache on the settle path) and re-encoded as data parts. Chunk bytes are reused verbatim with their checksums carried forward; only the footer is rebuilt. The rewrite budget is charged for the bytes that transited the auditor's process; the request budget for destination writes always and for source reads only when they went direct.
Publish create-once
Complete the upload conditionally: the first attempt to finish wins the key. A publish that loses the race does not replace the standing object: it reads the winner's own trailer and footer back and commits metadata describing the bytes that are durable, truncating at a whole-row boundary. A mid-row landing is corruption; the job refuses the swap loudly.
Commit visibility by CAS
One metastore write makes the output visible. A settle commits the window's catalog rows in one commit that swaps the first span row and retires the covered rows behind it. A GC or tail-fill revises the catalog entry from the expected (epoch, first, revision) to the new one, validate-or-abandon: no matching entry means another job got there first, and the output stays under its intent.
Arm the sources' watchers
Before the duty's own marker resolves, every source gets its next watcher: a chain slot's chain gets a trim-check one read grace out; any other source gets a reap-check with the marker's own probe coordinates. A killer that cannot re-find its kill (the tail-fill superseding a tail object) arms before step 5 instead.
Resolve the intent
Delete the output's orphan intent, but only once a ref names the object. The settle resolves each output's intent right after its commit. A GC revision leaves its intent standing; the orphan duty finds the ref naming it after the grace and relieves it. Then the duty's own marker resolves.
Revisions
A rewrite whose head record is still live covers the same first sequence and epoch as the object it replaces, and publish-new → repoint → reap-old demands the new object land at a different key. The revision suffix is that difference: revision 0 is the settle output at the bare derived key, and every rewrite or tail-fill lands at ~{revision} = the covered entries' maximum plus one. Overwriting in place would tear on a crash between the PUT and the repoint: refs whose offsets index into replaced bytes. Because the revision is a pure function of inputs that cannot change until the commit CAS, a crashed job's re-run derives the same key and converges; the abandoned loser of any race is an orphan under its intent.
Outputs are always standard class
Every rewrite output settles into standard storage, whatever class its sources are. Source classes come from the chain constants (a slot), the ref (a settled object), the intent (the orphan window), or the marker payload (a probe), and every consumer resolves them against the process's one class→store authority. The consequence for an Express stream: the only express-resident objects in existence are its live chain slots, held for at most the express age cap; its settled history is standard class.
The probe
The probe is the reclamation judgment for one settled object. It costs one ranged footer GET, then one ref lookup per chunk (deleted streams' chunks judged dead without a read, the rest fanned 128 abreast), and folds the live and dead byte tallies into a verdict:
| Verdict | Meaning |
|---|---|
| stand | some span is live; dead bytes are below max(64 MiB, 0.5 × total) |
| gc-worthy | some span is live; dead bytes reach the thresholds, so the live spans should move out |
| reapable | every span is dead |
| gone | the object no longer exists: an earlier, interrupted pass deleted it |
A chunk is live when a ref at its first sequence names this object and covers it. A ref that names another object, or none at all, means the span was trimmed, retired, or moved: dead either way.
The reap is two-phase
A reap-check marker carries {class, footer window, confirmed}. Its probe's all-dead verdict does not delete on an unconfirmed marker: it arms a confirmed marker one read grace out and resolves. Only a confirmed marker's all-dead probe deletes. Deletion is therefore provably no earlier than the first all-dead observation plus the read grace, which is no earlier than the last death plus the read grace, with no death timestamps stored anywhere, and regardless of when the marker was armed, how many times it collapsed, or clock skew between armers. The cost is one extra probe per object actually reaped.
Confirmed reaps spool per storage class and leave in batched deletes of up to 1000 keys; markers resolve after their batch lands, so a failed flush re-probes and finds the objects gone.
The read grace is the reader-safety contract: --auditor-read-grace (default 10m) must exceed twice any reader's cached-footer lifetime, and the read-plan TTL must be at most half of it.
Chain slots: cursors, never probes
Slots are never probed. Every absorbed slot's batch marker is armed before the checkpoint covers it, so for any slot below the checkpoint, marker standing ⇔ rewrite still owed. The trim-check advances the chain's rewritten cursor to the earliest standing marker's first slot (or the checkpoint when none stands), then deletes [trimmed, rewritten) from every bucket of the epoch's set and CASes the trimmed cursor. Slot deletion tolerates absence and the cursors refuse regressions, so a cut anywhere re-runs and converges. The auditor is the one owner of slot deletion; the absorb never trims.
Orphan judgment
An intent older than --auditor-orphan-grace (default 1h) is judged by a point stat (a keyed limit-1 listing) and, when the object exists, by its footer, recovered from the object's tail:
| Finding | Judgment |
|---|---|
| no object at the key | done; nothing to reclaim |
| shorter than a trailer, or the tail does not decode | the upload never completed → delete |
| footer intact, no ref names any chunk | never visible, or wholly superseded → delete |
| footer intact, a ref names a chunk | visible → resolve the intent; the normal lifecycle owns it |
A structural verdict deletes; a transient read fault propagates, so the judge never deletes on a read it could not complete.
Crash points
Every sequence above is an instance of one proof obligation: no interleaving of duties and cuts leaves an object with zero watchers while bytes still exist.
| Crash after | The re-run sees | Why it converges |
|---|---|---|
| the intent, before the PUT | an intent and no object | the orphan judge finds nothing at the key and relieves the intent |
| the PUT, before the commit | an intent and an unreferenced object | the retry derives the same key, meets the standing object, and describes it; or the orphan judge deletes it after the grace |
| a lost create-once race | a standing object covering less than the walk | the commit truncates at the standing coverage; the remainder stays in transit for the next pass |
| the catalog commit, before the intent resolves | a referenced object under an intent | the orphan judge finds a ref naming it and resolves the intent |
| the commit, before the source's watcher is armed | the duty's marker still standing | the marker re-runs: the source judges covered and its watcher is armed |
| arming, before the marker resolves | a duplicate arm | deterministic keys collapse; the second resolve is idempotent |
| a killer's arm-before-kill, before the kill | a premature reap-check | the probe finds a live span and stands down; one GET |
| the unconfirmed all-dead probe | a confirmed marker one grace out | the confirmed probe re-verifies before deleting |
| the batch delete, before its markers resolve | markers over gone objects | the re-probe judges gone and resolves |
| the rewritten-cursor CAS | the same standing markers | the cursor re-derives; a racing advance loses the CAS and re-reads |
| a teardown walk mid-kill | some refs gone, reap-checks armed | the walk re-derives from what stands; arming was deferred to a barrier before any delete |
| a settle-rewrite member | its marker standing, other members' outputs referenced | completed members judge as covered; only the failed member moves |
Deletes route through the object cache
When --object-cache-instances names a cache tier, the audit lane routes its deletes through it (the settle's own reads stay direct or go through the fetch cache), so reclaiming an object evicts every cached copy at once. A cache failure falls back to the direct client; the delete itself is idempotent either way.
Source documents
docs/design/012-footer-reclamation.mddocs/adr/0029-settled-catalog.mdinternal/storage/rewrite/denorm.go,internal/storage/rewrite/streamsettle.gointernal/storage/reclaim/prober.go,internal/storage/reclaim/orphan.go,internal/storage/reclaim/walker.gointernal/auditor/trimcheck.go