Settlement

An acknowledged append is durable the moment its flight lands in a chain slot, but a chain slot is a star's writing history, not a stream's. Everything that happens next (refs that make the record queryable from any star, settled objects that give each stream its own history in standard storage, and the retirement of the slots left behind) is metadata and background work that never touches the ack path. This page follows the byte from acknowledgement to rest; the auditor's own machinery is in the auditor.

The absorb turns slots into refs

The absorb is the leader's background consumer of its own chain. It takes released flights from the leader's flight index and commits, per stream and per batch, one span ref: an immutable metastore row keyed by the stream's first record in the batch, whose value carries a per-slot index (slot, first ID, footer window) plus the stream's cumulative bytes and time bounds. The row is its own seal: it carries the epoch and extent every later guard reads. When the batch's rows are down, the chain's checkpoint advances past the batch, and the backlog tokens those flights held return to producers. Metadata only: the absorb never touches object storage.

It is wake-on-work: a coalescing nudge from the release path runs a pass as soon as flights resolve, so tokens release at commit latency instead of once per pass interval. Two knobs shape a pass at high stream density:

Admitted throughput is bounded by backlog capacity divided by pass duration, which is why the absorb's pass is the write path's throughput formula. See write path for the chain the absorb consumes and backpressure for the budget it releases.

The transit window

Once ref-covered, a record is readable from any star, but its bytes still sit in a transit bundle, interleaved with every other stream that flushed in the same flight. The span of a stream in that form is its transit window, and it is the only span a reader ever meets as chain extents. Two timers per class bound it:

• The size trigger and its linger. The absorb detects, from the span rows' chained byte totals, when a stream's un-settled bytes cross 8 MiB; the batch's marker then ripens at birth + linger. The linger batches crossings into rounds, so a hot stream settles once per round with everything it accumulated, never once per threshold. A deterministic per-stream jitter of up to half the linger spreads cohorts of streams born in the same minute.
• The age cap. A batch with no crossing ripens at birth + age cap: the hard bound on how long any committed byte stays transit-resident, and therefore also the bound on slot lifetime, span-row accumulation, and how long a decommission takes.

The express cap is short by design: express-class storage bills for the transit window, so the express chain is the accumulation medium only up to its cap, and every settled object lands in the standard bucket regardless of stream class. Reads have no side effects on any of this: nothing a reader does triggers a settle. The worst a reader meets is the transit window, at most 8 MiB plus a linger's worth of appends, which the cold reader's prefetch turns into a throughput detail.

The group settle

The auditor schedules settling in chain coordinates, never by stream. An era is a contiguous run of one chain's slots whose batch markers have ripened; one group settle job owns it. The job first judges each marker on the plain store (a marker whose streams are all deleted resolves without a byte staged), then warms exactly the era's bundles into the auditor's fetch cache on local scratch, and assembles every qualifying stream's settled object from that warm copy through a bounded worker pool. Each bundle is read once, however many streams it carries; one settled object per stream comes out. A linger-ripened era settles a live stream only if its standing window is at least the trigger (below that, settling would mint the sub-target fragments the policy forbids), while a cap-ripened settle takes everything standing. A marker left with residue re-schedules to its age cap.

Assembly is server-side where it can be: a bundle extent of 5 MiB or more is a range copy, smaller extents are ranged GET plus upload part, mixed in one multipart upload. Each settle first extends the stream's standing tail object toward the output cap by server-side copy, so a stream has at most one sub-cap settled object, always the last. The publish is create-once: a job that loses a race to a concurrent settler reads the standing object's footer back and commits metadata describing the durable bytes, never replacing them.

The commit is what makes settled bytes visible. Settled objects have derivable keys (streams/<bucket>/<stream>/<epoch>/<firstSeq>, with a ~<revision> suffix for later rewrites), so no key is stored. The job creates any non-pivot catalog rows, then compare-and-swaps the span ref at the window's first sequence into the catalog row covering the window (that single CAS is the visibility commit), then deletes the remaining covered span refs in ascending order, then resolves the batch markers. A concurrent floor lookup lands either on a surviving span ref, still valid because the slots outlive it, or on the catalog row. A crash at any step re-runs to convergence; each stream's settle converges independently inside the job.

Below a chain's settle frontier (the first key of its due range) no settle actor ever writes again; it only advances.

Trim-check retires the slots

A chain slot is never probed. Its batch marker doubles as the not-yet-settled flag: when the markers below a position are gone, the chain's rewritten cursor advances past it, and the trim-check deletes [trimmed, rewritten) across the epoch's whole bucket set as pure cursor arithmetic, behind the read grace, so a reader holding a footer it fetched moments ago is never cut off. The auditor is the one owner of slot deletion, for live and settled epochs alike. Trim, GC, and retention on settled objects are described in reclamation and retention.

Settling a stopped chain

Settlement in the crash-recovery sense restores "refs cover every acked byte" for a chain whose writer is gone, before any successor serves. It runs a fixed walk: fence the epoch to its cut (or adopt a recorded one), walk the backlog's footers above the checkpoint and feed each slot through the same absorb consumer the live leader uses (so the two paths cannot diverge), then probe one pipeline window above the cut for garbage slots and reap them, and mark the chain settled. Every step is idempotent and CAS-guarded; concurrent settlers serialize on one claim, and whoever arrives second finds the work done.

Three things drive it:

• Demand. A successor's open gate that finds a predecessor's epoch unsettled starts or joins a star-local settle runner and parks the open on its completion, bounded by the client's own context. A client cancel stops the waiting, never the walk.
• A graceful close. A closing star announces durable settle markers for its own unsettled chains (the crash insurance) and nudges each peer to settle them, so a rolling deploy starts settlement before the first blocked client arrives.
• The settle duty. The auditor's background janitor settles epochs nothing demands: a dead star's chains, and any epoch below its class's mint head, which is terminally halted whatever the registry says. A live star's head epoch is deferred, never displaced, with one exception: decommission settles a drained star's chains while it stands, because a star that is draining and leads nothing will never append again, and the terminate gate needs every epoch settled.

A live leader that hands one stream off runs the cheap sibling instead: the release flush commits refs over that stream's resident span from its own flight index before the lease releases. The epoch stays open, no fence is needed, and the successor opens against a covered frontier. See leadership for the handoff it belongs to and epochs for the fence.

Source documents

• docs/design/013-ref-spans.md
• docs/design/016-denorm-policy.md
• docs/design/017-settled-catalog-plan.md
• docs/design/018-settle-at-density.md
• docs/adr/0024-settle-superseded-epochs.md
• docs/adr/0026-demand-driven-settlement.md
• docs/adr/0027-settlement-runner.md
• docs/adr/0029-settled-catalog.md
• docs/adr/0031-one-flusher-owns-order.md