Write path

A batch is acknowledged only once it is durable in object storage, and the metastore is never touched on the way. This page follows a batch from the producer session to that acknowledgement.

From append to ack

1. A producer session (the AppendSession stream, or a one-shot Append) hands a batch to the stream's leader. The facade's sequencer assigns each record the next contiguous sequence number; a match_seq_num miss rejects only that batch and reports the tail.
2. The pool is star-wide: every admitted, uncut batch sits in one FIFO whose oldest byte anchors the flush deadline. Its budget is --s3-memory-buffer-mib (default 256; Express fanout buffers separately under --s3-express-memory-buffer-mib, default 32). At the budget, admission blocks.
3. The flusher, one per star, cuts the pool into a flight and claims the flight's chain slot inline. Because one loop both cuts and claims, cut order is slot order by construction, and per-stream record order across flights follows from arrival order in the pool. Uploads parallelize below the claim; ordering never depends on them.
4. The commit is a conditional PUT. The star renders the flight once as a transit bundle and writes it with PutIfAbsent to the same key in every bucket of the set. The slot is durable when W buckets confirm.
5. Acknowledgements release in slot order. A flight acks only when its slot is durable at W and every earlier slot has acked. An ack, once released, is never disowned.
6. Nothing touched the metastore. The absorb, running in the leader's own process and woken by each release, later commits one span ref per stream per batch and advances the chain's checkpoint by CAS; see settlement. Until then the leader's flight index serves the tail window to reads.

Both classes, one protocol

Both are the same write core over one loglet. At K=3/W=2 an Express ack costs the second-fastest confirm, so one slow or down zone is invisible to producers; a straggling copy completes in the background or is tolerated absent (reads fall through the set, deletes fan to all of it).

One shared chain, two forms

The chain is one epoch's flights as a contiguous sequence of chain slots at derivable keys, chains/<star>/<epoch>/<seq>, and it is shared: a star runs one loglet per class (the Standard chain, the Express fanout), and every stream the star leads rides that loglet together. Stream data then takes two forms:

• In transit: the write form. Each slot holds a transit bundle: one flight's records across all of the star's streams, grouped by when they arrived, durable at the copy quorum. It is interleaved, cheap to write, and short-lived.
• Settled: the read form. The auditor denormalizes bundles into per-stream settled objects: contiguous, read-optimized, standard-priced, the shape history retires into and long-term retention lives in.

Behind the chain sits one contract every backend implements. The incumbent's sequencer assigns contiguous slot sequences, fans each flight to every bucket, releases acknowledgements in order at the copy quorum, and halts permanently the first time it observes a foreign occupant in a slot: epochs never resume. Any successor runs fence (returning the cut), walk (every committed slot by point lookups, never a listing), trim, and the garbage probe against the durable artifact. See epochs.

Chain slot lifetime

A slot is created at most once per bucket (the PUT is conditional on absence), and the key is identical across the set. A slot is committed when flight copies hold the copy quorum, and dead when poison written by a fencer occupies the blocking set of K−W+1 buckets. The two states are mutually exclusive and permanent: W flight copies and K−W+1 poisons cannot both fit in K buckets, and neither a flight copy nor a poison can ever be revoked. There is no torn state. Committed slots live until the absorb has taken them and the settle has moved their bytes; trim then deletes them from every bucket behind the trimmed cursor. Poison stays until the star is decommissioned.

The slot envelope

Every slot begins with a fixed 60-byte envelope: magic, version, kind (flight or poison), epoch, slot sequence, the writer's nonce, the object size, the footer's offset and length, and a CRC-32C. One bounded GET at offset zero classifies a slot and locates its footer with nothing but the key. The bundle behind it is laid out at base offset 60, so every coordinate a ref stores is object-absolute. The nonce settles ghost PUTs: a writer whose PUT failed ambiguously probes the envelope, and its own nonce means the flight committed.

Pipelining and ordering

The sequencer keeps up to a pipeline window of slots assigned but not yet released (--s3-parallel-uploads on Standard). When the window is full, claiming a slot blocks, which backpressures the flusher, the pool, and finally producer admission. Acknowledgements release strictly in slot order whichever PUT finishes first, so one slow slot head-of-line blocks every later slot on that star for its duration: the price of an ordering that needs no coordination. The window is minted onto the epoch record: it bounds how far assignment can run ahead of release, and so how many slots a fence must arbitrate and a garbage probe must cover.

The backlog budget

The backlog is the chain's released-but-unabsorbed span: flights acked but not yet taken into refs. --s3-chain-backlog-flights (default 1024) bounds it in flights and bytes, so one number bounds the leader's flight index, a successor's settlement time (and so takeover latency), the leader-only read window, and chain storage. Tokens free as the absorb's checkpoint advances.

At the bound, publishes block and producers block behind them. This is deliberate and loud: a budget stall is never normal, so the stall counter carries a diagnosed cause (absorb lagging, metastore slow, or epoch settling) and the star logs it. Alert on backlog age, never rate.

When the chain halts

A slot PUT refused with a foreign envelope, or a flight that cannot reach W within its deadline, halts the epoch: no acknowledgement at or past that slot will ever release. The star does not stay down. It fences its own epoch, mints the next one (the mint CAS doubles as the deposed-check: losing it means a successor exists, and the star sheds the leadership instead), acknowledges late the flights the fence found committed below the cut, and republishes the rest into the new epoch's first slots from the still-pooled payloads. Producers see one latency blip, not an error, and no sequence number is committed twice: the cut, not the ack ledger, decides which epoch covers each flight. See epochs.

What the ack costs

The trade between the classes is simple: Standard acks slower and costs less; Express acks faster and costs more. Standard batches toward larger, request-priced writes: latency spent for cost. Express pays for its speed twice, in byte-priced express storage in every zone of the set and in cross-zone transfer for the remote copies, but only over the transit window; once settled, history is standard-priced for both classes, so the premium covers the ack, never the retention. Either way the metastore's bill does not grow with ingest: tens of operations per second per star.

Source documents

• docs/design/011-virtual-log.md
• docs/design/009-chain-commit.md
• docs/design/010-write-path-economics.md
• docs/adr/0031-one-flusher-owns-order.md