Epochs

Every acknowledged record belongs to exactly one epoch of one star's chain, and every leadership change ends the old epoch at a definite position, the cut, before a successor writes. This page explains how the cut is computed, why it is safe, and how departures certify coverage so a successor never reassigns an acknowledged sequence number.

Epochs are per-star mint counters

A star's chain is a sequence of epochs. Each epoch is minted by a compare-and-set increment of the star's counter in the durable metastore (chains/<star>), followed by the creation of the epoch record (chains/<star>/<epoch>) carrying its mint-time constants (storage class, bucket zones, copy quorum, pipeline window) and its cursors (checkpoint, trimmed, rewritten). The counter records a head per storage class, so a Standard roll and an Express roll on the same star cannot depose each other. Only a returned mint can write a slot; a crash between the increment and the record burns a number and leaks nothing.

A star's writer starting, an in-process turnover after a halt, and a planned configuration change each mint an epoch. Nothing resumes one. Epochs compare only within one star, which is why safety never rests on comparing them in metadata: it rests on the chain itself. The writer lease names the current holder; it is advisory, and the epoch is never derived from it.

Fencing an epoch

An epoch ends when a successor fences it. Usually the successor is another star that takes over the stream after its leader died or was drained (a takeover). It can also be the same star starting a fresh epoch after a halt or a configuration change (a roll), or a later leader that opens the stream and finds an old epoch still unsettled.

Object storage cannot reject a write because the writer is stale, so the fence cannot lock the old writer out. Instead it takes the space the old writer would need: it writes poison into the slots the incumbent has not filled yet, in the same buckets the incumbent writes to. Once poison holds enough of a slot, that slot can never be acknowledged, and the first such slot is the cut.

The fence then walks the chain slot by slot. It first finds roughly where the chain ends by probing whether slots exist: it doubles the step until a probe finds nothing, then binary searches (each probe is one small GET per bucket). Starting a little below that estimate, it looks at every slot in all K buckets and classifies it:

1. Committed: flight copies visible in at least W buckets. Include it, advance.
2. Repair before poison: a flight visible in at least one bucket but fewer than W. Copy the identical bytes into every empty bucket, then re-observe. Repair is safe because one writer per epoch means a slot has exactly one possible content; the copy can only complete what the writer started.
3. Poison: no flight visible anywhere reachable. Write poison into one empty bucket at a time, in the bucket set's fixed order, re-observing after each write. If a flight appears meanwhile, the incumbent raced in; go back to repair. Completing a slot always wins over killing it.
4. Dead, the cut: poison occupies at least K−W+1 buckets (the blocking set), so the slot can never reach W and nothing at or past it will ever acknowledge. The walk stops here.

When unreachable buckets hold the deciding vote (a frontier slot showing one flight and one poison with the third zone down), the fence blocks and retries rather than guess, surfacing an undecided error on context expiry. Alert on its age.

Why it is safe

With B = K−W+1 as the blocking set:

• No acknowledged flight is ever cut. An acked slot holds W flight copies, all create-only and immutable, so poison can occupy at most K−W < B buckets. The walk can only classify it committed.
• Nothing at or above the cut is ever acknowledged, by anyone. Poison holds B buckets permanently, so the incumbent can never assemble W there; by the in-order ack rule it cannot release anything past that slot either, even if later PUTs landed, even if it never observes the poison.
• The cut is stable. Both terminal states are monotone, so a crashed or racing fencer re-derives the same value. Any successor can finish a dead fencer's walk.
• Contiguity holds below the cut. The walk advances only past committed slots, so [0, cut) has no holes.

At K=1/W=1 the protocol degenerates to a single poison slot: whoever wins the next free slot ends the epoch, and a writer that loses once halts.

Publishing the cut

The fence returns the cut; the caller publishes it on the epoch record by CAS, moving the record to the fenced state. Racing fencers computed the same value, so the CAS is a publication, not an arbitration; a disagreeing cut is a programming error, never retried. Once recorded, a later fence is a read.

Slots at or above the cut may still exist (pipelined or straggling PUTs, even complete at W). Nothing maps them and nothing acks them. Because assignment never runs more than the pipeline window past release, that garbage is confined to (cut, cut + window]; settlement probes exactly that span and deletes what it finds, leaving poison standing until the star is decommissioned: a fully partitioned zombie's PUT can arrive arbitrarily late, and only standing poison keeps it harmless.

The unratified tail

Flights that landed at or above the cut were never acknowledged, so their records were never promised. In a takeover they are garbage: the successor reassigns their sequence numbers, and the producer's retry re-sends the batch. In an in-process roll the same flights are republished from the still-pooled payloads into the new epoch's first slots, so producers see a delayed ack rather than an error. Either way no sequence number is burned and no record is covered twice: the cut partitions every flight into exactly one of absorbed from the old epoch or republished in the new.

Takeover, end to end

Settlement is metadata only: no bytes move. It restores the coverage invariant, refs cover every acknowledged byte, before the successor serves or appends. Typical failover is the metastore session timeout (--oxia-session-timeout, default 15s, the fleet's failure-detection floor) plus the walk, inside the SDK's 30s retry budget.

Clean release versus abandonment

Leadership is never given up without either coverage or death. Every departure (a graceful close, a cross-star handoff, a drain, a same-star supersede by a newer fencing token, a re-establish after a poisoned open, an eviction on session loss) exits through one door: Finish. Finish waits for the stream's in-flight batches to resolve, runs the release flush (span refs into the transit bundles, straight from the leader's flight index, so the epoch stays alive for every other stream), and then stamps the clean-release mark on the row covering the stream's tail: its final write, landed only after the coverage it certifies is durable.

The cross-star open gate passes on that evidence. The mark is single-use: the next open consumes it before admitting its first append, so it can only certify the tenure that stamped it. A departure without a mark is an abandonment, and abandonments resolve through settlement, never through trust:

• a predecessor epoch below its class's head is superseded, so it settles inline on the held open, whatever the registry says about the star;
• an alive predecessor's unmarked head defers the open with a retryable refusal until the mark lands, the star restarts (superseding the head), or the registry converges on its death (the registry's word is only ever a reason to wait);
• a Finish that cannot land parks the leadership in limbo (lease retained, retried every 5s, opens bouncing retryable), because coverage outranks availability.

The last-writer record

A crash that lands between an acknowledgement and the first absorb leaves acked records in the dead epoch's slots and no ref row naming the star. So every open durably records stream → star (streams/<id>/writer) before its first acknowledgement can exist. The gate judges that record with precedence over the newest ref row's builder, which lags a writer change by one absorb pass; a takeover therefore always finds a predecessor to settle instead of reading the stream as fresh.

The fencing token is policy; the epoch is safety

The Fence verb installs a producer's fencing token, and exact equality judges it at open and one-shot append: an application's single-writer rule, cooperative on Any streams and mandatory on Fenced streams. It never decides correctness. A stale token-holder is superseded at the serving layer; a stale writer is locked out at the storage layer, because its slots can no longer reach the copy quorum. The epoch guard is the correctness fence; the lease and the token are coordination above it.

Source documents

• docs/design/011-virtual-log.md
• docs/design/009-chain-commit.md
• docs/adr/0025-clean-release-gate.md
• docs/adr/0030-alive-defers-and-the-last-writer.md
• docs/adr/0032-same-star-departures-pay-coverage.md