Architecture
Leadership
Who serves a stream and how that moves — leases, placement, handoff, rebalance, saturation, and the three drains
Every stream has exactly one sequencer at a time, and the star that holds it is the stream's leader. Leadership is not elected and not assigned by a controller: it is claimed lazily by the first producer to open the stream, moves only when the leader gives it up, and is never what keeps a stale writer out; the epoch guard does that. This page explains the three things that are easy to conflate, how a new leadership is placed, and every way one ends.
Leadership, lease, and epoch guard
A leadership is live, in-memory serving state on one star: the shared sequencer cursor, the live producer handles, the idle timer, and the lease it holds. It is born at the first Open, joined by later handles, and released when the last one closes, or abandoned on handoff, delete, or shutdown. One star holds many leaderships; a stream with none is vacant.
The lease is only the advisory coordination record the leadership holds: a session-bound row in the metastore's ephemeral namespace saying who holds the role now. It stops cooperating stars from fighting over a stream and lets the fleet route producers and hot reads to the holder. A star can briefly hold a lease with no leadership behind it (a departure still certifying coverage, or an acquisition whose response was lost), and the lease dies with the star's session.
Neither gates safety. The epoch guard is the arbiter: every leadership change bumps the stream's epoch by compare-and-set, and a deposed writer's chain is poisoned at the storage layer so its late slots can never reach the copy quorum. A zombie holder is fenced by the epoch, never by the lease. See epochs.
The fleet surface
Coordination lives in one ephemeral namespace, reached through one constructed fleet handle and consumed by components as narrow slices:
| Record | What it carries | Lifetime |
|---|---|---|
| Liveness record | identity, --location, advertised address, saturated and draining flags | the star's session |
| Node stats record | fixed-size load aggregates (leaders, offered append bytes, hot-read bytes, saturation, stranded, drained), never per-stream maps | the star's session; a sibling of the liveness record, so stats churn never rewrites routing |
| Writer lease | the holding star and when it acquired | the holder's session |
The session timeout (--oxia-session-timeout, default 15s) is the fleet's failure-detection floor: a dead star's leases and liveness survive this long, and failover waits it out. A decommissioned star is refused at registration forever.
Placing a new leadership
When a producer opens a vacant stream, the star it reached resolves the designated leader from one snapshot of the live registry. Saturated and draining stars are ineligible. Among the rest, the stream's two highest rendezvous-scored stars are the candidates, and the top one wins unless the runner-up carries significantly less leader-anchored load: a gap above 20% of the pair's mean and above 1 MiB/s. Inside that band the per-stream hash decides, so a burst of opens spreads across the fleet instead of herding onto whichever star last published a low number. A fleet with no eligible star refuses the open as at capacity: backpressure, not a queue.
If the designate is another star, the open is answered with NOT_OWNER naming it. The cluster client follows the redirect, remembers the holder for that stream, and forgets it on any other failure from it. Leadership stays where it landed: designation drift never moves a live leader.
Idle timeout and vacancy
On an Any stream a leadership outlives its last producer handle by the idle timeout (30 seconds): a handle that opens inside the window rejoins the live role with no lease change and no recovery, and only an expiry the window passes unclaimed releases the leadership, leaving the stream vacant. A Fenced stream has no idle timeout: its last close releases at once, because a handoff to the next fenced producer must not wait. The console labels a vacant stream "idle" for the operator; in these pages idle names the trigger, vacant the state.
Handoff moves metadata, never data
A handoff is one stream's leadership moving to another star: the leader abandons it, the fenced producer re-routes, the successor continues the sequence. No record moves: the data is in object storage and the metastore, and any star can serve it.
The departing leader first runs Finish: it waits for in-flight batches to resolve, commits refs over the stream's resident span from its own flight index (the release flush), and stamps the clean-release mark on the stream's last segment. Only then does the lease release. The successor's open gate passes on that mark (evidence of coverage, never inference from liveness), so the new sequencer starts above everything acknowledged. A Finish that cannot land parks the leadership in limbo, lease retained, and retries; opens bounce with a retryable refusal until it resolves. Coverage outranks availability. A producer racing the release parks briefly on the designate (up to a 2 s transfer grace, clipped to its own deadline) so the redirect lands after the lease has moved.
Rebalance: three triggers, one mechanism
A star's rebalance loop ticks every 5 seconds, measures its own load, and, only under pressure, hands off streams to cooler peers. A comfortable star never rebalances, however uneven the fleet.
- Saturation relief. A saturated star hands off one stream per tick, hottest first among its top 8 candidates, until relieved. A move must be a stable improvement: the target's load plus twice the stream's must stay below this star's, so the target does not hand it straight back.
- Expansion spread. A loaded-but-unsaturated star above a utilization floor of 0.5 gently re-homes streams whose re-resolution would land on an underused peer (one attempt per ~30 s), so new capacity never sits cold.
- Imbalance correction. A star more than 20% above the fleet mean on offered append bytes or led-stream count (floors: 1 MiB/s, 500 streams), sustained for three consecutive ticks, sheds half its excess toward stars below the mean, at most 200 moves per pass.
Saturation is absolute: a resource at its ceiling with queued work behind it. The gate judges the maximum across axes, smoothed, with hysteresis (saturated above 0.85, relieved below 0.70), and admits an axis only if a handoff relieves it:
| Axis | Meter | Denominator |
|---|---|---|
| CPU | process busy time | core count |
| Network | appends in, hot reads out, express fanout copies | --net-capacity (bytes/sec; zero turns the axis off) |
| Memory | retained bytes: admission budgets + tail window cache + sessions × stream window | --memory-ceiling (bytes; zero turns the axis off) |
| Backlog | age of the oldest un-absorbed flight, per class | 2 minutes |
A saturated star refuses new leadership and is skipped by placement. If a pass runs with the gate open and sheds nothing (no live peer could absorb even one of its streams), the star publishes stranded, cleared by the first successful handoff or by the gate closing. Stranded reads as "add capacity"; stranded that persists after new capacity registers convicts an indivisible hot stream, which only partitioning above nova can fix.
Shed is a different word. It is the backpressure refusal of new work at an explicit bound: a producer session past --max-producer-sessions (default 4096), an open on a fleet at capacity. Moving existing leadership is always a handoff. See backpressure.
Three drains
A drain empties a star of leadership through the same handoff machinery, never into a fleet that cannot re-home the streams.
| Drain | Trigger | Grace | Ends when |
|---|---|---|---|
| Serving drain | operator order | --drain-session-grace (30s) for in-flight sessions | the star leads nothing; reversible until then |
| Decommission drain | the same order, kept until the terminate gate | same | sticky: retries until leading nothing; decommission then settles the drained star's chains and declares it gone for good |
| Exit drain | a stop signal | 5 s constant, then --exit-drain-budget (30s) overall | budget spent, or a pass that sheds nothing; the remainder falls to close's certify path |
A draining star flips its readiness, refuses new leadership, and placement skips it: a drain is a routing fact. A stop signal adopts an in-flight serving drain rather than restarting it, and once the exit drain has begun no drain can be reversed. Nothing about a drain persists across restart: a star holds no data, so a drained star is simply one leading nothing. See node lifecycle.
Departures run once; the gate heals leaks
Every departure (idle expiry, handoff, drain, fenced supersede, delete, shutdown) claims the leadership once under one lock and exits through the same clean-release door; a second initiator that finds the claim taken walks away. Lease releases run on their own clock, detached from the caller's context, so a cancelled open cannot leave its lease standing.
One leak survives: an acquisition whose response was lost, leaving a lease with no leadership behind it on a healthy star. The open gate tells it apart from a departure. While the holder is departing (limbo, or a drain), an open at the holder redirects to the designate; otherwise a held-but-not-led lease is an orphaned lease, and the open serves: the same-owner acquire re-attaches the row and leadership re-establishes over it. Every open reaches the holder, so any orphan heals on first touch; each heal is counted and logged, dated by the lease's acquisition stamp.
Source documents
docs/adr/0023-coord-fleet.mddocs/adr/0030-alive-defers-and-the-last-writer.mddocs/adr/0032-same-star-departures-pay-coverage.mddocs/design/000-decisions.mdinternal/coordpackage documentation