Scaling and retiring stars

A star holds no data, so every capacity change is a leadership question, never a data-movement one. This page covers what happens when a star joins, how you empty one, what makes its retirement durable, what a stop signal does, and how the Kubernetes controller walks a star out on your behalf.

Adding capacity is registration

A new star registers in the fleet and appears in the live set within one stats tick (5 seconds). Being the least-loaded star, it wins new placements at once. Existing leadership moves to it only under pressure: a saturated star hands off led streams one per tick until relieved, and a loaded-but-unsaturated star above a modest floor gently re-homes streams whose re-resolution lands on a severely underused peer: the expansion spread, paced far slower, so new capacity never sits cold but a comfortable, evenly-loaded fleet never rebalances. Every move is a handoff: the leader abandons the stream, the fenced producer re-routes to the new leader, metadata only. See leadership.

Placement needs somewhere to go. Keep at least three stars so a drain always has a target, and keep more stars than an Express stream's bucket set needs.

The serving drain

nova node drain empties a star of leadership in two phases:

1. Refuse new leadership. The star flips draining: /readyz answers 503, placement stops designating it within one tick, and an open that lands on it anyway is redirected to re-resolve. Producers already attached keep running.
2. After the grace, hand off. Once --drain-session-grace (default 30s) passes, the star hands off every led stream through the same handoff rebalancing drives; producers re-home in one hop. Sessions shorter than the grace finish untouched; longer ones get one clean handoff regardless, so a longer grace buys nothing but a longer wait. Size it to your typical short-lived session.

The drain never hands off into a fleet that cannot re-home: a stream whose bucket set exceeds the surviving fleet stays led, loudly, and the blocked exit shows at the terminate gate, never as client errors. The embedded auditor quiesces at drain start: no new duty claims, the in-flight job gets the same grace, and the auditor leaves the ring so its seat is dealt to survivors.

nova --addr star-3:7780 node drain
nova --addr star-3:7780 health       # "drained: safe to decommission" once leading=0
nova --addr star-3:7780 node drain --stop

The drain is reversible until terminate and is not persisted: a restarted star comes back serving, and a controller re-issues the verb. health prints <status> node=<id> location=<loc> version=<v> leading=<n> and, when the gate is open, drained: safe to decommission. nova fleet lists every live star with its location and address.

Decommission

nova --addr <any-star>:7780 node decommission --star <star-id>

Decommission writes a durable tombstone for the StarID: the registry refuses that star from now on, and every bookkeeping consumer settles against its permanent absence. The verb is gated on the terminate gate (drained ∧ leading == 0, read from the target's own health) and refuses a registered star that is not fully drained. Before judging the gate it settles the drained star's chains: a star that is draining and leads nothing will never append again, so its open chain epochs are fenced to their cut and every acked byte gets a ref, under a 30-second budget; a walk that outruns the budget keeps running and the next ask rejoins it. Any star can retire any drained star: the verb goes through a peer.

An absent star (the unplanned-loss corpse) always passes the gate; its chains settle through the settle duty's dead-star judgment. Because settling a chain is terminal for that process, a drain reversed after its decommission was attempted sheds the next append until the star restarts.

Identity is bound to the data directory: a wiped disk returns as a fresh star, and replacement hardware always mints a new StarID. There is no un-decommission. Nova never infers permanence from absence (a partitioned star that outlived any timeout would resurrect bookkeeping the fleet already settled), so the verb is always operator-issued, by a person or the controller.

The exit drain

SIGTERM, SIGINT, or stdin EOF (with --shutdown-on-stdin-close) runs the exit drain: the same handoff pass, bounded and irreversible. Draining flips on, the auditor stops claiming, a 5-second constant grace passes, then led streams are handed off under --exit-drain-budget (default 30s). Whatever remains falls to close's certify path; close always follows, budget spent or not. A pass that hands off nothing and holds nothing in flight ends the drain immediately: the fleet cannot absorb this star now, and retrying until it can is decommission's job. A stop signal adopts an in-flight operator drain (no second grace), and once shutdown begins drain --stop is refused.

Signals escalate: a first signal drains then closes; a second cancels the drain and goes straight to close (still certified); a third is the OS default. Automation never second-signals.

Size the supervisor's kill deadline above the exit budget plus the close guard plus margin: 75 seconds in the compose stack, the Fly kill_timeout, and the controller's terminationGracePeriodSeconds. A shorter deadline SIGKILLs a star mid-close and turns a clean roll into the crash path.

Health and readiness

Both endpoints ride the metrics listener (--metrics-listen), which comes up last on start and goes down first on close, so its reachability brackets the serving window.

An audit-only process has no registration; it is ready when its ring announcement is live, and reads not-ready again once it withdraws on exit.

The controller's ladder

On Kubernetes the controller walks a star out through the ladder: one star per pass, the last ledger entry, one rung per pass. Scale-down and an upgrade's blue both leave by it.

• Drain is self-targeted: the verb has no target field and must reach the victim, and it is re-issued every pass because it does not survive a restart.
• The terminate gate and the decommission go through a witness (a ready star that is not the subject, the newest ready green first), so an old-image victim is retired by fixed code and no roll depends on the victim's own handler.
• A rung that cannot advance is held, never failed: the condition DepartureHeld=True names the rung (GateOpen, DecommissionRefused, DrainUnreachable, NoWitness) and carries the verb's own message; the pass persists the ledger and re-asks on the ladder's 5-second beat. Every ladder call carries a 10-second deadline; expiry is held, and a longer settle walk keeps running server-side.
• A victim whose pod vanishes is a corpse: absence is judged through the witness's overview, the decommission is issued with drained=false so its chains settle under the duty's dead-star judgment, and the entry retires without a drain. An unplanned loss outside the ladder is reaped only after serving.reapAfter (default 10 minutes), judged through a live witness: a dark fleet is a visibility problem, never a decommission spree.

The ledger persists on every exit path and adopts standing pods it does not name, so a controller restart resumes mid-walk from status, never from memory.

Upgrades

An upgrade is surge-then-drain: one green star in, one blue star walked out through the ladder above, capacity never dipping below N. Because a handoff moves metadata only, the newcomer is warm the moment it takes its first leadership. Outside Kubernetes the same sequence by hand is: start the new star, drain the old one, wait for the gate, decommission, stop the process. A pre-drained star closes immediately on SIGTERM; pre-draining before deletion is deliberately the fastest clean roll.

Source documents

• docs/design/000-decisions.md (drain, decommission, exit drain, ladder entries)
• docs/adr/0011-node-scale-down.md, docs/adr/0018-retirement-at-release.md
• operator/README.md
• internal/daemon/drain.go, internal/daemon/health.go