Operations
Failure modes
What breaks, what Nova does about it, what you will see, and the procedures that put it right
Nova's failure posture follows one law: availability may pay, acked data never does. This page walks each dependency loss and each kind of star failure, says what stalls and what keeps working, names the signal you will see, and gives the recovery procedures that are yours to run.
Object storage outage
Object storage is the durability tier, so with it gone nothing new can become durable. What matters is how the fleet waits.
Appends block; they never OOM. A Standard append is admitted into the pool against --s3-memory-buffer-mib (default 256 MiB per star); once that budget is held, admission blocks producers. Behind it the chain's backlog budget (--s3-chain-backlog-flights, default 1024) bounds how many published-but-unabsorbed flights the leader retains. Every accumulation that tracks the outage is capped, so a star rides out a long outage at a fixed memory footprint, and producers see their calls wait until their own deadline. --s3-attempt-timeout (default 0s, unbounded) caps one S3 attempt so a straggler is cut and retried rather than held; size it well above your largest routine PUT's p50.
Express fails fast. An Express flight must reach its copy quorum within --s3-express-publish-deadline (default 2s); past it the append fails and the producer retries. The quorum (--s3-express-ack-quorum, default 2) is a hard floor, never degraded, so one express bucket down is invisible and two make the class unavailable until one returns. Express admission is bounded separately (--s3-express-memory-buffer-mib, default 32).
Reads split by tier. Hot reads inside a leader's tail window keep serving from memory. Reads of settled history need the object: the object cache falls back to the direct path, and with that gone too the read fails through with a storage error. Subscriptions parked on a leader's wake stay parked; nothing is lost.
Signal: nova_objectstore_attempt_errors_total and nova_objectstore_duration_seconds rise first; nova_direct_pending_bytes climbs to its limit; nova_write_latency_seconds stretches to the producers' deadlines. Nothing needs restarting. When storage returns, admission drains in order and acks resume.
A star crash
A star holds no data, so a crash loses nothing acked; it loses time.
- The dead star's leases and liveness record survive until its metastore session expires:
--oxia-session-timeout, default15s, the fleet's failure-detection floor. - The first open on a led stream after that acquires the lease and runs settlement: it fences the dead epoch to its cut (poison slots in the bucket set's fixed order, so a stale writer can never reach the copy quorum again), walks the backlog's footers, and commits refs pointing into the transit bundles (metadata only, no data movement), then probes one pipeline window above the cut for garbage. Only then does the successor serve.
- The SDK follows the redirect and resends unacked batches under epoch and sequence dedup; no sequence number is burned or reassigned.
Typical failover is around 20 seconds (the session timeout plus a healthy residue's walk), inside the SDK's 30-second retry budget. A recovery that cannot fit is an incident, and applications see it. Keep the session timeout above your worst-case process stall (GC pauses, a frozen VM): a star that misses its session renewal while alive is fenced exactly like a dead one, and its producers re-home. Vacant streams (no lease held) need no failover at all: reads serve committed state without waking anything.
Metastore unavailability
No acknowledgement depends on the metastore. Both classes commit by conditional PUTs into chain slots and ack at the copy quorum; the chain of loglets in the metastore is touched at reconfiguration and absorb cadence only. So a producer with an established leadership keeps getting acks through a metastore outage, for a while. What stalls:
| Needs the metastore | What you see |
|---|---|
| Acquiring a leadership (a new stream, a failover, a handoff) | opens block or fail; the SDK retries |
| Placement and rebalance | the fleet freezes where it is |
| The absorb (refs, segments, the checkpoint) | the backlog cannot drain; once the budget fills, producers block with stall cause metastore_slow or absorb_lag |
| Settlement, epoch mint, and rolls | failovers wait; a halted epoch's re-mint waits |
| The auditor's every duty | reclamation pauses; nova_auditor_due_age_seconds climbs |
| Registration renewals | when the outage outlives the session timeout every lease lapses at once; the whole fleet fails over when the metastore returns |
The last row is why the session timeout is a tuning decision: too short and a metastore blip becomes a fleet-wide re-election; too long and a dead star's streams are unavailable for that long. --oxia-write-window (default 4) and --oxia-max-batch-kib (default 128) bound the client's pipelining per shard.
Losing the metastore's data is a different matter: the durable namespace is forward-only and never snapshot-restored; recovery is reconstruct-from-S3, and a tooled procedure is open work. Run oxia with replication factor 3 on its own pool.
The object cache dies
Harmless. The object cache is never authoritative: a read it cannot serve falls back to a direct ranged GET, a sick instance is charged and then ejected (nova_objectcache_ejections_total, nova_objectcache_ring_size), and writes never touch it. The cost is read amplification against object storage until the tier returns.
A saturated or stranded star
A star at a resource ceiling (nova_node_saturated, nova_node_utilization above 0.85 on some axis) refuses new leadership and hands off led streams one per tick until relieved. If a full rebalance pass finds no live peer that fits even one of its streams, it publishes stranded (nova_node_stranded, the star summary's stranded=true). Stranded reads as add capacity; the Kubernetes controller does so at once. Stranded persisting after new capacity registers convicts an indivisible hot stream: one stream carries more than half the star, and no peer can take it. Partitioning is the layer above's lever; Nova cannot split a stream.
Reading a budget stall
Appends that wait with zero errors are backpressure, and a backlog budget stall is a fault signature, never normal. A healthy backlog at any fill turns over within a pass. The counter carries its diagnosis:
nova_chain_backlog_stall_total{cause=…} | Meaning |
|---|---|
absorb_lag | the absorb cannot keep up with ingest: metastore latency, or too many streams per batch |
metastore_slow | the metastore itself is the bottleneck |
settling | a prior epoch's settlement is holding the chain |
Read it beside nova_chain_backlog_age_seconds (the age, not the fill), nova_direct_backlog_wait_seconds, and nova_kv_duration_seconds. The goroutine dump at /debug/pprof/goroutine?debug=1 names the blocked stage.
Local disk
A serving star keeps no data on local disk. --dir holds the node-id file that binds the StarID to the volume: restart the same volume and the same star returns; wipe it and a fresh star registers. There is no disk-full mode on the serving path any more, because there is no serving-path disk.
The one disk that matters is the auditor's fetch cache scratch (--auditor-fetch-cache-dir), the whole-object read-through cache that dedups the settle's transit-bundle reads. Startup wipes its contents: point it at dedicated scratch, never at data. It is budget-bounded (--auditor-fetch-cache-mib, default 40960), a read the budget cannot admit falls back to a ranged GET, and its loss at any instant is harmless. On Kubernetes the budget derives at 90% of the declared scratch so admission backpressure, not kubelet eviction, is the valve.
The orphaned lease heals itself
A lease acquire whose response is lost leaves a row stamped with an owner that holds no leadership: a ghost. The open gate distinguishes a genuine departure (a draining writer, or one parked waiting to certify coverage) from an orphan: an open landing on a held-but-not-led lease with no departure behind it serves, re-attaching the row through the ordinary same-owner acquire. Every open reaches the holder, so any orphan heals on first touch. nova_dataplane_orphaned_lease_total counts each heal (flat zero is the only healthy reading), and the star warns once per stream per window, dated by the lease's acquisition time.
Wipe-first restore
While Nova is pre-release, crossing a stored-format boundary means wiping and restarting fresh. Never wipe one plane alone: metadata describing absent objects misparses, and objects without metadata are garbage nothing reclaims, because leaks are found by intent markers, not sweeps.
- Compose:
docker compose -f deploy/docker/docker-compose.yml down -vdrops oxia and MinIO together. - Kubernetes: with the pools scaled to zero, delete the oxia Lease, the
oxia-statusConfigMap, and both PVC sets (wal-…anddata-oxia-…), in that order, or the instance-id livelock keeps pods from converging; then empty the bucket (the bucket and control plane stay), then restore the pool counts.
Both planes, always
A kept bucket over a wiped metastore is a bill for orphans with no watcher; a kept metastore over an emptied bucket is a catalog of nothing. Wipe both, or neither.
Recovery procedure
For a lost star, do nothing: leases lapse at the session timeout and the next open settles the chain. Retire the identity once you know it is gone for good: nova node decommission --star <id> through any live star. An absent star passes the terminate gate; its chains settle through the settle duty's dead-star judgment, and the tombstone lets every consumer settle its bookkeeping. Never decommission on a timeout: a partitioned star that comes back would resurrect what the fleet already settled.
For a wedged star (appends stall, health fine, no stall cause moving), drain it and let handoffs re-home its streams before restarting it; a restart alone runs the crash path. For a metastore outage, wait: restarting stars fixes nothing, and every lease re-election you force is one more the fleet must absorb when the metastore returns.
What the chaos suite proves
Every CI run replays seeded fault schedules against real components with object storage as the injection surface: a dropped PUT is a dropped ack, a dropped GET a failed cold read, and a fault decision is a pure function of the seed and the call, so a failing seed replays byte-for-byte. Over a served star the suite checks linearizability of concurrent producers with a model checker (no acknowledged append lost, every read consistent with a single total order, sequence numbers contiguous) under bounces, failovers, cold-read brownouts, and combined nemeses. The guarantees in guarantees are enforced, not asserted.
Source documents
docs/design/007-production-readiness.md,docs/design/006-chaos-matrix.mddocs/design/000-decisions.md(single-stall, virtual log, clean-release, orphaned-lease entries)deploy/docker/README.md,deploy/pulumi/README.mdnovatest/chaos/chaos.go