Architecture
Backpressure
Memory is bounded everywhere — every budget, what it bounds, and what a client sees when it is reached
Nova is built never to run out of memory, including through an object-storage outage. The rule is uniform: every queue, buffer, cache, in-flight set, and goroutine population has an explicit limit, and reaching it applies backpressure (the caller blocks, or the work is shed with an error), never unbounded growth. A shed is always logged; silent shedding reads as "everything is fine" when it is not. This page lists the budgets that make the rule true and what happens at each one.
Block or shed, never grow
Two behaviors exist at a bound, and the choice is deliberate per bound:
- Block. Work that is already admitted and must eventually complete waits for room. Appends block on the admission buffer and the backlog budget; the Gateway's vended fetches wait for the fetch pool; the auditor's duties wait for their rate budgets. Blocking reaches the client through gRPC flow control or a slower response, and it is measured as an age or a wait, never hidden.
- Shed. Nova refuses new work that has not been admitted, with
ResourceExhausted(or the fleet's at-capacity refusal for a new leadership) and a log line naming the cap. Producer sessions and subscriptions past their caps shed; existing sessions are never cut to make room.
Moving existing leadership off a star is a handoff, not a shed; see leadership.
The budgets
| Budget | Flag | Default | Bounds what | At the limit |
|---|---|---|---|---|
| Admission buffer (Standard) | --s3-memory-buffer-mib | 256 | bytes of appends retained from decode to resolution: pooled batches and flights in flight | appends block; the transport stops reading frames, so backpressure reaches producers through gRPC flow control |
| Express buffer | --s3-express-memory-buffer-mib | 32 | bundle bytes buffered for the express fanout | admission blocks |
| Backlog budget | --s3-chain-backlog-flights | 1024 | flights the Standard chain may hold acknowledged but un-absorbed: leader tail memory, takeover settlement time, and chain storage together | appends block until the absorb's checkpoint catches up; every stall is counted with a diagnosed cause |
| Tail window cache | --tail-window-cache-mib | 128 | acknowledged payloads the leader keeps in memory for hot reads, shared across classes | oldest-admitted evicted; an over-budget flight is skipped and logged; a miss falls back to ranged GETs |
| Producer sessions | --max-producer-sessions | 4096 | concurrent producer sessions per star | new opens shed with ResourceExhausted |
| Subscriptions | — | 16384 | concurrent subscriptions per star | new subscriptions shed with ResourceExhausted |
| gRPC stream window | --grpc-stream-window-kib | 64 (the protocol floor) | inbound bytes buffered ahead of the handlers, per stream; sessions × window is the transport's memory plane | the peer stops sending on that stream |
| Batch and record caps | --max-batch-bytes, --max-batch-records, --max-record-bytes | 100MiB, 65536, 1MiB | one append batch and one record | the append is rejected |
| Read response caps | --max-read-bytes, --max-read-records | 8MiB, 8192 | one read response | the page returns short; one record always serves |
| Auditor request budget | --auditor-s3-budget | 50 | object-store requests per second the auditor issues | work waits; the backlog surfaces as due-queue age |
| Auditor rewrite budget | --auditor-rewrite-mib | 64 | MiB per second of bytes assembled through the star by rewrites (server-side copies are free of it) | rewrites wait |
| Auditor workers | --auditor-workers | 2 | concurrent background jobs | jobs queue |
| Fetch cache | --auditor-fetch-cache-mib | 40960 | scratch bytes the settle's whole-object cache keeps resident | a read the budget cannot admit falls back to a ranged GET |
| Gateway fetch pool | --fetch-inflight | 64MiB | vended-fetch bytes in flight across every response | responses wait; the wait is a metric |
| Gateway footer cache | --footer-cache-bytes | 32MiB | decoded footers | eviction |
| Object cache load | --object-cache-load-factor | 1.25 | in-flight claims per cache instance, relative to the mean | a hot key spills to the next instance; a dead instance's share goes direct |
The auditor's budgets are the same rule in a different place: a breach defers work rather than growing a queue, and deferred work surfaces on the one axis nova alerts on, backlog age, never rate. See auditor tuning.
What the admission buffer charges
A record's retained life begins before the write path sees it: the transport decodes a batch and may hold a window of decoded batches per session awaiting dispatch. The admission buffer therefore charges a batch the moment its decoded form exists (content bytes plus a measured per-record and per-batch overhead, so small records are not undercounted) and releases it exactly once, at resolution, where the tail window cache's own budget takes ownership of the same slices. The two budgets tile a body's whole life with no gap and no double count. A batch heavier than the whole budget is clamped so it can still be admitted; it never deadlocks against a cap it cannot fit under.
That accounting is also the memory saturation axis: --memory-ceiling is the denominator, and the numerator is exactly admission budgets + tail window cache + sessions × stream window. A star near its ceiling reads saturated, refuses new leadership, and starts handing streams off. The oldest admitted-but-unresolved age is a first-class gauge, and the shipped alerts fire on that age: a wedged drain must read as an incident, never as frozen counters.
What a client sees
| Situation | Client experience |
|---|---|
| Admission buffer full | Append blocks; the session's stream stops being read; latency rises, no error |
| Backlog budget full | Append blocks until the absorb retires flights; the star logs the stall with its cause |
| Express quorum unreachable | appends fail fast after --s3-express-publish-deadline (default 2s) rather than pile up; the producer retries |
| Session cap reached | the open fails with ResourceExhausted; the SDK's retry budget decides |
| Subscription cap reached | Subscribe fails with ResourceExhausted |
| Fleet at capacity (every star saturated or draining) | the open is refused; the SDK re-resolves and retries |
| Star saturated | new leadership is placed elsewhere; existing streams keep running and may be handed off |
A backlog stall is a fault signature, never normal. The star diagnoses the cause at the stall (the absorb lagging, the metastore slow, or a predecessor chain still settling), and each is a different operator action. See failure modes.
Why an outage cannot OOM a star
Walk the write path during an outage. Flights cannot land in chain slots, so nothing is acknowledged and nothing enters the tail window cache; the pool fills toward the admission buffer, and at 256 MiB admission blocks. Producers' frames are now buffered by the transport, but only up to each session's pinned stream window (64 KiB by default), so the transport plane is bounded by sessions × window, and the session count is itself capped. Every retained byte on the star is therefore under one of three bounded budgets, and their sum plus fixed overhead is the star's closed memory formula. Nothing scales with the outage's duration; the only thing that grows is the age of the oldest admitted batch, which is exactly the signal that alerts.
The same shape holds for the metastore. If oxia slows, the absorb cannot advance the checkpoint, the backlog budget fills, and appends block at the token bound: leader memory is capped at the backlog's flights, and the star logs the stall with absorb_lag or metastore_slow as its cause. The Express class adds a fail-fast rather than a deeper queue: a flight that cannot confirm its copy quorum inside the publish deadline fails, and the producer retries, so express memory never waits on a dead zone.
Reads are bounded by construction: a page is capped in records and bytes, the cold reader keeps at most 8 refs in flight, a subscription owns no queue (a slow consumer slows its own pulls, and a lagging one reads history from the cache tier rather than pinning leader memory), and the tail window cache evicts by its budget alone.
Source documents
README.mddocs/design/000-decisions.mdcmd/novadandcmd/novagatewayflag help