Concepts
Guarantees
What nova promises per stream — linearizability, durability at ack, contiguous stable ids, bounded memory — what it does not, and the limits
This page states nova's promises precisely enough to build on, says what is deliberately not promised, and lists the limits a caller runs into. Everything here is enforced by code and tests, not asserted.
Every stream is linearizable
A stream has exactly one sequencer at a time. Every acknowledged append is totally ordered with every other, and Read, Subscribe, and CheckTail are linearizable: an acknowledged append is reflected by any subsequent read, whichever star serves it. A reader can never observe a gap: records become visible in sequence order, whole batches at a time.
This holds across leader changes. A deterministic-simulation chaos suite checks the linearizability of concurrent producers' histories under injected faults (object-storage brownouts, star bounces, partitions) on every CI run.
Durable at acknowledgement
An append is acknowledged only after object storage has it:
| Class | Acknowledged when |
|---|---|
| Standard | The flight's conditional PUT into the standard bucket succeeded, and every earlier flight of the chain has been acknowledged |
| Express | The flight's PUT was confirmed by the copy quorum (W=2 of the per-zone express buckets, a hard floor never degraded) and every earlier flight has been acknowledged |
No metastore operation sits on the acknowledgement path, and there is no local durability tier: a star holds no data on disk, so losing one loses nothing acknowledged. A flight that cannot commit (a refused slot, an unreachable quorum) fails its appends; nova never acknowledges speculatively.
Sequence numbers are contiguous and permanent
Sequence numbers run 0, 1, 2, … with no gaps; a gap is corruption,
never intent, and every layer validates contiguity. Trimming never
renumbers, and a failover never burns or reuses a number: an
unacknowledged tail is discarded whole, so the next acknowledged record
follows the last acknowledged one. A sequence number names the same
record forever.
A deposed writer cannot write late
Every leadership change bumps the stream's epoch with a compare-and-set, and one seal protocol (fence, walk, repair, poison) computes the previous epoch's definitive end in the storage layer. A writer that lost leadership finds its next slot poisoned: its flight can never reach the copy quorum, so nothing it writes after being deposed is ever acknowledged or visible. This is epoch fencing, and it is the safety mechanism; fencing tokens and producer policy are cooperative policy layered above it.
Batches commit whole
A batch is contiguous (one unbroken range, never interleaved with another producer's records), acknowledged whole (a partial ack cannot be observed), and committed whole (it becomes durable and visible in its entirety or leaves no observable trace). Readers never see batch framing.
At-least-once appends, exactly-once conditionals
A plain append retried after an ambiguous outcome may land twice. A
conditional append (match_seq_num) is exactly-once across retries: a
resend re-evaluates its match and misses rather than duplicating. The
committed tail reported on every ack, rejection, and CheckTail is
durable, readable, and monotonic, never a speculative cursor.
Subscriptions deliver in order
A subscription delivers records in sequence order. The SDK resumes a
broken subscription at the last delivered sequence number + 1, and an
SSE consumer that reconnects with Last-Event-ID resumes at the same
point, so a reconnect neither skips nor repeats a record.
Memory is bounded
Every queue, buffer, cache, and in-flight population on a star has an
explicit budget, and reaching it applies backpressure rather than
growing: producers block at the pooled-append budget
(--s3-memory-buffer-mib, 256 MiB), the chain backlog
(--s3-chain-backlog-flights, 1024), and a stream's throughput bound;
new work is shed at a population cap: producer sessions
(--max-producer-sessions, 4096 per star), subscriptions (16384 per
star), SSE connections (4096 per HTTP host). A shed caller gets
ResourceExhausted on gRPC, 429 or 503 with Retry-After over
HTTP, and a log line records it every time. Nova is built never to run out of memory, including through
an object-storage outage; the bounds are in
backpressure.
What is not promised
- Ordering across streams. Sequence numbers and timestamps relate records within one stream only.
- Exactly-once for plain appends. Use
match_seq_num, or dedup on sequence numbers and your own record identity. - Fencing tokens as access control. On
anystreams a token-less writer always passes; tokens coordinate cooperating writers. Locking a caller out is authorization's job. - Wall-clock-exact timestamps. Stamps are clamped monotone within the stream and, unless uncapped, capped at arrival time; two records may share a stamp, and a clamped stamp may be later than the clock said.
- Prompt physical reclamation. A trim is logically instant; the bytes go when the auditor reaches them.
- A tail from a read plan. A plan partitions responsibility; only the ordinary verbs answer where the stream ends.
- The tail in a read response as a bound on the future. It is the tail as of that response.
Limits
| Limit | Default | Where |
|---|---|---|
| Record size (content bytes) | 1 MiB | novad --max-record-bytes |
| Batch size | 100 MiB | novad --max-batch-bytes |
| Records per batch | 65536 | novad --max-batch-records |
| Headers per record | 100 | fixed |
| Header name | non-empty | fixed |
| Records per read response | 8192 | novad --max-read-records |
| Bytes per read response | 8 MiB | novad --max-read-bytes |
| SDK wire batch | 1 MiB | fixed |
| Session in-flight window | 16 batches | advertised at open |
| Bucket name | 3 to 63 bytes, [a-z0-9-], no leading or trailing hyphen | fixed |
| Stream key | 1 to 512 bytes of UTF-8, no control characters | fixed |
| Fencing token | ≤ 36 UTF-8 bytes | fixed |
| Streams per list page | 1000 | fixed |
| Read plan entries | 256 | fixed |
| Producer sessions per star | 4096 | novad --max-producer-sessions |
| Subscriptions per star | 16384 | fixed |
| SSE connections per HTTP host | 4096 | fixed |
| Express copy quorum | 2 (hard floor) | novad --s3-express-ack-quorum |
Error taxonomy
| Condition | gRPC | HTTP | Go SDK |
|---|---|---|---|
| Wrong star for a leader-served verb | FailedPrecondition + NotOwner{holder_addr} | never surfaces (proxied or followed) | followed transparently |
match_seq_num miss | OutOfRange + Sequence{tail} | 412 with tail | SequenceError (ErrSequence) |
| Fencing token mismatch | FailedPrecondition + Fence{installed_token} | 412 with fencing_token | FenceError (ErrFencedOut) |
| Read below the trim watermark | OutOfRange | 410 | ErrTrimmed |
| Record, header, or timestamp rule broken | InvalidArgument | 400 | ErrRecordTooLarge |
| Stream or bucket missing | NotFound | 404 | ErrStreamNotFound |
| Address already taken | AlreadyExists | 409 | passed through |
| Population cap reached | ResourceExhausted | 429 | retried within the budget, then ErrUnavailable |
| Convergible (draining, settling, lease moving) | FailedPrecondition / Unavailable | 503 + Retry-After: 1 | retried within the budget |
| Bad or missing credential | Unauthenticated / PermissionDenied | 401 / 403 | ErrUnauthenticated / passed through |
The full list with every status is in the limits reference.
Source documents
docs/design/009-chain-commit.mddocs/design/011-virtual-log.mddocs/design/006-chaos-matrix.mddocs/design/000-decisions.mdinternal/transport/grpc/dataplane/status.go,internal/transport/http/dataplane/errors.go