novadocs
Concepts

Concepts

Guarantees

What nova promises per stream — linearizability, durability at ack, contiguous stable ids, bounded memory — what it does not, and the limits

This page states nova's promises precisely enough to build on, says what is deliberately not promised, and lists the limits a caller runs into. Everything here is enforced by code and tests, not asserted.

Every stream is linearizable

A stream has exactly one sequencer at a time. Every acknowledged append is totally ordered with every other, and Read, Subscribe, and CheckTail are linearizable: an acknowledged append is reflected by any subsequent read, whichever star serves it. A reader can never observe a gap: records become visible in sequence order, whole batches at a time.

This holds across leader changes. A deterministic-simulation chaos suite checks the linearizability of concurrent producers' histories under injected faults (object-storage brownouts, star bounces, partitions) on every CI run.

Durable at acknowledgement

An append is acknowledged only after object storage has it:

ClassAcknowledged when
StandardThe flight's conditional PUT into the standard bucket succeeded, and every earlier flight of the chain has been acknowledged
ExpressThe flight's PUT was confirmed by the copy quorum (W=2 of the per-zone express buckets, a hard floor never degraded) and every earlier flight has been acknowledged

No metastore operation sits on the acknowledgement path, and there is no local durability tier: a star holds no data on disk, so losing one loses nothing acknowledged. A flight that cannot commit (a refused slot, an unreachable quorum) fails its appends; nova never acknowledges speculatively.

Sequence numbers are contiguous and permanent

Sequence numbers run 0, 1, 2, … with no gaps; a gap is corruption, never intent, and every layer validates contiguity. Trimming never renumbers, and a failover never burns or reuses a number: an unacknowledged tail is discarded whole, so the next acknowledged record follows the last acknowledged one. A sequence number names the same record forever.

A deposed writer cannot write late

Every leadership change bumps the stream's epoch with a compare-and-set, and one seal protocol (fence, walk, repair, poison) computes the previous epoch's definitive end in the storage layer. A writer that lost leadership finds its next slot poisoned: its flight can never reach the copy quorum, so nothing it writes after being deposed is ever acknowledged or visible. This is epoch fencing, and it is the safety mechanism; fencing tokens and producer policy are cooperative policy layered above it.

Batches commit whole

A batch is contiguous (one unbroken range, never interleaved with another producer's records), acknowledged whole (a partial ack cannot be observed), and committed whole (it becomes durable and visible in its entirety or leaves no observable trace). Readers never see batch framing.

At-least-once appends, exactly-once conditionals

A plain append retried after an ambiguous outcome may land twice. A conditional append (match_seq_num) is exactly-once across retries: a resend re-evaluates its match and misses rather than duplicating. The committed tail reported on every ack, rejection, and CheckTail is durable, readable, and monotonic, never a speculative cursor.

Subscriptions deliver in order

A subscription delivers records in sequence order. The SDK resumes a broken subscription at the last delivered sequence number + 1, and an SSE consumer that reconnects with Last-Event-ID resumes at the same point, so a reconnect neither skips nor repeats a record.

Memory is bounded

Every queue, buffer, cache, and in-flight population on a star has an explicit budget, and reaching it applies backpressure rather than growing: producers block at the pooled-append budget (--s3-memory-buffer-mib, 256 MiB), the chain backlog (--s3-chain-backlog-flights, 1024), and a stream's throughput bound; new work is shed at a population cap: producer sessions (--max-producer-sessions, 4096 per star), subscriptions (16384 per star), SSE connections (4096 per HTTP host). A shed caller gets ResourceExhausted on gRPC, 429 or 503 with Retry-After over HTTP, and a log line records it every time. Nova is built never to run out of memory, including through an object-storage outage; the bounds are in backpressure.

What is not promised

  • Ordering across streams. Sequence numbers and timestamps relate records within one stream only.
  • Exactly-once for plain appends. Use match_seq_num, or dedup on sequence numbers and your own record identity.
  • Fencing tokens as access control. On any streams a token-less writer always passes; tokens coordinate cooperating writers. Locking a caller out is authorization's job.
  • Wall-clock-exact timestamps. Stamps are clamped monotone within the stream and, unless uncapped, capped at arrival time; two records may share a stamp, and a clamped stamp may be later than the clock said.
  • Prompt physical reclamation. A trim is logically instant; the bytes go when the auditor reaches them.
  • A tail from a read plan. A plan partitions responsibility; only the ordinary verbs answer where the stream ends.
  • The tail in a read response as a bound on the future. It is the tail as of that response.

Limits

LimitDefaultWhere
Record size (content bytes)1 MiBnovad --max-record-bytes
Batch size100 MiBnovad --max-batch-bytes
Records per batch65536novad --max-batch-records
Headers per record100fixed
Header namenon-emptyfixed
Records per read response8192novad --max-read-records
Bytes per read response8 MiBnovad --max-read-bytes
SDK wire batch1 MiBfixed
Session in-flight window16 batchesadvertised at open
Bucket name3 to 63 bytes, [a-z0-9-], no leading or trailing hyphenfixed
Stream key1 to 512 bytes of UTF-8, no control charactersfixed
Fencing token≤ 36 UTF-8 bytesfixed
Streams per list page1000fixed
Read plan entries256fixed
Producer sessions per star4096novad --max-producer-sessions
Subscriptions per star16384fixed
SSE connections per HTTP host4096fixed
Express copy quorum2 (hard floor)novad --s3-express-ack-quorum

Error taxonomy

ConditiongRPCHTTPGo SDK
Wrong star for a leader-served verbFailedPrecondition + NotOwner{holder_addr}never surfaces (proxied or followed)followed transparently
match_seq_num missOutOfRange + Sequence{tail}412 with tailSequenceError (ErrSequence)
Fencing token mismatchFailedPrecondition + Fence{installed_token}412 with fencing_tokenFenceError (ErrFencedOut)
Read below the trim watermarkOutOfRange410ErrTrimmed
Record, header, or timestamp rule brokenInvalidArgument400ErrRecordTooLarge
Stream or bucket missingNotFound404ErrStreamNotFound
Address already takenAlreadyExists409passed through
Population cap reachedResourceExhausted429retried within the budget, then ErrUnavailable
Convergible (draining, settling, lease moving)FailedPrecondition / Unavailable503 + Retry-After: 1retried within the budget
Bad or missing credentialUnauthenticated / PermissionDenied401 / 403ErrUnauthenticated / passed through

The full list with every status is in the limits reference.

Source documents

  • docs/design/009-chain-commit.md
  • docs/design/011-virtual-log.md
  • docs/design/006-chaos-matrix.md
  • docs/design/000-decisions.md
  • internal/transport/grpc/dataplane/status.go, internal/transport/http/dataplane/errors.go

On this page