Architecture
Overview
The components of a Nova constellation, what talks to what, and the invariants every page below assumes
Nova is a distributed system at its core: a fleet of identical stars scales horizontally, and any star serves any stream. It also runs as a single binary. One novad next to a metastore and a bucket is a complete constellation, and novad --dev collapses even those into one process with in-memory backends for local work. Whatever the shape, the serving fleet holds no data: records become durable when object storage has them, the metastore holds only coordinates, and a background auditor turns freshly committed data into its long-lived form. This page names the parts, draws the wiring, and states the rules that shape every mechanism described in the deeper pages.
Components
A running novad is a star; the running system (stars, the audit pool, and the metadata plane behind them) is the constellation.
| Component | What it is | What it holds |
|---|---|---|
novad (star) | The serving plane, the admin plane, and by default an embedded auditor, in one process. Any star serves any stream: history from anywhere, the live tail via the leader. | Nothing durable. Its identity comes from --node-id or a node-id file in --dir; record bytes never touch local disk. |
| Audit-only pool | The same image run with --audit-only: no serving plane, no star registration, only the auditor's duties. Deploy it so background I/O never competes with serving; pair it with --no-audit on the stars. | An optional scratch directory (--auditor-fetch-cache-dir) for the settle's whole-object fetch cache. |
novagateway | The internet-facing tier: TLS, edge token verification, per-bucket hosts, CORS, limits. Re-encodes the data plane as JSON and SSE and speaks gRPC into the fleet, following leader redirects on the caller's behalf. | Only the caller's forwarded credential. novad itself is never internet-exposed. |
| Metastore (oxia) | A replicated, linearizable key-value store behind a small interface. Reached as two services: --oxia for the hierarchical keyspaces and --oxia-naming for the natural-sorted naming keyspace. | Naming, stream descriptors, chain epoch records, refs and catalog rows, due-queues, leases and liveness. Never a record body. |
| Object storage | One standard bucket (--s3-bucket) and, when Express is enabled, a bucket set of express-class directory buckets, one per availability zone (--s3-express-buckets zone=bucket,…). | Every record byte: transit bundles in chain slots, and settled per-stream objects. |
| Object cache | A pool of read-through caching proxies in front of the standard bucket (--object-cache-instances). Stars and the Gateway route cold reads through it; the instance for a key is chosen by ring hash with bounded load. | Cached pages only. Never authoritative: a miss falls through, a failure falls back to a direct ranged GET, and the auditor's deletes route through it so cached bytes die with the object. |
The two planes
Every star serves both planes on one gRPC listener (--listen, default :7780):
- Data plane,
DataPlaneService: producer sessions, one-shot appends, reads, subscriptions, tail checks, trim, stream and bucket lifecycle. The Go SDK speaks it directly. The same surface is re-encoded as HTTP by the data bridge onnovad(--http-listen, for callers inside the trust domain) and by the Gateway for everyone else. A star that does not lead a stream answers a redirect; the SDK, the bridge, and the Gateway all follow it. - Admin plane: health, the fleet overview, node stats, drain and decommission, credentials, and token minting.
novaand the controller use it. A read-only JSON mirror of its query RPCs (the admin bridge) serves the admin UI on the metrics listener (--metrics-listen,--admin-ui).
What talks to what
The write path never touches the metastore: a producer's acknowledgement is the conditional PUT of a flight into the star's chain, at the copy quorum of the bucket set (write path). The metastore sees leadership changes, absorb commits, and reclamation: decisions, not traffic (metadata).
Invariants
The serving fleet is stateless. A star keeps no record data on local disk. Losing one loses nothing; adding one adds throughput; draining one is a leadership handoff, never a data move. The tail a leader serves from memory is always also durable in its chain, and a successor recovers it by walking the chain, not by asking the dead star.
Memory is bounded everywhere. Every pool, queue, cache, and in-flight set has an explicit limit, and reaching it applies backpressure: block or shed, always logged, never silent growth. The append pool (--s3-memory-buffer-mib, default 256), the chain backlog (--s3-chain-backlog-flights, default 1024), the tail window cache (--tail-window-cache-mib, default 128), producer sessions (--max-producer-sessions, default 4096), and the auditor's fetch cache all carry a budget. A stall at a budget is a diagnosed fault signature, not normal operation (backpressure).
Every metadata mutation is a compare-and-set. Records are written against an expected version; there is no blind read-modify-write. The metastore has no multi-key transaction, so every multi-record change is an ordered, idempotent sequence that converges when re-run.
Nothing scans a keyspace or a bucket. Work announces itself: the write that creates a condition arms a due marker for it in the same operation, and the auditor drains due-queues in order. Reads are point lookups (FloorGet) and bounded prefix ranges; object reads are ranged GETs at coordinates a footer or a ref supplied. Object leaks are found by intent markers, not sweeps.
Every multi-step sequence is crash-safe and idempotent. Commit, settle, rewrite, trim, and teardown each enumerate their crash points; a re-run after a crash at any step converges to the same state.
Sequence numbers are contiguous. A stream's records are numbered 0, 1, 2, … with no gaps, ever. A gap is corruption, never intent, and audits rely on count = last − first + 1. Trimming never renumbers.
Where the code lives
| Package | Role |
|---|---|
internal/star | The serving plane: leaderships, the sequencer, producer admission, reads, subscriptions, stream and bucket CRUD |
internal/stream | The domain model: ids, descriptors, policies (a leaf; imports nothing internal) |
internal/kv, internal/metadata | The swappable metastore (oxia, mem) and every record schema and CAS discipline over it |
internal/coord | The ephemeral namespace: liveness, node stats, writer leases, placement, rebalance |
internal/storage/chain | The loglet (slots, fence, walk, trim), the durability commit, the absorb, and settlement |
internal/storage/durability | The write core (pool, flusher, flight index), assembled once per class |
internal/storage/bundle, codec, objectstore | The object format, its encodings, and the object clients (standard, express fanout, object cache) |
internal/storage/rewrite, reclaim, internal/auditor | Settle, retention, reap, teardown, orphan judgment, and the due-queue scheduler that drives them |
internal/transport, internal/gateway | The gRPC and HTTP planes, scoped-token auth, the Gateway's edge |
internal/daemon | The assembled star: config, backends, serving, drain, audit, close |
Go deeper
Write path
From append to acknowledgement: pool, flusher, chain slots, the copy quorum, both classes
Epochs
Leadership epochs, the fence walk, the cut, and why a deposed writer cannot commit
Storage layout
Bundles, footers, chunks, stored records, and the object key scheme
Metadata
The metastore: namespaces, key families, CAS, locality, and how it scales
Read path
The tail window, cold reads, the object cache, and vended read plans
Leadership
Leases, placement, handoff, rebalance, saturation, drain
Settlement
Absorb and settle: transit to settled, linger, age cap, eras
Backpressure
Every bound, what fills it, and what happens at the limit
The Gateway
The internet-facing tenant tier: the trust boundary, the middleware stack, plan-driven serving
The object cache
The shared read-through tier: ring routing, fallback, zone scoping
The auditor
The background role: duties, due-queues, reclamation, tuning
Source documents
README.mdCONTEXT.mddocs/design/001-modules.mddocs/design/011-virtual-log.md