Architecture
Metadata
The metastore — oxia behind a small KV interface, its three namespaces, key families, CAS discipline, locality, and scaling
The metastore holds every coordinate and no record: where a stream's records live, who leads it, which epochs are open, and what work is due. It sits off the append path entirely, so its load scales with decisions (leadership changes, absorb commits, reclamation), never with traffic.
A small interface over oxia
Nova reaches the metastore through a lowest-common-denominator key-value interface so the backend can be swapped; oxia is the production backend and an in-memory store runs the same conformance suite. The store is linearizable and versioned:
| Operation | What it does |
|---|---|
Get, Put(expected), Delete(expected) | Point access. Every write names an expected version: a concrete version, must not exist (a safe create), or any (unconditional; used only where the record has one writer by construction). |
PutEphemeral | A session-bound write: the record vanishes when the client's session ends, judged server-side. Liveness and leases ride it. |
FloorGet(prefix, probe) | The greatest key at or below the probe within a prefix: the ref lookup. |
Scan(prefix, after) | The direct children of a path prefix, ascending, early-stoppable, bounded by the caller. |
DeleteChildren, DeleteRange | The deletion mirrors of a scan, single-shard within a locality. |
PutIndexed, FloorIndex, ScanIndex, GetByIndex | Secondary indexes maintained by the backend, atomically with the record. |
Within(localityKey) | A colocation view: every operation routes by the locality key so related records share one shard. |
Values are protobuf. A metadata mutation is a read-modify-write of the whole value under CAS, and protobuf preserves unknown fields across that round-trip, so during a rolling upgrade an older star that rewrites a record for its own reason carries a newer star's field through untouched.
Three namespaces
Namespaces split by lifecycle, not by volume (oxia reshards volume on its own):
| Namespace | Flag | Holds | Posture |
|---|---|---|---|
| records | --oxia-records-namespace (default: the client's default) | stream descriptors and names, refs and catalog rows, trim watermarks, last-writer and fencing-token rows, chain epoch records, buckets, credentials | the durable system of record; forward-only, never restored from a snapshot, because a restore would move epochs and trim points backward at once |
| work | --oxia-work-namespace | the auditor's due-queues and orphan intents | high-churn and fully reconstructible; isolated so its compaction stays off the record log |
| ephemeral | --oxia-ephemeral-namespace | star liveness, node stats, writer leases, the auditor ring, job claims | session-scoped; never backed up; a wipe is a non-event for correctness |
The records namespace is reached as two services. --oxia is the hierarchical keyspace, where / is structure and a scan yields direct children. --oxia-naming is a natural-sorted service for the naming keyspace, where stream keys are opaque caller bytes and / is data: point operations hash-route by full key, and prefix listing runs in plain bytewise order at any depth. The two sort orders are separate types in code, because pointing one contract at the other silently truncates ranges.
Key families
Naming (natural-sorted)
| Key | Record | Index |
|---|---|---|
streams/<bucket>/<key> | the stream descriptor at its address; the must-not-exist create is the uniqueness gate | stream-id: the reverse lookup by stream id, swapped atomically on create, tombstone, and replace |
buckets/<name> | the bucket and its default stream configuration | |
credentials/<key-id> | an api-key credential (secret stored hashed) |
Per-stream rows (records, locality streams/<id>)
| Key | Record |
|---|---|
streams/<id>/refs/<firstSeq> | the refs keyspace, keyed by the first record covered. Three kinds share it: a span ref (the absorb's row, one per stream per batch, whose entries name chain slots and footer windows), a single-object ref, and a catalog row (a run of settled objects, up to 128 entries). The row also carries the chain facts: the epoch, the builder star, and the clean-release mark. |
streams/<id>/trim | the monotonic trim watermark |
streams/<id>/writer | the last-writer record: which star held the stream, written before its first acknowledgement |
streams/<id>/producer | the fencing token |
streams/<id>/lease (ephemeral) | the writer lease: the holder's identity, session-bound |
Chains (records, locality chains/<star>)
| Key | Record |
|---|---|
chains/<star> | the mint counter, with a head epoch per storage class |
chains/<star>/<epoch> | one epoch: state (active, fenced, settled), the mint-time constants (class, bucket zones, quorum, window), the cursors (checkpoint, trimmed, rewritten), and the cut once a fence publishes it |
Due markers and intents (work, locality due/<type>/<shard>)
| Key | Meaning |
|---|---|
due/<type>/<shard>/<readyAt>~<target> | one unit of work becoming due at readyAt (the target is the stream, chain, or object it concerns), for settle, denorm, retention, reap, gc, teardown, and trimcheck |
due/orphan/<shard>/!<escaped-object-key> | an orphan intent: this object may exist before any ref names it |
Shards are 64 fixed partitions by hash of the target. Due entries fuse order and target into one final path segment so each shard's queue is a single-shard ordered scan: exactly the work that is due, nothing else. A by-stream index on markers lets the admin plane show one stream's pending background work with a bounded prefix scan instead of a queue sweep.
Fleet (ephemeral)
| Key | Record |
|---|---|
cluster/stars/<star-id> | liveness and routing facts: location, address, draining, saturated |
cluster/stats/<star-id> | the node stats record: fixed-size load aggregates, refreshed often, kept beside the liveness record so its churn never rewrites routing |
cluster/auditors/<star-id> | the auditor ring: membership deals due-queue shards by rendezvous |
auditor/lease/… | job claims: at most one auditor runs a given job at a time |
Every mutation is a compare-and-set
No record is ever written by blind read-modify-write. Creates use must not exist; updates carry the version they read; a mismatch means someone else moved first, and the caller re-reads and re-decides. The metastore offers no multi-key transaction, so every change touching several records is an ordered, idempotent sequence. The settle, for example, publishes the object, pivot-swaps the first row (the one CAS that commits visibility), deletes the covered span rows ascending, then resolves markers and the intent, in that order, so a concurrent floor lookup lands on a valid row at every step. A crash anywhere re-runs to convergence, and the tests enumerate the crash points.
Locality
A stream's whole world (id-keyed rows, refs, trim, writer, producer, lease) shares the locality key streams/<id>, so every per-stream operation is single-shard, ordered, and early-stoppable, and a teardown is a fixed set of single-shard deletes. Distinct streams hash across shards; a star's chain records colocate under chains/<star>; a due-queue shard under due/<type>/<shard>. The naming keyspace deliberately has no locality: a bucket is unbounded, and pinning its rows together would build a hot region shard splits could never divide. One stream on one shard is a feature (exactly one ordering authority per stream), and throughput never concentrates because the metastore sees objects, not appends.
Point lookups, never scans
A read resolves through one FloorGet on the refs keyspace: the row whose first record is at or below the probe, whether it is a span ref or a catalog row, and a binary search over the entries in that one value. Time-addressed reads and age retention take one floor lookup on the by-ts secondary index (written atomically with every ref commit) and confirm against the primary row. The only ordered cross-shard scan in the system is the operator's stream listing, bounded and paged.
There is no "list all streams" on any hot path and no keyspace sweep anywhere: work self-announces. The write that creates a condition (a batch absorbed, a trim advanced, an object published, a stream deleted) arms the due marker for it, and the auditor drains due-queues in order. See due-queues.
Sessions, pipelining, and batching
--oxia-session-timeout (default 15s) is the fleet's failure-detection floor: a dead star's leases and liveness survive this long, and failover waits it out. Keep it above the worst process stall you expect.
The client pipelines writes: --oxia-write-window (default 4) batches in flight per shard, ordered even across failovers, and --oxia-max-batch-kib (default 128) caps one batch. With one batch in flight per shard these two numbers bound a star's row throughput, which is what the absorb's commit fanout (--absorb-fanout, default 4096) is tuned against.
Scaling by heat
Oxia shards split, and splits are forever, so the plane scales by attributing load before acting. Every star and auditor keeps a 4096-bucket windowed histogram of its own metadata operations placed on oxia's hash ring (a stream's ops accrue to the bucket of its locality key) and publishes it on its node stats record. The controller merges those histograms, slices them by oxia's shard ranges, and splits the hottest shard at its load-weighted median: dividing the load, never merely the keyspace. A shard whose single hottest bucket carries half its load is quarantined onto a private shard; a private shard that stays hot names an indivisible hot stream: a signal for the layer above, not another split. Servers follow shards; latency raises an alarm but never scales anything. The controller's side is on the Kubernetes page.
Source documents
docs/adr/0003-keyspace-partitioning-namespaces.mddocs/adr/0005-protobuf-kv-values.mddocs/adr/0013-kv-secondary-indexes.mddocs/adr/0023-coord-fleet.mddocs/adr/0029-settled-catalog.mddocs/design/013-ref-spans.md