Storage layout

Every record byte lives in object storage in one of two forms: a transit bundle written by the write path into a chain slot, or a settled object the auditor later builds per stream. Both share one artifact format, and every key is derivable from metadata coordinates: nothing is ever found by listing a bucket.

The key scheme

Slot keys are transient and walkable: a reader holding only a star, an epoch, and a sequence reaches any slot with one point GET, and lexical order equals slot order. Settled keys nest under the stream's bucket so an operator can apply storage-class transitions, cost attribution, or erasure by prefix; they are a pure function of what the object covers, so a crashed settle retries onto the same key and converges. No key carries a UUID, and there is no separate index object: the catalog row in the metastore is the settled index.

An object's storage class is not in its key. It is recorded on every metadata record that names the object (the ref or catalog entry, the orphan intent, the due marker) and resolved against the star's class-to-store map. Settle outputs always land in standard class: the only express-resident objects are live chain slots.

The bundle format

A bundle is the artifact both forms share: stream-grouped chunks, an embedded footer, and a fixed trailer. It is readable with nothing but the object itself.

[ slot envelope — 60 bytes, transit bundles only ]
[ chunk: stream A, records 1000..1130 ]   ← stored records, back to back
[ chunk: stream B, records 7..9 ]
[ chunk: stream C, records 44100..44107 ]
[ footer:  directory entries | chunk index ]
[ trailer — 40 bytes ]

The slot envelope (transit only) is 60 bytes: magic CHNE, version 1, kind (flight or poison), epoch, slot sequence, the writer's nonce, the object size, the footer's offset and length, and a CRC-32C. A poison slot is an envelope and nothing else. One bounded GET at offset zero classifies a slot and locates its footer. Bundle coordinates are laid out at base offset 60, so every offset a ref stores is object-absolute; refs into slots resolve exactly like refs into any bundle.

A chunk is one stream's contiguous run of records inside one object: a region, and the unit a ref maps to. Its bytes are a bare concatenation of stored records with no absolute offsets inside, so a chunk range relocates verbatim: the settle copies chunks server-side into per-stream objects without decoding them.

The footer has two parts. The directory holds one entry per chunk: stream id, first record, record count, data offset, a CRC-32C over the chunk's bytes (DataCRC), then per-record tables of stored-record lengths, timestamps, and batch-end flags. The chunk index holds one fixed 52-byte row per chunk (stream, first record, count, chunk bytes, and the directory entry's offset, length, and CRC), sorted by (stream, first record). A point reader binary-searches the index and decodes exactly one entry, verified by its own CRC; a probe that only needs extents reads the index alone.

The trailer is 40 bytes: magic (CBDL), format version, directory offset and length, directory CRC, index length, index CRC. A reader fetches the footer with one ranged GET at the coordinates a ref or the envelope supplied, or recovers it from the trailer when it holds only the object size.

The stored record

A chunk's data region is a concatenation of stored records: one flag byte, a width-packed header table when the record has headers, then the body. The byte layout, with worked examples, is on the records page; what matters here is what the record does not carry. It has no length of its own: the directory's per-record length frames the whole stored record, and the body is the remainder after the header table. It has no sequence number or timestamp: the directory's first record and per-record timestamp table supply both. And it has no checksum: integrity is container-granular, through the chunk's DataCRC at rest and the slot envelope's CRC in flight. Every cap and budget meters content bytes (body plus header names and values); the framing is unmetered.

Transit bundles

A transit bundle is one flight: the pool's batches, sorted by stream and first record, rendered once and written by conditional PUT into the star's next chain slot; see the write path. It mixes many streams by design; that is what makes one PUT amortize across the whole star's traffic. Its refs are span refs: one metastore row per stream per absorb batch, whose entries name the slots the stream appears in and the footer window of each; slot keys are derived, never stored.

A transit bundle lives until the settle has moved every stream's chunks out of it and the auditor's trim cursor passes it. Its residency is bounded by the class's linger and age cap (--denorm-linger 5m and --denorm-age-cap 24h for standard; --express-denorm-linger 5m and --express-denorm-age-cap 1h for express); see settlement.

Settled objects

A settled object holds one stream's contiguous records, assembled by server-side copy of chunks out of transit bundles and rewritten as a bundle of its own. The catalog row in the metastore maps a run of them; each entry carries the epoch, first record, revision, footer window, byte count, time bounds, and storage class from which the key derives.

Three rules keep the settled tier convergent:

• Create-once. A settled publish completes conditionally (If-None-Match: *). A retry that loses the race reads the standing object's trailer and footer back and commits metadata describing those bytes, never replacing them. A settled object, once complete, is immutable at its key.
• Revisions, never overwrites. A rewrite that re-covers the same first record (the tail-fill that extends a stream's last object toward the output cap, a GC rewrite that drops dead records) lands at ~<revision+1>, repoints the catalog entry, and only then arms the old object's reap. A crash between any two steps leaves both generations standing.
• One sub-cap tail. Each settle first extends the stream's standing tail object toward --denorm-output-cap-mib (default 256) and then emits any remainder at full size, so a stream has at most one object below the cap, always its last.

The express bucket set

--s3-express-buckets names one express-class directory bucket per availability zone (zone=bucket,…). Every chain slot fans to the whole set under the same key; reads prefer the bucket serving the star's --location and fall through on a miss or a down zone; deletes fan to every bucket and tolerate absence. Only live chain slots are express-resident: an Express stream's settled objects are standard class, so the express bill covers the transit window, not the retention. Directory buckets do not list in lexicographic order; nothing in Nova depends on list order against any backend.

Lifecycle rules and versioning

Only the auditor deletes objects. A bucket lifecycle expiry rule is forbidden: a stalled settle plus auto-expiry would delete referenced data. The one lifecycle rule the reference deployment installs is AbortIncompleteMultipartUpload after 7 days, which reclaims the parts of a settle that died mid-assembly; the reference bucket also enables default server-side encryption.

Every artifact is sealed by a version-stamped trailer. A reader accepts a range of versions and dispatches the directory layout by version, so a future format change bumps the version, adds a decode branch, and keeps old data readable across a rolling upgrade. Two failures stay distinct on purpose: an unsupported version means the bytes are intact and the software is wrong (deploy the right build); a corrupt artifact (bad magic, out-of-range bounds, checksum mismatch) means damaged data. The current format is version 1 for both the trailer and the slot envelope.

Source documents

• docs/design/025-stored-record.md
• docs/design/013-ref-spans.md
• docs/adr/0010-object-key-scheme.md
• docs/adr/0015-artifact-format-versioning.md
• docs/adr/0029-settled-catalog.md