Records

A record is the unit you append: a body, optional headers, a timestamp, and a sequence number. This page defines what a record carries, how it is sized, whose clock stamps it, how batches are framed, and the bytes nova writes for one record in object storage. How records are numbered and where a stream ends is on the streams page.

What a record carries

Each record carries:

Headers are as opaque as the body: nova stores and returns them, nothing more. Names are non-empty; duplicate names are legal and keep their order; at most 100 headers ride one record. When a record breaks these rules, nova refuses its whole batch as InvalidArgument.

A record's size is its content bytes: the body plus every header name and value. That is the one number every cap and budget meters: the per-record cap (1 MiB by default), the per-batch cap (100 MiB), read page budgets, and the leader's memory accounting. Nova never counts encoding overhead against you.

Timestamps

Every stored record carries a millisecond timestamp, and timestamps never decrease along a stream: each stamp is clamped to be at least the previous record's. That monotonicity is what makes time addressable: a read or trim "from time T" resolves to the first record stamped at or after T by search, never by scanning.

Whose clock stamps a record is the stream's timestamping rule, fixed at creation (or copied from the bucket's defaults):

Under client-prefer and client-require, an offered stamp is capped at arrival time: a client cannot stamp a record in the future. Setting timestamping_uncapped removes that ceiling, which lets an application use its own monotone scale (a log sequence number, for example) as the record time. Uncapped and age-based retention are mutually exclusive: age retention reads record time as wall clock, and an uncapped scale would silently mis-trim, so nova refuses the combination at create and update.

Refusals under client-require and arrival are batch-atomic and surface as InvalidArgument.

On the wire and in the SDK, record time is always int64 milliseconds, never a time.Time, because on an uncapped stream the scale is the application's, not the clock's.

Batches

An append carries a batch of one or more records. The batch is sequenced as a unit under one cursor hold, so its records occupy one unbroken range, never interleaved with another producer's. It is acknowledged whole: the ack (or the one-shot response) covers the entire batch, or the call fails, and you never observe a partial ack. It is committed whole, too: a batch becomes durable and visible in its entirety or leaves no trace, so a crash between flights loses whole batches, never parts of one.

Batch framing is a producer-side concept: readers see records, never batch boundaries. Every record of one batch shares one clamped timestamp, so a batch's start and end positions carry the same time.

A plain append retried after an ambiguous outcome (the connection died between send and ack) may land the batch twice: appends are at-least-once unless you use match_seq_num. See producers for the conditional form.

Codec

In object storage a record is a stored record: one flag byte, a header table when there are headers, then the body. That is all. The sequence number and timestamp are not in these bytes; the chunk that holds the record keeps them in its directory (the first sequence number and a per-record timestamp table), so a record's position in the chunk is its number. See the storage layout for the chunk and its directory.

┌──────┬───────┬─────────────────────────────┬────────────────┬──────┐
│ flag │ count │ (nameLen, valueLen) × count │ names + values │ body │
└──────┴───────┴─────────────────────────────┴────────────────┴──────┘
        └── header table: only when the record has headers ───┘

flag byte, bits from the least significant

  bits 0–1  count width          00 = no headers: the body follows at once
                                 01 = the count is one byte
                                 10, 11 = reserved
  bits 2–3  name-length width    00…11 = 1…4 bytes, little-endian
  bits 4–5  value-length width   the same scale
  bits 6–7  reserved             must be zero

header table, only when the record has headers

  count                          one byte
  (nameLen, valueLen) × count    fixed-width pairs, in header order
  name₀ value₀ name₁ value₁ …    the bytes, in the same order

body                             the rest: no length of its own (the chunk
                                 directory frames the whole stored record)

The writer picks the smallest widths that fit this record's longest header name and longest value, so a record with one 300-byte value pays two-byte value lengths for its own table and nothing for any other record. The table is one fixed-width block: a decoder reads it, adds up the extent, and slices every name and value without scanning the bytes.

The body carries no length. The chunk directory records each stored record's total length, and the body is whatever remains after the header table. A record has no checksum of its own either: the chunk's DataCRC covers every stored record in it at rest, and the slot envelope's CRC covers the same bytes in flight.

Two records, byte for byte:

body "hi", no headers:
  00 68 69                      3 stored bytes, 2 content bytes

body "hi", one header k=v:
  01                            flag: one-byte count, 1-byte lengths
  01                            one header
  01 01                         len("k"), len("v")
  6b 76                         "k" "v"
  68 69                         "hi"
                                8 stored bytes, 4 content bytes

The headerless form is exactly the zero flag byte followed by the body: one byte of framing per record. Every cap and budget meters content bytes, so the framing never counts against a record, a batch, or a read page.

A decoder is strict about the flag. It refuses a set reserved bit, a count width of 10 or 11, a zero count under a one-byte count width, and a headerless flag with any width bits set (the only headerless flag is 0x00), and it refuses a record whose table or header bytes run past the framed length. Each refusal is corruption, never a format the reader is missing: there is one version of this layout, and any future use of the reserved bits is a format change that arrives with a bump of the container's format version.

Limits at a glance

The full table, with error codes, is in the limits reference.

Source documents

• proto/nova/v1/dataplane.proto
• docs/design/025-stored-record.md
• internal/storage/codec/stored.go