Streams

A stream is an ordered, immutable sequence of records with exactly one sequencer at a time. It lives in one bucket and is addressed as (bucket, key). This page covers how a stream numbers its records, what the tail is and how the wire names a point on a stream, and the verbs that create, inspect, list, and delete streams. What a record itself carries, whose clock stamps it, and how batches are framed is on the records page.

Sequence numbers

Sequence numbers run 0, 1, 2, … with no gaps. The stream's sequencer assigns them at admission, and every layer below validates that the next record is exactly the expected one. A gap is evidence of corruption, never of intent; audits rely on count = last − first + 1.

Trimming does not renumber. When you trim history below sequence number N, records 0…N−1 become unreadable but N still names the same record it always did. A sequence number is a permanent name for one record.

Sequence numbers are per stream. There is no ordering relation between records of different streams.

The tail

The tail is where the stream ends: the sequence number of the last record that is durable and readable. On the wire it is always encoded as a position: one past the last committed record, together with that record's timestamp:

message Position {
  uint64 seq_num = 1;      // one past the last record at a boundary
  int64 timestamp_ms = 2;  // that record's time; 0 when none is known
}

So a fresh stream's tail position is {seq_num: 0, timestamp_ms: 0}; after three records it is {seq_num: 3, …}, and 3 is the sequence number the next append will receive. timestamp_ms is 0 when no record's time is known: a virgin stream, or one whose whole history was trimmed away.

Every answer that names a point on a stream uses this shape:

The tail is the committed tail: durable, readable, and monotonic. It never regresses, and it is never a speculative cursor. An append whose match_seq_num misses is told the tail, and tail.seq_num is exactly the next value to try.

Stream lifecycle

Observing verbs never create a stream. Only an explicit CreateStream, or first use in a bucket with a create flag set, does; see create on first use.

A deleted address can be reused, but the stream created there is a new one with a fresh ID and an empty history. The address and ID section explains why.

nova stream create nova://apps/orders [--class standard|express] [--producer any|fenced] [--retention-max-age 168h] [--throughput N]
nova stream get nova://apps/orders
nova stream list --bucket apps [--limit N] [--cursor <cursor>]
nova stream delete nova://apps/orders

curl -s -X POST localhost:8080/v1/streams/apps/orders -d '{}'
curl -s localhost:8080/v1/streams/apps/orders
curl -s -X DELETE localhost:8080/v1/streams/apps/orders

Reconfiguring a stream

A stream's configuration is set at creation and can be changed later with ReconfigureStream.

Changing retention takes effect immediately. The other fields (class, producer policy, timestamping, throughput, batch) do not: the stream's leader reads them once, when it takes the stream, and keeps using those values until it gives the stream up. To change one of them, the leader has to be replaced, and that is what a handoff does. The verb writes the new descriptor, the leader finishes what is in flight and releases the stream, and the next leader opens it under the new configuration. A live producer sees one reconnect, which the SDK hides.

When the call returns, every later append uses the new configuration. Records already acknowledged keep the class they were written under, so a stream can hold Standard history behind an Express head. A reconfigure that changes nothing causes no handoff. Switching a stream to Express on a star with no Express bucket set is refused before anything is written.

nova stream reconfigure nova://apps/orders/eu --retention-max-age 0
nova stream reconfigure nova://apps/orders/eu --class express

curl -s -X PATCH localhost:8080/v1/streams/apps/orders%2Feu -d '{"throughput_max_bytes_per_second":1048576}'

Buckets are different: UpdateBucket replaces a bucket's configuration whole, and it reaches only streams created afterwards; see buckets.

Listing streams

ListStreams pages one bucket's streams in a stable order, up to 1000 per page, with an opaque cursor for the next page: pass the next a page returned as start_after for the following one; it is empty when the listing is complete. There is no key-prefix filter today, and no listing across buckets: a bucket is the scope of every stream enumeration on the data plane (the fleet-wide view belongs to the operator's admin plane). Listings return each stream's descriptor (address, ID, and configuration) but not its fencing token; GetStream adds that.

nova ls apps                                # the bucket's streams: address and class per line
nova stream list --bucket apps --limit 50   # the same, with paging flags

curl -s 'localhost:8080/v1/streams/apps?limit=50'

Paging the buckets themselves is a separate verb with a wider scope; see listing buckets.

Source documents

• proto/nova/v1/dataplane.proto
• docs/design/020-stream-addressing.md
• docs/design/000-decisions.md