Modeling streams

Streams are cheap enough to model at the natural grain of your domain: one per entity rather than one topic sharded by key. This guide shows the patterns that fall out of that, how to design keys and buckets so listing and governance work for you, and which per-stream choices to make where.

Model at the natural grain

Creating a stream is one metadata write, and the per-stream cost is the same at ten streams or ten million. So put one stream where you would otherwise multiplex through a partitioned topic and hope the key hashes well:

Each stream is independently ordered, independently retained, and independently readable from any star. Ordering across streams is not promised, so the grain you pick is the grain of your ordering guarantee: if two things must be ordered against each other, they belong in one stream.

Design keys as stable identities

A key is opaque UTF-8 that nova never parses, and / is a grouping convention; S3 reads it the same way. Put the dimension you group by first, so keys read well in listings and tooling:

tenant-42/orders/8a1f…       one order's history
tenant-42/orders/9c02…
tenant-42/audit               the tenant's audit log
device/eu-west/sensor-0017   a device
run/2026-08-24/build-3391    a workflow run

orders, orders/, and orders/eu are three unrelated keys; a key never implies its prefixes exist, and listing is per bucket (there is no key-prefix filter), so put streams you want to list together in one bucket. Keys are immutable: encode identity, not state that changes (no status words, no owners that reassign). Remember that a stream recreated at the same address is a new stream: if you delete and recreate tenant-42/audit, its history starts at 0 again.

Use buckets for governance, not for grouping

A bucket is where configuration lives: every stream created in it starts from its defaults, and its create flags decide whether first use creates streams. Group by policy, not by domain hierarchy; that is what key prefixes are for.

# Long-lived, throughput-bound, fenced writers.
nova bucket create ledger --default-class standard

# Latency-sensitive, short retention, create-on-append for many small streams.
nova bucket create sessions --default-class express --create-on-append

Set the strict values in the defaults and let streams relax nothing: scalar fields (retention age and bytes, throughput, uncapped) cannot be overridden back to zero by a create, so a bucket with a bound is a bucket whose every stream has at least that bound. Put the permissive values in the bucket's defaults and tighten per stream when you need to.

You can set everything the default configuration holds (class, producer policy, timestamping, retention, batch, throughput) over HTTP when you create the bucket:

curl -s -X POST localhost:8080/v1/buckets/sessions -d '{
  "create_stream_on_append": true,
  "default_class": "express",
  "default_producer_policy": "any",
  "default_retention_max_age_ms": 86400000,
  "default_timestamping_mode": "arrival"
}'

Let first use create the stream

With create_stream_on_append on the bucket, producers never run a create step: the first append to a new key mints the stream from the bucket's defaults, exactly once, and concurrent first appends all land on the one stream. This is the shape for stream-per-device or stream-per-session models where the set of keys is open-ended.

addr := client.Address{Bucket: "sessions", Key: "chat/" + sessionID}
res, err := c.Append(ctx, addr, client.Bodies(event)...)

create_stream_on_read does the same for subscribers that may arrive before their producer: the stream is created empty and the subscription waits at sequence number 0.

Choose the class per bucket

Both keep their history in standard storage; Express only changes what the head of the stream costs and how fast it acknowledges. The class is set at creation (a reconfigure can move a stream later, at the cost of one handoff), so a bucket whose streams share a latency need is the right unit; see Storage classes.

Choose timestamping and retention per bucket

• client-prefer (the default) takes the producer's stamp when offered, capped at arrival time. Right for most event data.
• client-require refuses records without a stamp, for pipelines where a missing event time is a bug.
• arrival refuses client stamps and stamps arrival time instead, for logs and audits where the server's clock is the truth.
• uncapped lets an application stamp on its own monotone scale (a log sequence number). It excludes age retention, since retention reads record time as wall clock.

Age retention (retention_max_age_ms) advances the trim watermark for you with seconds-scale precision. Put it on the bucket for anything with a natural horizon (sessions, telemetry, delivery attempts) and leave it unbounded for ledgers and audit logs you trim by hand, if ever.

Single-writer designs: fenced producers

When each stream has one logical writer (an aggregate's command handler, a device's uplink, a workflow engine), use fenced so nova refuses a second one, and rotate the fencing token at every failover:

token := fmt.Sprintf("gen-%d", generation)       // any value ≤ 36 bytes
_, err := c.Fence(ctx, addr, token)              // supersedes the incumbent
p, err := c.OpenStream(ctx, addr, client.WithFencingToken(token))

Fence returns the tail as the admission boundary: everything at or below it was written by the previous writer. The token is policy for your own writers; the guarantee that a deposed writer cannot append late is epoch fencing, enforced underneath regardless.

Use any for fan-in (many devices, request handlers, or workers appending to one stream), where interleaving in arrival order is the point.

Optimistic concurrency with match_seq_num

When several writers may act on one stream but each write depends on the stream's current state (event sourcing's "apply command to aggregate at version N"), read to the tail, decide, and append conditionally on the tail you saw:

tail, err := c.CheckTail(ctx, addr)
events := decide(load(ctx, c, addr, tail.SeqNum))

res, err := c.AppendMatch(ctx, addr, client.SeqNum(tail.SeqNum), client.Bodies(events...)...)
var seq *client.SequenceError
if errors.As(err, &seq) {
	// Someone appended first: reload from seq.Tail and try again.
}

A miss rejects only that batch and reports the current tail; the one-shot form is always definitive. On a session, Producer.AppendMatch does the same and stays usable after a miss.

Use CheckTail as the truth about "now"

CheckTail is strongly consistent and served by the stream's leader. Use it to bound a catch-up read ("read everything up to the tail I saw, then subscribe from there"), to resolve an ambiguous append (check the tail, read the suffix, look for your record), and as the version in optimistic concurrency. It never creates a stream and never moves leadership.

A worked example: orders

nova bucket create orders --default-class express --create-on-append
nova bucket create orders-audit --default-class standard

• orders/<order-id>: one Express stream per order, fenced, written by the order's command handler with match_seq_num; read by projections that subscribe from 0.
• orders-audit/<tenant>: one Standard stream per tenant, any, arrival timestamping, appended by every service that touches the tenant's orders; retained for as long as compliance says, then trimmed by time.
• Listing the orders bucket gives you the order streams; listing orders-audit gives you tenants.

Source documents

• docs/design/020-stream-addressing.md
• docs/design/000-decisions.md