Storage classes

Standard and Express compared, in where a write lands, how it becomes durable, what that costs in latency, and why history always ends up in standard storage

Every stream has a class, set at creation, that decides where the head of the stream is written and how a write becomes durable. There are two. Standard writes each flight once into standard object storage and lets the storage's own replication make it durable. Express writes each flight into express-class buckets in two or more availability zones and acknowledges when a quorum of them confirm. Both are durable across zones at acknowledgement. They differ in how they get there, in latency, and in cost; and whichever you pick, long-term history settles into standard storage.

Standard: one PUT into replicated storage

Standard object storage replicates on its own. S3 Standard stores every object across at least three availability zones and is designed for 99.999999999% (eleven nines) durability, so a single successful PUT is already a multi-zone commit. Nova takes that at face value: the flusher pools a producer's batches, cuts a flight, and writes it with one conditional PUT into the next chain slot. That PUT is the commit and the acknowledgement; nothing else on the path has to agree.

The price is latency. A flight is cut when the pooled payload reaches 8 MiB or when the oldest pooled byte has waited 200 ms, and the PUT itself takes tens to hundreds of milliseconds, so an append is acknowledged a few hundred milliseconds after it arrives. PUTs pipeline up to 128 deep, and acknowledgements release in slot order, so throughput does not pay for that latency. The flags are in the novad reference.

Use Standard where a few hundred milliseconds to durable is fine and volume matters: bulk event capture, telemetry, change feeds, audit history.

Express: single-digit milliseconds, one zone per bucket

Express-class buckets (S3 Express One Zone) answer a PUT in single-digit milliseconds, but each bucket lives in exactly one availability zone. A PUT into one of them is durable only within that zone; if the zone is lost, so is the object. Nova therefore supplies the cross-zone replication itself.

A star running Express is configured with a bucket set: one express-class bucket per zone (--s3-express-buckets, as zone=bucket pairs). The flusher writes each flight to every bucket in the set in parallel and acknowledges the append when W of them confirm. W is 2 by default and a hard floor, so at the acknowledgement the flight exists in at least two zones. A zone that goes down mid-write is invisible to producers while two zones still answer. Copies that straggle keep landing in the background, and a copy that never lands is tolerated: reads fall through the set, and deletes fan to all of it.

Because the acknowledgement waits for the second-fastest PUT rather than the slowest, a slow zone hides behind the others, and a durable, two-zone acknowledgement costs on the order of one express PUT. When a quorum cannot be reached within the publish deadline (2 s by default) the flight's appends fail fast and producers retry. Nova never quietly degrades to W=1 or falls back to the standard bucket.

Express is a star capability

Only stars that run with --s3-express-buckets serve a stream created as Express. Today, nova accepts a create that names Express on a fleet without it, and the first append fails; make sure every star that can lead the stream has the bucket set configured. In --dev, pass --s3-express-buckets zone=name,… to run an in-memory bucket set.

Use Express for latency-sensitive streams that still need multi-zone durability at acknowledgement.

History always settles into standard storage

The class decides where the head of a stream lands, not where its history lives. In both classes a background settle relocates records out of their chain slots into per-stream settled objects in standard-class storage, and the read path serves that history the same way whatever the stream's class.

For Express this matters twice over. Express buckets bill for the bytes they hold, so a flight sits in its express slots only for the transit window, bounded by the express age cap (--express-denorm-age-cap, 1 h by default; Standard's cap is 24 h). Tail reads inside that window are express-fast, and reads prefer the bucket in the star's own zone. Everything older is served from standard storage, exactly as a Standard stream's history is. The express bill covers the head of the stream, never the retention.

When both senses of "class" meet, say which: an Express stream's settled objects are standard storage class.

Side by side

	Standard	Express
Where a flight lands	One standard bucket	K express-class buckets, one per zone
Who replicates across zones	The storage (S3 Standard: at least three zones, eleven nines)	Nova, by fanning each flight to the bucket set
Copies at acknowledgement	1 PUT, already regional	W of K (default 2, hard floor)
Acknowledgement latency	Batch interval (200 ms) plus one standard PUT: a few hundred milliseconds	The second-fastest of K express PUTs: single-digit milliseconds
Zone outage during writes	Absorbed by the storage	Invisible while W zones answer; past the deadline, appends fail fast
Cost of the head	Standard storage per PUT and per byte	Express storage for the transit window (1 h age cap by default)
Where history lives	Standard storage	Standard storage; the settle moves it there
Read path	Same	Same; tail reads prefer the star's own zone
Needs on the star	The standard bucket	`--s3-express-buckets` and one express bucket per zone
Choose it for	Throughput-bound ingestion	Latency-sensitive streams needing multi-zone durability at acknowledgement

Choosing at creation

The class is set when the stream is created, explicitly or copied from the bucket's defaults. It can change later: a reconfigure to the other class hands the leadership off, and the stream's history keeps each record under the class it was written in. The natural place to decide is the bucket:

A bucket whose streams are Express by default

nova bucket create trades --default-class express
nova stream create --bucket trades --key eu/orders

Or per stream, overriding the bucket's default

nova stream create --bucket demo --key hot --class express

Over HTTP

curl -s -X POST localhost:8080/v1/streams/demo/hot -d '{"class":"express"}'

In Go, client.StreamOptions{Class: client.Express}; the zero value client.ClassUnspecified takes the bucket's default.

For the chain protocol itself (slots, the copy quorum, fencing, and settlement), see the write path.

Source documents

docs/design/010-write-path-economics.md
docs/design/011-virtual-log.md
docs/design/000-decisions.md