Benchmarking with novabench

novabench generates load against a Nova cluster through the public SDK over real gRPC and measures what a real caller sees. It never spawns stars and never touches internal packages; provisioning is yours, and it needs only seed addresses. Each run drives exactly one measured verb (append, read, or subscribe), so every number is attributable; mixed load is several processes side by side.

novabench <verb> -addr <seeds> [flags] [stream-id...]

How a run works

1. Setup: connect, create streams (append) or validate targets (read, subscribe). A failure here is a setup failure: exit 3, no numbers.
2. Warmup (-warmup, default 10s): load flows, nothing counts.
3. Measured window (-duration, default 1m0s): counts, bytes, and latencies land in fixed-memory HDR histograms.
4. Drain and report: in-flight operations complete, the result goes to stdout, the run's streams are deleted (unless -keep). The drain is bounded at the run's end plus the retry budget plus 5 seconds: acks still pending resolve as counted errors, never silently, and the exit code is 1.

The load model

• -rate 0 (default) is the closed loop: workers keep the pipeline full to find the ceiling. Use it for throughput targets. Latency percentiles under saturation include the queueing you created, by design.
• -rate N is the open loop: operations issue on a fixed arrival schedule (N per second per stream or reader) and latency is measured from the scheduled send time, so a stalled server shows up as queueing delay in the tail instead of silently slowing the arrival rate. Use it for honest p99 tracking at a chosen load point. -spread gives each stream a random phase inside its interval, for uniform aggregate arrival instead of synchronized bursts.

Shared flags

Verbs

append

Creates fresh streams and writes to them; the run owns its streams and deletes them on exit.

Latency is issue to durable ack, per append call.

read

Ranged reads against existing streams, given as arguments (from a prior append -keep). The population run and the measurement run are separate on purpose: what happens between them (the tail settling, a rewrite, a star restart) is yours to arrange, and that is how you steer hot-tail versus cold-storage reads.

Latency is per read call.

subscribe

Tails existing streams.

Reports delivery throughput and per-record lag (assigned timestamp to receipt). Over a backlog that is catch-up age; beside a live append process it is end-to-end delivery lag.

A fourth verb, delete, tears down streams from an id list: cleanup, not a measurement.

Output

Human summary by default; -json writes exactly one JSON object to stdout (progress goes to stderr):

{
  "verb": "append",
  "config": {"streams": 4, "window": 32, "size": 262144, "rate": 0, "...": "..."},
  "duration_s": 60,
  "records": 75000, "bytes": 19660800000,
  "throughput_mibps": 312.4, "ops_per_s": 1250,
  "latency_ms": {"p50": 1.8, "p90": 2.9, "p99": 4.2, "p999": 9.1, "max": 15.0},
  "errors": {"append": 0},
  "kept_streams": ["<id>"]
}

Baseline comparison lives outside the tool: store the JSON and diff runs. novabench measures; it does not judge.

jq '{mibps: .throughput_mibps, p99: .latency_ms.p99, errs: .errors}' ceiling.json

Reading the numbers honestly

• errors > 0 means exit 1. The numbers include failed operations; do not compare them against a clean baseline.
• ops_per_s versus records. An op is one append or read call; records are what landed: times -batch for appends, up to -limit for reads.
• Closed-loop latency is queueing. At the ceiling, p99 measures the depth you chose with -window, not the server. Use -rate for latency.
• One machine is a shape, not a number. Three stars on one laptop share one NIC, one disk under MinIO, and one oxia; use it to see how throughput scales with -window or -streams, not for the production figure.
• Wire bytes are 2 to 3× client bytes. Append, read, and express fanout all cross the star's NIC; size --net-capacity and machine choice from that, not from throughput_mibps.

Errors and exit codes

Operation failures mid-run are counted, never fatal: a run that aborted on the first shed could never characterize saturation. A permanently broken producer stops only its own worker.

Where to run it

A dev star, no infrastructure

go run ./cmd/novad --dev &            # in-memory single star on :7780
go run ./cmd/novabench append -addr localhost:7780 -size 4KiB -duration 10s -warmup 2s

The --dev star keeps its object store in memory, so this exercises the full append path with nothing else running.

The compose stack

Bring up the compose stack with in-network advertise addresses and run the bench on the compose network; the host-port default routes every redirected byte through Docker's port forwarding, which is fine for a demo and wrong for a number:

NOVA_ADVERTISE_1=novad-1:7780 NOVA_ADVERTISE_2=novad-2:7780 NOVA_ADVERTISE_3=novad-3:7780 \
  docker compose -f deploy/docker/docker-compose.yml up -d --build

docker run --rm --network nova_default your/bench-image \
  append -addr novad-1:7780,novad-2:7780,novad-3:7780 \
  -streams 4 -window 32 -size 256KiB -duration 60s -json

A representative ladder, with SEEDS set to the three stars:

# 1. The append ceiling: closed loop
novabench append -addr $SEEDS -streams 4 -window 32 -size 256KiB -duration 60s -json > ceiling.json

# 2. Honest p99 at a fixed load point: 200 appends/s per stream
novabench append -addr $SEEDS -rate 200 -streams 2 -size 4KiB -duration 60s -json > p99.json

# 3. Express (on MinIO this exercises the fanout protocol, not express latency)
novabench append -addr $SEEDS -class express -streams 8 -size 64KiB -duration 60s

# 4. Populate, then read two ways
ID=$(novabench append -addr $SEEDS -keep -json -size 4KiB -duration 30s | jq -r '.kept_streams[0]')
novabench read -addr $SEEDS -pattern seq    -duration 30s $ID
novabench read -addr $SEEDS -pattern random -limit 10 -duration 30s $ID

# 5. End-to-end delivery lag: a subscriber beside a live writer
novabench append -addr $SEEDS -rate 500 -size 1KiB -duration 90s -keep -json > writer.json &
novabench subscribe -addr $SEEDS -duration 60s -json <the writer's stream id>

Watch the run on Grafana's Overview and Chain rows; nova_chain_backlog_stall_total moving during a bench names the bottleneck.

Fly

deploy/fly/bench.sh fans the bench out over N ephemeral machines inside the private network and aggregates their JSON; the seed -addr is injected. See Nova on Fly.io.

deploy/fly/bench.sh --org <your-org> --machines 3 -- append -streams 4 -size 64KiB -duration 60s -json

Kubernetes (Pulumi)

deploy/pulumi/bench.sh runs the bench as a Kubernetes Job (one fresh Job per invocation) and fans out --machines N identical pods, soft-spread across bench nodes. Give the generator its own pool first so it cannot perturb the numbers; the script exits non-zero on any failed pod:

pulumi config set benchNodeCount 1 && pulumi up
deploy/pulumi/bench.sh --machines 3 -- append -streams 8 -size 256KiB -duration 60s -json
pulumi config set benchNodeCount 0 && pulumi up      # stop paying for the pool

--skip-build reuses the last image; --cpu and --memory size the pods. Finished Jobs self-delete after 24 hours.

Source documents

• cmd/novabench/README.md
• deploy/fly/README.md, deploy/pulumi/README.md
• docs/design/000-decisions.md (the novabench entry)