The local compose stack

The compose stack under deploy/docker/ is the smallest complete Nova: three stars in three locations, the metastore, an S3 endpoint, the Gateway, the object cache tier, and a wired-up Grafana. Use it to try the API, drive load, and watch the dashboards react.

What runs

Every novad container runs as the non-root nova user (uid 10001) with all capabilities dropped, and gets a 75-second stop grace: the exit drain's 30-second budget, the close guard, and margin. Docker's default 10 seconds would SIGKILL a star mid-close.

Bring it up

docker compose -f deploy/docker/docker-compose.yml up -d --build

The first run builds the novad image (Go 1.26) and pulls the rest. Then:

Host ports are offset (3001, 9091, 9201) so this stack coexists with the integration-test stack. Metrics only populate under traffic: histograms stay empty until you drive load; the node gauges and runtime metrics show at once.

Driving load

The cluster spreads leadership across stars and redirects a client to a stream's leader using that star's advertised address. Each star advertises host.docker.internal:<its published port> by default, a name that resolves on the host (Docker Desktop writes it into /etc/hosts) and inside the compose network (the host-gateway entry), so a host-side client follows redirects without configuration:

seq 1 100000 | ./nova append nova://demo/orders/eu --addr localhost:7781

If host.docker.internal does not resolve on your host (some Linux setups), add 127.0.0.1 host.docker.internal to /etc/hosts; the client fails fast naming the unresolvable address rather than burning its retry budget.

That default routes every redirected byte through Docker's port forwarding: fine for demos, wrong for measuring. For benchmarking, advertise the in-network names and run the load client on the compose network:

NOVA_ADVERTISE_1=novad-1:7780 NOVA_ADVERTISE_2=novad-2:7780 \
NOVA_ADVERTISE_3=novad-3:7780 \
  docker compose -f deploy/docker/docker-compose.yml up -d --build

# A one-off shell on the network; dial novad-1:7780 and friends from inside.
docker compose -f deploy/docker/docker-compose.yml run --rm --entrypoint sh novad-1

# Or attach your own bench image.
docker run --rm --network nova_default your/bench -targets novad-1:7780,novad-2:7780,novad-3:7780

Watch the load land across stars on the Grafana Overview row (leadership spread, append throughput) and the write-path latencies as you push harder. See benchmarking for the novabench verbs.

Express on MinIO

Every star runs with Express enabled: -s3-express-buckets names three MinIO buckets standing in for one express-class bucket per location. MinIO has no S3 Express, so this exercises the fanout protocol (quorum publish, zone-preferring reads, fan-all reaps), not express latency. Drive it with novabench append -class express ... from inside the network.

Against AWS, point each pair at an S3 Express One Zone directory bucket and label zones by AZ id, the stable identifier the bucket name embeds: -s3-express-buckets use1-az4=nova--use1-az4--x-s3,... with --location use1-az4 on the star. novad refuses a zone label that contradicts the bucket's embedded AZ id, and refuses -s3-path-style with directory buckets. Leave -s3-endpoint empty on AWS so the SDK routes each bucket to its zonal endpoint.

The data bridge

Each star also serves the data plane re-encoded for HTTP (-http-listen, published as 7791 through 7793) for trusted callers that cannot speak gRPC. Same gates and auth mode as the wire; a foreign-led stream is proxied one hop to its holder, so any star answers any stream:

curl -X POST localhost:7791/v1/buckets/demo
curl -X POST localhost:7791/v1/streams/demo/orders%2Feu -d '{"class":"standard"}'
curl -X POST localhost:7791/v1/streams/demo/orders%2Feu/records \
     -d '{"records":[{"body":"one"},{"body":"two"}]}'
curl localhost:7791/v1/streams/demo/orders%2Feu/records/tail
curl -N 'localhost:7791/v1/streams/demo/orders%2Feu/records/subscribe?seq_num=0'

The key is one percent-encoded path segment (orders/eu → orders%2Feu); quote the URL or the shell eats the escape. Anything you place in front of the bridge must pass escaped slashes through unchanged and keep idle timeouts above the 15-second SSE heartbeat. The Gateway at localhost:8080 serves the same surface for the whole fleet with holder redirects absorbed; see the HTTP API guide.

The object cache

Both the stars and the Gateway carry the same -object-cache-instances list, so their ring picks agree and both warm the same instances; every failure falls back to the direct client. Watch nova_objectcache_requests_total and nova_objectcache_fallbacks_total on any star's or the Gateway's /metrics. The blunt proof that vended reads ride the cache: settle a stream, read it once, docker compose stop minio, read again (the page still serves; nova_gateway_vended_bytes climbs), then start minio.

The audit-split topology

By default each star runs its embedded auditor. To move reclamation onto its own container (the production shape when settle and reclaim traffic must not contend with ingest), flip both halves with one command:

NOVA_NO_AUDIT=true docker compose --profile audit-split \
  -f deploy/docker/docker-compose.yml up -d --build

NOVA_NO_AUDIT=true puts -no-audit on the three stars; the profile adds novad-audit, whose metrics land on localhost:9194 and in the Grafana auditor panels. Duty routing is rendezvous over the auditor ring, so with the stars out of the ring every duty lands on the audit container. Run the profile without NOVA_NO_AUDIT to A/B them: stars and the audit container then share the ring and split the 64 duty shards. See the auditor for what those duties are.

Teardown and the wipe rule

docker compose -f deploy/docker/docker-compose.yml down -v

down -v drops the oxia and MinIO volumes together. While Nova is pre-release this is also the format-change procedure: crossing a stored-format boundary requires a wipe, and the wipe must take both planes as one. Never wipe one alone: metadata describing absent objects misparses, and objects without metadata are garbage nothing reclaims. Failure modes has the same rule for Kubernetes.

Troubleshooting

oxia exits with SIGBUS: bus error under write load. The Docker Desktop VM's disk is full. oxia memory-maps its WAL, and a write into a page the filesystem can no longer materialize arrives as SIGBUS: oxia starts fine, then dies at the first sustained burst, and every metadata call on the stars times out. Check and reclaim:

docker run --rm alpine df -h /        # the VM's disk, not the host's
docker builder prune -f               # build cache
docker compose -f deploy/docker/docker-compose.yml down -v   # bench data

Budget for it: a sustained bench's payload lands in MinIO (chain slots plus the settled objects the auditor produces coexist until reclamation catches up), with oxia's WAL behind the metadata. A two-minute run at ~30 MiB/s is ~3 GB of client bytes and several GB in MinIO before reclamation. down -v between heavy runs, or grow the VM disk.

novad crash-loops with a permission error on /data after an image rebuild. Volumes created by an older root image are still owned by root. Recreate them (down -v) or re-own them in place:

for n in 1 2 3; do
  docker run --rm -v nova_novad-$n-data:/data alpine chown -R 10001:10001 /data
done

endpoint with name nova-X already exists in network nova_default. A container died uncleanly and its network endpoint leaked:

docker network disconnect -f nova_default nova-X
docker compose -f deploy/docker/docker-compose.yml up -d --wait

A star looks stuck (appends stall, zero errors). Appends that stall without erroring are backpressure: the backlog budget is full because the absorb or the object store is stalled. The goroutine dump names the blocked stage:

curl -s localhost:9191/debug/pprof/goroutine?debug=1 | less
go tool pprof "localhost:9191/debug/pprof/profile?seconds=10"

Every metadata call times out but oxia looks healthy. Something else is squatting the host ports: a host-run novad or the integration-test stack fighting over 7781 through 7783.

Source documents

• deploy/docker/README.md
• deploy/docker/docker-compose.yml
• deploy/docker/Dockerfile