Nova on Fly.io

deploy/fly/ stands up a private Nova on Fly.io (three stars, an HA metastore, and a Tigris bucket) with two scripts: one idempotent pipeline for everything, and a fast loop that redeploys only novad. Nothing gets a public IP; clients run inside the same private network.

What deploy.sh brings up

nova-dev-oxia-coord-1..3   oxia coordinators — raft HA trio; shard assignment + leader election
nova-dev-oxia-1..3         oxia data servers — one app per server, so each keeps a stable <app>.internal name
nova-dev-novad             novad — one app, 3 machines, one small volume each
nova-dev-data              Tigris bucket (Fly-native S3), the durable tier
nova-dev-bench             novabench runners — ephemeral machines that exist only while bench.sh runs

Everything lives on the org's private 6PN network with **no public IPs and no [[services]]**. Nova's redirect protocol names specific stars, so an opaque load balancer on the data path would break it; clients must be on the network (another Fly app, or fly proxy` / WireGuard from a laptop).

The oxia layout follows the upstream bare-metal guide: three data servers and three raft coordinators, with Nova's three namespaces (nova for durable records, nova-work for the reconstructible due-queues, nova-ephemeral for leases and liveness) at replication factor 3 and notifications off; Nova subscribes to nothing.

Authentication is off (-auth defaults to off): the trust boundary is the private network. Turning it on later is novad configuration only.

Deploy

Prerequisites: the fly CLI authenticated, an org. Docker is not required locally; Fly builds remotely.

deploy/fly/deploy.sh --org <your-org> --region iad

The script re-roots itself at the repository root and is idempotent: every step checks before it creates, so re-run it after a partial failure or to roll out a new build. Deploy order matters on first run and the script encodes it: data servers → coordinators (they need the servers' DNS names resolvable to converge shard assignment) → bucket → novad. App names are fixed (nova-dev-*) because they are baked into the oxia cluster config and the coordinators' raft bootstrap list.

The fast loop

deploy/fly/deploy-novad.sh [--region <region>]

Assumes deploy.sh has run once. Machines restart one at a time against the /readyz check; drains and leadership handoff are Nova's own job. The machine's kill_timeout is 75 seconds, so the exit drain's 30-second budget and the close guard fit inside it.

Derived at runtime

The novad image carries an entrypoint that assembles node-specific flags from the machine environment:

• Advertise address: <machine-id>.vm.nova-dev-novad.internal:7780, stable for a machine's lifetime. A machine replacement mints a new one, and the star re-announces itself in the registry.
• Location: the machine ID, so every star is its own placement zone and leadership spreads across all three machines.
• oxia seed: the entrypoint probes nova-dev-oxia-{1..3} and dials the first that answers; one oxia server down never blocks a novad boot. Nova's oxia client discovers the shard layout from whichever server answers.
• S3: fly storage create injects AWS_* secrets and BUCKET_NAME; the entrypoint maps bucket, endpoint, and region onto NOVAD_S3_*, and the credentials flow through the SDK default chain.

The fly.toml pins the gRPC flow-control windows, names the three oxia namespaces, mounts the identity volume at /data, and declares the /readyz check on port 9090 (15-second interval, 30-second grace).

Verify

fly checks list -a nova-dev-novad        # 3 machines passing /readyz
fly logs -a nova-dev-novad               # "novad serving" per star, then the star summary every minute

# From inside the network:
fly ssh console -a nova-dev-novad -C 'nova -addr localhost:7780 fleet'

Metrics

Each fly.toml declares a [metrics] section, so Fly's managed Prometheus scrapes every machine automatically: novad on 9090, oxia (both roles) on 8080. Explore them in the org's managed Grafana at https://fly-metrics.net, or query the org datasource at https://api.fly.io/prometheus/<org-slug> with a Fly API token. The repo's dashboards/ JSON imports there against that datasource; see observability for what the rows show.

Benchmarking

bench.sh drives novabench from inside the private network, fanned out over N ephemeral machines. Each run builds and pushes the bench image, replaces whatever a previous run left in nova-dev-bench, launches N machines that each run one novabench process, waits for them to exit, prints per-machine results plus an aggregate (with -json), and destroys the machines:

deploy/fly/bench.sh --org <your-org> --machines 3 -- \
  append -streams 4 -size 64KiB -duration 60s -json

Everything after -- is a novabench command line (see benchmarking); the seed -addr is injected as nova-dev-novad.internal:7780, which resolves to all three machines; any star redirects to the right leader. The script needs jq.

Operations

• Roll a new build: deploy-novad.sh (the full deploy.sh works too but re-walks oxia first).
• Change oxia namespaces or servers: edit deploy/fly/oxia/cluster.yaml and redeploy the three coordinator apps; data servers do not read it.
• Grow a volume: fly volumes extend.
• Scale out: create another nova_data volume and fly scale count 4 -a nova-dev-novad. A new star registers and starts taking placements within one stats tick; see node lifecycle.

Tear down

for app in nova-dev-novad nova-dev-bench nova-dev-oxia-{1,2,3} nova-dev-oxia-coord-{1,2,3}; do
  fly apps destroy "$app" -y
done
fly storage destroy nova-dev-data     # the bucket and its data

Production notes

A proxy in front of the data bridge. Enabling -http-listen (NOVAD_HTTP_LISTEN) serves the HTTP data plane on a second in-cluster listener with the wire's auth mode and TLS identity. Two obligations bind anything in front of it (or in front of the Gateway):

1. Escaped slashes pass through unchanged. Stream keys travel as one percent-encoded path segment (orders%2Feu); a proxy that decodes or rejects %2F breaks every slashed key. Envoy: path_with_escaped_slashes_action: KEEP_UNCHANGED; nginx: a bare proxy_pass http://upstream; forwards the raw path, and normalize-and-merge features stay off for /v1/streams/.
2. Idle timeouts outlive the SSE heartbeat. Subscriptions are long-lived and quiet between commits; the bridge writes a comment heartbeat every 15 seconds. Set proxy idle and read timeouts comfortably above it, and keep response buffering off for text/event-stream.

Placement spread is single-region here. Location = machine ID spreads leadership across machines, all in one Fly region. For real fault domains, pin each oxia data server and novad volume to a different region (--region per app, and derive the location from FLY_REGION), and revisit replicationFactor and latency accordingly.

Sizing. novad runs performance-8x/16gb, oxia shared-cpu-2x/2gb. A star holds no data on local disk (its write path goes straight to object storage and cold reads are ranged GETs), so CPU and network size the star, not volume bandwidth. The volume-bandwidth concern moves to oxia: Fly ties volume bandwidth to the machine tier (shared-cpu machines cap at 16 to 32 MiB/s and the block rate limiter reads as multi-second fsyncs; performance machines get 48 to 128 MiB/s). Bump the oxia data servers before benchmarking seriously: metadata commit latency rides their volumes, the one fsync-bound path left. The WAL and data share one volume on the oxia servers; upstream recommends separate disks.

Backups. The nova namespace is forward-only, never snapshot-restored; disaster recovery is reconstruct-from-S3. Tigris holds the durable tier; consider its replication and versioning options for the bucket.

Auth. Enable -auth and TLS before any semi-trusted network touches these apps; see authentication.

Source documents

• deploy/fly/README.md
• deploy/fly/deploy.sh, deploy/fly/deploy-novad.sh, deploy/fly/bench.sh
• deploy/fly/novad/fly.toml, deploy/fly/novad/entrypoint.sh