The object cache

The object cache is a pool of caching proxies between Nova's readers and the standard bucket. It dedups the fleet's cold fetches and absorbs backfills, and it changes nothing about durability: it is never authoritative, losing it at any instant is harmless, and every read it cannot serve falls back to the direct path. Writes never touch it; durability is still the PUT.

Why it exists

Serving nodes hold no data on local disk, so every cold read is a ranged GET against object storage, and plan-driven serving turns the Gateway into a cold-read fleet of its own, fetching settled objects for backfills and SSE catch-up. Without a cache, ten consumers replaying the same stream cost ten identical GETs. The object cache gives those reads a shared, disk-backed tier: transit bundles (where multi-tenant colocation makes one warmed bundle serve many streams' reads) and settled objects, cached in pages on the instances' local disks.

Two consumers ride it: stars, whose cold reads take the cache route on the serving lane, and the Gateway, whose vended fetches route their presigned URLs through the same instances; both carry the same instance list, so both warm the same pages. The auditor's settle reads stay on its own pod-local fetch cache (a read-every-bundle-once workload would size this pool for ingest bandwidth), but the auditor's deletes do route through the proxy, so cache invalidation rides deletion. Express is out of scope entirely: only live chain slots are express-resident, and they never route here.

Never authoritative, never a stall

The house rule, sharpened: the cache may never fail a read, and, since a hung read is worse than a failed one, may never stall one. Every cache-routed request carries the ordinary direct ranged GET as its substitute:

A delete that fails through the proxy is still issued direct; a delete is never lost. Every fallback is counted by cause and every ejection logged; silent shedding lies.

Routing

There is no balancer and no shared storage; both jobs go to client-side routing, computed in one shared ring package that stars and the Gateway both use:

• Key affinity. A consistent-hash ring (128 vnodes per instance, hashing instance URLs, so the same set routes identically regardless of list order) sends each object key to the same instance from every process. That determinism is the fleet's dedup: the proxy coalesces concurrent downloads of one key only per instance, and affinity makes that per-instance coalescing global.
• Bounded load. An instance's in-flight claims are capped at ⌈--object-cache-load-factor × mean⌉ (default 1.25); a hot key's overflow spills to the next instance in ring order, deliberately replicating hot keys, trading bounded duplication for hot-key fan-out instead of saturating one NIC.
• One ring per process. Bounded load and ejection span every lane in the process, or the bound lies.

Zone-scoped pools

A single regional ring would send roughly two thirds of every consumer's cache reads across an AZ boundary and pay egress both ways. So the pool is deployed as per-zone sub-pools, and every consumer rings over its own zone's instances only: stars know their --location, the Gateway runs per-zone deployments, and no instance list ever names another zone. Cache traffic crosses no AZ boundary, by construction.

The trade is bounded: a key hot in N zones is cached N times (disk is cheap; egress is not), each zone's coalescing is its own, and a delete invalidates the deleting auditor's zone at once while other zones' copies age out on the 1-hour TTL backstop. Copies stranded by a membership change share the same staleness class, with the same bound. A zone's outage degrades that zone to direct reads: the fallback posture, verbatim.

The range outside the signature

The proxy can widen a small ranged GET to page bounds only if the byte range is outside the request signature: it holds no credentials and cannot re-sign a rewritten Range. Every cache-routed read is therefore built so Range travels unsigned: stars and auditors add the header after signing (SigV4 permits unsigned extra headers), and the Gateway's presigned URLs carry query-string auth, which never signs Range; the fetcher just rewrites the scheme and routes via the proxy.

Two consequences are the tier's recorded security posture. Cacheability requires the proxy to see plaintext, so cache-routed traffic rides plain HTTP on the cluster network. And since a warm instance serves hits without object storage re-checking the caller, a warm instance is an unauthenticated read oracle for what it holds: the object cache is cluster-internal only, never tenant-reachable, the same topological boundary that keeps novad off the internet.

The instances

An instance is a forked S3 caching proxy (pinned to an exact commit, image built in Nova's CI), run in proxy-only mode with local disk per instance and deliberately no shared storage: an instance replaced means its key share goes cold and refetches, which the fallback posture makes harmless. Global read-caching is off; a rules file allowlists exactly the Nova bucket's keys:

{
  "rules": [
    {
      "pattern": "nova/**",
      "read_cache_enabled": true,
      "page_widening": true,
      "page_size": 16777216,
      "get_ttl": "1h"
    }
  ]
}

• 16 MiB pages match the settle's born-at-target object size, so a first small read caches the whole typical object and later ranges are sliced from disk.
• get_ttl: 1h is the invalidation backstop behind deletes-through-the-proxy.
• Nothing else is cached, and nothing else is even routed: the allowlist is the second fence behind the routing decision.

The upstream proxy fails open into passthrough: misconfiguration produces no errors, only a zero hit rate. The leased behaviors (page widening on unsigned ranges, presigned-expiry checks beating warm keys, DELETE invalidating, per-instance coalescing) are each pinned by a conformance test that every version bump re-proves, and the proxy's signed-range-skip counter is the standing tripwire: it is ≈ 0 for Nova-originated traffic or widening has silently died.

Operating it

The pool is network-bound: place it on network-guaranteed nodes (a shared-NIC neighbor is the proven p99 killer), 2 to 3 instances per zone to start, disk sized to the hot working set (transit-window bundles plus the settled objects behind active backfills), never to retention, because a miss costs a fetch, not correctness.

Both consumers take the same flags, and should carry the same per-zone list:

On Kubernetes the Constellation spec's objectCache block (image, slice, dnsServers, metricsEndpoint) renders one zone-pinned pool per location and threads the per-zone lists into every consumer; see Kubernetes. The compose stack runs two instances in front of MinIO with the production allowlist posture.

Watch it from both sides. The proxy pushes its own metrics (hit ratio, S3 traffic saved, bytes served, evictions, failures, coalescing) as OTLP into Prometheus (the Nova — Object Cache dashboard reads them), and Nova's clients export nova.objectcache.requests, .fallbacks (by cause: connect, first byte, error, ejected), .spills, .ejections, and .ring.size. Two alerts matter: an instance ejected beyond a few minutes, and a hit ratio collapsed while request volume is nonzero (the fail-open detector). See observability.

Source documents

• docs/design/026-object-cache.md
• internal/storage/objectcache/: the ring
• internal/storage/objectstore/s3/: the dual-route store
• internal/gateway/cacheroute.go: the Gateway's route
• deploy/docker/objectcache/: instance config and rules