v3 engine: default InMemory object store with a persistent log_directory loses all metadata on restart, then refuses envoys with depot.corrupt ("record is lost") #5932

Summary

On a single self-hosted engine (rivetdev/engine:3.0.0-preview.2), we set storage.log_directory to a Docker volume and left object_storage unset. The primary object store then defaults to InMemory.

The depot logs survive a restart; everything in the primary store does not: tablet manifests, tablet SlateDB cold stores (volume records), topology and leases. After any restart, the engine:

• comes back with manifest_version=3;
• forgets namespaces, pools and actors;
• for every volume whose data had already been offloaded to the (now empty) cold store, logs a volume has no stored record but a replica reports a cold position above zero; the record is lost…;
• refuses every envoy connect with depot.corrupt: The volume's stored data failed a consistency check. from then on.

Another restart does not help, because the on-disk replicas keep reporting cold > 0 against a cold store that no longer exists.

The engine gives no warning at startup, and nothing documents that object_storage is required for durability. A persistent log_directory makes the install look durable. This combination (durable log, volatile cold store) turns every restart into data loss, and on a busy engine into a permanent depot.corrupt.

We first read the split-time warnings below as the cause. They are noise: the engine re-splits from one tablet to 2 × cores tablets after each restart, because the manifests are gone.

Versions

• Engine: rivetdev/engine:3.0.0-preview.2 (starting rivet version=3.0.0). v3.0.0-preview.3 and 3.x have no changes under engine/packages/depot-*.
• Single node, 32 cores (so desired_tablets=64), Linux, Docker. Disk was 30% full.
• Environment:

  RIVET__STORAGE__LOG_DIRECTORY=/data/storage   # a Docker volume
  RIVET__STORAGE__AUTO_PROVISION_PARTITIONS=1
  RIVET__TELEMETRY__ENABLED=false
  RIVET__AUTH__ADMIN_TOKEN=…
  # no RIVET__OBJECT_STORAGE__*
  

Root cause

• ObjectStoreBackend has #[default] InMemory (config/src/config/object_storage.rs#L57-L60). create_object_stores builds InMemory::new() for it (engine-context/src/production/object_storage.rs#L135). Nothing checks this against a persistent storage.log_directory.
• The tablet cold store (SlateDB, holding FsVolumeRecord rows), the tablet manifest and topology all use that primary store. The depot-log shards (/data/storage/shards) are on disk. /data/storage/cold-cache is only SlateDB's disk cache: the volume we kept has cold-cache/ and shards/ and no manifests.
• After a volume's first record is flushed to the cold store (write_to_cold_after_ms), its log is trimmed and the replica's cold position rises above 0. After a restart, record_lost (tablet_maintainer_machine/state/filesystem.rs#L523-L539) sees no stored record and cold.volume_seq > 0, and quarantines the incarnation. depot-client maps VolumeRecordMissing to FsReadError::Corrupt / depot.corrupt, which is permanent. Every later boot reaches the same verdict.
• The restart in our case came from the designed fail-stop under disk pressure: log shard failed; restart required reason="write to disk timed out" (write_to_disk_timeout_ms = 30 s), then state machine requested fail-stop; exiting process. Before that, the log had hours of failed to write index … filesystem_overloaded. A plain docker restart does the same thing (see reproduction).

Evidence, from /depot/repair/v1/volume/inspect on a copy of the data volume: recordMissing: true, stored: null, the replica InSync with cold = appended = {volumeSeq: 20}, and placement.agree: true (manifest and maintainer agree, so this is not a stale grant). On every boot the manifest starts again at manifest_version=3.

Log excerpt (first incident, 2026-10-28)

22:49:22.839 level=error message="log shard failed; restart required" reason="write to disk timed out"
22:49:22.839 level=error message="state machine requested fail-stop; exiting process" machine=log_shard_machine seq=96731
22:49:38.459 level=info  message="starting rivet" version=3.0.0
22:49:39.447 level=info  message="tablet grant written" tablet_id=ed6733f2-… from=None grant_epoch=1 manifest_version=3
22:49:39.670 level=warn  message="a volume has no stored record but a replica reports a cold position above zero; the record is lost, so the incarnation stays unregistered and its name is refused until the incarnation is deleted" key="VolumeLogKey(... partition_key: b\"\\x15\\x1c\\x02default\\0\" ...)" observed="Some(VolumeLogPosition { volume_seq: 4989, client_epoch: 55834574849 })"
22:49:42.143 level=error message="a volume holds a claim epoch of a newer grant; refusing new claims" grant_epoch=8 min=55834574929
22:49:59.599 level=warn  message="no member holds the split parent's maintainer log; its ops are not imported" ...
22:49:59.601 level=warn  message="unknown filter policy '_bf:p=depot-fs-formal-row-v2' in SST, skipping" ... step="CloneLeft { manifest_id: 9 }"
22:50:xx     ... every envoy connect: depot.corrupt: The volume's stored data failed a consistency check.

64 incarnations were quarantined, including the engine's own volumes for the namespace and pools (\x02default\0\x02workspace\0…). A later restart, and a fresh engine on a copy of the volume, quarantine the same 64.

Reproduction (about 3 minutes)

1. Start a fresh engine with a persistent log directory and no object storage. WRITE_TO_COLD_AFTER_MS only speeds up the offload:

   docker network create repro-net
   docker run -d --name repro-engine --network repro-net --network-alias rivet-engine \
     -e RIVET__AUTH__ADMIN_TOKEN=$TOKEN -e RIVET__STORAGE__LOG_DIRECTORY=/data/storage \
     -e RIVET__STORAGE__AUTO_PROVISION_PARTITIONS=1 -e RIVET__STORAGE__WRITE_TO_COLD_AFTER_MS=5000 \
     -v repro-engine-data:/data/storage rivetdev/engine:3.0.0-preview.2
   curl -X PUT -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
     "http://rivet-engine:6420/pools/repro?namespace=default" -d '{}'

2. Run a RivetKit worker with one actor: state: { n: 0 }, and a bump action that does c.state.n += 1. Call bump 20 times on each of 8 actors (getOrCreate(["repro-<i>"])). Every actor reports n=20.

3. Wait about 2 minutes, then docker restart repro-engine. This is a graceful stop.

4. Observe:

   • the engine logs manifest_version=3 again;
   • envoys are refused with not_found: The node pool does not exist., and GET /namespaces shows default re-created at the restart time;
   • after recreating the pool, the same actors return n=0: their state is silently gone.

   On a busier engine, where volumes (including the engine's own metadata volumes) had cold positions above 0 before the restart, step 4 becomes the permanent depot.corrupt described above.

With RIVET__OBJECT_STORAGE__PRIMARY__FILE_SYSTEM__PATH=/data/storage/objects added, the same steps keep the manifest (no reset to 3) and all 8 actors read back n=20 after the restart. Our production engine uses S3 and comes back intact after restarts.

Suggestions

• Refuse to start, or at least log an error, when storage.log_directory is persistent and object_storage.primary is InMemory. Alternatively, default the primary store to FileSystem under log_directory when one is set.
• Document that object_storage is required for a durable self-hosted v3 engine.
• When a record is missing on a tablet whose manifest was just bootstrapped (manifest_version starting from scratch while replicas exist on disk), say so in the warning and in depot.corrupt. "failed a consistency check" sends people looking for disk corruption.

We still have the engine log (~15.8k lines) and the corrupt data volume (~154 MB) and can share them privately on request. We are not attaching them here because they may contain user data.