Stale async reference registration after dropping a reference field via PATCH #3002

Bug Description

When a field with reference + async_reference: true is dropped from a collection via PATCH /collections/:name, the field is removed from that collection's schema, but the corresponding entry in CollectionManager::referenced_ins — and the async_referenced_ins map on the referenced collection — is never removed.

From then on, every newly indexed document in the referenced collection fails with a 400 pointing at a field that no longer exists in any schema.

The stale entry cannot be cleared through the API: it survives deleting and recreating the referencing collection, and survives deleting and recreating the referenced collection. It is written to the $REFERENCED_INS key at shutdown and reloaded verbatim, so it can also survive restarts — though some restarts clear it as a side effect (details below), which makes the failure look intermittent.

Reproduction Steps

### Run Typesense via Docker ########################################
set -x

export TYPESENSE_API_KEY=xyz
export TYPESENSE_HOST=http://localhost:8108

docker stop typesense-repro 2>/dev/null
docker rm typesense-repro 2>/dev/null
rm -rf "$(pwd)"/typesense-data-dir-repro
mkdir "$(pwd)"/typesense-data-dir-repro

docker run -d -p 8108:8108 --name typesense-repro \
  -v"$(pwd)"/typesense-data-dir-repro:/data \
  typesense/typesense:30.2 \
  --data-dir /data \
  --api-key=$TYPESENSE_API_KEY \
  --enable-cors

# Wait till typesense is ready.
until curl -s -o /dev/null -w "%{http_code}" "$TYPESENSE_HOST/health" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | grep -q "200"; do
  sleep 2
done

### 1. Referenced collection #########################################
curl -s "$TYPESENSE_HOST/collections" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '
  {
    "name": "customers",
    "fields": [
      {"name": "name", "type": "string"}
    ]
  }' | jq

### 2. Referencing collection, with an async reference ###############
curl -s "$TYPESENSE_HOST/collections" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '
  {
    "name": "invoices",
    "fields": [
      {"name": "number", "type": "string"},
      {"name": "customer_id", "type": "string", "reference": "customers.id", "async_reference": true, "optional": true}
    ]
  }' | jq

### 3. Baseline: indexing a customer works ###########################
curl -s "$TYPESENSE_HOST/collections/customers/documents" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '{"id": "c1", "name": "Acme"}' | jq

### 4. Drop the reference field from `invoices` ######################
curl -s "$TYPESENSE_HOST/collections/invoices" \
  -X PATCH \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '{"fields": [{"name": "customer_id", "drop": true}]}' | jq

# `customer_id` is gone from the schema:
curl -s "$TYPESENSE_HOST/collections/invoices" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq '.fields[].name'

### 5. Index another customer -> 400 ################################
curl -s "$TYPESENSE_HOST/collections/customers/documents" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '{"id": "c2", "name": "Globex"}' | jq

docker stop typesense-repro
docker rm typesense-repro

Expected vs Actual

Expected behavior

Step 5 succeeds, exactly as step 3 did. After step 4 no collection declares an async reference to customers, so indexing a customer should have no reference work to do.

Actual behavior

Step 4 reports the drop as successful, and the schema check prints only "number" for invoices. Step 5 then fails:

{
  "message": "Error while updating async reference field `customer_id` of collection `invoices`: Could not find field `customer_id` in the schema."
}

Every subsequent insert into customers fails the same way.

Environment

• Typesense version: v30.2 (typesense/typesense:30.2), single node, default config. Code reading suggests master (e6607fd) is also affected; not built to confirm.
• Operating system: official Linux container image, run under Docker 28.5.2 on macOS 15.6 (arm64)
• Client library & version: none — plain curl

Schema / Configuration

Referenced collection:

{
  "name": "customers",
  "fields": [
    { "name": "name", "type": "string" }
  ]
}

Referencing collection:

{
  "name": "invoices",
  "fields": [
    { "name": "number", "type": "string" },
    { "name": "customer_id", "type": "string", "reference": "customers.id", "async_reference": true, "optional": true }
  ]
}

Alter payload that triggers the bug:

{ "fields": [{ "name": "customer_id", "drop": true }] }

Additional Context

Recovery attempts

Run as one cumulative session on a single node immediately after the repro above, with no restarts in between:

Only the last one actually removes the registration, because drop_collection is the only code path that cleans it up and it needs the reference field to be present in the schema when it runs. That requires the operator to know the name of a field that may have been deleted weeks earlier, plus a full reindex of the referencing collection.

Restart behaviour is inconsistent

Restarts sometimes clear the bad state and sometimes don't, which makes this hard to diagnose in production:

• Repro above, then docker restart with no snapshot in between: still fails. The persisted $REFERENCED_INS value in the data dir does contain the stale entry, and it is reloaded verbatim.
• Repro, then POST /operations/snapshot, then restart: fixed. $REFERENCED_INS is only written in dispose(), so a snapshot taken before the first graceful shutdown contains no such key; on restore, _populate_referenced_ins re-derives a correct map from the collection metas.
• Repro, then delete/recreate both collections, then restart with no snapshot: fixed. Raft log replay re-executes the earlier DELETE /collections/invoices at a point where the reference field is still present, so drop_collection cleans up as a side effect. Visible in the startup log as E collection_manager.cpp:937] Referenced collection 'customers' not found.

Analysis

Line numbers from v30.2. An async reference is tracked in three places:

1. reference_fields / search_schema on the referencing collection — the only one visible via the API
2. CollectionManager::referenced_ins — global referenced_coll -> referencing_coll -> reference_info_t
3. Collection::async_referenced_ins on the referenced collection — what drives Index::update_async_references

The PATCH drop path only cleans up (1):

// src/collection.cpp:7280-7297 — Collection::validate_alter_payload
if(!field_it->reference.empty()) {
    reference_fields.erase(field_name);          // local map only
    if (field_it->nested) { object_reference_fields.erase(field_name); }
    rebuild_read_state_snapshot_unlocked();
    // ... drops the _sequence_id helper field
}

remove_referenced_ins does not appear anywhere in collection.cpp in v30.2. Its only caller is drop_collection, which iterates the dropped collection's current reference fields:

// src/collection_manager.cpp:925-942 — CollectionManager::drop_collection
auto reference_fields = collection->get_reference_fields();
for (const auto& item: reference_fields) {
    remove_referenced_ins(ref_coll_name, actual_coll_name);
    ref_coll->remove_referenced_in(actual_coll_name, field_name, reference_info.is_async, reference_info.field);
}

Once the field has been PATCH-dropped, deleting the collection can no longer clean up after it — the loop has nothing to iterate.

Recreating the referenced collection re-attaches the stale entry: create_collection replays referenced_ins verbatim (src/collection_manager.cpp:788-807), and Collection::add_referenced_in (src/collection.cpp:8526-8551) only validates the referenced side (customers.id, and id is exempt from the check entirely). It never verifies that the referencing collection still has the field, or that it exists at all. The load path does the same with no validation (src/collection_manager.cpp:220-232).

The error originates at src/collection.cpp:152-154 (Collection::update_async_references_with_lock), reached from Index::update_async_references. The "delete the referencing collection" workaround only appears to fix things because of this guard:

// src/index.cpp:1229-1236
auto referencing_coll = cm.get_collection(referencing_collection_name);
if (referencing_coll == nullptr) {
    // ...collections get created and indexed in parallel on server restart...
    continue;
}

The stale entry is skipped, not removed, so the failure returns as soon as the referencing collection exists again.

On master (e6607fd): the cleanup call now exists in Collection::batch_alter_data, but is gated on the field being dropped and re-added pointing at a different target:

// src/collection.cpp:7216-7240 (master)
auto it = updated_reference_fields.find(f.name);
if (it != updated_reference_fields.end() && f.reference != (it->second.collection + it->second.field)) {
    CollectionManager::get_instance().remove_referenced_ins_with_lock(name, erase_it->second);
    continue;
}
reference_fields.erase(erase_it);

For a plain drop, updated_reference_fields.erase(field_name) has already run (src/collection.cpp:7728), so the lookup misses and remove_referenced_ins_with_lock is never called. The leak looks unchanged for this case.

Related

• [Feature Request] Allow Updating Dangling Async References After Collection Recreation #2680 — dangling reference values after collection recreation. Adjacent but distinct: this is a dangling reference registration, where no schema anywhere declares the reference.