Bug Description
It looks like delete by query doesn't play nice if the field targeted by the delete query is not configured as stored (store).
For stored fields, delete by query works exactly as expected.
For non-stored fields, the documents are indeed deleted from the index as expected, but they somehow still count towards the search result hit count and the facet counts.
In the repro script below, the failing test case produces a result like this:
{
"facet_counts": [{
"counts": [{
"count": 2,
"highlighted": "facet1",
"value": "facet1"
}
],
"field_name": "text",
"sampled": false,
"stats": {
"total_values": 1
}
}
],
"found": 2,
"hits": [],
"out_of": 2,
"page": 1,
"request_params": {
"collection_name": "testindex2",
"first_q": "*",
"per_page": 10,
"q": "*"
},
"search_cutoff": false,
"search_time_ms": 1
}...after all documents with the "facet1" value have been deleted by query.
Reproduction Steps
set -x
export TYPESENSE_HOST=http://localhost:8108
export TYPESENSE_API_KEY=xyz
docker stop typesense-repro 2>/dev/null
docker rm typesense-repro 2>/dev/null
rm -rf "$(pwd)"/typesense-data-dir-repro
mkdir "$(pwd)"/typesense-data-dir-repro
# Wait for Typesense to be ready
docker run -d -p 8108:8108 --name typesense-repro \
-v"$(pwd)"/typesense-data-dir-repro:/data \
typesense/typesense:29.0 \
--data-dir /data \
--api-key=$TYPESENSE_API_KEY \
--enable-cors
# Wait till typesense is ready.
until curl -s -o /dev/null -w "%{http_code}" "${TYPESENSE_HOST}/health" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | grep -q "200"; do
sleep 2
done
curl -s "${TYPESENSE_HOST}/debug" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}"
# 1. create two identical indexes here - only difference is that the first of them stores the field, while the second does not.
curl -s "${TYPESENSE_HOST}/collections" \
-X POST \
-H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-d '{
"name": "testindex1",
"fields": [
{"name": "text", "type": "string", "facet": true, "store": true }
],
"default_sorting_field": ""
}'
curl -s "${TYPESENSE_HOST}/collections" \
-X POST \
-H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-d '{
"name": "testindex2",
"fields": [
{"name": "text", "type": "string", "facet": true, "store": false }
],
"default_sorting_field": ""
}'
# 2. add identical documents to the two indexes
curl -s "${TYPESENSE_HOST}/collections/testindex1/documents/import?action=create" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-H "Content-Type: text/plain" \
-X POST \
-d '{"id": "1","text": "facet1"}
{"id": "2","text": "facet2"}
{"id": "3","text": "facet1"}
{"id": "4","text": "facet2"}'
curl -s "${TYPESENSE_HOST}/collections/testindex2/documents/import?action=create" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-H "Content-Type: text/plain" \
-X POST \
-d '{"id": "1","text": "facet1"}
{"id": "2","text": "facet2"}
{"id": "3","text": "facet1"}
{"id": "4","text": "facet2"}'
# 3. delete documents with "facet1"
curl -s "${TYPESENSE_HOST}/collections/testindex1/documents/?filter_by=text:=facet1" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-X DELETE
curl -s "${TYPESENSE_HOST}/collections/testindex2/documents/?filter_by=text:=facet1" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-X DELETE
##################
### TEST CASES ###
##################
# ✔️ filter documents in the first index by "facet1"
# => yields no facets and result count = 0
# ==> works as expected
curl -g "${TYPESENSE_HOST}/collections/testindex1/documents/search?q=*&filter_by=text:=facet1&facet_by=text" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}"
# ❌ filter documents in the second index by "facet1"
# => yields facet count = 2 for "facet1" and result count = 2, although no documents are returned
# ==> expected the same result as above
curl -g "${TYPESENSE_HOST}/collections/testindex2/documents/search?q=*&filter_by=text:=facet1&facet_by=text" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}"
docker stop typesense-repro
docker rm typesense-reproExpected vs Actual
Expected behavior
Deleted documents should not count against the search result hit count, nor the facet counts, no matter the field configuration.
Actual behavior
Deleted documents count against the search result hit count and the facet counts, when fields are not stored.
Environment
- Typesense version:
v29.0
- Operating system:
Windows 11
Schema / Configuration
Include any relevant schema definitions, JSON payloads, or config files:
{
"name": "testindex2",
"fields": [{"name": "text", "type": "string", "facet": true, "store": false}],
"default_sorting_field": ""
}
Bug Description
It looks like delete by query doesn't play nice if the field targeted by the delete query is not configured as stored (
store).For stored fields, delete by query works exactly as expected.
For non-stored fields, the documents are indeed deleted from the index as expected, but they somehow still count towards the search result hit count and the facet counts.
In the repro script below, the failing test case produces a result like this:
{ "facet_counts": [{ "counts": [{ "count": 2, "highlighted": "facet1", "value": "facet1" } ], "field_name": "text", "sampled": false, "stats": { "total_values": 1 } } ], "found": 2, "hits": [], "out_of": 2, "page": 1, "request_params": { "collection_name": "testindex2", "first_q": "*", "per_page": 10, "q": "*" }, "search_cutoff": false, "search_time_ms": 1 }...after all documents with the "facet1" value have been deleted by query.
Reproduction Steps
Expected vs Actual
Expected behavior
Deleted documents should not count against the search result hit count, nor the facet counts, no matter the field configuration.
Actual behavior
Deleted documents count against the search result hit count and the facet counts, when fields are not stored.
Environment
v29.0Windows 11Schema / Configuration
Include any relevant schema definitions, JSON payloads, or config files:
{ "name": "testindex2", "fields": [{"name": "text", "type": "string", "facet": true, "store": false}], "default_sorting_field": "" }