What happens
VectorSearchOptions documents that "When multiple filters are used, they are combined with an AND operator" (python/semantic_kernel/data/vector.py, class VectorSearchOptions). InMemoryCollection combines a list of filters with OR instead.
Where
python/semantic_kernel/connectors/in_memory.py, InMemoryCollection._get_filtered_records:
filtered_records: dict[TKey, AttributeDict] = {}
for key, record in self.inner_storage.items():
for filter in callable_filters:
if self._run_filter(filter, record):
filtered_records[key] = record
A record is kept as soon as any one filter matches. Every other connector (Postgres " AND ", Chroma $and, Mongo $and, Qdrant must, Weaviate all_of, Redis &, Azure AI Search " and ") uses AND.
Repro (checked on current main, 9974625)
import asyncio
from dataclasses import dataclass
from typing import Annotated
from semantic_kernel.connectors.in_memory import InMemoryCollection
from semantic_kernel.data.vector import VectorStoreField, vectorstoremodel
@vectorstoremodel
@dataclass
class R:
id: Annotated[str, VectorStoreField("key")]
color: Annotated[str, VectorStoreField("data", is_indexed=True)]
size: Annotated[int, VectorStoreField("data", is_indexed=True)]
vec: Annotated[list[float] | None, VectorStoreField("vector", dimensions=2)] = None
async def main():
c = InMemoryCollection(record_type=R, collection_name="x")
await c.upsert([R("1", "red", 1, [1, 0]), R("2", "red", 9, [1, 0]),
R("3", "blue", 1, [1, 0]), R("4", "blue", 9, [1, 0])])
r = await c.search(vector=[1, 0], top=10,
filter=["lambda x: x.color == 'red'", "lambda x: x.size > 5"])
print(sorted([x.record.id async for x in r.results]))
asyncio.run(main())
Expected: ['2'] (red AND size > 5).
Actual: ['1', '2', '4'] (red OR size > 5).
The same happens when the filters are lambdas instead of strings.
Impact
Any multi-filter search against the in-memory store returns records that violate part of the filter. Code that works on a real store returns extra results here, and tests written against the in-memory store give wrong results.
I have a fix ready (all(...) over the filters) with tests, and will open the PR right away.
What happens
VectorSearchOptionsdocuments that "When multiple filters are used, they are combined with an AND operator" (python/semantic_kernel/data/vector.py, classVectorSearchOptions).InMemoryCollectioncombines a list of filters with OR instead.Where
python/semantic_kernel/connectors/in_memory.py,InMemoryCollection._get_filtered_records:A record is kept as soon as any one filter matches. Every other connector (Postgres
" AND ", Chroma$and, Mongo$and, Qdrantmust, Weaviateall_of, Redis&, Azure AI Search" and ") uses AND.Repro (checked on current main, 9974625)
Expected:
['2'](red AND size > 5).Actual:
['1', '2', '4'](red OR size > 5).The same happens when the filters are lambdas instead of strings.
Impact
Any multi-filter search against the in-memory store returns records that violate part of the filter. Code that works on a real store returns extra results here, and tests written against the in-memory store give wrong results.
I have a fix ready (
all(...)over the filters) with tests, and will open the PR right away.