Repository navigation
Python: Support list literals in Chroma, MongoDB, Pinecone and Weaviate filters - #14558
PRABHU KIRAN VANDRANKI (VANDRANKI) wants to merge 2 commits into
Conversation
GitHub Actions
Active
GitHub Actions
Active
semantic-kernel-automation
Bot
added
the
python
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Chroma and Weaviate currently alter apostrophes in list values, causing valid membership filters to miss matching records.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds list-literal support to vector-store filter parsers and fixes Pinecone negation handling.
Changes:
- Parses list literals for Chroma, MongoDB Atlas, Pinecone, and Weaviate.
- Correctly converts Pinecone
not (... in [...])to$nin. - Adds connector-specific unit coverage.
| File | Description |
|---|---|
python/semantic_kernel/connectors/chroma.py |
Adds list parsing and updates return typing. |
python/semantic_kernel/connectors/mongodb.py |
Adds list parsing. |
python/semantic_kernel/connectors/pinecone.py |
Adds list parsing and repairs negation handling. |
python/semantic_kernel/connectors/weaviate.py |
Adds list parsing for contains_any. |
python/tests/unit/connectors/memory/test_chroma.py |
Tests $in and $nin lists. |
python/tests/unit/connectors/memory/mongodb_atlas/test_mongodb_atlas_collection.py |
Tests MongoDB list filters. |
python/tests/unit/connectors/memory/test_pinecone.py |
Tests list and negated filters. |
python/tests/unit/connectors/memory/weaviate/test_weaviate_collection.py |
Tests contains_any translation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| return value | ||
| raise VectorStoreOperationException(f"Unsupported constant type: {type(value)}") | ||
| case ast.List(): | ||
| return [self._lambda_parser(elt) for elt in node.elts] |
There was a problem hiding this comment.
Confirmed. The list branch went through the string constant branch, so x.content in ["O'Reilly"] became {"content": {"$in": ["O''Reilly"]}}.
Fixed in 89346f5. List elements that are str or bytes constants are now passed through unchanged. test_chroma_collection_build_filter_in_list_keeps_apostrophes fails on the previous head and passes now. This does not depend on #14560, which fixes the same doubling for scalar values like x.content == "O'Brien".
| return value | ||
| raise VectorStoreOperationException(f"Unsupported constant type: {type(value)}") | ||
| case ast.List(): | ||
| return [self._lambda_parser(elt) for elt in node.elts] # type: ignore |
There was a problem hiding this comment.
Confirmed. contains_any received ["O''Reilly", "b"] for x.content in ["O'Reilly", 'b'], because list elements were parsed through the string constant branch.
Fixed in 89346f5. List elements that are str or bytes constants now keep their value as written. test_weaviate_collection_build_filter_in_list_keeps_apostrophes fails on the previous head (["O''Reilly", 'b']) and passes now. The scalar case is handled separately in #14560.
…tten List elements were parsed through the string constant branch, which doubles apostrophes, so x.content in ["O'Reilly"] filtered on O''Reilly. List elements now pass string and bytes constants through unchanged.
GitHub Actions
Active
Motivation and Context
Filter lambdas that use
inornot inwith a list literal always fail on the Chroma, MongoDB Atlas, Pinecone and Weaviate collections withNotImplementedError: Unsupported AST node: <class 'ast.List'>. Their_lambda_parserimplementations already handleast.Inandast.NotIn, but none of them parse the list on the right-hand side. Qdrant already does.Pinecone also computes
$ninfornot (x in [...])withnext(operand.values()), which raisesTypeError: 'dict_values' object is not an iteratorfor everynotfilter.Fixes #14557
Description
case ast.List(): return [self._lambda_parser(elt) for elt in node.elts]to_lambda_parserinmongodb.pyandpinecone.py, the same wayqdrant.pydoes it.ast.Listcase to_lambda_parserinchroma.pyandweaviate.pytoo. There, string and bytes elements are passed through as written instead of through the scalar string branch, which doubles apostrophes (x.content in ["O'Reilly"]would otherwise filter onO''Reilly). The scalar case is fixed separately in Python: Stop doubling apostrophes in Chroma and Weaviate filter values #14560.pinecone.py, replace thenext(operand.values())calls withnext(iter(operand.items())).not (x in [...])now becomes$nin, and any othernotraises the documentedNotImplementedErrorinstead of aTypeError._lambda_parserreturn annotation to includelist[Any].TextandNumfilters reject a list on the right side, so it needs a separate decision.Tests added for each connector, including an apostrophe case for Chroma and Weaviate. They fail on
mainand pass with this change.tests/unit/dataand the Chroma, Pinecone, MongoDB, Weaviate and Redis unit tests pass, and ruff and mypy report nothing new for the changed lines.Contribution Checklist