Skip to content

Commit 352315c

Browse files
caseyclementsdevin-ai-integration[bot]ntkathole
committed
feat: Add OnlineStore for MongoDB (feast-dev#6025)
* Initial commit on INTPYTHON-297-MongoDB-Feast-Integration Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Added mongodb to project.optional-dependencies in pyproject.toml. Now pymongo is found as extra Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Checkpoint. Passing tests.unit.online_store.test_online_writes.TestOnlineWrites with default args of MongoDBOnlineStoreConfig. Lots to do. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Handle Nan in dfs for test_online_writes.py. Now all tests in the module pass Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Removed suffix of implementation: mongodb_openai -> mongodb Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Moved MongoDBOnlineStore to feast.infra.online_store.contrib Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Formatting Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Refactor online_read that converts bson to proto. Left two methods for now. simply and transformming Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Remove file added early during discovery Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Formatting Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Added version of test that uses testcontainers mongodb instead of assuming one is running Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Create Make target for universal tests Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Cleanup Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Removed temporary integration tests requiring one to spin up own mongodb server. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Format Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Typing Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Implemented ASync API and Tests Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Removed offline store stubs. The first PR will only contain the OnlineStore Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Moved mongodb_online_store out of cobtrib package. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Add documentation. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Cleanups and docstrings Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Fixed another reference to contrib dir Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Typos Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Made _convert_raw_docs_to_proto staticmethods private Signed-off-by: Casey Clements <casey.clements@mongodb.com> * After benchmarking two alogithm for conevrting read results from bson to proto, removed the naive one. It was outcompeted 3X across dimensions Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Add extra unit tests Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Formatting Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Fixes in pyproject.toml Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Fixed Detect secrets false positives. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Update sdk/python/feast/infra/online_stores/mongodb_online_store/mongodb.py Set _client and _collection to None upon synchronous teardown. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Fix CI: guard pymongo imports and skip test module when pymongo unavailable In the unit-test-python CI job, pymongo is not installed (it is an optional extra). This caused ModuleNotFoundError during pytest collection. Two changes: 1. mongodb.py: wrap pymongo imports in try/except -> FeastExtrasDependencyImportError (consistent with redis.py, dynamodb.py, datastore.py pattern) 2. test_mongodb_online_retrieval.py: add pytest.importorskip(\pymongo\) so the entire test module is skipped gracefully when pymongo is absent Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Fix: return (None, None) when entity doc exists but feature view was never written MongoDB stores all feature views for an entity in a single document. If FV 'driver_stats' is written, an entity doc exists for driver_1. A subsequent read for FV 'pricing' (never written) was previously returning (None, {all_features: ValueProto()}) - a truthy feature dict with all-empty protos - instead of the correct (None, None) sentinel. Feast and downstream callers use (None, None) to signal entity absence. A non-None feature dict means 'entity found, values are null', which causes different behaviour in the feature retrieval pipeline. Fix: after confirming the entity doc exists, also check that the feature view key is present in doc['features']. If absent, return (None, None) rather than a dict of empty protos. Adds test_convert_raw_docs_entity_exists_but_fv_not_written to cover the multi-feature-view colocation scenario identified in code review. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Update pixi.lock after adding mongodb optional dependency to pyproject.toml Any change to pyproject.toml invalidates the pixi.lock manifest hash, causing 'pixi install --locked' to fail in CI even when the changed section (mongodb optional extras) is not used by any pixi environment. Regenerated with: pixi install (v0.63.2) Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: catch FeastExtrasDependencyImportError in doctest runner Stores like mongodb, redis, and dynamodb raise FeastExtrasDependencyImportError at import time when their optional Python extras are not installed. test_all.py only caught ModuleNotFoundError, so any such import caused the entire test_docstrings() function to abort rather than gracefully skipping the module. Extend the except clause to also catch FeastExtrasDependencyImportError so the doctest run completes for all other modules when an optional extra is absent in the test environment. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: update PYTEST_PLUGINS path for mongodb online store tests Directory was renamed from tests/integration/feature_repos to tests/universal/feature_repos in upstream/master. Update the PYTEST_PLUGINS module path in test-python-universal-mongodb-online to match the new location. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: broaden import exception handling in doctest runner to catch TypeError and other third-party import errors qdrant_client raises TypeError at import time when a gRPC EnumTypeWrapper is used with the | union operator on Python < 3.12. The previous fix only caught ModuleNotFoundError and FeastExtrasDependencyImportError, leaving the test runner vulnerable to any other exception raised by third-party libraries during pkgutil.walk_packages. Changes: - Catch bare Exception in the import try/except block so any import-time error from an optional dependency causes a graceful skip rather than an abort of the entire test run. - Initialize temp_module = None before the try block and add a continue guard so that a failed import never leaves a stale module reference to be used in the subsequent doctest execution block. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: pass onerror to pkgutil.walk_packages to suppress non-ImportError package import failures FeastExtrasDependencyImportError inherits from FeastError -> Exception, not from ImportError. pkgutil.walk_packages calls __import__ internally when recursing into sub-packages, and only silently swallows ImportError subclasses; any other exception is re-raised, crashing the entire test run. Passing onerror=lambda _: None tells walk_packages to skip any package that fails to import during the walk phase, regardless of the exception type. The inner importlib.import_module try/except already handles the same errors for the explicit import step used to collect doctests. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: update stale tests.integration.feature_repos imports to tests.universal.feature_repos The upstream directory tests/integration/feature_repos was renamed to tests/universal/feature_repos. Three files still referenced the old path: - mongodb_repo_configuration.py: IntegrationTestRepoConfig and MongoDBOnlineStoreCreator imports - tests/.../online_store/mongodb.py: OnlineStoreCreator import - tests/unit/online_store/test_mongodb_online_retrieval.py: TAGS import Updating all three unblocks test collection and allows make test-python-universal-mongodb-online to run locally. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * feat: add mongodb to ValidOnlineStoreDBStorePersistenceTypes in feast-operator The Python ONLINE_STORE_CLASS_FOR_TYPE now includes 'mongodb', so the operator's parity check (test-datasources) fails unless the Go API also lists it. Add 'mongodb' to ValidOnlineStoreDBStorePersistenceTypes in both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * feat: add Feast driver metadata to MongoDB client instantiations Add DriverInfo with the Feast name and version to both the sync MongoClient and async AsyncMongoClient so that MongoDB can identify traffic originating from a Feast AI integration. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * docs: update MongoDB online store status from alpha to preview Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: add mongodb to kubebuilder Enum annotations for OnlineStoreDBStorePersistence The +kubebuilder:validation:Enum annotation on the Type field of OnlineStoreDBStorePersistence was not updated when mongodb was added to ValidOnlineStoreDBStorePersistenceTypes. This annotation drives CRD OpenAPI schema validation at Kubernetes admission time, so any FeatureStore CR specifying type: mongodb would be rejected by the API server. Updated both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Update +GOLANGCI_LINT_VERSION to fix upstream issue golang/go#74462 Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: raise ValueError in _get_client_async for invalid config type, consistent with sync counterpart Signed-off-by: Casey Clements <casey.clements@mongodb.com> * fix: Added mongodb to operator yamls Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * Small change suggested by ntkathole Signed-off-by: Casey Clements <casey.clements@mongodb.com> * Factor out write logic into utility function making sync/async essentially identical. Signed-off-by: Casey Clements <casey.clements@mongodb.com> --------- Signed-off-by: Casey Clements <casey.clements@mongodb.com> Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ntkathole <nikhilkathole2683@gmail.com>
1 parent 154dc18 commit 352315c

25 files changed

Lines changed: 8296 additions & 89 deletions

File tree

‎.secrets.baseline‎

Lines changed: 78 additions & 78 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Makefile‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -535,6 +535,25 @@ test-python-universal-elasticsearch-online: ## Run Python Elasticsearch online s
535535
not test_snowflake" \
536536
sdk/python/tests
537537

538+
test-python-universal-mongodb-online: ## Run Python MongoDB online store integration tests
539+
PYTHONPATH='.' \
540+
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.mongodb_online_store.mongodb_repo_configuration \
541+
PYTEST_PLUGINS=sdk.python.tests.universal.feature_repos.universal.online_store.mongodb \
542+
python -m pytest -n 8 --integration \
543+
-k "not test_universal_cli and \
544+
not test_go_feature_server and \
545+
not test_feature_logging and \
546+
not test_reorder_columns and \
547+
not test_logged_features_validation and \
548+
not test_lambda_materialization_consistency and \
549+
not test_offline_write and \
550+
not test_push_features_to_offline_store and \
551+
not gcs_registry and \
552+
not s3_registry and \
553+
not test_universal_types and \
554+
not test_snowflake" \
555+
sdk/python/tests
556+
538557
test-python-universal-milvus-online: ## Run Python Milvus online store integration tests
539558
PYTHONPATH='.' \
540559
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.milvus_online_store.milvus_repo_configuration \

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@
131131
* [ScyllaDB](reference/online-stores/scylladb.md)
132132
* [SingleStore](reference/online-stores/singlestore.md)
133133
* [Milvus](reference/online-stores/milvus.md)
134+
* [MongoDB](reference/online-stores/mongodb.md)
134135
* [Registries](reference/registries/README.md)
135136
* [Local](reference/registries/local.md)
136137
* [S3](reference/registries/s3.md)

‎docs/reference/online-stores/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,10 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
5050
[mysql.md](mysql.md)
5151
{% endcontent-ref %}
5252

53+
{% content-ref url="mongodb.md" %}
54+
[mongodb.md](mongodb.md)
55+
{% endcontent-ref %}
56+
5357
{% content-ref url="hazelcast.md" %}
5458
[hazelcast.md](hazelcast.md)
5559
{% endcontent-ref %}

0 commit comments

Comments
 (0)