-
Notifications
You must be signed in to change notification settings - Fork 1.5k
feat: Modernize precommit hooks and optimize test performance #5929
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
e7c9113
031a978
a23152e
ae52fcc
88cefb3
759dd9e
ff4548e
de69e92
c8bbf87
46ed9d9
df45285
6bc12c2
eb6b346
5417ab2
6f6c736
9dae77f
63c8f3b
2068303
0da7c1d
8848a45
4904104
60466b2
97cd848
386c7cf
d8b156c
0b6d274
c530cf6
dec75eb
f50366b
282558a
f8051e1
0e111fc
12b4a72
2d54924
c69a5c8
cf72f4e
ad90593
3e827bc
9552822
9c499ad
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Change install-python-dependencies-ci from uv pip sync --system to uv sync --extra ci - This ensures CI uses the same uv-managed virtualenv as local development - All make targets now consistently use uv run for tool execution - Fixes mypy type stub access issues in CI Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
|
|
@@ -55,12 +55,12 @@ protos: compile-protos-python compile-protos-docs ## Compile protobufs for Pytho | |||||||
| build: protos build-docker ## Build protobufs and Docker images | ||||||||
|
|
||||||||
| format-python: ## Format Python code | ||||||||
| cd $(ROOT_DIR) && uv run ruff check --fix sdk/python/feast/ sdk/python/tests/ | ||||||||
| cd $(ROOT_DIR) && uv run ruff format sdk/python/feast/ sdk/python/tests/ | ||||||||
| uv run ruff check --fix sdk/python/feast/ sdk/python/tests/ | ||||||||
| uv run ruff format sdk/python/feast/ sdk/python/tests/ | ||||||||
|
|
||||||||
| lint-python: ## Lint Python code | ||||||||
| cd $(ROOT_DIR) && uv run ruff check sdk/python/feast/ sdk/python/tests/ | ||||||||
| cd $(ROOT_DIR) && uv run sh -c "cd sdk/python && mypy feast" | ||||||||
| uv run ruff check sdk/python/feast/ sdk/python/tests/ | ||||||||
| uv run bash -c "cd sdk/python && mypy feast" | ||||||||
|
|
||||||||
| # New combined target | ||||||||
| precommit-check: format-python lint-python ## Run all precommit checks | ||||||||
|
|
@@ -74,7 +74,7 @@ install-precommit: ## Install precommit hooks (runs on commit, not push) | |||||||
|
|
||||||||
| # Manual full type check | ||||||||
| mypy-full: ## Full MyPy type checking with all files | ||||||||
| cd ${ROOT_DIR} && uv run sh -c "cd sdk/python && mypy feast tests" | ||||||||
| uv run bash -c "cd sdk/python && mypy feast tests" | ||||||||
|
|
||||||||
| # Run precommit on all files | ||||||||
| precommit-all: ## Run all precommit hooks on all files | ||||||||
|
|
@@ -95,21 +95,12 @@ install-python-dependencies-minimal: ## Install minimal Python dependencies usin | |||||||
| uv pip sync --require-hashes sdk/python/requirements/py$(PYTHON_VERSION)-minimal-requirements.txt | ||||||||
| uv pip install --no-deps -e .[minimal] | ||||||||
|
|
||||||||
| ##@ Python SDK - system | ||||||||
| # the --system flag installs dependencies in the global python context | ||||||||
| # instead of a venv which is useful when working in a docker container or ci. | ||||||||
| ##@ Python SDK - CI (uses uv project management) | ||||||||
| # Uses uv sync for consistent behavior between local and CI environments | ||||||||
|
|
||||||||
| # Used in github actions/ci | ||||||||
| # formerly install-python-ci-dependencies-uv | ||||||||
| install-python-dependencies-ci: ## Install Python CI dependencies using uv (system) | ||||||||
| @if [ "$$(uname -s)" = "Linux" ]; then \ | ||||||||
| echo "Installing dependencies with torch CPU index for Linux..."; \ | ||||||||
| uv pip sync --system --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txt; \ | ||||||||
| else \ | ||||||||
| echo "Installing dependencies from PyPI for macOS..."; \ | ||||||||
| uv pip sync --system sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txt; \ | ||||||||
| fi | ||||||||
| uv pip install --system --no-deps -e . | ||||||||
| install-python-dependencies-ci: ## Install Python CI dependencies using uv sync | ||||||||
| uv sync --extra ci | ||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 CI dependency installation loses PyTorch CPU-only handling for Linux The Click to expandImpactThe old implementation had explicit logic to:
Old code ( install-python-dependencies-ci:
pip uninstall torch torchvision -y || true
@if [ "$$(uname -s)" = "Linux" ]; then \
uv pip sync --system --extra-index-url https://download.pytorch.org/whl/cpu ...New code ( install-python-dependencies-ci:
uv sync --extra ciWithout this special handling on Linux CI, the build may:
The Was this helpful? React with 👍 or 👎 to provide feedback.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 The PR changes Click to expandRoot CauseThe Original Code (from master):install-python-dependencies-ci:
uv pip sync --system --extra-index-url https://download.pytorch.org/whl/cpu ...
uv pip install --system --no-deps -e .New Code (Makefile:102-103):install-python-dependencies-ci:
uv sync --extra ciImpact
Recommendation: Revert to using install-python-dependencies-ci:
uv pip sync --system sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txt
uv pip install --system --no-deps -e .Was this helpful? React with 👍 or 👎 to provide feedback.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 CI dependency installation bypasses locked requirements files due to missing uv.lock The Click to expandProblemThe old implementation used locked, hashed requirements files: uv pip sync --system --require-hashes sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txtThe new implementation uses: uv sync --extra ciHowever, Impact
Actual vs Expected
Recommendation: Either: (1) Revert to using Was this helpful? React with 👍 or 👎 to provide feedback. |
||||||||
|
|
||||||||
| # Used in github actions/ci | ||||||||
| install-hadoop-dependencies-ci: ## Install Hadoop dependencies | ||||||||
|
|
@@ -170,33 +161,33 @@ benchmark-python-local: ## Run integration + benchmark tests for Python (local d | |||||||
| ##@ Tests | ||||||||
|
|
||||||||
| test-python-unit: ## Run Python unit tests (use pattern=<pattern> to filter tests, e.g., pattern=milvus, pattern=test_online_retrieval.py, pattern=test_online_retrieval.py::test_get_online_features_milvus) | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest -n 8 --color=yes $(if $(pattern),-k "$(pattern)") sdk/python/tests | ||||||||
| uv run python -m pytest -n 8 --color=yes $(if $(pattern),-k "$(pattern)") sdk/python/tests | ||||||||
|
|
||||||||
| # Fast unit tests only | ||||||||
| test-python-unit-fast: ## Run fast unit tests only (no external dependencies) | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest sdk/python/tests/unit -n auto -x --tb=short | ||||||||
| uv run python -m pytest sdk/python/tests/unit -n auto -x --tb=short | ||||||||
|
|
||||||||
| # Changed files only (requires pytest-testmon) | ||||||||
| test-python-changed: ## Run tests for changed files only | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest --testmon -n 8 --tb=short sdk/python/tests | ||||||||
| uv run python -m pytest --testmon -n 8 --tb=short sdk/python/tests | ||||||||
|
|
||||||||
| # Quick smoke test for PRs | ||||||||
| test-python-smoke: ## Quick smoke test for development | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest \ | ||||||||
| uv run python -m pytest \ | ||||||||
| sdk/python/tests/unit/test_unit_feature_store.py \ | ||||||||
| sdk/python/tests/unit/test_repo_operations_validate_feast_project_name.py \ | ||||||||
| -n 4 --tb=short | ||||||||
|
devin-ai-integration[bot] marked this conversation as resolved.
|
||||||||
|
|
||||||||
| test-python-integration: ## Run Python integration tests (CI) | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest --tb=short -v -n 8 --integration --color=yes --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \ | ||||||||
| uv run python -m pytest --tb=short -v -n 8 --integration --color=yes --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \ | ||||||||
| -k "(not snowflake or not test_historical_features_main)" \ | ||||||||
| -m "not rbac_remote_integration_test" \ | ||||||||
| --log-cli-level=INFO -s \ | ||||||||
| sdk/python/tests | ||||||||
|
|
||||||||
| # Integration tests with better parallelization | ||||||||
| test-python-integration-parallel: ## Run integration tests with enhanced parallelization | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest sdk/python/tests/integration \ | ||||||||
| uv run python -m pytest sdk/python/tests/integration \ | ||||||||
| -n auto --dist loadscope \ | ||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 The new Root Cause and ImpactAll existing integration test targets consistently use
But the new target at The 300s value appears to have been copied from the new Impact: Integration tests running via this target will fail with timeout errors for any test taking over 5 minutes, which is common for integration tests that interact with external services.
Suggested change
Was this helpful? React with 👍 or 👎 to provide feedback. |
||||||||
| --timeout=300 --tb=short -v \ | ||||||||
| --integration --color=yes --durations=20 | ||||||||
|
devin-ai-integration[bot] marked this conversation as resolved.
Comment on lines
+200
to
+204
|
||||||||
|
|
@@ -207,7 +198,7 @@ test-python-integration-local: ## Run Python integration tests (local dev mode) | |||||||
| HADOOP_HOME=$$HOME/hadoop \ | ||||||||
| CLASSPATH="$$( $$HADOOP_HOME/bin/hadoop classpath --glob ):$$CLASSPATH" \ | ||||||||
| HADOOP_USER_NAME=root \ | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \ | ||||||||
| uv run python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \ | ||||||||
| -k "not test_lambda_materialization and not test_snowflake_materialization" \ | ||||||||
| -m "not rbac_remote_integration_test" \ | ||||||||
| --log-cli-level=INFO -s \ | ||||||||
|
|
@@ -216,7 +207,7 @@ test-python-integration-local: ## Run Python integration tests (local dev mode) | |||||||
| test-python-integration-rbac-remote: ## Run Python remote RBAC integration tests | ||||||||
| FEAST_IS_LOCAL_TEST=True \ | ||||||||
| FEAST_LOCAL_ONLINE_CONTAINER=True \ | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \ | ||||||||
| uv run python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \ | ||||||||
| -k "not test_lambda_materialization and not test_snowflake_materialization" \ | ||||||||
| -m "rbac_remote_integration_test" \ | ||||||||
| --log-cli-level=INFO -s \ | ||||||||
|
|
@@ -225,7 +216,7 @@ test-python-integration-rbac-remote: ## Run Python remote RBAC integration tests | |||||||
| test-python-integration-container: ## Run Python integration tests using Docker | ||||||||
| @(docker info > /dev/null 2>&1 && \ | ||||||||
| FEAST_LOCAL_ONLINE_CONTAINER=True \ | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest -n 8 --integration sdk/python/tests \ | ||||||||
| uv run python -m pytest -n 8 --integration sdk/python/tests \ | ||||||||
| ) || echo "This script uses Docker, and it isn't running - please start the Docker Daemon and try again!"; | ||||||||
|
|
||||||||
| test-python-universal-spark: ## Run Python Spark integration tests | ||||||||
|
|
@@ -597,7 +588,7 @@ test-python-universal-couchbase-online: ## Run Python Couchbase online store int | |||||||
| sdk/python/tests | ||||||||
|
|
||||||||
| test-python-universal: ## Run all Python integration tests | ||||||||
| cd $(ROOT_DIR) && uv run python -m pytest -n 8 --integration sdk/python/tests | ||||||||
| uv run python -m pytest -n 8 --integration sdk/python/tests | ||||||||
|
|
||||||||
| ##@ Java | ||||||||
|
|
||||||||
|
|
||||||||
Uh oh!
There was an error while loading. Please reload this page.