Skip to content

feat(spark): SparkSource query+path and pre-computed offline read for BatchFeatureView #6440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ntkathole merged 8 commits into from
Jul 20, 2026
Merged

feat(spark): SparkSource query+path and pre-computed offline read for BatchFeatureView #6440

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0055589
feat: allow query + path in SparkSource for offline materialization
abhijeet-dhumal May 7, 2026
3535a8c
feat: read from offline path in get_historical_features for BFVs
abhijeet-dhumal May 7, 2026
5dc7b6d
fix: graceful fallback when offline path is not readable
abhijeet-dhumal May 7, 2026
7639c2a
style: ruff format spark.py and spark_source.py
abhijeet-dhumal May 27, 2026
b2c76ff
fix: allow offline-only BatchFeatureView to skip online validation in…
abhijeet-dhumal May 27, 2026
17405f3
fix(spark): narrow exception handling in offline path fallback
abhijeet-dhumal Jun 2, 2026
de321f8
fix(test): lower performance benchmark threshold from 1.5x to 1.2x
abhijeet-dhumal Jun 9, 2026
16798e4
refactor: fold pre-computed path into _apply_bfv_transformations per …
abhijeet-dhumal Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: read from offline path in get_historical_features for BFVs 
Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
  • Loading branch information
@abhijeet-dhumal @ntkathole
abhijeet-dhumal authored and ntkathole committed Jul 20, 2026
commit 3535a8cea700c72103a71b91c8edf4fbfbd609f3
72 changes: 70 additions & 2 deletions sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,72 @@ class SparkFeatureViewQueryContext(offline_utils.FeatureViewQueryContext):
max_date_partition: Optional[str]


def _apply_bfv_transformations_for_historical(

@ntkathole ntkathole Jun 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think instead of writing new helper, better approach is to extend the existing _apply_bfv_transformations with a pre-computed-path shortcut (adding the "read from parquet if offline=True and path exists" branch at the top), rather than creating a parallel function that reimplements UDF execution.

@abhijeet-dhumal abhijeet-dhumal Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed 🙌
Refactored exactly as you suggested. _apply_bfv_transformations_for_historical is deleted. _apply_bfv_transformations now has two branches:

  1. Pre-computed path shortcut (offline=True + batch_source.path) which reads parquet directly, falls back on error
  2. UDF execution via has_transformation() / get_transformation_function() which is unchanged from upstream

No duplication now.. Please review !

spark_session: SparkSession,
feature_views: List[FeatureView],
query_context: List[offline_utils.FeatureViewQueryContext],
) -> List[offline_utils.FeatureViewQueryContext]:
"""
For BatchFeatureViews, redirect get_historical_features to read from the
pre-materialized offline store (batch_source.path) when available, avoiding
expensive UDF re-execution on raw data.

Precedence:
1. offline=True + batch_source.path set -> read pre-computed parquet
2. Python/pandas UDF present -> execute UDF on raw source (fallback)
3. Otherwise -> pass through unchanged
"""
from dataclasses import replace

fv_by_name = {fv.projection.name_to_use(): fv for fv in feature_views}
new_contexts = []

for ctx in query_context:
fv = fv_by_name.get(ctx.name)
if fv is None or not isinstance(fv, BatchFeatureView):
new_contexts.append(ctx)
continue

if (
getattr(fv, "offline", False)
and isinstance(fv.batch_source, SparkSource)
and fv.batch_source.path
):
tmp_view = f"__feast_offline_{ctx.name}_{uuid.uuid4().hex[:8]}"
file_format = fv.batch_source.file_format or "parquet"
df = spark_session.read.format(file_format).load(fv.batch_source.path)
df.createOrReplaceTempView(tmp_view)
ctx = replace(ctx, table_subquery=tmp_view)
elif (
hasattr(fv, "feature_transformation")

@ntkathole ntkathole Jun 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use has_transformation() and get_transformation_function() instead

@abhijeet-dhumal abhijeet-dhumal Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done ✅
The UDF detection now uses has_transformation() and get_transformation_function() from feast.feature_view_utils, same as the existing _apply_bfv_transformations already did. The raw hasattr/getattr chains for feature_transformation are gone.

and fv.feature_transformation is not None
and (
getattr(fv.feature_transformation, "mode", None)
in ("python", "pandas")
or getattr(
getattr(fv.feature_transformation, "mode", None), "value", None
)
in ("python", "pandas")
)
):
udf = getattr(fv.feature_transformation, "udf", None) or getattr(
fv, "udf", None
)
if udf is not None:
temp_view_name = f"__feast_bfv_{ctx.name}_{uuid.uuid4().hex[:8]}"
spark_session.conf.set("spark.sql.runSQLOnFiles", "true")
raw_df = spark_session.sql(
f"SELECT * FROM {ctx.table_subquery}"
)
transformed_df = udf(raw_df)
transformed_df.createOrReplaceTempView(temp_view_name)
ctx = replace(ctx, table_subquery=temp_view_name)

new_contexts.append(ctx)

return new_contexts


class SparkOfflineStore(OfflineStore):
@staticmethod
def pull_latest_from_table_or_query(
Expand Down Expand Up @@ -298,8 +364,10 @@ def get_historical_features(
entity_df_event_timestamp_range,
)

query_context = _apply_bfv_transformations(
spark_session, feature_views, query_context
query_context = _apply_bfv_transformations_for_historical(

@ntkathole ntkathole Jun 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so previous _apply_bfv_transformations helper not removed from code?

@abhijeet-dhumal abhijeet-dhumal Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch 🙌
the previous version had a separate _apply_bfv_transformations_for_historical that duplicated UDF execution logic. Removed it and folded the pre-computed path shortcut directly into _apply_bfv_transformations as the first branch. The call site now uses the single function.
Please review !

spark_session=spark_session,
feature_views=feature_views,
query_context=query_context,
)

spark_query_context = [
Expand Down