Skip to content

feat: Compute Engine Initial Implementation #5223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
franciscojavierarceo merged 21 commits into from
Apr 16, 2025
Merged

feat: Compute Engine Initial Implementation #5223

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1183a9a
add compute engine
haoxu0 Apr 4, 2025
2825ee4
fix linting
haoxu0 Apr 4, 2025
4210f68
fix linting
haoxu0 Apr 4, 2025
a398075
fix linting
haoxu0 Apr 4, 2025
68c48dc
fix linting
haoxu0 Apr 7, 2025
1c9ae31
add doc
haoxu0 Apr 8, 2025
25af94e
add test
haoxu0 Apr 9, 2025
ed0cdf4
add integration test
haoxu0 Apr 10, 2025
6b57e94
update API
haoxu0 Apr 12, 2025
227f8f4
update API
haoxu0 Apr 12, 2025
e9362de
update API
haoxu0 Apr 13, 2025
68cf242
update API
haoxu0 Apr 13, 2025
3a5cf92
update API
haoxu0 Apr 13, 2025
c1ba3d6
fix linting
haoxu0 Apr 13, 2025
95f757d
update doc
haoxu0 Apr 13, 2025
e5445a2
update doc
haoxu0 Apr 13, 2025
263024b
Merge branch 'master' into compute-engine
haoxu0 Apr 13, 2025
2433064
update test
haoxu0 Apr 13, 2025
87e51c7
update doc
haoxu0 Apr 15, 2025
c698b64
Merge branch 'master' into compute-engine
haoxu0 Apr 15, 2025
0877c03
Merge branch 'master' into compute-engine
haoxu0 Apr 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix linting 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
  • Loading branch information
@haoxu0
haoxu0 committed Apr 4, 2025
commit a398075ae38155a6eca9a42a2d5068fea8c9aca6
21 changes: 21 additions & 0 deletions sdk/python/feast/infra/compute_engines/dag/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
from dataclasses import dataclass, field
from typing import Dict, List, Union

import pandas as pd

from feast.entity import Entity
from feast.infra.compute_engines.dag.value import DAGValue
from feast.infra.offline_stores.offline_store import OfflineStore
from feast.infra.online_stores.online_store import OnlineStore
from feast.repo_config import RepoConfig


@dataclass
class ExecutionContext:
project: str
repo_config: RepoConfig
offline_store: OfflineStore
online_store: OnlineStore
entity_defs: List[Entity]
entity_df: Union[pd.DataFrame, None] = None
node_outputs: Dict[str, DAGValue] = field(default_factory=dict)
21 changes: 0 additions & 21 deletions sdk/python/feast/infra/compute_engines/dag/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,7 @@
from dataclasses import dataclass, field
from enum import Enum
from typing import Dict, List, Union

import pandas as pd

from feast.entity import Entity
from feast.infra.compute_engines.dag.value import DAGValue
from feast.infra.offline_stores.offline_store import OfflineStore
from feast.infra.online_stores.online_store import OnlineStore
from feast.repo_config import RepoConfig


class DAGFormat(str, Enum):
SPARK = "spark"
PANDAS = "pandas"
ARROW = "arrow"

@franciscojavierarceo franciscojavierarceo Apr 4, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native python?

@haoxu0 haoxu0 Apr 5, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it's only a placeholder for now. The native python will be interesting as it works with on demand feature view. Will do it in the next PR



@dataclass
class ExecutionContext:
project: str
repo_config: RepoConfig
offline_store: OfflineStore
online_store: OnlineStore
entity_defs: List[Entity]
entity_df: Union[pd.DataFrame, None] = None
node_outputs: Dict[str, DAGValue] = field(default_factory=dict)
2 changes: 1 addition & 1 deletion sdk/python/feast/infra/compute_engines/dag/node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

from infra.compute_engines.dag.value import DAGValue

from feast.infra.compute_engines.dag.model import ExecutionContext
from feast.infra.compute_engines.dag.context import ExecutionContext


class DAGNode(ABC):
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/feast/infra/compute_engines/dag/plan.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import List

from feast.infra.compute_engines.dag.model import ExecutionContext
from feast.infra.compute_engines.dag.context import ExecutionContext
from feast.infra.compute_engines.dag.node import DAGNode
from feast.infra.compute_engines.dag.value import DAGValue

Expand Down
2 changes: 1 addition & 1 deletion sdk/python/feast/infra/compute_engines/spark/compute.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import pyarrow as pa

from feast.infra.compute_engines.base import ComputeEngine, HistoricalRetrievalTask
from feast.infra.compute_engines.dag.model import ExecutionContext
from feast.infra.compute_engines.dag.context import ExecutionContext
from feast.infra.compute_engines.spark.spark_dag_builder import SparkDAGBuilder
from feast.infra.materialization.batch_materialization_engine import (
MaterializationJob,
Expand Down
3 changes: 2 additions & 1 deletion sdk/python/feast/infra/compute_engines/spark/node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
from pyspark.sql import functions as F

from feast.infra.compute_engines.base import HistoricalRetrievalTask
from feast.infra.compute_engines.dag.model import DAGFormat, ExecutionContext
from feast.infra.compute_engines.dag.context import ExecutionContext
from feast.infra.compute_engines.dag.model import DAGFormat
from feast.infra.compute_engines.dag.node import DAGNode
from feast.infra.compute_engines.dag.value import DAGValue
from feast.infra.materialization.batch_materialization_engine import MaterializationTask
Expand Down