Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update doc
Signed-off-by: hao-xu5 <hxu44@apple.com>
  • Loading branch information
haoxu0 committed Nov 5, 2025
commit 4919fba4c6d46c02e90c2f02f5a194308b0c34c7
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@
* [Type System](reference/type-system.md)
* [Data sources](reference/data-sources/README.md)
* [Overview](reference/data-sources/overview.md)
* [Table formats](reference/data-sources/table-formats.md)
* [File](reference/data-sources/file.md)
* [Snowflake](reference/data-sources/snowflake.md)
* [BigQuery](reference/data-sources/bigquery.md)
Expand Down
52 changes: 9 additions & 43 deletions docs/reference/data-sources/spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Spark data sources are tables or files that can be loaded from some Spark store (e.g. Hive or in-memory). They can also be specified by a SQL query.

**New in Feast:** SparkSource now supports advanced table formats including **Apache Iceberg**, **Delta Lake**, and **Apache Hudi**, enabling ACID transactions, time travel, and schema evolution capabilities.
**New in Feast:** SparkSource now supports advanced table formats including **Apache Iceberg**, **Delta Lake**, and **Apache Hudi**, enabling ACID transactions, time travel, and schema evolution capabilities. See the [Table Formats guide](table-formats.md) for detailed documentation.

## Disclaimer

Expand Down Expand Up @@ -55,17 +55,16 @@ my_spark_source = SparkSource(
)
```

### Table Format Support
### Table Format Examples

SparkSource now supports advanced table formats for modern data lakehouse architectures:
SparkSource supports advanced table formats for modern data lakehouse architectures. For detailed documentation, configuration options, and best practices, see the **[Table Formats guide](table-formats.md)**.

#### Apache Iceberg

```python
from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource
from feast.table_format import IcebergFormat

# Basic Iceberg configuration
iceberg_format = IcebergFormat(
catalog="my_catalog",
namespace="my_database"
Expand All @@ -79,31 +78,12 @@ my_spark_source = SparkSource(
)
```

Time travel with Iceberg:

```python
# Read from a specific snapshot
iceberg_format = IcebergFormat(
catalog="spark_catalog",
namespace="lakehouse"
)
iceberg_format.set_property("snapshot-id", "123456789")

my_spark_source = SparkSource(
name="historical_features",
path="spark_catalog.lakehouse.features",
table_format=iceberg_format,
timestamp_field="event_timestamp"
)
```

#### Delta Lake

```python
from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource
from feast.table_format import DeltaFormat

# Basic Delta configuration
delta_format = DeltaFormat()

my_spark_source = SparkSource(
Expand All @@ -114,30 +94,14 @@ my_spark_source = SparkSource(
)
```

Time travel with Delta:

```python
# Read from a specific version
delta_format = DeltaFormat()
delta_format.set_property("versionAsOf", "5")

my_spark_source = SparkSource(
name="historical_transactions",
path="s3://my-bucket/delta-tables/transactions",
table_format=delta_format,
timestamp_field="transaction_timestamp"
)
```

#### Apache Hudi

```python
from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource
from feast.table_format import HudiFormat

# Basic Hudi configuration
hudi_format = HudiFormat(
table_type="COPY_ON_WRITE", # or "MERGE_ON_READ"
table_type="COPY_ON_WRITE",
record_key="user_id",
precombine_field="updated_at"
)
Expand All @@ -150,15 +114,17 @@ my_spark_source = SparkSource(
)
```

For advanced configuration including time travel, incremental queries, and performance tuning, see the **[Table Formats guide](table-formats.md)**.

## Configuration Options

The full set of configuration options is available [here](https://rtd.feast.dev/en/master/#feast.infra.offline_stores.contrib.spark_offline_store.spark_source.SparkSource).

### Table Format Options

- **IcebergFormat**: See [Python API reference](https://rtd.feast.dev/en/master/#feast.table_format.IcebergFormat)
- **DeltaFormat**: See [Python API reference](https://rtd.feast.dev/en/master/#feast.table_format.DeltaFormat)
- **HudiFormat**: See [Python API reference](https://rtd.feast.dev/en/master/#feast.table_format.HudiFormat)
- **IcebergFormat**: See [Table Formats - Iceberg](table-formats.md#apache-iceberg)
- **DeltaFormat**: See [Table Formats - Delta Lake](table-formats.md#delta-lake)
- **HudiFormat**: See [Table Formats - Hudi](table-formats.md#apache-hudi)

## Supported Types

Expand Down
Loading