Expected Behavior
Metadata diffs such as created_timestamp and last_updated_timestamp shouldn't be included in the output of feast plan for BigQuery tables created outside of Feast (e.g. with table_create_disposition set to CREATE_NEVER). This prevents permadiffs in the plan.
Current Behavior
Feast plan currently creates a permadiff when BigQuery tables are created outside of Feast. This diff is never executed because of the table_create_disposition being set to CREATE_NEVER thus it is clutter and not relevant as there is no way this plan is every 'actioned'. This is challenging for us as we write out plans to Slack channels and it creates an enormous amount of clutter.
Updated feature view [1m[33mmy_feature_view[0m
batch_source: [1m[33mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
created_timestamp {
seconds: 1755626119
nanos: 37380000
}
last_updated_timestamp {
seconds: 1755626140
nanos: 128836000
}
}
[0m -> [1m[92mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
created_timestamp {
seconds: 1755632676
nanos: 249135000
}
last_updated_timestamp {
seconds: 1755632676
nanos: 249135000
}
}
Steps to reproduce
- Create BigQuery table outside of Feast.
- Set
create_table_disposition to CREATE_NEVER in YAML config.
- Add table to Feast registry with code similar to:
my_feature_view_source = BigQuerySource(
name="my_feature_view_batch",
table="my_feature_view_batch",
timestamp_field="feature_timestamp",
)
my_feature_view_push = PushSource(
name="my_feature_view",
batch_source=my_feature_view,
)
- Run
feast plan and feast apply using some principal without access to change anything in BigQuery.
- Run another
feast plan and observe the proposed metadata changes that can never be actioned.
Specifications
- Version:
feast[gcp,aws,grpcio,postgres]==0.51.0
- Platform: Debian, Py 3.12
- Subsystem: N/a
Possible Solution
- Exclude
created_at from the plan as this is immutable.
- Where
create_table_disposition is set to CREATE_NEVER, plans should fail when changes are required to the table.
last_updated_timestamp should never be included in the plan when create_table_disposition is set to CREATE_NEVER as no update to the table is taking place.
Expected Behavior
Metadata diffs such as
created_timestampandlast_updated_timestampshouldn't be included in the output offeast planfor BigQuery tables created outside of Feast (e.g. withtable_create_dispositionset toCREATE_NEVER). This prevents permadiffs in the plan.Current Behavior
Feast plan currently creates a permadiff when BigQuery tables are created outside of Feast. This diff is never executed because of the
table_create_dispositionbeing set toCREATE_NEVERthus it is clutter and not relevant as there is no way this plan is every 'actioned'. This is challenging for us as we write out plans to Slack channels and it creates an enormous amount of clutter.Steps to reproduce
create_table_dispositiontoCREATE_NEVERin YAML config.feast planandfeast applyusing some principal without access to change anything in BigQuery.feast planand observe the proposed metadata changes that can never be actioned.Specifications
feast[gcp,aws,grpcio,postgres]==0.51.0Possible Solution
created_atfrom the plan as this is immutable.create_table_dispositionis set toCREATE_NEVER, plans should fail when changes are required to the table.last_updated_timestampshould never be included in the plan whencreate_table_dispositionis set toCREATE_NEVERas no update to the table is taking place.