Skip to content

BigQuery table metadata causing Plan Permadiff #5573

Description

@danbaron63

Expected Behavior

Metadata diffs such as created_timestamp and last_updated_timestamp shouldn't be included in the output of feast plan for BigQuery tables created outside of Feast (e.g. with table_create_disposition set to CREATE_NEVER). This prevents permadiffs in the plan.

Current Behavior

Feast plan currently creates a permadiff when BigQuery tables are created outside of Feast. This diff is never executed because of the table_create_disposition being set to CREATE_NEVER thus it is clutter and not relevant as there is no way this plan is every 'actioned'. This is challenging for us as we write out plans to Slack channels and it creates an enormous amount of clutter.

Updated feature view [1m[33mmy_feature_view[0m
    batch_source: [1m[33mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
 table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
 created_timestamp {
   seconds: 1755626119
   nanos: 37380000
 }
 last_updated_timestamp {
   seconds: 1755626140
   nanos: 128836000
 }
}
[0m -> [1m[92mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
 table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
 created_timestamp {
   seconds: 1755632676
   nanos: 249135000
 }
 last_updated_timestamp {
   seconds: 1755632676
   nanos: 249135000
 }
}

Steps to reproduce

  1. Create BigQuery table outside of Feast.
  2. Set create_table_disposition to CREATE_NEVER in YAML config.
  3. Add table to Feast registry with code similar to:
my_feature_view_source = BigQuerySource(
    name="my_feature_view_batch",
    table="my_feature_view_batch",
    timestamp_field="feature_timestamp",
)

my_feature_view_push = PushSource(
    name="my_feature_view",
    batch_source=my_feature_view,
)
  1. Run feast plan and feast apply using some principal without access to change anything in BigQuery.
  2. Run another feast plan and observe the proposed metadata changes that can never be actioned.

Specifications

  • Version: feast[gcp,aws,grpcio,postgres]==0.51.0
  • Platform: Debian, Py 3.12
  • Subsystem: N/a

Possible Solution

  • Exclude created_at from the plan as this is immutable.
  • Where create_table_disposition is set to CREATE_NEVER, plans should fail when changes are required to the table.
  • last_updated_timestamp should never be included in the plan when create_table_disposition is set to CREATE_NEVER as no update to the table is taking place.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions