feat: Add LanceFormat table format with version/tag pinning - #6925

Adds Lance to the existing TableFormat abstraction from #5650, rather than introducing a separate data source. Addresses the core of #6899.

Why TableFormat and not a new source

TableFormat already models Iceberg, Delta and Hudi as formats a source can carry, with catalog/namespace addressing and a properties bag. Lance fits that shape exactly, so it needs no new source class, no new offline store, and no changes to any existing consumer.

LanceFormat(
    catalog="my_catalog",
    namespace="my_namespace",
    version=3,            # or tag="candidate"
)

It works with SparkSource unchanged

SparkSource already drives its reader generically:

reader = spark_session.read.format(self.table_format.format_type.value)
for key, value in self.table_format.properties.items():
    reader = reader.option(key, value)

So mirroring the pin into properties is what makes this fall out for free. Verified:

SparkSource accepted it   : LanceFormat
spark .format(...) value  : lance
reader .option(...) pairs :
    lance.catalog = polaris_dev
    lance.namespace = ml_features
    lance.version = 3
SparkSource proto rt      : LanceFormat version = 3

Pin semantics

version / tag is the Lance-shaped instance of #5782. Two invariants:

• version >= 1 is enforced. Lance dataset versions start at 1 and the proto treats 0 as unset, so without the guard version=0 would not round-trip — to_proto/from_proto read 0 as absent. Rejecting it keeps the Python and proto semantics in agreement rather than silently dropping a pin.
• version and tag are mutually exclusive, since a tag already resolves to a version.

On the broader question @jfw-ppi raised in #5782 — whether a pin can change the response shape — the position I'd argue for is that a pin selects data, never shape: the declared FeatureView schema stays the contract, and a pinned version whose schema disagrees should fail explicitly. That belongs in whatever consumes the pin, so it is not in this PR, but the format carries enough information to enforce it.

Proto regeneration, deliberately constrained

Two things worth flagging, both about not doing the obvious thing.

1. Generated with grpcio-tools==1.62.3, not the pinned 1.84.0.

Regenerating with the pinned toolchain emits gencode that opens with:

_runtime_version.ValidateProtobufRuntimeVersion(_runtime_version.Domain.PUBLIC, 7, 35, 1, ...)

pyproject.toml declares protobuf>=4.24.0. That call would hard-fail for anyone on protobuf 4/5/6 — and google.protobuf.runtime_version does not exist in 4.x at all, so it is an ImportError on that one module while every other proto still imports. Using 1.62.3 emits 4.25.1-level gencode, matching every other checked-in proto, and keeps the declared floor honest.

Worth noting independently: the checked-in protos are at gencode 4.25.1 while the requirements pin grpcio-tools==1.84.0 / protobuf==7.36.2, so a full regeneration on master today would touch ~70 files and raise the effective protobuf floor. That looks like something to decide on purpose rather than as a side effect of a feature PR.

2. DataSource_pb2 is left untouched on purpose.

Regenerating also rewrites DataSource_pb2.py/.pyi (~125 lines), but that churn is pre-existing — I verified it reproduces on pristine master with no changes at all. The checked-in copy is missing _CONNECTIONREF_PARAMSENTRY entries, i.e. it is stale against DataSource.proto. Happy to fix that separately; it does not belong here.

Net result is 5 files, and only DataFormat_pb2 regenerated.

Scope

This adds the format descriptor. It does not add a Lance reader or offline store — that is the follow-on discussed in #6899, and it is why the tests here cover LanceFormat semantics rather than reading real datasets (no new test dependency on pylance).

Also explicitly not proposing Lance as an online store: it is a format plus indexes, not a low-latency KV service, and its write path is columnar while online_write_batch is row-oriented proto.

Testing

11 new tests in test_table_format.py (24 total in the file): creation, minimal construction, version pin, tag pin, version < 1 rejection, version+tag rejection, dict/json/proto round-trips, unpinned not coming back as version 0, and factory dispatch.

• test_table_format.py — 24 passed
• test_utils.py, test_data_sources.py, test_types.py — 52 passed, 1 skipped
• mypy feast/table_format.py — clean, no issues
• ruff check / ruff format --check — clean

Related: #6899, #5782, #5650, #6499