Skip to content

feat(node): add blobMode to return blob bytes from queries - #4353

Open
BilalAtique wants to merge 4 commits into
lancedb:mainfrom
BilalAtique:feat/node-query-blob-mode
Open

BilalAtique wants to merge 4 commits into
lancedb:mainfrom
BilalAtique:feat/node-query-blob-mode

Conversation

@BilalAtique

Copy link
Copy Markdown
Contributor

Closes #4187.

Node queries can now return blob bytes instead of descriptors. QueryExecutionOptions gets blobMode?: "descriptions" | "bytes", used by toArrow() and toArray():

const rows = await table.query().where("id = 1").toArray({ blobMode: "bytes" });
rows[0].image; // Uint8Array with the blob bytes

"descriptions" stays the default, so nothing changes for existing callers.

How it works

With "bytes", the query also reads _rowid, then fills each top-level blob v2 column with the bytes from fetchBlobs (as LargeBinary). _rowid stays out of the result unless the caller asked for it with withRowId(). Renamed columns work too, for example select(new Map([["picture", "image"]])). Blobs nested in a struct or a list keep their descriptors.

This is the same approach Python uses for blob v2. to_pandas(blob_mode="bytes") reads descriptors and _rowid, then swaps in the bytes via fetch_blobs (from #3578), and the plan agreed in #4186 does the same for to_arrow and to_list. I also looked at passing Lance's BlobHandling::AllBinary through the Rust query instead. I didn't go that way because QueryRequest and the remote query protocol have no field for it, so it would only work on local tables. Going through fetchBlobs works on remote tables as well, and it works for vector, full-text and take queries, not just plain scans.

Two details:

  • The native query builder mutates the query in place, so calling withRowId() on the caller's query would leak _rowid into its later executions. Instead, native execute takes an optional withRowId flag that adds _rowid to a clone for that one execution.
  • The bytes column is a new field, not a clone of the descriptor field, so the blob v2 marker from the descriptor doesn't carry over to a binary column.

To get at fetchBlobs, the query classes now carry the table they read (Query, VectorQuery, TakeQuery, and the AutoQuery behind search()). The TypeScript API is unchanged apart from the new option and the BlobMode type.

Tests

New tests in table.test.ts cover toArrow and toArray with blobMode: "bytes" (column type, no blob marker, nulls), _rowid only when requested, a renamed blob column, a take query, a second execution of the same query still returning descriptors and no _rowid, a selection with no blob column, an empty result, and an invalid blobMode. The whole Node suite passes. The Node docs under docs/src/js are regenerated.

A "lazy" mode that returns BlobFile handles could only work in toArray(), since Arrow can't hold JS objects, so I left it for later.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the enhancement New feature or request label Sep 29, 2026
lancedb-gatekeeper[bot]

This comment was marked as outdated.

@lancedb-gatekeeper lancedb-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 29, 2026
The bytes come from a pinned snapshot. The query reruns (up to three times) when the table version changes while it runs.
@lancedb-gatekeeper lancedb-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 29, 2026

@lancedb-gatekeeper lancedb-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The earlier mixed-version race and documentation gap are addressed. Blob bytes are fetched from a pinned table snapshot, and a changed query version causes a full retry; the blobMode example now appears in the generated API reference.

If the table changes during all three attempts, this mode returns an error. That visible, documented limit is acceptable here, but callers on heavily updated tables may need to retry the read later.

@lancedb-gatekeeper lancedb-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(Typescript): Expose blob mode in queries

1 participant