docs: add guide for using LanceDB in VS Code extensions - #371

Closes lancedb/lancedb#1849

Why

lancedb/lancedb#1849 asks for an example of LanceDB in a VS Code extension, because people kept running into problems there (lancedb/lancedb#927, lancedb/lancedb#1713). An earlier attempt, lancedb/lancedb#3682, added a README link, and @prrao87 asked for something else instead: a quick guide in this repo, under Integrations › AI Platforms & Frameworks, showing how to use LanceDB in an extension and what results look like. The author of #3682 agreed and closed it. This PR is that guide.

What it adds

• docs/integrations/ai/vscode.mdx: a new page, also added to docs.json. It builds a small extension with two commands. One indexes the workspace's Markdown into a LanceDB table using the built-in huggingface embedding function, so no API key is needed. The other searches it by meaning. The page covers the parts that are specific to VS Code:
  • storing the table under context.storageUri instead of a relative path, which is the cause of #927;
  • @types/node ≥ 22 and skipLibCheck, which covers #1713 and the DOM-type errors from apache-arrow/onnxruntime typings;
  • pinning apache-arrow inside LanceDB's peer range;
  • marking @lancedb/lancedb and @huggingface/transformers as esbuild externals, and removing node_modules/** from the template's .vscodeignore;
  • building one VSIX per platform with vsce package --target, with a table mapping each target to LanceDB's native package;
  • keeping the Transformers.js model cache in globalStorageUri, which only works if it's loaded with import();
  • a troubleshooting table listing each error message with its cause and fix.
• tests/ts/vscode_search.ts and tests/ts/vscode_search.test.ts: the LanceDB code the page shows (connect, chunk, index, search), plus jest tests for it.
• tests/ts/vscode_extension.ts: the extension's src/extension.ts. Jest can't run it because it needs the VS Code extension host, so it was tested in a real extension instead (see below). This follows the precedent of tests/ts/integrations.ts.
• docs/snippets/vscode_search.mdx and docs/snippets/vscode_extension.mdx: generated with scripts/mdx_snippets_gen.py.

How I tested it

• Snippet tests. vscode_search.test.ts passes (2 tests) with the repo's pinned @lancedb/lancedb 0.31.0. On Windows the npm test script fails before it starts, because it runs node on the .bin/jest shell shim. I ran node --experimental-vm-modules node_modules/jest/bin/jest.js --testEnvironment jest-environment-node-single-context vscode_search instead.
• Lint. biome ci is clean for the three new files.
• Real extension. I built a sample extension from exactly the code on the page, scaffolded with yo code (TypeScript + esbuild) and using @lancedb/lancedb 0.39.0. I ran it with @vscode/test-cli in VS Code 1.140 on Windows x64. The integration test runs both commands against a six-file workspace and checks the top hits. The "Sample results" block on the page is copied from that run.
• Packaged VSIX. I packaged vsce package --target win32-x64 (200 MB, or 148 MB with the onnxruntime trimming in the Tip). Then I unpacked it outside the project and ran the same tests against the unpacked copy, so they couldn't pick up the project's node_modules. They pass.
• Negative checks behind the troubleshooting rows:
  • With node_modules/** left in .vscodeignore, activation fails with Cannot find module '@lancedb/lancedb' and the commands report command '…' not found.
  • Without the externals, esbuild fails with No loader is configured for ".node" files.
  • Installing LanceDB into a project that already has apache-arrow 21 fails with ERESOLVE.
  • require("@huggingface/transformers") returns a different env object from import().
• Docs build. mint dev renders the page, its 14 code blocks and the sidebar entry. Locally, mint broken-links reports the same 81 entries with and without this change, none on the new page.

Not tested:

• macOS. I couldn't reproduce the read-only working directory from #927; the page's fix (an absolute storage path) doesn't depend on the OS.
• Linux and macOS VSIX builds.

Notes for reviewers

• Running scripts/mdx_snippets_gen.py -s tests/ts also rewrites docs/snippets/storage.mdx. It drops TsStorageGoosefsConnect, which exists only in the .mdx file and not in tests/ts/storage.test.ts, and it reorders the COS and GooseFS exports. I left that file out of this PR, but the next person who regenerates TS snippets will hit the same thing.
• I added _distance to .select(...), because LanceDB warns that it will stop adding the column automatically.

AI disclosure

I used Claude Code to research the issue and its history, write the sample extension, the tests and this page, and draft this description. I ran every check listed above myself, and the outputs quoted on the page come from those runs.

🤖 Generated with Claude Code