mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-01 00:06:52 +00:00
088f4d670ef5b5aa3fdcde9ecb7c262eece9b184
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nishil Rathod
|
84fc48198b |
docs: add guide for using LanceDB in VS Code extensions
Closes #1849 This was lancedb/docs#371. After lancedb/docs#355 moved the open-source pages, their navigation and the snippet tests into this repository's `docs/web` and `docs/web-tests`, that PR could no longer merge there, so I've moved the same change here. ## Why #1849 asks for an example of LanceDB in a VS Code extension, because people kept running into problems there (#927, #1713). An earlier attempt, #3682, added a README link. @prrao87 asked for something else instead: a quick guide on the docs site, under **Integrations › AI Platforms & Frameworks**, showing how to use LanceDB in an extension and what results look like. The author of #3682 agreed and closed it. This PR is that guide. ## What it adds - **`docs/web/integrations/ai/vscode.mdx`**: a new page, also added to `docs/web/docs.json`. It builds a small extension with two commands. One indexes the workspace's Markdown into a LanceDB table using the built-in `huggingface` embedding function, so no API key is needed. The other searches it by meaning. The page covers the parts that are specific to VS Code: - storing the table under `context.storageUri` instead of a relative path, which is the cause of #927; - `@types/node` ≥ 22 and `skipLibCheck`, which covers #1713 and the DOM-type errors from `apache-arrow`/onnxruntime typings; - pinning `apache-arrow` inside LanceDB's peer range; - marking `@lancedb/lancedb` and `@huggingface/transformers` as esbuild externals, and removing `node_modules/**` from the template's `.vscodeignore`; - storing each file's URI so a hit opens in the right folder of a multi-root workspace; - building one VSIX per platform with `vsce package --target`, with a table mapping each target to LanceDB's native package, and a note that the sample's local model (via `onnxruntime-node`) doesn't run on Alpine; - keeping the Transformers.js model cache in `globalStorageUri`, which only works if it's loaded with `import()`; - a troubleshooting table listing each error message with its cause and fix. - **`docs/web-tests/ts/vscode_search.ts`** and **`vscode_search.test.ts`**: the LanceDB code the page shows (connect, chunk, index, search), plus jest tests for it. - **`docs/web-tests/ts/vscode_extension.ts`**: the extension's `src/extension.ts`. Jest can't run it because it needs the VS Code extension host, so it was tested in a real extension instead (see below). This follows the precedent of `integrations.ts`. - **`docs/web/snippets/vscode_search.mdx`** and **`vscode_extension.mdx`**: generated with the command in CONTRIBUTING.md. ## Changes from lancedb/docs#371 - **New paths only.** The page, tests and snippets are the same content, at their paths in this repository. - **One link.** `/tables/update` doesn't exist under `docs/web`, so the `mergeInsert` link now points to `/ingestion/merge-insert`. - **Heading anchors.** Every heading now has an explicit `{#anchor}`, as CONTRIBUTING.md asks. ## How I tested it - **Snippet tests.** In this repository at `c2c8269f`, `vscode_search.test.ts` passes (2 tests) with the pinned `@lancedb/lancedb` 0.31.0. On Windows the `npm test` script fails before it starts, because it runs `node` on the `.bin/jest` shell shim. I ran `node --experimental-vm-modules node_modules/jest/bin/jest.js --testEnvironment jest-environment-node-single-context vscode_search` instead. - **Lint.** `biome ci` is clean for the three new files, and they carry the SPDX header that `license-header-check.yml` looks for. - **Snippets.** I regenerated them with CONTRIBUTING.md's command, `mdx_snippets_gen.py -s docs/web-tests/py -s docs/web-tests/ts -s docs/web-tests/rs -o docs/web/snippets`, run with `python` since I don't have `uv`. It reproduces the two new snippet files exactly, once the Windows CRLF endings are normalised. As CONTRIBUTING.md warns, it also rewrites `build_with_ai_agents.mdx`, `integrations.mdx`, `search.mdx` and `storage.mdx`. I left those out. - **Links.** `mint broken-links` from `docs/web` reports no entries on the new page, and its three internal links (`/embedding/quickstart`, `/ingestion/merge-insert`, `/search/full-text-search`) resolve to pages here. - **Real extension.** I built a sample extension from exactly the code on the page, scaffolded with `yo code` (TypeScript + esbuild) and using `@lancedb/lancedb` 0.39.0. I ran it with `@vscode/test-cli` in VS Code 1.140 on Windows x64. The integration test runs both commands against a six-file workspace and checks the top hits. The "Sample results" block on the page is copied from that run, and after the review fixes below it reproduces line for line in VS Code 1.141. - **Multi-root workspace.** A second test configuration opens two folders that both contain `docs/auth.md`. It types a query and picks the top hit through the real picker, then checks that the right folder's file opens at the right line. It fails on the code before the review fix and passes after it. - **Packaged VSIX.** I packaged `vsce package --target win32-x64` (200 MB, or 148 MB with the onnxruntime trimming in the Tip). Then I unpacked it outside the project and ran the same tests against the unpacked copy, so they couldn't pick up the project's `node_modules`. They pass. - **Negative checks behind the troubleshooting rows:** - With `node_modules/**` left in `.vscodeignore`, activation fails with `Cannot find module '@lancedb/lancedb'` and the commands report `command '…' not found`. - Without the externals, esbuild fails with `No loader is configured for ".node" files`. - Installing LanceDB into a project that already has `apache-arrow` 21 fails with `ERESOLVE`. - `require("@huggingface/transformers")` returns a different `env` object from `import()`. - **Rendering.** For lancedb/docs#371, `mint dev` rendered the page, its 14 code blocks and the sidebar entry. I haven't re-rendered it from `docs/web`; only the paths, one link and the heading anchors changed. **Not tested:** - macOS. I couldn't reproduce the read-only working directory from #927; the page's fix (an absolute storage path) doesn't depend on the OS. - Linux and macOS VSIX builds. ## Notes for reviewers - I added `_distance` to `.select(...)`, because LanceDB warns that it will stop adding the column automatically. ## AI disclosure I used Claude Code to research the issue and its history, write the sample extension, the tests and this page, move the change here, and draft this description. I ran every check listed above myself, and the outputs quoted on the page come from those runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
 Wyatt Alt
|
b4cbaf2f67 |
docs: show stateful class Functions
The Functions page only showed a plain function, so it never said where a class's `__init__` runs. Users loading a large model need to know it is not the client: registration reads the signature and ships the source. This adds a section on class Functions: `__init__` runs once per instance on the worker, then `__call__` per row or batch and `close()` on retirement, state is per instance, and `__init__` parameters are initialization fields set per binding. The example's test asserts that registering the class does not construct it. |
||
  
|
659ea11ac4 |
fix(embeddings): default Cohere to embed-multilingual-v3.0, the v2.0 models are retired (#4698)
Cohere retired `embed-english-v2.0`, `embed-english-light-v2.0` and `embed-multilingual-v2.0` on April 4, 2026 ([deprecations](https://docs.cohere.com/docs/deprecations), which lists `embed-english-v3.0`, `embed-multilingual-v3.0` and `embed-v4.0` as the embedding alternatives). `CohereEmbeddingFunction` still defaulted to `embed-multilingual-v2.0`, so every call without an explicit `name` fails. The default is now `embed-multilingual-v3.0`, the multilingual v3 model, and the retired models are removed from the docstring and the Cohere integration page (the page also listed `embed-english-v2.0` as the default, which did not match the code). The docs example, its generated snippet, the docs test and the slow Cohere test use the new name. This changes the default vector size: `ndims()` returns 1024 for `embed-multilingual-v3.0`, where the old default gave 768. Tables created from a schema that uses `cohere.ndims()` get 1024-dimension vectors from now on. A table that already holds 768-dimension vectors from the retired model needs a re-embed with a current model, since the old model can no longer be called. The `ndims()` entries for the v2 names are left as they are. Test: `test_cohere_default_model_is_not_a_retired_v2_model` in `python/python/tests/test_embeddings.py` checks the default name and that `ndims()` is 1024. The repository's pytest session could not be run for this change (its `conftest.py` needs a newer native module than the released wheel); the body of that test was run against the released `lancedb` 0.40.0 with this version of `cohere.py` in place, where it passes, and it returns `embed-multilingual-v2.0` / 768 without the change. ruff check and format are clean on the touched Python files. No Cohere request was made. This change was made with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 lancedb-gatefixer[bot]andXuanwo
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Drew
|
436c209580 | docs: document Blob V2 in the multimodal guide (#4623) | ||
 Brendan Clement
|
06aa526ae1 |
docs: correct data workflows, SQL and operations (#4534)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`. |
||
|
Brendan Clement
|
758f738b8c |
docs: complete ingestion, search and integration examples (#4533)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. Ingestion, search and integration examples are incomplete or inconsistent across languages, and some guidance misstates SDK behavior. This brings the guides and their example sources together. It includes complete Python, TypeScript and Rust workflows, isolated provider and reranker examples, the Superlinked corrections, and guidance on retries that can duplicate writes. Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after #4491 is squash-merged. Local assembly, Mint validation and link checks pass. Example code is unchanged from the previously validated package. Publication stays held through integration. |
||
|
Brendan Clement
|
486f6c66bb |
docs: revise getting started and concepts (#4491)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The introductory pages are inconsistent, and the old PR split asked reviewers to review some pages twice. This brings the final quickstart, connection guide, architecture and Concepts pages together. It clarifies OSS and Enterprise, simplifies naming rules and includes the navigation and diagram fixes from #4489. Function and Secret details on the Concepts pages are MDX comments until their release, and CONTRIBUTING.md explains how to write docs before a release. Includes #4532. Ingestion, search and integrations are in #4533; data workflows, SQL and operations are in #4534. Each MDX file has one PR owner. Local assembly, Mint validation and link checks pass. The Function pages stay published from `main` until #4534 hides them, so hold publication until the complete stack is integrated. |
||
|
Drew
|
52da9ed934 | feat(python): add blob_mode to to_arrow and to_list (#4529) | ||
 Jack YeandBrendan Clement
|
47b7babc08 |
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (
|