mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-30 15:26:52 +00:00
005a9a353a04874c95586c7c734871d8955b1bfd
731
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Taylor
|
a065b4b9c6 |
feat(python): add NVIDIA Nemotron embedding provider
Adds a Python `nvidia` embedding provider for Nemotron 3.5 Embed 1B and 8B. It works with NVIDIA's hosted API and self-hosted NIM `/v1` endpoints, and uses the existing `openai` client. The OpenAI provider can't send NVIDIA's passage/query input types or pick Nemotron's dimensions. - Documents are embedded as `passage`, searches as `query`. - Default dimensions are 2048 (1B) and 4096 (8B), and `dim` is checked against each model's supported range. Self-hosted model IDs need an explicit `dim`. - Response count, order, dimensions and values are checked. Null and empty text get a null vector without a request. - Each request is retried on its own with LanceDB's retry policy. Invalid input or responses fail immediately. - `NVIDIA_API_KEY` is only sent to the hosted API. Custom endpoints need an explicit `api_key` (a `$var:` reference), and endpoints restored from table metadata must be allowed with `allow_endpoint`. - Request dimensions are sent through `extra_body`, preserving the NVIDIA wire contract while supporting the declared minimum `openai==1.6.1`. Python and text only. Docs are in `integrations/embedding/nvidia`. Originally authored by Taylor (@tsmithv11). ### Testing - `test_nvidia_embeddings.py`: 45 passed, 2 deselected with both the locked `openai==2.44.0` and minimum `openai==1.6.1`. These use the real OpenAI client with mock HTTP transport and real local tables. - Embedding, registry, field-metadata and trust suites: 163 passed, 4 skipped, 10 deselected. - Historical live experiments: hosted 8B passed a five-request smoke test; hosted 1B returned HTTP 500. No live request was made in this stage, so current hosted 1B availability was not verified. Enterprise coverage uses the repository mock HTTP server, not a live deployment. |
||
 BHAWESHBHASKAR
|
891774343e |
feat(python): add CombinedFieldsQuery for BM25F full text search
Add a typed Python `CombinedFieldsQuery` for BM25F search across multiple columns. The query mirrors Lance's combined-fields semantics with ordered per-column boosts and AND/OR term matching. Convert it in both directions at the PyO3 boundary, including nested Boolean queries, and document the Python typed convenience entry. Closes #4652. |
||
 geoffgscott
|
7ea10fa1e0 |
feat(node): expose targetPartitionSize on IVF index options (#4594)
The Node SDK's vector index options only take `numPartitions`. The Rust index builders (`target_partition_size` in `rust/lancedb/src/index/vector.rs`) and the Python SDK already support a target partition size, so this adds `targetPartitionSize` to `IvfPqOptions`, `IvfRqOptions`, `IvfFlatOptions`, `HnswPqOptions` and `HnswSqOptions` and passes it to their builders. When both are set, `numPartitions` is used, as it is for Rust and Python callers. The reason to want it: an index built from a partition count records no target partition size, so `table.optimize()` splits and joins it against the index type's default and can merge small partitions down to one or two. Building with a target records it in the index details, and optimize keeps to it. https://github.com/lance-format/lance/pull/10154 fixes this on the Lance side for indexes built from a partition count; this lets Node users set the size directly. The new tests build IvfPq, IvfFlat, IvfRq, IvfHnswPq and IvfHnswSq indexes with `targetPartitionSize` and check the size recorded in the index details, and check that `numPartitions` wins when both are set. Each option's doc comment states that index type's default target. Docs regenerated with `pnpm run docs`. |
||
 lancedb-gatefixer[bot]
|
741cb9ad4e |
fix(python): implement remote namespace and table existence checks (#4725)
Remote Python connections inherited the unsupported-operation stubs for `namespace_exists` and `table_exists`, so both raised `NotImplementedError` even when namespace and table operations worked. Add the missing remote overrides, async methods, and PyO3 bindings using the existing namespace describe and table open paths. Only an explicit not-found result returns `False`; permission, transport, and server errors propagate. This preserves authentication, dynamic headers, retries, and nested namespace handling, and rechecks cached tables before reporting that they exist. The async namespace helper also recognizes native `NamespaceNotFound` errors so missing local namespaces return `False`. The table and namespace guides document Python remote support and distinguish it from the unsupported `exists` routes used by direct REST namespace clients. Validation: - All 40 focused regression cases pass, covering successful and missing lookups, local and remote namespaces, dynamic headers, permission/server errors, invalid table identifiers, and cached tables that disappear. - The original inherited stubs reproduce the reported `NotImplementedError`; the local namespace regression reproduced the native missing-namespace `RuntimeError` before its fix. - Full remote connection suite: 255 passed. Local namespace tests: 14 passed. - `cargo clippy --quiet -p lancedb-python --features remote --lib`, `cargo fmt --all`, Python Ruff lint, and Python formatting checks pass. - Root `ruff check .` reports 48 existing errors in documentation examples under `docs/web-tests`; the Python subtree passes and those unrelated files are unchanged. Fixes #4722 <!-- lance-gatekeeper-fix:v1 agent=4d369a0681756fdb5a8b35853d7f5607 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Nishil Rathod
|
84fc48198b |
docs: add guide for using LanceDB in VS Code extensions
Closes #1849 This was lancedb/docs#371. After lancedb/docs#355 moved the open-source pages, their navigation and the snippet tests into this repository's `docs/web` and `docs/web-tests`, that PR could no longer merge there, so I've moved the same change here. ## Why #1849 asks for an example of LanceDB in a VS Code extension, because people kept running into problems there (#927, #1713). An earlier attempt, #3682, added a README link. @prrao87 asked for something else instead: a quick guide on the docs site, under **Integrations › AI Platforms & Frameworks**, showing how to use LanceDB in an extension and what results look like. The author of #3682 agreed and closed it. This PR is that guide. ## What it adds - **`docs/web/integrations/ai/vscode.mdx`**: a new page, also added to `docs/web/docs.json`. It builds a small extension with two commands. One indexes the workspace's Markdown into a LanceDB table using the built-in `huggingface` embedding function, so no API key is needed. The other searches it by meaning. The page covers the parts that are specific to VS Code: - storing the table under `context.storageUri` instead of a relative path, which is the cause of #927; - `@types/node` ≥ 22 and `skipLibCheck`, which covers #1713 and the DOM-type errors from `apache-arrow`/onnxruntime typings; - pinning `apache-arrow` inside LanceDB's peer range; - marking `@lancedb/lancedb` and `@huggingface/transformers` as esbuild externals, and removing `node_modules/**` from the template's `.vscodeignore`; - storing each file's URI so a hit opens in the right folder of a multi-root workspace; - building one VSIX per platform with `vsce package --target`, with a table mapping each target to LanceDB's native package, and a note that the sample's local model (via `onnxruntime-node`) doesn't run on Alpine; - keeping the Transformers.js model cache in `globalStorageUri`, which only works if it's loaded with `import()`; - a troubleshooting table listing each error message with its cause and fix. - **`docs/web-tests/ts/vscode_search.ts`** and **`vscode_search.test.ts`**: the LanceDB code the page shows (connect, chunk, index, search), plus jest tests for it. - **`docs/web-tests/ts/vscode_extension.ts`**: the extension's `src/extension.ts`. Jest can't run it because it needs the VS Code extension host, so it was tested in a real extension instead (see below). This follows the precedent of `integrations.ts`. - **`docs/web/snippets/vscode_search.mdx`** and **`vscode_extension.mdx`**: generated with the command in CONTRIBUTING.md. ## Changes from lancedb/docs#371 - **New paths only.** The page, tests and snippets are the same content, at their paths in this repository. - **One link.** `/tables/update` doesn't exist under `docs/web`, so the `mergeInsert` link now points to `/ingestion/merge-insert`. - **Heading anchors.** Every heading now has an explicit `{#anchor}`, as CONTRIBUTING.md asks. ## How I tested it - **Snippet tests.** In this repository at `c2c8269f`, `vscode_search.test.ts` passes (2 tests) with the pinned `@lancedb/lancedb` 0.31.0. On Windows the `npm test` script fails before it starts, because it runs `node` on the `.bin/jest` shell shim. I ran `node --experimental-vm-modules node_modules/jest/bin/jest.js --testEnvironment jest-environment-node-single-context vscode_search` instead. - **Lint.** `biome ci` is clean for the three new files, and they carry the SPDX header that `license-header-check.yml` looks for. - **Snippets.** I regenerated them with CONTRIBUTING.md's command, `mdx_snippets_gen.py -s docs/web-tests/py -s docs/web-tests/ts -s docs/web-tests/rs -o docs/web/snippets`, run with `python` since I don't have `uv`. It reproduces the two new snippet files exactly, once the Windows CRLF endings are normalised. As CONTRIBUTING.md warns, it also rewrites `build_with_ai_agents.mdx`, `integrations.mdx`, `search.mdx` and `storage.mdx`. I left those out. - **Links.** `mint broken-links` from `docs/web` reports no entries on the new page, and its three internal links (`/embedding/quickstart`, `/ingestion/merge-insert`, `/search/full-text-search`) resolve to pages here. - **Real extension.** I built a sample extension from exactly the code on the page, scaffolded with `yo code` (TypeScript + esbuild) and using `@lancedb/lancedb` 0.39.0. I ran it with `@vscode/test-cli` in VS Code 1.140 on Windows x64. The integration test runs both commands against a six-file workspace and checks the top hits. The "Sample results" block on the page is copied from that run, and after the review fixes below it reproduces line for line in VS Code 1.141. - **Multi-root workspace.** A second test configuration opens two folders that both contain `docs/auth.md`. It types a query and picks the top hit through the real picker, then checks that the right folder's file opens at the right line. It fails on the code before the review fix and passes after it. - **Packaged VSIX.** I packaged `vsce package --target win32-x64` (200 MB, or 148 MB with the onnxruntime trimming in the Tip). Then I unpacked it outside the project and ran the same tests against the unpacked copy, so they couldn't pick up the project's `node_modules`. They pass. - **Negative checks behind the troubleshooting rows:** - With `node_modules/**` left in `.vscodeignore`, activation fails with `Cannot find module '@lancedb/lancedb'` and the commands report `command '…' not found`. - Without the externals, esbuild fails with `No loader is configured for ".node" files`. - Installing LanceDB into a project that already has `apache-arrow` 21 fails with `ERESOLVE`. - `require("@huggingface/transformers")` returns a different `env` object from `import()`. - **Rendering.** For lancedb/docs#371, `mint dev` rendered the page, its 14 code blocks and the sidebar entry. I haven't re-rendered it from `docs/web`; only the paths, one link and the heading anchors changed. **Not tested:** - macOS. I couldn't reproduce the read-only working directory from #927; the page's fix (an absolute storage path) doesn't depend on the OS. - Linux and macOS VSIX builds. ## Notes for reviewers - I added `_distance` to `.select(...)`, because LanceDB warns that it will stop adding the column automatically. ## AI disclosure I used Claude Code to research the issue and its history, write the sample extension, the tests and this page, move the change here, and draft this description. I ran every check listed above myself, and the outputs quoted on the page come from those runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
  Will JonesandClaude Opus 5.5
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 Wyatt Alt
|
b4cbaf2f67 |
docs: show stateful class Functions
The Functions page only showed a plain function, so it never said where a class's `__init__` runs. Users loading a large model need to know it is not the client: registration reads the signature and ships the source. This adds a section on class Functions: `__init__` runs once per instance on the worker, then `__call__` per row or batch and `close()` on retirement, state is per instance, and `__init__` parameters are initialization fields set per binding. The example's test asserts that registering the class does not construct it. |
||
 JiMyung Lee
|
057a39a1c7 |
docs: row counts in list_indices can be set for remote tables
Remote list_indices fills num_indexed_rows / num_unindexed_rows from the server's enriched response and only leaves them unset for servers that do not report them. The Node and Rust documentation previously implied those fields were always unavailable for remote tables. Synchronize the generated TypeScript API reference with the Node source comments. Closes #4301 Co-authored-by: desty <lee.ji.myung@gmail.com> |
||
 Joshua Snider
|
edb79d38e3 |
docs: correct the documented default retry statuses (#3964)
#2349 changed the default retryable status codes to `[409, 429, 500, 502, 503, 504]`, but only the Rust docstring was updated. The Python and TypeScript bindings — and the generated TypeScript API reference — still document the old `[429, 500, 502, 503]`. The actual default is set in `ResolvedRetryConfig::try_from` in `rust/lancedb/src/remote/retry.rs`, and `rust/lancedb/src/remote/client.rs` already documents it correctly. This brings the other three in line. Docs only — no behavior change. Split out from #3963 per the "one problem at a time" guidance in `CONTRIBUTING.md`. |
||
 J Ganesh Kumar Reddy
|
12d86ea251 |
feat(nodejs): add col() expression builder for camelCase column names
## Summary Fixes #3211 by exposing the shared Rust/DataFusion expression builder through Node bindings. Case-sensitive references such as `col("firstName")` can be composed and passed to `where()`, while SQL strings remain supported. The expression-object representation follows [the design requested on #3290](https://github.com/lancedb/lancedb/pull/3290#discussion_r3632371998). That earlier PR remains open; no claim is made that the issue was unclaimed or that maintainers have selected this implementation. ## Review corrections - Construct the named native struct correctly and parse casts through `arrow_schema::DataType`. - Preserve numeric values: use Int64 only for JavaScript safe integers; retain Float64 otherwise. Explicit `litInt` rejects unsupported numbers with a documented `RangeError`. - Correct test syntax/API calls and align SQL assertions with the shared dialect, including reserved and case-sensitive names. - Use typed, isolated temporary test tables with cleanup, and cover an actual large-number filter threshold. - Document the public methods and commit generated API documentation. Synced with current main `0bceaaf3` through a normal merge. The diff is limited to seven Node source/test/config files and eleven generated documentation files; no Rust core change. ## Validation [Linux validation run](https://github.com/Ganesh1729-ui/lancedb/actions/runs/38244794288) on Node 24 / Rust 1.97.0 with the repository's `ci` profile and `remote,lancedb/metrics-otel` features: - Rebuilt native addon, TypeScript build, Biome, Rust formatting and native Clippy pass. - Expression/query/Arrow/table suites: **1,129 passed**; expression suite repeated: **42 passed**. - Using the original JavaScript numeric dispatch with the rebuilt addon: **8 regressions fail, 2 controls pass**. - Strict TypeDoc generation passes. Final local typing/docs/lint checks pass; the PR source equals the validated source except for the separate validation workflow, which is excluded from this PR. Full provider/model/S3, other feature combinations and OS/Node matrices were not run in this follow-up. Upstream checks and maintainer review remain separate. ## AI assistance AI was used for implementation, review and testing. No prior human review is claimed. |
||
 Lance Release
|
ece54d8917 | Bump version: 0.41.0-beta.16 → 0.41.0-beta.17 | ||
 Brendan Clement
|
882be89778 |
docs: clarify how to contribute website documentation
The contributor guide mixes up shared documentation, Enterprise placeholders and the SDK reference. It also describes section merging that the assembler doesn't do. This replaces the overlapping instructions with guidance on where to write, how whole-page replacement works, how to preview changes and how to keep unfinished docs hidden. It explains that source changes enter a saved release edition before they appear on staging, and points SDK contributors to the existing reference guide. |
||
 
|
659ea11ac4 |
fix(embeddings): default Cohere to embed-multilingual-v3.0, the v2.0 models are retired (#4698)
Cohere retired `embed-english-v2.0`, `embed-english-light-v2.0` and `embed-multilingual-v2.0` on April 4, 2026 ([deprecations](https://docs.cohere.com/docs/deprecations), which lists `embed-english-v3.0`, `embed-multilingual-v3.0` and `embed-v4.0` as the embedding alternatives). `CohereEmbeddingFunction` still defaulted to `embed-multilingual-v2.0`, so every call without an explicit `name` fails. The default is now `embed-multilingual-v3.0`, the multilingual v3 model, and the retired models are removed from the docstring and the Cohere integration page (the page also listed `embed-english-v2.0` as the default, which did not match the code). The docs example, its generated snippet, the docs test and the slow Cohere test use the new name. This changes the default vector size: `ndims()` returns 1024 for `embed-multilingual-v3.0`, where the old default gave 768. Tables created from a schema that uses `cohere.ndims()` get 1024-dimension vectors from now on. A table that already holds 768-dimension vectors from the retired model needs a re-embed with a current model, since the old model can no longer be called. The `ndims()` entries for the v2 names are left as they are. Test: `test_cohere_default_model_is_not_a_retired_v2_model` in `python/python/tests/test_embeddings.py` checks the default name and that `ndims()` is 1024. The repository's pytest session could not be run for this change (its `conftest.py` needs a newer native module than the released wheel); the body of that test was run against the released `lancedb` 0.40.0 with this version of `cohere.py` in place, where it passes, and it returns `embed-multilingual-v2.0` / 768 without the change. ruff check and format are clean on the touched Python files. No Cohere request was made. This change was made with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Lance Release
|
45f987a29b | Bump version: 0.41.0-beta.15 → 0.41.0-beta.16 | ||
|
Wyatt Alt
|
bb6aa1a147 |
feat: add a cascade option to drop_database (#4697)
The remote catalog could only drop an empty database: it always asked
the server for a restricted drop.
`DropDatabaseRequest::cascade`, `drop_database(..., cascade=True)` in
Python and `dropDatabase(name, { cascade: true })` in TypeScript ask the
server to drop everything in the database first. The name stays taken
until the server finishes cleaning up the dropped objects' storage. A
restricted drop remains the default.
|
||
 Jack YeandXuanwo
|
45dbf5eff5 |
fix: use SQL for all remote materialized view APIs (#4673)
Remote materialized-view APIs now use SQL for the full lifecycle. Create and refresh already submit SQL; this change moves describe, paginated listing, and drop onto `DESCRIBE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS`, and `DROP MATERIALIZED VIEW`. Async create, refresh, and drop return the existing Job handle with `job_type = "query"`. The handle has one `id`: the connection-scoped SQL query ID, also accepted by `describe_query`. Submission starts the query, the job drains its result once, and repeated waits reuse the completed result. Synchronous drop waits for catalog removal; physical cleanup proceeds in the background. Listing uses namespace-scoped keyset pagination with opaque resume tokens. Streamed and deferred SQL responses share the same path. Adds transport coverage and updates Python/TypeScript documentation for query-backed drop jobs. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
ac99ab173b |
fix(python): support native blob conversion in pandas and batches (#4647)
Plain local blob v2 scans tried to open a pylance dataset before reaching LanceDB's byte-fetch path, so `to_pandas(blob_mode="bytes")` failed when the optional dependency was absent. Batch readers also had no `blob_mode` option. Route supported blob v2 pandas conversions through the native Arrow path. Add `blob_mode` to sync and async batch readers, convert payloads during iteration, and expose the converted schema without fetching the first batch. Apply conversion after reranking and retain the default descriptor behavior. Clean up scanner error punctuation and remove the misleading plain-scan suggestion. Bytes mode retains the existing `to_arrow` restrictions for legacy/list blobs, remote tables, and namespace query pushdown. Lazy pandas handles and legacy pandas scans still require pylance; sync FTS/hybrid batching remains unsupported. Validation: - UV-managed editable Python development build completed successfully. - The absent-pylance regression tests fail against the original query implementation and pass with this change. - Blob and query suites: 403 passed, 1 skipped, 1 deselected. - Remote query suite: 190 passed. - Python-tree Ruff lint/format checks and the async reader Pyright check pass. Repository-wide Ruff reports existing errors outside the Python tree. Fixes #4645 <!-- lance-gatekeeper-fix:v1 agent=fd85998f14bb0b3c92b2d050df93a615 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Dan Tasse
|
6e4f523e14 |
docs: remove logical-column from docs (#4693)
We've removed this feature from LanceDB Enterprise and no longer privilege the "lancedb:logical-column" key. So, removing that from these docs. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
2cd635d41c |
fix: correct remote materialized view refresh results and options (#4672)
Remote materialized-view refreshes drained the SQL result and then fabricated `mode=rebuild`, `rows_written=<total view rows>`, and `source_version=0`. This made incremental refreshes report incorrect statistics. Make these three result fields optional across Rust, Python, and TypeScript, and leave them unset for remote SQL refreshes. Remove the post-refresh row count request. Local refreshes continue to populate all metrics, and `version` remains the view version observed after completion. Remote per-operation statistics remain unavailable until the SQL job adapter can retrieve them. Expose the forced-rebuild option through the Rust builder, synchronous/asynchronous Python refresh and job methods, and TypeScript refresh. Remote `full=true` sends `WITH (full = true)`; the default lets the server choose its refresh mode. Local source-version pinning and remote job cancellation are preserved. Local refresh regression tests respect the `experimental-functions` build gate, while remote refresh coverage also runs in default builds. Validation: - Rust: 20 materialized-view tests and 21 remote SQL tests passed with `remote,experimental-functions`; `cargo fmt --all` passed. - Python: rebuilt the extension in the uv-managed environment. The default build passed 23 tests with 27 feature-dependent skips. The opt-in build passed 61 tests with 3 expected default-build-only skips, covering the materialized-view suite and 14 async query-serialization cases from current main. Python Ruff lint and formatting checks passed. - TypeScript: default and opt-in native builds and TypeScript compilation passed. The default build passed 5 tests with 10 expected feature-dependent skips; the opt-in build passed all 14 materialized-view tests with the default-build-only test skipped. Lint/format checks and documentation generation passed. - Repository-wide `ruff check .` reports 48 pre-existing violations in unchanged `docs/web-tests` files; `ruff check python` passes. Fixes #4670 <!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
c167b99fa0 |
fix: read legacy blob columns through blob APIs (#4649)
Legacy blob v1 columns were rejected by LanceDB's blob validation and omitted from `blob_columns()`, even though the pinned Lance reader already supports them. Accept both blob versions in read validation and discovery so `fetch_blobs`, `fetch_blob_files`, and `fetch_blob_ranges` can read legacy payloads through the existing native reader. Keep list-nested blobs excluded from row-based reads and document how to fetch legacy blobs using explicit row IDs. Regression coverage checks sync and async APIs, independently written Lance datasets, ordering and duplicates across fragments, null and empty blobs, lazy seeking, ranges, and deleted rows. Fixes #4644 Validation: - Both new Python regressions failed on the original implementation with the reported legacy-column rejection and empty discovery result, then passed with the fix. - Python blob suite: 202 passed, 1 skipped. - Remote blob tests: 29 passed. - Focused Rust blob unit tests with `remote`: 13 passed. - Rust blob integration suite with `remote`: 59 passed, including legacy storage 2.0 and 2.1. - Rust compile and Clippy checks with `remote`, tests, and examples passed. - Python package Ruff lint and format checks passed; Rust formatting passed. - Repository-wide `ruff check .` reports 49 existing errors in unchanged `docs/web-tests` examples; the Python package is clean. <!-- lance-gatekeeper-fix:v1 agent=02b7edcbfa8ff0e7fad4615159249b3e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
83b7275137 |
fix(remote): make HTTP timeout errors actionable (#4650)
Remote `create_table` reader uploads can exceed the read deadline while waiting for response headers, but reqwest's main error message only says "error sending request". Make HTTP and retry errors explicitly identify timeouts and name the relevant `client_config.timeout_config` settings and environment variables, preserving typed sources, request metadata, and retry classification. Document that the read timeout covers waiting for headers during upload. This implements the issue's diagnostic option: creation still uses one streaming request, so large or slow readers may require a longer `client_config.timeout_config.read_timeout`. Validation: - 48 focused Rust client, error, and retry tests passed. Regression coverage holds an upload open until its read timeout and also exercises TLS connection timeouts, with and without retries. - 8 Python create-table and HTTP/retry error tests passed in the uv-managed environment after rebuilding the extension, including reader creation with read and overall timeouts. - `cargo check --quiet --features remote --tests --examples` passed. - `cargo clippy --quiet --features remote --tests --examples` passed. - Rust formatting and Python package Ruff lint/format checks passed. - Repository-wide `ruff check .` reports 49 existing findings under `docs/web-tests`; those files are unchanged. Fixes #4646 <!-- lance-gatekeeper-fix:v1 agent=ce1c040fa581f1748166ecb686d9c9c0 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
8a4effa06a |
fix: gate experimental local functions behind an opt-in feature (#4688)
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files. |
||
|
Lance Release
|
37b46f6b4f | Bump version: 0.41.0-beta.14 → 0.41.0-beta.15 | ||
|
lancedb-gatefixer[bot]andXuanwo
|
e3dce2cbad |
fix(rust): prevent oversized SQ remap panics (#3861)
## Summary - preflight IVF-SQ and IVF-HNSW-SQ storage segments touched by compaction - preserve unknown legacy fragment coverage as conservatively affected - cover both fixed-size vector and multivector SQ storage dimensions - reject only affected partitions whose fixed-size-list child exceeds Arrow's `u32` take-index limit - return an actionable `InvalidInput` error before Arrow can panic ## Root cause Arrow 58 builds fixed-size-list child indices with `u32`. For a large SQ partition, `partition_rows * vector_dimension` can exceed `u32::MAX`; the child range then wraps inside `take_fixed_size_list`, producing the reported `ArrayData::slice` assertion during index remapping. The preflight now reads raw segment metadata, mirrors Lance's fragment-bitmap selection (including `None` as affected/unknown), opens only affected vector segments, and checks their actual storage quantizer and partition sizes. It derives the stored dimension for both `FixedSizeList` vectors and flattened `List<FixedSizeList>` multivectors. Stable-row-ID datasets, deferred remapping, custom remappers, unaffected segments, non-SQ indexes, safe partitions, and no-op compactions are unchanged. Because the upstream Arrow kernel still has the limit, an affected index must be recreated with default IVF partitioning (or enough partitions to stay below the reported maximum) before compaction. This changes the failure from a process panic to a targeted recovery error; it does not silently alter index structure. ## Validation - `cargo fmt --all -- --check` - `cargo test --quiet --features aws,remote -p lancedb table::optimize::tests --lib` - `cargo test --quiet --features remote -p lancedb table::create_index::tests::test_create_index_ivf_hnsw_sq` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples -- -A clippy::redundant-pub-crate` (the allowance is for pre-existing Rust 1.97 warnings in remote-table code) Fixes #2866 <!-- lance-gatekeeper-fix:v1 agent=ba2f07b1ee6f9e68a0de1961c164f70e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Drew
|
436c209580 | docs: document Blob V2 in the multimodal guide (#4623) | ||
 KarthikSandXuanwo
|
180eaaf68e |
fix!: reject unsupported remote connection and table options (#4486)
Fixes #4399. Remote connections currently discard local-only settings, including table write options that determine storage format. The synchronous Python client also binds the fourth positional `create_table` argument to `on_bad_vectors`, so an intended overwrite is sent as a create. This change aligns the remote Python signature with `DBConnection.create_table`, preserves the legacy remote timeout keywords, and raises `ValueError` for other unknown connection keywords. The shared Rust client rejects unsupported connection, create, and open options before sending HTTP requests; open validation also runs before a table-cache hit. Python reports these as `NotImplementedError`, including explicit `enable_v2_manifest_paths=False`. Supported Enterprise connection-level storage options remain available. This implements option 2 from the issue: explicit rejection. It deliberately does not add unverified server write parameters. Calls that previously supplied ignored options must omit them to use server-managed storage and caching. Database URI prefixes are already preserved on current main and are unchanged here. Regression tests cover Python sync/async, TypeScript create/open calls, and the Rust remote client, including empty-table creation, overwrite mode, and cached table handles. Python API documentation describes the behavior and corrected positional order. Reconciled with main at `0ea932da`, preserving upstream blob and error-handling tests and updating the Python test that previously expected open options to be ignored. Validation on the merged source: - Rebuilt the Python native extension; 304 tests passed across remote options, database, remote database, and session suites. - Rust remote database suite: 135 passed. Rebuilt the Node native extension; TypeScript remote suite: 62 passed. TypeScript compilation and documentation generation passed. - Full Python SDK lint and formatting pass (159 files); full TypeScript lint and formatting pass (66 files). Rust formatting and `git diff --check` pass. - Repository-wide Python lint/format reports existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust 1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci` build profile. Remote tests use HTTP fixtures; this does not claim validation against the original ARM64 single-node deployment. Documentation review follow-up: added a draft Enterprise migration note in `docs/web/basics/sdk.mdx`. It explains that `db://` connections reject local session/cache and per-table storage/write options, while supported connection-level Enterprise storage options remain available. Following the repository's release conventions, the note stays in an MDX comment below the migration list until the behavior is released. Vale 3.24.0 with the pinned LanceDB style package reports no findings; `git diff --check` passes. This follow-up changes documentation only. BREAKING CHANGE: Remote connections now reject unsupported local session/cache and per-table storage/write options instead of silently ignoring them. Callers must omit these options and use server-managed storage and caching; supported Enterprise connection-level storage options remain available. Integration with current main preserves its synchronous API signatures, including the remote `create_table(mode=None)` default, and the table compatibility defaults. The positional-order repair is already on main via #4608; this PR retains the distinct unsupported-option rejection behavior. CI follow-up: the inherited IVF search-effort test occasionally measured recall 0.375 against its unchanged 0.5 threshold after random index training. A matching local reproducer observed this once in 30 builds. The fixture now places the eight exact neighbors together, preserving the 16-partition setup, two writes, all probe/precedence assertions, and the recall threshold. The adjusted fixture returned recall 1.0 in 100 builds; the full query and remote suites passed (102 tests). This stabilizes test data and does not change search behavior or promise production recall. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
KarthikSandXuanwo
|
3dd579785e |
fix(python)!: align synchronous table method signatures (#4498)
Closes #4396. Calls written against `Table` can fail or bind different arguments when used with `RemoteTable`. Align the affected sync signatures, positional order, and defaults; preserve renamed remote keywords as deprecated aliases. Share legacy vector configuration and tokenizer inference, forward index tuning and timeout options, and accept `fast_search` consistently. Preserve current upstream conversion behavior: local `to_polars(batch_size=...)` returns a LazyFrame; remote tables accept the same argument signature but raise the documented unsupported-operation error and direct callers to `table.search().to_polars()`. The abstract `Table` method retains its unsupported-operation error, with the implementation confined to `LanceTable`. Regression coverage checks argument binding, deprecated aliases, option forwarding, native behavior through both wrappers, HTTP index requests, and remote conversion rejection. The merge preserves current tokenizer defaults, remote query defaults, HNSW-PQ support, and unsupported-operation handling. Runtime validation at `c4fb01f2` (Linux x86_64, uv-managed Python 3.12.15): - Focused Ruff lint and formatting passed for all four changed Python files. - Rebuilt and installed the current native extension successfully with the repository's `ci` Cargo profile. - Seven regression suites passed: `test_errors.py`, `test_table_signatures.py`, `test_remote_db.py`, `test_table.py`, `test_index.py`, `test_fts.py`, and `test_query.py`: **874 passed, 1 deselected**, with slow and S3 tests excluded. - CI caught an abstract-base conversion regression in the initial merge. The existing error-contract test reproduced it locally (1 failed, 7 passed); restoring the implementation to `LanceTable` fixes it while preserving signature parity. - Full-root Ruff reports 49 lint errors and 11 files needing formatting, all in unchanged upstream `docs/web-tests` files. The four changed Python files pass both checks. - Validated against upstream `197403e8`, using Lance 15.0.0-beta.1 and LanceDB 0.41.0-beta.14 bindings. Historical validation before this merge: 618 tests passed and one was deselected. That result does not establish correctness of the current merge. The original client-side reproduction produced nine argument-binding errors without contacting a server; the initial regression suite failed 32 cases before the fix and passed 39 after it. Documentation review follow-up: restored the `batch_size` parameter description on the abstract `Table.to_polars` method so it appears in the public API reference. This adds six docstring lines and changes no runtime behavior. After this edit, all 47 focused error-contract and signature tests pass; changed-file Ruff lint/format and `git diff --check` pass. Repository-wide Ruff still reports the same 49 findings and 11 formatting files in unchanged `docs/web-tests`. The full documentation site and seven-suite runtime run were not repeated for this documentation-only follow-up. The reported ARM64 production-server deployment has not been recreated. HTTP integration tests use the repository's local mock server. BREAKING CHANGE: Remote synchronous table methods now use the positional argument order and defaults documented by `Table`. Migrate calls using the former remote-only positional order to named arguments, particularly `create_index` and `search`; pass `index_type` by keyword to `create_scalar_index`. Remote scalar and vector index creation now default to `replace=True`; specify `replace=False` to retain rejection of an existing index. Renamed remote keywords remain supported as deprecated aliases, as documented above. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
66357275bb |
fix: inherit Lance defaults for IVF RQ quantization (#4585)
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding Lance's five-bit default. The shared native index creation path also substituted one bit when the Rust or Node builder left the value unset. Make Python's bit count optional and forward only explicit values. Use `RQBuildParams::default()` for an unset native bit count, and omit unset values when dispatching to pylance for accelerator builds. Update the Python, Rust, Node, and vector-index documentation. New indexes with an unset bit count now inherit Lance's default, currently five bits, across Python, Rust, and Node. Five-bit vector codes use roughly five times the storage of one-bit codes, plus other index overhead. This also changes reader compatibility: indexes created with `num_bits >= 2` use a newer on-disk layout that some older LanceDB versions cannot read. Upgrade readers to versions that support multi-bit IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)` in Python) when sharing indexes with readers limited to one-bit IVF_RQ. Validation: - Reproduced six regression failures before the fix: one-bit default metadata, rejected `None`, and incorrect remote/accelerator forwarding. - After merging `main` through `b762ec22`, rebuilt both native bindings against Lance 15.0.0-beta.1. Python validation passed with 106 tests and 15 optional Ollama skips. Coverage includes the full index suite, omitted and explicit bit counts, reading Lance-created IVF_RQ indexes, accelerator forwarding, CUDA-unavailability handling, timeout validation, remote requests, and table/namespace listing behavior. - Rust formatting and Python package Ruff lint/format checks pass. Node `pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run docs`, the full connection suite, and the existing IVF_RQ regression pass. Generated documentation adds no further changes. The merge preserves upstream's explicit RaBitQ heading anchor alongside the `IvfRq()` example and five-bit default/reader-compatibility guidance. Repository-wide `ruff check .` still reports 49 existing errors in unchanged documentation examples under `docs/web-tests`; root formatting also finds 11 unchanged documentation example files. The Python package checks pass. Fixes #4582 <!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
7ed03e1c40 |
fix(python): honor bad-vector policies for pandas NaNs (#4503)
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN mask and Rust's default append validator ignored null children, so `add()` silently stored incomplete vectors despite `on_bad_vectors="error"`. Reject null components in the existing Rust ingestion validator, and include null components in Python's bad-vector detection and filling. `drop` removes affected rows, `fill` repairs missing components while preserving valid ones, and `null` replaces affected vectors. Explicit null components in Arrow and list inputs follow the same policies. Rust allows whole null vectors and retains the explicit `NaNVectorBehavior::Keep` opt-out. The ingestion guide and synchronous/asynchronous `create_table()` option docs describe null components, all four policies, and filling that preserves valid components. Validation: - Reproduced the default pandas append failure with the regression test before changing the Rust validator. - Rebuilt the native Python extension from the merged sources. The Python table and utility suites passed: 481 passed, 1 skipped. Coverage includes create/append/overwrite, pandas/rows/Arrow inputs, every bad-vector policy, float16/32/64, sliced and chunked arrays, all-null components, and hidden children of null lists. - All 43 focused Rust tests passed: 7 validator tests and 36 ingestion tests. - Rust Clippy with `remote`, tests, and examples passed. Ruff lint passes for the Python package, formatting checks pass for modified Python files, and `cargo fmt --all` is clean. - Documentation follow-up: all 56 focused vector-policy tests passed; Python package lint and creation-docstring formatting checks passed. Repository-wide Ruff reports 49 existing errors in untouched documentation examples. Create/overwrite retain their existing `RuntimeError` wrapping for errors raised while consuming lazy Arrow readers. Fixes #4435 <!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Lance Release
|
de0c5edc7d | Bump version: 0.41.0-beta.13 → 0.41.0-beta.14 | ||
|
Jack Ye
|
dcfef7c5ab |
feat(python): open lazy blob files from SQL descriptors (#4668)
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
|
||
|
Jack Ye
|
9af7757053 |
fix: execute remote materialized view creation through SQL (#4665)
Remote materialized-view creation currently expects a REST response containing a registry job ID, while refresh already submits SQL. Route both through the SQL query backend and preserve Python's synchronous and asynchronous return types. ### Query to Job mapping `SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through the same backend as `execute_query_async()`. Once submission succeeds, it retains the returned `Query` and starts one client-side task that consumes its result reader to completion. `Job::from_query_task()` wraps that task, its cached outcome, a single ID and a cancellation callback. Python exposes this as `Job[T]` or `AsyncJob[T]`. The existing `job_type` property maps SQL-backed handles to the new `"query"` kind. The shared `id` contains that query's UUID string, so Python callers can inspect it with `db.describe_query(job.id)` on the submitting connection. No separate `query_id`, backend enum or `type()` API is added. | Handle | `job_type` | `id` | Available lookup | | --- | --- | --- | --- | | SQL-backed | `"query"`, known at submission | Connection-scoped query UUID string | `describe_query(job.id)` | | Registry-backed | Existing server-reported kind, after `status()` or `refresh()` | Registry job ID string | Registry job APIs, descriptions and events | | Local | `None` | `None` | The current in-process handle | | Job concept | SQL-backed behavior | | --- | --- | | Submission | Await initial SQL submission before returning the handle. Submission errors propagate immediately. | | `status()` | Tracks the adapter task: running during result consumption/finalization, finished after both succeed, failed if either fails, or cancelled after cancellation. | | `wait()` | Returns the operation's typed terminal result. One task consumes the query reader once; repeated waits reuse its cached outcome without resubmitting SQL or reopening the reader. | | `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported cancellation leaves the task running; an accepted request stops the task. Cancelling a finished operation is a no-op. | | Registry details | Query-backed handles expose no registry description, event history, pause/resume or reattachment after a client restart. `refresh()` updates cached lifecycle state. | The Job tracks command completion rather than exposing SQL batches. A wait timeout does not cancel the query. Query IDs remain scoped to the submitting connection and cannot be used with registry APIs or as durable resume tokens. Python `describe_query()` accepts both UUID objects and the string `id` exposed by query-backed jobs. Rust, Python and TypeScript expose the same classification through their existing job-type accessors. ### Materialized-view operations - **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace and optional auto-refresh settings. After command completion, resolve the job to `None`. Synchronous creation waits on the job, then opens and returns the materialized view. - **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same adapter. After command completion, read the refreshed table's version and row count to construct the existing typed refresh result. Synchronous refresh waits on the same path. Remote creation requires the SQL service. Execution placement remains a server decision. REST endpoints are unchanged. The job guide documents the ID/job-type contract and query-backed limitations. Regression coverage exercises SQL submission, namespaces, refresh options, job-type/ID preservation through typed result mapping, repeated waits, failures and cancellation rejection. No local builds or tests were run. |
||
|
Jack Ye
|
24c2669a50 |
feat(sql): follow adaptive streaming and polling responses (#4664)
SQL execution now starts with one Flight call. The server can stream
rows immediately or return a polling continuation, which the SDK follows
through the existing query handle and reader APIs. The same behavior
applies with and without parameters.
Both initial transports share direct-stream lifecycle handling. Handoffs
are explicitly negotiated; existing servers can continue returning
ordinary streams. Add coverage for streaming, deferred results,
cancellation, malformed decisions, and legacy responses.
```mermaid
flowchart TD
A[execute_query or execute_query_async] --> B{Parameters supplied?}
B -->|No| C[DoGet: SQL command ticket]
B -->|Yes| D[DoExchange: SQL and Arrow parameter row]
C --> E[Server chooses execution path]
D --> E
E -->|Stream| F[Initial response: streaming decision and Arrow batches]
E -->|Defer| G[Initial response: PollInfo continuation]
G --> H[SDK follows PollFlightInfo]
H -->|Still running| H
H -->|Result endpoints available| I[SDK fetches tickets with DoGet]
F --> J[Query reader yields Arrow batches]
I --> J
```
`execute_query` opens the reader; `execute_query_async` returns its
query handle. The caller does not select the transport or execution
path.
|
||
|
Jack Ye
|
6b6778e684 |
feat: infer lazy blob reader Function inputs (#4663)
Infer Blob V2 input access from the Python Function signature. `video:
BlobReader` requests a lazy, seekable reader; `video: bytes` requests
the complete payload. Users do not need a separate `blob_inputs`
registration option.
## Python experience
For a video table joined to a table of frame timestamps, the Function
can pass its reader directly to a media decoder:
```python
import io
import av
from lancedb import BlobReader, udf
@udf(pip=["av", "Pillow"])
def extract_frame(video: BlobReader, timestamp_s: float) -> bytes:
with av.open(video) as container:
stream = container.streams.video[0]
container.seek(int(timestamp_s / stream.time_base), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time >= timestamp_s:
output = io.BytesIO()
frame.to_image().save(output, format="JPEG")
return output.getvalue()
raise ValueError("No frame at or after the requested timestamp")
```
After registering `extract_frame`, the query keeps the usual Function
call:
```sql
SELECT f.video_id, f.timestamp_s,
extract_frame(v.video, f.timestamp_s) AS image
FROM videos AS v
JOIN frames AS f ON v.id = f.video_id
```
The returned JPEG can feed a subsequent image-model step. The reader
provides `read`, `readinto`, `seek`, `tell`, `size()`, and
`read_range(offset, length)`; range reads leave the sequential cursor
unchanged. Each row invocation owns its reader, which closes after the
call. Decoder seeks can request the parts of the MP4 needed to locate
and decode the requested frame; the amount read depends on the file
layout and codec.
Functions that need the whole value can keep a bytes signature:
```python
@udf
def video_length(video: bytes) -> int:
return len(video)
```
## Execution contract
- Signature inference maps `BlobReader` to a Blob V2 field and records
`blob_access="reader"`. `bytes` keeps its binary Arrow type and records
`blob_access="bytes"`; it can also bind to a Blob V2 source. Explicit
schemas support the same choice, including a Blob V2 field annotated as
`bytes`.
- Reader execution carries a compact blob reference into the Function
worker. Payload reads happen when Python calls the reader, with requests
split into at most 1 MiB ranges. The full video does not need to enter
the Arrow input batch just to invoke the Function.
- Bytes execution materializes the full source before the Function runs
and remains subject to eager-input memory admission. Calling
`reader.read()` without a size also materializes the remaining contents
in Python; lazy access depends on how the Function consumes it.
- Input access metadata survives Python/Rust Function serialization and
registration. Packaged Functions import the worker-provided reader
implementation. This PR defines the SDK contract; execution requires
corresponding server support.
- This changes blob inputs. The example still returns ordinary image
bytes; it does not add streamed outputs or run an image model.
Adds coverage for signature inference, explicit schemas, nullable
readers, signature round trips, seeking, bounded range reads, buffering,
and invalid responses.
|
||
|
alexcastgonzandXuanwo
|
8cf115b081 |
fix(python): replace the retired gpt-4-turbo-preview default in OpenaiReranker (#4613)
OpenAI shuts down `gpt-4-turbo` and its snapshots on 2026-10-23 (https://developers.openai.com/api/docs/deprecations), so `OpenaiReranker` fails for every user who relies on the default `gpt-4-turbo-preview` after that date. This changes the default to `gpt-5.6-sol`, the replacement OpenAI names for `gpt-4-turbo` on that page, in the class, its docstring and the integration docs page. `_rerank` also stops sending `temperature=0`: the GPT-5 models only accept the default temperature, so keeping it would fail the new default on its first call. The reranker still asks for `json_object` output and the rest of the request is unchanged. The new test calls `_rerank` with a mocked client and checks that no `temperature` is sent. It fails on main and passes with the change. No request was made to OpenAI. This change was made with Claude Code. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
KarthikSandXuanwo
|
702389654e |
fix(python): align pylance extra and preview install indexes (#4556)
Fixes #4248. The preview index still contains pylance 1.0.0b6. uv's default first-index resolution can select it, or backtrack to an old LanceDB release when installing the extra. Meanwhile, the extra still permits pylance 5 even though current main bundles Lance 14. Raise the optional dependency to `pylance>=14.0.0b12`, the version already pinned in the test extra, and keep the lockfile metadata consistent. Document both Fury indexes and uv's `unsafe-best-match` strategy in the Python README and the source-owned quickstart (`docs/web/quickstart.mdx`). Explain the index trust requirement and recommend installing the extra to enforce the version floor. Add CI checks that reject older Lance majors and ensure the pylance and tests extras remain compatible. Run these checks when either dependency manifest changes, so a future Lance major bump cannot silently leave the Python minimum behind. ### Validation - Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects `pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and the unexpected `blob_handling` scanner argument using a real blob table. - Resolved the modified `python/pyproject.toml` with the pylance extra against both indexes: selects `pylance==14.0.0b18`, matching the current Rust pin. - Ran both documented pip and uv installations in isolated environments: `lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both package dependency checks pass. - New regression fails on the original dependency bound (27 rejected-version cases incorrectly accepted), then passes with the fix. Full CI-script suite: 3 tests pass. - `uv lock --check` with both indexes passes; focused Ruff checks pass; `git diff --check` passes. - Repository-wide Ruff reports 49 existing errors in untouched `docs/web-tests` files. Unrelated formatting changes were discarded. Runtime smoke checks used released native wheels; source dependency resolution was tested directly from the modified pyproject. No Rust or Python runtime code changes. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 LuQQiu
|
b762ec221b |
docs: add Enterprise distributed search reference page (#4625)
Adds the Enterprise reference page at `docs/web/enterprise/distributed-search.mdx`, following the same arrangement as Enterprise architecture. The page describes distributed search and the embedded execution model, with links to shared search guides. Enterprise architecture links to it, and the official sidebar lists it immediately after Architecture in the existing Deployment group. Adds stable anchors for the RaBitQ and `approx_mode` sections linked by the Enterprise guide. Companion: https://github.com/lancedb/sophon/pull/8986 supplies the detailed Enterprise overlay at the same path. Merge this reference-page PR first so the overlay has an existing route to replace. Validation: - Combined-site assembly with both PR branches passed (309 files). - Mintlify build validation and link, anchor, and redirect checks passed. |
||
 Colin Patrick McCabe
|
631ea41a63 |
feat!: return lazy iterators for table and namespace listings (#4599)
Make table and namespace listings consistent with other resource
listings across Rust, Python, and TypeScript. Callers can iterate over
individual names across all pages, with configurable page sizes and
continuation tokens.
In order to avoid breaking existing Python users, this change adds some
compatibility methods to the returned Python iterators. Code such as:
```
for t in db.list_tables().tables:
print(t)
```
will continue to work as expected because the iterator has a `tables`
field and a `page_token` field, which function as expected to reveal the
first page of results. In other words, it is never necessary to use the
iterator __as an iterator__. It may be used as if it were an old-style
response object. (As the iterator progresses, the `tables` list may
switch to the second, third, etc. page of results.) Similar
compatibility code allows the async python API to keep working as well.
|
||
|
lancedb-gatefixer[bot]andXuanwo
|
f5f4463b15 |
fix(node): materialize nested struct blobs in bytes mode (#4611)
`blobMode: "bytes"` left blobs inside structs as descriptors because projection matching ignored parent structs and byte replacement only rebuilt top-level fields. Expand direct struct projections and aliases to their blob leaves, then recursively replace the corresponding Arrow fields and data with `LargeBinary`. Track literal output names separately from descendant paths so scalar aliases stay intact and every overlapping blob projection is converted. Preserve null structs, null and empty blobs, sibling fields, schema metadata, and the existing row ID and snapshot behavior. Blobs inside lists retain their descriptors. Update the API docs to describe struct support. Regression coverage includes full and parent-struct projections, leaf and struct aliases, dotted output names, overlapping projections in both orders, nested structs, nulls, empty blobs and results, multiple batches, list siblings, query reuse, row IDs, and concurrent writes. Regression tests reproduce the nested-blob and projection-collision failures before the corresponding changes. Validation: - `pnpm build` - `pnpm lint` - `pnpm run docs` - `pnpm test __test__/nested_blob.test.ts __test__/blob.test.ts __test__/query.test.ts --runInBand --verbose=false` — 142 passed - `pnpm test __test__/table.test.ts --runInBand --verbose=false -t 'when dealing with blob columns'` — 60 passed Fixes #4609 <!-- lance-gatekeeper-fix:v1 agent=76edaac77d044c468d569b58efa2104a generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Ishita Singh
|
e5873d9763 |
fix(node): support nested blobs with blobMode 'bytes' (#4610)
Closes #4609. `blobMode: "bytes"` only resolved top-level blob columns, so a blob nested in a struct came back as its descriptor. Now struct projections that contain blob columns are resolved too: the bytes are fetched per blob column and the struct is rebuilt with `LargeBinary` leaves, preserving struct validity so a null struct stays null. Sibling fields pass through untouched, aliases work for both `info.image AS image` and `info AS details`, and blobs nested in lists still keep their descriptors. Tested with `npx tsc`, `biome check`, and the full `table.test.ts` suite (580 tests, including new regressions for select-all, struct selection, aliases, null structs, null and empty blobs, sibling fields, empty results, and single-row batches). Not run against remote tables. |
||
 Captain
|
a6011dcd95 |
fix(python): accept NumPy text arrays in Ollama embeddings (#4605)
`OllamaEmbeddings.compute_source_embeddings(np.array(["hello", "world"]))` raises a validation error before sending a request. The embedding interface accepts NumPy arrays, but the adapter passes the array directly to Ollama's string-or-sequence request field. Convert NumPy text arrays to lists before the batched request. This preserves input order, the single request, model options and `keep_alive`. The tests cover Unicode/object/sliced arrays, empty input, existing list/Arrow/query inputs and API error propagation. The added input-contract tests also cover two-dimensional arrays and byte strings, with equivalent list inputs as controls. The real Ollama SDK rejects 2-D inputs and invalid UTF-8 locally, before transport. Valid UTF-8 bytes are decoded normally. No extra adapter validation is needed for these cases. Validation on Python 3.12.14 / macOS arm64: - Built the Rust/PyO3 bindings using the repository's uv/maturin environment. - The initial regression run had 5 failures and 4 passing controls on the base; all 9 passed with the fix. With the 6 additional input-contract cases, the related embedding and trust suites now total 55 passed / 11 skipped. - All Python Ruff lint and formatting checks pass. Root checks still report the same 49 lint errors and 11 documentation formatting files reproduced on the untouched base. - Separate review exercised 13 public API cases. Another 7 groups passed through a real loopback HTTP transport, including local validation, UTF-8 serialization and empty-batch model-error propagation. The regression tests use the real Ollama 0.6.2 SDK with HTTPX MockTransport. The loopback check uses an HTTP surrogate, not an Ollama daemon. No live model inference was run. A live Ollama server was unavailable locally. [The current server handler](https://github.com/ollama/ollama/blob/2cd63f7a15039ccd790e97b31edc091fa1707c56/server/routes.go#L1212) explicitly returns HTTP 200 with empty embeddings for empty input after model lookup and runner setup. The adapter preserves the existing empty-list request and its model/setup errors. This is a source check, not a live-server test. Developed with OpenAI Codex, with separate agent review and executed local checks. |
||
|
Captain
|
f2c6aa35c2 |
fix(python): normalize VoyageAI image modes before JPEG encoding (#4576)
PIL images, PNG bytes and local image paths in VoyageAI source embeddings are saved directly as JPEG. RGBA, grayscale-with-alpha (LA) and palette (P) images therefore raise `OSError: cannot write mode ... as JPEG` before the provider is called. Convert to RGB at all three JPEG encoding sites. The caller’s original PIL image remains unchanged. Document that alpha is discarded and callers should composite onto a chosen background when it matters. Validation: - New image regressions + existing VoyageAI tests: **18 failed / 26 passed before → 44 passed after**. Tests cover both multimodal models, PIL/bytes/Path, RGBA/LA/P/RGB, decoded JPEG pixels and invalid images. - Independent local LanceDB table ingestion smoke: three PNG modes stored with 1024-dimensional vectors, original bytes preserved. Uses the freshly built current native extension. - Adjacent local embedding tests: 15 passed, 1 skipped (no OpenAI key), 9 deselected. The separate Flickr download test failed with `RemoteDisconnected`; it is excluded from the local result. - Pyright: 0 errors; CI-pinned Ruff 0.9.9 lint and Python formatting passed. External VoyageAI calls are stubbed; real Pillow conversion and LanceDB ingestion execute locally. No live provider or full-repository test result is claimed. AI assistance was used for implementation and validation. |
||
|
Captain
|
0cb5bd0a5d |
fix(python): accept VoyageAI text batch containers (#4558)
Direct VoyageAI `compute_source_embeddings()` calls reject `list[str]` and NumPy text batches before reaching the provider, although both belong to the declared `TEXT` input type. For example, `["first", "second"]` raises `ValueError: Input type <class 'list'> not allowed with text model.` Normalize these containers to a list before the existing string-entry validation. The regression cases exercise the public embedding API with a stub provider: input order, list/NumPy batches, Arrow controls, empty batches, and rejection of non-text entries and ordinary two-dimensional arrays. The VoyageAI integration page now documents the accepted text containers. Verified on current main `7b3ba4c` with a locally compiled native extension (0.41.0b7, Python 3.12.12): - VoyageAI suite: 7 failed / 20 passed before the fix; 27 passed afterward. - Adjacent embedding suite: 13 passed, 4 skipped, 8 slow cases deselected. - Python CI's pinned Ruff 0.9.9 lint/format checks pass; configured Pyright reports zero errors/warnings; all applicable pre-commit hooks pass. The provider is stubbed; no live Voyage API request or model-quality evaluation was performed. AI disclosure: I used Codex for investigation, implementation, tests, review, and drafting this description. |
||
 Aviral Saxena
|
d826b6115a |
fix(rust): explain client-side model files when remote tokenize fails (#4519)
Addresses option 2 of #4440. tokenize() builds the FTS analyzer on the client. For jieba/* and lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and when they’re missing you only got lance’s raw Invalid directory path: ... error, even though server-side search works. Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps the original cause and adds that tokenize() runs on the client, which directory the files need to be in, and that for remote tables they must match the server’s. I also updated the tokenize docs (Rust, Python, TS, generated JS docs) to say a mismatch can’t be detected, so tokens may differ from the server’s index. Behavior when the files exist is unchanged. Detecting a mismatch needs a server-side tokenize endpoint (option 1), so I left it out. The hint isn’t limited to remote tables, since Table::as_native is slated for removal and the wording fits local tables too. Tests: Rust test with mocked jieba/default index_details and an empty model dir, one checking other tokenizer errors don’t get the hint, and Python sync/async tests. |
||
  Preston ZhaoandPreston Zhao
|
cc9f01ae26 |
fix(node): accept exist_ok in createTable (#4432)
TypeScript CreateTableOptions.mode omitted exist_ok even though createTable already forwards that mode to the native client. closes: #2900 --------- Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com> |
||
|
Lance Release
|
9f0f6400bc | Bump version: 0.41.0-beta.12 → 0.41.0-beta.13 | ||
|
Xuanwo
|
a5079b0fc6 | chore: update lance dependency to v15.0.0-beta.1 (#4628) | ||
|
Jack Ye
|
f9680806c9 |
feat(python): expose materialized view auto-refresh creation options (#4621)
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
|
||
|
Jack Ye
|
a4544c5472 |
feat: configure automatic refresh when declaring computed columns (#4607)
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
|