mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
28ba34998a9633b6fb513cdd4d3fdaba0dcfb3fe
73
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Xuanwo
|
28ba34998a |
docs: replace pylance calls with native lancedb equivalents (#4811)
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot integration guide with `Table.fetch_blob_files`, document how to open `.lance` datasets directly with `lancedb` alongside a `pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in installation docs that the optional `pylance` extra is only needed for engine/connector authors or format-level fragment APIs. |
||
 XuanwoandEC2 Default User
|
8d10f25121 |
feat(nodejs): expose structured job progress (#4815)
TypeScript job handles now expose structured progress through `progress()` and an optional synchronous callback to `wait()`. Snapshots preserve unknown fields and missing counters, and job listings include progress and pause state. Existing `status()` strings and no-argument `wait()` behavior remain compatible. Wait callbacks observe the existing backend poller. Observer exceptions are logged without changing the job outcome. A best-effort ETA is available only from compatible samples with positive progress, and is omitted during initialization, queueing, or pause. Part of ENT-3500. --------- Co-authored-by: EC2 Default User <ec2-user@ip-172-31-12-104.us-east-2.compute.internal> |
||
|
Xuanwo
|
1544ad65a8 |
feat: expose structured job progress to SDK callers (#4808)
Job callers currently lose the server's structured progress when describing, listing, or waiting for a job. Expose Rust and Python progress snapshots and optional wait observers using the existing backend poller. Python supports callbacks, a caller-owned tqdm-like bar, or `progress=True`; `status()` still returns a string and the default wait behavior is unchanged. Preserve unknown progress fields and phases from newer servers, and report omitted counters as unknown. Each wait calibrates a caller-owned bar from its first complete snapshot, then uses a high-water mark within the same execution epoch; a new epoch or smaller plan resets the bar. ETA is best-effort and omitted for missing, stale, paused, or incompatible samples. Newly submitted index and refresh handles forward observers through their freshness wrapper while retaining successful-wait read consistency. The Jobs guide includes draft Python/Rust snapshot and observer examples for release; TypeScript progress support remains tracked under ENT-3500. Validation on the development EC2 host: the initial 11 Python progress tests, REST fixture caller, and two Rust progress regressions passed. After fixing freshness-wrapper forwarding, rebuilt the Python extension and ran two submitted-handle Python tests plus the new Rust refresh and index regressions (one test each); all passed. Strict Clippy, rustfmt, and targeted Ruff passed. The final commit only wraps the new guide section in an MDX draft and clarifies language support. Repository-wide Ruff reports 48 existing findings in six unchanged docs/web-tests files. This publishes the SDK portion of ENT-3500. Server-side live-split and terminal-progress changes and the combined server/SDK end-to-end scenario remain in progress. Refs ENT-3500. BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress: None`, `phase: None`, and `paused: false`. Literals for `JobDescription` must add `progress: None`, `phase: None`, `queue_reason: None`, `waiting_for_job_id: None`, and `paused_at_millis: None` when that metadata is unavailable. Existing Python calls remain compatible. |
||
|
Xuanwo
|
ad70597435 |
feat(python): expose Function batch size and row timeout controls (#4807)
Python UDF registration could not set `batch_size` or `row_timeout_seconds`, and the Rust registration model discarded these fields before submitting SQL. Expose both optional controls and preserve them through registration, with invalid values rejected and unchanged canonical bytes when the controls are omitted. The timeout applies to each Python call: one row for scalar Functions and one batch for vectorized Functions. Initialization keeps its existing service deadline. Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust registration-contract tests, 2 outbound-SQL tests, and all 103 Python slice2 tests passed after rebuilding the native extension. Rust formatting and the repository Python CI checks (Ruff 0.9.9, python/ scope) passed. A real REST + Flight SQL server integration also passed using the rebuilt SDK: `batch_size=2` limits observed calls in a direct SELECT over 8 rows, initial materialized-view creation, and append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime passthrough; it does not exercise GPU initialization. Refs ENT-3246. The original GPU initialization and video-decoding scenario still needs verification on a current deployment; this SDK change does not close that investigation. The additional repository-root style checks report existing docs/web-tests findings (nine format files and 48 lint diagnostics); all affected files are unchanged from main. These are outside the Python CI job scope. Rust compatibility: callers constructing `FunctionRegistrationRequest` must initialize the two new optional fields (`None` preserves defaults); the request no longer implements `Eq`. The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds creation-guide documentation and integrates main’s namespace repr/optional OCI embedding changes. The Function registration implementation is unchanged. All 103 Python slice2 tests passed again on EC2 using the unchanged native extension; Rust and server-bridge evidence remains applicable. |
||
|
Xuanwo
|
2d70bec53a |
feat: add UTM-tagged docs links to SDK warnings and enterprise-only errors (#4797)
Add a central documentation link registry (`lancedb::docs::DocLink`) in the Rust core that builds `docs.lancedb.com` URLs with UTM parameters (`utm_source=lancedb_sdk`, `utm_medium=warning|error`, `utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any `#anchor` fragment and without user-identifying data. - Initialize `utm_content` per language binding (`python`, `nodejs`, `cli`, defaulting to `rust`) and expose the link builder to Python so `DBConnection` and `Job` stubs reuse the core definitions. - Append published documentation links to Enterprise-only `NotSupported` / `NotImplementedError` errors across database, job, catalog, and table operations. - Emit rate-limited maintenance warnings (once per table per process when `warn` logging is enabled) when opening a native table with at least 100 small fragments (`< 100k` physical rows each) or at least 1,000,000 unindexed rows on an optimizable user index, reading only manifest metadata without opening data or `_indices/` files, and document the thresholds in the compaction and reindexing guides. - Verify published/in-repo `docs/web/` targets and `#anchor` fragments in unit tests and include extracted SDK links in the scheduled `docs-link-check` workflow. Linear: OSS-2772 |
||
|
Xuanwo
|
77af3bd746 |
fix(python): warn when Functions code packages omit resource files (#4804)
Packages passed through `@udf(code=[...])` silently omit non-Python resources, so registration can appear successful before a worker fails to open a data file. Warn during function preparation with the omitted paths and document the source-only boundary next to `code=`. The diagnostic explains how to install a distribution that includes the data and remove the package from `code=` to avoid shadowing that installation. Python cache files do not produce warnings. Excluded directories are reported at their boundary, preserving the existing traversal and Python-source artifact format. This implements the warning-and-documentation alternative in ENT-3492; it does not add resource uploads. On EC2, the regression fails on baseline `f9389f113` because no warning is emitted; all 60 packaging tests pass at `e23445b96`, using the matching native extension built from unchanged Rust sources. CI-scoped Python lint with Ruff 0.9.9 passes. Whole-site Mintlify validation reports the same four snippet-import warnings on the baseline and this head; repository-root Ruff likewise reports the same 48 findings under `docs/web-tests`. These checks are not clean passes, but their failures do not involve this change. Fixes ENT-3492 |
||
  LeoandLeo
|
f2bcb156ad |
feat: expose approx_mode on vector queries
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing. Co-authored-by: Leo <leo@insight.co> |
||
 Srijan Keshri
|
90b2b264d9 |
feat(python): accept DataFrames and readers in add_columns
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader. Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them. SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits. Fixes #1959. |
||
 Wyatt Alt
|
8ee7698545 |
feat(python): allow Blob v2 under a list in Function outputs
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
|
||
 lancedb-gatefixer[bot]
|
7466dc9d1d |
fix(python): reject embedding source updates without vectors
Fixes #4761 `Table.update` does not compute embeddings, so changing an embedding source column could leave the previous embedding in place. Source-column updates now raise `ValueError` unless every associated vector column is supplied in the same call. The guard reads Python embedding metadata and native `lancedb::column_definitions` without loading embedding models. It uses Lance's field-path parser for quoted targets and covers synchronous and asynchronous local and remote tables. Supplying every associated vector column, vector-only updates, and unrelated updates remain supported. This is a breaking change for source-only updates. The API documentation and Update rows guide explain the requirement and the `merge_insert` alternative for regenerating embeddings. Validation at the reviewed head covered 44 targeted public update cases and the full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable exact-head checks and workflows completed successfully. |
||
|
lancedb-gatefixer[bot]
|
c18aa7b033 |
fix: reject renaming or dropping embedding columns
Renaming or dropping an embedding source or vector column leaves schema metadata pointing at missing fields, causing later `add()` calls to fail. Reject these operations in local schema evolution before committing any changes. The guard reads Python/Node embedding column metadata and native Rust `lancedb::column_definitions` without loading embedding models. It supports explicit and default Rust vector-column names and respects quoted column paths. Unrelated column changes and nullability updates remain supported. The restriction is documented in the Python, Rust, and TypeScript APIs. Regression coverage includes both SDK metadata formats, native Rust definitions with explicit and default destinations, atomic rejection of batched changes, and continued embedding generation after reopening. Fixes #4763 Validation at the exact PR head and GitHub test-merge: - Hosted macOS Python wheel suite: 3061 passed, 162 skipped, 70 deselected, including all 8 sync and 4 async embedding schema-evolution cases. - Hosted Windows Rust suite: 1900 library tests passed, 8 embedding integration tests passed, and 107 doctests passed, including the native explicit/default destination and binding guard cases. - Hosted Linux Python 3.10 rerun: 3084 passed, 159 skipped, 50 deselected; the previously observed remote-fork timeout passed on the replacement job. - All 10 applicable exact-head workflow runs completed successfully. The existing main-only Windows ARM job was skipped by its workflow condition. |
||
|
lancedb-gatefixer[bot]
|
ea9194a7f3 |
fix(python): retry transient job polling errors
Fixes #4773. Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately. Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation. |
||
  TaylorandBrendan Clement
|
9e65bd2c27 |
docs: use Enterprise terminology across website and SDK documentation (#4787)
Use “LanceDB Enterprise” throughout the staged website guides, SDK reference sources, generated API pages, and user-facing SDK messages. A separate branded “LanceDB Distributed” name adds terminology without clarifying behavior; describe where operations run directly. - Update `docs/web`, including the architecture diagram title and alt text. Use Enterprise in headings, table labels, and prose; simplify redundant product/execution-model wording and name the product only once in the diagram alt text. - Update Python docstrings and the Python/Java reference pages, TypeScript comments and native declaration comments, and Rust API documentation. Regenerate the TypeScript reference, including 24 replacements across six pages. - Update SDK warnings/errors and existing test expectations to match, along with contributor guidance and Vale rules. Remove the obsolete rule distinguishing Enterprise from Distributed. - Keep lowercase “distributed” for technical descriptions. API identifiers, anchors, asset paths, and execution behavior are unchanged. No branded-name exceptions are proposed. Validation: - Python and Node native builds succeeded; focused Python tests passed (36 passed, 4 skipped), and focused Node tests passed (14). - TypeDoc generation and the complete MkDocs reference build passed. MkDocs retains an existing unresolved `lancedb.functions.SecretReference` cross-reference. - Rust formatting, Node lint/format, changed-file Python lint/format, 32 Vale tests, and whitespace checks passed. Repository-wide Python lint has 48 existing findings in six untouched files. - Combined documentation assembly passed. Mintlify validation and link/anchor/redirect checks previously passed for both site builds; no code snippets changed. Companion PRs: [saved editions, terminology guide, and shared rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview and writing rules](https://github.com/lancedb/sophon/pull/9511). --------- Co-authored-by: Brendan Clement <brendan.clement@lancedb.com> |
||
 Yang Cen
|
2d86532de9 |
docs(search): clarify auto probing and query tuning (#4796)
The vector-search guides do not clearly explain when to use Auto, search effort or a fixed probe count. They also describe `minimum_nprobes` as the exact starting count, although Auto can search more partitions from the start. Explain how to choose public query controls for recall and latency, with short executable examples. Clarify that probe limits apply per segment, calibration targets are not recall guarantees, and more probes cannot remove quantization error. Keep necessary client/server version guidance in a short note. Validation: - The existing example test passes against published `lancedb==0.41.0`, covering all four snippets, result counts, filtering, finite distances and recall against an exact cosine query. The test body takes 0.20 s. Generated snippets match their source. - Vale reports no findings on added prose. Changed-function Python formatting and `git diff --check` pass. Repository-wide Ruff has 137 findings identical to the previous scan, including seven pre-existing findings in the example file. - Mint 4.2.888 strict validation reports the same four `RsExampleBody`/`RsExampleTail` import warnings as pristine base `0b20965d`, with no additional warnings. |
||
|
Xuanwo
|
63aa06ef7d |
feat(python): expose Function GPU classes and placement (#4791)
## Summary - Add sync/async `DBConnection.list_gpu_classes()` returning typed `GpuClass` records for `GET /v1/gpu/classes`. Records include the class name, Kubernetes resource, GPU model and memory, per-worker GPU maximum, and worker CPU/memory/ephemeral-storage limits. - Add `udf(gpu_class=...)` for immutable Function-level GPU placement. It requires `gpu=True`; Function versions expose their registered `gpu_class`. - Add `gpu_class` and `gpu_count` to `refresh_column` / `refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as top-level `backfill_column` fields and validated before any request. `gpu_count` is the aggregate GPU budget across Function workers. - Validate GPU class names with the server lowercase 63-character DNS-label rule. - Document the discovery-to-registration flow, configured hardware versus free capacity, System Usage permission, and per-refresh GPU placement in the API and web guides. Older servers without `/v1/gpu/classes` produce a clear `NotImplementedError` instead of a bare 404. Part of ENT-3470. The client follows the in-flight Sophon contract on `xuanwo/ent-3470-fix` for GPU class discovery and backfill placement options. ## Breaking change `PythonRuntimeSpec::PythonV2` adds `gpu_class`, and `RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers constructing either value with all fields explicitly must supply the new fields. ## Validation At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python extension build, pyright, ruff, and Python Function/runtime, discovery, refresh placement, and refresh regressions (147 passed, 6 skipped). At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright, Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed). The final commit only updates documentation and Rust doc comments. |
||
 Jack Ye
|
eeb244fa57 |
fix: preserve all function revisions through SQL history (#4786)
`list_functions()` promises every published immutable revision. Request
`SHOW FUNCTIONS HISTORY` so the SQL server's latest-only default does
not hide older revisions, and preserve the server's numeric revision
ordering when constructing page cursors (`1`, `2`, `10`).
```python
[(fn.name, fn.version) for fn in db.list_functions(page_limit=2)]
# [("embed", "1"), ("embed", "2"), ("embed", "10")]
```
```sql
SHOW FUNCTIONS; -- latest visible revision per name
SHOW FUNCTIONS HISTORY WITH (limit = 3, include_definition = true);
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '2', limit = 3);
```
Each listing page uses the existing SQL query and polling transport; the
continuation token stores the last returned name/revision and remains
bound to its database and namespace. Function mutations keep their
query-backed job handles: `Job.id` is the query's ID and `job_type` is
`query`, so waiting observes that same SQL execution. No registry job or
second query ID is added.
Update the SQL and Python documentation for exact-revision drops:
sibling revisions and grants survive, and only the last visible
revision's removal releases the name and schedules object cleanup.
Extend the existing Flight SQL regression to paginate revisions `1`,
`2`, and `10` and verify the explicit history statement. Formatting and
static Python checks pass; local builds and tests were intentionally not
run.
|
||
|
Jack Ye
|
cbc24fa84f |
fix: decode native SQL view metadata (#4785)
View APIs now read the native `name`, `query`, and `schema` columns
returned by `CREATE VIEW` and `DESCRIBE VIEW`. The schema JSON carries
the defining database and namespace; the requested namespace identifies
where the view lives. The unreleased JSON-envelope response is removed
without a compatibility fallback.
```python
view = db.create_view("adults", "SELECT name FROM people WHERE age >= 18")
view = db.describe_view("adults")
```
```sql
CREATE VIEW "adults" AS SELECT name FROM people WHERE age >= 18;
DESCRIBE VIEW "adults";
```
Both Python calls execute SQL through the query-backed job adapter and
wait on the resulting query. A job handle uses that query's `id` and
type `query`; reading metadata consumes its completed result rather than
resubmitting the DDL. Creation returns the definition from the creating
execution, with no extra describe request.
|
||
|
Jack Ye
|
42e8e7dfca |
fix: align Secret APIs with native SQL syntax (#4784)
Secret APIs now submit the native SQL statements with the credential
bound as an Arrow string parameter:
```sql
CREATE SECRET "api-key" TYPE = GENERIC_STRING SECRET_STRING = $1;
ALTER SECRET "api-key" SET SECRET_STRING = $1;
DESCRIBE SECRET "api-key";
```
`create_secret` and `alter_secret` execute SQL through the existing
query-backed job adapter and wait for that query to complete. The job's
`id` is the query ID and its type is `query`; credentials stay out of
SQL text. `describe_secret` reads the native `name`,
`created_at_millis`, and `updated_at_millis` columns. This replaces the
unreleased `AS $1` syntax and JSON-envelope response without a
compatibility fallback.
Python methods and namespace handling remain the same:
```python
db.create_secret("api-key", "credential", namespace_path=["team"])
db.alter_secret("api-key", "rotated", namespace_path=["team"])
info = db.describe_secret("api-key", namespace_path=["team"])
```
|
||
 BHAWESHBHASKAR
|
a9146d536a |
feat(python): expose the NGram scalar index
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
|
||
 alexcastgonz
|
e9b2f7e6e1 |
fix(python): use dim as the embedding size for OpenAI-compatible models
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model names outside the three built-in OpenAI names, so schemas can be built for compatible models. Preserve existing model defaults and request behavior. Add regression coverage for the resulting Arrow schema dimensions and for all known OpenAI model dimensions. |
||
  ryan-haynesandRyan Haynes
|
d316cbbed4 |
fix!: support opt-in PKCE for device-code login (#4747)
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
|
||
 Colin Patrick McCabe
|
2044179a8b |
feat: add a CLI release binary (#4751)
Package the standalone lancedb CLI with every stable and beta GitHub release. Publish archives for Linux x86_64, Linux ARM64, and Apple Silicon macOS, along with SHA-256 checksums, licenses, and third-party notices. Synchronize the CLI version with the SDKs and validate that release tags match both manifests. Add PR and manual packaging dry runs, runtime compatibility checks, and smoke tests of extracted executables. GitHub release creation now waits for all three archives to pass validation; SDK registry publishing remains independent. Update installation instructions and the release checklist. Linux binaries target glibc 2.28, retaining existing x86_64 CPU requirements; macOS binaries target macOS 14 or newer. |
||
|
Jack Ye
|
ac7930fa77 |
feat: use SQL for remote view lifecycle
Stacked on #4675. Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination. `drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`. |
||
|
Jack YeandXuanwo
|
45e00d4de7 |
feat: use parameterized SQL for remote secrets
Stacked on #4674. Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens. The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jack Ye
|
564474f51a |
feat: use SQL for remote function lifecycle
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates. Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`. |
||
 Brendan ClementandClaude Opus 5.5
|
ca343a5c06 |
docs: fix table, accuracy and structure problems found in the pre-launch review (#4745)
## Summary Fixes the problems a page-by-page review of the docs found before launch, after checking each one against the current source and, where noted, the code. The matching fixes for the dataset and Geneva pages, which live in `lancedb/docs`, are in lancedb/docs#402. **Tables** - `integrations/embedding/cohere`: each input-type description was split across two table rows and wrapped in literal quotes (`"`search_document`"`). Now one row per type. - `integrations/embedding/gemini`: removes the same stray quotes from the task-type table. - `integrations/ai/langchain`: the MMR search table's `kwargs` row was missing a cell, so its description sat under Type. - Six reranker pages (answerdotai, cohere, colbert, jina, openai, voyageai): closes the backtick before `_relevance_score`. - `indexing/vector-index`: finishes a cell cut off at "with scalar". - `sql/dql/dedupe`: labels the empty header cell in the resolution table. **Corrections** - `curation/sampling`: sampling works in LanceDB Embedded through the Python SDK, so the page is available in both, with the SQL sections marked Enterprise. - `feature-engineering/index`, `add-computed-column`: automatic refresh is on by default for each computed column and takes effect once a deployment enables its scheduler. `refresh-column` already said so; these two implied it was off. - `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates `array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also merges scalar-index's two "nested fields" sections into one. - `integrations/embedding/ibm`: lists the four current models from `CURRENT_MODELS` (it listed a deprecated one) and explains that the default stays a legacy ID so existing tables reload with their model. - `integrations/reranking/openai`: the class is `OpenaiReranker`, not `OpenAI()`. - `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit` throughout, each step runs one command, each step passes the file it writes to the next, and `save-as` uses `-f`/`-o`. - `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the pipeline reads, and the run command runs the file the page creates. - `search/vector-search`: the multivector walkthrough connected to Enterprise in step 1 and a local path in step 2. It now runs locally throughout, with a note on connecting to Enterprise, and is renamed so it doesn't read as a duplicate of the short section above. **Structure and wording** - `integrations/embedding/nvidia`: adds headings, and drops test-suite details aimed at contributors. - `integrations/ai/genkit`: makes the retrieval test a real heading and describes the query step. - `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a `<Note>`. - `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading, not bold text. - `curation/windowing`: sidebar title matches the curation overview ("Rank and Deduplicate"). - `deployment/storage`, `basics/agentic-skills`: fixes a garbled sentence and a stale reference to the quickstart. **Checked:** no new Vale alerts on the changed pages. Assembled together with lancedb/docs#402, the site passes `mint validate` and `mint broken-links --check-anchors --check-redirects`. Every changed page was reviewed in a local preview. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colin Patrick McCabe
|
301ccad3aa |
feat: add a LanceDB command line (#4600)
Add a LanceDB command line with the following functionality: - `lancedb auth`: log in with OAuth, check cached OAuth session status, or log out. - `lancedb profile`: Create, edit, list, show, select, and delete connection profiles. - When no subcommand is given, we will attempt to create a SQL shell. Update ci/check_cargo_lock.py and its regression tests to allow pruning dependency edges while their target packages remain in Cargo.lock. Removing the CLI from the Python source distribution can disable optional features such as Axum tracing, even though other SDK dependencies still require the tracing package. |
||
|
Brendan ClementandClaude Opus 5.5
|
2347e97e56 |
docs: list features in the landing page comparison table (#4743)
## Summary Replaces the generated availability table under **Compare features** on the landing page with a hand-written list of features, grouped by area, with separate Embedded and Enterprise columns. The generated table listed every page that declares `availability`, including overview and basic CRUD pages. Rows such as "Overview" appeared three times without context, and pages like Insert, Update, Database and the integrations showed up as if they were features. | Area | Features | | --- | --- | | Search | Vector, full-text and hybrid search, filtering, reranking, query plans, distributed search | | Indexing | Vector, full-text and scalar indexes | | SQL and curation | SQL, aggregation, sampling, joins, ranking and deduplication, near-duplicate detection, chunking, changes since a version, views | | Feature engineering | Computed columns, materialized views, embedding registry | | Training | PyTorch data loader, permutations, distributed training | | Data management | Branches, compaction and cleanup, background jobs, Real Time engine | | Security | Authentication, authorization | Each ✅/❌ matches the linked page's `availability` on `main`; SQL, which declares none, is Enterprise-only. Functions are left out until it's settled whether they are published. The same list, limited to 1.10 features, is in the Enterprise 1.10 editions in lancedb/docs#398. Checked by assembling the docs site with this change: `mint validate` passes and `mint broken-links --check-anchors --check-redirects` finds no broken links. Vale reports no new warnings. After this merges, the committed `lancedb` skill in lancedb/docs needs a rebuild (`make assemble && make skill`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brendan ClementandClaude Opus 5.5
|
dcddd541c8 |
docs: link home page sections through /index (#4740)
## Summary Changes links to home page sections from `/#<anchor>` to `/index#<anchor>`. The docs site's Assemble CI fails `mint broken-links --check-anchors` on every `/#architecture` link since #4732 moved the Architecture page onto the home page. The anchor exists; Mint 4.2.888's checker can't resolve any anchor on the root page `/`, but resolves `/index#…`. `docs.lancedb.com/index` serves the home page, so readers land in the same place. Verified by assembling the site with this change and the matching Enterprise docs change: `mint validate` passes and `mint broken-links --check-anchors --check-redirects` finds no broken links. The `docs.json` redirects keep `/#architecture`, which the checker accepts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brendan ClementandClaude Opus 5.5
|
1d15bb3ada |
docs: address staging review comments (#4715)
## Summary Addresses some comments on the staging branch. Rebased onto #4732, which rebuilt `index.mdx` and the quickstart and reverted the landing copy to the pre-v0.4 headings; the final copy is reapplied here. | Comment | Page | Change | | --- | --- | --- | | Taylor: drop the comma | `index.mdx` | "one, unified platform" → "one unified platform" | | Taylor: "Preparing t…" + "Data doesn't slow down" | `index.mdx` | "Training data and experimentation slow down…" → "Preparing training data and running experiments slow down…" | | Fei: subtitles should align with the new home page | `index.mdx` | Follows the final homepage copy (v0.4): "Train on the right data faster" with the hero subtitle, "Accelerate the data flywheel for model research" with its intro, and the five v0.4 sections as steps, each linking to the page that backs it (search, curation, branches, versions, computed columns and materialized views, `StreamingDataset`). Wording the docs can't back yet is left out: "zero-copy", "lineage", "cached across clouds". Anchors renamed to match; nothing links to the old ones | | Fei: quickstart seems OSS-only | `quickstart.mdx` | Tip at the top: for Enterprise only the connection changes, linking the Enterprise connection in step 2 | Fei's "list all Enterprise features as a catalog" comment is now covered by #4732's feature comparison table (`/#compare`), so the separate table this PR added to `basics/offerings.mdx` was dropped along with that page. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brendan ClementandClaude Opus 5.5
|
8440574f2a |
docs: use the new overview diagrams on the landing page (#4714)
## Summary Updates the overview diagrams on the landing page. Rebased onto #4732, which moved the architecture and offerings content into `index.mdx` and drew it as Mermaid diagrams; this PR replaces those with the guide's diagrams. - **Architecture** (`#architecture`): the suite diagram (`lancedb-suite.svg`) replaces the first Mermaid diagram. Its Enterprise boxes read Search, Data exploration & curation, Feature engineering, Training, matching the guide's product hierarchy. - **LanceDB Embedded** (`#1-lancedb-oss`): adds the guide's Embedded diagram. Its storage box reads "Local Disk or Object Storage", since Embedded supports both. - **LanceDB Enterprise** (`#2-lancedb-enterprise`): the guide's Distributed diagram replaces the second Mermaid diagram. This also drops that diagram's "column + MV refresh" label; automatic refresh is still gated. - **Functions** are still unpublished, so the "Functions ·" label is removed from the Distributed diagram's jobs fleet. At release, restore the designer's original image from the Notion guide. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Jack Ye
|
ddb318567a |
docs: complete SQL reference, enrich Functions drafts, rename OSS to Embedded (#4732)
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table. |
||
|
Taylor
|
a065b4b9c6 |
feat(python): add NVIDIA Nemotron embedding provider
Adds a Python `nvidia` embedding provider for Nemotron 3.5 Embed 1B and 8B. It works with NVIDIA's hosted API and self-hosted NIM `/v1` endpoints, and uses the existing `openai` client. The OpenAI provider can't send NVIDIA's passage/query input types or pick Nemotron's dimensions. - Documents are embedded as `passage`, searches as `query`. - Default dimensions are 2048 (1B) and 4096 (8B), and `dim` is checked against each model's supported range. Self-hosted model IDs need an explicit `dim`. - Response count, order, dimensions and values are checked. Null and empty text get a null vector without a request. - Each request is retried on its own with LanceDB's retry policy. Invalid input or responses fail immediately. - `NVIDIA_API_KEY` is only sent to the hosted API. Custom endpoints need an explicit `api_key` (a `$var:` reference), and endpoints restored from table metadata must be allowed with `allow_endpoint`. - Request dimensions are sent through `extra_body`, preserving the NVIDIA wire contract while supporting the declared minimum `openai==1.6.1`. Python and text only. Docs are in `integrations/embedding/nvidia`. Originally authored by Taylor (@tsmithv11). ### Testing - `test_nvidia_embeddings.py`: 45 passed, 2 deselected with both the locked `openai==2.44.0` and minimum `openai==1.6.1`. These use the real OpenAI client with mock HTTP transport and real local tables. - Embedding, registry, field-metadata and trust suites: 163 passed, 4 skipped, 10 deselected. - Historical live experiments: hosted 8B passed a five-request smoke test; hosted 1B returned HTTP 500. No live request was made in this stage, so current hosted 1B availability was not verified. Enterprise coverage uses the repository mock HTTP server, not a live deployment. |
||
|
BHAWESHBHASKAR
|
891774343e |
feat(python): add CombinedFieldsQuery for BM25F full text search
Add a typed Python `CombinedFieldsQuery` for BM25F search across multiple columns. The query mirrors Lance's combined-fields semantics with ordered per-column boosts and AND/OR term matching. Convert it in both directions at the PyO3 boundary, including nested Boolean queries, and document the Python typed convenience entry. Closes #4652. |
||
|
lancedb-gatefixer[bot]
|
741cb9ad4e |
fix(python): implement remote namespace and table existence checks (#4725)
Remote Python connections inherited the unsupported-operation stubs for `namespace_exists` and `table_exists`, so both raised `NotImplementedError` even when namespace and table operations worked. Add the missing remote overrides, async methods, and PyO3 bindings using the existing namespace describe and table open paths. Only an explicit not-found result returns `False`; permission, transport, and server errors propagate. This preserves authentication, dynamic headers, retries, and nested namespace handling, and rechecks cached tables before reporting that they exist. The async namespace helper also recognizes native `NamespaceNotFound` errors so missing local namespaces return `False`. The table and namespace guides document Python remote support and distinguish it from the unsupported `exists` routes used by direct REST namespace clients. Validation: - All 40 focused regression cases pass, covering successful and missing lookups, local and remote namespaces, dynamic headers, permission/server errors, invalid table identifiers, and cached tables that disappear. - The original inherited stubs reproduce the reported `NotImplementedError`; the local namespace regression reproduced the native missing-namespace `RuntimeError` before its fix. - Full remote connection suite: 255 passed. Local namespace tests: 14 passed. - `cargo clippy --quiet -p lancedb-python --features remote --lib`, `cargo fmt --all`, Python Ruff lint, and Python formatting checks pass. - Root `ruff check .` reports 48 existing errors in documentation examples under `docs/web-tests`; the Python subtree passes and those unrelated files are unchanged. Fixes #4722 <!-- lance-gatekeeper-fix:v1 agent=4d369a0681756fdb5a8b35853d7f5607 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Nishil Rathod
|
84fc48198b |
docs: add guide for using LanceDB in VS Code extensions
Closes #1849 This was lancedb/docs#371. After lancedb/docs#355 moved the open-source pages, their navigation and the snippet tests into this repository's `docs/web` and `docs/web-tests`, that PR could no longer merge there, so I've moved the same change here. ## Why #1849 asks for an example of LanceDB in a VS Code extension, because people kept running into problems there (#927, #1713). An earlier attempt, #3682, added a README link. @prrao87 asked for something else instead: a quick guide on the docs site, under **Integrations › AI Platforms & Frameworks**, showing how to use LanceDB in an extension and what results look like. The author of #3682 agreed and closed it. This PR is that guide. ## What it adds - **`docs/web/integrations/ai/vscode.mdx`**: a new page, also added to `docs/web/docs.json`. It builds a small extension with two commands. One indexes the workspace's Markdown into a LanceDB table using the built-in `huggingface` embedding function, so no API key is needed. The other searches it by meaning. The page covers the parts that are specific to VS Code: - storing the table under `context.storageUri` instead of a relative path, which is the cause of #927; - `@types/node` ≥ 22 and `skipLibCheck`, which covers #1713 and the DOM-type errors from `apache-arrow`/onnxruntime typings; - pinning `apache-arrow` inside LanceDB's peer range; - marking `@lancedb/lancedb` and `@huggingface/transformers` as esbuild externals, and removing `node_modules/**` from the template's `.vscodeignore`; - storing each file's URI so a hit opens in the right folder of a multi-root workspace; - building one VSIX per platform with `vsce package --target`, with a table mapping each target to LanceDB's native package, and a note that the sample's local model (via `onnxruntime-node`) doesn't run on Alpine; - keeping the Transformers.js model cache in `globalStorageUri`, which only works if it's loaded with `import()`; - a troubleshooting table listing each error message with its cause and fix. - **`docs/web-tests/ts/vscode_search.ts`** and **`vscode_search.test.ts`**: the LanceDB code the page shows (connect, chunk, index, search), plus jest tests for it. - **`docs/web-tests/ts/vscode_extension.ts`**: the extension's `src/extension.ts`. Jest can't run it because it needs the VS Code extension host, so it was tested in a real extension instead (see below). This follows the precedent of `integrations.ts`. - **`docs/web/snippets/vscode_search.mdx`** and **`vscode_extension.mdx`**: generated with the command in CONTRIBUTING.md. ## Changes from lancedb/docs#371 - **New paths only.** The page, tests and snippets are the same content, at their paths in this repository. - **One link.** `/tables/update` doesn't exist under `docs/web`, so the `mergeInsert` link now points to `/ingestion/merge-insert`. - **Heading anchors.** Every heading now has an explicit `{#anchor}`, as CONTRIBUTING.md asks. ## How I tested it - **Snippet tests.** In this repository at `c2c8269f`, `vscode_search.test.ts` passes (2 tests) with the pinned `@lancedb/lancedb` 0.31.0. On Windows the `npm test` script fails before it starts, because it runs `node` on the `.bin/jest` shell shim. I ran `node --experimental-vm-modules node_modules/jest/bin/jest.js --testEnvironment jest-environment-node-single-context vscode_search` instead. - **Lint.** `biome ci` is clean for the three new files, and they carry the SPDX header that `license-header-check.yml` looks for. - **Snippets.** I regenerated them with CONTRIBUTING.md's command, `mdx_snippets_gen.py -s docs/web-tests/py -s docs/web-tests/ts -s docs/web-tests/rs -o docs/web/snippets`, run with `python` since I don't have `uv`. It reproduces the two new snippet files exactly, once the Windows CRLF endings are normalised. As CONTRIBUTING.md warns, it also rewrites `build_with_ai_agents.mdx`, `integrations.mdx`, `search.mdx` and `storage.mdx`. I left those out. - **Links.** `mint broken-links` from `docs/web` reports no entries on the new page, and its three internal links (`/embedding/quickstart`, `/ingestion/merge-insert`, `/search/full-text-search`) resolve to pages here. - **Real extension.** I built a sample extension from exactly the code on the page, scaffolded with `yo code` (TypeScript + esbuild) and using `@lancedb/lancedb` 0.39.0. I ran it with `@vscode/test-cli` in VS Code 1.140 on Windows x64. The integration test runs both commands against a six-file workspace and checks the top hits. The "Sample results" block on the page is copied from that run, and after the review fixes below it reproduces line for line in VS Code 1.141. - **Multi-root workspace.** A second test configuration opens two folders that both contain `docs/auth.md`. It types a query and picks the top hit through the real picker, then checks that the right folder's file opens at the right line. It fails on the code before the review fix and passes after it. - **Packaged VSIX.** I packaged `vsce package --target win32-x64` (200 MB, or 148 MB with the onnxruntime trimming in the Tip). Then I unpacked it outside the project and ran the same tests against the unpacked copy, so they couldn't pick up the project's `node_modules`. They pass. - **Negative checks behind the troubleshooting rows:** - With `node_modules/**` left in `.vscodeignore`, activation fails with `Cannot find module '@lancedb/lancedb'` and the commands report `command '…' not found`. - Without the externals, esbuild fails with `No loader is configured for ".node" files`. - Installing LanceDB into a project that already has `apache-arrow` 21 fails with `ERESOLVE`. - `require("@huggingface/transformers")` returns a different `env` object from `import()`. - **Rendering.** For lancedb/docs#371, `mint dev` rendered the page, its 14 code blocks and the sidebar entry. I haven't re-rendered it from `docs/web`; only the paths, one link and the heading anchors changed. **Not tested:** - macOS. I couldn't reproduce the read-only working directory from #927; the page's fix (an absolute storage path) doesn't depend on the OS. - Linux and macOS VSIX builds. ## Notes for reviewers - I added `_distance` to `.select(...)`, because LanceDB warns that it will stop adding the column automatically. ## AI disclosure I used Claude Code to research the issue and its history, write the sample extension, the tests and this page, move the change here, and draft this description. I ran every check listed above myself, and the outputs quoted on the page come from those runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
 Will JonesandClaude Opus 5.5
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Wyatt Alt
|
b4cbaf2f67 |
docs: show stateful class Functions
The Functions page only showed a plain function, so it never said where a class's `__init__` runs. Users loading a large model need to know it is not the client: registration reads the signature and ships the source. This adds a section on class Functions: `__init__` runs once per instance on the worker, then `__call__` per row or batch and `close()` on retirement, state is per instance, and `__init__` parameters are initialization fields set per binding. The example's test asserts that registering the class does not construct it. |
||
|
|
659ea11ac4 |
fix(embeddings): default Cohere to embed-multilingual-v3.0, the v2.0 models are retired (#4698)
Cohere retired `embed-english-v2.0`, `embed-english-light-v2.0` and `embed-multilingual-v2.0` on April 4, 2026 ([deprecations](https://docs.cohere.com/docs/deprecations), which lists `embed-english-v3.0`, `embed-multilingual-v3.0` and `embed-v4.0` as the embedding alternatives). `CohereEmbeddingFunction` still defaulted to `embed-multilingual-v2.0`, so every call without an explicit `name` fails. The default is now `embed-multilingual-v3.0`, the multilingual v3 model, and the retired models are removed from the docstring and the Cohere integration page (the page also listed `embed-english-v2.0` as the default, which did not match the code). The docs example, its generated snippet, the docs test and the slow Cohere test use the new name. This changes the default vector size: `ndims()` returns 1024 for `embed-multilingual-v3.0`, where the old default gave 768. Tables created from a schema that uses `cohere.ndims()` get 1024-dimension vectors from now on. A table that already holds 768-dimension vectors from the retired model needs a re-embed with a current model, since the old model can no longer be called. The `ndims()` entries for the v2 names are left as they are. Test: `test_cohere_default_model_is_not_a_retired_v2_model` in `python/python/tests/test_embeddings.py` checks the default name and that `ndims()` is 1024. The repository's pytest session could not be run for this change (its `conftest.py` needs a newer native module than the released wheel); the body of that test was run against the released `lancedb` 0.40.0 with this version of `cohere.py` in place, where it passes, and it returns `embed-multilingual-v2.0` / 768 without the change. ruff check and format are clean on the touched Python files. No Cohere request was made. This change was made with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jack YeandXuanwo
|
45dbf5eff5 |
fix: use SQL for all remote materialized view APIs (#4673)
Remote materialized-view APIs now use SQL for the full lifecycle. Create and refresh already submit SQL; this change moves describe, paginated listing, and drop onto `DESCRIBE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS`, and `DROP MATERIALIZED VIEW`. Async create, refresh, and drop return the existing Job handle with `job_type = "query"`. The handle has one `id`: the connection-scoped SQL query ID, also accepted by `describe_query`. Submission starts the query, the job drains its result once, and repeated waits reuse the completed result. Synchronous drop waits for catalog removal; physical cleanup proceeds in the background. Listing uses namespace-scoped keyset pagination with opaque resume tokens. Streamed and deferred SQL responses share the same path. Adds transport coverage and updates Python/TypeScript documentation for query-backed drop jobs. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
ac99ab173b |
fix(python): support native blob conversion in pandas and batches (#4647)
Plain local blob v2 scans tried to open a pylance dataset before reaching LanceDB's byte-fetch path, so `to_pandas(blob_mode="bytes")` failed when the optional dependency was absent. Batch readers also had no `blob_mode` option. Route supported blob v2 pandas conversions through the native Arrow path. Add `blob_mode` to sync and async batch readers, convert payloads during iteration, and expose the converted schema without fetching the first batch. Apply conversion after reranking and retain the default descriptor behavior. Clean up scanner error punctuation and remove the misleading plain-scan suggestion. Bytes mode retains the existing `to_arrow` restrictions for legacy/list blobs, remote tables, and namespace query pushdown. Lazy pandas handles and legacy pandas scans still require pylance; sync FTS/hybrid batching remains unsupported. Validation: - UV-managed editable Python development build completed successfully. - The absent-pylance regression tests fail against the original query implementation and pass with this change. - Blob and query suites: 403 passed, 1 skipped, 1 deselected. - Remote query suite: 190 passed. - Python-tree Ruff lint/format checks and the async reader Pyright check pass. Repository-wide Ruff reports existing errors outside the Python tree. Fixes #4645 <!-- lance-gatekeeper-fix:v1 agent=fd85998f14bb0b3c92b2d050df93a615 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Dan Tasse
|
6e4f523e14 |
docs: remove logical-column from docs (#4693)
We've removed this feature from LanceDB Enterprise and no longer privilege the "lancedb:logical-column" key. So, removing that from these docs. |
||
|
Xuanwo
|
8a4effa06a |
fix: gate experimental local functions behind an opt-in feature (#4688)
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
e3dce2cbad |
fix(rust): prevent oversized SQ remap panics (#3861)
## Summary - preflight IVF-SQ and IVF-HNSW-SQ storage segments touched by compaction - preserve unknown legacy fragment coverage as conservatively affected - cover both fixed-size vector and multivector SQ storage dimensions - reject only affected partitions whose fixed-size-list child exceeds Arrow's `u32` take-index limit - return an actionable `InvalidInput` error before Arrow can panic ## Root cause Arrow 58 builds fixed-size-list child indices with `u32`. For a large SQ partition, `partition_rows * vector_dimension` can exceed `u32::MAX`; the child range then wraps inside `take_fixed_size_list`, producing the reported `ArrayData::slice` assertion during index remapping. The preflight now reads raw segment metadata, mirrors Lance's fragment-bitmap selection (including `None` as affected/unknown), opens only affected vector segments, and checks their actual storage quantizer and partition sizes. It derives the stored dimension for both `FixedSizeList` vectors and flattened `List<FixedSizeList>` multivectors. Stable-row-ID datasets, deferred remapping, custom remappers, unaffected segments, non-SQ indexes, safe partitions, and no-op compactions are unchanged. Because the upstream Arrow kernel still has the limit, an affected index must be recreated with default IVF partitioning (or enough partitions to stay below the reported maximum) before compaction. This changes the failure from a process panic to a targeted recovery error; it does not silently alter index structure. ## Validation - `cargo fmt --all -- --check` - `cargo test --quiet --features aws,remote -p lancedb table::optimize::tests --lib` - `cargo test --quiet --features remote -p lancedb table::create_index::tests::test_create_index_ivf_hnsw_sq` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples -- -A clippy::redundant-pub-crate` (the allowance is for pre-existing Rust 1.97 warnings in remote-table code) Fixes #2866 <!-- lance-gatekeeper-fix:v1 agent=ba2f07b1ee6f9e68a0de1961c164f70e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Drew
|
436c209580 | docs: document Blob V2 in the multimodal guide (#4623) | ||
 KarthikSandXuanwo
|
180eaaf68e |
fix!: reject unsupported remote connection and table options (#4486)
Fixes #4399. Remote connections currently discard local-only settings, including table write options that determine storage format. The synchronous Python client also binds the fourth positional `create_table` argument to `on_bad_vectors`, so an intended overwrite is sent as a create. This change aligns the remote Python signature with `DBConnection.create_table`, preserves the legacy remote timeout keywords, and raises `ValueError` for other unknown connection keywords. The shared Rust client rejects unsupported connection, create, and open options before sending HTTP requests; open validation also runs before a table-cache hit. Python reports these as `NotImplementedError`, including explicit `enable_v2_manifest_paths=False`. Supported Enterprise connection-level storage options remain available. This implements option 2 from the issue: explicit rejection. It deliberately does not add unverified server write parameters. Calls that previously supplied ignored options must omit them to use server-managed storage and caching. Database URI prefixes are already preserved on current main and are unchanged here. Regression tests cover Python sync/async, TypeScript create/open calls, and the Rust remote client, including empty-table creation, overwrite mode, and cached table handles. Python API documentation describes the behavior and corrected positional order. Reconciled with main at `0ea932da`, preserving upstream blob and error-handling tests and updating the Python test that previously expected open options to be ignored. Validation on the merged source: - Rebuilt the Python native extension; 304 tests passed across remote options, database, remote database, and session suites. - Rust remote database suite: 135 passed. Rebuilt the Node native extension; TypeScript remote suite: 62 passed. TypeScript compilation and documentation generation passed. - Full Python SDK lint and formatting pass (159 files); full TypeScript lint and formatting pass (66 files). Rust formatting and `git diff --check` pass. - Repository-wide Python lint/format reports existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust 1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci` build profile. Remote tests use HTTP fixtures; this does not claim validation against the original ARM64 single-node deployment. Documentation review follow-up: added a draft Enterprise migration note in `docs/web/basics/sdk.mdx`. It explains that `db://` connections reject local session/cache and per-table storage/write options, while supported connection-level Enterprise storage options remain available. Following the repository's release conventions, the note stays in an MDX comment below the migration list until the behavior is released. Vale 3.24.0 with the pinned LanceDB style package reports no findings; `git diff --check` passes. This follow-up changes documentation only. BREAKING CHANGE: Remote connections now reject unsupported local session/cache and per-table storage/write options instead of silently ignoring them. Callers must omit these options and use server-managed storage and caching; supported Enterprise connection-level storage options remain available. Integration with current main preserves its synchronous API signatures, including the remote `create_table(mode=None)` default, and the table compatibility defaults. The positional-order repair is already on main via #4608; this PR retains the distinct unsupported-option rejection behavior. CI follow-up: the inherited IVF search-effort test occasionally measured recall 0.375 against its unchanged 0.5 threshold after random index training. A matching local reproducer observed this once in 30 builds. The fixture now places the eight exact neighbors together, preserving the 16-partition setup, two writes, all probe/precedence assertions, and the recall threshold. The adjusted fixture returned recall 1.0 in 100 builds; the full query and remote suites passed (102 tests). This stabilizes test data and does not change search behavior or promise production recall. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
66357275bb |
fix: inherit Lance defaults for IVF RQ quantization (#4585)
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding Lance's five-bit default. The shared native index creation path also substituted one bit when the Rust or Node builder left the value unset. Make Python's bit count optional and forward only explicit values. Use `RQBuildParams::default()` for an unset native bit count, and omit unset values when dispatching to pylance for accelerator builds. Update the Python, Rust, Node, and vector-index documentation. New indexes with an unset bit count now inherit Lance's default, currently five bits, across Python, Rust, and Node. Five-bit vector codes use roughly five times the storage of one-bit codes, plus other index overhead. This also changes reader compatibility: indexes created with `num_bits >= 2` use a newer on-disk layout that some older LanceDB versions cannot read. Upgrade readers to versions that support multi-bit IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)` in Python) when sharing indexes with readers limited to one-bit IVF_RQ. Validation: - Reproduced six regression failures before the fix: one-bit default metadata, rejected `None`, and incorrect remote/accelerator forwarding. - After merging `main` through `b762ec22`, rebuilt both native bindings against Lance 15.0.0-beta.1. Python validation passed with 106 tests and 15 optional Ollama skips. Coverage includes the full index suite, omitted and explicit bit counts, reading Lance-created IVF_RQ indexes, accelerator forwarding, CUDA-unavailability handling, timeout validation, remote requests, and table/namespace listing behavior. - Rust formatting and Python package Ruff lint/format checks pass. Node `pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run docs`, the full connection suite, and the existing IVF_RQ regression pass. Generated documentation adds no further changes. The merge preserves upstream's explicit RaBitQ heading anchor alongside the `IvfRq()` example and five-bit default/reader-compatibility guidance. Repository-wide `ruff check .` still reports 49 existing errors in unchanged documentation examples under `docs/web-tests`; root formatting also finds 11 unchanged documentation example files. The Python package checks pass. Fixes #4582 <!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
7ed03e1c40 |
fix(python): honor bad-vector policies for pandas NaNs (#4503)
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN mask and Rust's default append validator ignored null children, so `add()` silently stored incomplete vectors despite `on_bad_vectors="error"`. Reject null components in the existing Rust ingestion validator, and include null components in Python's bad-vector detection and filling. `drop` removes affected rows, `fill` repairs missing components while preserving valid ones, and `null` replaces affected vectors. Explicit null components in Arrow and list inputs follow the same policies. Rust allows whole null vectors and retains the explicit `NaNVectorBehavior::Keep` opt-out. The ingestion guide and synchronous/asynchronous `create_table()` option docs describe null components, all four policies, and filling that preserves valid components. Validation: - Reproduced the default pandas append failure with the regression test before changing the Rust validator. - Rebuilt the native Python extension from the merged sources. The Python table and utility suites passed: 481 passed, 1 skipped. Coverage includes create/append/overwrite, pandas/rows/Arrow inputs, every bad-vector policy, float16/32/64, sliced and chunked arrays, all-null components, and hidden children of null lists. - All 43 focused Rust tests passed: 7 validator tests and 36 ingestion tests. - Rust Clippy with `remote`, tests, and examples passed. Ruff lint passes for the Python package, formatting checks pass for modified Python files, and `cargo fmt --all` is clean. - Documentation follow-up: all 56 focused vector-policy tests passed; Python package lint and creation-docstring formatting checks passed. Repository-wide Ruff reports 49 existing errors in untouched documentation examples. Create/overwrite retain their existing `RuntimeError` wrapping for errors raised while consuming lazy Arrow readers. Fixes #4435 <!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jack Ye
|
9af7757053 |
fix: execute remote materialized view creation through SQL (#4665)
Remote materialized-view creation currently expects a REST response containing a registry job ID, while refresh already submits SQL. Route both through the SQL query backend and preserve Python's synchronous and asynchronous return types. ### Query to Job mapping `SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through the same backend as `execute_query_async()`. Once submission succeeds, it retains the returned `Query` and starts one client-side task that consumes its result reader to completion. `Job::from_query_task()` wraps that task, its cached outcome, a single ID and a cancellation callback. Python exposes this as `Job[T]` or `AsyncJob[T]`. The existing `job_type` property maps SQL-backed handles to the new `"query"` kind. The shared `id` contains that query's UUID string, so Python callers can inspect it with `db.describe_query(job.id)` on the submitting connection. No separate `query_id`, backend enum or `type()` API is added. | Handle | `job_type` | `id` | Available lookup | | --- | --- | --- | --- | | SQL-backed | `"query"`, known at submission | Connection-scoped query UUID string | `describe_query(job.id)` | | Registry-backed | Existing server-reported kind, after `status()` or `refresh()` | Registry job ID string | Registry job APIs, descriptions and events | | Local | `None` | `None` | The current in-process handle | | Job concept | SQL-backed behavior | | --- | --- | | Submission | Await initial SQL submission before returning the handle. Submission errors propagate immediately. | | `status()` | Tracks the adapter task: running during result consumption/finalization, finished after both succeed, failed if either fails, or cancelled after cancellation. | | `wait()` | Returns the operation's typed terminal result. One task consumes the query reader once; repeated waits reuse its cached outcome without resubmitting SQL or reopening the reader. | | `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported cancellation leaves the task running; an accepted request stops the task. Cancelling a finished operation is a no-op. | | Registry details | Query-backed handles expose no registry description, event history, pause/resume or reattachment after a client restart. `refresh()` updates cached lifecycle state. | The Job tracks command completion rather than exposing SQL batches. A wait timeout does not cancel the query. Query IDs remain scoped to the submitting connection and cannot be used with registry APIs or as durable resume tokens. Python `describe_query()` accepts both UUID objects and the string `id` exposed by query-backed jobs. Rust, Python and TypeScript expose the same classification through their existing job-type accessors. ### Materialized-view operations - **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace and optional auto-refresh settings. After command completion, resolve the job to `None`. Synchronous creation waits on the job, then opens and returns the materialized view. - **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same adapter. After command completion, read the refreshed table's version and row count to construct the existing typed refresh result. Synchronous refresh waits on the same path. Remote creation requires the SQL service. Execution placement remains a server decision. REST endpoints are unchanged. The job guide documents the ID/job-type contract and query-backed limitations. Regression coverage exercises SQL submission, namespaces, refresh options, job-type/ID preservation through typed result mapping, repeated waits, failures and cancellation rejection. No local builds or tests were run. |