Expose `FunctionRetryPolicy` on Function declarations and refresh
overrides, together with `error_policy` and recorded attempt counts.
Python uses the Rust validation rules before sending a request; local
refreshes reject these Enterprise options.
The policy bounds calls and backoff, supports
class/message/provider-code matching, and can disable retries with
`max_attempts=1`. Batch retry and skip operate on the whole failed call.
Requires Enterprise server support for the new policy.
Validation: rebuilt the Python extension; 47 targeted Python tests and
10 Rust tests passed. Python `make check`, Rust formatting, and `cargo
clippy --features remote --tests --examples -- -D warnings` passed. The
additional repository-root Ruff sweep reports existing errors under
`docs/web-tests`; the Python workspace checks pass.
Draft user guidance covers the supporting server requirement, safe
repetition of partial work and whole-batch exhaustion. The guide
sections stay hidden until release. Vale 3.24.0 passed for the added
draft prose; no paired deployed SDK/server retry acceptance was run.
Companion server: https://github.com/lancedb/sophon/pull/9812.
BREAKING CHANGE: Rust callers constructing `FunctionRegistrationRequest`
or `FunctionErrorRecord` with struct literals must supply the new
`retry_policy` or `attempts` field, respectively.
- Removes the stale `/enterprise` redirect that hijacked the real
LanceDB Enterprise page, sending it to the Embedded vs Enterprise
comparison page instead.
- Removes the Merge page from the Data Ingestion nav (confusing parity
story between Embedded and Enterprise); moves RTE Write up to right
below Delete and renames it **RTE Write (Upsert)** to take over that
slot.
- Removes the Early access badge from Tiered Index Cache and restores
its full sidebar title (was shortened to "Index Cache" to fit the
badge).
- Moves Changes below Dedupe in Data Curation.
- Matches Advanced Conflict Resolution's sidebar title to its page title
(previously just "Conflict Resolution" in the sidebar).
- Marks Job Framework as Enterprise-only.
- Distributed RTE Write diagram now shows the `metrics` table split
across multiple shards, illustrating the per-table sharding described in
the page's own text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The refresh guide and SQL reference do not warn that a `WHERE` refresh
recomputes filled rows, and the guide describes resources as per-worker
allocations. Clarify the aggregate Function budget and expression-only
`num_workers`, show the output-null predicate with its stale-value and
successful-null limitations in both references, and document
refresh-level `error_policy='skip'`. Link both guides to the current SQL
options route.
Validation: Vale 3.24.0 found no issues on changed lines; `git diff
--check` and the new link targets passed. Mint 4.2.888 reports the same
four existing `RsExampleBody`/`RsExampleTail` import warnings in
`reranking/index.mdx` and `search/vector-search.mdx` on both pristine
base `5e38120b` and this branch. No additional warnings. Behavior was
checked against the owning Sophon implementation; no runtime code
changed or runtime tests were run.
Companion to https://github.com/lancedb/sophon/pull/9806 for ENT-3279.
Page visibility and saved documentation editions are unchanged. DOC-90
retains publication and matching-shipped-release verification.
## Summary
- **Define Storage Integration**: un-gates the Python (and Rust, for S3)
`create_storage_integration` examples, drafted against #4618 and held
back until it shipped -- it's now in v0.42.0-beta.0+, and the signature
matches the drafts exactly.
- **Naming rules**: drops the `Branch` row and its "Learn more" link,
consistent with de-emphasizing branches elsewhere in this nav.
- **Distributed RTE Write**: removes the REST path, the
`__lance_mem_wal` index name, and "WAL node" terminology in favor of an
abstract framing -- a query node routes writes to RTE nodes, each of
which can host shards from many different tables. Both diagram variants
are updated to match.
- **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop
Helm-specific and exact-percentage detail (disk-budget math,
`startup_prewarm_block_query_ready`/`healthPort`/failure-policy table)
in favor of a short feature → architecture → why-it-matters shape, each
closing with a contact-sales pointer. Data Page Cache's on-demand
prewarm section is removed outright.
## Note
I wasn't able to get a clean visual render of the updated Distributed
RTE Write diagram in local preview -- Mermaid isn't rendering in
headless Chrome in this environment for *any* page right now, confirmed
against an unmodified reference page showing the identical symptom. The
diagram source follows the exact pattern already shipped and working in
the 21 diagrams from the earlier brand-theming pass, and `mint
validate`/`mint broken-links` both pass clean.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
`take_row_ids` used an `IN` filter, so the backend could reorder rows
and collapse duplicate IDs. That can silently misalign training rows
with labels or sample weights.
Restore the requested order and duplicate occurrences in the shared Rust
take path for native and remote tables, and retain that information
through Python query serialization. Missing or deleted IDs are omitted.
Explicit Rust `order_by` overrides request order, and pagination applies
after restoring occurrences. Empty projections and `_rowid` projection
aliases retain their output schema.
RTE tables reject unsupported takes with the existing base-table
guidance before checking the internal row-ID projection used for
ordering. An explicit `use_rte(False)` reads the base table with the
same requested order and occurrences.
A pre-existing fixture defect also blocked CI:
`test_table_add_in_threadpool` responded without reading uploads,
allowing a connection reset during response decoding. This PR includes a
fix confined to that test's handler: consume each request body before
responding and verify one create row plus ten inserted rows. This was
not a regression in the take implementation. Production write retries,
concurrency and HTTP client code are unchanged.
The selective sampling guide scopes the row-ID guarantees to SDK
releases containing this fix, preserves the distinct offset-take
semantics, and explains that ordered takes collect and combine matching
rows before output batching. It recommends smaller ID requests and
narrow projections because `max_batch_length` does not cap working
memory. This uses the existing REST query protocol and requires a new
SDK release; no deployed service was changed.
Validation:
- Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX
compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed
lines. Full-site Mint validation reports four existing snippet-import
warnings on the reranking/vector-search pages, identical with the
original page restored in a baseline comparison.
- The applied threadpool test passed locally after the fixture change.
The identical candidate previously passed 10 trials with the ordinary
reader and 10 with an unbuffered reader; each trial verifies all
uploaded rows. Those checks used macOS arm64/Python 3.14/default
features, not the full failing CI environment. The normal new-head macOS
Python 3.13/experimental-functions job remains the required platform
check.
- Python package Ruff format/check passed. Repository-root Ruff still
reports 59 existing errors and six formatting differences in unchanged
`docs/web-tests` files; none are in changed files.
- At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed;
5 targeted cases and strict workspace Clippy passed. The fixture-only
follow-up changes no production Rust/Python code, so that evidence is
reused.
- Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests
and 24 Python take/getitems tests passed; the focused 49-test suite also
covered offset dataloaders. A local 9,216-row, four-fragment
reproduction returned 200 seeded random batches of 64 rows with zero
ordering or label-alignment mismatches. The ordering and projection
implementation is unchanged by the follow-ups.
- No live Enterprise run was performed.
Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469).
The client registers Functions, but the decorator that packages one was
called `udf` and produced a `UdfDefinition`. Rename them to `function`
and `FunctionDefinition`, and say "function" instead of "UDF" in the
docstrings, error messages and docs around them.
`udf` and `UdfDefinition` stay as deprecated aliases that emit a
`DeprecationWarning`. The source packager strips either spelling of the
decorator, so `@udf` and `@function` produce the same artifact and
digest, and the wire contract is unchanged. DataFusion and Lance UDF
types, and Geneva's `@udf` in the integration docs, are unrelated and
untouched.
Labels early access features and hides features we aren't documenting
yet.
**Early access label:** Functions (4 pages, plus the GPU refresh,
checkpointing and adaptive execution pages), Secrets (3 pages), Tiered
Index Cache, Blob V2, and merge insert, which is labelled for Enterprise
only. Two sidebar titles are shorter so the tag fits.
**Hidden:** branching, JSON columns and geospatial.
- The JSON functions page and the agent branch-experiments tutorial go
in `.mintignore`.
- `tables/branching` stays reachable but out of the nav, as #4859 left
it, because SDK error messages link to it.
- Other pages' mentions of these features become draft comments, so
they're easy to restore.
**Secrets page:** drops references to manual KMS setup, which the
control plane handles. It keeps the encryption-at-rest facts, states the
Azure value-size limit inline, and points a 501 error to support.
## Summary
- Hides `tools/cli.mdx` from the sidebar and from sitemap/search via
`.mintignore`, unlinking its one reference in `basics/sql.mdx`. No SDK
doc-link depends on this page.
- Removes `tables/branching.mdx` from the Management nav group, but
leaves it reachable (no `.mintignore` entry): the SDK's
`diff_branch()`/`cherry_pick()` error messages link to
`/tables/branching#diff-a-branch` and `#merge-a-branch`, and
`rust/lancedb/src/docs.rs`'s link-resolution test asserts those paths
aren't `.mintignore`'d.
- Adds a Console row to the comparison table, linking to the companion
change's new `enterprise/console` page.
## Companion change
lancedb/sophon#9783 adds `enterprise/console.mdx` and inserts it into
the "Getting Started" nav group this PR touches.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The Search group covers full-text and hybrid search as well as vector
search, and full-text search is lexical, not semantic. This renames the
group to "Search" in the sidebar and in the Embedded vs Enterprise
comparison table.
`mint validate` and `mint broken-links --check-anchors` pass on the
assembled site.
## Summary
- Adds a new top-level "Introduction" nav section before "Getting
Started": LanceDB, LanceDB Embedded, LanceDB Enterprise, Embedded vs
Enterprise.
- Trims the landing page's combined
Architecture/Embedded/Enterprise/Compare block down to a short pointer
section (keeping its `#architecture` anchor), moving that content into
two new pages: `basics/lancedb-embedded.mdx` and
`basics/embedded-vs-enterprise.mdx`.
- `basics/embedded-vs-enterprise.mdx` consolidates the
feature-comparison table and the Enterprise overview's own comparison
content (previously duplicated across multiple pages and, within this
page, repeated across a dimension table, several narrative subsections,
and a usage-differences list) into one comparison page with each section
covering a distinct angle.
- Repoints the `/enterprise` and `/enterprise/deployment` redirects at
`/basics/embedded-vs-enterprise`, since the previous destination only
resolves once the Enterprise overlay is applied and fails
`rust/lancedb/src/docs.rs`'s link-resolution test against this repo's
own `docs/web` tree.
## Companion change
lancedb/sophon#9718 trims the Enterprise overview page to drop the
content that moved here, and relocates its nav insertion into the new
Introduction group.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
Follow-up to the Mermaid brand-theming PR, addressing review feedback:
- Diagram text was hard to read at the previous 15px font size; bumped
to 18px, light and dark alike.
- The dark-mode orange accent border was brighter and more saturated
than its fill or the default border, reading as too loud against the
dark background. It now reuses the same muted orange already used for
the light-mode accent border, which was fine as is.
## Companion change
lancedb/sophon#9719 makes the same two changes across its own diagrams.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
Every Mermaid diagram now renders in the brand palette, with separate
light and dark variants that switch with the site's theme toggle.
Default node styling stays neutral; one accent color marks each
diagram's actual subject, chosen per diagram. A couple of diagrams
changed orientation where their old layout was too wide to read
comfortably.
Companion change in lancedb/sophon uses the same colors and pattern.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The removed Enterprise landing page left the `/enterprise` and
`/enterprise/deployment` redirects pointing at a missing page, breaking
the SDK documentation-link test. Point both redirects at the existing
architecture overview, which describes Embedded and Enterprise.
The exact failing Rust test fails on current main and passes with these
redirect changes. This repairs the inherited failure blocking #4821.
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:
**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.
**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.
**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.
**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.
**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.
Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot
integration guide with `Table.fetch_blob_files`, document how to open
`.lance` datasets directly with `lancedb` alongside a
`pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in
installation docs that the optional `pylance` extra is only needed for
engine/connector authors or format-level fragment APIs.
TypeScript job handles now expose structured progress through
`progress()` and an optional synchronous callback to `wait()`. Snapshots
preserve unknown fields and missing counters, and job listings include
progress and pause state. Existing `status()` strings and no-argument
`wait()` behavior remain compatible.
Wait callbacks observe the existing backend poller. Observer exceptions
are logged without changing the job outcome. A best-effort ETA is
available only from compatible samples with positive progress, and is
omitted during initialization, queueing, or pause.
Part of ENT-3500.
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-12-104.us-east-2.compute.internal>
Job callers currently lose the server's structured progress when
describing, listing, or waiting for a job. Expose Rust and Python
progress snapshots and optional wait observers using the existing
backend poller. Python supports callbacks, a caller-owned tqdm-like bar,
or `progress=True`; `status()` still returns a string and the default
wait behavior is unchanged.
Preserve unknown progress fields and phases from newer servers, and
report omitted counters as unknown. Each wait calibrates a caller-owned
bar from its first complete snapshot, then uses a high-water mark within
the same execution epoch; a new epoch or smaller plan resets the bar.
ETA is best-effort and omitted for missing, stale, paused, or
incompatible samples.
Newly submitted index and refresh handles forward observers through
their freshness wrapper while retaining successful-wait read
consistency. The Jobs guide includes draft Python/Rust snapshot and
observer examples for release; TypeScript progress support remains
tracked under ENT-3500.
Validation on the development EC2 host: the initial 11 Python progress
tests, REST fixture caller, and two Rust progress regressions passed.
After fixing freshness-wrapper forwarding, rebuilt the Python extension
and ran two submitted-handle Python tests plus the new Rust refresh and
index regressions (one test each); all passed. Strict Clippy, rustfmt,
and targeted Ruff passed. The final commit only wraps the new guide
section in an MDX draft and clarifies language support. Repository-wide
Ruff reports 48 existing findings in six unchanged docs/web-tests files.
This publishes the SDK portion of ENT-3500. Server-side live-split and
terminal-progress changes and the combined server/SDK end-to-end
scenario remain in progress. Refs ENT-3500.
BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress:
None`, `phase: None`, and `paused: false`. Literals for `JobDescription`
must add `progress: None`, `phase: None`, `queue_reason: None`,
`waiting_for_job_id: None`, and `paused_at_millis: None` when that
metadata is unavailable. Existing Python calls remain compatible.
Python UDF registration could not set `batch_size` or
`row_timeout_seconds`, and the Rust registration model discarded these
fields before submitting SQL. Expose both optional controls and preserve
them through registration, with invalid values rejected and unchanged
canonical bytes when the controls are omitted.
The timeout applies to each Python call: one row for scalar Functions
and one batch for vectorized Functions. Initialization keeps its
existing service deadline.
Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust
registration-contract tests, 2 outbound-SQL tests, and all 103 Python
slice2 tests passed after rebuilding the native extension. Rust
formatting and the repository Python CI checks (Ruff 0.9.9, python/
scope) passed. A real REST + Flight SQL server integration also passed
using the rebuilt SDK: `batch_size=2` limits observed calls in a direct
SELECT over 8 rows, initial materialized-view creation, and
append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime
passthrough; it does not exercise GPU initialization.
Refs ENT-3246. The original GPU initialization and video-decoding
scenario still needs verification on a current deployment; this SDK
change does not close that investigation.
The additional repository-root style checks report existing
docs/web-tests findings (nine format files and 48 lint diagnostics); all
affected files are unchanged from main. These are outside the Python CI
job scope.
Rust compatibility: callers constructing `FunctionRegistrationRequest`
must initialize the two new optional fields (`None` preserves defaults);
the request no longer implements `Eq`.
The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds
creation-guide documentation and integrates main’s namespace
repr/optional OCI embedding changes. The Function registration
implementation is unchanged. All 103 Python slice2 tests passed again on
EC2 using the unchanged native extension; Rust and server-bridge
evidence remains applicable.
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.
- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.
Linear: OSS-2772
Packages passed through `@udf(code=[...])` silently omit non-Python
resources, so registration can appear successful before a worker fails
to open a data file. Warn during function preparation with the omitted
paths and document the source-only boundary next to `code=`. The
diagnostic explains how to install a distribution that includes the data
and remove the package from `code=` to avoid shadowing that
installation.
Python cache files do not produce warnings. Excluded directories are
reported at their boundary, preserving the existing traversal and
Python-source artifact format. This implements the
warning-and-documentation alternative in ENT-3492; it does not add
resource uploads.
On EC2, the regression fails on baseline `f9389f113` because no warning
is emitted; all 60 packaging tests pass at `e23445b96`, using the
matching native extension built from unchanged Rust sources.
CI-scoped Python lint with Ruff 0.9.9 passes. Whole-site Mintlify
validation reports the same four snippet-import warnings on the baseline
and this head; repository-root Ruff likewise reports the same 48
findings under `docs/web-tests`. These checks are not clean passes, but
their failures do not involve this change.
Fixes ENT-3492
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.
Co-authored-by: Leo <leo@insight.co>
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.
Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.
SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.
Fixes#1959.
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
Fixes#4761
`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.
The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.
This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.
Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
Renaming or dropping an embedding source or vector column leaves schema metadata pointing at missing fields, causing later `add()` calls to fail.
Reject these operations in local schema evolution before committing any changes. The guard reads Python/Node embedding column metadata and native Rust `lancedb::column_definitions` without loading embedding models. It supports explicit and default Rust vector-column names and respects quoted column paths. Unrelated column changes and nullability updates remain supported.
The restriction is documented in the Python, Rust, and TypeScript APIs. Regression coverage includes both SDK metadata formats, native Rust definitions with explicit and default destinations, atomic rejection of batched changes, and continued embedding generation after reopening.
Fixes#4763
Validation at the exact PR head and GitHub test-merge:
- Hosted macOS Python wheel suite: 3061 passed, 162 skipped, 70 deselected, including all 8 sync and 4 async embedding schema-evolution cases.
- Hosted Windows Rust suite: 1900 library tests passed, 8 embedding integration tests passed, and 107 doctests passed, including the native explicit/default destination and binding guard cases.
- Hosted Linux Python 3.10 rerun: 3084 passed, 159 skipped, 50 deselected; the previously observed remote-fork timeout passed on the replacement job.
- All 10 applicable exact-head workflow runs completed successfully. The existing main-only Windows ARM job was skipped by its workflow condition.
Fixes#4773.
Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.
Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
The vector-search guides do not clearly explain when to use Auto, search
effort or a fixed probe count. They also describe `minimum_nprobes` as
the exact starting count, although Auto can search more partitions from
the start.
Explain how to choose public query controls for recall and latency, with
short executable examples. Clarify that probe limits apply per segment,
calibration targets are not recall guarantees, and more probes cannot
remove quantization error. Keep necessary client/server version guidance
in a short note.
Validation:
- The existing example test passes against published `lancedb==0.41.0`,
covering all four snippets, result counts, filtering, finite distances
and recall against an exact cosine query. The test body takes 0.20 s.
Generated snippets match their source.
- Vale reports no findings on added prose. Changed-function Python
formatting and `git diff --check` pass. Repository-wide Ruff has 137
findings identical to the previous scan, including seven pre-existing
findings in the example file.
- Mint 4.2.888 strict validation reports the same four
`RsExampleBody`/`RsExampleTail` import warnings as pristine base
`0b20965d`, with no additional warnings.
## Summary
- Add sync/async `DBConnection.list_gpu_classes()` returning typed
`GpuClass` records for `GET /v1/gpu/classes`. Records include the class
name, Kubernetes resource, GPU model and memory, per-worker GPU maximum,
and worker CPU/memory/ephemeral-storage limits.
- Add `udf(gpu_class=...)` for immutable Function-level GPU placement.
It requires `gpu=True`; Function versions expose their registered
`gpu_class`.
- Add `gpu_class` and `gpu_count` to `refresh_column` /
`refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as
top-level `backfill_column` fields and validated before any request.
`gpu_count` is the aggregate GPU budget across Function workers.
- Validate GPU class names with the server lowercase 63-character
DNS-label rule.
- Document the discovery-to-registration flow, configured hardware
versus free capacity, System Usage permission, and per-refresh GPU
placement in the API and web guides.
Older servers without `/v1/gpu/classes` produce a clear
`NotImplementedError` instead of a bare 404.
Part of ENT-3470. The client follows the in-flight Sophon contract on
`xuanwo/ent-3470-fix` for GPU class discovery and backfill placement
options.
## Breaking change
`PythonRuntimeSpec::PythonV2` adds `gpu_class`, and
`RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers
constructing either value with all fields explicitly must supply the new
fields.
## Validation
At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python
extension build, pyright, ruff, and Python Function/runtime, discovery,
refresh placement, and refresh regressions (147 passed, 6 skipped).
At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright,
Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed).
The final commit only updates documentation and Rust doc comments.
`list_functions()` promises every published immutable revision. Request
`SHOW FUNCTIONS HISTORY` so the SQL server's latest-only default does
not hide older revisions, and preserve the server's numeric revision
ordering when constructing page cursors (`1`, `2`, `10`).
```python
[(fn.name, fn.version) for fn in db.list_functions(page_limit=2)]
# [("embed", "1"), ("embed", "2"), ("embed", "10")]
```
```sql
SHOW FUNCTIONS; -- latest visible revision per name
SHOW FUNCTIONS HISTORY WITH (limit = 3, include_definition = true);
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '2', limit = 3);
```
Each listing page uses the existing SQL query and polling transport; the
continuation token stores the last returned name/revision and remains
bound to its database and namespace. Function mutations keep their
query-backed job handles: `Job.id` is the query's ID and `job_type` is
`query`, so waiting observes that same SQL execution. No registry job or
second query ID is added.
Update the SQL and Python documentation for exact-revision drops:
sibling revisions and grants survive, and only the last visible
revision's removal releases the name and schedules object cleanup.
Extend the existing Flight SQL regression to paginate revisions `1`,
`2`, and `10` and verify the explicit history statement. Formatting and
static Python checks pass; local builds and tests were intentionally not
run.
View APIs now read the native `name`, `query`, and `schema` columns
returned by `CREATE VIEW` and `DESCRIBE VIEW`. The schema JSON carries
the defining database and namespace; the requested namespace identifies
where the view lives. The unreleased JSON-envelope response is removed
without a compatibility fallback.
```python
view = db.create_view("adults", "SELECT name FROM people WHERE age >= 18")
view = db.describe_view("adults")
```
```sql
CREATE VIEW "adults" AS SELECT name FROM people WHERE age >= 18;
DESCRIBE VIEW "adults";
```
Both Python calls execute SQL through the query-backed job adapter and
wait on the resulting query. A job handle uses that query's `id` and
type `query`; reading metadata consumes its completed result rather than
resubmitting the DDL. Creation returns the definition from the creating
execution, with no extra describe request.
Secret APIs now submit the native SQL statements with the credential
bound as an Arrow string parameter:
```sql
CREATE SECRET "api-key" TYPE = GENERIC_STRING SECRET_STRING = $1;
ALTER SECRET "api-key" SET SECRET_STRING = $1;
DESCRIBE SECRET "api-key";
```
`create_secret` and `alter_secret` execute SQL through the existing
query-backed job adapter and wait for that query to complete. The job's
`id` is the query ID and its type is `query`; credentials stay out of
SQL text. `describe_secret` reads the native `name`,
`created_at_millis`, and `updated_at_millis` columns. This replaces the
unreleased `AS $1` syntax and JSON-envelope response without a
compatibility fallback.
Python methods and namespace handling remain the same:
```python
db.create_secret("api-key", "credential", namespace_path=["team"])
db.alter_secret("api-key", "rotated", namespace_path=["team"])
info = db.describe_secret("api-key", namespace_path=["team"])
```
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model
names outside the three built-in OpenAI names, so schemas can be built for
compatible models.
Preserve existing model defaults and request behavior. Add regression coverage
for the resulting Arrow schema dimensions and for all known OpenAI model
dimensions.
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
Package the standalone lancedb CLI with every stable and beta GitHub
release. Publish archives for Linux x86_64, Linux ARM64, and Apple
Silicon macOS, along with SHA-256 checksums, licenses, and third-party
notices.
Synchronize the CLI version with the SDKs and validate that release tags
match both manifests. Add PR and manual packaging dry runs, runtime
compatibility checks, and smoke tests of extracted executables. GitHub
release creation now waits for all three archives to pass validation;
SDK registry publishing remains independent.
Update installation instructions and the release checklist. Linux
binaries target glibc 2.28, retaining existing x86_64 CPU requirements;
macOS binaries target macOS 14 or newer.
Stacked on #4675.
Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.
`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
Stacked on #4674.
Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens.
The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates.
Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`.
## Summary
Fixes the problems a page-by-page review of the docs found before
launch, after checking each one against the current source and, where
noted, the code. The matching fixes for the dataset and Geneva pages,
which live in `lancedb/docs`, are in lancedb/docs#402.
**Tables**
- `integrations/embedding/cohere`: each input-type description was split
across two table rows and wrapped in literal quotes
(`"`search_document`"`). Now one row per type.
- `integrations/embedding/gemini`: removes the same stray quotes from
the task-type table.
- `integrations/ai/langchain`: the MMR search table's `kwargs` row was
missing a cell, so its description sat under Type.
- Six reranker pages (answerdotai, cohere, colbert, jina, openai,
voyageai): closes the backtick before `_relevance_score`.
- `indexing/vector-index`: finishes a cell cut off at "with scalar".
- `sql/dql/dedupe`: labels the empty header cell in the resolution
table.
**Corrections**
- `curation/sampling`: sampling works in LanceDB Embedded through the
Python SDK, so the page is available in both, with the SQL sections
marked Enterprise.
- `feature-engineering/index`, `add-computed-column`: automatic refresh
is on by default for each computed column and takes effect once a
deployment enables its scheduler. `refresh-column` already said so;
these two implied it was off.
- `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates
`array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also
merges scalar-index's two "nested fields" sections into one.
- `integrations/embedding/ibm`: lists the four current models from
`CURRENT_MODELS` (it listed a deprecated one) and explains that the
default stays a legacy ID so existing tables reload with their model.
- `integrations/reranking/openai`: the class is `OpenaiReranker`, not
`OpenAI()`.
- `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit`
throughout, each step runs one command, each step passes the file it
writes to the next, and `save-as` uses `-f`/`-o`.
- `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the
pipeline reads, and the run command runs the file the page creates.
- `search/vector-search`: the multivector walkthrough connected to
Enterprise in step 1 and a local path in step 2. It now runs locally
throughout, with a note on connecting to Enterprise, and is renamed so
it doesn't read as a duplicate of the short section above.
**Structure and wording**
- `integrations/embedding/nvidia`: adds headings, and drops test-suite
details aimed at contributors.
- `integrations/ai/genkit`: makes the retrieval test a real heading and
describes the query step.
- `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a
`<Note>`.
- `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading,
not bold text.
- `curation/windowing`: sidebar title matches the curation overview
("Rank and Deduplicate").
- `deployment/storage`, `basics/agentic-skills`: fixes a garbled
sentence and a stale reference to the quickstart.
**Checked:** no new Vale alerts on the changed pages. Assembled together
with lancedb/docs#402, the site passes `mint validate` and `mint
broken-links --check-anchors --check-redirects`. Every changed page was
reviewed in a local preview.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add a LanceDB command line with the following functionality:
- `lancedb auth`: log in with OAuth, check cached OAuth session status,
or log out.
- `lancedb profile`: Create, edit, list, show, select, and delete
connection profiles.
- When no subcommand is given, we will attempt to create a SQL shell.
Update ci/check_cargo_lock.py and its regression tests to allow
pruning dependency edges while their target packages remain in
Cargo.lock. Removing the CLI from the Python source distribution
can disable optional features such as Axum tracing, even though
other SDK dependencies still require the tracing package.
## Summary
Replaces the generated availability table under **Compare features** on
the landing page with a hand-written list of features, grouped by area,
with separate Embedded and Enterprise columns.
The generated table listed every page that declares `availability`,
including overview and basic CRUD pages. Rows such as "Overview"
appeared three times without context, and pages like Insert, Update,
Database and the integrations showed up as if they were features.
| Area | Features |
| --- | --- |
| Search | Vector, full-text and hybrid search, filtering, reranking,
query plans, distributed search |
| Indexing | Vector, full-text and scalar indexes |
| SQL and curation | SQL, aggregation, sampling, joins, ranking and
deduplication, near-duplicate detection, chunking, changes since a
version, views |
| Feature engineering | Computed columns, materialized views, embedding
registry |
| Training | PyTorch data loader, permutations, distributed training |
| Data management | Branches, compaction and cleanup, background jobs,
Real Time engine |
| Security | Authentication, authorization |
Each ✅/❌ matches the linked page's `availability` on `main`; SQL, which
declares none, is Enterprise-only. Functions are left out until it's
settled whether they are published.
The same list, limited to 1.10 features, is in the Enterprise 1.10
editions in lancedb/docs#398.
Checked by assembling the docs site with this change: `mint validate`
passes and `mint broken-links --check-anchors --check-redirects` finds
no broken links. Vale reports no new warnings.
After this merges, the committed `lancedb` skill in lancedb/docs needs a
rebuild (`make assemble && make skill`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Changes links to home page sections from `/#<anchor>` to
`/index#<anchor>`.
The docs site's Assemble CI fails `mint broken-links --check-anchors` on
every `/#architecture` link since #4732 moved the Architecture page onto
the home page. The anchor exists; Mint 4.2.888's checker can't resolve
any anchor on the root page `/`, but resolves `/index#…`.
`docs.lancedb.com/index` serves the home page, so readers land in the
same place.
Verified by assembling the site with this change and the matching
Enterprise docs change: `mint validate` passes and `mint broken-links
--check-anchors --check-redirects` finds no broken links. The
`docs.json` redirects keep `/#architecture`, which the checker accepts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Addresses some comments on the staging branch. Rebased onto #4732, which
rebuilt `index.mdx` and the quickstart and reverted the landing copy to
the pre-v0.4 headings; the final copy is reapplied here.
| Comment | Page | Change |
| --- | --- | --- |
| Taylor: drop the comma | `index.mdx` | "one, unified platform" → "one
unified platform" |
| Taylor: "Preparing t…" + "Data doesn't slow down" | `index.mdx` |
"Training data and experimentation slow down…" → "Preparing training
data and running experiments slow down…" |
| Fei: subtitles should align with the new home page | `index.mdx` |
Follows the final homepage copy (v0.4): "Train on the right data faster"
with the hero subtitle, "Accelerate the data flywheel for model
research" with its intro, and the five v0.4 sections as steps, each
linking to the page that backs it (search, curation, branches, versions,
computed columns and materialized views, `StreamingDataset`). Wording
the docs can't back yet is left out: "zero-copy", "lineage", "cached
across clouds". Anchors renamed to match; nothing links to the old ones
|
| Fei: quickstart seems OSS-only | `quickstart.mdx` | Tip at the top:
for Enterprise only the connection changes, linking the Enterprise
connection in step 2 |
Fei's "list all Enterprise features as a catalog" comment is now covered
by #4732's feature comparison table (`/#compare`), so the separate table
this PR added to `basics/offerings.mdx` was dropped along with that
page.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Updates the overview diagrams on the landing page. Rebased onto #4732,
which moved the architecture and offerings content into `index.mdx` and
drew it as Mermaid diagrams; this PR replaces those with the guide's
diagrams.
- **Architecture** (`#architecture`): the suite diagram
(`lancedb-suite.svg`) replaces the first Mermaid diagram. Its Enterprise
boxes read Search, Data exploration & curation, Feature engineering,
Training, matching the guide's product hierarchy.
- **LanceDB Embedded** (`#1-lancedb-oss`): adds the guide's Embedded
diagram. Its storage box reads "Local Disk or Object Storage", since
Embedded supports both.
- **LanceDB Enterprise** (`#2-lancedb-enterprise`): the guide's
Distributed diagram replaces the second Mermaid diagram. This also drops
that diagram's "column + MV refresh" label; automatic refresh is still
gated.
- **Functions** are still unpublished, so the "Functions ·" label is
removed from the Distributed diagram's jobs fleet. At release, restore
the designer's original image from the Notion guide.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause,
and
`dedupe()`/`duplicate_pairs()` (verified against current sophon source;
these were recently renamed from
`vector_dedup`/`vector_duplicate_pairs`).
Fix the blob-column default-projection claim and add the previously
undocumented `blob_descriptor()`.
- Enrich the gated Functions draft (still `.mintignore`'d, pre-release):
chained Functions over blob outputs, GPU-produced blobs, Secrets,
multi-input/output binding, `code=[module]` packaging.
- Upgrade the materialized-view and computed-column auto-refresh drafts
with
precise, verified catch-up semantics, and add the missing `ALTER
MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the
automatic-refresh scheduler is still dev/staging-only).
- Thread one running example (video frames → GPU extract/caption → dedup
→
auto-refreshing materialized view) through Functions, materialized
views,
and the LeRobotDataset integration guide instead of a separate tutorial.
- Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout
(52 files); activate the existing availability-comparison mechanism on
`basics/offerings.mdx` for a full feature-by-feature comparison table.
Adds a Python `nvidia` embedding provider for Nemotron 3.5 Embed 1B and 8B. It works with NVIDIA's hosted API and self-hosted NIM `/v1` endpoints, and uses the existing `openai` client. The OpenAI provider can't send NVIDIA's passage/query input types or pick Nemotron's dimensions.
- Documents are embedded as `passage`, searches as `query`.
- Default dimensions are 2048 (1B) and 4096 (8B), and `dim` is checked against each model's supported range. Self-hosted model IDs need an explicit `dim`.
- Response count, order, dimensions and values are checked. Null and empty text get a null vector without a request.
- Each request is retried on its own with LanceDB's retry policy. Invalid input or responses fail immediately.
- `NVIDIA_API_KEY` is only sent to the hosted API. Custom endpoints need an explicit `api_key` (a `$var:` reference), and endpoints restored from table metadata must be allowed with `allow_endpoint`.
- Request dimensions are sent through `extra_body`, preserving the NVIDIA wire contract while supporting the declared minimum `openai==1.6.1`.
Python and text only. Docs are in `integrations/embedding/nvidia`.
Originally authored by Taylor (@tsmithv11).
### Testing
- `test_nvidia_embeddings.py`: 45 passed, 2 deselected with both the locked `openai==2.44.0` and minimum `openai==1.6.1`. These use the real OpenAI client with mock HTTP transport and real local tables.
- Embedding, registry, field-metadata and trust suites: 163 passed, 4 skipped, 10 deselected.
- Historical live experiments: hosted 8B passed a five-request smoke test; hosted 1B returned HTTP 500. No live request was made in this stage, so current hosted 1B availability was not verified. Enterprise coverage uses the repository mock HTTP server, not a live deployment.
Add a typed Python `CombinedFieldsQuery` for BM25F search across multiple columns.
The query mirrors Lance's combined-fields semantics with ordered per-column boosts and AND/OR term matching. Convert it in both directions at the PyO3 boundary, including nested Boolean queries, and document the Python typed convenience entry.
Closes#4652.
Remote Python connections inherited the unsupported-operation stubs for
`namespace_exists` and `table_exists`, so both raised
`NotImplementedError` even when namespace and table operations worked.
Add the missing remote overrides, async methods, and PyO3 bindings using
the existing namespace describe and table open paths. Only an explicit
not-found result returns `False`; permission, transport, and server
errors propagate. This preserves authentication, dynamic headers,
retries, and nested namespace handling, and rechecks cached tables
before reporting that they exist.
The async namespace helper also recognizes native `NamespaceNotFound`
errors so missing local namespaces return `False`. The table and
namespace guides document Python remote support and distinguish it from
the unsupported `exists` routes used by direct REST namespace clients.
Validation:
- All 40 focused regression cases pass, covering successful and missing
lookups, local and remote namespaces, dynamic headers, permission/server
errors, invalid table identifiers, and cached tables that disappear.
- The original inherited stubs reproduce the reported
`NotImplementedError`; the local namespace regression reproduced the
native missing-namespace `RuntimeError` before its fix.
- Full remote connection suite: 255 passed. Local namespace tests: 14
passed.
- `cargo clippy --quiet -p lancedb-python --features remote --lib`,
`cargo fmt --all`, Python Ruff lint, and Python formatting checks pass.
- Root `ruff check .` reports 48 existing errors in documentation
examples under `docs/web-tests`; the Python subtree passes and those
unrelated files are unchanged.
Fixes#4722
<!-- lance-gatekeeper-fix:v1 agent=4d369a0681756fdb5a8b35853d7f5607
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Closes#1849
This was lancedb/docs#371. After lancedb/docs#355 moved the open-source pages, their navigation and the snippet tests into this repository's `docs/web` and `docs/web-tests`, that PR could no longer merge there, so I've moved the same change here.
## Why
#1849 asks for an example of LanceDB in a VS Code extension, because people kept running into problems there (#927, #1713). An earlier attempt, #3682, added a README link. @prrao87 asked for something else instead: a quick guide on the docs site, under **Integrations › AI Platforms & Frameworks**, showing how to use LanceDB in an extension and what results look like. The author of #3682 agreed and closed it. This PR is that guide.
## What it adds
- **`docs/web/integrations/ai/vscode.mdx`**: a new page, also added to `docs/web/docs.json`. It builds a small extension with two commands. One indexes the workspace's Markdown into a LanceDB table using the built-in `huggingface` embedding function, so no API key is needed. The other searches it by meaning. The page covers the parts that are specific to VS Code:
- storing the table under `context.storageUri` instead of a relative path, which is the cause of #927;
- `@types/node` ≥ 22 and `skipLibCheck`, which covers #1713 and the DOM-type errors from `apache-arrow`/onnxruntime typings;
- pinning `apache-arrow` inside LanceDB's peer range;
- marking `@lancedb/lancedb` and `@huggingface/transformers` as esbuild externals, and removing `node_modules/**` from the template's `.vscodeignore`;
- storing each file's URI so a hit opens in the right folder of a multi-root workspace;
- building one VSIX per platform with `vsce package --target`, with a table mapping each target to LanceDB's native package, and a note that the sample's local model (via `onnxruntime-node`) doesn't run on Alpine;
- keeping the Transformers.js model cache in `globalStorageUri`, which only works if it's loaded with `import()`;
- a troubleshooting table listing each error message with its cause and fix.
- **`docs/web-tests/ts/vscode_search.ts`** and **`vscode_search.test.ts`**: the LanceDB code the page shows (connect, chunk, index, search), plus jest tests for it.
- **`docs/web-tests/ts/vscode_extension.ts`**: the extension's `src/extension.ts`. Jest can't run it because it needs the VS Code extension host, so it was tested in a real extension instead (see below). This follows the precedent of `integrations.ts`.
- **`docs/web/snippets/vscode_search.mdx`** and **`vscode_extension.mdx`**: generated with the command in CONTRIBUTING.md.
## Changes from lancedb/docs#371
- **New paths only.** The page, tests and snippets are the same content, at their paths in this repository.
- **One link.** `/tables/update` doesn't exist under `docs/web`, so the `mergeInsert` link now points to `/ingestion/merge-insert`.
- **Heading anchors.** Every heading now has an explicit `{#anchor}`, as CONTRIBUTING.md asks.
## How I tested it
- **Snippet tests.** In this repository at `c2c8269f`, `vscode_search.test.ts` passes (2 tests) with the pinned `@lancedb/lancedb` 0.31.0. On Windows the `npm test` script fails before it starts, because it runs `node` on the `.bin/jest` shell shim. I ran `node --experimental-vm-modules node_modules/jest/bin/jest.js --testEnvironment jest-environment-node-single-context vscode_search` instead.
- **Lint.** `biome ci` is clean for the three new files, and they carry the SPDX header that `license-header-check.yml` looks for.
- **Snippets.** I regenerated them with CONTRIBUTING.md's command, `mdx_snippets_gen.py -s docs/web-tests/py -s docs/web-tests/ts -s docs/web-tests/rs -o docs/web/snippets`, run with `python` since I don't have `uv`. It reproduces the two new snippet files exactly, once the Windows CRLF endings are normalised. As CONTRIBUTING.md warns, it also rewrites `build_with_ai_agents.mdx`, `integrations.mdx`, `search.mdx` and `storage.mdx`. I left those out.
- **Links.** `mint broken-links` from `docs/web` reports no entries on the new page, and its three internal links (`/embedding/quickstart`, `/ingestion/merge-insert`, `/search/full-text-search`) resolve to pages here.
- **Real extension.** I built a sample extension from exactly the code on the page, scaffolded with `yo code` (TypeScript + esbuild) and using `@lancedb/lancedb` 0.39.0. I ran it with `@vscode/test-cli` in VS Code 1.140 on Windows x64. The integration test runs both commands against a six-file workspace and checks the top hits. The "Sample results" block on the page is copied from that run, and after the review fixes below it reproduces line for line in VS Code 1.141.
- **Multi-root workspace.** A second test configuration opens two folders that both contain `docs/auth.md`. It types a query and picks the top hit through the real picker, then checks that the right folder's file opens at the right line. It fails on the code before the review fix and passes after it.
- **Packaged VSIX.** I packaged `vsce package --target win32-x64` (200 MB, or 148 MB with the onnxruntime trimming in the Tip). Then I unpacked it outside the project and ran the same tests against the unpacked copy, so they couldn't pick up the project's `node_modules`. They pass.
- **Negative checks behind the troubleshooting rows:**
- With `node_modules/**` left in `.vscodeignore`, activation fails with `Cannot find module '@lancedb/lancedb'` and the commands report `command '…' not found`.
- Without the externals, esbuild fails with `No loader is configured for ".node" files`.
- Installing LanceDB into a project that already has `apache-arrow` 21 fails with `ERESOLVE`.
- `require("@huggingface/transformers")` returns a different `env` object from `import()`.
- **Rendering.** For lancedb/docs#371, `mint dev` rendered the page, its 14 code blocks and the sidebar entry. I haven't re-rendered it from `docs/web`; only the paths, one link and the heading anchors changed.
**Not tested:**
- macOS. I couldn't reproduce the read-only working directory from #927; the page's fix (an absolute storage path) doesn't depend on the OS.
- Linux and macOS VSIX builds.
## Notes for reviewers
- I added `_distance` to `.select(...)`, because LanceDB warns that it will stop adding the column automatically.
## AI disclosure
I used Claude Code to research the issue and its history, write the sample extension, the tests and this page, move the change here, and draft this description. I ran every check listed above myself, and the outputs quoted on the page come from those runs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Yang Cen
Jack YeandClaude Sonnet 5
Wyatt Alt
Brendan Clement
Xuanwo
LeoandLeo
Srijan Keshri
lancedb-gatefixer[bot]
TaylorandBrendan Clement
BHAWESHBHASKAR
alexcastgonz
ryan-haynesandRyan Haynes
Colin Patrick McCabe
Nishil Rathod