The removed Enterprise landing page left the `/enterprise` and
`/enterprise/deployment` redirects pointing at a missing page, breaking
the SDK documentation-link test. Point both redirects at the existing
architecture overview, which describes Embedded and Enterprise.
The exact failing Rust test fails on current main and passes with these
redirect changes. This repairs the inherited failure blocking #4821.
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:
**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.
**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.
**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.
**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.
**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.
Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot
integration guide with `Table.fetch_blob_files`, document how to open
`.lance` datasets directly with `lancedb` alongside a
`pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in
installation docs that the optional `pylance` extra is only needed for
engine/connector authors or format-level fragment APIs.
TypeScript job handles now expose structured progress through
`progress()` and an optional synchronous callback to `wait()`. Snapshots
preserve unknown fields and missing counters, and job listings include
progress and pause state. Existing `status()` strings and no-argument
`wait()` behavior remain compatible.
Wait callbacks observe the existing backend poller. Observer exceptions
are logged without changing the job outcome. A best-effort ETA is
available only from compatible samples with positive progress, and is
omitted during initialization, queueing, or pause.
Part of ENT-3500.
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-12-104.us-east-2.compute.internal>
On remote tables, `create_index` returns as soon as the build is scheduled. Until it finishes, a vector index is missing from `list_indices`, a scalar or FTS index shows 0 indexed rows, and queries don't use the index. The docs didn't say any of this, so it was easy to assume the index was ready.
This adds that to the `create_index` and `list_indices` docs in Python (`Table` and `AsyncTable`) and TypeScript, and points to `wait_timeout` / `waitTimeoutSeconds`, `wait_for_index`, and `create_index_async` for blocking until the index is ready. It also explains that completion tracking is tied to the submitted build only when the server returns a job ID, while legacy name-based waits can return early when replacing an existing index.
Closes#4405
Job callers currently lose the server's structured progress when
describing, listing, or waiting for a job. Expose Rust and Python
progress snapshots and optional wait observers using the existing
backend poller. Python supports callbacks, a caller-owned tqdm-like bar,
or `progress=True`; `status()` still returns a string and the default
wait behavior is unchanged.
Preserve unknown progress fields and phases from newer servers, and
report omitted counters as unknown. Each wait calibrates a caller-owned
bar from its first complete snapshot, then uses a high-water mark within
the same execution epoch; a new epoch or smaller plan resets the bar.
ETA is best-effort and omitted for missing, stale, paused, or
incompatible samples.
Newly submitted index and refresh handles forward observers through
their freshness wrapper while retaining successful-wait read
consistency. The Jobs guide includes draft Python/Rust snapshot and
observer examples for release; TypeScript progress support remains
tracked under ENT-3500.
Validation on the development EC2 host: the initial 11 Python progress
tests, REST fixture caller, and two Rust progress regressions passed.
After fixing freshness-wrapper forwarding, rebuilt the Python extension
and ran two submitted-handle Python tests plus the new Rust refresh and
index regressions (one test each); all passed. Strict Clippy, rustfmt,
and targeted Ruff passed. The final commit only wraps the new guide
section in an MDX draft and clarifies language support. Repository-wide
Ruff reports 48 existing findings in six unchanged docs/web-tests files.
This publishes the SDK portion of ENT-3500. Server-side live-split and
terminal-progress changes and the combined server/SDK end-to-end
scenario remain in progress. Refs ENT-3500.
BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress:
None`, `phase: None`, and `paused: false`. Literals for `JobDescription`
must add `progress: None`, `phase: None`, `queue_reason: None`,
`waiting_for_job_id: None`, and `paused_at_millis: None` when that
metadata is unavailable. Existing Python calls remain compatible.
Python UDF registration could not set `batch_size` or
`row_timeout_seconds`, and the Rust registration model discarded these
fields before submitting SQL. Expose both optional controls and preserve
them through registration, with invalid values rejected and unchanged
canonical bytes when the controls are omitted.
The timeout applies to each Python call: one row for scalar Functions
and one batch for vectorized Functions. Initialization keeps its
existing service deadline.
Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust
registration-contract tests, 2 outbound-SQL tests, and all 103 Python
slice2 tests passed after rebuilding the native extension. Rust
formatting and the repository Python CI checks (Ruff 0.9.9, python/
scope) passed. A real REST + Flight SQL server integration also passed
using the rebuilt SDK: `batch_size=2` limits observed calls in a direct
SELECT over 8 rows, initial materialized-view creation, and
append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime
passthrough; it does not exercise GPU initialization.
Refs ENT-3246. The original GPU initialization and video-decoding
scenario still needs verification on a current deployment; this SDK
change does not close that investigation.
The additional repository-root style checks report existing
docs/web-tests findings (nine format files and 48 lint diagnostics); all
affected files are unchanged from main. These are outside the Python CI
job scope.
Rust compatibility: callers constructing `FunctionRegistrationRequest`
must initialize the two new optional fields (`None` preserves defaults);
the request no longer implements `Eq`.
The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds
creation-guide documentation and integrates main’s namespace
repr/optional OCI embedding changes. The Function registration
implementation is unchanged. All 103 Python slice2 tests passed again on
EC2 using the unchanged native extension; Rust and server-bridge
evidence remains applicable.
Adds `OCIGenAIEmbeddings`, registered as `"oci-genai"`, embedding text with the Cohere models hosted on Oracle Cloud Infrastructure Generative AI (default `cohere.embed-v4.0`) via the `oci` SDK, alongside the existing Bedrock, Gemini and watsonx functions.
- Auth delegated to the OCI SDK: `API_KEY` / `SECURITY_TOKEN` profiles from the OCI config file, or `INSTANCE_PRINCIPAL` / `RESOURCE_PRINCIPAL`. No credentials are stored on the function or in table metadata; `compartment_id` can come from `OCI_COMPARTMENT_ID` so the OCID need not be baked into metadata either.
- Endpoint derived from `region` (default `us-chicago-1`) or overridden with `service_endpoint`; dedicated AI cluster endpoint OCIDs use `DedicatedServingMode`.
- Asymmetric Cohere input types (`SEARCH_DOCUMENT` for the source column, `SEARCH_QUERY` for queries); `truncate` and `output_dimensions` (embed-v4.0: 256/512/1024/1536) forwarded.
- Batched at the service limit of 96 inputs; empty inputs are not sent and come back as `None`.
- `ndims()` uses `output_dimensions` when set, otherwise a single cached probe request, so there is no model→dims table to keep in sync with the catalog.
- SDK retries disabled in favour of LanceDB's `max_retries` backoff; 401/403 service errors carry `status_code` so they surface without retrying, with an actionable message (expired session token, IAM policy, region, model).
- `oci>=2.165.1` added to the `embeddings` extra (the floor where `EmbedTextDetails.output_dimensions` exists).
Tests: 22 mocked unit tests (auth paths, batching, input types, config validation, retry behaviour, pickling, table-metadata round trip) and a live test skipped unless `OCI_COMPARTMENT_ID` is set, which passes against us-chicago-1 for embed-v4.0 at 1536 and 512 dims and embed-multilingual-v3.0 at 1024. Verified end to end with a real LanceDB table (`LanceModel` with `SourceField`/`VectorField`, 6 rows, two semantic queries ranked correctly). `ruff format --check`, `ruff check`, pyright and typos are clean.
Docs: a page under `docs/src/embeddings/available_embedding_models/text_embedding_functions/` in the shape of the remaining `voyageai_embedding.md`; the function is exported from `lancedb.embeddings` so the API reference picks it up. The hosted docs' available-models table lives outside this repo and would need a separate update.
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.
- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.
Linear: OSS-2772
Running a Function body in a local test or another engine currently
requires discovering undocumented APIs. Document direct calls to
decorated Functions and classes, including initialization fields,
instance reuse, and cleanup, and explain how they differ from registered
`FunctionVersion` applications.
Add examples of `BlobReader(read_range, size)` backed by an in-memory
byte-range callback, including cursor-independent reads. The local
examples and rendered API documentation were validated on EC2 at
`df6076e1ae605f425a380d282c4f25b845a3165f`; integration with current
main leaves these documented call paths unchanged.
Fixes ENT-3561.
Packages passed through `@udf(code=[...])` silently omit non-Python
resources, so registration can appear successful before a worker fails
to open a data file. Warn during function preparation with the omitted
paths and document the source-only boundary next to `code=`. The
diagnostic explains how to install a distribution that includes the data
and remove the package from `code=` to avoid shadowing that
installation.
Python cache files do not produce warnings. Excluded directories are
reported at their boundary, preserving the existing traversal and
Python-source artifact format. This implements the
warning-and-documentation alternative in ENT-3492; it does not add
resource uploads.
On EC2, the regression fails on baseline `f9389f113` because no warning
is emitted; all 60 packaging tests pass at `e23445b96`, using the
matching native extension built from unchanged Rust sources.
CI-scoped Python lint with Ruff 0.9.9 passes. Whole-site Mintlify
validation reports the same four snippet-import warnings on the baseline
and this head; repository-root Ruff likewise reports the same 48
findings under `docs/web-tests`. These checks are not clean passes, but
their failures do not involve this change.
Fixes ENT-3492
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.
Co-authored-by: Leo <leo@insight.co>
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.
Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.
SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.
Fixes#1959.
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
Fixes#4761
`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.
The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.
This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.
Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
Renaming or dropping an embedding source or vector column leaves schema metadata pointing at missing fields, causing later `add()` calls to fail.
Reject these operations in local schema evolution before committing any changes. The guard reads Python/Node embedding column metadata and native Rust `lancedb::column_definitions` without loading embedding models. It supports explicit and default Rust vector-column names and respects quoted column paths. Unrelated column changes and nullability updates remain supported.
The restriction is documented in the Python, Rust, and TypeScript APIs. Regression coverage includes both SDK metadata formats, native Rust definitions with explicit and default destinations, atomic rejection of batched changes, and continued embedding generation after reopening.
Fixes#4763
Validation at the exact PR head and GitHub test-merge:
- Hosted macOS Python wheel suite: 3061 passed, 162 skipped, 70 deselected, including all 8 sync and 4 async embedding schema-evolution cases.
- Hosted Windows Rust suite: 1900 library tests passed, 8 embedding integration tests passed, and 107 doctests passed, including the native explicit/default destination and binding guard cases.
- Hosted Linux Python 3.10 rerun: 3084 passed, 159 skipped, 50 deselected; the previously observed remote-fork timeout passed on the replacement job.
- All 10 applicable exact-head workflow runs completed successfully. The existing main-only Windows ARM job was skipped by its workflow condition.
Fixes#4773.
Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.
Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
The vector-search guides do not clearly explain when to use Auto, search
effort or a fixed probe count. They also describe `minimum_nprobes` as
the exact starting count, although Auto can search more partitions from
the start.
Explain how to choose public query controls for recall and latency, with
short executable examples. Clarify that probe limits apply per segment,
calibration targets are not recall guarantees, and more probes cannot
remove quantization error. Keep necessary client/server version guidance
in a short note.
Validation:
- The existing example test passes against published `lancedb==0.41.0`,
covering all four snippets, result counts, filtering, finite distances
and recall against an exact cosine query. The test body takes 0.20 s.
Generated snippets match their source.
- Vale reports no findings on added prose. Changed-function Python
formatting and `git diff --check` pass. Repository-wide Ruff has 137
findings identical to the previous scan, including seven pre-existing
findings in the example file.
- Mint 4.2.888 strict validation reports the same four
`RsExampleBody`/`RsExampleTail` import warnings as pristine base
`0b20965d`, with no additional warnings.
Update the Rust workspace and Java SDK dependencies to [Lance
v15.0.0-beta.8](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.8),
preserve the optional `manifest_size` field in remote version responses,
and verify directory-namespace renames preserve registration and data
both within and across namespaces. Update the Rust and TypeScript API
documentation to reflect directory-namespace rename support.
Rust callers constructing `Version` literals must now provide
`manifest_size` (`None` when unavailable). Java is intentionally pinned
to beta.8 even while Maven artifacts are still publishing;
dependency-resolution CI failures caused solely by that publishing delay
are accepted for this upgrade. Python package pins remain unchanged.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
## Summary
- Add sync/async `DBConnection.list_gpu_classes()` returning typed
`GpuClass` records for `GET /v1/gpu/classes`. Records include the class
name, Kubernetes resource, GPU model and memory, per-worker GPU maximum,
and worker CPU/memory/ephemeral-storage limits.
- Add `udf(gpu_class=...)` for immutable Function-level GPU placement.
It requires `gpu=True`; Function versions expose their registered
`gpu_class`.
- Add `gpu_class` and `gpu_count` to `refresh_column` /
`refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as
top-level `backfill_column` fields and validated before any request.
`gpu_count` is the aggregate GPU budget across Function workers.
- Validate GPU class names with the server lowercase 63-character
DNS-label rule.
- Document the discovery-to-registration flow, configured hardware
versus free capacity, System Usage permission, and per-refresh GPU
placement in the API and web guides.
Older servers without `/v1/gpu/classes` produce a clear
`NotImplementedError` instead of a bare 404.
Part of ENT-3470. The client follows the in-flight Sophon contract on
`xuanwo/ent-3470-fix` for GPU class discovery and backfill placement
options.
## Breaking change
`PythonRuntimeSpec::PythonV2` adds `gpu_class`, and
`RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers
constructing either value with all fields explicitly must supply the new
fields.
## Validation
At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python
extension build, pyright, ruff, and Python Function/runtime, discovery,
refresh placement, and refresh regressions (147 passed, 6 skipped).
At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright,
Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed).
The final commit only updates documentation and Rust doc comments.
## Summary
- Parse structured `error_code` and `resource` fields from remote HTTP
error bodies and carry them through Rust/PyO3.
- Add `SecretNotFoundError`, `FunctionNotFoundError`,
`NamespaceNotFoundError`, and `AlreadyExistsError` as `HttpError`
subclasses.
- Expose `error_code`, `resource_kind`, `resource_name`, and
`namespace_path` on HTTP exceptions while preserving `status_code`,
`request_id`, and existing `except HttpError` handlers.
- Do not retry structured domain errors that would otherwise become
`RetryError` (notably `already_exists` on 409).
Older servers that do not provide the structured fields continue to
raise a plain `HttpError`.
## Compatibility
This consumes the stable resource error contract added in
lancedb/sophon#9516 (merge commit `35858561b3`). The existing
`TableNotFoundError` behavior is unchanged.
Part of ENT-3373.
`list_functions()` promises every published immutable revision. Request
`SHOW FUNCTIONS HISTORY` so the SQL server's latest-only default does
not hide older revisions, and preserve the server's numeric revision
ordering when constructing page cursors (`1`, `2`, `10`).
```python
[(fn.name, fn.version) for fn in db.list_functions(page_limit=2)]
# [("embed", "1"), ("embed", "2"), ("embed", "10")]
```
```sql
SHOW FUNCTIONS; -- latest visible revision per name
SHOW FUNCTIONS HISTORY WITH (limit = 3, include_definition = true);
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '2', limit = 3);
```
Each listing page uses the existing SQL query and polling transport; the
continuation token stores the last returned name/revision and remains
bound to its database and namespace. Function mutations keep their
query-backed job handles: `Job.id` is the query's ID and `job_type` is
`query`, so waiting observes that same SQL execution. No registry job or
second query ID is added.
Update the SQL and Python documentation for exact-revision drops:
sibling revisions and grants survive, and only the last visible
revision's removal releases the name and schedules object cleanup.
Extend the existing Flight SQL regression to paginate revisions `1`,
`2`, and `10` and verify the explicit history statement. Formatting and
static Python checks pass; local builds and tests were intentionally not
run.
View APIs now read the native `name`, `query`, and `schema` columns
returned by `CREATE VIEW` and `DESCRIBE VIEW`. The schema JSON carries
the defining database and namespace; the requested namespace identifies
where the view lives. The unreleased JSON-envelope response is removed
without a compatibility fallback.
```python
view = db.create_view("adults", "SELECT name FROM people WHERE age >= 18")
view = db.describe_view("adults")
```
```sql
CREATE VIEW "adults" AS SELECT name FROM people WHERE age >= 18;
DESCRIBE VIEW "adults";
```
Both Python calls execute SQL through the query-backed job adapter and
wait on the resulting query. A job handle uses that query's `id` and
type `query`; reading metadata consumes its completed result rather than
resubmitting the DDL. Creation returns the definition from the creating
execution, with no extra describe request.
Secret APIs now submit the native SQL statements with the credential
bound as an Arrow string parameter:
```sql
CREATE SECRET "api-key" TYPE = GENERIC_STRING SECRET_STRING = $1;
ALTER SECRET "api-key" SET SECRET_STRING = $1;
DESCRIBE SECRET "api-key";
```
`create_secret` and `alter_secret` execute SQL through the existing
query-backed job adapter and wait for that query to complete. The job's
`id` is the query ID and its type is `query`; credentials stay out of
SQL text. `describe_secret` reads the native `name`,
`created_at_millis`, and `updated_at_millis` columns. This replaces the
unreleased `AS $1` syntax and JSON-envelope response without a
compatibility fallback.
Python methods and namespace handling remain the same:
```python
db.create_secret("api-key", "credential", namespace_path=["team"])
db.alter_secret("api-key", "rotated", namespace_path=["team"])
info = db.describe_secret("api-key", namespace_path=["team"])
```
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
Fixes#4750
Remote missing-table errors retained HTTP metadata in Rust but the Python binding converted them to a plain `ValueError`, discarding the request ID and status. This raises a `TableNotFoundError` that preserves the original message, request ID, status code, and chained HTTP cause while remaining compatible with existing `ValueError` and `lance_namespace.errors.TableNotFoundError` handlers. Local and non-HTTP errors retain their previous behavior.
Validation covers sync and async opens across cached/uncached and root/nested paths, exception pickling, local compatibility, the remote and error test suites, Clippy, formatting, Ruff, and all applicable PR checks including CodeQL.
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model
names outside the three built-in OpenAI names, so schemas can be built for
compatible models.
Preserve existing model defaults and request behavior. Add regression coverage
for the resulting Arrow schema dimensions and for all known OpenAI model
dimensions.
Adds `HUGGINGFACE` and an optional token Secret reference to the
existing storage integration methods. The same SQL transport handles
create, describe, list and drop. Only a Secret reference is stored in
integration metadata; mixed-provider credential options are rejected.
```python
from lancedb.storage_integration import SecretReference
info = db.create_storage_integration(
"private_hub", "hf://datasets/acme/private/raw/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference(
"hf_token", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```
Omit the token Secret for anonymous access at Hugging Face's default
anonymous
rate limits. A tokenless integration does not use a cluster's Hugging
Face token:
```python
info = db.create_storage_integration(
"public_hub", "hf://datasets/acme/public/raw/",
storage_provider="HUGGINGFACE",
)
```
```sql
CREATE STORAGE INTEGRATION public_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```
```sql
CREATE STORAGE INTEGRATION private_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
HUGGINGFACE_TOKEN_SECRET = shared.credentials.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing SQL transport]
SQL --> Metadata[Integration and token Secret reference]
```
Adds Rust request construction, SQL rendering, typed metadata decoding,
sync/async Flight SQL regressions and public documentation. Local
builds/tests are excluded by request; CI provides execution validation.
Adds GCP inputs to the existing database-owned storage integration
methods. GCP takes one Secret reference; the Secret stores the complete
application credentials JSON as an opaque string. The same methods and
SQL transport handle create, describe, list and drop.
```python
info = db.create_storage_integration(
"gcs", "gs://customer-data/raw/", storage_provider="GCP",
gcp_credentials_json_secret=SecretReference(
"gcp_json", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("gcs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("gcs")
```
```sql
CREATE STORAGE INTEGRATION gcs
STORAGE_PROVIDER = 'GCP'
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = shared.credentials.gcp_json;
DESCRIBE STORAGE INTEGRATION gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION gcs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing Flight SQL transport]
SQL --> Config[Configuration and one Secret reference]
```
Adds Rust request construction, SQL rendering and typed metadata
decoding, plus sync/async Flight SQL regressions across the three Arrow
string encodings. Rejects missing or mixed-cloud credential options.
Local builds/tests are excluded by request; validation runs in CI.
Register Azure Blob integrations through Flight SQL using an explicit
storage account and three Secret references for tenant ID, client ID and
client secret. Extends the database-owned integration APIs with
provider-specific inputs while keeping the same create, describe, list
and drop operations.
```python
info = db.create_storage_integration(
"customer_blobs", "az://customer-data/raw/",
storage_provider="AZURE", azure_storage_account="customeraccount",
azure_tenant_id_secret="tenant", azure_client_id_secret="client",
azure_client_secret_secret="credential",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("customer_blobs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("customer_blobs")
```
Sync and async connections accept the same options. Use
`SecretReference` for cross-database or namespace qualification. Values
remain in Secrets; returned metadata includes only references and setup
guidance.
```sql
CREATE STORAGE INTEGRATION customer_blobs
STORAGE_PROVIDER = 'AZURE'
STORAGE_PREFIX = 'az://customer-data/raw/'
STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
STORAGE_AZURE_TENANT_ID_SECRET = public.tenant
STORAGE_AZURE_CLIENT_ID_SECRET = public.client
STORAGE_AZURE_CLIENT_SECRET_SECRET = public.credential;
DESCRIBE STORAGE INTEGRATION customer_blobs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_blobs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Flight SQL]
SQL --> Metadata[Configuration and Secret references]
```
Azure supports service principal client credentials only. Adds SQL
rendering, provider validation and Python Flight SQL regressions across
Utf8, LargeUtf8 and Utf8View results. Local builds and tests are
excluded as requested; validation runs in CI.
Add `on_transform_error="fill"` to `StreamingDataset`, replacing failed
transform rows at their original positions with a deep-copied fill value or
the result of `repair(exc, row_batch)`.
Fill mode preserves split length, elastic step layout, worker accounting, and
checkpoint progress. Repair failures remain fatal, and `rows_filled` is shared
across workers.
Validated with focused fill and adjacent dataloader tests (10 and 6 passed)
and the complete applicable CI run for
6b51b33e56: four workflows and fifteen checks
all succeeded.
Closes#3916
Replaces #4322, keeping only its property graph defined over tables.
A property graph maps node labels and relationship types to existing tables of its namespace, in the shape of SQL/PGQ's `CREATE PROPERTY GRAPH`. It stores only that definition: a query reads the tables as nodes and edges through SQL `GRAPH_TABLE`, as they are when it runs.
- **Rust:** `Connection::create_property_graph` takes a `PropertyGraphDefinition` of `NodeTable`s and `EdgeTable`s. `describe_property_graph`, `drop_property_graph` and `list_property_graphs` manage it.
- **Python:** `lancedb.graph` has `NodeTable`, `EdgeTable` and `Endpoint`. Sync and async connections gain `create_property_graph`, `open_property_graph`, `drop_property_graph` and `list_property_graphs`, and a `PropertyGraph` handle describes its graph.
- **Support:** remote connections only. Local connections raise `NotImplementedError` in Python and return `NotSupported` in Rust.
```python
import lancedb
from lancedb.graph import EdgeTable, Endpoint, NodeTable
db = lancedb.connect("db://demo", api_key="...")
db.create_property_graph(
"social",
nodes=[NodeTable("person", key="person_id", label="Person")],
edges=[
EdgeTable(
"knows",
label="KNOWS",
source=Endpoint("src_id", references=("person", "person_id")),
destination=Endpoint("dst_id", references=("person", "person_id")),
)
],
)
db.execute_query("""
SELECT * FROM GRAPH_TABLE (social
MATCH (p:Person)-[:KNOWS]->(f:Person) WHERE p.age > 30
COLUMNS (p.name AS person, f.name AS friend))
""").read_all()
```
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
Package the standalone lancedb CLI with every stable and beta GitHub
release. Publish archives for Linux x86_64, Linux ARM64, and Apple
Silicon macOS, along with SHA-256 checksums, licenses, and third-party
notices.
Synchronize the CLI version with the SDKs and validate that release tags
match both manifests. Add PR and manual packaging dry runs, runtime
compatibility checks, and smoke tests of extracted executables. GitHub
release creation now waits for all three archives to pass validation;
SDK registry publishing remains independent.
Update installation instructions and the release checklist. Linux
binaries target glibc 2.28, retaining existing x86_64 CPU requirements;
macOS binaries target macOS 14 or newer.
Stacked on #4675.
Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.
`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
Stacked on #4674.
Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens.
The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates.
Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`.
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.
### Python experience
```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```
For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.
### SQL experience
```sql
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```
CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.
Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
## Summary
Fixes the problems a page-by-page review of the docs found before
launch, after checking each one against the current source and, where
noted, the code. The matching fixes for the dataset and Geneva pages,
which live in `lancedb/docs`, are in lancedb/docs#402.
**Tables**
- `integrations/embedding/cohere`: each input-type description was split
across two table rows and wrapped in literal quotes
(`"`search_document`"`). Now one row per type.
- `integrations/embedding/gemini`: removes the same stray quotes from
the task-type table.
- `integrations/ai/langchain`: the MMR search table's `kwargs` row was
missing a cell, so its description sat under Type.
- Six reranker pages (answerdotai, cohere, colbert, jina, openai,
voyageai): closes the backtick before `_relevance_score`.
- `indexing/vector-index`: finishes a cell cut off at "with scalar".
- `sql/dql/dedupe`: labels the empty header cell in the resolution
table.
**Corrections**
- `curation/sampling`: sampling works in LanceDB Embedded through the
Python SDK, so the page is available in both, with the SQL sections
marked Enterprise.
- `feature-engineering/index`, `add-computed-column`: automatic refresh
is on by default for each computed column and takes effect once a
deployment enables its scheduler. `refresh-column` already said so;
these two implied it was off.
- `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates
`array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also
merges scalar-index's two "nested fields" sections into one.
- `integrations/embedding/ibm`: lists the four current models from
`CURRENT_MODELS` (it listed a deprecated one) and explains that the
default stays a legacy ID so existing tables reload with their model.
- `integrations/reranking/openai`: the class is `OpenaiReranker`, not
`OpenAI()`.
- `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit`
throughout, each step runs one command, each step passes the file it
writes to the next, and `save-as` uses `-f`/`-o`.
- `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the
pipeline reads, and the run command runs the file the page creates.
- `search/vector-search`: the multivector walkthrough connected to
Enterprise in step 1 and a local path in step 2. It now runs locally
throughout, with a note on connecting to Enterprise, and is renamed so
it doesn't read as a duplicate of the short section above.
**Structure and wording**
- `integrations/embedding/nvidia`: adds headings, and drops test-suite
details aimed at contributors.
- `integrations/ai/genkit`: makes the retrieval test a real heading and
describes the query step.
- `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a
`<Note>`.
- `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading,
not bold text.
- `curation/windowing`: sidebar title matches the curation overview
("Rank and Deduplicate").
- `deployment/storage`, `basics/agentic-skills`: fixes a garbled
sentence and a stale reference to the quickstart.
**Checked:** no new Vale alerts on the changed pages. Assembled together
with lancedb/docs#402, the site passes `mint validate` and `mint
broken-links --check-anchors --check-redirects`. Every changed page was
reviewed in a local preview.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add a LanceDB command line with the following functionality:
- `lancedb auth`: log in with OAuth, check cached OAuth session status,
or log out.
- `lancedb profile`: Create, edit, list, show, select, and delete
connection profiles.
- When no subcommand is given, we will attempt to create a SQL shell.
Update ci/check_cargo_lock.py and its regression tests to allow
pruning dependency edges while their target packages remain in
Cargo.lock. Removing the CLI from the Python source distribution
can disable optional features such as Axum tracing, even though
other SDK dependencies still require the tracing package.
## Summary
Replaces the generated availability table under **Compare features** on
the landing page with a hand-written list of features, grouped by area,
with separate Embedded and Enterprise columns.
The generated table listed every page that declares `availability`,
including overview and basic CRUD pages. Rows such as "Overview"
appeared three times without context, and pages like Insert, Update,
Database and the integrations showed up as if they were features.
| Area | Features |
| --- | --- |
| Search | Vector, full-text and hybrid search, filtering, reranking,
query plans, distributed search |
| Indexing | Vector, full-text and scalar indexes |
| SQL and curation | SQL, aggregation, sampling, joins, ranking and
deduplication, near-duplicate detection, chunking, changes since a
version, views |
| Feature engineering | Computed columns, materialized views, embedding
registry |
| Training | PyTorch data loader, permutations, distributed training |
| Data management | Branches, compaction and cleanup, background jobs,
Real Time engine |
| Security | Authentication, authorization |
Each ✅/❌ matches the linked page's `availability` on `main`; SQL, which
declares none, is Enterprise-only. Functions are left out until it's
settled whether they are published.
The same list, limited to 1.10 features, is in the Enterprise 1.10
editions in lancedb/docs#398.
Checked by assembling the docs site with this change: `mint validate`
passes and `mint broken-links --check-anchors --check-redirects` finds
no broken links. Vale reports no new warnings.
After this merges, the committed `lancedb` skill in lancedb/docs needs a
rebuild (`make assemble && make skill`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fixes#4733
PyArrow expects Blob V2 storage structs when constructing a record batch
and cannot infer the extension array from raw bytes. LanceDB already
converts binary values during ingestion, but batch writers had no
supported array constructor.
Add `lancedb.schema.blob_array(payloads)` for iterables of inline bytes
and nulls, using the same extension type resolver as `blob()`. The
helper preserves empty blobs and null blobs and works with both
pylance's type and LanceDB's fallback. Document the descriptor fields,
exclusive inline/external representations, batch construction, and
existing binary conversion in synchronous and asynchronous
`create_table` / `add` APIs.
Validation:
- Reproduced the reported `ArrowTypeError` on the original code; both
new regression cases fail against the original schema module.
- 256 focused blob tests and schema doctests passed; one optional PyAV
test skipped.
- Both new regression cases also passed with PyArrow 25, with and
without pylance imports.
- Python package Ruff checks and formatting passed.
- MkDocs build passed, with an existing unrelated `SecretReference`
cross-reference warning. Repository-wide Ruff reports 48 existing errors
in `docs/web-tests`.
<!-- lance-gatekeeper-fix:v1 agent=20d1a296d76ff5eb914c0848428a747a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
Changes links to home page sections from `/#<anchor>` to
`/index#<anchor>`.
The docs site's Assemble CI fails `mint broken-links --check-anchors` on
every `/#architecture` link since #4732 moved the Architecture page onto
the home page. The anchor exists; Mint 4.2.888's checker can't resolve
any anchor on the root page `/`, but resolves `/index#…`.
`docs.lancedb.com/index` serves the home page, so readers land in the
same place.
Verified by assembling the site with this change and the matching
Enterprise docs change: `mint validate` passes and `mint broken-links
--check-anchors --check-redirects` finds no broken links. The
`docs.json` redirects keep `/#architecture`, which the checker accepts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Addresses some comments on the staging branch. Rebased onto #4732, which
rebuilt `index.mdx` and the quickstart and reverted the landing copy to
the pre-v0.4 headings; the final copy is reapplied here.
| Comment | Page | Change |
| --- | --- | --- |
| Taylor: drop the comma | `index.mdx` | "one, unified platform" → "one
unified platform" |
| Taylor: "Preparing t…" + "Data doesn't slow down" | `index.mdx` |
"Training data and experimentation slow down…" → "Preparing training
data and running experiments slow down…" |
| Fei: subtitles should align with the new home page | `index.mdx` |
Follows the final homepage copy (v0.4): "Train on the right data faster"
with the hero subtitle, "Accelerate the data flywheel for model
research" with its intro, and the five v0.4 sections as steps, each
linking to the page that backs it (search, curation, branches, versions,
computed columns and materialized views, `StreamingDataset`). Wording
the docs can't back yet is left out: "zero-copy", "lineage", "cached
across clouds". Anchors renamed to match; nothing links to the old ones
|
| Fei: quickstart seems OSS-only | `quickstart.mdx` | Tip at the top:
for Enterprise only the connection changes, linking the Enterprise
connection in step 2 |
Fei's "list all Enterprise features as a catalog" comment is now covered
by #4732's feature comparison table (`/#compare`), so the separate table
this PR added to `basics/offerings.mdx` was dropped along with that
page.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Updates the overview diagrams on the landing page. Rebased onto #4732,
which moved the architecture and offerings content into `index.mdx` and
drew it as Mermaid diagrams; this PR replaces those with the guide's
diagrams.
- **Architecture** (`#architecture`): the suite diagram
(`lancedb-suite.svg`) replaces the first Mermaid diagram. Its Enterprise
boxes read Search, Data exploration & curation, Feature engineering,
Training, matching the guide's product hierarchy.
- **LanceDB Embedded** (`#1-lancedb-oss`): adds the guide's Embedded
diagram. Its storage box reads "Local Disk or Object Storage", since
Embedded supports both.
- **LanceDB Enterprise** (`#2-lancedb-enterprise`): the guide's
Distributed diagram replaces the second Mermaid diagram. This also drops
that diagram's "column + MV refresh" label; automatic refresh is still
gated.
- **Functions** are still unpublished, so the "Functions ·" label is
removed from the Distributed diagram's jobs fleet. At release, restore
the designer's original image from the Notion guide.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause,
and
`dedupe()`/`duplicate_pairs()` (verified against current sophon source;
these were recently renamed from
`vector_dedup`/`vector_duplicate_pairs`).
Fix the blob-column default-projection claim and add the previously
undocumented `blob_descriptor()`.
- Enrich the gated Functions draft (still `.mintignore`'d, pre-release):
chained Functions over blob outputs, GPU-produced blobs, Secrets,
multi-input/output binding, `code=[module]` packaging.
- Upgrade the materialized-view and computed-column auto-refresh drafts
with
precise, verified catch-up semantics, and add the missing `ALTER
MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the
automatic-refresh scheduler is still dev/staging-only).
- Thread one running example (video frames → GPU extract/caption → dedup
→
auto-refreshing materialized view) through Functions, materialized
views,
and the LeRobotDataset integration guide instead of a separate tutorial.
- Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout
(52 files); activate the existing availability-comparison mechanism on
`basics/offerings.mdx` for a full feature-by-feature comparison table.
Adds a Python `nvidia` embedding provider for Nemotron 3.5 Embed 1B and 8B. It works with NVIDIA's hosted API and self-hosted NIM `/v1` endpoints, and uses the existing `openai` client. The OpenAI provider can't send NVIDIA's passage/query input types or pick Nemotron's dimensions.
- Documents are embedded as `passage`, searches as `query`.
- Default dimensions are 2048 (1B) and 4096 (8B), and `dim` is checked against each model's supported range. Self-hosted model IDs need an explicit `dim`.
- Response count, order, dimensions and values are checked. Null and empty text get a null vector without a request.
- Each request is retried on its own with LanceDB's retry policy. Invalid input or responses fail immediately.
- `NVIDIA_API_KEY` is only sent to the hosted API. Custom endpoints need an explicit `api_key` (a `$var:` reference), and endpoints restored from table metadata must be allowed with `allow_endpoint`.
- Request dimensions are sent through `extra_body`, preserving the NVIDIA wire contract while supporting the declared minimum `openai==1.6.1`.
Python and text only. Docs are in `integrations/embedding/nvidia`.
Originally authored by Taylor (@tsmithv11).
### Testing
- `test_nvidia_embeddings.py`: 45 passed, 2 deselected with both the locked `openai==2.44.0` and minimum `openai==1.6.1`. These use the real OpenAI client with mock HTTP transport and real local tables.
- Embedding, registry, field-metadata and trust suites: 163 passed, 4 skipped, 10 deselected.
- Historical live experiments: hosted 8B passed a five-request smoke test; hosted 1B returned HTTP 500. No live request was made in this stage, so current hosted 1B availability was not verified. Enterprise coverage uses the repository mock HTTP server, not a live deployment.
Xuanwo
Lance Release
Jack YeandClaude Sonnet 5
Gaétan
Fede Kamelhar
LeoandLeo
Srijan Keshri
Wyatt Alt
lancedb-gatefixer[bot]
TaylorandBrendan Clement
Yang Cen
LanceDB RobotandJack Ye
BHAWESHBHASKAR
alexcastgonz
importcpp
ryan-haynesandRyan Haynes
Colin Patrick McCabe