## Summary
- Adds a new top-level "Introduction" nav section before "Getting
Started": LanceDB, LanceDB Embedded, LanceDB Enterprise, Embedded vs
Enterprise.
- Trims the landing page's combined
Architecture/Embedded/Enterprise/Compare block down to a short pointer
section (keeping its `#architecture` anchor), moving that content into
two new pages: `basics/lancedb-embedded.mdx` and
`basics/embedded-vs-enterprise.mdx`.
- `basics/embedded-vs-enterprise.mdx` consolidates the
feature-comparison table and the Enterprise overview's own comparison
content (previously duplicated across multiple pages and, within this
page, repeated across a dimension table, several narrative subsections,
and a usage-differences list) into one comparison page with each section
covering a distinct angle.
- Repoints the `/enterprise` and `/enterprise/deployment` redirects at
`/basics/embedded-vs-enterprise`, since the previous destination only
resolves once the Enterprise overlay is applied and fails
`rust/lancedb/src/docs.rs`'s link-resolution test against this repo's
own `docs/web` tree.
## Companion change
lancedb/sophon#9718 trims the Enterprise overview page to drop the
content that moved here, and relocates its nav insertion into the new
Introduction group.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
Follow-up to the Mermaid brand-theming PR, addressing review feedback:
- Diagram text was hard to read at the previous 15px font size; bumped
to 18px, light and dark alike.
- The dark-mode orange accent border was brighter and more saturated
than its fill or the default border, reading as too loud against the
dark background. It now reuses the same muted orange already used for
the light-mode accent border, which was fine as is.
## Companion change
lancedb/sophon#9719 makes the same two changes across its own diagrams.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Remote SQL failures currently become generic runtime errors, and HTTP
table lookup can discard server error metadata. Preserve transport
status, current request ID, SQL job ID and worker request ID across Rust
and Python. Select typed column, table, reference and schema exceptions
only when an explicit server code matches the transport status.
Remote `HttpError` and `RuntimeError` handlers remain compatible. Failed
SQL jobs retain `JobFailedError` with a typed SQL cause when available.
Missing metadata stays absent; message text never determines the
exception class. Local missing-column errors retain their type and
qualified-field suggestions, and `JobFailure.source` documents
structured remote Flight SQL failures.
This branch integrates main at `a9ad4997`, including its released Lance
15.0.0-beta.9 dependency update and remote optimize support. The only
textual conflict was two independent additions to `test_remote_db.py`;
both test groups and all original assertions are retained. The feature
has no dependency or lockfile delta against current main.
SDK runtime validation on `03ee633a` (runtime source unchanged in
`f3f2d0b8`), macOS ARM64, pinned Rust 1.97.0 and the frozen uv Python
3.14 environment:
- Rebuilt the native extension from the integrated source. All 22
targeted Python tests passed: native local errors, HTTP/Flight metadata,
legacy sync/async table-not-found behavior, and the three main
remote-optimize regressions.
- All 52 Rust SQL, metadata/source and nested-field tests passed,
including the new main optimize cases.
- Root and Python-binding Rust Clippy with `-D warnings`, Rust
formatting, Python package Ruff and conflict-file formatting passed.
- Repository-wide Ruff reports 59 existing documentation findings, and
its format check identifies six documentation files. Every affected file
is byte-identical to current main; these checks are not reported as
passing.
- Direct integration review and AST comparison confirm both conflicting
test groups were preserved. Earlier exception-constructor/pickling
evidence remains applicable to the unchanged Python error modules.
The unlocked build exhausted disk again on a runner with the same label
but only 9.4 GiB free on a 72 GiB root filesystem. Keep the existing
runner and dependency caches, serialize Cargo compilation, and remove
only completed test/bench executables reported by Cargo. This job never
runs those binaries; normal binaries, libraries and build-script outputs
remain available. Fresh resolution, the ci profile, all features, and
test/bench compilation and linking are retained.
Three probes using Cargo and target-path validation pass with `-D
warnings`: all fixture targets link while ordinary outputs remain
usable; an undefined symbol still produces Cargo exit 101 and its linker
diagnostic; files outside the target directory are never removed. The
[current-head build-no-lock
job](https://github.com/lancedb/lancedb/actions/runs/41911613851/job/126860364065)
passed on `f3f2d0b8`: all compilation/linking completed, 20 completed
test executables were removed, and `target` retained about 7.3 GiB. This
successful run had a 145 GiB root with about 78 GiB free afterward; it
was not an identical rerun of the failed 72 GiB image. The driver tests
also passed in this job. All previous-head jobs ended naturally before
the push; new-head CI and review gates still apply.
Typed remote input exceptions require the server metadata contract.
Older servers retain generic transport errors and existing missing-table
compatibility. The optimize tests use mock servers and do not establish
downstream server acceptance.
BREAKING CHANGE: Rust callers with exhaustive matches on
`lancedb::Error` must handle the new `FieldNotFound` and `Sql` variants.
Local `Error::ColumnNotFound` failures now raise
`lancedb.exceptions.ColumnNotFoundError`, a `ValueError` subclass.
Python callers that previously caught only `RuntimeError` for this local
failure must catch `ColumnNotFoundError` or `ValueError`.
`Table::optimize()` previously returned `NotSupported` for remote
tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE
TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL.
Only `OptimizeAction::Compact` without a custom `remap_options` or
non-default `CompactionOptions` is supported remotely.
`OptimizeAction::All`, `::Prune`, and `::Index` all return a clear
`NotSupported` error instead of silently doing the wrong thing. A
branch-scoped table handle submits a `WITH (branch = '<branch>')`
clause; the statement doesn't validate the branch exists itself, so an
unknown branch surfaces as an async job failure rather than an error
from this call.
The server's SQL job adapter doesn't expose compaction metrics, so
`OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a
placeholder rather than real numbers — it must still be `Some`, since
callers (including the Python/Node bindings) unwrap it unconditionally
on success. This mirrors how `refresh_materialized_view_async` already
handles the same limitation for its own metrics.
The Python and Node `optimize()` convenience methods call a new shared
`Table::optimize_full`, which runs `Compact` and then, on a remote
table, skips `Prune` and `Index` instead of attempting and failing on
them — returning zeroed prune stats rather than submitting a compaction
job and then raising. Python's `RemoteTable.optimize()` also now
forwards `compaction_options` to the Rust layer, so a non-default value
is rejected before submission instead of being silently dropped.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
Every Mermaid diagram now renders in the brand palette, with separate
light and dark variants that switch with the site's theme toggle.
Default node styling stays neutral; one accent color marks each
diagram's actual subject, chosen per diagram. A couple of diagrams
changed orientation where their old layout was too wide to read
comfortably.
Companion change in lancedb/sophon uses the same colors and pattern.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Python Function declarations can now pin Hugging Face snapshots with
`models=[HuggingFaceModel(repo=..., revision=...)]`. The native Rust
registration path carries those declarations into `CREATE FUNCTION`, and
create/get/list results retain `image.models`. Functions without
declarations preserve their existing payload shape. The API reference
includes a model-loading example that initializes once per Function
instance.
Validation:
- The real SQL mock-service regression fails on the previous native
implementation and passes after the fix. It covers immediate/deferred
execution, requests with/without models, and create/get/list responses.
- Rust Function wire tests, strict Clippy, the public-type doctest, and
formatting.
- Rebuilt Python extension and targeted registration/golden tests.
The service-side snapshot preparation and cache are delivered separately
for ENT-3483.
The removed Enterprise landing page left the `/enterprise` and
`/enterprise/deployment` redirects pointing at a missing page, breaking
the SDK documentation-link test. Point both redirects at the existing
architecture overview, which describes Embedded and Enterprise.
The exact failing Rust test fails on current main and passes with these
redirect changes. This repairs the inherited failure blocking #4821.
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:
**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.
**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.
**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.
**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.
**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.
Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot
integration guide with `Table.fetch_blob_files`, document how to open
`.lance` datasets directly with `lancedb` alongside a
`pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in
installation docs that the optional `pylance` extra is only needed for
engine/connector authors or format-level fragment APIs.
TypeScript job handles now expose structured progress through
`progress()` and an optional synchronous callback to `wait()`. Snapshots
preserve unknown fields and missing counters, and job listings include
progress and pause state. Existing `status()` strings and no-argument
`wait()` behavior remain compatible.
Wait callbacks observe the existing backend poller. Observer exceptions
are logged without changing the job outcome. A best-effort ETA is
available only from compatible samples with positive progress, and is
omitted during initialization, queueing, or pause.
Part of ENT-3500.
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-12-104.us-east-2.compute.internal>
On remote tables, `create_index` returns as soon as the build is scheduled. Until it finishes, a vector index is missing from `list_indices`, a scalar or FTS index shows 0 indexed rows, and queries don't use the index. The docs didn't say any of this, so it was easy to assume the index was ready.
This adds that to the `create_index` and `list_indices` docs in Python (`Table` and `AsyncTable`) and TypeScript, and points to `wait_timeout` / `waitTimeoutSeconds`, `wait_for_index`, and `create_index_async` for blocking until the index is ready. It also explains that completion tracking is tied to the submitted build only when the server returns a job ID, while legacy name-based waits can return early when replacing an existing index.
Closes#4405
Job callers currently lose the server's structured progress when
describing, listing, or waiting for a job. Expose Rust and Python
progress snapshots and optional wait observers using the existing
backend poller. Python supports callbacks, a caller-owned tqdm-like bar,
or `progress=True`; `status()` still returns a string and the default
wait behavior is unchanged.
Preserve unknown progress fields and phases from newer servers, and
report omitted counters as unknown. Each wait calibrates a caller-owned
bar from its first complete snapshot, then uses a high-water mark within
the same execution epoch; a new epoch or smaller plan resets the bar.
ETA is best-effort and omitted for missing, stale, paused, or
incompatible samples.
Newly submitted index and refresh handles forward observers through
their freshness wrapper while retaining successful-wait read
consistency. The Jobs guide includes draft Python/Rust snapshot and
observer examples for release; TypeScript progress support remains
tracked under ENT-3500.
Validation on the development EC2 host: the initial 11 Python progress
tests, REST fixture caller, and two Rust progress regressions passed.
After fixing freshness-wrapper forwarding, rebuilt the Python extension
and ran two submitted-handle Python tests plus the new Rust refresh and
index regressions (one test each); all passed. Strict Clippy, rustfmt,
and targeted Ruff passed. The final commit only wraps the new guide
section in an MDX draft and clarifies language support. Repository-wide
Ruff reports 48 existing findings in six unchanged docs/web-tests files.
This publishes the SDK portion of ENT-3500. Server-side live-split and
terminal-progress changes and the combined server/SDK end-to-end
scenario remain in progress. Refs ENT-3500.
BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress:
None`, `phase: None`, and `paused: false`. Literals for `JobDescription`
must add `progress: None`, `phase: None`, `queue_reason: None`,
`waiting_for_job_id: None`, and `paused_at_millis: None` when that
metadata is unavailable. Existing Python calls remain compatible.
Python UDF registration could not set `batch_size` or
`row_timeout_seconds`, and the Rust registration model discarded these
fields before submitting SQL. Expose both optional controls and preserve
them through registration, with invalid values rejected and unchanged
canonical bytes when the controls are omitted.
The timeout applies to each Python call: one row for scalar Functions
and one batch for vectorized Functions. Initialization keeps its
existing service deadline.
Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust
registration-contract tests, 2 outbound-SQL tests, and all 103 Python
slice2 tests passed after rebuilding the native extension. Rust
formatting and the repository Python CI checks (Ruff 0.9.9, python/
scope) passed. A real REST + Flight SQL server integration also passed
using the rebuilt SDK: `batch_size=2` limits observed calls in a direct
SELECT over 8 rows, initial materialized-view creation, and
append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime
passthrough; it does not exercise GPU initialization.
Refs ENT-3246. The original GPU initialization and video-decoding
scenario still needs verification on a current deployment; this SDK
change does not close that investigation.
The additional repository-root style checks report existing
docs/web-tests findings (nine format files and 48 lint diagnostics); all
affected files are unchanged from main. These are outside the Python CI
job scope.
Rust compatibility: callers constructing `FunctionRegistrationRequest`
must initialize the two new optional fields (`None` preserves defaults);
the request no longer implements `Eq`.
The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds
creation-guide documentation and integrates main’s namespace
repr/optional OCI embedding changes. The Function registration
implementation is unchanged. All 103 Python slice2 tests passed again on
EC2 using the unchanged native extension; Rust and server-bridge
evidence remains applicable.
Adds `OCIGenAIEmbeddings`, registered as `"oci-genai"`, embedding text with the Cohere models hosted on Oracle Cloud Infrastructure Generative AI (default `cohere.embed-v4.0`) via the `oci` SDK, alongside the existing Bedrock, Gemini and watsonx functions.
- Auth delegated to the OCI SDK: `API_KEY` / `SECURITY_TOKEN` profiles from the OCI config file, or `INSTANCE_PRINCIPAL` / `RESOURCE_PRINCIPAL`. No credentials are stored on the function or in table metadata; `compartment_id` can come from `OCI_COMPARTMENT_ID` so the OCID need not be baked into metadata either.
- Endpoint derived from `region` (default `us-chicago-1`) or overridden with `service_endpoint`; dedicated AI cluster endpoint OCIDs use `DedicatedServingMode`.
- Asymmetric Cohere input types (`SEARCH_DOCUMENT` for the source column, `SEARCH_QUERY` for queries); `truncate` and `output_dimensions` (embed-v4.0: 256/512/1024/1536) forwarded.
- Batched at the service limit of 96 inputs; empty inputs are not sent and come back as `None`.
- `ndims()` uses `output_dimensions` when set, otherwise a single cached probe request, so there is no model→dims table to keep in sync with the catalog.
- SDK retries disabled in favour of LanceDB's `max_retries` backoff; 401/403 service errors carry `status_code` so they surface without retrying, with an actionable message (expired session token, IAM policy, region, model).
- `oci>=2.165.1` added to the `embeddings` extra (the floor where `EmbedTextDetails.output_dimensions` exists).
Tests: 22 mocked unit tests (auth paths, batching, input types, config validation, retry behaviour, pickling, table-metadata round trip) and a live test skipped unless `OCI_COMPARTMENT_ID` is set, which passes against us-chicago-1 for embed-v4.0 at 1536 and 512 dims and embed-multilingual-v3.0 at 1024. Verified end to end with a real LanceDB table (`LanceModel` with `SourceField`/`VectorField`, 6 rows, two semantic queries ranked correctly). `ruff format --check`, `ruff check`, pyright and typos are clean.
Docs: a page under `docs/src/embeddings/available_embedding_models/text_embedding_functions/` in the shape of the remaining `voyageai_embedding.md`; the function is exported from `lancedb.embeddings` so the API reference picks it up. The hosted docs' available-models table lives outside this repo and would need a separate update.
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.
- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.
Linear: OSS-2772
Running a Function body in a local test or another engine currently
requires discovering undocumented APIs. Document direct calls to
decorated Functions and classes, including initialization fields,
instance reuse, and cleanup, and explain how they differ from registered
`FunctionVersion` applications.
Add examples of `BlobReader(read_range, size)` backed by an in-memory
byte-range callback, including cursor-independent reads. The local
examples and rendered API documentation were validated on EC2 at
`df6076e1ae605f425a380d282c4f25b845a3165f`; integration with current
main leaves these documented call paths unchanged.
Fixes ENT-3561.
Packages passed through `@udf(code=[...])` silently omit non-Python
resources, so registration can appear successful before a worker fails
to open a data file. Warn during function preparation with the omitted
paths and document the source-only boundary next to `code=`. The
diagnostic explains how to install a distribution that includes the data
and remove the package from `code=` to avoid shadowing that
installation.
Python cache files do not produce warnings. Excluded directories are
reported at their boundary, preserving the existing traversal and
Python-source artifact format. This implements the
warning-and-documentation alternative in ENT-3492; it does not add
resource uploads.
On EC2, the regression fails on baseline `f9389f113` because no warning
is emitted; all 60 packaging tests pass at `e23445b96`, using the
matching native extension built from unchanged Rust sources.
CI-scoped Python lint with Ruff 0.9.9 passes. Whole-site Mintlify
validation reports the same four snippet-import warnings on the baseline
and this head; repository-root Ruff likewise reports the same 48
findings under `docs/web-tests`. These checks are not clean passes, but
their failures do not involve this change.
Fixes ENT-3492
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.
Co-authored-by: Leo <leo@insight.co>
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.
Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.
SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.
Fixes#1959.
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
Fixes#4761
`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.
The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.
This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.
Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
Renaming or dropping an embedding source or vector column leaves schema metadata pointing at missing fields, causing later `add()` calls to fail.
Reject these operations in local schema evolution before committing any changes. The guard reads Python/Node embedding column metadata and native Rust `lancedb::column_definitions` without loading embedding models. It supports explicit and default Rust vector-column names and respects quoted column paths. Unrelated column changes and nullability updates remain supported.
The restriction is documented in the Python, Rust, and TypeScript APIs. Regression coverage includes both SDK metadata formats, native Rust definitions with explicit and default destinations, atomic rejection of batched changes, and continued embedding generation after reopening.
Fixes#4763
Validation at the exact PR head and GitHub test-merge:
- Hosted macOS Python wheel suite: 3061 passed, 162 skipped, 70 deselected, including all 8 sync and 4 async embedding schema-evolution cases.
- Hosted Windows Rust suite: 1900 library tests passed, 8 embedding integration tests passed, and 107 doctests passed, including the native explicit/default destination and binding guard cases.
- Hosted Linux Python 3.10 rerun: 3084 passed, 159 skipped, 50 deselected; the previously observed remote-fork timeout passed on the replacement job.
- All 10 applicable exact-head workflow runs completed successfully. The existing main-only Windows ARM job was skipped by its workflow condition.
Fixes#4773.
Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.
Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
The vector-search guides do not clearly explain when to use Auto, search
effort or a fixed probe count. They also describe `minimum_nprobes` as
the exact starting count, although Auto can search more partitions from
the start.
Explain how to choose public query controls for recall and latency, with
short executable examples. Clarify that probe limits apply per segment,
calibration targets are not recall guarantees, and more probes cannot
remove quantization error. Keep necessary client/server version guidance
in a short note.
Validation:
- The existing example test passes against published `lancedb==0.41.0`,
covering all four snippets, result counts, filtering, finite distances
and recall against an exact cosine query. The test body takes 0.20 s.
Generated snippets match their source.
- Vale reports no findings on added prose. Changed-function Python
formatting and `git diff --check` pass. Repository-wide Ruff has 137
findings identical to the previous scan, including seven pre-existing
findings in the example file.
- Mint 4.2.888 strict validation reports the same four
`RsExampleBody`/`RsExampleTail` import warnings as pristine base
`0b20965d`, with no additional warnings.
Update the Rust workspace and Java SDK dependencies to [Lance
v15.0.0-beta.8](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.8),
preserve the optional `manifest_size` field in remote version responses,
and verify directory-namespace renames preserve registration and data
both within and across namespaces. Update the Rust and TypeScript API
documentation to reflect directory-namespace rename support.
Rust callers constructing `Version` literals must now provide
`manifest_size` (`None` when unavailable). Java is intentionally pinned
to beta.8 even while Maven artifacts are still publishing;
dependency-resolution CI failures caused solely by that publishing delay
are accepted for this upgrade. Python package pins remain unchanged.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
## Summary
- Add sync/async `DBConnection.list_gpu_classes()` returning typed
`GpuClass` records for `GET /v1/gpu/classes`. Records include the class
name, Kubernetes resource, GPU model and memory, per-worker GPU maximum,
and worker CPU/memory/ephemeral-storage limits.
- Add `udf(gpu_class=...)` for immutable Function-level GPU placement.
It requires `gpu=True`; Function versions expose their registered
`gpu_class`.
- Add `gpu_class` and `gpu_count` to `refresh_column` /
`refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as
top-level `backfill_column` fields and validated before any request.
`gpu_count` is the aggregate GPU budget across Function workers.
- Validate GPU class names with the server lowercase 63-character
DNS-label rule.
- Document the discovery-to-registration flow, configured hardware
versus free capacity, System Usage permission, and per-refresh GPU
placement in the API and web guides.
Older servers without `/v1/gpu/classes` produce a clear
`NotImplementedError` instead of a bare 404.
Part of ENT-3470. The client follows the in-flight Sophon contract on
`xuanwo/ent-3470-fix` for GPU class discovery and backfill placement
options.
## Breaking change
`PythonRuntimeSpec::PythonV2` adds `gpu_class`, and
`RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers
constructing either value with all fields explicitly must supply the new
fields.
## Validation
At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python
extension build, pyright, ruff, and Python Function/runtime, discovery,
refresh placement, and refresh regressions (147 passed, 6 skipped).
At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright,
Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed).
The final commit only updates documentation and Rust doc comments.
## Summary
- Parse structured `error_code` and `resource` fields from remote HTTP
error bodies and carry them through Rust/PyO3.
- Add `SecretNotFoundError`, `FunctionNotFoundError`,
`NamespaceNotFoundError`, and `AlreadyExistsError` as `HttpError`
subclasses.
- Expose `error_code`, `resource_kind`, `resource_name`, and
`namespace_path` on HTTP exceptions while preserving `status_code`,
`request_id`, and existing `except HttpError` handlers.
- Do not retry structured domain errors that would otherwise become
`RetryError` (notably `already_exists` on 409).
Older servers that do not provide the structured fields continue to
raise a plain `HttpError`.
## Compatibility
This consumes the stable resource error contract added in
lancedb/sophon#9516 (merge commit `35858561b3`). The existing
`TableNotFoundError` behavior is unchanged.
Part of ENT-3373.
`list_functions()` promises every published immutable revision. Request
`SHOW FUNCTIONS HISTORY` so the SQL server's latest-only default does
not hide older revisions, and preserve the server's numeric revision
ordering when constructing page cursors (`1`, `2`, `10`).
```python
[(fn.name, fn.version) for fn in db.list_functions(page_limit=2)]
# [("embed", "1"), ("embed", "2"), ("embed", "10")]
```
```sql
SHOW FUNCTIONS; -- latest visible revision per name
SHOW FUNCTIONS HISTORY WITH (limit = 3, include_definition = true);
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '2', limit = 3);
```
Each listing page uses the existing SQL query and polling transport; the
continuation token stores the last returned name/revision and remains
bound to its database and namespace. Function mutations keep their
query-backed job handles: `Job.id` is the query's ID and `job_type` is
`query`, so waiting observes that same SQL execution. No registry job or
second query ID is added.
Update the SQL and Python documentation for exact-revision drops:
sibling revisions and grants survive, and only the last visible
revision's removal releases the name and schedules object cleanup.
Extend the existing Flight SQL regression to paginate revisions `1`,
`2`, and `10` and verify the explicit history statement. Formatting and
static Python checks pass; local builds and tests were intentionally not
run.
View APIs now read the native `name`, `query`, and `schema` columns
returned by `CREATE VIEW` and `DESCRIBE VIEW`. The schema JSON carries
the defining database and namespace; the requested namespace identifies
where the view lives. The unreleased JSON-envelope response is removed
without a compatibility fallback.
```python
view = db.create_view("adults", "SELECT name FROM people WHERE age >= 18")
view = db.describe_view("adults")
```
```sql
CREATE VIEW "adults" AS SELECT name FROM people WHERE age >= 18;
DESCRIBE VIEW "adults";
```
Both Python calls execute SQL through the query-backed job adapter and
wait on the resulting query. A job handle uses that query's `id` and
type `query`; reading metadata consumes its completed result rather than
resubmitting the DDL. Creation returns the definition from the creating
execution, with no extra describe request.
Secret APIs now submit the native SQL statements with the credential
bound as an Arrow string parameter:
```sql
CREATE SECRET "api-key" TYPE = GENERIC_STRING SECRET_STRING = $1;
ALTER SECRET "api-key" SET SECRET_STRING = $1;
DESCRIBE SECRET "api-key";
```
`create_secret` and `alter_secret` execute SQL through the existing
query-backed job adapter and wait for that query to complete. The job's
`id` is the query ID and its type is `query`; credentials stay out of
SQL text. `describe_secret` reads the native `name`,
`created_at_millis`, and `updated_at_millis` columns. This replaces the
unreleased `AS $1` syntax and JSON-envelope response without a
compatibility fallback.
Python methods and namespace handling remain the same:
```python
db.create_secret("api-key", "credential", namespace_path=["team"])
db.alter_secret("api-key", "rotated", namespace_path=["team"])
info = db.describe_secret("api-key", namespace_path=["team"])
```
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
Fixes#4750
Remote missing-table errors retained HTTP metadata in Rust but the Python binding converted them to a plain `ValueError`, discarding the request ID and status. This raises a `TableNotFoundError` that preserves the original message, request ID, status code, and chained HTTP cause while remaining compatible with existing `ValueError` and `lance_namespace.errors.TableNotFoundError` handlers. Local and non-HTTP errors retain their previous behavior.
Validation covers sync and async opens across cached/uncached and root/nested paths, exception pickling, local compatibility, the remote and error test suites, Clippy, formatting, Ruff, and all applicable PR checks including CodeQL.
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model
names outside the three built-in OpenAI names, so schemas can be built for
compatible models.
Preserve existing model defaults and request behavior. Add regression coverage
for the resulting Arrow schema dimensions and for all known OpenAI model
dimensions.
Adds `HUGGINGFACE` and an optional token Secret reference to the
existing storage integration methods. The same SQL transport handles
create, describe, list and drop. Only a Secret reference is stored in
integration metadata; mixed-provider credential options are rejected.
```python
from lancedb.storage_integration import SecretReference
info = db.create_storage_integration(
"private_hub", "hf://datasets/acme/private/raw/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference(
"hf_token", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```
Omit the token Secret for anonymous access at Hugging Face's default
anonymous
rate limits. A tokenless integration does not use a cluster's Hugging
Face token:
```python
info = db.create_storage_integration(
"public_hub", "hf://datasets/acme/public/raw/",
storage_provider="HUGGINGFACE",
)
```
```sql
CREATE STORAGE INTEGRATION public_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```
```sql
CREATE STORAGE INTEGRATION private_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
HUGGINGFACE_TOKEN_SECRET = shared.credentials.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing SQL transport]
SQL --> Metadata[Integration and token Secret reference]
```
Adds Rust request construction, SQL rendering, typed metadata decoding,
sync/async Flight SQL regressions and public documentation. Local
builds/tests are excluded by request; CI provides execution validation.
Adds GCP inputs to the existing database-owned storage integration
methods. GCP takes one Secret reference; the Secret stores the complete
application credentials JSON as an opaque string. The same methods and
SQL transport handle create, describe, list and drop.
```python
info = db.create_storage_integration(
"gcs", "gs://customer-data/raw/", storage_provider="GCP",
gcp_credentials_json_secret=SecretReference(
"gcp_json", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("gcs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("gcs")
```
```sql
CREATE STORAGE INTEGRATION gcs
STORAGE_PROVIDER = 'GCP'
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = shared.credentials.gcp_json;
DESCRIBE STORAGE INTEGRATION gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION gcs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing Flight SQL transport]
SQL --> Config[Configuration and one Secret reference]
```
Adds Rust request construction, SQL rendering and typed metadata
decoding, plus sync/async Flight SQL regressions across the three Arrow
string encodings. Rejects missing or mixed-cloud credential options.
Local builds/tests are excluded by request; validation runs in CI.
Register Azure Blob integrations through Flight SQL using an explicit
storage account and three Secret references for tenant ID, client ID and
client secret. Extends the database-owned integration APIs with
provider-specific inputs while keeping the same create, describe, list
and drop operations.
```python
info = db.create_storage_integration(
"customer_blobs", "az://customer-data/raw/",
storage_provider="AZURE", azure_storage_account="customeraccount",
azure_tenant_id_secret="tenant", azure_client_id_secret="client",
azure_client_secret_secret="credential",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("customer_blobs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("customer_blobs")
```
Sync and async connections accept the same options. Use
`SecretReference` for cross-database or namespace qualification. Values
remain in Secrets; returned metadata includes only references and setup
guidance.
```sql
CREATE STORAGE INTEGRATION customer_blobs
STORAGE_PROVIDER = 'AZURE'
STORAGE_PREFIX = 'az://customer-data/raw/'
STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
STORAGE_AZURE_TENANT_ID_SECRET = public.tenant
STORAGE_AZURE_CLIENT_ID_SECRET = public.client
STORAGE_AZURE_CLIENT_SECRET_SECRET = public.credential;
DESCRIBE STORAGE INTEGRATION customer_blobs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_blobs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Flight SQL]
SQL --> Metadata[Configuration and Secret references]
```
Azure supports service principal client credentials only. Adds SQL
rendering, provider validation and Python Flight SQL regressions across
Utf8, LargeUtf8 and Utf8View results. Local builds and tests are
excluded as requested; validation runs in CI.
Add `on_transform_error="fill"` to `StreamingDataset`, replacing failed
transform rows at their original positions with a deep-copied fill value or
the result of `repair(exc, row_batch)`.
Fill mode preserves split length, elastic step layout, worker accounting, and
checkpoint progress. Repair failures remain fatal, and `rows_filled` is shared
across workers.
Validated with focused fill and adjacent dataloader tests (10 and 6 passed)
and the complete applicable CI run for
6b51b33e56: four workflows and fifteen checks
all succeeded.
Closes#3916
Replaces #4322, keeping only its property graph defined over tables.
A property graph maps node labels and relationship types to existing tables of its namespace, in the shape of SQL/PGQ's `CREATE PROPERTY GRAPH`. It stores only that definition: a query reads the tables as nodes and edges through SQL `GRAPH_TABLE`, as they are when it runs.
- **Rust:** `Connection::create_property_graph` takes a `PropertyGraphDefinition` of `NodeTable`s and `EdgeTable`s. `describe_property_graph`, `drop_property_graph` and `list_property_graphs` manage it.
- **Python:** `lancedb.graph` has `NodeTable`, `EdgeTable` and `Endpoint`. Sync and async connections gain `create_property_graph`, `open_property_graph`, `drop_property_graph` and `list_property_graphs`, and a `PropertyGraph` handle describes its graph.
- **Support:** remote connections only. Local connections raise `NotImplementedError` in Python and return `NotSupported` in Rust.
```python
import lancedb
from lancedb.graph import EdgeTable, Endpoint, NodeTable
db = lancedb.connect("db://demo", api_key="...")
db.create_property_graph(
"social",
nodes=[NodeTable("person", key="person_id", label="Person")],
edges=[
EdgeTable(
"knows",
label="KNOWS",
source=Endpoint("src_id", references=("person", "person_id")),
destination=Endpoint("dst_id", references=("person", "person_id")),
)
],
)
db.execute_query("""
SELECT * FROM GRAPH_TABLE (social
MATCH (p:Person)-[:KNOWS]->(f:Person) WHERE p.age > 30
COLUMNS (p.name AS person, f.name AS friend))
""").read_all()
```
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
Package the standalone lancedb CLI with every stable and beta GitHub
release. Publish archives for Linux x86_64, Linux ARM64, and Apple
Silicon macOS, along with SHA-256 checksums, licenses, and third-party
notices.
Synchronize the CLI version with the SDKs and validate that release tags
match both manifests. Add PR and manual packaging dry runs, runtime
compatibility checks, and smoke tests of extracted executables. GitHub
release creation now waits for all three archives to pass validation;
SDK registry publishing remains independent.
Update installation instructions and the release checklist. Linux
binaries target glibc 2.28, retaining existing x86_64 CPU requirements;
macOS binaries target macOS 14 or newer.
Stacked on #4675.
Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.
`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
Stacked on #4674.
Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens.
The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates.
Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`.
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.
### Python experience
```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```
For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.
### SQL experience
```sql
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```
CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.
Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
Jack YeandClaude Sonnet 5
Lance Release
Yang Cen
Bruno RamirezandClaude Sonnet 5
Xuanwo
Gaétan
Fede Kamelhar
LeoandLeo
Srijan Keshri
Wyatt Alt
lancedb-gatefixer[bot]
TaylorandBrendan Clement
LanceDB RobotandJack Ye
BHAWESHBHASKAR
alexcastgonz
importcpp
ryan-haynesandRyan Haynes
Colin Patrick McCabe