docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)

## Summary

Adopts the product terminology guide across the docs site pages, SDK
docstrings and error messages. "LanceDB OSS" and local mode become
**LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB
Distributed**, which LanceDB Enterprise provides. Where free vs paid is
the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB
Enterprise". This is a first trial of the Embedded name, so reviewers
can judge it in context.

- **Embedded vs Distributed** for where execution runs (`optimize()`,
consistency, branching, maintenance). Main pages define it once:
"LanceDB Enterprise runs LanceDB Distributed: …"
- **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://`
region, the brute-force KNN limit) and marked legacy. "Cloud and
Enterprise" becomes Enterprise or Distributed.
- **Plan executor** becomes "executor" or the fleet name.
- **Error messages** change, e.g. `optimize is not supported for remote
tables.` → `... in LanceDB Distributed.` Tests that match on them are
updated.
- **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`,
`DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the
terms in prose.

Unchanged: identifiers (`availability.oss`, the `remote` module and
feature, `RemoteTable`, `db://`), heading anchors, internal code
comments, and Geneva.

## Evidence

- Rust lib tests for the changed messages and the Python tests in
`test_remote_db.py`, `test_materialized_views.py` and `test_table.py`
pass locally.
- Not run locally:
- The Node tests: `remote.test.ts` and `table.test.ts` have updated
assertions.
  - Vale.
  - The `experimental_functions`-marked materialized-view tests.
- `docs/src` API reference was hand-edited to match the docstrings
rather than regenerated.

## Merge Risk

**Door:** two-way; prose and message text only.
**Blast Radius:** `docs`, `error-messages`: callers matching on old
error strings would break.

Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon
(Enterprise pages).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Will Jones
2026-10-28 14:04:41 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent b4cbaf2f67
commit 4d7a64712f
148 changed files with 981 additions and 755 deletions
+1 -1
View File
@@ -8,7 +8,7 @@
{
"name": "lancedb",
"source": "./plugins/lancedb",
"description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables.",
"description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work with both LanceDB Embedded and LanceDB Enterprise.",
"version": "0.1.0",
"author": {
"name": "LanceDB"
+10 -2
View File
@@ -1,8 +1,16 @@
# LanceDB prose rules
These five rule files are copied unchanged from the `LanceDB.zip` asset of
`lancedb/docs` release `vale-lancedb-v0.1.0`. The archive SHA-256 is
`Dashes`, `PlainLanguage`, `ProductNames`, `VagueHeadings` and `VaguePhrases`
come from the `LanceDB.zip` asset of `lancedb/docs` release
`vale-lancedb-v0.1.0`, whose archive SHA-256 is
`0730a43e919774121068811d5687c61dfc7a0421cff78e1393f5e565b622cb0f`.
`ProductNames` no longer swaps in "LanceDB OSS".
The product-terminology rules were added here: `EmbeddedName`,
`RemoteExecution`, `OSSProduct`, `DistributedEnterprise`, `InternalNames`
(warnings) and `LegacyCloud` (suggestion). They enforce the names in the
docs repository's `STYLE.md#names`. Run every rule's tests from
`.github/vale` with `vale test --coverage styles`.
Keeping the rules here lets contributors and CI run Vale without access to the
source repository. The root `.vale.ini` preserves the release's MDX settings
@@ -0,0 +1,19 @@
extends: existence
message: "'%s' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
tokens:
- 'Distributed(?:\s+(?:and|or)\s+|\s*/\s*)(?:LanceDB\s+)?Enterprise'
- 'Enterprise(?:\s+(?:and|or)\s+|\s*/\s*)(?:LanceDB\s+)?Distributed'
tests:
- name: flags Distributed and Enterprise side by side
input: Choose Distributed or Enterprise. LanceDB Enterprise and LanceDB Distributed differ, and so does Distributed/Enterprise.
want: |
1:8:LanceDB.DistributedEnterprise:'Distributed or Enterprise' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
1:43:LanceDB.DistributedEnterprise:'Enterprise and LanceDB Distributed' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
1:98:LanceDB.DistributedEnterprise:'Distributed/Enterprise' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
- name: leaves the defining sentence and the Embedded pair alone
input: |
LanceDB Enterprise runs LanceDB Distributed. Compare Embedded and Distributed execution. Distributed search and enterprise teams are fine.
want: ''
@@ -0,0 +1,22 @@
extends: substitution
message: "Use '%s' instead of '%s'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# The in-process execution model is LanceDB Embedded, whether the data is on
# local disk or in object storage. Identifiers such as the `oss` product key
# are code, and code is never prose.
swap:
'LanceDB\s+(?:OSS|Oss|oss|Local|local)': LanceDB Embedded
'[Ll]ocal\s+mode': LanceDB Embedded
tests:
- name: replaces the old names of the execution model
input: Start with LanceDB OSS, also called LanceDB Local, and move off local mode later.
want: |
1:12:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'LanceDB OSS'.
1:37:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'LanceDB Local'.
1:65:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'local mode'.
- name: leaves the current name, other uses of local, and code alone
input: |
LanceDB Embedded reads local disk or object storage. Run LanceDB locally, set `availability.oss`, or call `LanceDB OSS` in a literal.
want: ''
@@ -0,0 +1,25 @@
extends: substitution
message: "Use '%s' instead of the internal name '%s'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# Internal component names stay out of public text. Environment variables such
# as `PHALANX_*` and Helm keys such as `planExecutorNodes` are code. Geneva is
# a product name and is not listed.
swap:
Phalanx: query node
'[Pp]lan\s+[Ee]xecutor': executor
'[Pp]lan\s+[Ee]xecutors': executors
Sophon: LanceDB Enterprise
tests:
- name: replaces internal component names
input: Phalanx sends work to a Plan Executor. Plan executors cache data in Sophon.
want: |
1:1:LanceDB.InternalNames:Use 'query node' instead of the internal name 'Phalanx'.
1:25:LanceDB.InternalNames:Use 'executor' instead of the internal name 'Plan Executor'.
1:40:LanceDB.InternalNames:Use 'executors' instead of the internal name 'Plan executors'.
1:69:LanceDB.InternalNames:Use 'LanceDB Enterprise' instead of the internal name 'Sophon'.
- name: leaves Geneva, configuration keys and code alone
input: |
Geneva runs on the executor + cache fleet. Set `PHALANX_REMOTE_FTS_SCORER` and `planExecutorNodes`.
want: ''
@@ -0,0 +1,21 @@
extends: existence
message: "'%s' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: suggestion
ignorecase: false
# A suggestion, because `db://` URIs, `api_key` and `region` for the hosted
# service are still documented as LanceDB Cloud.
tokens:
- 'LanceDB\s+Cloud'
- 'Cloud\s*/\s*Enterprise'
- 'Enterprise\s*/\s*Cloud'
tests:
- name: suggests checking each LanceDB Cloud mention
input: Connect to LanceDB Cloud with an API key. Cloud/Enterprise users get the cache.
want: |
1:12:LanceDB.LegacyCloud:'LanceDB Cloud' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'.
1:43:LanceDB.LegacyCloud:'Cloud/Enterprise' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'.
- name: leaves other clouds and code alone
input: |
Deploy into your own cloud account on Google Cloud. Pass `LanceDB Cloud` as a literal.
want: ''
@@ -0,0 +1,28 @@
extends: existence
message: "Avoid '%s'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# Bare OSS used as a product, as in "OSS and Enterprise" or "in OSS". "LanceDB
# OSS" is reported by LanceDB.EmbeddedName.
tokens:
- 'OSS(?:\s+(?:and|or|vs\.?|versus)\s+|\s*/\s*)(?:LanceDB\s+)?Enterprise'
- 'Enterprise(?:\s+(?:and|or|vs\.?|versus)\s+|\s*/\s*)OSS'
- '(?:in|on|from|using|than)\s+OSS'
tests:
- name: flags OSS presented as a product
input: Compare OSS vs Enterprise. It works in OSS and on OSS, unlike OSS/Enterprise.
want: |
1:9:LanceDB.OSSProduct:Avoid 'OSS vs Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:37:LanceDB.OSSProduct:Avoid 'in OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:48:LanceDB.OSSProduct:Avoid 'on OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:63:LanceDB.OSSProduct:Avoid 'OSS/Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
- name: flags the pair in either order
input: Both OSS and LanceDB Enterprise ship it, as do Enterprise or OSS.
want: |
1:6:LanceDB.OSSProduct:Avoid 'OSS and LanceDB Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:48:LanceDB.OSSProduct:Avoid 'Enterprise or OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
- name: leaves open source, identifiers and code alone
input: |
LanceDB Embedded (open source) and LanceDB Enterprise share the format. Set `oss: available` or read `in OSS` literally.
want: ''
@@ -9,7 +9,6 @@ swap:
'(?:Lancedb|LanceDb|lanceDB|LANCEDB|Lance DB)': LanceDB
'LanceDB (?:cloud|CLOUD)': LanceDB Cloud
'LanceDB (?:enterprise|ENTERPRISE)': LanceDB Enterprise
'LanceDB (?:oss|Oss)': LanceDB OSS
'(?:HuggingFace|Huggingface|Hugging face)': Hugging Face
Langchain: LangChain
Typescript: TypeScript
@@ -0,0 +1,33 @@
extends: existence
message: "Avoid '%s'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# Remote storage, remote URLs and a git remote are other things and stay. Class
# names such as `RemoteTable` are code, and code is never prose.
tokens:
- '[Rr]emote\s+(?:mode|tables?|connections?|databases?)'
tests:
- name: flags remote as the execution model
input: In remote mode, a remote table is opened through a remote connection.
want: |
1:4:LanceDB.RemoteExecution:Avoid 'remote mode'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
1:19:LanceDB.RemoteExecution:Avoid 'remote table'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
1:52:LanceDB.RemoteExecution:Avoid 'remote connection'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
- name: flags plurals and a heading
input: |
## Remote tables
Remote databases hold remote tables.
want: |
1:4:LanceDB.RemoteExecution:Avoid 'Remote tables'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
3:1:LanceDB.RemoteExecution:Avoid 'Remote databases'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
3:23:LanceDB.RemoteExecution:Avoid 'remote tables'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
- name: leaves other remotes and code alone
input: |
Push to the git remote, read remote storage and fetch a remote URL. `open_table` returns a `RemoteTable`; see `remote table` in the API.
```python
remote_table = db.open_table("remote table")
```
want: ''
+8 -7
View File
@@ -72,7 +72,7 @@ are wrappers around the Rust library.
* `node`: **Deprecated** Typescript package source code
* `java`: Java package source code
* `docs/src`: SDK reference, built with mkdocs
* `docs/web`: website guides for LanceDB OSS and the APIs used by both OSS and Enterprise
* `docs/web`: website guides for LanceDB Embedded and the APIs used by both Embedded and Distributed
* `docs/web-tests`: the tests the documentation's code examples are extracted from
## Documentation
@@ -83,14 +83,14 @@ it is ready for readers.
### Choose the right place
- `docs/web/` holds website guides for LanceDB OSS and the APIs used by both
OSS and Enterprise.
- `docs/web/` holds website guides for LanceDB Embedded and the APIs used by both
Embedded and Distributed.
- `docs/src/` holds the SDK reference. See [docs/README.md](docs/README.md) for
its build and API-generation instructions.
- `docs/web-tests/` and `rust/lancedb/tests/` hold tests used to generate the
website's code examples.
Document SDK APIs here, and explain which operations work in OSS and Enterprise.
Document SDK APIs here, and explain which operations work in Embedded and Distributed.
Put Enterprise-specific deployment and operating instructions in the Enterprise
documentation.
@@ -150,7 +150,8 @@ example.
### Say where a feature works
Use `availability` in a page's frontmatter to describe the support covered by
that page. Each product is either `available` or `unavailable`; use the summary
that page. Each product (`oss` for LanceDB Embedded, `enterprise` for LanceDB
Enterprise) is either `available` or `unavailable`; use the summary
and body for setup requirements, SDK differences and unsupported operations.
For example, a page about signing in to Enterprise could declare:
@@ -159,8 +160,8 @@ availability:
oss: unavailable
enterprise: available
summary: >-
These sign-in instructions apply to LanceDB Enterprise. Embedded
LanceDB has no server to sign in to.
These sign-in instructions apply to LanceDB Enterprise. LanceDB
Embedded has no server to sign in to.
```
Check these claims with the feature owner. Don't use `varies`, treat an API
+2 -2
View File
@@ -45,8 +45,8 @@ LanceDB is a central location where developers can build, train and analyze thei
- **Advanced Features**: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
### **Products**:
- **Open Source & Local**: 100% open source, runs locally or in your cloud. No vendor lock-in.
- **Cloud and Enterprise**: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
- **LanceDB Embedded (open source)**: 100% open source, runs in your application's process; your data can live on local disk or in object storage. No vendor lock-in.
- **LanceDB Enterprise**: Production-scale distributed search, indexing, and maintenance. Plus, complete data sovereignty and security.
### **Ecosystem**:
- **Columnar Storage**: Built on the Lance columnar format for efficient storage and analytics.
+3 -3
View File
@@ -1,9 +1,9 @@
openapi: 3.1.0
info:
version: 1.0.0
title: LanceDB Cloud API
title: LanceDB REST API
description: |
LanceDB Cloud API is a RESTful API that allows users to access and modify data stored in LanceDB Cloud.
The LanceDB REST API is a RESTful API that allows users to access and modify data stored in LanceDB Enterprise or LanceDB Cloud (legacy).
Table actions are considered temporary resource creations and all use POST method.
contact:
name: LanceDB support
@@ -12,7 +12,7 @@ info:
servers:
- url: https://{db}.{region}.api.lancedb.com
description: LanceDB Cloud REST endpoint.
description: LanceDB Cloud (legacy) REST endpoint. For LanceDB Enterprise, use your deployment's endpoint.
variables:
db:
default: ""
+6 -6
View File
@@ -1,10 +1,10 @@
# Java SDK
The LanceDB Java SDK provides a convenient way to interact with LanceDB Cloud and Enterprise deployments using the Lance REST Namespace API.
The LanceDB Java SDK provides a convenient way to interact with LanceDB Enterprise deployments using the Lance REST Namespace API.
!!! note
The Java SDK currently only works for LanceDB remote database that connects to LanceDB Cloud and Enterprise.
Local database support is a work in progress. Check [LANCEDB-2848](https://github.com/lancedb/lancedb/issues/2848) for the latest progress.
The Java SDK currently only works with LanceDB Distributed, connecting to LanceDB Enterprise or LanceDB Cloud (legacy).
LanceDB Embedded support is a work in progress. Check [LANCEDB-2848](https://github.com/lancedb/lancedb/issues/2848) for the latest progress.
## Installation
@@ -20,7 +20,7 @@ Add the following dependency to your `pom.xml`:
## Quick Start
### Connecting to LanceDB Cloud
### Connecting to LanceDB Cloud (legacy)
```java
import com.lancedb.LanceDbNamespaceClientBuilder;
@@ -51,8 +51,8 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|--------|-------------|----------|
| `apiKey(String)` | LanceDB API key | Yes |
| `database(String)` | Database name | Yes |
| `endpoint(String)` | Custom endpoint URL for Enterprise deployments | No |
| `region(String)` | AWS region (default: "us-east-1") | No |
| `endpoint(String)` | Custom endpoint URL for LanceDB Enterprise deployments | No |
| `region(String)` | LanceDB Cloud (legacy) region (default: "us-east-1") | No |
| `config(String, String)` | Additional configuration parameters | No |
### Opening a Table with Vended Credentials
+3 -3
View File
@@ -180,9 +180,9 @@ abstract createMaterializedView(
Define a materialized view named `name` over the table `source`.
Local creation and refresh require a source build with the
LanceDB Embedded creation and refresh require a source build with the
`experimental-functions` Cargo feature, disabled in published packages.
Remote Cloud/Enterprise operations do not require this feature.
LanceDB Distributed operations do not require this feature.
The view is populated before creation returns. Set `withNoData` to create
only its definition and empty backing table. The view is a normal table:
@@ -827,7 +827,7 @@ abstract renameTable(
Rename a table.
Currently only supported by LanceDB Cloud. Local OSS connections and
Currently only supported in LanceDB Distributed. LanceDB Embedded connections and
namespace-backed connections (via [connectNamespace](../functions/connectNamespace.md)) reject with
a "not supported" error.
+14 -13
View File
@@ -69,9 +69,9 @@ abstract addColumns(newColumnTransforms): Promise<AddColumnsResult>
Add new columns with defined values.
Local computed-column declaration and refresh require a source build with
the `experimental-functions` Cargo feature. Published packages disable it;
remote Cloud/Enterprise operations remain available.
LanceDB Embedded computed-column declaration and refresh require a source
build with the `experimental-functions` Cargo feature. Published packages
disable it; LanceDB Distributed operations remain available.
The `{ computed }` form stores the expression rather than evaluating it
now: the column is committed with no values, and rows get them from
@@ -83,7 +83,7 @@ computed, so a mutated input is reflected by the next refresh. While a
declaration reads a column, that column cannot be renamed, retyped or
dropped.
On LanceDB Cloud and Enterprise the expression is planned by the
In LanceDB Distributed the expression is planned by the
server, and the refresh runs as a server job -- see
[Table#refreshColumnAsync](Table.md#refreshcolumnasync).
@@ -679,7 +679,7 @@ The per-row errors Function refreshes recorded on this table.
A refresh running under a skip policy records each row it skipped with
the input that failed and the error. This lists those records, newest
job first, plus a summary for any fragment whose per-row detail was
capped. LanceDB Cloud and Enterprise only; reading errors needs read
capped. LanceDB Distributed only; reading errors needs read
access to the table, since a message carries the value that failed.
#### Parameters
@@ -932,7 +932,7 @@ Prewarm one or more columns of data in the table.
Prewarming is generally useful for columns used in filters or projections.
Large columns (e.g. high-dimensional vectors or binary data) may not be
practical to prewarm.
This feature is currently only supported on remote tables.
This feature is currently only supported in LanceDB Distributed.
#### Returns
@@ -1043,8 +1043,8 @@ Local refresh requires a source build with `experimental-functions`.
Rows appended since the last refresh are filled by the next one, and
rows whose inputs changed since they were computed are recomputed;
everything else is left as it is. Local tables only: a remote refresh
runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
everything else is left as it is. LanceDB Embedded only: in LanceDB
Distributed a refresh runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
#### Parameters
@@ -1068,14 +1068,15 @@ abstract refreshColumnAsync(column): Promise<Job>
Like [Table#refreshColumn](Table.md#refreshcolumn), but returns a handle to the refresh
job instead of blocking until it completes.
Local refresh requires a source build with `experimental-functions`.
Remote Cloud/Enterprise jobs do not require this feature.
LanceDB Embedded refresh requires a source build with
`experimental-functions`. LanceDB Distributed jobs do not require this
feature.
The job may already be complete when returned; callers must not assume
the column is filled until [Job.wait](Job.md#wait) resolves. Invalid input --
an unknown column, or one that is not computed -- rejects here rather
than failing the job. On local tables the job runs in-process; on
LanceDB Cloud and Enterprise it is the server's backfill job.
than failing the job. In LanceDB Embedded the job runs in-process; in
LanceDB Distributed it is the server's backfill job.
#### Parameters
@@ -1304,7 +1305,7 @@ Tokenize a full-text search query using the tokenizer configured on an FTS index
Specify exactly one of `column` or `indexName`.
Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
the client process from index metadata. For remote tables, this means the
the client process from index metadata. In LanceDB Distributed, this means the
same tokenizer model files must also exist locally (under
`LANCE_LANGUAGE_MODEL_HOME`) and must match the files the server uses. A
mismatch cannot be detected: the returned tokens may differ from what the
+5 -5
View File
@@ -20,15 +20,15 @@ Connect to a LanceDB instance at the given URI.
Accepted formats:
- `/path/to/database` - local database
- `/path/to/database` - database on the local file system
- `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
- `db://host:port` - remote database (LanceDB cloud)
- `db://host:port` - LanceDB Enterprise or LanceDB Cloud (legacy) database
### Parameters
* **uri**: `string`
The uri of the database. If the database uri starts
with `db://` then it connects to a remote database.
with `db://` then it connects to LanceDB Enterprise or LanceDB Cloud (legacy).
* **options?**: `Partial`&lt;[`ConnectionOptions`](../interfaces/ConnectionOptions.md)&gt;
The options to use when connecting to the database
@@ -80,9 +80,9 @@ Connect to a LanceDB instance at the given URI.
Accepted formats:
- `/path/to/database` - local database
- `/path/to/database` - database on the local file system
- `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
- `db://host:port` - remote database (LanceDB cloud)
- `db://host:port` - LanceDB Enterprise or LanceDB Cloud (legacy) database
### Parameters
+1 -1
View File
@@ -18,7 +18,7 @@ function connectNamespace(
Connect to a LanceDB database through a namespace.
Unlike [connect](connect.md), which routes by URI scheme (local path vs.
`db://` cloud), `connectNamespace` always returns a namespace-backed
`db://`), `connectNamespace` always returns a namespace-backed
connection. The `implName` selects the namespace implementation:
- `"dir"` — directory namespace, configured with [DirNamespaceConfig](../interfaces/DirNamespaceConfig.md).
+2 -2
View File
@@ -18,8 +18,8 @@ optional allowExternalBlobOutsideBases: boolean;
Whether blob URIs outside registered external bases may be written.
Defaults to `false`. This option is supported only for local/native
tables; remote tables return an error when it is enabled. An enabled write
Defaults to `false`. This option is supported only in LanceDB Embedded;
LanceDB Distributed returns an error when it is enabled. An enabled write
stores the absolute URI reference without registering a base or copying
the external object into the database. The object must remain accessible
when the blob is read later.
+1 -1
View File
@@ -80,7 +80,7 @@ optional userId: string;
User identifier for tracking purposes.
This is sent as the `x-lancedb-user-id` header in requests to LanceDB Cloud/Enterprise.
This is sent as the `x-lancedb-user-id` header in requests to LanceDB Enterprise.
It can be set directly, or via the `LANCEDB_USER_ID` environment variable.
Alternatively, set `LANCEDB_USER_ID_ENV_KEY` to specify another environment
variable that contains the user ID value.
+10 -10
View File
@@ -14,7 +14,7 @@
optional apiKey: string;
```
(For LanceDB cloud only): the API key to use with LanceDB Cloud.
(For LanceDB Enterprise or LanceDB Cloud only): the API key to use.
Can also be set via the environment variable `LANCEDB_API_KEY`.
@@ -26,7 +26,7 @@ Can also be set via the environment variable `LANCEDB_API_KEY`.
optional clientConfig: ClientConfig;
```
(For LanceDB cloud only): configuration for the remote HTTP client.
(For LanceDB Enterprise or LanceDB Cloud only): configuration for the HTTP client.
***
@@ -36,8 +36,8 @@ optional clientConfig: ClientConfig;
optional hostOverride: string;
```
(For LanceDB cloud only): the host to use for LanceDB cloud. Used
for testing purposes.
(For LanceDB Enterprise only): the host of the LanceDB Enterprise
deployment to connect to.
***
@@ -47,7 +47,7 @@ for testing purposes.
optional manifestEnabled: boolean;
```
(For LanceDB OSS only): use directory namespace manifests as the source
(For LanceDB Embedded only): use directory namespace manifests as the source
of truth for table metadata. Existing directory-listed root tables are
migrated into the manifest on access.
@@ -59,7 +59,7 @@ migrated into the manifest on access.
optional namespaceClientProperties: Record<string, string>;
```
(For LanceDB OSS only): extra properties for the backing namespace
(For LanceDB Embedded only): extra properties for the backing namespace
client used by manifest-enabled native connections.
***
@@ -70,7 +70,7 @@ client used by manifest-enabled native connections.
optional oauthConfig: NativeOAuthConfig;
```
(For LanceDB cloud only): OAuth configuration for IdP-based
(For LanceDB Enterprise only): OAuth configuration for IdP-based
authentication (e.g., Azure Entra ID). When set, token acquisition
and refresh are handled entirely in Rust. TypeScript users should pass
the public `OAuthConfig` type exported from `@lancedb/lancedb`.
@@ -105,7 +105,7 @@ storage, raising per-read latency and cost.
optional region: string;
```
(For LanceDB cloud only): the region to use for LanceDB cloud.
(For LanceDB Cloud only): the region to use for LanceDB Cloud (legacy).
Defaults to 'us-east-1'.
***
@@ -116,7 +116,7 @@ Defaults to 'us-east-1'.
optional session: Session;
```
(For LanceDB OSS only): the session to use for this connection. Holds
(For LanceDB Embedded only): the session to use for this connection. Holds
shared caches and other session-specific state.
***
@@ -127,6 +127,6 @@ shared caches and other session-specific state.
optional storageOptions: Record<string, string>;
```
(For LanceDB OSS only): configuration for object storage.
(For LanceDB Embedded only): configuration for object storage.
The available options are described at https://docs.lancedb.com/storage/
+10 -10
View File
@@ -31,7 +31,7 @@ optional createdAt: Date;
When the index was created.
`undefined` for remote tables or indices created before timestamps were tracked.
`undefined` in LanceDB Distributed or indices created before timestamps were tracked.
***
@@ -43,8 +43,8 @@ optional indexDetails: any;
Index-type-specific details parsed as a JavaScript object.
Falls back to a raw string if JSON parsing fails. `undefined` for
remote tables or when details are unavailable.
Falls back to a raw string if JSON parsing fails. `undefined` in
LanceDB Distributed or when details are unavailable.
***
@@ -66,7 +66,7 @@ optional indexUuid: string;
The UUID of the first segment of the index.
`undefined` for remote tables, which do not yet surface this.
`undefined` in LanceDB Distributed, which does not yet surface this.
***
@@ -78,7 +78,7 @@ optional indexVersion: number;
The on-disk index format version.
`undefined` for remote tables.
`undefined` in LanceDB Distributed.
***
@@ -100,7 +100,7 @@ optional numIndexedRows: number;
The number of rows indexed, across all segments.
May be `undefined` for remote tables if the server does not report it.
May be `undefined` in LanceDB Distributed if the server does not report it.
***
@@ -112,7 +112,7 @@ optional numSegments: number;
The number of segments that make up the index.
`undefined` for remote tables.
`undefined` in LanceDB Distributed.
***
@@ -124,7 +124,7 @@ optional numUnindexedRows: number;
The number of rows not yet covered by this index.
May be `undefined` for remote tables if the server does not report it.
May be `undefined` in LanceDB Distributed if the server does not report it.
***
@@ -136,7 +136,7 @@ optional sizeBytes: number;
The total size in bytes of all index files across all segments.
`undefined` for remote tables or indices without size tracking.
`undefined` in LanceDB Distributed or indices without size tracking.
***
@@ -148,4 +148,4 @@ optional typeUrl: string;
The protobuf type URL, a precise type identifier for the index.
`undefined` for remote tables.
`undefined` in LanceDB Distributed.
@@ -14,7 +14,7 @@
optional mode: string;
```
How the view was brought up to date: "rebuild", or absent for remote SQL refreshes.
How the view was brought up to date: "rebuild", or absent for LanceDB Distributed SQL refreshes.
***
@@ -24,7 +24,7 @@ How the view was brought up to date: "rebuild", or absent for remote SQL refresh
optional rowsWritten: number;
```
Rows written by this refresh, or absent for remote SQL refreshes.
Rows written by this refresh, or absent for LanceDB Distributed SQL refreshes.
***
@@ -34,7 +34,7 @@ Rows written by this refresh, or absent for remote SQL refreshes.
optional sourceVersion: number;
```
Source version refreshed to, or absent for remote SQL refreshes.
Source version refreshed to, or absent for LanceDB Distributed SQL refreshes.
***
+1 -1
View File
@@ -21,4 +21,4 @@ optional signal: AbortSignal;
```
Cancels the read. The call rejects with `signal.reason`, and the native
read stops, including in-flight requests to a remote table.
read stops, including in-flight requests to a LanceDB Distributed table.
+17 -17
View File
@@ -27,8 +27,8 @@ connect, per-table storage and write options on create, and `index_cache_size`,
Omit these options to use server-managed storage and caching. Connection-level
`storage_options` supported by an Enterprise deployment remain available.
The synchronous remote `create_table` uses the same positional order as the
local API: `name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
The synchronous LanceDB Distributed `create_table` uses the same positional order as the
LanceDB Embedded API: `name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
embedding_functions`. Prefer keyword arguments for optional parameters.
## Connections (Synchronous)
@@ -227,14 +227,14 @@ async for entry in catalog.authz.list_acls(subject=Subject.role("reader")):
## Remote SQL
Submit SQL against a remote LanceDB database through the connection.
Submit SQL against a LanceDB Distributed database through the connection.
The connected database and `default_namespace_path=["public"]` are used for
unqualified tables. Fully qualified references can still query other databases
and namespaces available to the same deployment. `execute_query` returns a
reader as soon as its initial result stream is available. `execute_query_async`
returns a query handle once submission is accepted; use it to inspect progress, open a reader,
or cancel the query. The SQL client is initialized by the first query and
retained for the lifetime of the remote connection. Query ids are random,
retained for the lifetime of the connection. Query ids are random,
connection-scoped references rather than encoded SQL or durable resume tokens.
Both APIs start with one Flight call. The server can stream results on that
@@ -308,24 +308,24 @@ listing a storage directory.
## Tables (Synchronous)
Local and remote tables use the same arguments for index creation, deletion,
LanceDB Embedded and LanceDB Distributed tables use the same arguments for index creation, deletion,
index statistics, and search. For example, `table.delete(where="id = 1")`,
`table.create_index("cosine", 2)` (the deprecated vector API), and
`table.search([1.0, 1.0], fast_search=True)` work with either backend.
`fast_search=True` excludes rows that have not been indexed.
`create_scalar_index` defaults to `replace=True` on both backends. Both scalar
and full-text index creation accept `wait_timeout`. Remote maintenance methods
accept the local options: cleanup and compaction warn that they are no-ops,
and full-text index creation accept `wait_timeout`. LanceDB Distributed maintenance methods
accept the LanceDB Embedded options: cleanup and compaction warn that they are no-ops,
while `optimize` reports an unsupported operation. Removed Tantivy options
produce the same explanatory errors on both backends.
The old remote keyword names `predicate` for `delete`, `index_name` for
The old LanceDB Distributed keyword names `predicate` for `delete`, `index_name` for
`drop_index`, `index_uuid` for `index_stats`, and `column` for `create_fts_index`
remain available with deprecation warnings. Use `where`, `name`, `index_name`,
and `field_names`, respectively. For positional calls, use the argument order
documented by `Table` below; specify index options by keyword when migrating
code that used the former remote-only argument order.
code that used the former LanceDB Distributed-only argument order.
::: lancedb.table.Table
@@ -429,14 +429,14 @@ scoped to each invocation and support `read`, `readinto`, `seek`, `tell`,
## Materialized Views (Synchronous)
In the 0.41 SDK release, local materialized-view creation and refresh require a
In the 0.41 SDK release, LanceDB Embedded materialized-view creation and refresh require a
source build with the `experimental-functions` Cargo feature. Published Python
packages leave it disabled. Opening, reading, listing and dropping existing views
remain available, as do remote Cloud and Enterprise operations. The same feature
requirement applies to local SQL computed-column declaration and refresh; ordinary
remain available, as do LanceDB Distributed operations. The same feature
requirement applies to LanceDB Embedded SQL computed-column declaration and refresh; ordinary
eager `add_columns` transforms remain available.
On Cloud and Enterprise, creation accepts `auto_refresh_enabled` and
In LanceDB Distributed, creation accepts `auto_refresh_enabled` and
`auto_refresh_interval_seconds`:
```python
@@ -581,13 +581,13 @@ pylance, LanceDB uses a matching `lance.blob.v2` extension type so blob columns
still work. Native table queries return descriptors. Call
[`fetch_blob_files`][lancedb.table.Table.fetch_blob_files] for lazy reads or
[`fetch_blobs`][lancedb.table.Table.fetch_blobs] for eager bytes.
Local tables support these fetch APIs for both legacy blob v1 columns
LanceDB Embedded supports these fetch APIs for both legacy blob v1 columns
(`lance-encoding:blob`) and blob v2 columns. For legacy columns, obtain row IDs
with `search().with_row_id(True).to_arrow()` and pass the result to the fetch API.
One missing or deleted row ID rejects the entire blob-read batch; the APIs do
not return partial results.
Remote SQL returns blob bytes by default. Select `blob_descriptor(video)` to
LanceDB Distributed SQL returns blob bytes by default. Select `blob_descriptor(video)` to
return an openable reference instead, then use
[`open_blob_file`][lancedb.table.Table.open_blob_file] or the lazy iterable
[`open_blob_files`][lancedb.table.Table.open_blob_files]:
@@ -604,7 +604,7 @@ for row in rows.to_pylist():
frame = extract_frame(video, row["timestamp"])
```
This example uses a remote SQL connection that produces source-bearing
This example uses a LanceDB Distributed SQL connection that produces source-bearing
descriptors. The descriptor contains `kind`, `position`, `size`, `blob_id`,
`blob_uri`, and a `source` struct with `uri`, `version`, `committed_at_micros`,
`branch`, `column`, `row_reference`, and `row_is_address`. The opener verifies
@@ -636,7 +636,7 @@ async for video in async_videos.open_blob_files(rows["video"]):
```
Payload bytes load on reads. Opening an external blob with unspecified size
can issue a metadata-only HEAD request. Remote reads use authenticated range
can issue a metadata-only HEAD request. LanceDB Distributed reads use authenticated range
requests that validate the captured URI and commit timestamp.
Blobs inside lists can be written and queried as descriptors. They are omitted
+3 -3
View File
@@ -64,13 +64,13 @@ The branch experiments leave you with the results side by side, and they
deliberately never touch `main`. Once you've picked a winner, apply it to
`main` yourself by rerunning that experiment's validated transformation
directly against the `main` table. Rerunning the reviewed operation is the
reliable path on both OSS and Enterprise: it replays exactly the transformation
reliable path on both LanceDB Embedded and LanceDB Enterprise: it replays exactly the transformation
you validated on the branch, and it works the same regardless of how the branch
evolved.
Suppose `nomic-embed-text` wins. Rerun the same Nomic embedding transformation
you validated on `embed-nomic` against `main` in batches, then call
`table.optimize()` on OSS. Verify a bounded vector search on `main`, and keep
`table.optimize()` on LanceDB Embedded. Verify a bounded vector search on `main`, and keep
the losing branch around until you're confident in the result.
See [Branches](/tables/branching) for more on how branches, versions, and tags
@@ -96,5 +96,5 @@ branch that's isolated from `main` until you decide you want the new derived
column in `main`.
Once you've chosen a winner, rerun that validated transformation to ingest it into `main`
yourself, on either LanceDB OSS or Enterprise. See our documentation on [branches](/tables/branching)
yourself, on either LanceDB Embedded or LanceDB Enterprise. See our documentation on [branches](/tables/branching)
for more information.
+1 -1
View File
@@ -9,7 +9,7 @@ description: "API reference for LanceDB"
is an OpenAPI protocol that enables reading, writing and managing Lance tables by connecting
those metadata services or building a custom metadata server in a standardized way.
LanceDB OSS allows you to interface with Lance tables via the REST Namespace.
LanceDB Embedded allows you to interface with Lance tables via the REST Namespace.
LanceDB Enterprise provides an extended REST API with
additional endpoints for managing tables and data.
If you have specific needs or questions about the Enterprise REST API Namespace,
+17 -13
View File
@@ -1,14 +1,17 @@
---
title: "Architecture"
sidebarTitle: "Architecture"
description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with embedded LanceDB OSS."
description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with LanceDB Embedded."
icon: cubes
keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"]
---
LanceDB OSS is an embedded library for working with Lance tables. LanceDB
Enterprise is a multimodal lakehouse for data curation, feature engineering,
search, and training data access. Its architecture separates query serving,
LanceDB Embedded is an open-source library for working with Lance tables. It runs
in your application's process; your data can live on local disk or in object
storage. LanceDB Enterprise is a multimodal lakehouse for data curation, feature
engineering, search, and training data access. LanceDB Enterprise runs LanceDB
Distributed: query execution, indexing and maintenance run on the Enterprise
service, outside your application. Its architecture separates query serving,
read execution, caching, and background work. Both use the open
[Lance](/basics/lance) format.
@@ -17,23 +20,23 @@ read the resulting tables into training frameworks. Enterprise separates the
resources that serve shared data from those that build indexes and run
background work, so teams can size them for these different workloads.
![LanceDB OSS runs the application and embedded library in one process, using local disk or object storage. Enterprise applications connect to query serving, with separate read execution and caching, indexing and background workers, and durable object storage. Both use the Lance format, but they do not share a storage location automatically.](/static/assets/images/overview/lancedb-architecture.svg)
![LanceDB Embedded runs the application and embedded library in one process, using local disk or object storage. Enterprise applications connect to query serving, with separate read execution and caching, indexing and background workers, and durable object storage. Both use the Lance format, but they do not share a storage location automatically.](/static/assets/images/overview/lancedb-architecture.svg)
The diagram shows the main compute and storage boundaries, not every request
path. The control plane governs the Enterprise service.
## A shared format, different storage choices {#storage}
Lance stores table data, versions, and indexes. OSS can read and write tables on
Lance stores table data, versions, and indexes. LanceDB Embedded can read and write tables on
local disk or in object storage. Enterprise keeps durable table data and index
artifacts in object storage, independently of the machines serving requests.
Its caches accelerate reads; they are not the durable source of truth.
Sharing a format does not mean that an Enterprise deployment automatically
opens an existing OSS database. Plan how to move or connect your data and
opens an existing LanceDB Embedded database. Plan how to move or connect your data and
configure access for your deployment.
## OSS: an embedded library inside your application {#embedded}
## LanceDB Embedded: a library inside your application {#embedded}
With an embedded connection, the LanceDB library plans and executes queries,
builds indexes, and updates tables inside your process. There is no separate
@@ -44,9 +47,10 @@ single-node data pipelines. Your application controls the compute resources and
when to run [table maintenance](/maintenance/compaction). You also manage access
to the local files or object storage that hold your tables.
## Enterprise: a multimodal lakehouse for shared workloads {#distributed}
## LanceDB Enterprise: distributed execution for shared workloads {#distributed}
Applications connect to remote tables through a LanceDB SDK. The service handles
A LanceDB SDK connects to LanceDB Enterprise, and the service runs operations
with LanceDB Distributed. The service handles
requests across a fleet, with components sized for different kinds of work:
- **Query serving** receives requests, resolves tables, and plans operations.
@@ -81,20 +85,20 @@ and request flows.
## Application changes for Enterprise {#differences}
Core operations remain familiar: create a table, add data, and run a search.
With OSS, the application executes those operations. With Enterprise, the SDK
With LanceDB Embedded, the application executes those operations. With Enterprise, the SDK
sends requests to the service, and the service manages execution and configured
background maintenance.
To use Enterprise, configure the cluster endpoint, database URI and credentials,
and make your data available in the Enterprise database. Review SDK methods
that behave differently on remote tables, including result retrieval and
that behave differently with LanceDB Distributed, including result retrieval and
background jobs. [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise)
describes these application changes. If you plan to use materialized views, check the
[stable row ID requirements](/basics/limits#schema) before creating source tables.
## Choose for the workload you need to run {#choosing}
Choose OSS when you want an embedded library and can manage its resources and
Choose LanceDB Embedded when you want an embedded library and can manage its resources and
maintenance in your application.
Choose Enterprise when multiple applications or teams need a shared multimodal
+8 -8
View File
@@ -158,13 +158,13 @@ LANCEDB_REGION=us-east-1
LANCEDB_HOST_OVERRIDE=https://your-enterprise-endpoint.com
```
If you're using LanceDB OSS, no connection settings are required, as it runs as an
If you're using LanceDB Embedded, no connection settings are required, as it runs as an
embedded retrieval library. A simple prompt like this should work:
```text Agent prompt
# If using OSS
# If using LanceDB Embedded
Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB
OSS table.
Embedded table.
# If using enterprise
Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB
@@ -208,7 +208,7 @@ to `batch_size` rows at a time, providing an iterable of batches for the ingesti
shown next.
#### Table maintenance {#table-maintenance}
For LanceDB OSS, the plugin instructs the agent to call
For LanceDB Embedded, the plugin instructs the agent to call
`table.optimize()` after the ingestion loop. This compacts small fragments, cleans up
old versions according to the retention policy, and incorporates new data into indexes.
@@ -225,7 +225,7 @@ This example dataset has only eight rows, so the default batch size writes it in
one call. Larger inputs should still avoid single-row write commits.
</Note>
### 3\. Run the OSS pipeline {#3-run-the-oss-pipeline}
### 3\. Run the Embedded pipeline {#3-run-the-oss-pipeline}
```bash
uv run python ingest_multimodal.py
@@ -258,9 +258,9 @@ The plugin shown in this tutorial should generalize reasonably well to other use
If you find any issues, open [an issue](https://github.com/lancedb/lancedb-agent-plugins/issues)
on GitHub, clearly describing the intended behavior.
You can choose OSS or Enterprise based on how the work will run:
You can choose LanceDB Embedded or LanceDB Enterprise based on how the work will run:
Start with [LanceDB OSS](/quickstart) during the early stages of a project
Start with [LanceDB Embedded](/quickstart) during the early stages of a project
when an agent is helping you prototype,
explore a dataset, or run small workflows on a subset of the data. The application owns
the storage and lifecycle work, so ask the agent to validate inputs, write in
@@ -271,7 +271,7 @@ Choose [LanceDB Enterprise](/enterprise) when the resulting table becomes
shared production infrastructure, and the workload needs distributed capacity,
private deployment, or platform-managed operations. The data format and the core
table API stay the same, so the pipeline's table and search code carries over:
the agent connects to a remote `db://` table, leaves maintenance to the
the agent connects to an Enterprise `db://` table, leaves maintenance to the
deployment instead of calling `optimize()`, and submits long-running work as
jobs. See [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
+10 -7
View File
@@ -1,7 +1,7 @@
---
title: "Connect to LanceDB"
sidebarTitle: "Connect"
description: "Use the embedded LanceDB OSS library with tables in a local directory or object storage, or connect to LanceDB Enterprise."
description: "Use LanceDB Embedded with tables in a local directory or object storage, or connect to LanceDB Enterprise."
icon: plug
keywords: ["connect", "uri", "embedded", "object storage", "s3", "enterprise", "db://"]
---
@@ -20,15 +20,18 @@ import {
TsConnectObjectStorage,
} from '/snippets/connection.mdx';
The URI you pass to `connect` decides where the work runs. The same Python,
TypeScript and Rust SDKs open all three kinds;
[OSS and Enterprise](/basics/offerings) describes how the two offerings differ.
The URI you pass to `connect` decides where the work runs. LanceDB Embedded
runs in your application's process; your data can live on local disk or in
object storage. LanceDB Enterprise runs LanceDB Distributed: query execution,
indexing and maintenance run on the Enterprise service, outside your
application. The same Python, TypeScript and Rust SDKs open all three kinds;
[Embedded and Enterprise](/basics/offerings) describes how the two offerings differ.
| URI | Where queries run |
| --- | --- |
| A local path, such as `./data` | OSS: in your process |
| `s3://`, `gs://` or `az://` | OSS: in your process, reading object storage |
| `db://` | LanceDB Enterprise: queries run on your cluster |
| A local path, such as `./data` | LanceDB Embedded: in your process |
| `s3://`, `gs://` or `az://` | LanceDB Embedded: in your process, reading object storage |
| `db://` | LanceDB Enterprise: LanceDB Distributed runs queries on your cluster |
## A local directory {#local}
+2 -2
View File
@@ -25,7 +25,7 @@ rewrite data; adding a derived column can write the new column independently.
A read of a table uses a consistent snapshot. A consistent snapshot can still
be older than another writer's latest commit. [Consistency](/tables/consistency)
explains how OSS and Enterprise choose the version a reader sees.
explains how LanceDB Embedded and LanceDB Enterprise choose the version a reader sees.
For an experiment that must keep using the same data, select a specific retained
table version rather than relying on a connection's default freshness.
@@ -46,7 +46,7 @@ describes commit behavior and compatibility between operations.
## Storage durability {#durability}
A committed table version is stored in the table’s configured storage. LanceDB
OSS uses your local filesystem or object store; LanceDB Enterprise uses the
Embedded uses your local filesystem or object store; LanceDB Enterprise uses the
cluster’s configured storage.
Table version history does not replace backups. Configure storage redundancy
+7 -6
View File
@@ -8,7 +8,7 @@ availability:
oss: available
enterprise: available
summary: >-
Job handles track operations in OSS and Enterprise. Enterprise also
Job handles track operations in LanceDB Embedded and LanceDB Enterprise. Enterprise also
supports listing server jobs, reopening them by ID and reading their events.
---
@@ -19,7 +19,7 @@ column refresh. Use it to check progress, wait for completion and inspect
failures.
Methods such as Python's `create_index_async` and TypeScript's
`createIndexAsync` return job handles. In OSS, the work runs in your application
`createIndexAsync` return job handles. In LanceDB Embedded, the work runs in your application
process; in Enterprise, it runs on server workers or through the SQL query
backend. A returned handle can represent an operation that has already finished,
so check its status before
@@ -58,12 +58,13 @@ Event responses have [row limits](/basics/limits#jobs). An explicit limit can
request more than the default, but it does not guarantee a complete event
history.
## OSS and Enterprise execution {#in-process}
## Embedded and Distributed execution {#in-process}
The following table describes local operations and Enterprise registry jobs.
The following table describes LanceDB Embedded operations and registry jobs on
LanceDB Distributed (provided by LanceDB Enterprise).
SQL-backed job handles follow the query lifecycle described below.
| | OSS | Enterprise registry job |
| | Embedded | Enterprise registry job |
| --- | --- | --- |
| Execution | In your application process | On server workers |
| Job identity | No server ID | A server ID for looking up the job |
@@ -72,7 +73,7 @@ SQL-backed job handles follow the query lifecycle described below.
### SQL-backed job handles
Remote materialized-view CREATE, REFRESH, and DROP use the SQL query backend.
On Enterprise, materialized-view CREATE, REFRESH, and DROP use the SQL query backend.
Python's `create_materialized_view_async()`, `mv.refresh_async()`, and
`drop_materialized_view_async()` return job handles with `status()`, `wait()`,
and `cancel()`. Synchronous creation, refresh, and drop wait on the same SQL
+1 -1
View File
@@ -44,7 +44,7 @@ framework runs the model; LanceDB provides access to the training data.
## Shared compute and maintenance in Enterprise {#where}
LanceDB OSS runs as an embedded library in your application. LanceDB Enterprise
LanceDB Embedded runs as an embedded library in your application. LanceDB Enterprise
provides shared infrastructure for applications and teams working with larger
multimodal workloads.
+2 -2
View File
@@ -14,7 +14,7 @@ multimodal AI. It defines how data is stored and organized into versioned tables
<span id="embedded" />
<span id="enterprise" />
LanceDB OSS and LanceDB Enterprise both use Lance as their storage foundation.
LanceDB Embedded and LanceDB Enterprise both use Lance as their storage foundation.
The format is developed as an independent open-source project, with its own
specification and tools.
@@ -42,4 +42,4 @@ those fields together.
- [Lance format specification](https://lance.org/format): how the format works.
- [Table versioning](/management/table#versioning): read and manage table versions.
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
- [Architecture](/architecture): how LanceDB OSS and Enterprise build on Lance.
- [Architecture](/architecture): how LanceDB Embedded and LanceDB Enterprise build on Lance.
+1 -1
View File
@@ -14,7 +14,7 @@ Databases, namespaces, tables, views and branches all have names. Choose names
that describe their purpose, such as `documents`, `prod` or `training_2026`.
Consistent names make data easier to find and application code easier to read.
Use the following rules for names that work across LanceDB OSS and Enterprise.
Use the following rules for names that work across LanceDB Embedded and LanceDB Enterprise.
| Object | Characters | Length |
| --- | --- | --- |
+8 -5
View File
@@ -1,6 +1,6 @@
---
title: "LanceDB OSS or Enterprise?"
sidebarTitle: "OSS and Enterprise"
title: "LanceDB Embedded or Enterprise?"
sidebarTitle: "Embedded and Enterprise"
description: "Start with an embedded library, or use the LanceDB Enterprise multimodal lakehouse for shared AI workloads."
icon: compass
keywords: ["oss", "enterprise", "embedded", "deployment", "availability", "comparison"]
@@ -9,15 +9,18 @@ keywords: ["oss", "enterprise", "embedded", "deployment", "availability", "compa
LanceDB gives you two ways to work with multimodal data, both built on the open
[Lance](/basics/lance) format.
**LanceDB OSS** is an embedded library. Run it inside your application
with tables on local disk or in object storage. You control the application,
**LanceDB Embedded (open source)** is a library that runs in your
application's process; your data can live on local disk or in object storage.
You control the application,
its resources, and table maintenance, without a separate database service.
[Try the quickstart](/quickstart) to create a table and run your first search.
**LanceDB Enterprise** is a multimodal lakehouse for data curation, feature
engineering, search, and training data access. It adds query-serving capacity,
data caching, and background operations for teams working with shared
multimodal datasets.
multimodal datasets. LanceDB Enterprise runs LanceDB Distributed: query
execution, indexing and maintenance run on the Enterprise service, outside your
application.
[Connect to Enterprise](/basics/connect#enterprise) when your cluster is ready.
<span id="labels" />
+1 -1
View File
@@ -17,7 +17,7 @@ in the last month.
| Interface | When to use it |
| --- | --- |
| **SDK query builders** | Build vector, full-text and hybrid searches, apply filters and choose the fields to return. Available in OSS and Enterprise. |
| **SDK query builders** | Build vector, full-text and hybrid searches, apply filters and choose the fields to return. Available in LanceDB Embedded and LanceDB Enterprise. |
| **Enterprise SQL** | Write SQL queries for filtering, aggregation, joins and other data curation tasks. Connect through Python or a Flight SQL client. |
Both interfaces work with LanceDB tables. Choose the one that supports your
+10 -7
View File
@@ -8,13 +8,16 @@ availability:
oss: available
enterprise: available
summary: >-
Python, TypeScript and Rust support both LanceDB OSS and LanceDB Enterprise.
Python, TypeScript and Rust support both LanceDB Embedded and LanceDB Enterprise.
Java provides a client for the Enterprise REST API.
---
Use a LanceDB SDK to create tables, add data, build indexes and run searches
from your application. Python, TypeScript and Rust work with both the embedded
OSS library and LanceDB Enterprise.
from your application. Python, TypeScript and Rust work with both LanceDB
Embedded and LanceDB Enterprise. LanceDB Embedded runs in your application's
process; your data can live on local disk or in object storage. LanceDB
Enterprise runs LanceDB Distributed: query execution, indexing and maintenance
run on the Enterprise service, outside your application.
The [quickstart](/quickstart) includes installation instructions and complete
examples in all three languages. For connection settings, see
@@ -73,16 +76,16 @@ For PyTorch worker setup and connection factories, see
## Moving to LanceDB Enterprise {#moving-to-enterprise}
The core table workflow stays familiar: connect, create or open a table, write
records, and query them. Enterprise runs those operations on remote compute and
manages background maintenance.
records, and query them. Enterprise runs those operations with LanceDB
Distributed and manages background maintenance.
- **Configure the connection.** Replace the local path with your database URI,
and supply the endpoint, region and credentials provided by your administrator.
[Connect to Enterprise](/basics/connect#enterprise) shows these settings.
- **Retrieve query results.** Python's remote tables do not support whole-table
- **Retrieve query results.** Python tables on Enterprise do not support whole-table
`to_arrow()` or `to_pandas()` calls. Use a query and select the columns and
number of rows you need, for example `table.search().limit(10).to_arrow()`.
- **Handle table listings and namespaces.** Remote table listings are paginated.
- **Handle table listings and namespaces.** Enterprise table listings are paginated.
Use the namespace and pagination options described in
[Namespaces](/management/namespace).
- **Use server-side jobs.** Index builds and feature refreshes can run as
+4 -4
View File
@@ -8,7 +8,7 @@ availability:
oss: unavailable
enterprise: available
summary: >-
Enterprise supports SQL through Arrow Flight SQL. Use the SDK table APIs for embedded OSS tables. See [Connection settings](#sql-is-remote) to connect.
Enterprise supports SQL through Arrow Flight SQL. Use the SDK table APIs for LanceDB Embedded tables. See [Connection settings](#sql-is-remote) to connect.
---
Use SQL in LanceDB Enterprise to filter records, join tables, calculate
@@ -52,7 +52,7 @@ API and SQL endpoints may use different hosts or ports.
See [Connect to Enterprise](/basics/connect#enterprise) for other connection options.
SQL runs in Enterprise. Connections to a local directory or an object-storage
bucket use the embedded OSS library and cannot execute SQL statements. Use
bucket use LanceDB Embedded and cannot execute SQL statements. Use
SDK table operations to query those tables.
## Run a query in Python {#sql-from-python}
@@ -94,7 +94,7 @@ handles, streaming, and cancellation.
Use **SQL** for joins, aggregations, window functions, and analytical queries
written in SQL. Use **SDK table operations** to build applications around table
management, ingestion, filtering, and search, including applications that run
with the embedded OSS library.
with LanceDB Embedded.
You can combine them. For example, create and populate a table with an SDK,
then query it with SQL to prepare a training dataset. In Python, both workflows
@@ -130,7 +130,7 @@ server version supports this form, then use
[`EXPLAIN`](/sql/dql/explain) to inspect the query plan.
The [SDK vector search API](/search/vector-search) is also available in
Enterprise and OSS. Use it when you prefer the SDK query builder or need to
LanceDB Enterprise and LanceDB Embedded. Use it when you prefer the SDK query builder or need to
search embedded tables.
For full-text search in SQL, use the [`fts()`](/sql/functions#fts) table
+2 -2
View File
@@ -23,8 +23,8 @@ records belong, and save the selected data or the rules used to produce it.
| [Rank and deduplicate](/curation/windowing) | Which records to keep when several describe the same item. |
The analytical examples in this section use **Enterprise SQL**. Connect with
the [SQL guide](/basics/sql). For sampling through the Python SDK in OSS or
Enterprise, see [Sample without SQL](/curation/sampling#training).
the [SQL guide](/basics/sql). For sampling through the Python SDK in LanceDB Embedded or
LanceDB Enterprise, see [Sample without SQL](/curation/sampling#training).
## Prepare a labeled document dataset {#shape}
+2 -2
View File
@@ -8,7 +8,7 @@ availability:
oss: unavailable
enterprise: available
summary: >-
Enterprise supports SQL `TABLESAMPLE`. For sampling in OSS and Enterprise through the Python SDK, see [Sample without SQL](#training).
Enterprise supports SQL `TABLESAMPLE`. For sampling in LanceDB Embedded and LanceDB Enterprise through the Python SDK, see [Sample without SQL](#training).
---
import { PySampleDataset } from '/snippets/curation_sampling.mdx';
@@ -143,7 +143,7 @@ for sampling. `TABLESAMPLE` cannot be applied directly to a subquery result.
## Sample without SQL {#training}
The Python SDK provides a permutation builder for selecting and ordering rows
in OSS and Enterprise. A split selects a subset; a shuffle changes the order
in LanceDB Embedded and LanceDB Enterprise. A split selects a subset; a shuffle changes the order
in which records are read. `Permutation.identity(table)` retains stored order.
<Accordion title="Complete Python example: read a random subset">
+2 -2
View File
@@ -11,7 +11,7 @@ LanceDB emits internal metrics (currently object store request counts, bytes tra
The bridge is available in the Python and TypeScript SDKs. It is a thin wrapper over LanceDB's `metrics` recorder; your application supplies and configures the OpenTelemetry SDK.
<Note>
This page covers LanceDB OSS instrumentation. LanceDB Enterprise clusters emit their own metrics from the server side. See [Enterprise monitoring](#enterprise) below.
This page covers LanceDB Embedded instrumentation. LanceDB Enterprise clusters emit their own metrics from the server side. See [Enterprise monitoring](#enterprise) below.
</Note>
## Available metrics {#what-you-get}
@@ -113,7 +113,7 @@ export them with data enrichment. No client-side instrumentation is needed.
| Component | Metrics |
| --- | --- |
| Query node | RPS, query latency, error codes, slow-query count, CPU/memory utilization |
| Plan executor | SSD cache hit/miss rate, CPU/memory utilization |
| Executor | SSD cache hit/miss rate, CPU/memory utilization |
[Contact support](/deployment/support) for the comprehensive metric list and
help integrating them with your monitoring dashboard.
+2 -2
View File
@@ -1,7 +1,7 @@
---
title: "Embedded storage configuration"
sidebarTitle: "Embedded Storage"
description: "Choose and configure local disk or object storage for tables accessed by the embedded LanceDB OSS library."
description: "Choose and configure local disk or object storage for tables accessed by LanceDB Embedded."
icon: database
keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"]
availability:
@@ -13,7 +13,7 @@ availability:
the deployment's own configuration instead.
---
This page is about the embedded library, where your connection chooses and
This page is about LanceDB Embedded, where your connection chooses and
configures the storage backend. A LanceDB Enterprise deployment sets up its own
storage when it is deployed, so a `db://` connection passes none of these
options except an Azure storage account name. See
+2 -2
View File
@@ -1,12 +1,12 @@
---
title: "Support"
sidebarTitle: "Support"
description: "Where to get help: community Discord for OSS, dedicated support for Enterprise."
description: "Where to get help: community Discord for LanceDB Embedded (open source), dedicated support for LanceDB Enterprise."
icon: circle-question
keywords: ["support", "help", "discord", "community"]
---
## LanceDB OSS {#oss}
## LanceDB Embedded (open source) {#oss}
For a quick question, the
[Discord community](https://discord.gg/AUEWnJ7Txb) has channels for
+4 -4
View File
@@ -35,7 +35,7 @@ Modern machine learning models can be trained to convert raw data into embedding
of floating point numbers. The position of an embedding in vector space captures the semantics of
the data, so vectors that are close to each other are considered similar.
LanceDB provides an embedding function registry in OSS as well as its Enterprise versions
LanceDB provides an embedding function registry in LanceDB Embedded as well as LanceDB Enterprise
([see below](#embeddings-in-lancedb-enterprise))
that automatically generates vector embeddings during data ingestion. Query text is embedded
automatically too, in the client, for local and Enterprise tables alike, with SDK-specific query
@@ -198,7 +198,7 @@ You can find all supported embedding models in the [integrations](/integrations/
<Badge color="red">Enterprise</Badge>
In LanceDB Enterprise, embedding generation during data ingestion is client-side and the resulting vectors are
stored on the remote table.
stored in the Enterprise table.
<Info>
The Enterprise server does not currently generate embeddings from query text on its own. Any automatic
@@ -207,13 +207,13 @@ query-time embedding happens on the client side.
### How string queries are interpreted {#how-string-queries-are-interpreted}
For the Python remote client, `table.search("hello")` can take two different paths:
When the Python SDK connects to Enterprise, `table.search("hello")` can take two different paths:
- If the selected vector column has embedding metadata
(i.e., the table schema stores the source-column, vector-column, and
embedding-function mapping created from fields like `SourceField()` and
`VectorField()` during table creation), then the embeddings are computed in the Python client process.
The client uses the same local LanceDB embedding registry used by OSS tables to
The client uses the same local LanceDB embedding registry used by LanceDB Embedded tables to
reconstruct the embedding function from schema metadata, compute the query vector in
the client process, and send that vector to Enterprise for search.
```python
+4 -4
View File
@@ -42,11 +42,11 @@ from lancedb.embeddings import get_registry
## Step 2: Connect to LanceDB {#step-2-connect-to-lancedb}
Establish a connection to your LanceDB OSS directory or Enterprise cluster:
Establish a connection to your LanceDB Embedded directory or Enterprise cluster:
<CodeGroup>
```python Python icon="python"
# Enter your LanceDB connection URI for OSS or Enterprise here
# Enter your LanceDB connection URI for LanceDB Embedded or Enterprise here
db = lancedb.connect(...)
```
@@ -126,8 +126,8 @@ LanceDB preserves them instead of filling only the missing rows.
## Step 6: Query with Automatic Embedding {#step-6-query-with-automatic-embedding}
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. In the Python remote
client, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. When the Python
SDK connects to Enterprise, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
+2 -2
View File
@@ -9,7 +9,7 @@ availability:
enterprise: available
summary: >-
Requests to a deployment carry an API key or an OAuth token, whichever it is
configured to accept. Embedded LanceDB has no server to sign in to; see
configured to accept. LanceDB Embedded has no server to sign in to; see
[Architecture](/architecture).
---
@@ -423,5 +423,5 @@ control. All three SDKs deliberately omit the secret from their debug/repr outpu
so accidental log lines do not expose the value.
</Warning>
Embedded LanceDB has nothing to authenticate to: your process opens the files
LanceDB Embedded has nothing to authenticate to: your process opens the files
directly, and the object store's own credentials are the only gate.
+1 -1
View File
@@ -16,5 +16,5 @@ Deployment is specific to your cloud and account, so the procedure is published
with LanceDB Enterprise. [Contact us](https://www.lancedb.com/contact) to get
started.
Embedded LanceDB has nothing to deploy: it is a library your process links
LanceDB Embedded has nothing to deploy: it is a library your process links
against. See [Architecture](/architecture#embedded).
+4 -4
View File
@@ -8,17 +8,17 @@ availability:
enterprise: available
summary: >-
LanceDB Enterprise distributes index execution and table-data reads across
plan executors. Embedded LanceDB executes searches in the application process.
executors. LanceDB Embedded executes searches in the application process.
---
Distributed search spreads index execution, table-data reads, and caching across
multiple serving machines in LanceDB Enterprise. Query nodes plan queries and
combine results. Plan executors cache index structures and table data and execute
the remote work. See [Enterprise architecture](/enterprise/architecture).
combine results. Executors cache index structures and table data and execute
the distributed work. See [Enterprise architecture](/enterprise/architecture).
## Embedded search {#embedded-search}
LanceDB OSS runs inside your application process. It searches indexes and reads
LanceDB Embedded runs inside your application process. It searches indexes and reads
table data using that process's resources, including its index cache. Reading a
table from object storage does not by itself distribute query execution across
machines.
+1 -1
View File
@@ -1,7 +1,7 @@
---
title: "LanceDB Enterprise"
sidebarTitle: "Overview"
description: "What a deployment adds over embedded LanceDB."
description: "What a deployment adds over LanceDB Embedded."
icon: building
keywords: ["enterprise", "deployment", "cluster"]
---
+1 -1
View File
@@ -16,5 +16,5 @@ Compliance posture (certifications, data residency, encryption at rest and in
transit, audit logging) is a property of a deployment, so it is documented
with LanceDB Enterprise.
For embedded LanceDB, these are properties of the object store you point it at
For LanceDB Embedded, these are properties of the object store you point it at
and the process you run it in; LanceDB adds no layer of its own.
@@ -16,9 +16,9 @@ Declaring it is cheap on any table size, because nothing is evaluated until
[a refresh](/feature-engineering/refresh-column) asks for it.
<Note>
In the 0.41 SDK release, local computed-column declaration and refresh require a
In the 0.41 SDK release, computed-column declaration and refresh in LanceDB Embedded require a
source build with the `experimental-functions` Cargo feature. Published Python
and TypeScript packages leave it disabled. Remote Cloud and Enterprise operations
and TypeScript packages leave it disabled. LanceDB Enterprise operations
do not require this feature. Ordinary eager `add_columns` transforms remain available.
</Note>
+3 -3
View File
@@ -70,7 +70,7 @@ changes incrementally for supported queries; other queries require a full rebuil
See [refresh behavior](/feature-engineering/materialized-views#incremental) for
version-specific details.
Enterprise runs materialization on remote compute. Use the SDK and refresh
Enterprise runs materialization on the Enterprise service. Use the SDK and refresh
options supported by your server version; see
[Enterprise refreshes](/feature-engineering/materialized-views#availability).
@@ -78,8 +78,8 @@ options supported by your server version; see
| | What it produces | Filled by | Available |
| --- | --- | --- | --- |
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | OSS and Enterprise |
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | OSS and Enterprise |
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | Embedded and Enterprise |
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Embedded and Enterprise |
| [Geneva](/geneva) | columns filled by Python | the `geneva` package | <Badge color="red">Enterprise</Badge> |
{/*
@@ -8,7 +8,7 @@ availability:
oss: available
enterprise: available
summary: >-
Local creation and refresh require a source build with the experimental-functions Cargo feature. Enterprise runs the work on remote compute. See [Enterprise refreshes](#availability) for server compatibility requirements.
Creation and refresh in LanceDB Embedded require a source build with the experimental-functions Cargo feature. Enterprise runs the work on the Enterprise service. See [Enterprise refreshes](#availability) for server compatibility requirements.
---
import {
@@ -21,10 +21,10 @@ filtered or transformed dataset that you can query, index and search repeatedly.
Refresh the view after its source data changes.
<Note>
In the 0.41 SDK release, local materialized-view creation and refresh require a
In the 0.41 SDK release, materialized-view creation and refresh in LanceDB Embedded require a
source build with the `experimental-functions` Cargo feature. Published Python
and TypeScript packages leave it disabled. Opening, reading, listing and dropping
existing views remain available, as do remote Cloud and Enterprise operations.
existing views remain available, as do LanceDB Enterprise operations.
</Note>
<Note>
@@ -84,7 +84,7 @@ when maintaining an existing pipeline.
<Badge color="red">Enterprise</Badge>
Enterprise runs materialized-view creation and refresh on remote compute.
Enterprise runs materialized-view creation and refresh on the Enterprise service.
For SQL, see [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) and
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view).
@@ -16,9 +16,9 @@ new rows and recompute values whose inputs changed. The example below continues
with the `doubled` column from the computed-column guide.
<Note>
In the 0.41 SDK release, local declaration and refresh require a source build
In the 0.41 SDK release, declaration and refresh in LanceDB Embedded require a source build
with the `experimental-functions` Cargo feature. Published Python and TypeScript
packages leave it disabled. Remote Cloud and Enterprise operations do not require
packages leave it disabled. LanceDB Enterprise operations do not require
this feature; existing computed-column data remains readable in default builds.
</Note>
@@ -82,9 +82,9 @@ table.add_columns(
Both options apply to every computed output declared in that call, including
registered Function outputs. Automatic refresh is enabled by default. When
the interval is omitted, the server determines when to refresh. Local tables
persist only an explicitly configured interval; automatic execution requires a
Cloud or Enterprise server with its separate `column_refresh.enabled` switch
the interval is omitted, the server determines when to refresh. LanceDB Embedded
tables persist only an explicitly configured interval; automatic execution requires a
LanceDB Enterprise deployment with its separate `column_refresh.enabled` switch
turned on. That server switch is disabled in the base configuration.
The interval is a positive minimum delay after the last successful refresh
+1 -1
View File
@@ -8,7 +8,7 @@ availability:
oss: unavailable
enterprise: available
summary: >-
Enterprise supports catalog privileges when its administrator configures enforcement. Embedded LanceDB has no catalog authorization layer: access is whatever the object store allows.
Enterprise supports catalog privileges when its administrator configures enforcement. LanceDB Embedded has no catalog authorization layer: access is whatever the object store allows.
---
[Authentication](/enterprise/authentication) establishes who is calling.
+9 -6
View File
@@ -29,13 +29,16 @@ For a deeper look at how this works in training pipelines, start with [Why Lance
## LanceDB suite {#lancedb-suite}
The LanceDB suite includes LanceDB OSS, an open-source embedded retrieval library, and LanceDB Enterprise,
The LanceDB suite includes LanceDB Embedded (open source), a retrieval library, and LanceDB Enterprise,
a multimodal lakehouse platform for the full AI data lifecycle.
OSS is easy to set up on a local machine for search and regular-scale workflows. LanceDB Enterprise is built
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
It is easy to set up on a local machine for search and regular-scale workflows. LanceDB Enterprise is built
for teams that need scale without building bespoke infrastructure for curation,
feature engineering, search and retrieval, and efficient training data access.
LanceDB Enterprise runs LanceDB Distributed: query execution, indexing and maintenance
run on the Enterprise service, outside your application.
![LanceDB suite: OSS search and Enterprise multimodal lakehouse on Lance format](/static/assets/images/overview/lancedb-suite.svg)
![LanceDB suite: LanceDB Embedded search and Enterprise multimodal lakehouse on Lance format](/static/assets/images/overview/lancedb-suite.svg)
## Why teams use LanceDB {#why-teams-use-lancedb}
@@ -74,11 +77,11 @@ feature engineering, search and retrieval, and efficient training data access.
## From local development to production scale {#from-local-development-to-production-scale}
LanceDB OSS and LanceDB Enterprise share the same Lance format and table model. Start locally with the embedded OSS
library, then move to Enterprise when your team needs distributed scale, managed infrastructure, private deployment,
LanceDB Embedded and LanceDB Enterprise share the same Lance format and table model. Start locally with
LanceDB Embedded, then move to Enterprise when your team needs distributed scale, managed infrastructure, private deployment,
or higher-throughput curation, feature engineering, search and retrieval, and training workflows.
### 1\. LanceDB OSS {#1-lancedb-oss}
### 1\. LanceDB Embedded {#1-lancedb-oss}
The fastest way to get started is the open-source embedded library, with client SDKs in Python, TypeScript
and Rust. Run it locally in just a few steps, which lets you explore datasets, curate data, and run search and retrieval workloads
for agents. Start here:
+1 -1
View File
@@ -14,7 +14,7 @@ availability:
import { PyFtsIndexAsync as FtsIndexAsync, PyFtsIndexCreate as FtsIndexCreate, PyFtsIndexNested as FtsIndexNested, PyFtsIndexWait as FtsIndexWait } from '/snippets/indexing.mdx';
LanceDB provides performant full-text search based on BM25, allowing you to incorporate keyword-based search in your retrieval solutions. This page shows
examples on how to create and configure FTS indexes in LanceDB OSS and Enterprise, using the synchronous and asynchronous APIs.
examples on how to create and configure FTS indexes in LanceDB Embedded and LanceDB Enterprise, using the synchronous and asynchronous APIs.
<Note>
In LanceDB Enterprise, `create_fts_index` API returns immediately, but index building happens asynchronously.
+4 -4
View File
@@ -65,7 +65,7 @@ LanceDB Enterprise supports incremental reindexing through an automated backgrou
The benefit of using LanceDB Enterprise is that it automates the reindexing process
and operates continuously in the background, minimizing the impact on latency under high loads.
In OSS, you must manually manage the reindexing cadence based on your data growth and performance needs.
In LanceDB Embedded, you must manually manage the reindexing cadence based on your data growth and performance needs.
## Disk utilization {#disk-utilization}
@@ -73,7 +73,7 @@ Compaction by itself does not immediately free disk space, and can temporarily i
compacted files are written before old-version files are deleted. Disk space is reclaimed when old versions
are pruned during cleanup. Set retention only as low as your rollback and time-travel requirements allow.
If you need to reclaim space more aggressively in OSS, use a shorter retention window:
If you need to reclaim space more aggressively in LanceDB Embedded, use a shorter retention window:
<CodeGroup>
```python Python icon=Python
from datetime import timedelta
@@ -109,11 +109,11 @@ GPU acceleration changes how the vector index is built, not the lifecycle after
embedded library, rows appended later still need `optimize()` before they are part of the index; a
deployment adds them in the background. `fast_search()` only searches indexed rows.
### Manual GPU indexing in LanceDB OSS {#gpu-manual-gpu-indexing-in-lancedb-oss}
### Manual GPU indexing in LanceDB Embedded {#gpu-manual-gpu-indexing-in-lancedb-oss}
You can use the Python SDK to manually create the `IVF_PQ` index on a GPU. You'll need
[PyTorch>2.0](https://pytorch.org/). Note that GPU-based indexing is currently only
supported by the synchronous SDK in LanceDB OSS.
supported by the synchronous SDK in LanceDB Embedded.
Specify the values `cuda` or `mps` (on Apple Silicon) for the `accelerator` parameter
to enable GPU training on your device.
+4 -4
View File
@@ -159,7 +159,7 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil
The `distance_type` can be specified at search time in addition to build time. If
you do this, set it to the same value used to build the vector index. If the search
metric does not match the index metric, LanceDB cannot use the index and falls
back to a costly flat search. On LanceDB Cloud, the brute-force KNN safety limit
back to a costly flat search. On LanceDB Cloud (legacy hosted service), the brute-force KNN safety limit
can reject that search on a large table instead of executing it. An Enterprise
deployment applies the same limit only when it is configured to.
@@ -279,7 +279,7 @@ In LanceDB Enterprise, vector indexes are managed **automatically**. The system
- GPU-based `IVF_PQ` builds are available on request. [Contact support](/deployment/support) to enable them for your deployment. Build time depends on dataset size, dimensions and GPU resources.
<Badge color="gray">Open-Source</Badge>
LanceDB OSS users can manually create vector indexes by calling `table.create_index()`.
LanceDB Embedded users can manually create vector indexes by calling `table.create_index()`.
See the above sections for guidance on manually tuning index parameters as data changes.
<Note>
@@ -290,7 +290,7 @@ In the embedded library the index is built before `create_index()` returns.
</Note>
<Note>
Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal
Rows appended after an initial index build remain outside the index until refreshed manually (Embedded) or automatically (Enterprise). Normal
search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
fallback and searches only indexed rows.
</Note>
@@ -330,7 +330,7 @@ Each entry returned by `list_indices()` also carries detailed per-index metadata
<Note>
These fields are populated for local tables. In LanceDB Enterprise, `list_indices()` returns them except `num_unindexed_rows`; use `index_stats()` for index coverage.
These fields are populated for LanceDB Embedded tables. In LanceDB Enterprise, `list_indices()` returns them except `num_unindexed_rows`; use `index_stats()` for index coverage.
</Note>
#### Custom Index Names
+1 -1
View File
@@ -8,7 +8,7 @@ availability:
oss: available
enterprise: available
summary: >-
Create and open tables in OSS and Enterprise.
Create and open tables in LanceDB Embedded and LanceDB Enterprise.
---
import {
+2 -2
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Delete rows through the SDKs in OSS and Enterprise. Storage reclamation follows compaction and retention policy.
Delete rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Storage reclamation follows compaction and retention policy.
---
import { RsExample as RsDeleteRows } from '/snippets/ingestion_delete.mdx';
@@ -70,7 +70,7 @@ Compaction rewrites the live data layout; cleanup removes eligible retained
versions and files. Retained history can continue to reference data that no
longer appears in the latest table version.
In OSS, `optimize()` provides maintenance operations. In Enterprise, managed
In LanceDB Embedded, `optimize()` provides maintenance operations. In Enterprise, managed
maintenance and retention follow the service configuration. Choose retention
according to the history you need, rather than shortening it as part of a
routine delete example. See [Compaction](/maintenance/compaction) and
+2 -2
View File
@@ -8,7 +8,7 @@ availability:
oss: available
enterprise: available
summary: >-
Create tables and write records through the SDKs in OSS and Enterprise. Enterprise also provides SQL write statements.
Create tables and write records through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also provides SQL write statements.
---
import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx';
@@ -32,7 +32,7 @@ with a stable key when the incoming data should update existing records.
## Connect and prepare the data {#connect-to-lancedb}
Use the [connection guide](/basics/connect) to connect to OSS or Enterprise.
Use the [connection guide](/basics/connect) to connect to LanceDB Embedded or LanceDB Enterprise.
The SDK write operations in this section work with both. Enterprise also
supports [SQL write statements](/sql#dml); an SDK filter expression does not
require a separate SQL connection.
+4 -4
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Append records through the SDKs in OSS and Enterprise. For large Enterprise writes, see [large upload settings](#bulk-ingests-into-a-remote-table).
Append records through the SDKs in LanceDB Embedded and LanceDB Enterprise. For large Enterprise writes, see [large upload settings](#bulk-ingests-into-a-remote-table).
---
import {
@@ -167,7 +167,7 @@ Wait for the write operation's result, then check the expected records.
Progress callbacks describe activity; they do not establish that a write
succeeded. If a request fails after the server may have committed it, blindly
repeating an append can duplicate rows. The client's automatic retries can do
the same: on a remote table, the Python and TypeScript 0.40.0 clients resend an
the same: on an Enterprise table, the Python and TypeScript 0.40.0 clients resend an
append after a retryable status such as `504`, even when the server had already
committed it, and `add()` still returns success. Reconcile using your record
IDs, or choose an upsert policy designed for replay.
@@ -176,9 +176,9 @@ IDs, or choose an upsert policy designed for replay.
<Badge color="red">Enterprise</Badge>
This section applies only to remote tables (`db://` connections). Embedded (local) writes are unaffected.
This section applies only to LanceDB Enterprise tables (`db://` connections). LanceDB Embedded writes are unaffected.
When you write to a remote LanceDB Enterprise table, the client sends the write as one HTTP `insert` request unless it splits the data into more than one partition, which it does for a large input it can measure or when you set [`write_parallelism`](#write-parallelism). A partitioned write goes up as a multipart write: the client cuts each partition into one or more `insert` parts and uploads them under a single upload id. Each part is streamed. Its size and duration still affect resource use and
When you write to a LanceDB Enterprise table, the client sends the write as one HTTP `insert` request unless it splits the data into more than one partition, which it does for a large input it can measure or when you set [`write_parallelism`](#write-parallelism). A partitioned write goes up as a multipart write: the client cuts each partition into one or more `insert` parts and uploads them under a single upload id. Each part is streamed. Its size and duration still affect resource use and
timeouts: it has to finish within the client read timeout while the server writes it to object storage, so on very large ingests an oversized part can run past that timeout and surface as:
```text
+2 -2
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Merge insert is available through the SDKs in OSS and Enterprise. Use it for key-based updates and inserts.
Merge insert is available through the SDKs in LanceDB Embedded and LanceDB Enterprise. Use it for key-based updates and inserts.
---
import {
@@ -230,7 +230,7 @@ when changing the input used to generate it.
Rust's `when_matched_update_all_expr` and
`when_not_matched_by_source_delete_expr` accept DataFusion expressions for
local tables. Use the string predicate variants for Enterprise tables.
LanceDB Embedded tables. Use the string predicate variants for Enterprise tables.
## Complete write workflow {#complete-example}
+2 -2
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Update rows through the SDKs in OSS and Enterprise. Enterprise also supports the SQL UPDATE statement.
Update rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also supports the SQL UPDATE statement.
---
import {
@@ -71,7 +71,7 @@ Rust's `column` argument is an expression, so the string value includes SQL quot
In this separate example, Bob keeps his name and his login count becomes **21**.
Python's `values_sql` and TypeScript's `valuesSql` accept expressions.
These SDK expressions work with embedded OSS tables; they do not require an
These SDK expressions work with LanceDB Embedded tables; they do not require an
Enterprise SQL endpoint. Enterprise also offers [`UPDATE`](/sql/dml/update).
## Verify the affected data {#verify}
+1 -1
View File
@@ -139,7 +139,7 @@ We get the response based on the transcript's contents and title as expected.
## Why this works well {#why-this-works-well}
To start, LanceDB OSS can run from a local directory, so transcript data can stay on your machine when you are using the OSS stack.
To start, LanceDB Embedded can run from a local directory, so transcript data can stay on your machine.
- You do not need to maintain a separate transcript parser in your application code.
- You do not need to hand-roll chunking and retrieval orchestration across multiple modules.
+1 -1
View File
@@ -29,7 +29,7 @@ prompt, plus lexical (keyword) search over past sessions. Both are useful, but k
misses paraphrases of what you originally typed, the exact thing you need when recalling a
fact you phrased differently months ago.
LanceDB OSS is an embedded retrieval library, which makes it a natural fit here:
LanceDB Embedded is a retrieval library that runs inside your application, which makes it a natural fit here:
- **No server to stand up**: it reads and writes a table on local disk, so the plugin ships
as a dependency rather than a service to operate.
+2 -2
View File
@@ -40,7 +40,7 @@ In this example, we will be fetching movie information from the [Open Movie Data
3. **Specify necessary credentials and/or embedding model details:**
To fetch data from the OMDb API, you will need to pass a valid API key into your pipeline. Depending on whether you're using LanceDB OSS or LanceDB Enterprise, you also may need to provide the necessary credentials to connect to the LanceDB instance. These can be pasted inside `.dlt/secrets.toml`.
To fetch data from the OMDb API, you will need to pass a valid API key into your pipeline. Depending on whether you're using LanceDB Embedded or LanceDB Enterprise, you also may need to provide the necessary credentials to connect to the LanceDB instance. These can be pasted inside `.dlt/secrets.toml`.
dlt's LanceDB integration also allows you to automatically embed the data during ingestion. Depending on the embedding model chosen, you may need to paste the necessary credentials inside `.dlt/secrets.toml`:
```toml
@@ -52,7 +52,7 @@ In this example, we will be fetching movie information from the [Open Movie Data
embedding_model = "all-MiniLM-L6-v2"
[destination.lancedb.credentials]
uri = ".lancedb"
api_key = "api_key" # API key to connect to LanceDB Enterprise. Leave out if you are using LanceDB OSS.
api_key = "api_key" # API key to connect to LanceDB Enterprise. Leave out if you are using LanceDB Embedded.
embedding_model_provider_api_key = "embedding_model_provider_api_key" # Not needed for providers that don't need authentication (ollama, sentence-transformers).
```
See [here](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb#configure-the-destination) for more information and for a list of available models and model providers.
+1 -1
View File
@@ -6,7 +6,7 @@ description: "Learn how to use the DuckDB-Lance extension to query Lance tables
LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables.
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for local tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for LanceDB Embedded tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
## Install {#install}
+1 -1
View File
@@ -43,7 +43,7 @@ Run vector search and keep the results as a Polars `DataFrame` for further proce
## Work with LazyFrames {#work-with-lazyframes}
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for local tables only; with LanceDB Enterprise, `search(...).to_polars()` returns an eager DataFrame of query results.
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for LanceDB Embedded tables only; with LanceDB Enterprise, `search(...).to_polars()` returns an eager DataFrame of query results.
<CodeBlock filename="Python" language="Python" icon="python">
{PyPlatformsPolarsLazyframe}
+1 -1
View File
@@ -48,7 +48,7 @@ log from growing without bound:
| --- | --- |
| `prewarm_page_cache` | Loads table data into a query node's page cache ahead of a request needing it. |
| `prewarm_index_cache` | The same, for index structures. |
| `startup_prewarm_idx` | Runs cache prewarming as part of a plan executor's own startup, rather than on a schedule, when the deployment enables it. |
| `startup_prewarm_idx` | Runs cache prewarming as part of an executor's own startup, rather than on a schedule, when the deployment enables it. |
## Registry housekeeping {#registry}
+4 -3
View File
@@ -18,7 +18,7 @@ those versions referenced.
<Note>
**Enterprise job type:** `cleanup`. Distinct from `compact`: compaction
merges files, cleanup removes them. `optimize()` in OSS runs both in one call;
merges files, cleanup removes them. `optimize()` in LanceDB Embedded runs both in one call;
Enterprise schedules them as separate jobs, and runs cleanup only when the
deployment's configuration enables it.
</Note>
@@ -89,8 +89,9 @@ directly can corrupt current or historical snapshots. Upgrade all clients and
maintenance workers to a version that understands Managed blobs before using
them, and reopen handles created before that upgrade.
For local tables, `optimize()` performs compaction and cleanup. Enterprise runs
these as background jobs; the remote Python table does not support `optimize()`.
For LanceDB Embedded tables, `optimize()` performs compaction and cleanup.
Enterprise runs these as background jobs; Python tables connected to Enterprise
do not support `optimize()`.
See [Background jobs](/maintenance/background-jobs) for the Enterprise path.
## Disk usage does not drop immediately {#disk}
+5 -5
View File
@@ -24,7 +24,7 @@ Run optimization as regular table maintenance, particularly after many small
writes or a large ingest.
{/*
Draft for the OSS compaction SQ partition check release. Keep this section
Draft for the LanceDB Embedded compaction SQ partition check release. Keep this section
unpublished until that behavior ships.
### Recover from an oversized SQ partition {#oversized-sq-partition}
@@ -62,7 +62,7 @@ needs to.
<Note>
**Enterprise job type:** `compact`. Runs automatically as data accumulates,
separately from [`cleanup`](/maintenance/cleanup). The two are distinct jobs
even though `optimize()` bundles both in one OSS call.
even though `optimize()` bundles both in one LanceDB Embedded call.
</Note>
## Retraining indexes {#retrain}
@@ -113,7 +113,7 @@ Two things accumulate on a long-lived table as more and more data gets added to
- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
In LanceDB Embedded, you run them yourself via `optimize()`, which bundles both into a single call:
```python Python icon="python"
from datetime import timedelta
@@ -129,7 +129,7 @@ until the first supported release is confirmed.
### Tune compaction for wide rows {#tune-wide-rows}
For an existing local table, derive a fragment row target from the expected
For an existing LanceDB Embedded table, derive a fragment row target from the expected
row size. These examples use rows averaging 1 KiB and a 64 KiB fragment
target; choose sizes for your workload. Smaller scan batches and one compaction
task at a time can also reduce memory use.
@@ -148,7 +148,7 @@ separately when needed; Python's `optimize()` includes those steps.
TypeScript's `optimize()` does not currently expose the fragment row target,
scan batch size, or compaction thread count. These tuning examples apply to
Python and Rust local tables.
Python and Rust tables in LanceDB Embedded.
Planning remains row based. The row target bounds later passes after existing
oversized fragments have been rewritten; the first pass rewrites those
+14 -12
View File
@@ -8,8 +8,9 @@ availability:
oss: available
enterprise: available
summary: >-
OSS opens tables in a local directory or object storage. Enterprise connects
to remote compute with a database URI and credentials.
LanceDB Embedded opens tables in a local directory or object storage.
LanceDB Enterprise connects with a database URI and credentials and runs
operations with LanceDB Distributed.
---
import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
@@ -22,14 +23,15 @@ objects, and configure how your application reaches the data.
| URI | Offering | Where queries run |
| --- | --- | --- |
| `./.lancedb` | OSS, with local storage | In your application |
| `s3://bucket/path`, `gs://bucket/path`, `az://container/path` | OSS, with object storage | In your application |
| `db://my-database` | Enterprise | On remote compute |
| `./.lancedb` | LanceDB Embedded, with local storage | In your application |
| `s3://bucket/path`, `gs://bucket/path`, `az://container/path` | LanceDB Embedded, with object storage | In your application |
| `db://my-database` | LanceDB Enterprise | On the Enterprise service (LanceDB Distributed) |
[Connect to LanceDB](/basics/connect) has examples for each connection type.
A connection to object storage still uses the embedded library: your process
reads and writes the bucket directly. Enterprise runs operations remotely and
manages the supporting compute and maintenance.
A connection to object storage still uses LanceDB Embedded: your process
reads and writes the bucket directly. LanceDB Enterprise runs LanceDB
Distributed: query execution, indexing and maintenance run on the Enterprise
service, outside your application.
## Connect to Enterprise {#deployment}
@@ -60,7 +62,7 @@ manager. See [Authentication](/enterprise/authentication) for OAuth and
## Storage options {#storage-options}
For OSS tables in object storage, `storage_options` configures credentials,
For LanceDB Embedded tables in object storage, `storage_options` configures credentials,
endpoints and options for new tables. This example enables stable row IDs for
tables created through the connection:
@@ -91,7 +93,7 @@ See [Storage](/deployment/storage) for provider credentials and other options.
## Read consistency {#consistency}
For OSS connections, `read_consistency_interval` controls how often reads check
For LanceDB Embedded connections, `read_consistency_interval` controls how often reads check
for table versions committed by other writers. For example, check at most once
every five seconds:
@@ -112,7 +114,7 @@ const db = await lancedb.connect("./.lancedb", { readConsistencyInterval: 5 });
```
</CodeGroup>
By default, an OSS table handle does not automatically pick up another
By default, an Embedded table handle does not automatically pick up another
process's writes. Reopen the table, check out its latest version, or configure
a consistency interval when you need those updates. A zero interval checks on
every read, with the cost of a metadata check.
@@ -139,7 +141,7 @@ versions and lifecycle operations.
<Warning>
Dropping a table removes its history as well as its current data. You cannot
recover a dropped table through table versioning. The OSS `drop_database()`
recover a dropped table through table versioning. The LanceDB Embedded `drop_database()`
and `drop_all_tables()` operations remove all tables without a confirmation prompt.
</Warning>
+7 -6
View File
@@ -8,7 +8,7 @@ availability:
oss: unavailable
enterprise: available
summary: >-
Enterprise supports Functions when enabled for your Enterprise cluster; the embedded library cannot register or run them.
Enterprise supports Functions when enabled for your Enterprise cluster; LanceDB Embedded cannot register or run them.
---
import {
@@ -31,8 +31,8 @@ rather than pulling rows to your process.
Registering and running a function needs a LanceDB Enterprise deployment with
Functions enabled, which they are not by default. On a
local connection `create_function` raises `NotImplementedError`; to derive a
column locally, declare a
LanceDB Embedded connection `create_function` raises `NotImplementedError`; to
derive a column there, declare a
[SQL computed column](/feature-engineering/add-computed-column) instead.
<Note>
@@ -282,14 +282,15 @@ as `bytes`; this feature does not add streamed outputs.
## Declare the environment {#environment}
The function runs remotely, so its dependencies are part of its definition:
The function runs on the Enterprise service, not in your application, so its
dependencies are part of its definition:
| | |
| --- | --- |
| `pip` | requirements for the remote environment |
| `pip` | requirements for the function's execution environment |
| `conda`, `conda_channels` | conda packages instead of pip; channels in priority order |
| `env` | environment variables |
| `python_version` | remote Python major/minor, defaulting to yours |
| `python_version` | Python major/minor for the execution environment, defaulting to yours |
| `gpu` | whether every execution requires a GPU |
These are part of the **immutable function version**. Changing a dependency
+2 -2
View File
@@ -78,8 +78,8 @@ That also means dropping one removes stored rows, unlike dropping a
## Enterprise compatibility {#deployment}
Enterprise supports creating and refreshing materialized views on remote
compute. Use the SDK version supported by your deployment, or the
Enterprise supports creating and refreshing materialized views on the
Enterprise service. Use the SDK version supported by your deployment, or the
[SQL interface](/sql/ddl/materialized-view). See
[Enterprise refreshes](/feature-engineering/materialized-views#availability)
for compatibility considerations when upgrading an existing pipeline.
+3 -3
View File
@@ -22,7 +22,7 @@ import {
} from '/snippets/connection.mdx';
LanceDB organizes Lance tables through a catalog-level abstraction defined by
the *namespace spec*. LanceDB OSS is an embedded library for accessing those
the *namespace spec*. LanceDB Embedded is an embedded library for accessing those
tables, and LanceDB Enterprise is a multimodal lakehouse that provides shared
compute and catalog services on top of the Lance format.
If you're coming from traditional databases or lakehouses, you can think of a namespace as the catalog path that says where a table name lives.
@@ -76,7 +76,7 @@ directory. Namespaces can nest inside it, as in this hierarchy:
The hierarchy is logical, not a tree of folders: a directory namespace records it in a
`__manifest` table in the root directory, and stores each table in a directory of its own there.
As a user of LanceDB OSS, you might never notice namespaces at first, because a plain connection
As a user of LanceDB Embedded, you might never notice namespaces at first, because a plain connection
uses only the root namespace, which is implicit. Connecting to this namespace is as simple as
connecting to the catalog root:
@@ -165,7 +165,7 @@ println!("Created table: {}", table.name());
</CodeGroup>
<Info>
- For simple use cases in LanceDB OSS, you don't need to go too deep into namespaces.
- For simple use cases in LanceDB Embedded, you don't need to go too deep into namespaces.
- To integrate LanceDB with external catalogs, understand how each namespace implementation resolves table names and locations in your organization's setup.
</Info>
+9 -8
View File
@@ -248,7 +248,7 @@ set to the exact version a model saw.
<Warning>
A shallow clone depends on the source's files continuing to exist. If the source
is [compacted](/maintenance/compaction) past the retention window, the version
the clone points at can be pruned. Local OSS connections support shallow clones
the clone points at can be pruned. LanceDB Embedded connections support shallow clones
only; `is_shallow=False` is not implemented. If the copy must outlive the
source's history, copy its data into an independent table instead. A deployment
that supports cloning also makes shallow clones only, and one whose catalog
@@ -462,8 +462,9 @@ pylance. Pandas conversions that need the Lance scanner, including plain-scan
`"lazy"` handles and legacy blobs, still require pylance. `to_arrow()`,
`to_list()`, `to_batches()`, and the fetch methods do not require pylance.
`blob_mode="bytes"` requires a local table snapshot, so remote and in-memory
tables raise `NotImplementedError`. On a remote table, `Table.to_pandas()` and
`blob_mode="bytes"` requires a LanceDB Embedded table snapshot, so Enterprise
and in-memory tables raise `NotImplementedError`. On an Enterprise table,
`Table.to_pandas()` and
`Table.to_arrow()` aren't supported, and a query's `to_pandas()` also raises in
the default `"lazy"` mode because it would open `BlobFile` handles. Query with
`blob_mode="descriptions"`, then read bytes with `fetch_blobs` or
@@ -495,7 +496,7 @@ descriptors. Both `"bytes"` and `"descriptions"` hide automatic row IDs;
explicitly requested row IDs remain visible.
Bytes mode has the same restrictions as `to_arrow(blob_mode="bytes")`,
including no legacy blobs, blobs inside lists, remote tables, or namespace
including no legacy blobs, blobs inside lists, Enterprise tables, or namespace
`QueryTable` pushdown. If concurrent writers can overwrite the table, use a
separate reader checked out to a fixed version before querying. Otherwise,
an overwrite can invalidate the row IDs used to fetch bytes and raise
@@ -686,7 +687,7 @@ between inline values and Managed objects.
Enterprise Functions can consume blob columns and
return new blobs. Their output payloads are managed by the destination table.
Local tables can also derive blob columns with
LanceDB Embedded tables can also derive blob columns with
[SQL computed columns](/feature-engineering/add-computed-column).
Deleting a row or replacing its blob does not immediately remove the old
@@ -741,8 +742,8 @@ Write raw bytes to a V1 blob column:
#### 3\. Convert blob tables to pandas {#multimodal-3-convert-blob-tables-to-pandas}
On local tables with V1 blob columns, `to_pandas()` in Python accepts `blob_mode`
(`"lazy"`, `"bytes"`, or `"descriptions"`). Remote tables raise
On LanceDB Embedded tables with V1 blob columns, `to_pandas()` in Python accepts
`blob_mode` (`"lazy"`, `"bytes"`, or `"descriptions"`). Enterprise tables raise
`NotImplementedError`, and `"bytes"` is not supported on in-memory tables.
Extra keyword arguments such as `split_blocks` or `self_destruct` are forwarded
to the PyArrow / Lance pandas conversion:
@@ -1537,7 +1538,7 @@ mixing in lower-level Lance file operations.
**System Operations**
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
In LanceDB OSS, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
In LanceDB Embedded, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
which is when old-version files are removed and disk space is reclaimed. A deployment prunes them in the background
when its configuration enables [cleanup](/maintenance/cleanup); `optimize()` is not available there.
+3 -2
View File
@@ -7,7 +7,8 @@ icon: rocket
import { PyConnect, TsConnect, RsConnect, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartVectorSearch1, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1 } from '/snippets/quickstart.mdx';
This quickstart uses a local database and works without an account or API key.
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
Follow the steps below, or jump to the [complete examples](#code) to copy a runnable program.
@@ -154,7 +155,7 @@ Use the database URI, API key, and endpoint provided by your administrator or th
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/connect#enterprise) for an example.
</Note>
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between Embedded and Distributed execution, see [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
</Accordion>
<Accordion title="Connect directly to object storage instead">
+1 -1
View File
@@ -8,7 +8,7 @@ availability:
oss: available
enterprise: available
summary: >-
Reranking is available with OSS and Enterprise.
Reranking is available with LanceDB Embedded and LanceDB Enterprise.
---
import {
+10 -9
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Query plans are available in OSS and Enterprise.
Query plans are available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyQueryPlan, TsQueryPlanBody, TsQueryPlanTail } from '/snippets/search_examples.mdx';
@@ -26,7 +26,7 @@ the query and adds measurements, including execution time and rows processed.
Use the first to inspect query structure and the second to investigate work
actually performed. Operator names and available measurements can differ
between OSS and Enterprise. SQL provides [EXPLAIN and EXPLAIN ANALYZE](/sql/dql/explain).
between LanceDB Embedded and LanceDB Enterprise. SQL provides [EXPLAIN and EXPLAIN ANALYZE](/sql/dql/explain).
<span id="try-both-methods-on-a-local-table"></span>
@@ -161,7 +161,7 @@ CoalesceBatchesExec: target_batch_size=1024
#### 5\. Data Retrieval {#5-data-retrieval}
- `RemoteTake` is the deployment's operator for this step; a local plan shows Lance's `Take` or a `LanceRead` over the selected row IDs, depending on the library version
- `RemoteTake` is the deployment's operator for this step; a LanceDB Embedded plan shows Lance's `Take` or a `LanceRead` over the selected row IDs, depending on the library version
- Handles efficient data retrieval from remote storage locations
- Fetches specific rows and columns needed for the final output
- Optimizes network bandwidth by only retrieving required data
@@ -328,25 +328,26 @@ ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier
</Accordion>
### Distributed metrics on remote tables {#distributed-metrics-on-remote-tables}
### Distributed metrics on LanceDB Enterprise {#distributed-metrics-on-remote-tables}
<Badge color="red">Enterprise</Badge>
When you call `analyze_plan` against a LanceDB Enterprise deployment that distributes its queries across a pool of workers, the query runs on those workers. By default the returned plan aggregates each operator's metrics into a single value, matching the local single-node output. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
When you call `analyze_plan` against a LanceDB Enterprise deployment that distributes its queries across a pool of workers, the query runs on those workers. By default the returned plan aggregates each operator's metrics into a single value, matching the single-node output of LanceDB Embedded. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
| Mode | What it shows | When to use it |
| :------------ | :---------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
| `aggregate` | One summary value per operator (default). Same shape as local `analyze_plan`. | Everyday performance checks and comparisons against a local plan. |
| `aggregate` | One summary value per operator (default). Same shape as Embedded `analyze_plan`. | Everyday performance checks and comparisons against an Embedded plan. |
| `per_worker` | Metrics reported separately for each worker that participated in the query. | Diagnosing stragglers or uneven work distribution across the cluster. |
| `full` | Both the aggregate summary and the per-worker breakdown. | Deep investigations where you need the totals *and* the per-worker detail in a single output. |
Python exposes this option through `distributed_metrics`. In TypeScript, pass
the same mode to `analyzePlan()` as its `distributedMetrics` argument. Local
Python queries ignore the distributed mode and return local measurements.
the same mode to `analyzePlan()` as its `distributedMetrics` argument. Python
queries on LanceDB Embedded ignore the distributed mode and return single-node
measurements.
<CodeGroup>
```python Python icon="python"
# Show a per-worker breakdown for a remote query
# Show a per-worker breakdown for a query on LanceDB Enterprise
plan = (
table.search(query_embed)
.where("identifier > 0 AND identifier < 1000000")
+1 -1
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Metadata filtering is available in OSS and Enterprise.
Metadata filtering is available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyFilterSearch, TsFilterSearchBody, TsFilterSearchTail } from '/snippets/search_examples.mdx';
+5 -5
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Full-text search is available in OSS and Enterprise.
Full-text search is available in LanceDB Embedded and LanceDB Enterprise.
---
import {
@@ -131,7 +131,7 @@ tbl
```
</CodeGroup>
OSS builds the index before the call returns. Enterprise builds it in the
LanceDB Embedded builds the index before the call returns. Enterprise builds it in the
background. Wait for completion with Python’s `table.wait_for_index(["text_idx"])`
or TypeScript’s `await tbl.waitForIndex(["text_idx"], 300)` before querying it.
Use your actual index name if you chose a different one.
@@ -187,7 +187,7 @@ values rather than probabilities.
### Keeping the index up to date {#keeping-the-index-up-to-date}
In OSS, new rows can remain outside the FTS index until you optimize the table.
In LanceDB Embedded, new rows can remain outside the FTS index until you optimize the table.
Search includes those rows by scanning them, but that work can increase query
latency. Call `optimize()` to update the index:
@@ -207,7 +207,7 @@ tbl.optimize(OptimizeAction::All).await?;
```
</CodeGroup>
Choose an OSS maintenance schedule based on write volume and query latency.
Choose a LanceDB Embedded maintenance schedule based on write volume and query latency.
Inspect index coverage with `table.index_stats(...)` and optimize when the
unindexed data adds too much work. `.fast_search()` skips unindexed rows, so
use it only when omitting recent records is acceptable.
@@ -289,7 +289,7 @@ export LANCE_LANGUAGE_MODEL_HOME=/path/to/lance/language_models
For example, `jieba/default` is resolved under `<model-home>/jieba/default/...`, `lindera/ipadic` under `<model-home>/lindera/ipadic/...`, and `lindera/ko-dic` under `<model-home>/lindera/ko-dic/...`.
Model files must be installed where indexing and queries run: on your machine
for OSS, or on the Enterprise cluster for a remote table.
for LanceDB Embedded, or on the Enterprise cluster for an Enterprise table.
<Note>
Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. If you use another stemming language, such as Arabic, Greek, Romanian, Tamil, or Turkish, set `remove_stop_words=False` or pass `custom_stop_words`.
+4 -4
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Hybrid search is available in OSS and Enterprise.
Hybrid search is available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
@@ -84,9 +84,9 @@ import { Utf8 } from "apache-arrow";
</CodeGroup>
### 2\. Connect to LanceDB {#2-connect-to-lancedb}
Choose a local OSS connection or use your Enterprise connection details.
Choose a LanceDB Embedded connection or use your Enterprise connection details.
<Badge color="green">OSS</Badge>
<Badge color="green">Embedded</Badge>
<CodeGroup>
```python Python icon="python"
@@ -389,7 +389,7 @@ which relevant rows reach the candidate set before evaluating how a reranker
orders them. Record latency as well as relevance: increasing candidate counts
or applying a model-based reranker adds work.
Reranking runs in the application for both OSS and Enterprise connections.
Reranking runs in the application for both Embedded and Enterprise connections.
See [Reranking](/reranking) for model-free fusion, model-based rerankers,
custom implementations and score interpretation.
+4 -4
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Vector, full-text and hybrid search are available in OSS and Enterprise.
Vector, full-text and hybrid search are available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyVectorSearch, TsVectorSearchBody, TsVectorSearchTail } from '/snippets/search_examples.mdx';
@@ -106,9 +106,9 @@ in [Data loading](/training/data-loading). The SDK buffers Enterprise
query responses before exposing batches. Set a query limit rather than relying
on batch iteration alone to bound memory.
Direct Lance dataset access through Python's `table.to_lance()` is an OSS
interface. Use the SDK query builders for reads that need to work with both
OSS and Enterprise.
Direct Lance dataset access through Python's `table.to_lance()` is a LanceDB
Embedded interface. Use the SDK query builders for reads that need to work with
both LanceDB Embedded and LanceDB Enterprise.
## Investigate a slow query {#diagnostics}
+3 -3
View File
@@ -7,7 +7,7 @@ availability:
oss: available
enterprise: available
summary: >-
Vector search is available in OSS and Enterprise.
Vector search is available in LanceDB Embedded and LanceDB Enterprise.
---
import {
@@ -131,7 +131,7 @@ Without a vector index, searches use `l2` by default. You can specify `cosine` o
When a vector index exists, the search metric must match the metric used to build
the index. If you explicitly request a different metric, LanceDB cannot use that
index and falls back to an exhaustive (brute-force) search. On LanceDB Cloud, the
index and falls back to an exhaustive (brute-force) search. On LanceDB Cloud (legacy hosted service), the
brute-force KNN safety limit can reject that search on a large table instead of
executing it. LanceDB Enterprise doesn't have the limit.
@@ -623,7 +623,7 @@ tbl.create_index(metric="cosine", vector_column_name="vector")
**Indexing matters more for multivector tables than for single-vector ones.**
A brute-force scan over a multivector column has to compare every query vector to every document vector in every row, so the cost grows with both the row count and the number of vectors per row.
In LanceDB OSS, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.
In LanceDB Embedded, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.
On LanceDB Cloud, the brute-force KNN safety check applies a stricter row threshold to multivector columns; roughly 10× lower than for single-vector columns. So an unindexed multivector table can be rejected with a "vector search would use brute-force KNN" error well before a comparable single-vector table would. LanceDB Enterprise doesn't have the limit. Build the index before you start hitting it from production traffic, even if the dataset is small enough that you'd skip indexing for a single-vector workload.
</Info>
@@ -1,6 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" width="800" height="676" viewBox="0 0 800 676" role="img" aria-labelledby="title desc">
<title id="title">LanceDB OSS and Enterprise architecture</title>
<desc id="desc">OSS runs the application and embedded library in one process with local disk or object storage. Enterprise separates query serving, read execution and caching, and indexing and background workers. The control plane governs the service. Enterprise persists tables in object storage. Both offerings use the Lance format, without automatically sharing storage.</desc>
<title id="title">LanceDB Embedded and Enterprise architecture</title>
<desc id="desc">LanceDB Embedded runs the application and embedded library in one process with local disk or object storage. Enterprise separates query serving, read execution and caching, and indexing and background workers. The control plane governs the service. Enterprise persists tables in object storage. Both offerings use the Lance format, without automatically sharing storage.</desc>
<defs>
<marker id="arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto" markerUnits="userSpaceOnUse"><path d="M1 1 7 4 1 7" fill="none" stroke="#596776" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"/></marker>
<style>
@@ -15,7 +15,7 @@
<rect width="800" height="676" rx="20" fill="#ffffff"/>
<rect x="12" y="12" width="380" height="580" rx="16" fill="#f3f6f9" stroke="#d4dce6"/>
<text class="title" x="36" y="53">LanceDB OSS</text>
<text class="title" x="36" y="53">LanceDB Embedded</text>
<text class="note" x="36" y="83">Embedded library</text>
<rect x="36" y="110" width="332" height="296" rx="10" fill="#ffffff" stroke="#c4cfdd"/>
<text class="label" x="202" y="164" text-anchor="middle">Application +</text>

Before

Width:  |  Height:  |  Size: 4.3 KiB

After

Width:  |  Height:  |  Size: 4.3 KiB

@@ -1,6 +1,6 @@
<svg width="2048" height="540" viewBox="0 0 2048 540" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="title desc">
<title id="title">LanceDB suite</title>
<desc id="desc">LanceDB OSS supports search. LanceDB Enterprise is a multimodal lakehouse for curation, feature engineering, search and retrieval, and training. Both are built on the Lance open lakehouse format for multimodal AI.</desc>
<desc id="desc">LanceDB Embedded supports search. LanceDB Enterprise is a multimodal lakehouse for curation, feature engineering, search and retrieval, and training. Both are built on the Lance open lakehouse format for multimodal AI.</desc>
<defs>
<linearGradient id="enterpriseGradient" x1="512" y1="90" x2="2020" y2="90" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#E9C9F6"/>
@@ -54,7 +54,7 @@
<circle cx="94" cy="126" r="9"/>
<circle cx="126" cy="126" r="9"/>
</g>
<text x="162" y="117" style="fill:#241712 !important;" font-family="Inter, Arial, sans-serif" font-size="44" font-weight="760">LanceDB OSS</text>
<text x="162" y="113" style="fill:#241712 !important;" font-family="Inter, Arial, sans-serif" font-size="34" font-weight="760">LanceDB Embedded</text>
<text x="260" y="291" text-anchor="middle" style="fill:#241712 !important;" font-family="Inter, Arial, sans-serif" font-size="42" font-weight="500">Search</text>
<g style="fill:#241712 !important;">

Before

Width:  |  Height:  |  Size: 5.8 KiB

After

Width:  |  Height:  |  Size: 5.9 KiB

+6 -6
View File
@@ -63,18 +63,18 @@ scoped to that branch. The [comparison table](#branches-vs-tags-vs-versions) at
the end of this page lays out when to reach for each.
<Note>
Branches are supported on local and namespace-backed tables in LanceDB OSS, as
well as on LanceDB Enterprise (remote) tables.
Branches are supported on local and namespace-backed tables in LanceDB Embedded,
as well as on LanceDB Enterprise tables.
</Note>
## Connect to a table {#connect-to-a-table}
The branch API is identical no matter how you connect: only the connection
itself differs between OSS and Enterprise. Establish a connection (`db`) and open
itself differs between LanceDB Embedded and LanceDB Enterprise. Establish a connection (`db`) and open
a `table` (see [Ingesting data](/ingestion)), then use the same branch calls
in every example that follows.
### LanceDB OSS {#lancedb-oss}
### LanceDB Embedded {#lancedb-oss}
Point LanceDB at a local directory (or an object-storage URI) to use it as an
embedded library.
@@ -99,7 +99,7 @@ embedded library.
<Badge color="red">Enterprise</Badge>
Branching on LanceDB Enterprise works the same way as on OSS, apart from the differences
Branching on LanceDB Enterprise works the same way as on LanceDB Embedded, apart from the differences
noted below, but you connect to your
Enterprise deployment with a `db://` URI, an API key, and your
region and endpoint. Replace the placeholders below, including `us-east-1` if your
@@ -282,7 +282,7 @@ inspects what a branch has changed relative to `main`, and `cherry_pick`
place, without reissuing the branch's writes.
<Info>
`diff` and `cherry_pick` work on Enterprise (remote) tables only. On a local
`diff` and `cherry_pick` work on LanceDB Enterprise tables only. On an Embedded
table they fail as not supported: Python raises `NotImplementedError`, and
TypeScript and Rust return an error. A cherry-pick promotes added columns; use
the [upsert](#upsert-selected-branch-rows-into-main) or rerun patterns above for
+8 -6
View File
@@ -7,8 +7,9 @@ availability:
oss: available
enterprise: available
summary: >-
Configure read freshness in OSS and Enterprise. OSS checks for table updates;
Enterprise uses the connection setting to tighten its configured freshness bound.
Configure read freshness in LanceDB Embedded and LanceDB Enterprise. Embedded
checks for table updates; Enterprise uses the connection setting to tighten its
configured freshness bound.
---
import { PyReadConsistency } from '/snippets/read_consistency.mdx';
@@ -26,11 +27,12 @@ for the experiment.
## Visibility of writes {#visibility}
In **OSS**, a table handle sees its own committed writes. By default, it does
In **LanceDB Embedded**, a table handle sees its own committed writes. By default, it does
not automatically check for changes made through other handles or processes.
Use a read consistency interval or `checkout_latest()` to pick up those changes.
In **Enterprise**, readers use the service's configured freshness policy.
In **LanceDB Distributed** (provided by LanceDB Enterprise), readers use the
service's configured freshness policy.
A connection can request fresher data. When a read must include a completed
write, use a zero interval or call `checkout_latest()` after the write, with a
client and server that support these freshness controls.
@@ -40,13 +42,13 @@ client and server that support these freshness controls.
Set `read_consistency_interval` when connecting. This controls when reads
pick up newer table versions; it does not change how writes commit.
| Setting | OSS | Enterprise |
| Setting | Embedded | Distributed |
| --- | --- | --- |
| **Unset** | Keep the open table version until explicitly refreshed or changed by that handle. | Use the service's configured freshness policy. |
| **Zero** | Check for the latest committed version on every read. | Request the latest committed version on every read. |
| **Non-zero interval** | Check for updates on a read once the interval since the last check has elapsed. | Request data no older than the interval allows; the service may return fresher data. |
`checkout_latest()` explicitly refreshes an OSS table handle. On Enterprise it
`checkout_latest()` explicitly refreshes an Embedded table handle. On Distributed it
asks subsequent reads on that handle to reflect data at least as recent as the
refresh request. To read a fixed historical version instead, use
[table versioning](/management/table#versioning).
+1 -1
View File
@@ -59,7 +59,7 @@ for batch in dataloader:
The `StreamingDataset` wraps a LanceDB `Table` and yields plain Python dictionaries by default. Whether iterating directly or through a PyTorch `DataLoader`, only one iterator can be simultaneously active on a `StreamingDataset` instance.
<Note>
`StreamingDataset` is built on the `permutation` API and works with both LanceDB OSS and LanceDB Enterprise. The underlying table data can live on local disk or object storage.
`StreamingDataset` is built on the `permutation` API and works with both LanceDB Embedded and LanceDB Enterprise. The underlying table data can live on local disk or object storage.
</Note>
To understand the wide range of scenarios handled by `StreamingDataset`, consider a model trained with stochastic gradient descent (SGD) and distributed data parallelism (DDP). In this example, we need to load batches onto multiple GPUs
+3 -3
View File
@@ -238,7 +238,7 @@ described above.
## Using multiple DataLoader workers {#using-multiple-dataloader-workers}
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables in LanceDB Enterprise are picklable, so each worker reopens the table after it starts. A local table is not, so wrap it in a `Permutation`, as below, to read it from several workers.
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables in LanceDB Enterprise are picklable, so each worker reopens the table after it starts. A LanceDB Embedded table is not, so wrap it in a `Permutation`, as below, to read it from several workers.
Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and became the default on Linux and other supported POSIX platforms in Python
3.14. macOS and Windows default to `spawn`. See
@@ -263,9 +263,9 @@ dataloader = torch.utils.data.DataLoader(
)
```
### Remote tables in DataLoader workers {#remote-tables-in-dataloader-workers}
### Enterprise tables in DataLoader workers {#remote-tables-in-dataloader-workers}
Remote LanceDB Enterprise tables (`db://...`) work the same way: workers reopen the table from the pickled connection state. Replace the URI, API key, region and endpoint below with your deployment details. In application code, load credentials from environment variables or a secrets manager.
LanceDB Enterprise tables (`db://...`) work the same way: workers reopen the table from the pickled connection state. Replace the URI, API key, region and endpoint below with your deployment details. In application code, load credentials from environment variables or a secrets manager.
```py Python icon=Python
import lancedb
@@ -6,7 +6,7 @@ description: "Use LanceDB as the retrieval layer for NVIDIA RAG Blueprint with a
## What this tutorial shows {#what-this-tutorial-shows}
If you are using [NVIDIA RAG Blueprints](https://build.nvidia.com/blueprints) and want to evaluate LanceDB in that stack, this tutorial gives you a concrete starting point. It shows how to use LanceDB as the retrieval layer for a Docker-based NVIDIA RAG deployment with a small, script-driven reference integration where LanceDB OSS is embedded directly in the NVIDIA containers, the collection is prepared ahead of time, and the RAG server retrieves from it for search and generation. The example is intentionally retrieval-only, but it also includes hybrid search and reranker selection so you can see how LanceDB fits into a realistic NVIDIA retrieval workflow.
If you are using [NVIDIA RAG Blueprints](https://build.nvidia.com/blueprints) and want to evaluate LanceDB in that stack, this tutorial gives you a concrete starting point. It shows how to use LanceDB as the retrieval layer for a Docker-based NVIDIA RAG deployment with a small, script-driven reference integration where LanceDB Embedded runs directly in the NVIDIA containers, the collection is prepared ahead of time, and the RAG server retrieves from it for search and generation. The example is intentionally retrieval-only, but it also includes hybrid search and reranker selection so you can see how LanceDB fits into a realistic NVIDIA retrieval workflow.
<Tip>
The runnable example for this tutorial lives in the
@@ -31,7 +31,7 @@ yet teach NVIDIA's ingestor how to write new documents into LanceDB automaticall
## Deployment model {#deployment-model}
This reference integration uses **LanceDB OSS as an embedded retrieval library**, not as a separate
This reference integration uses **LanceDB Embedded as an embedded retrieval library**, not as a separate
database service. In practice, `APP_VECTORSTORE_NAME` is set to `lancedb`, `APP_VECTORSTORE_URL`
points to a local filesystem path inside the NVIDIA containers, the LanceDB collection is prepared
ahead of time, and the NVIDIA RAG server loads the LanceDB adapter to retrieve directly from that
+3 -3
View File
@@ -2,9 +2,9 @@
## Configuration and Initialization
### LanceDB Cloud
### LanceDB Cloud (legacy)
For LanceDB Cloud, use the simplified builder API:
For LanceDB Cloud, the legacy hosted service, use the simplified builder API:
```java
import com.lancedb.LanceDbNamespaceClientBuilder;
@@ -19,7 +19,7 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
### LanceDB Enterprise
For Enterprise deployments, use your custom endpoint:
For LanceDB Enterprise deployments, use your custom endpoint:
```java
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
@@ -22,7 +22,7 @@ import java.util.Optional;
/**
* Util class to help construct a {@link LanceNamespace} for LanceDB.
*
* <p>For LanceDB Cloud, use the simplified builder API:
* <p>For LanceDB Cloud (legacy hosted service), use the simplified builder API:
*
* <pre>{@code
* import org.lance.namespace.LanceNamespace;
@@ -27,7 +27,7 @@ import java.io.IOException;
import java.io.UncheckedIOException;
/**
* Minimal HTTP client for LanceDB Cloud and Enterprise routes that the Lance Namespace
* Minimal HTTP client for LanceDB Enterprise routes that the Lance Namespace
* specification does not cover.
*
* <p>Most table operations reach LanceDB through {@link org.lance.namespace.LanceNamespace}, which
@@ -22,7 +22,7 @@ import java.util.Optional;
import java.util.OptionalLong;
/**
* The RTE write path for one LanceDB Cloud or Enterprise table.
* The RTE write path for one LanceDB Enterprise table.
*
* <p>Installing an {@link RTEWriteSpec} routes {@code mergeInsert} upserts through Lance's RTE — an
* LSM-style append — instead of the standard merge path. Rows land in an in-memory memtable, freeze
+7 -5
View File
@@ -115,14 +115,14 @@ describe("remote connection", () => {
...options,
mode: "overwrite",
}),
).rejects.toThrow(/write_options.*remote/);
).rejects.toThrow(/write_options.*LanceDB Distributed/);
await expect(
db.createEmptyTable(
"test",
new Schema([new Field("value", new Int64(), true)]),
options,
),
).rejects.toThrow(/write_options.*remote/);
).rejects.toThrow(/write_options.*LanceDB Distributed/);
},
);
expect(requests).toEqual([]);
@@ -140,7 +140,9 @@ describe("remote connection", () => {
async (db) => {
await expect(
db.openTable("test", undefined, options),
).rejects.toThrow(/not supported for remote/);
).rejects.toThrow(
/not supported for table access in LanceDB Distributed/,
);
},
);
expect(requests).toEqual([]);
@@ -316,7 +318,7 @@ describe("remote connection", () => {
],
];
for (const [run, operation] of cases) {
let message = `LanceDBError: not supported: ${operation} is not supported for remote tables.`;
let message = `LanceDBError: not supported: ${operation} is not supported in LanceDB Distributed.`;
if (operation === "IVF_HNSW_PQ") {
message += " Please use IVF_HNSW_SQ instead.";
}
@@ -463,7 +465,7 @@ describe("remote connection", () => {
table.add([{ id: 1n }], {
allowExternalBlobOutsideBases: true,
}),
).rejects.toThrow("only supported on local tables");
).rejects.toThrow("only supported in LanceDB Embedded");
expect(insertRequests).toBe(0);
await expect(table.add([{ id: 2n }])).resolves.toMatchObject({
+4 -4
View File
@@ -4526,14 +4526,14 @@ describe.each(arrowVersions)(
];
const table = await db.createTable("prewarm_data_test", data);
// prewarmData is only supported on remote tables. We verify the call
// prewarmData is only supported in LanceDB Distributed. We verify the call
// is wired through napi and surfaces the expected error for both
// arg shapes (undefined and string[]).
await expect(table.prewarmData()).rejects.toThrow(
"prewarm_data is currently only supported on remote tables",
"prewarm_data is currently only supported in LanceDB Distributed",
);
await expect(table.prewarmData(["text"])).rejects.toThrow(
"prewarm_data is currently only supported on remote tables",
"prewarm_data is currently only supported in LanceDB Distributed",
);
});
@@ -5506,7 +5506,7 @@ describe("computed columns", () => {
const db = await connect(tmpDir.name);
const table = await db.createTable("errors_local", [{ x: 1 }]);
await expect(table.functionErrors()).rejects.toThrow(
"LanceDB Cloud and Enterprise",
"recorded only in LanceDB Distributed",
);
});
+1 -1
View File
@@ -274,7 +274,7 @@ export class BlobFile {
export type BlobReadOptions = {
/**
* Cancels the read. The call rejects with `signal.reason`, and the native
* read stops, including in-flight requests to a remote table.
* read stops, including in-flight requests to a LanceDB Distributed table.
*/
signal?: AbortSignal;
};

Some files were not shown because too many files have changed in this diff Show More