mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-30 15:26:52 +00:00
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
b4cbaf2f67
commit
4d7a64712f
@@ -8,7 +8,7 @@
|
||||
{
|
||||
"name": "lancedb",
|
||||
"source": "./plugins/lancedb",
|
||||
"description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables.",
|
||||
"description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work with both LanceDB Embedded and LanceDB Enterprise.",
|
||||
"version": "0.1.0",
|
||||
"author": {
|
||||
"name": "LanceDB"
|
||||
|
||||
+10
-2
@@ -1,8 +1,16 @@
|
||||
# LanceDB prose rules
|
||||
|
||||
These five rule files are copied unchanged from the `LanceDB.zip` asset of
|
||||
`lancedb/docs` release `vale-lancedb-v0.1.0`. The archive SHA-256 is
|
||||
`Dashes`, `PlainLanguage`, `ProductNames`, `VagueHeadings` and `VaguePhrases`
|
||||
come from the `LanceDB.zip` asset of `lancedb/docs` release
|
||||
`vale-lancedb-v0.1.0`, whose archive SHA-256 is
|
||||
`0730a43e919774121068811d5687c61dfc7a0421cff78e1393f5e565b622cb0f`.
|
||||
`ProductNames` no longer swaps in "LanceDB OSS".
|
||||
|
||||
The product-terminology rules were added here: `EmbeddedName`,
|
||||
`RemoteExecution`, `OSSProduct`, `DistributedEnterprise`, `InternalNames`
|
||||
(warnings) and `LegacyCloud` (suggestion). They enforce the names in the
|
||||
docs repository's `STYLE.md#names`. Run every rule's tests from
|
||||
`.github/vale` with `vale test --coverage styles`.
|
||||
|
||||
Keeping the rules here lets contributors and CI run Vale without access to the
|
||||
source repository. The root `.vale.ini` preserves the release's MDX settings
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
extends: existence
|
||||
message: "'%s' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both."
|
||||
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
|
||||
level: warning
|
||||
ignorecase: false
|
||||
tokens:
|
||||
- 'Distributed(?:\s+(?:and|or)\s+|\s*/\s*)(?:LanceDB\s+)?Enterprise'
|
||||
- 'Enterprise(?:\s+(?:and|or)\s+|\s*/\s*)(?:LanceDB\s+)?Distributed'
|
||||
tests:
|
||||
- name: flags Distributed and Enterprise side by side
|
||||
input: Choose Distributed or Enterprise. LanceDB Enterprise and LanceDB Distributed differ, and so does Distributed/Enterprise.
|
||||
want: |
|
||||
1:8:LanceDB.DistributedEnterprise:'Distributed or Enterprise' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
|
||||
1:43:LanceDB.DistributedEnterprise:'Enterprise and LanceDB Distributed' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
|
||||
1:98:LanceDB.DistributedEnterprise:'Distributed/Enterprise' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
|
||||
- name: leaves the defining sentence and the Embedded pair alone
|
||||
input: |
|
||||
LanceDB Enterprise runs LanceDB Distributed. Compare Embedded and Distributed execution. Distributed search and enterprise teams are fine.
|
||||
want: ''
|
||||
@@ -0,0 +1,22 @@
|
||||
extends: substitution
|
||||
message: "Use '%s' instead of '%s'."
|
||||
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
|
||||
level: warning
|
||||
ignorecase: false
|
||||
# The in-process execution model is LanceDB Embedded, whether the data is on
|
||||
# local disk or in object storage. Identifiers such as the `oss` product key
|
||||
# are code, and code is never prose.
|
||||
swap:
|
||||
'LanceDB\s+(?:OSS|Oss|oss|Local|local)': LanceDB Embedded
|
||||
'[Ll]ocal\s+mode': LanceDB Embedded
|
||||
tests:
|
||||
- name: replaces the old names of the execution model
|
||||
input: Start with LanceDB OSS, also called LanceDB Local, and move off local mode later.
|
||||
want: |
|
||||
1:12:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'LanceDB OSS'.
|
||||
1:37:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'LanceDB Local'.
|
||||
1:65:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'local mode'.
|
||||
- name: leaves the current name, other uses of local, and code alone
|
||||
input: |
|
||||
LanceDB Embedded reads local disk or object storage. Run LanceDB locally, set `availability.oss`, or call `LanceDB OSS` in a literal.
|
||||
want: ''
|
||||
@@ -0,0 +1,25 @@
|
||||
extends: substitution
|
||||
message: "Use '%s' instead of the internal name '%s'."
|
||||
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
|
||||
level: warning
|
||||
ignorecase: false
|
||||
# Internal component names stay out of public text. Environment variables such
|
||||
# as `PHALANX_*` and Helm keys such as `planExecutorNodes` are code. Geneva is
|
||||
# a product name and is not listed.
|
||||
swap:
|
||||
Phalanx: query node
|
||||
'[Pp]lan\s+[Ee]xecutor': executor
|
||||
'[Pp]lan\s+[Ee]xecutors': executors
|
||||
Sophon: LanceDB Enterprise
|
||||
tests:
|
||||
- name: replaces internal component names
|
||||
input: Phalanx sends work to a Plan Executor. Plan executors cache data in Sophon.
|
||||
want: |
|
||||
1:1:LanceDB.InternalNames:Use 'query node' instead of the internal name 'Phalanx'.
|
||||
1:25:LanceDB.InternalNames:Use 'executor' instead of the internal name 'Plan Executor'.
|
||||
1:40:LanceDB.InternalNames:Use 'executors' instead of the internal name 'Plan executors'.
|
||||
1:69:LanceDB.InternalNames:Use 'LanceDB Enterprise' instead of the internal name 'Sophon'.
|
||||
- name: leaves Geneva, configuration keys and code alone
|
||||
input: |
|
||||
Geneva runs on the executor + cache fleet. Set `PHALANX_REMOTE_FTS_SCORER` and `planExecutorNodes`.
|
||||
want: ''
|
||||
@@ -0,0 +1,21 @@
|
||||
extends: existence
|
||||
message: "'%s' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'."
|
||||
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
|
||||
level: suggestion
|
||||
ignorecase: false
|
||||
# A suggestion, because `db://` URIs, `api_key` and `region` for the hosted
|
||||
# service are still documented as LanceDB Cloud.
|
||||
tokens:
|
||||
- 'LanceDB\s+Cloud'
|
||||
- 'Cloud\s*/\s*Enterprise'
|
||||
- 'Enterprise\s*/\s*Cloud'
|
||||
tests:
|
||||
- name: suggests checking each LanceDB Cloud mention
|
||||
input: Connect to LanceDB Cloud with an API key. Cloud/Enterprise users get the cache.
|
||||
want: |
|
||||
1:12:LanceDB.LegacyCloud:'LanceDB Cloud' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'.
|
||||
1:43:LanceDB.LegacyCloud:'Cloud/Enterprise' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'.
|
||||
- name: leaves other clouds and code alone
|
||||
input: |
|
||||
Deploy into your own cloud account on Google Cloud. Pass `LanceDB Cloud` as a literal.
|
||||
want: ''
|
||||
@@ -0,0 +1,28 @@
|
||||
extends: existence
|
||||
message: "Avoid '%s'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property."
|
||||
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
|
||||
level: warning
|
||||
ignorecase: false
|
||||
# Bare OSS used as a product, as in "OSS and Enterprise" or "in OSS". "LanceDB
|
||||
# OSS" is reported by LanceDB.EmbeddedName.
|
||||
tokens:
|
||||
- 'OSS(?:\s+(?:and|or|vs\.?|versus)\s+|\s*/\s*)(?:LanceDB\s+)?Enterprise'
|
||||
- 'Enterprise(?:\s+(?:and|or|vs\.?|versus)\s+|\s*/\s*)OSS'
|
||||
- '(?:in|on|from|using|than)\s+OSS'
|
||||
tests:
|
||||
- name: flags OSS presented as a product
|
||||
input: Compare OSS vs Enterprise. It works in OSS and on OSS, unlike OSS/Enterprise.
|
||||
want: |
|
||||
1:9:LanceDB.OSSProduct:Avoid 'OSS vs Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
|
||||
1:37:LanceDB.OSSProduct:Avoid 'in OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
|
||||
1:48:LanceDB.OSSProduct:Avoid 'on OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
|
||||
1:63:LanceDB.OSSProduct:Avoid 'OSS/Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
|
||||
- name: flags the pair in either order
|
||||
input: Both OSS and LanceDB Enterprise ship it, as do Enterprise or OSS.
|
||||
want: |
|
||||
1:6:LanceDB.OSSProduct:Avoid 'OSS and LanceDB Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
|
||||
1:48:LanceDB.OSSProduct:Avoid 'Enterprise or OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
|
||||
- name: leaves open source, identifiers and code alone
|
||||
input: |
|
||||
LanceDB Embedded (open source) and LanceDB Enterprise share the format. Set `oss: available` or read `in OSS` literally.
|
||||
want: ''
|
||||
@@ -9,7 +9,6 @@ swap:
|
||||
'(?:Lancedb|LanceDb|lanceDB|LANCEDB|Lance DB)': LanceDB
|
||||
'LanceDB (?:cloud|CLOUD)': LanceDB Cloud
|
||||
'LanceDB (?:enterprise|ENTERPRISE)': LanceDB Enterprise
|
||||
'LanceDB (?:oss|Oss)': LanceDB OSS
|
||||
'(?:HuggingFace|Huggingface|Hugging face)': Hugging Face
|
||||
Langchain: LangChain
|
||||
Typescript: TypeScript
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
extends: existence
|
||||
message: "Avoid '%s'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product."
|
||||
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
|
||||
level: warning
|
||||
ignorecase: false
|
||||
# Remote storage, remote URLs and a git remote are other things and stay. Class
|
||||
# names such as `RemoteTable` are code, and code is never prose.
|
||||
tokens:
|
||||
- '[Rr]emote\s+(?:mode|tables?|connections?|databases?)'
|
||||
tests:
|
||||
- name: flags remote as the execution model
|
||||
input: In remote mode, a remote table is opened through a remote connection.
|
||||
want: |
|
||||
1:4:LanceDB.RemoteExecution:Avoid 'remote mode'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
|
||||
1:19:LanceDB.RemoteExecution:Avoid 'remote table'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
|
||||
1:52:LanceDB.RemoteExecution:Avoid 'remote connection'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
|
||||
- name: flags plurals and a heading
|
||||
input: |
|
||||
## Remote tables
|
||||
|
||||
Remote databases hold remote tables.
|
||||
want: |
|
||||
1:4:LanceDB.RemoteExecution:Avoid 'Remote tables'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
|
||||
3:1:LanceDB.RemoteExecution:Avoid 'Remote databases'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
|
||||
3:23:LanceDB.RemoteExecution:Avoid 'remote tables'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
|
||||
- name: leaves other remotes and code alone
|
||||
input: |
|
||||
Push to the git remote, read remote storage and fetch a remote URL. `open_table` returns a `RemoteTable`; see `remote table` in the API.
|
||||
|
||||
```python
|
||||
remote_table = db.open_table("remote table")
|
||||
```
|
||||
want: ''
|
||||
+8
-7
@@ -72,7 +72,7 @@ are wrappers around the Rust library.
|
||||
* `node`: **Deprecated** Typescript package source code
|
||||
* `java`: Java package source code
|
||||
* `docs/src`: SDK reference, built with mkdocs
|
||||
* `docs/web`: website guides for LanceDB OSS and the APIs used by both OSS and Enterprise
|
||||
* `docs/web`: website guides for LanceDB Embedded and the APIs used by both Embedded and Distributed
|
||||
* `docs/web-tests`: the tests the documentation's code examples are extracted from
|
||||
|
||||
## Documentation
|
||||
@@ -83,14 +83,14 @@ it is ready for readers.
|
||||
|
||||
### Choose the right place
|
||||
|
||||
- `docs/web/` holds website guides for LanceDB OSS and the APIs used by both
|
||||
OSS and Enterprise.
|
||||
- `docs/web/` holds website guides for LanceDB Embedded and the APIs used by both
|
||||
Embedded and Distributed.
|
||||
- `docs/src/` holds the SDK reference. See [docs/README.md](docs/README.md) for
|
||||
its build and API-generation instructions.
|
||||
- `docs/web-tests/` and `rust/lancedb/tests/` hold tests used to generate the
|
||||
website's code examples.
|
||||
|
||||
Document SDK APIs here, and explain which operations work in OSS and Enterprise.
|
||||
Document SDK APIs here, and explain which operations work in Embedded and Distributed.
|
||||
Put Enterprise-specific deployment and operating instructions in the Enterprise
|
||||
documentation.
|
||||
|
||||
@@ -150,7 +150,8 @@ example.
|
||||
### Say where a feature works
|
||||
|
||||
Use `availability` in a page's frontmatter to describe the support covered by
|
||||
that page. Each product is either `available` or `unavailable`; use the summary
|
||||
that page. Each product (`oss` for LanceDB Embedded, `enterprise` for LanceDB
|
||||
Enterprise) is either `available` or `unavailable`; use the summary
|
||||
and body for setup requirements, SDK differences and unsupported operations.
|
||||
For example, a page about signing in to Enterprise could declare:
|
||||
|
||||
@@ -159,8 +160,8 @@ availability:
|
||||
oss: unavailable
|
||||
enterprise: available
|
||||
summary: >-
|
||||
These sign-in instructions apply to LanceDB Enterprise. Embedded
|
||||
LanceDB has no server to sign in to.
|
||||
These sign-in instructions apply to LanceDB Enterprise. LanceDB
|
||||
Embedded has no server to sign in to.
|
||||
```
|
||||
|
||||
Check these claims with the feature owner. Don't use `varies`, treat an API
|
||||
|
||||
@@ -45,8 +45,8 @@ LanceDB is a central location where developers can build, train and analyze thei
|
||||
- **Advanced Features**: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
|
||||
|
||||
### **Products**:
|
||||
- **Open Source & Local**: 100% open source, runs locally or in your cloud. No vendor lock-in.
|
||||
- **Cloud and Enterprise**: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
|
||||
- **LanceDB Embedded (open source)**: 100% open source, runs in your application's process; your data can live on local disk or in object storage. No vendor lock-in.
|
||||
- **LanceDB Enterprise**: Production-scale distributed search, indexing, and maintenance. Plus, complete data sovereignty and security.
|
||||
|
||||
### **Ecosystem**:
|
||||
- **Columnar Storage**: Built on the Lance columnar format for efficient storage and analytics.
|
||||
|
||||
+3
-3
@@ -1,9 +1,9 @@
|
||||
openapi: 3.1.0
|
||||
info:
|
||||
version: 1.0.0
|
||||
title: LanceDB Cloud API
|
||||
title: LanceDB REST API
|
||||
description: |
|
||||
LanceDB Cloud API is a RESTful API that allows users to access and modify data stored in LanceDB Cloud.
|
||||
The LanceDB REST API is a RESTful API that allows users to access and modify data stored in LanceDB Enterprise or LanceDB Cloud (legacy).
|
||||
Table actions are considered temporary resource creations and all use POST method.
|
||||
contact:
|
||||
name: LanceDB support
|
||||
@@ -12,7 +12,7 @@ info:
|
||||
|
||||
servers:
|
||||
- url: https://{db}.{region}.api.lancedb.com
|
||||
description: LanceDB Cloud REST endpoint.
|
||||
description: LanceDB Cloud (legacy) REST endpoint. For LanceDB Enterprise, use your deployment's endpoint.
|
||||
variables:
|
||||
db:
|
||||
default: ""
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Java SDK
|
||||
|
||||
The LanceDB Java SDK provides a convenient way to interact with LanceDB Cloud and Enterprise deployments using the Lance REST Namespace API.
|
||||
The LanceDB Java SDK provides a convenient way to interact with LanceDB Enterprise deployments using the Lance REST Namespace API.
|
||||
|
||||
!!! note
|
||||
The Java SDK currently only works for LanceDB remote database that connects to LanceDB Cloud and Enterprise.
|
||||
Local database support is a work in progress. Check [LANCEDB-2848](https://github.com/lancedb/lancedb/issues/2848) for the latest progress.
|
||||
The Java SDK currently only works with LanceDB Distributed, connecting to LanceDB Enterprise or LanceDB Cloud (legacy).
|
||||
LanceDB Embedded support is a work in progress. Check [LANCEDB-2848](https://github.com/lancedb/lancedb/issues/2848) for the latest progress.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -20,7 +20,7 @@ Add the following dependency to your `pom.xml`:
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Connecting to LanceDB Cloud
|
||||
### Connecting to LanceDB Cloud (legacy)
|
||||
|
||||
```java
|
||||
import com.lancedb.LanceDbNamespaceClientBuilder;
|
||||
@@ -51,8 +51,8 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
|--------|-------------|----------|
|
||||
| `apiKey(String)` | LanceDB API key | Yes |
|
||||
| `database(String)` | Database name | Yes |
|
||||
| `endpoint(String)` | Custom endpoint URL for Enterprise deployments | No |
|
||||
| `region(String)` | AWS region (default: "us-east-1") | No |
|
||||
| `endpoint(String)` | Custom endpoint URL for LanceDB Enterprise deployments | No |
|
||||
| `region(String)` | LanceDB Cloud (legacy) region (default: "us-east-1") | No |
|
||||
| `config(String, String)` | Additional configuration parameters | No |
|
||||
|
||||
### Opening a Table with Vended Credentials
|
||||
|
||||
@@ -180,9 +180,9 @@ abstract createMaterializedView(
|
||||
|
||||
Define a materialized view named `name` over the table `source`.
|
||||
|
||||
Local creation and refresh require a source build with the
|
||||
LanceDB Embedded creation and refresh require a source build with the
|
||||
`experimental-functions` Cargo feature, disabled in published packages.
|
||||
Remote Cloud/Enterprise operations do not require this feature.
|
||||
LanceDB Distributed operations do not require this feature.
|
||||
|
||||
The view is populated before creation returns. Set `withNoData` to create
|
||||
only its definition and empty backing table. The view is a normal table:
|
||||
@@ -827,7 +827,7 @@ abstract renameTable(
|
||||
|
||||
Rename a table.
|
||||
|
||||
Currently only supported by LanceDB Cloud. Local OSS connections and
|
||||
Currently only supported in LanceDB Distributed. LanceDB Embedded connections and
|
||||
namespace-backed connections (via [connectNamespace](../functions/connectNamespace.md)) reject with
|
||||
a "not supported" error.
|
||||
|
||||
|
||||
@@ -69,9 +69,9 @@ abstract addColumns(newColumnTransforms): Promise<AddColumnsResult>
|
||||
|
||||
Add new columns with defined values.
|
||||
|
||||
Local computed-column declaration and refresh require a source build with
|
||||
the `experimental-functions` Cargo feature. Published packages disable it;
|
||||
remote Cloud/Enterprise operations remain available.
|
||||
LanceDB Embedded computed-column declaration and refresh require a source
|
||||
build with the `experimental-functions` Cargo feature. Published packages
|
||||
disable it; LanceDB Distributed operations remain available.
|
||||
|
||||
The `{ computed }` form stores the expression rather than evaluating it
|
||||
now: the column is committed with no values, and rows get them from
|
||||
@@ -83,7 +83,7 @@ computed, so a mutated input is reflected by the next refresh. While a
|
||||
declaration reads a column, that column cannot be renamed, retyped or
|
||||
dropped.
|
||||
|
||||
On LanceDB Cloud and Enterprise the expression is planned by the
|
||||
In LanceDB Distributed the expression is planned by the
|
||||
server, and the refresh runs as a server job -- see
|
||||
[Table#refreshColumnAsync](Table.md#refreshcolumnasync).
|
||||
|
||||
@@ -679,7 +679,7 @@ The per-row errors Function refreshes recorded on this table.
|
||||
A refresh running under a skip policy records each row it skipped with
|
||||
the input that failed and the error. This lists those records, newest
|
||||
job first, plus a summary for any fragment whose per-row detail was
|
||||
capped. LanceDB Cloud and Enterprise only; reading errors needs read
|
||||
capped. LanceDB Distributed only; reading errors needs read
|
||||
access to the table, since a message carries the value that failed.
|
||||
|
||||
#### Parameters
|
||||
@@ -932,7 +932,7 @@ Prewarm one or more columns of data in the table.
|
||||
Prewarming is generally useful for columns used in filters or projections.
|
||||
Large columns (e.g. high-dimensional vectors or binary data) may not be
|
||||
practical to prewarm.
|
||||
This feature is currently only supported on remote tables.
|
||||
This feature is currently only supported in LanceDB Distributed.
|
||||
|
||||
#### Returns
|
||||
|
||||
@@ -1043,8 +1043,8 @@ Local refresh requires a source build with `experimental-functions`.
|
||||
|
||||
Rows appended since the last refresh are filled by the next one, and
|
||||
rows whose inputs changed since they were computed are recomputed;
|
||||
everything else is left as it is. Local tables only: a remote refresh
|
||||
runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
|
||||
everything else is left as it is. LanceDB Embedded only: in LanceDB
|
||||
Distributed a refresh runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
|
||||
|
||||
#### Parameters
|
||||
|
||||
@@ -1068,14 +1068,15 @@ abstract refreshColumnAsync(column): Promise<Job>
|
||||
|
||||
Like [Table#refreshColumn](Table.md#refreshcolumn), but returns a handle to the refresh
|
||||
job instead of blocking until it completes.
|
||||
Local refresh requires a source build with `experimental-functions`.
|
||||
Remote Cloud/Enterprise jobs do not require this feature.
|
||||
LanceDB Embedded refresh requires a source build with
|
||||
`experimental-functions`. LanceDB Distributed jobs do not require this
|
||||
feature.
|
||||
|
||||
The job may already be complete when returned; callers must not assume
|
||||
the column is filled until [Job.wait](Job.md#wait) resolves. Invalid input --
|
||||
an unknown column, or one that is not computed -- rejects here rather
|
||||
than failing the job. On local tables the job runs in-process; on
|
||||
LanceDB Cloud and Enterprise it is the server's backfill job.
|
||||
than failing the job. In LanceDB Embedded the job runs in-process; in
|
||||
LanceDB Distributed it is the server's backfill job.
|
||||
|
||||
#### Parameters
|
||||
|
||||
@@ -1304,7 +1305,7 @@ Tokenize a full-text search query using the tokenizer configured on an FTS index
|
||||
Specify exactly one of `column` or `indexName`.
|
||||
|
||||
Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
|
||||
the client process from index metadata. For remote tables, this means the
|
||||
the client process from index metadata. In LanceDB Distributed, this means the
|
||||
same tokenizer model files must also exist locally (under
|
||||
`LANCE_LANGUAGE_MODEL_HOME`) and must match the files the server uses. A
|
||||
mismatch cannot be detected: the returned tokens may differ from what the
|
||||
|
||||
@@ -20,15 +20,15 @@ Connect to a LanceDB instance at the given URI.
|
||||
|
||||
Accepted formats:
|
||||
|
||||
- `/path/to/database` - local database
|
||||
- `/path/to/database` - database on the local file system
|
||||
- `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
|
||||
- `db://host:port` - remote database (LanceDB cloud)
|
||||
- `db://host:port` - LanceDB Enterprise or LanceDB Cloud (legacy) database
|
||||
|
||||
### Parameters
|
||||
|
||||
* **uri**: `string`
|
||||
The uri of the database. If the database uri starts
|
||||
with `db://` then it connects to a remote database.
|
||||
with `db://` then it connects to LanceDB Enterprise or LanceDB Cloud (legacy).
|
||||
|
||||
* **options?**: `Partial`<[`ConnectionOptions`](../interfaces/ConnectionOptions.md)>
|
||||
The options to use when connecting to the database
|
||||
@@ -80,9 +80,9 @@ Connect to a LanceDB instance at the given URI.
|
||||
|
||||
Accepted formats:
|
||||
|
||||
- `/path/to/database` - local database
|
||||
- `/path/to/database` - database on the local file system
|
||||
- `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
|
||||
- `db://host:port` - remote database (LanceDB cloud)
|
||||
- `db://host:port` - LanceDB Enterprise or LanceDB Cloud (legacy) database
|
||||
|
||||
### Parameters
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ function connectNamespace(
|
||||
Connect to a LanceDB database through a namespace.
|
||||
|
||||
Unlike [connect](connect.md), which routes by URI scheme (local path vs.
|
||||
`db://` cloud), `connectNamespace` always returns a namespace-backed
|
||||
`db://`), `connectNamespace` always returns a namespace-backed
|
||||
connection. The `implName` selects the namespace implementation:
|
||||
|
||||
- `"dir"` — directory namespace, configured with [DirNamespaceConfig](../interfaces/DirNamespaceConfig.md).
|
||||
|
||||
@@ -18,8 +18,8 @@ optional allowExternalBlobOutsideBases: boolean;
|
||||
|
||||
Whether blob URIs outside registered external bases may be written.
|
||||
|
||||
Defaults to `false`. This option is supported only for local/native
|
||||
tables; remote tables return an error when it is enabled. An enabled write
|
||||
Defaults to `false`. This option is supported only in LanceDB Embedded;
|
||||
LanceDB Distributed returns an error when it is enabled. An enabled write
|
||||
stores the absolute URI reference without registering a base or copying
|
||||
the external object into the database. The object must remain accessible
|
||||
when the blob is read later.
|
||||
|
||||
@@ -80,7 +80,7 @@ optional userId: string;
|
||||
|
||||
User identifier for tracking purposes.
|
||||
|
||||
This is sent as the `x-lancedb-user-id` header in requests to LanceDB Cloud/Enterprise.
|
||||
This is sent as the `x-lancedb-user-id` header in requests to LanceDB Enterprise.
|
||||
It can be set directly, or via the `LANCEDB_USER_ID` environment variable.
|
||||
Alternatively, set `LANCEDB_USER_ID_ENV_KEY` to specify another environment
|
||||
variable that contains the user ID value.
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
optional apiKey: string;
|
||||
```
|
||||
|
||||
(For LanceDB cloud only): the API key to use with LanceDB Cloud.
|
||||
(For LanceDB Enterprise or LanceDB Cloud only): the API key to use.
|
||||
|
||||
Can also be set via the environment variable `LANCEDB_API_KEY`.
|
||||
|
||||
@@ -26,7 +26,7 @@ Can also be set via the environment variable `LANCEDB_API_KEY`.
|
||||
optional clientConfig: ClientConfig;
|
||||
```
|
||||
|
||||
(For LanceDB cloud only): configuration for the remote HTTP client.
|
||||
(For LanceDB Enterprise or LanceDB Cloud only): configuration for the HTTP client.
|
||||
|
||||
***
|
||||
|
||||
@@ -36,8 +36,8 @@ optional clientConfig: ClientConfig;
|
||||
optional hostOverride: string;
|
||||
```
|
||||
|
||||
(For LanceDB cloud only): the host to use for LanceDB cloud. Used
|
||||
for testing purposes.
|
||||
(For LanceDB Enterprise only): the host of the LanceDB Enterprise
|
||||
deployment to connect to.
|
||||
|
||||
***
|
||||
|
||||
@@ -47,7 +47,7 @@ for testing purposes.
|
||||
optional manifestEnabled: boolean;
|
||||
```
|
||||
|
||||
(For LanceDB OSS only): use directory namespace manifests as the source
|
||||
(For LanceDB Embedded only): use directory namespace manifests as the source
|
||||
of truth for table metadata. Existing directory-listed root tables are
|
||||
migrated into the manifest on access.
|
||||
|
||||
@@ -59,7 +59,7 @@ migrated into the manifest on access.
|
||||
optional namespaceClientProperties: Record<string, string>;
|
||||
```
|
||||
|
||||
(For LanceDB OSS only): extra properties for the backing namespace
|
||||
(For LanceDB Embedded only): extra properties for the backing namespace
|
||||
client used by manifest-enabled native connections.
|
||||
|
||||
***
|
||||
@@ -70,7 +70,7 @@ client used by manifest-enabled native connections.
|
||||
optional oauthConfig: NativeOAuthConfig;
|
||||
```
|
||||
|
||||
(For LanceDB cloud only): OAuth configuration for IdP-based
|
||||
(For LanceDB Enterprise only): OAuth configuration for IdP-based
|
||||
authentication (e.g., Azure Entra ID). When set, token acquisition
|
||||
and refresh are handled entirely in Rust. TypeScript users should pass
|
||||
the public `OAuthConfig` type exported from `@lancedb/lancedb`.
|
||||
@@ -105,7 +105,7 @@ storage, raising per-read latency and cost.
|
||||
optional region: string;
|
||||
```
|
||||
|
||||
(For LanceDB cloud only): the region to use for LanceDB cloud.
|
||||
(For LanceDB Cloud only): the region to use for LanceDB Cloud (legacy).
|
||||
Defaults to 'us-east-1'.
|
||||
|
||||
***
|
||||
@@ -116,7 +116,7 @@ Defaults to 'us-east-1'.
|
||||
optional session: Session;
|
||||
```
|
||||
|
||||
(For LanceDB OSS only): the session to use for this connection. Holds
|
||||
(For LanceDB Embedded only): the session to use for this connection. Holds
|
||||
shared caches and other session-specific state.
|
||||
|
||||
***
|
||||
@@ -127,6 +127,6 @@ shared caches and other session-specific state.
|
||||
optional storageOptions: Record<string, string>;
|
||||
```
|
||||
|
||||
(For LanceDB OSS only): configuration for object storage.
|
||||
(For LanceDB Embedded only): configuration for object storage.
|
||||
|
||||
The available options are described at https://docs.lancedb.com/storage/
|
||||
|
||||
@@ -31,7 +31,7 @@ optional createdAt: Date;
|
||||
|
||||
When the index was created.
|
||||
|
||||
`undefined` for remote tables or indices created before timestamps were tracked.
|
||||
`undefined` in LanceDB Distributed or indices created before timestamps were tracked.
|
||||
|
||||
***
|
||||
|
||||
@@ -43,8 +43,8 @@ optional indexDetails: any;
|
||||
|
||||
Index-type-specific details parsed as a JavaScript object.
|
||||
|
||||
Falls back to a raw string if JSON parsing fails. `undefined` for
|
||||
remote tables or when details are unavailable.
|
||||
Falls back to a raw string if JSON parsing fails. `undefined` in
|
||||
LanceDB Distributed or when details are unavailable.
|
||||
|
||||
***
|
||||
|
||||
@@ -66,7 +66,7 @@ optional indexUuid: string;
|
||||
|
||||
The UUID of the first segment of the index.
|
||||
|
||||
`undefined` for remote tables, which do not yet surface this.
|
||||
`undefined` in LanceDB Distributed, which does not yet surface this.
|
||||
|
||||
***
|
||||
|
||||
@@ -78,7 +78,7 @@ optional indexVersion: number;
|
||||
|
||||
The on-disk index format version.
|
||||
|
||||
`undefined` for remote tables.
|
||||
`undefined` in LanceDB Distributed.
|
||||
|
||||
***
|
||||
|
||||
@@ -100,7 +100,7 @@ optional numIndexedRows: number;
|
||||
|
||||
The number of rows indexed, across all segments.
|
||||
|
||||
May be `undefined` for remote tables if the server does not report it.
|
||||
May be `undefined` in LanceDB Distributed if the server does not report it.
|
||||
|
||||
***
|
||||
|
||||
@@ -112,7 +112,7 @@ optional numSegments: number;
|
||||
|
||||
The number of segments that make up the index.
|
||||
|
||||
`undefined` for remote tables.
|
||||
`undefined` in LanceDB Distributed.
|
||||
|
||||
***
|
||||
|
||||
@@ -124,7 +124,7 @@ optional numUnindexedRows: number;
|
||||
|
||||
The number of rows not yet covered by this index.
|
||||
|
||||
May be `undefined` for remote tables if the server does not report it.
|
||||
May be `undefined` in LanceDB Distributed if the server does not report it.
|
||||
|
||||
***
|
||||
|
||||
@@ -136,7 +136,7 @@ optional sizeBytes: number;
|
||||
|
||||
The total size in bytes of all index files across all segments.
|
||||
|
||||
`undefined` for remote tables or indices without size tracking.
|
||||
`undefined` in LanceDB Distributed or indices without size tracking.
|
||||
|
||||
***
|
||||
|
||||
@@ -148,4 +148,4 @@ optional typeUrl: string;
|
||||
|
||||
The protobuf type URL, a precise type identifier for the index.
|
||||
|
||||
`undefined` for remote tables.
|
||||
`undefined` in LanceDB Distributed.
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
optional mode: string;
|
||||
```
|
||||
|
||||
How the view was brought up to date: "rebuild", or absent for remote SQL refreshes.
|
||||
How the view was brought up to date: "rebuild", or absent for LanceDB Distributed SQL refreshes.
|
||||
|
||||
***
|
||||
|
||||
@@ -24,7 +24,7 @@ How the view was brought up to date: "rebuild", or absent for remote SQL refresh
|
||||
optional rowsWritten: number;
|
||||
```
|
||||
|
||||
Rows written by this refresh, or absent for remote SQL refreshes.
|
||||
Rows written by this refresh, or absent for LanceDB Distributed SQL refreshes.
|
||||
|
||||
***
|
||||
|
||||
@@ -34,7 +34,7 @@ Rows written by this refresh, or absent for remote SQL refreshes.
|
||||
optional sourceVersion: number;
|
||||
```
|
||||
|
||||
Source version refreshed to, or absent for remote SQL refreshes.
|
||||
Source version refreshed to, or absent for LanceDB Distributed SQL refreshes.
|
||||
|
||||
***
|
||||
|
||||
|
||||
@@ -21,4 +21,4 @@ optional signal: AbortSignal;
|
||||
```
|
||||
|
||||
Cancels the read. The call rejects with `signal.reason`, and the native
|
||||
read stops, including in-flight requests to a remote table.
|
||||
read stops, including in-flight requests to a LanceDB Distributed table.
|
||||
|
||||
+17
-17
@@ -27,8 +27,8 @@ connect, per-table storage and write options on create, and `index_cache_size`,
|
||||
Omit these options to use server-managed storage and caching. Connection-level
|
||||
`storage_options` supported by an Enterprise deployment remain available.
|
||||
|
||||
The synchronous remote `create_table` uses the same positional order as the
|
||||
local API: `name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
|
||||
The synchronous LanceDB Distributed `create_table` uses the same positional order as the
|
||||
LanceDB Embedded API: `name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
|
||||
embedding_functions`. Prefer keyword arguments for optional parameters.
|
||||
|
||||
## Connections (Synchronous)
|
||||
@@ -227,14 +227,14 @@ async for entry in catalog.authz.list_acls(subject=Subject.role("reader")):
|
||||
|
||||
## Remote SQL
|
||||
|
||||
Submit SQL against a remote LanceDB database through the connection.
|
||||
Submit SQL against a LanceDB Distributed database through the connection.
|
||||
The connected database and `default_namespace_path=["public"]` are used for
|
||||
unqualified tables. Fully qualified references can still query other databases
|
||||
and namespaces available to the same deployment. `execute_query` returns a
|
||||
reader as soon as its initial result stream is available. `execute_query_async`
|
||||
returns a query handle once submission is accepted; use it to inspect progress, open a reader,
|
||||
or cancel the query. The SQL client is initialized by the first query and
|
||||
retained for the lifetime of the remote connection. Query ids are random,
|
||||
retained for the lifetime of the connection. Query ids are random,
|
||||
connection-scoped references rather than encoded SQL or durable resume tokens.
|
||||
|
||||
Both APIs start with one Flight call. The server can stream results on that
|
||||
@@ -308,24 +308,24 @@ listing a storage directory.
|
||||
|
||||
## Tables (Synchronous)
|
||||
|
||||
Local and remote tables use the same arguments for index creation, deletion,
|
||||
LanceDB Embedded and LanceDB Distributed tables use the same arguments for index creation, deletion,
|
||||
index statistics, and search. For example, `table.delete(where="id = 1")`,
|
||||
`table.create_index("cosine", 2)` (the deprecated vector API), and
|
||||
`table.search([1.0, 1.0], fast_search=True)` work with either backend.
|
||||
`fast_search=True` excludes rows that have not been indexed.
|
||||
|
||||
`create_scalar_index` defaults to `replace=True` on both backends. Both scalar
|
||||
and full-text index creation accept `wait_timeout`. Remote maintenance methods
|
||||
accept the local options: cleanup and compaction warn that they are no-ops,
|
||||
and full-text index creation accept `wait_timeout`. LanceDB Distributed maintenance methods
|
||||
accept the LanceDB Embedded options: cleanup and compaction warn that they are no-ops,
|
||||
while `optimize` reports an unsupported operation. Removed Tantivy options
|
||||
produce the same explanatory errors on both backends.
|
||||
|
||||
The old remote keyword names `predicate` for `delete`, `index_name` for
|
||||
The old LanceDB Distributed keyword names `predicate` for `delete`, `index_name` for
|
||||
`drop_index`, `index_uuid` for `index_stats`, and `column` for `create_fts_index`
|
||||
remain available with deprecation warnings. Use `where`, `name`, `index_name`,
|
||||
and `field_names`, respectively. For positional calls, use the argument order
|
||||
documented by `Table` below; specify index options by keyword when migrating
|
||||
code that used the former remote-only argument order.
|
||||
code that used the former LanceDB Distributed-only argument order.
|
||||
|
||||
::: lancedb.table.Table
|
||||
|
||||
@@ -429,14 +429,14 @@ scoped to each invocation and support `read`, `readinto`, `seek`, `tell`,
|
||||
|
||||
## Materialized Views (Synchronous)
|
||||
|
||||
In the 0.41 SDK release, local materialized-view creation and refresh require a
|
||||
In the 0.41 SDK release, LanceDB Embedded materialized-view creation and refresh require a
|
||||
source build with the `experimental-functions` Cargo feature. Published Python
|
||||
packages leave it disabled. Opening, reading, listing and dropping existing views
|
||||
remain available, as do remote Cloud and Enterprise operations. The same feature
|
||||
requirement applies to local SQL computed-column declaration and refresh; ordinary
|
||||
remain available, as do LanceDB Distributed operations. The same feature
|
||||
requirement applies to LanceDB Embedded SQL computed-column declaration and refresh; ordinary
|
||||
eager `add_columns` transforms remain available.
|
||||
|
||||
On Cloud and Enterprise, creation accepts `auto_refresh_enabled` and
|
||||
In LanceDB Distributed, creation accepts `auto_refresh_enabled` and
|
||||
`auto_refresh_interval_seconds`:
|
||||
|
||||
```python
|
||||
@@ -581,13 +581,13 @@ pylance, LanceDB uses a matching `lance.blob.v2` extension type so blob columns
|
||||
still work. Native table queries return descriptors. Call
|
||||
[`fetch_blob_files`][lancedb.table.Table.fetch_blob_files] for lazy reads or
|
||||
[`fetch_blobs`][lancedb.table.Table.fetch_blobs] for eager bytes.
|
||||
Local tables support these fetch APIs for both legacy blob v1 columns
|
||||
LanceDB Embedded supports these fetch APIs for both legacy blob v1 columns
|
||||
(`lance-encoding:blob`) and blob v2 columns. For legacy columns, obtain row IDs
|
||||
with `search().with_row_id(True).to_arrow()` and pass the result to the fetch API.
|
||||
One missing or deleted row ID rejects the entire blob-read batch; the APIs do
|
||||
not return partial results.
|
||||
|
||||
Remote SQL returns blob bytes by default. Select `blob_descriptor(video)` to
|
||||
LanceDB Distributed SQL returns blob bytes by default. Select `blob_descriptor(video)` to
|
||||
return an openable reference instead, then use
|
||||
[`open_blob_file`][lancedb.table.Table.open_blob_file] or the lazy iterable
|
||||
[`open_blob_files`][lancedb.table.Table.open_blob_files]:
|
||||
@@ -604,7 +604,7 @@ for row in rows.to_pylist():
|
||||
frame = extract_frame(video, row["timestamp"])
|
||||
```
|
||||
|
||||
This example uses a remote SQL connection that produces source-bearing
|
||||
This example uses a LanceDB Distributed SQL connection that produces source-bearing
|
||||
descriptors. The descriptor contains `kind`, `position`, `size`, `blob_id`,
|
||||
`blob_uri`, and a `source` struct with `uri`, `version`, `committed_at_micros`,
|
||||
`branch`, `column`, `row_reference`, and `row_is_address`. The opener verifies
|
||||
@@ -636,7 +636,7 @@ async for video in async_videos.open_blob_files(rows["video"]):
|
||||
```
|
||||
|
||||
Payload bytes load on reads. Opening an external blob with unspecified size
|
||||
can issue a metadata-only HEAD request. Remote reads use authenticated range
|
||||
can issue a metadata-only HEAD request. LanceDB Distributed reads use authenticated range
|
||||
requests that validate the captured URI and commit timestamp.
|
||||
|
||||
Blobs inside lists can be written and queried as descriptors. They are omitted
|
||||
|
||||
@@ -64,13 +64,13 @@ The branch experiments leave you with the results side by side, and they
|
||||
deliberately never touch `main`. Once you've picked a winner, apply it to
|
||||
`main` yourself by rerunning that experiment's validated transformation
|
||||
directly against the `main` table. Rerunning the reviewed operation is the
|
||||
reliable path on both OSS and Enterprise: it replays exactly the transformation
|
||||
reliable path on both LanceDB Embedded and LanceDB Enterprise: it replays exactly the transformation
|
||||
you validated on the branch, and it works the same regardless of how the branch
|
||||
evolved.
|
||||
|
||||
Suppose `nomic-embed-text` wins. Rerun the same Nomic embedding transformation
|
||||
you validated on `embed-nomic` against `main` in batches, then call
|
||||
`table.optimize()` on OSS. Verify a bounded vector search on `main`, and keep
|
||||
`table.optimize()` on LanceDB Embedded. Verify a bounded vector search on `main`, and keep
|
||||
the losing branch around until you're confident in the result.
|
||||
|
||||
See [Branches](/tables/branching) for more on how branches, versions, and tags
|
||||
@@ -96,5 +96,5 @@ branch that's isolated from `main` until you decide you want the new derived
|
||||
column in `main`.
|
||||
|
||||
Once you've chosen a winner, rerun that validated transformation to ingest it into `main`
|
||||
yourself, on either LanceDB OSS or Enterprise. See our documentation on [branches](/tables/branching)
|
||||
yourself, on either LanceDB Embedded or LanceDB Enterprise. See our documentation on [branches](/tables/branching)
|
||||
for more information.
|
||||
|
||||
@@ -9,7 +9,7 @@ description: "API reference for LanceDB"
|
||||
is an OpenAPI protocol that enables reading, writing and managing Lance tables by connecting
|
||||
those metadata services or building a custom metadata server in a standardized way.
|
||||
|
||||
LanceDB OSS allows you to interface with Lance tables via the REST Namespace.
|
||||
LanceDB Embedded allows you to interface with Lance tables via the REST Namespace.
|
||||
LanceDB Enterprise provides an extended REST API with
|
||||
additional endpoints for managing tables and data.
|
||||
If you have specific needs or questions about the Enterprise REST API Namespace,
|
||||
|
||||
+17
-13
@@ -1,14 +1,17 @@
|
||||
---
|
||||
title: "Architecture"
|
||||
sidebarTitle: "Architecture"
|
||||
description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with embedded LanceDB OSS."
|
||||
description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with LanceDB Embedded."
|
||||
icon: cubes
|
||||
keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"]
|
||||
---
|
||||
|
||||
LanceDB OSS is an embedded library for working with Lance tables. LanceDB
|
||||
Enterprise is a multimodal lakehouse for data curation, feature engineering,
|
||||
search, and training data access. Its architecture separates query serving,
|
||||
LanceDB Embedded is an open-source library for working with Lance tables. It runs
|
||||
in your application's process; your data can live on local disk or in object
|
||||
storage. LanceDB Enterprise is a multimodal lakehouse for data curation, feature
|
||||
engineering, search, and training data access. LanceDB Enterprise runs LanceDB
|
||||
Distributed: query execution, indexing and maintenance run on the Enterprise
|
||||
service, outside your application. Its architecture separates query serving,
|
||||
read execution, caching, and background work. Both use the open
|
||||
[Lance](/basics/lance) format.
|
||||
|
||||
@@ -17,23 +20,23 @@ read the resulting tables into training frameworks. Enterprise separates the
|
||||
resources that serve shared data from those that build indexes and run
|
||||
background work, so teams can size them for these different workloads.
|
||||
|
||||

|
||||

|
||||
|
||||
The diagram shows the main compute and storage boundaries, not every request
|
||||
path. The control plane governs the Enterprise service.
|
||||
|
||||
## A shared format, different storage choices {#storage}
|
||||
|
||||
Lance stores table data, versions, and indexes. OSS can read and write tables on
|
||||
Lance stores table data, versions, and indexes. LanceDB Embedded can read and write tables on
|
||||
local disk or in object storage. Enterprise keeps durable table data and index
|
||||
artifacts in object storage, independently of the machines serving requests.
|
||||
Its caches accelerate reads; they are not the durable source of truth.
|
||||
|
||||
Sharing a format does not mean that an Enterprise deployment automatically
|
||||
opens an existing OSS database. Plan how to move or connect your data and
|
||||
opens an existing LanceDB Embedded database. Plan how to move or connect your data and
|
||||
configure access for your deployment.
|
||||
|
||||
## OSS: an embedded library inside your application {#embedded}
|
||||
## LanceDB Embedded: a library inside your application {#embedded}
|
||||
|
||||
With an embedded connection, the LanceDB library plans and executes queries,
|
||||
builds indexes, and updates tables inside your process. There is no separate
|
||||
@@ -44,9 +47,10 @@ single-node data pipelines. Your application controls the compute resources and
|
||||
when to run [table maintenance](/maintenance/compaction). You also manage access
|
||||
to the local files or object storage that hold your tables.
|
||||
|
||||
## Enterprise: a multimodal lakehouse for shared workloads {#distributed}
|
||||
## LanceDB Enterprise: distributed execution for shared workloads {#distributed}
|
||||
|
||||
Applications connect to remote tables through a LanceDB SDK. The service handles
|
||||
A LanceDB SDK connects to LanceDB Enterprise, and the service runs operations
|
||||
with LanceDB Distributed. The service handles
|
||||
requests across a fleet, with components sized for different kinds of work:
|
||||
|
||||
- **Query serving** receives requests, resolves tables, and plans operations.
|
||||
@@ -81,20 +85,20 @@ and request flows.
|
||||
## Application changes for Enterprise {#differences}
|
||||
|
||||
Core operations remain familiar: create a table, add data, and run a search.
|
||||
With OSS, the application executes those operations. With Enterprise, the SDK
|
||||
With LanceDB Embedded, the application executes those operations. With Enterprise, the SDK
|
||||
sends requests to the service, and the service manages execution and configured
|
||||
background maintenance.
|
||||
|
||||
To use Enterprise, configure the cluster endpoint, database URI and credentials,
|
||||
and make your data available in the Enterprise database. Review SDK methods
|
||||
that behave differently on remote tables, including result retrieval and
|
||||
that behave differently with LanceDB Distributed, including result retrieval and
|
||||
background jobs. [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise)
|
||||
describes these application changes. If you plan to use materialized views, check the
|
||||
[stable row ID requirements](/basics/limits#schema) before creating source tables.
|
||||
|
||||
## Choose for the workload you need to run {#choosing}
|
||||
|
||||
Choose OSS when you want an embedded library and can manage its resources and
|
||||
Choose LanceDB Embedded when you want an embedded library and can manage its resources and
|
||||
maintenance in your application.
|
||||
|
||||
Choose Enterprise when multiple applications or teams need a shared multimodal
|
||||
|
||||
@@ -158,13 +158,13 @@ LANCEDB_REGION=us-east-1
|
||||
LANCEDB_HOST_OVERRIDE=https://your-enterprise-endpoint.com
|
||||
```
|
||||
|
||||
If you're using LanceDB OSS, no connection settings are required, as it runs as an
|
||||
If you're using LanceDB Embedded, no connection settings are required, as it runs as an
|
||||
embedded retrieval library. A simple prompt like this should work:
|
||||
|
||||
```text Agent prompt
|
||||
# If using OSS
|
||||
# If using LanceDB Embedded
|
||||
Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB
|
||||
OSS table.
|
||||
Embedded table.
|
||||
|
||||
# If using enterprise
|
||||
Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB
|
||||
@@ -208,7 +208,7 @@ to `batch_size` rows at a time, providing an iterable of batches for the ingesti
|
||||
shown next.
|
||||
|
||||
#### Table maintenance {#table-maintenance}
|
||||
For LanceDB OSS, the plugin instructs the agent to call
|
||||
For LanceDB Embedded, the plugin instructs the agent to call
|
||||
`table.optimize()` after the ingestion loop. This compacts small fragments, cleans up
|
||||
old versions according to the retention policy, and incorporates new data into indexes.
|
||||
|
||||
@@ -225,7 +225,7 @@ This example dataset has only eight rows, so the default batch size writes it in
|
||||
one call. Larger inputs should still avoid single-row write commits.
|
||||
</Note>
|
||||
|
||||
### 3\. Run the OSS pipeline {#3-run-the-oss-pipeline}
|
||||
### 3\. Run the Embedded pipeline {#3-run-the-oss-pipeline}
|
||||
|
||||
```bash
|
||||
uv run python ingest_multimodal.py
|
||||
@@ -258,9 +258,9 @@ The plugin shown in this tutorial should generalize reasonably well to other use
|
||||
If you find any issues, open [an issue](https://github.com/lancedb/lancedb-agent-plugins/issues)
|
||||
on GitHub, clearly describing the intended behavior.
|
||||
|
||||
You can choose OSS or Enterprise based on how the work will run:
|
||||
You can choose LanceDB Embedded or LanceDB Enterprise based on how the work will run:
|
||||
|
||||
Start with [LanceDB OSS](/quickstart) during the early stages of a project
|
||||
Start with [LanceDB Embedded](/quickstart) during the early stages of a project
|
||||
when an agent is helping you prototype,
|
||||
explore a dataset, or run small workflows on a subset of the data. The application owns
|
||||
the storage and lifecycle work, so ask the agent to validate inputs, write in
|
||||
@@ -271,7 +271,7 @@ Choose [LanceDB Enterprise](/enterprise) when the resulting table becomes
|
||||
shared production infrastructure, and the workload needs distributed capacity,
|
||||
private deployment, or platform-managed operations. The data format and the core
|
||||
table API stay the same, so the pipeline's table and search code carries over:
|
||||
the agent connects to a remote `db://` table, leaves maintenance to the
|
||||
the agent connects to an Enterprise `db://` table, leaves maintenance to the
|
||||
deployment instead of calling `optimize()`, and submits long-running work as
|
||||
jobs. See [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: "Connect to LanceDB"
|
||||
sidebarTitle: "Connect"
|
||||
description: "Use the embedded LanceDB OSS library with tables in a local directory or object storage, or connect to LanceDB Enterprise."
|
||||
description: "Use LanceDB Embedded with tables in a local directory or object storage, or connect to LanceDB Enterprise."
|
||||
icon: plug
|
||||
keywords: ["connect", "uri", "embedded", "object storage", "s3", "enterprise", "db://"]
|
||||
---
|
||||
@@ -20,15 +20,18 @@ import {
|
||||
TsConnectObjectStorage,
|
||||
} from '/snippets/connection.mdx';
|
||||
|
||||
The URI you pass to `connect` decides where the work runs. The same Python,
|
||||
TypeScript and Rust SDKs open all three kinds;
|
||||
[OSS and Enterprise](/basics/offerings) describes how the two offerings differ.
|
||||
The URI you pass to `connect` decides where the work runs. LanceDB Embedded
|
||||
runs in your application's process; your data can live on local disk or in
|
||||
object storage. LanceDB Enterprise runs LanceDB Distributed: query execution,
|
||||
indexing and maintenance run on the Enterprise service, outside your
|
||||
application. The same Python, TypeScript and Rust SDKs open all three kinds;
|
||||
[Embedded and Enterprise](/basics/offerings) describes how the two offerings differ.
|
||||
|
||||
| URI | Where queries run |
|
||||
| --- | --- |
|
||||
| A local path, such as `./data` | OSS: in your process |
|
||||
| `s3://`, `gs://` or `az://` | OSS: in your process, reading object storage |
|
||||
| `db://` | LanceDB Enterprise: queries run on your cluster |
|
||||
| A local path, such as `./data` | LanceDB Embedded: in your process |
|
||||
| `s3://`, `gs://` or `az://` | LanceDB Embedded: in your process, reading object storage |
|
||||
| `db://` | LanceDB Enterprise: LanceDB Distributed runs queries on your cluster |
|
||||
|
||||
## A local directory {#local}
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ rewrite data; adding a derived column can write the new column independently.
|
||||
|
||||
A read of a table uses a consistent snapshot. A consistent snapshot can still
|
||||
be older than another writer's latest commit. [Consistency](/tables/consistency)
|
||||
explains how OSS and Enterprise choose the version a reader sees.
|
||||
explains how LanceDB Embedded and LanceDB Enterprise choose the version a reader sees.
|
||||
|
||||
For an experiment that must keep using the same data, select a specific retained
|
||||
table version rather than relying on a connection's default freshness.
|
||||
@@ -46,7 +46,7 @@ describes commit behavior and compatibility between operations.
|
||||
## Storage durability {#durability}
|
||||
|
||||
A committed table version is stored in the table’s configured storage. LanceDB
|
||||
OSS uses your local filesystem or object store; LanceDB Enterprise uses the
|
||||
Embedded uses your local filesystem or object store; LanceDB Enterprise uses the
|
||||
cluster’s configured storage.
|
||||
|
||||
Table version history does not replace backups. Configure storage redundancy
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Job handles track operations in OSS and Enterprise. Enterprise also
|
||||
Job handles track operations in LanceDB Embedded and LanceDB Enterprise. Enterprise also
|
||||
supports listing server jobs, reopening them by ID and reading their events.
|
||||
---
|
||||
|
||||
@@ -19,7 +19,7 @@ column refresh. Use it to check progress, wait for completion and inspect
|
||||
failures.
|
||||
|
||||
Methods such as Python's `create_index_async` and TypeScript's
|
||||
`createIndexAsync` return job handles. In OSS, the work runs in your application
|
||||
`createIndexAsync` return job handles. In LanceDB Embedded, the work runs in your application
|
||||
process; in Enterprise, it runs on server workers or through the SQL query
|
||||
backend. A returned handle can represent an operation that has already finished,
|
||||
so check its status before
|
||||
@@ -58,12 +58,13 @@ Event responses have [row limits](/basics/limits#jobs). An explicit limit can
|
||||
request more than the default, but it does not guarantee a complete event
|
||||
history.
|
||||
|
||||
## OSS and Enterprise execution {#in-process}
|
||||
## Embedded and Distributed execution {#in-process}
|
||||
|
||||
The following table describes local operations and Enterprise registry jobs.
|
||||
The following table describes LanceDB Embedded operations and registry jobs on
|
||||
LanceDB Distributed (provided by LanceDB Enterprise).
|
||||
SQL-backed job handles follow the query lifecycle described below.
|
||||
|
||||
| | OSS | Enterprise registry job |
|
||||
| | Embedded | Enterprise registry job |
|
||||
| --- | --- | --- |
|
||||
| Execution | In your application process | On server workers |
|
||||
| Job identity | No server ID | A server ID for looking up the job |
|
||||
@@ -72,7 +73,7 @@ SQL-backed job handles follow the query lifecycle described below.
|
||||
|
||||
### SQL-backed job handles
|
||||
|
||||
Remote materialized-view CREATE, REFRESH, and DROP use the SQL query backend.
|
||||
On Enterprise, materialized-view CREATE, REFRESH, and DROP use the SQL query backend.
|
||||
Python's `create_materialized_view_async()`, `mv.refresh_async()`, and
|
||||
`drop_materialized_view_async()` return job handles with `status()`, `wait()`,
|
||||
and `cancel()`. Synchronous creation, refresh, and drop wait on the same SQL
|
||||
|
||||
@@ -44,7 +44,7 @@ framework runs the model; LanceDB provides access to the training data.
|
||||
|
||||
## Shared compute and maintenance in Enterprise {#where}
|
||||
|
||||
LanceDB OSS runs as an embedded library in your application. LanceDB Enterprise
|
||||
LanceDB Embedded runs as an embedded library in your application. LanceDB Enterprise
|
||||
provides shared infrastructure for applications and teams working with larger
|
||||
multimodal workloads.
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ multimodal AI. It defines how data is stored and organized into versioned tables
|
||||
<span id="embedded" />
|
||||
<span id="enterprise" />
|
||||
|
||||
LanceDB OSS and LanceDB Enterprise both use Lance as their storage foundation.
|
||||
LanceDB Embedded and LanceDB Enterprise both use Lance as their storage foundation.
|
||||
The format is developed as an independent open-source project, with its own
|
||||
specification and tools.
|
||||
|
||||
@@ -42,4 +42,4 @@ those fields together.
|
||||
- [Lance format specification](https://lance.org/format): how the format works.
|
||||
- [Table versioning](/management/table#versioning): read and manage table versions.
|
||||
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
|
||||
- [Architecture](/architecture): how LanceDB OSS and Enterprise build on Lance.
|
||||
- [Architecture](/architecture): how LanceDB Embedded and LanceDB Enterprise build on Lance.
|
||||
|
||||
@@ -14,7 +14,7 @@ Databases, namespaces, tables, views and branches all have names. Choose names
|
||||
that describe their purpose, such as `documents`, `prod` or `training_2026`.
|
||||
Consistent names make data easier to find and application code easier to read.
|
||||
|
||||
Use the following rules for names that work across LanceDB OSS and Enterprise.
|
||||
Use the following rules for names that work across LanceDB Embedded and LanceDB Enterprise.
|
||||
|
||||
| Object | Characters | Length |
|
||||
| --- | --- | --- |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "LanceDB OSS or Enterprise?"
|
||||
sidebarTitle: "OSS and Enterprise"
|
||||
title: "LanceDB Embedded or Enterprise?"
|
||||
sidebarTitle: "Embedded and Enterprise"
|
||||
description: "Start with an embedded library, or use the LanceDB Enterprise multimodal lakehouse for shared AI workloads."
|
||||
icon: compass
|
||||
keywords: ["oss", "enterprise", "embedded", "deployment", "availability", "comparison"]
|
||||
@@ -9,15 +9,18 @@ keywords: ["oss", "enterprise", "embedded", "deployment", "availability", "compa
|
||||
LanceDB gives you two ways to work with multimodal data, both built on the open
|
||||
[Lance](/basics/lance) format.
|
||||
|
||||
**LanceDB OSS** is an embedded library. Run it inside your application
|
||||
with tables on local disk or in object storage. You control the application,
|
||||
**LanceDB Embedded (open source)** is a library that runs in your
|
||||
application's process; your data can live on local disk or in object storage.
|
||||
You control the application,
|
||||
its resources, and table maintenance, without a separate database service.
|
||||
[Try the quickstart](/quickstart) to create a table and run your first search.
|
||||
|
||||
**LanceDB Enterprise** is a multimodal lakehouse for data curation, feature
|
||||
engineering, search, and training data access. It adds query-serving capacity,
|
||||
data caching, and background operations for teams working with shared
|
||||
multimodal datasets.
|
||||
multimodal datasets. LanceDB Enterprise runs LanceDB Distributed: query
|
||||
execution, indexing and maintenance run on the Enterprise service, outside your
|
||||
application.
|
||||
[Connect to Enterprise](/basics/connect#enterprise) when your cluster is ready.
|
||||
|
||||
<span id="labels" />
|
||||
|
||||
@@ -17,7 +17,7 @@ in the last month.
|
||||
|
||||
| Interface | When to use it |
|
||||
| --- | --- |
|
||||
| **SDK query builders** | Build vector, full-text and hybrid searches, apply filters and choose the fields to return. Available in OSS and Enterprise. |
|
||||
| **SDK query builders** | Build vector, full-text and hybrid searches, apply filters and choose the fields to return. Available in LanceDB Embedded and LanceDB Enterprise. |
|
||||
| **Enterprise SQL** | Write SQL queries for filtering, aggregation, joins and other data curation tasks. Connect through Python or a Flight SQL client. |
|
||||
|
||||
Both interfaces work with LanceDB tables. Choose the one that supports your
|
||||
|
||||
+10
-7
@@ -8,13 +8,16 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Python, TypeScript and Rust support both LanceDB OSS and LanceDB Enterprise.
|
||||
Python, TypeScript and Rust support both LanceDB Embedded and LanceDB Enterprise.
|
||||
Java provides a client for the Enterprise REST API.
|
||||
---
|
||||
|
||||
Use a LanceDB SDK to create tables, add data, build indexes and run searches
|
||||
from your application. Python, TypeScript and Rust work with both the embedded
|
||||
OSS library and LanceDB Enterprise.
|
||||
from your application. Python, TypeScript and Rust work with both LanceDB
|
||||
Embedded and LanceDB Enterprise. LanceDB Embedded runs in your application's
|
||||
process; your data can live on local disk or in object storage. LanceDB
|
||||
Enterprise runs LanceDB Distributed: query execution, indexing and maintenance
|
||||
run on the Enterprise service, outside your application.
|
||||
|
||||
The [quickstart](/quickstart) includes installation instructions and complete
|
||||
examples in all three languages. For connection settings, see
|
||||
@@ -73,16 +76,16 @@ For PyTorch worker setup and connection factories, see
|
||||
## Moving to LanceDB Enterprise {#moving-to-enterprise}
|
||||
|
||||
The core table workflow stays familiar: connect, create or open a table, write
|
||||
records, and query them. Enterprise runs those operations on remote compute and
|
||||
manages background maintenance.
|
||||
records, and query them. Enterprise runs those operations with LanceDB
|
||||
Distributed and manages background maintenance.
|
||||
|
||||
- **Configure the connection.** Replace the local path with your database URI,
|
||||
and supply the endpoint, region and credentials provided by your administrator.
|
||||
[Connect to Enterprise](/basics/connect#enterprise) shows these settings.
|
||||
- **Retrieve query results.** Python's remote tables do not support whole-table
|
||||
- **Retrieve query results.** Python tables on Enterprise do not support whole-table
|
||||
`to_arrow()` or `to_pandas()` calls. Use a query and select the columns and
|
||||
number of rows you need, for example `table.search().limit(10).to_arrow()`.
|
||||
- **Handle table listings and namespaces.** Remote table listings are paginated.
|
||||
- **Handle table listings and namespaces.** Enterprise table listings are paginated.
|
||||
Use the namespace and pagination options described in
|
||||
[Namespaces](/management/namespace).
|
||||
- **Use server-side jobs.** Index builds and feature refreshes can run as
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: unavailable
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Enterprise supports SQL through Arrow Flight SQL. Use the SDK table APIs for embedded OSS tables. See [Connection settings](#sql-is-remote) to connect.
|
||||
Enterprise supports SQL through Arrow Flight SQL. Use the SDK table APIs for LanceDB Embedded tables. See [Connection settings](#sql-is-remote) to connect.
|
||||
---
|
||||
|
||||
Use SQL in LanceDB Enterprise to filter records, join tables, calculate
|
||||
@@ -52,7 +52,7 @@ API and SQL endpoints may use different hosts or ports.
|
||||
See [Connect to Enterprise](/basics/connect#enterprise) for other connection options.
|
||||
|
||||
SQL runs in Enterprise. Connections to a local directory or an object-storage
|
||||
bucket use the embedded OSS library and cannot execute SQL statements. Use
|
||||
bucket use LanceDB Embedded and cannot execute SQL statements. Use
|
||||
SDK table operations to query those tables.
|
||||
|
||||
## Run a query in Python {#sql-from-python}
|
||||
@@ -94,7 +94,7 @@ handles, streaming, and cancellation.
|
||||
Use **SQL** for joins, aggregations, window functions, and analytical queries
|
||||
written in SQL. Use **SDK table operations** to build applications around table
|
||||
management, ingestion, filtering, and search, including applications that run
|
||||
with the embedded OSS library.
|
||||
with LanceDB Embedded.
|
||||
|
||||
You can combine them. For example, create and populate a table with an SDK,
|
||||
then query it with SQL to prepare a training dataset. In Python, both workflows
|
||||
@@ -130,7 +130,7 @@ server version supports this form, then use
|
||||
[`EXPLAIN`](/sql/dql/explain) to inspect the query plan.
|
||||
|
||||
The [SDK vector search API](/search/vector-search) is also available in
|
||||
Enterprise and OSS. Use it when you prefer the SDK query builder or need to
|
||||
LanceDB Enterprise and LanceDB Embedded. Use it when you prefer the SDK query builder or need to
|
||||
search embedded tables.
|
||||
|
||||
For full-text search in SQL, use the [`fts()`](/sql/functions#fts) table
|
||||
|
||||
@@ -23,8 +23,8 @@ records belong, and save the selected data or the rules used to produce it.
|
||||
| [Rank and deduplicate](/curation/windowing) | Which records to keep when several describe the same item. |
|
||||
|
||||
The analytical examples in this section use **Enterprise SQL**. Connect with
|
||||
the [SQL guide](/basics/sql). For sampling through the Python SDK in OSS or
|
||||
Enterprise, see [Sample without SQL](/curation/sampling#training).
|
||||
the [SQL guide](/basics/sql). For sampling through the Python SDK in LanceDB Embedded or
|
||||
LanceDB Enterprise, see [Sample without SQL](/curation/sampling#training).
|
||||
|
||||
## Prepare a labeled document dataset {#shape}
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: unavailable
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Enterprise supports SQL `TABLESAMPLE`. For sampling in OSS and Enterprise through the Python SDK, see [Sample without SQL](#training).
|
||||
Enterprise supports SQL `TABLESAMPLE`. For sampling in LanceDB Embedded and LanceDB Enterprise through the Python SDK, see [Sample without SQL](#training).
|
||||
---
|
||||
|
||||
import { PySampleDataset } from '/snippets/curation_sampling.mdx';
|
||||
@@ -143,7 +143,7 @@ for sampling. `TABLESAMPLE` cannot be applied directly to a subquery result.
|
||||
## Sample without SQL {#training}
|
||||
|
||||
The Python SDK provides a permutation builder for selecting and ordering rows
|
||||
in OSS and Enterprise. A split selects a subset; a shuffle changes the order
|
||||
in LanceDB Embedded and LanceDB Enterprise. A split selects a subset; a shuffle changes the order
|
||||
in which records are read. `Permutation.identity(table)` retains stored order.
|
||||
|
||||
<Accordion title="Complete Python example: read a random subset">
|
||||
|
||||
@@ -11,7 +11,7 @@ LanceDB emits internal metrics (currently object store request counts, bytes tra
|
||||
The bridge is available in the Python and TypeScript SDKs. It is a thin wrapper over LanceDB's `metrics` recorder; your application supplies and configures the OpenTelemetry SDK.
|
||||
|
||||
<Note>
|
||||
This page covers LanceDB OSS instrumentation. LanceDB Enterprise clusters emit their own metrics from the server side. See [Enterprise monitoring](#enterprise) below.
|
||||
This page covers LanceDB Embedded instrumentation. LanceDB Enterprise clusters emit their own metrics from the server side. See [Enterprise monitoring](#enterprise) below.
|
||||
</Note>
|
||||
|
||||
## Available metrics {#what-you-get}
|
||||
@@ -113,7 +113,7 @@ export them with data enrichment. No client-side instrumentation is needed.
|
||||
| Component | Metrics |
|
||||
| --- | --- |
|
||||
| Query node | RPS, query latency, error codes, slow-query count, CPU/memory utilization |
|
||||
| Plan executor | SSD cache hit/miss rate, CPU/memory utilization |
|
||||
| Executor | SSD cache hit/miss rate, CPU/memory utilization |
|
||||
|
||||
[Contact support](/deployment/support) for the comprehensive metric list and
|
||||
help integrating them with your monitoring dashboard.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: "Embedded storage configuration"
|
||||
sidebarTitle: "Embedded Storage"
|
||||
description: "Choose and configure local disk or object storage for tables accessed by the embedded LanceDB OSS library."
|
||||
description: "Choose and configure local disk or object storage for tables accessed by LanceDB Embedded."
|
||||
icon: database
|
||||
keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"]
|
||||
availability:
|
||||
@@ -13,7 +13,7 @@ availability:
|
||||
the deployment's own configuration instead.
|
||||
---
|
||||
|
||||
This page is about the embedded library, where your connection chooses and
|
||||
This page is about LanceDB Embedded, where your connection chooses and
|
||||
configures the storage backend. A LanceDB Enterprise deployment sets up its own
|
||||
storage when it is deployed, so a `db://` connection passes none of these
|
||||
options except an Azure storage account name. See
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
---
|
||||
title: "Support"
|
||||
sidebarTitle: "Support"
|
||||
description: "Where to get help: community Discord for OSS, dedicated support for Enterprise."
|
||||
description: "Where to get help: community Discord for LanceDB Embedded (open source), dedicated support for LanceDB Enterprise."
|
||||
icon: circle-question
|
||||
keywords: ["support", "help", "discord", "community"]
|
||||
---
|
||||
|
||||
## LanceDB OSS {#oss}
|
||||
## LanceDB Embedded (open source) {#oss}
|
||||
|
||||
For a quick question, the
|
||||
[Discord community](https://discord.gg/AUEWnJ7Txb) has channels for
|
||||
|
||||
@@ -35,7 +35,7 @@ Modern machine learning models can be trained to convert raw data into embedding
|
||||
of floating point numbers. The position of an embedding in vector space captures the semantics of
|
||||
the data, so vectors that are close to each other are considered similar.
|
||||
|
||||
LanceDB provides an embedding function registry in OSS as well as its Enterprise versions
|
||||
LanceDB provides an embedding function registry in LanceDB Embedded as well as LanceDB Enterprise
|
||||
([see below](#embeddings-in-lancedb-enterprise))
|
||||
that automatically generates vector embeddings during data ingestion. Query text is embedded
|
||||
automatically too, in the client, for local and Enterprise tables alike, with SDK-specific query
|
||||
@@ -198,7 +198,7 @@ You can find all supported embedding models in the [integrations](/integrations/
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
In LanceDB Enterprise, embedding generation during data ingestion is client-side and the resulting vectors are
|
||||
stored on the remote table.
|
||||
stored in the Enterprise table.
|
||||
|
||||
<Info>
|
||||
The Enterprise server does not currently generate embeddings from query text on its own. Any automatic
|
||||
@@ -207,13 +207,13 @@ query-time embedding happens on the client side.
|
||||
|
||||
### How string queries are interpreted {#how-string-queries-are-interpreted}
|
||||
|
||||
For the Python remote client, `table.search("hello")` can take two different paths:
|
||||
When the Python SDK connects to Enterprise, `table.search("hello")` can take two different paths:
|
||||
|
||||
- If the selected vector column has embedding metadata
|
||||
(i.e., the table schema stores the source-column, vector-column, and
|
||||
embedding-function mapping created from fields like `SourceField()` and
|
||||
`VectorField()` during table creation), then the embeddings are computed in the Python client process.
|
||||
The client uses the same local LanceDB embedding registry used by OSS tables to
|
||||
The client uses the same local LanceDB embedding registry used by LanceDB Embedded tables to
|
||||
reconstruct the embedding function from schema metadata, compute the query vector in
|
||||
the client process, and send that vector to Enterprise for search.
|
||||
```python
|
||||
|
||||
@@ -42,11 +42,11 @@ from lancedb.embeddings import get_registry
|
||||
|
||||
## Step 2: Connect to LanceDB {#step-2-connect-to-lancedb}
|
||||
|
||||
Establish a connection to your LanceDB OSS directory or Enterprise cluster:
|
||||
Establish a connection to your LanceDB Embedded directory or Enterprise cluster:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
# Enter your LanceDB connection URI for OSS or Enterprise here
|
||||
# Enter your LanceDB connection URI for LanceDB Embedded or Enterprise here
|
||||
db = lancedb.connect(...)
|
||||
```
|
||||
|
||||
@@ -126,8 +126,8 @@ LanceDB preserves them instead of filling only the missing rows.
|
||||
|
||||
## Step 6: Query with Automatic Embedding {#step-6-query-with-automatic-embedding}
|
||||
|
||||
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. In the Python remote
|
||||
client, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
|
||||
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. When the Python
|
||||
SDK connects to Enterprise, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
|
||||
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
|
||||
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ availability:
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Requests to a deployment carry an API key or an OAuth token, whichever it is
|
||||
configured to accept. Embedded LanceDB has no server to sign in to; see
|
||||
configured to accept. LanceDB Embedded has no server to sign in to; see
|
||||
[Architecture](/architecture).
|
||||
---
|
||||
|
||||
@@ -423,5 +423,5 @@ control. All three SDKs deliberately omit the secret from their debug/repr outpu
|
||||
so accidental log lines do not expose the value.
|
||||
</Warning>
|
||||
|
||||
Embedded LanceDB has nothing to authenticate to: your process opens the files
|
||||
LanceDB Embedded has nothing to authenticate to: your process opens the files
|
||||
directly, and the object store's own credentials are the only gate.
|
||||
|
||||
@@ -16,5 +16,5 @@ Deployment is specific to your cloud and account, so the procedure is published
|
||||
with LanceDB Enterprise. [Contact us](https://www.lancedb.com/contact) to get
|
||||
started.
|
||||
|
||||
Embedded LanceDB has nothing to deploy: it is a library your process links
|
||||
LanceDB Embedded has nothing to deploy: it is a library your process links
|
||||
against. See [Architecture](/architecture#embedded).
|
||||
|
||||
@@ -8,17 +8,17 @@ availability:
|
||||
enterprise: available
|
||||
summary: >-
|
||||
LanceDB Enterprise distributes index execution and table-data reads across
|
||||
plan executors. Embedded LanceDB executes searches in the application process.
|
||||
executors. LanceDB Embedded executes searches in the application process.
|
||||
---
|
||||
|
||||
Distributed search spreads index execution, table-data reads, and caching across
|
||||
multiple serving machines in LanceDB Enterprise. Query nodes plan queries and
|
||||
combine results. Plan executors cache index structures and table data and execute
|
||||
the remote work. See [Enterprise architecture](/enterprise/architecture).
|
||||
combine results. Executors cache index structures and table data and execute
|
||||
the distributed work. See [Enterprise architecture](/enterprise/architecture).
|
||||
|
||||
## Embedded search {#embedded-search}
|
||||
|
||||
LanceDB OSS runs inside your application process. It searches indexes and reads
|
||||
LanceDB Embedded runs inside your application process. It searches indexes and reads
|
||||
table data using that process's resources, including its index cache. Reading a
|
||||
table from object storage does not by itself distribute query execution across
|
||||
machines.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: "LanceDB Enterprise"
|
||||
sidebarTitle: "Overview"
|
||||
description: "What a deployment adds over embedded LanceDB."
|
||||
description: "What a deployment adds over LanceDB Embedded."
|
||||
icon: building
|
||||
keywords: ["enterprise", "deployment", "cluster"]
|
||||
---
|
||||
|
||||
@@ -16,5 +16,5 @@ Compliance posture (certifications, data residency, encryption at rest and in
|
||||
transit, audit logging) is a property of a deployment, so it is documented
|
||||
with LanceDB Enterprise.
|
||||
|
||||
For embedded LanceDB, these are properties of the object store you point it at
|
||||
For LanceDB Embedded, these are properties of the object store you point it at
|
||||
and the process you run it in; LanceDB adds no layer of its own.
|
||||
|
||||
@@ -16,9 +16,9 @@ Declaring it is cheap on any table size, because nothing is evaluated until
|
||||
[a refresh](/feature-engineering/refresh-column) asks for it.
|
||||
|
||||
<Note>
|
||||
In the 0.41 SDK release, local computed-column declaration and refresh require a
|
||||
In the 0.41 SDK release, computed-column declaration and refresh in LanceDB Embedded require a
|
||||
source build with the `experimental-functions` Cargo feature. Published Python
|
||||
and TypeScript packages leave it disabled. Remote Cloud and Enterprise operations
|
||||
and TypeScript packages leave it disabled. LanceDB Enterprise operations
|
||||
do not require this feature. Ordinary eager `add_columns` transforms remain available.
|
||||
</Note>
|
||||
|
||||
|
||||
@@ -70,7 +70,7 @@ changes incrementally for supported queries; other queries require a full rebuil
|
||||
See [refresh behavior](/feature-engineering/materialized-views#incremental) for
|
||||
version-specific details.
|
||||
|
||||
Enterprise runs materialization on remote compute. Use the SDK and refresh
|
||||
Enterprise runs materialization on the Enterprise service. Use the SDK and refresh
|
||||
options supported by your server version; see
|
||||
[Enterprise refreshes](/feature-engineering/materialized-views#availability).
|
||||
|
||||
@@ -78,8 +78,8 @@ options supported by your server version; see
|
||||
|
||||
| | What it produces | Filled by | Available |
|
||||
| --- | --- | --- | --- |
|
||||
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | OSS and Enterprise |
|
||||
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | OSS and Enterprise |
|
||||
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | Embedded and Enterprise |
|
||||
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Embedded and Enterprise |
|
||||
| [Geneva](/geneva) | columns filled by Python | the `geneva` package | <Badge color="red">Enterprise</Badge> |
|
||||
|
||||
{/*
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Local creation and refresh require a source build with the experimental-functions Cargo feature. Enterprise runs the work on remote compute. See [Enterprise refreshes](#availability) for server compatibility requirements.
|
||||
Creation and refresh in LanceDB Embedded require a source build with the experimental-functions Cargo feature. Enterprise runs the work on the Enterprise service. See [Enterprise refreshes](#availability) for server compatibility requirements.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -21,10 +21,10 @@ filtered or transformed dataset that you can query, index and search repeatedly.
|
||||
Refresh the view after its source data changes.
|
||||
|
||||
<Note>
|
||||
In the 0.41 SDK release, local materialized-view creation and refresh require a
|
||||
In the 0.41 SDK release, materialized-view creation and refresh in LanceDB Embedded require a
|
||||
source build with the `experimental-functions` Cargo feature. Published Python
|
||||
and TypeScript packages leave it disabled. Opening, reading, listing and dropping
|
||||
existing views remain available, as do remote Cloud and Enterprise operations.
|
||||
existing views remain available, as do LanceDB Enterprise operations.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
@@ -84,7 +84,7 @@ when maintaining an existing pipeline.
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
Enterprise runs materialized-view creation and refresh on remote compute.
|
||||
Enterprise runs materialized-view creation and refresh on the Enterprise service.
|
||||
For SQL, see [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) and
|
||||
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view).
|
||||
|
||||
|
||||
@@ -16,9 +16,9 @@ new rows and recompute values whose inputs changed. The example below continues
|
||||
with the `doubled` column from the computed-column guide.
|
||||
|
||||
<Note>
|
||||
In the 0.41 SDK release, local declaration and refresh require a source build
|
||||
In the 0.41 SDK release, declaration and refresh in LanceDB Embedded require a source build
|
||||
with the `experimental-functions` Cargo feature. Published Python and TypeScript
|
||||
packages leave it disabled. Remote Cloud and Enterprise operations do not require
|
||||
packages leave it disabled. LanceDB Enterprise operations do not require
|
||||
this feature; existing computed-column data remains readable in default builds.
|
||||
</Note>
|
||||
|
||||
@@ -82,9 +82,9 @@ table.add_columns(
|
||||
|
||||
Both options apply to every computed output declared in that call, including
|
||||
registered Function outputs. Automatic refresh is enabled by default. When
|
||||
the interval is omitted, the server determines when to refresh. Local tables
|
||||
persist only an explicitly configured interval; automatic execution requires a
|
||||
Cloud or Enterprise server with its separate `column_refresh.enabled` switch
|
||||
the interval is omitted, the server determines when to refresh. LanceDB Embedded
|
||||
tables persist only an explicitly configured interval; automatic execution requires a
|
||||
LanceDB Enterprise deployment with its separate `column_refresh.enabled` switch
|
||||
turned on. That server switch is disabled in the base configuration.
|
||||
|
||||
The interval is a positive minimum delay after the last successful refresh
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: unavailable
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Enterprise supports catalog privileges when its administrator configures enforcement. Embedded LanceDB has no catalog authorization layer: access is whatever the object store allows.
|
||||
Enterprise supports catalog privileges when its administrator configures enforcement. LanceDB Embedded has no catalog authorization layer: access is whatever the object store allows.
|
||||
---
|
||||
|
||||
[Authentication](/enterprise/authentication) establishes who is calling.
|
||||
|
||||
+9
-6
@@ -29,13 +29,16 @@ For a deeper look at how this works in training pipelines, start with [Why Lance
|
||||
|
||||
## LanceDB suite {#lancedb-suite}
|
||||
|
||||
The LanceDB suite includes LanceDB OSS, an open-source embedded retrieval library, and LanceDB Enterprise,
|
||||
The LanceDB suite includes LanceDB Embedded (open source), a retrieval library, and LanceDB Enterprise,
|
||||
a multimodal lakehouse platform for the full AI data lifecycle.
|
||||
OSS is easy to set up on a local machine for search and regular-scale workflows. LanceDB Enterprise is built
|
||||
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
|
||||
It is easy to set up on a local machine for search and regular-scale workflows. LanceDB Enterprise is built
|
||||
for teams that need scale without building bespoke infrastructure for curation,
|
||||
feature engineering, search and retrieval, and efficient training data access.
|
||||
LanceDB Enterprise runs LanceDB Distributed: query execution, indexing and maintenance
|
||||
run on the Enterprise service, outside your application.
|
||||
|
||||

|
||||

|
||||
|
||||
## Why teams use LanceDB {#why-teams-use-lancedb}
|
||||
|
||||
@@ -74,11 +77,11 @@ feature engineering, search and retrieval, and efficient training data access.
|
||||
|
||||
## From local development to production scale {#from-local-development-to-production-scale}
|
||||
|
||||
LanceDB OSS and LanceDB Enterprise share the same Lance format and table model. Start locally with the embedded OSS
|
||||
library, then move to Enterprise when your team needs distributed scale, managed infrastructure, private deployment,
|
||||
LanceDB Embedded and LanceDB Enterprise share the same Lance format and table model. Start locally with
|
||||
LanceDB Embedded, then move to Enterprise when your team needs distributed scale, managed infrastructure, private deployment,
|
||||
or higher-throughput curation, feature engineering, search and retrieval, and training workflows.
|
||||
|
||||
### 1\. LanceDB OSS {#1-lancedb-oss}
|
||||
### 1\. LanceDB Embedded {#1-lancedb-oss}
|
||||
The fastest way to get started is the open-source embedded library, with client SDKs in Python, TypeScript
|
||||
and Rust. Run it locally in just a few steps, which lets you explore datasets, curate data, and run search and retrieval workloads
|
||||
for agents. Start here:
|
||||
|
||||
@@ -14,7 +14,7 @@ availability:
|
||||
import { PyFtsIndexAsync as FtsIndexAsync, PyFtsIndexCreate as FtsIndexCreate, PyFtsIndexNested as FtsIndexNested, PyFtsIndexWait as FtsIndexWait } from '/snippets/indexing.mdx';
|
||||
|
||||
LanceDB provides performant full-text search based on BM25, allowing you to incorporate keyword-based search in your retrieval solutions. This page shows
|
||||
examples on how to create and configure FTS indexes in LanceDB OSS and Enterprise, using the synchronous and asynchronous APIs.
|
||||
examples on how to create and configure FTS indexes in LanceDB Embedded and LanceDB Enterprise, using the synchronous and asynchronous APIs.
|
||||
|
||||
<Note>
|
||||
In LanceDB Enterprise, `create_fts_index` API returns immediately, but index building happens asynchronously.
|
||||
|
||||
@@ -65,7 +65,7 @@ LanceDB Enterprise supports incremental reindexing through an automated backgrou
|
||||
|
||||
The benefit of using LanceDB Enterprise is that it automates the reindexing process
|
||||
and operates continuously in the background, minimizing the impact on latency under high loads.
|
||||
In OSS, you must manually manage the reindexing cadence based on your data growth and performance needs.
|
||||
In LanceDB Embedded, you must manually manage the reindexing cadence based on your data growth and performance needs.
|
||||
|
||||
## Disk utilization {#disk-utilization}
|
||||
|
||||
@@ -73,7 +73,7 @@ Compaction by itself does not immediately free disk space, and can temporarily i
|
||||
compacted files are written before old-version files are deleted. Disk space is reclaimed when old versions
|
||||
are pruned during cleanup. Set retention only as low as your rollback and time-travel requirements allow.
|
||||
|
||||
If you need to reclaim space more aggressively in OSS, use a shorter retention window:
|
||||
If you need to reclaim space more aggressively in LanceDB Embedded, use a shorter retention window:
|
||||
<CodeGroup>
|
||||
```python Python icon=Python
|
||||
from datetime import timedelta
|
||||
@@ -109,11 +109,11 @@ GPU acceleration changes how the vector index is built, not the lifecycle after
|
||||
embedded library, rows appended later still need `optimize()` before they are part of the index; a
|
||||
deployment adds them in the background. `fast_search()` only searches indexed rows.
|
||||
|
||||
### Manual GPU indexing in LanceDB OSS {#gpu-manual-gpu-indexing-in-lancedb-oss}
|
||||
### Manual GPU indexing in LanceDB Embedded {#gpu-manual-gpu-indexing-in-lancedb-oss}
|
||||
|
||||
You can use the Python SDK to manually create the `IVF_PQ` index on a GPU. You'll need
|
||||
[PyTorch>2.0](https://pytorch.org/). Note that GPU-based indexing is currently only
|
||||
supported by the synchronous SDK in LanceDB OSS.
|
||||
supported by the synchronous SDK in LanceDB Embedded.
|
||||
|
||||
Specify the values `cuda` or `mps` (on Apple Silicon) for the `accelerator` parameter
|
||||
to enable GPU training on your device.
|
||||
|
||||
@@ -159,7 +159,7 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil
|
||||
The `distance_type` can be specified at search time in addition to build time. If
|
||||
you do this, set it to the same value used to build the vector index. If the search
|
||||
metric does not match the index metric, LanceDB cannot use the index and falls
|
||||
back to a costly flat search. On LanceDB Cloud, the brute-force KNN safety limit
|
||||
back to a costly flat search. On LanceDB Cloud (legacy hosted service), the brute-force KNN safety limit
|
||||
can reject that search on a large table instead of executing it. An Enterprise
|
||||
deployment applies the same limit only when it is configured to.
|
||||
|
||||
@@ -279,7 +279,7 @@ In LanceDB Enterprise, vector indexes are managed **automatically**. The system
|
||||
- GPU-based `IVF_PQ` builds are available on request. [Contact support](/deployment/support) to enable them for your deployment. Build time depends on dataset size, dimensions and GPU resources.
|
||||
|
||||
<Badge color="gray">Open-Source</Badge>
|
||||
LanceDB OSS users can manually create vector indexes by calling `table.create_index()`.
|
||||
LanceDB Embedded users can manually create vector indexes by calling `table.create_index()`.
|
||||
See the above sections for guidance on manually tuning index parameters as data changes.
|
||||
|
||||
<Note>
|
||||
@@ -290,7 +290,7 @@ In the embedded library the index is built before `create_index()` returns.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal
|
||||
Rows appended after an initial index build remain outside the index until refreshed manually (Embedded) or automatically (Enterprise). Normal
|
||||
search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
|
||||
fallback and searches only indexed rows.
|
||||
</Note>
|
||||
@@ -330,7 +330,7 @@ Each entry returned by `list_indices()` also carries detailed per-index metadata
|
||||
|
||||
|
||||
<Note>
|
||||
These fields are populated for local tables. In LanceDB Enterprise, `list_indices()` returns them except `num_unindexed_rows`; use `index_stats()` for index coverage.
|
||||
These fields are populated for LanceDB Embedded tables. In LanceDB Enterprise, `list_indices()` returns them except `num_unindexed_rows`; use `index_stats()` for index coverage.
|
||||
</Note>
|
||||
#### Custom Index Names
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Create and open tables in OSS and Enterprise.
|
||||
Create and open tables in LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import {
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Delete rows through the SDKs in OSS and Enterprise. Storage reclamation follows compaction and retention policy.
|
||||
Delete rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Storage reclamation follows compaction and retention policy.
|
||||
---
|
||||
|
||||
import { RsExample as RsDeleteRows } from '/snippets/ingestion_delete.mdx';
|
||||
@@ -70,7 +70,7 @@ Compaction rewrites the live data layout; cleanup removes eligible retained
|
||||
versions and files. Retained history can continue to reference data that no
|
||||
longer appears in the latest table version.
|
||||
|
||||
In OSS, `optimize()` provides maintenance operations. In Enterprise, managed
|
||||
In LanceDB Embedded, `optimize()` provides maintenance operations. In Enterprise, managed
|
||||
maintenance and retention follow the service configuration. Choose retention
|
||||
according to the history you need, rather than shortening it as part of a
|
||||
routine delete example. See [Compaction](/maintenance/compaction) and
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Create tables and write records through the SDKs in OSS and Enterprise. Enterprise also provides SQL write statements.
|
||||
Create tables and write records through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also provides SQL write statements.
|
||||
---
|
||||
|
||||
import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx';
|
||||
@@ -32,7 +32,7 @@ with a stable key when the incoming data should update existing records.
|
||||
|
||||
## Connect and prepare the data {#connect-to-lancedb}
|
||||
|
||||
Use the [connection guide](/basics/connect) to connect to OSS or Enterprise.
|
||||
Use the [connection guide](/basics/connect) to connect to LanceDB Embedded or LanceDB Enterprise.
|
||||
The SDK write operations in this section work with both. Enterprise also
|
||||
supports [SQL write statements](/sql#dml); an SDK filter expression does not
|
||||
require a separate SQL connection.
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Append records through the SDKs in OSS and Enterprise. For large Enterprise writes, see [large upload settings](#bulk-ingests-into-a-remote-table).
|
||||
Append records through the SDKs in LanceDB Embedded and LanceDB Enterprise. For large Enterprise writes, see [large upload settings](#bulk-ingests-into-a-remote-table).
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -167,7 +167,7 @@ Wait for the write operation's result, then check the expected records.
|
||||
Progress callbacks describe activity; they do not establish that a write
|
||||
succeeded. If a request fails after the server may have committed it, blindly
|
||||
repeating an append can duplicate rows. The client's automatic retries can do
|
||||
the same: on a remote table, the Python and TypeScript 0.40.0 clients resend an
|
||||
the same: on an Enterprise table, the Python and TypeScript 0.40.0 clients resend an
|
||||
append after a retryable status such as `504`, even when the server had already
|
||||
committed it, and `add()` still returns success. Reconcile using your record
|
||||
IDs, or choose an upsert policy designed for replay.
|
||||
@@ -176,9 +176,9 @@ IDs, or choose an upsert policy designed for replay.
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
This section applies only to remote tables (`db://` connections). Embedded (local) writes are unaffected.
|
||||
This section applies only to LanceDB Enterprise tables (`db://` connections). LanceDB Embedded writes are unaffected.
|
||||
|
||||
When you write to a remote LanceDB Enterprise table, the client sends the write as one HTTP `insert` request unless it splits the data into more than one partition, which it does for a large input it can measure or when you set [`write_parallelism`](#write-parallelism). A partitioned write goes up as a multipart write: the client cuts each partition into one or more `insert` parts and uploads them under a single upload id. Each part is streamed. Its size and duration still affect resource use and
|
||||
When you write to a LanceDB Enterprise table, the client sends the write as one HTTP `insert` request unless it splits the data into more than one partition, which it does for a large input it can measure or when you set [`write_parallelism`](#write-parallelism). A partitioned write goes up as a multipart write: the client cuts each partition into one or more `insert` parts and uploads them under a single upload id. Each part is streamed. Its size and duration still affect resource use and
|
||||
timeouts: it has to finish within the client read timeout while the server writes it to object storage, so on very large ingests an oversized part can run past that timeout and surface as:
|
||||
|
||||
```text
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Merge insert is available through the SDKs in OSS and Enterprise. Use it for key-based updates and inserts.
|
||||
Merge insert is available through the SDKs in LanceDB Embedded and LanceDB Enterprise. Use it for key-based updates and inserts.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -230,7 +230,7 @@ when changing the input used to generate it.
|
||||
|
||||
Rust's `when_matched_update_all_expr` and
|
||||
`when_not_matched_by_source_delete_expr` accept DataFusion expressions for
|
||||
local tables. Use the string predicate variants for Enterprise tables.
|
||||
LanceDB Embedded tables. Use the string predicate variants for Enterprise tables.
|
||||
|
||||
## Complete write workflow {#complete-example}
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Update rows through the SDKs in OSS and Enterprise. Enterprise also supports the SQL UPDATE statement.
|
||||
Update rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also supports the SQL UPDATE statement.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -71,7 +71,7 @@ Rust's `column` argument is an expression, so the string value includes SQL quot
|
||||
|
||||
In this separate example, Bob keeps his name and his login count becomes **21**.
|
||||
Python's `values_sql` and TypeScript's `valuesSql` accept expressions.
|
||||
These SDK expressions work with embedded OSS tables; they do not require an
|
||||
These SDK expressions work with LanceDB Embedded tables; they do not require an
|
||||
Enterprise SQL endpoint. Enterprise also offers [`UPDATE`](/sql/dml/update).
|
||||
|
||||
## Verify the affected data {#verify}
|
||||
|
||||
@@ -139,7 +139,7 @@ We get the response based on the transcript's contents and title as expected.
|
||||
|
||||
## Why this works well {#why-this-works-well}
|
||||
|
||||
To start, LanceDB OSS can run from a local directory, so transcript data can stay on your machine when you are using the OSS stack.
|
||||
To start, LanceDB Embedded can run from a local directory, so transcript data can stay on your machine.
|
||||
|
||||
- You do not need to maintain a separate transcript parser in your application code.
|
||||
- You do not need to hand-roll chunking and retrieval orchestration across multiple modules.
|
||||
|
||||
@@ -29,7 +29,7 @@ prompt, plus lexical (keyword) search over past sessions. Both are useful, but k
|
||||
misses paraphrases of what you originally typed, the exact thing you need when recalling a
|
||||
fact you phrased differently months ago.
|
||||
|
||||
LanceDB OSS is an embedded retrieval library, which makes it a natural fit here:
|
||||
LanceDB Embedded is a retrieval library that runs inside your application, which makes it a natural fit here:
|
||||
|
||||
- **No server to stand up**: it reads and writes a table on local disk, so the plugin ships
|
||||
as a dependency rather than a service to operate.
|
||||
|
||||
@@ -40,7 +40,7 @@ In this example, we will be fetching movie information from the [Open Movie Data
|
||||
|
||||
3. **Specify necessary credentials and/or embedding model details:**
|
||||
|
||||
To fetch data from the OMDb API, you will need to pass a valid API key into your pipeline. Depending on whether you're using LanceDB OSS or LanceDB Enterprise, you also may need to provide the necessary credentials to connect to the LanceDB instance. These can be pasted inside `.dlt/secrets.toml`.
|
||||
To fetch data from the OMDb API, you will need to pass a valid API key into your pipeline. Depending on whether you're using LanceDB Embedded or LanceDB Enterprise, you also may need to provide the necessary credentials to connect to the LanceDB instance. These can be pasted inside `.dlt/secrets.toml`.
|
||||
|
||||
dlt's LanceDB integration also allows you to automatically embed the data during ingestion. Depending on the embedding model chosen, you may need to paste the necessary credentials inside `.dlt/secrets.toml`:
|
||||
```toml
|
||||
@@ -52,7 +52,7 @@ In this example, we will be fetching movie information from the [Open Movie Data
|
||||
embedding_model = "all-MiniLM-L6-v2"
|
||||
[destination.lancedb.credentials]
|
||||
uri = ".lancedb"
|
||||
api_key = "api_key" # API key to connect to LanceDB Enterprise. Leave out if you are using LanceDB OSS.
|
||||
api_key = "api_key" # API key to connect to LanceDB Enterprise. Leave out if you are using LanceDB Embedded.
|
||||
embedding_model_provider_api_key = "embedding_model_provider_api_key" # Not needed for providers that don't need authentication (ollama, sentence-transformers).
|
||||
```
|
||||
See [here](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb#configure-the-destination) for more information and for a list of available models and model providers.
|
||||
|
||||
@@ -6,7 +6,7 @@ description: "Learn how to use the DuckDB-Lance extension to query Lance tables
|
||||
|
||||
LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables.
|
||||
|
||||
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for local tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
|
||||
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for LanceDB Embedded tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
|
||||
|
||||
|
||||
## Install {#install}
|
||||
|
||||
@@ -43,7 +43,7 @@ Run vector search and keep the results as a Polars `DataFrame` for further proce
|
||||
|
||||
## Work with LazyFrames {#work-with-lazyframes}
|
||||
|
||||
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for local tables only; with LanceDB Enterprise, `search(...).to_polars()` returns an eager DataFrame of query results.
|
||||
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for LanceDB Embedded tables only; with LanceDB Enterprise, `search(...).to_polars()` returns an eager DataFrame of query results.
|
||||
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyPlatformsPolarsLazyframe}
|
||||
|
||||
@@ -48,7 +48,7 @@ log from growing without bound:
|
||||
| --- | --- |
|
||||
| `prewarm_page_cache` | Loads table data into a query node's page cache ahead of a request needing it. |
|
||||
| `prewarm_index_cache` | The same, for index structures. |
|
||||
| `startup_prewarm_idx` | Runs cache prewarming as part of a plan executor's own startup, rather than on a schedule, when the deployment enables it. |
|
||||
| `startup_prewarm_idx` | Runs cache prewarming as part of an executor's own startup, rather than on a schedule, when the deployment enables it. |
|
||||
|
||||
## Registry housekeeping {#registry}
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ those versions referenced.
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `cleanup`. Distinct from `compact`: compaction
|
||||
merges files, cleanup removes them. `optimize()` in OSS runs both in one call;
|
||||
merges files, cleanup removes them. `optimize()` in LanceDB Embedded runs both in one call;
|
||||
Enterprise schedules them as separate jobs, and runs cleanup only when the
|
||||
deployment's configuration enables it.
|
||||
</Note>
|
||||
@@ -89,8 +89,9 @@ directly can corrupt current or historical snapshots. Upgrade all clients and
|
||||
maintenance workers to a version that understands Managed blobs before using
|
||||
them, and reopen handles created before that upgrade.
|
||||
|
||||
For local tables, `optimize()` performs compaction and cleanup. Enterprise runs
|
||||
these as background jobs; the remote Python table does not support `optimize()`.
|
||||
For LanceDB Embedded tables, `optimize()` performs compaction and cleanup.
|
||||
Enterprise runs these as background jobs; Python tables connected to Enterprise
|
||||
do not support `optimize()`.
|
||||
See [Background jobs](/maintenance/background-jobs) for the Enterprise path.
|
||||
|
||||
## Disk usage does not drop immediately {#disk}
|
||||
|
||||
@@ -24,7 +24,7 @@ Run optimization as regular table maintenance, particularly after many small
|
||||
writes or a large ingest.
|
||||
|
||||
{/*
|
||||
Draft for the OSS compaction SQ partition check release. Keep this section
|
||||
Draft for the LanceDB Embedded compaction SQ partition check release. Keep this section
|
||||
unpublished until that behavior ships.
|
||||
|
||||
### Recover from an oversized SQ partition {#oversized-sq-partition}
|
||||
@@ -62,7 +62,7 @@ needs to.
|
||||
<Note>
|
||||
**Enterprise job type:** `compact`. Runs automatically as data accumulates,
|
||||
separately from [`cleanup`](/maintenance/cleanup). The two are distinct jobs
|
||||
even though `optimize()` bundles both in one OSS call.
|
||||
even though `optimize()` bundles both in one LanceDB Embedded call.
|
||||
</Note>
|
||||
|
||||
## Retraining indexes {#retrain}
|
||||
@@ -113,7 +113,7 @@ Two things accumulate on a long-lived table as more and more data gets added to
|
||||
- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
|
||||
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
|
||||
|
||||
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
|
||||
In LanceDB Embedded, you run them yourself via `optimize()`, which bundles both into a single call:
|
||||
|
||||
```python Python icon="python"
|
||||
from datetime import timedelta
|
||||
@@ -129,7 +129,7 @@ until the first supported release is confirmed.
|
||||
|
||||
### Tune compaction for wide rows {#tune-wide-rows}
|
||||
|
||||
For an existing local table, derive a fragment row target from the expected
|
||||
For an existing LanceDB Embedded table, derive a fragment row target from the expected
|
||||
row size. These examples use rows averaging 1 KiB and a 64 KiB fragment
|
||||
target; choose sizes for your workload. Smaller scan batches and one compaction
|
||||
task at a time can also reduce memory use.
|
||||
@@ -148,7 +148,7 @@ separately when needed; Python's `optimize()` includes those steps.
|
||||
|
||||
TypeScript's `optimize()` does not currently expose the fragment row target,
|
||||
scan batch size, or compaction thread count. These tuning examples apply to
|
||||
Python and Rust local tables.
|
||||
Python and Rust tables in LanceDB Embedded.
|
||||
|
||||
Planning remains row based. The row target bounds later passes after existing
|
||||
oversized fragments have been rewritten; the first pass rewrites those
|
||||
|
||||
@@ -8,8 +8,9 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
OSS opens tables in a local directory or object storage. Enterprise connects
|
||||
to remote compute with a database URI and credentials.
|
||||
LanceDB Embedded opens tables in a local directory or object storage.
|
||||
LanceDB Enterprise connects with a database URI and credentials and runs
|
||||
operations with LanceDB Distributed.
|
||||
---
|
||||
|
||||
import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
|
||||
@@ -22,14 +23,15 @@ objects, and configure how your application reaches the data.
|
||||
|
||||
| URI | Offering | Where queries run |
|
||||
| --- | --- | --- |
|
||||
| `./.lancedb` | OSS, with local storage | In your application |
|
||||
| `s3://bucket/path`, `gs://bucket/path`, `az://container/path` | OSS, with object storage | In your application |
|
||||
| `db://my-database` | Enterprise | On remote compute |
|
||||
| `./.lancedb` | LanceDB Embedded, with local storage | In your application |
|
||||
| `s3://bucket/path`, `gs://bucket/path`, `az://container/path` | LanceDB Embedded, with object storage | In your application |
|
||||
| `db://my-database` | LanceDB Enterprise | On the Enterprise service (LanceDB Distributed) |
|
||||
|
||||
[Connect to LanceDB](/basics/connect) has examples for each connection type.
|
||||
A connection to object storage still uses the embedded library: your process
|
||||
reads and writes the bucket directly. Enterprise runs operations remotely and
|
||||
manages the supporting compute and maintenance.
|
||||
A connection to object storage still uses LanceDB Embedded: your process
|
||||
reads and writes the bucket directly. LanceDB Enterprise runs LanceDB
|
||||
Distributed: query execution, indexing and maintenance run on the Enterprise
|
||||
service, outside your application.
|
||||
|
||||
## Connect to Enterprise {#deployment}
|
||||
|
||||
@@ -60,7 +62,7 @@ manager. See [Authentication](/enterprise/authentication) for OAuth and
|
||||
|
||||
## Storage options {#storage-options}
|
||||
|
||||
For OSS tables in object storage, `storage_options` configures credentials,
|
||||
For LanceDB Embedded tables in object storage, `storage_options` configures credentials,
|
||||
endpoints and options for new tables. This example enables stable row IDs for
|
||||
tables created through the connection:
|
||||
|
||||
@@ -91,7 +93,7 @@ See [Storage](/deployment/storage) for provider credentials and other options.
|
||||
|
||||
## Read consistency {#consistency}
|
||||
|
||||
For OSS connections, `read_consistency_interval` controls how often reads check
|
||||
For LanceDB Embedded connections, `read_consistency_interval` controls how often reads check
|
||||
for table versions committed by other writers. For example, check at most once
|
||||
every five seconds:
|
||||
|
||||
@@ -112,7 +114,7 @@ const db = await lancedb.connect("./.lancedb", { readConsistencyInterval: 5 });
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
By default, an OSS table handle does not automatically pick up another
|
||||
By default, an Embedded table handle does not automatically pick up another
|
||||
process's writes. Reopen the table, check out its latest version, or configure
|
||||
a consistency interval when you need those updates. A zero interval checks on
|
||||
every read, with the cost of a metadata check.
|
||||
@@ -139,7 +141,7 @@ versions and lifecycle operations.
|
||||
|
||||
<Warning>
|
||||
Dropping a table removes its history as well as its current data. You cannot
|
||||
recover a dropped table through table versioning. The OSS `drop_database()`
|
||||
recover a dropped table through table versioning. The LanceDB Embedded `drop_database()`
|
||||
and `drop_all_tables()` operations remove all tables without a confirmation prompt.
|
||||
</Warning>
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: unavailable
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Enterprise supports Functions when enabled for your Enterprise cluster; the embedded library cannot register or run them.
|
||||
Enterprise supports Functions when enabled for your Enterprise cluster; LanceDB Embedded cannot register or run them.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -31,8 +31,8 @@ rather than pulling rows to your process.
|
||||
|
||||
Registering and running a function needs a LanceDB Enterprise deployment with
|
||||
Functions enabled, which they are not by default. On a
|
||||
local connection `create_function` raises `NotImplementedError`; to derive a
|
||||
column locally, declare a
|
||||
LanceDB Embedded connection `create_function` raises `NotImplementedError`; to
|
||||
derive a column there, declare a
|
||||
[SQL computed column](/feature-engineering/add-computed-column) instead.
|
||||
|
||||
<Note>
|
||||
@@ -282,14 +282,15 @@ as `bytes`; this feature does not add streamed outputs.
|
||||
|
||||
## Declare the environment {#environment}
|
||||
|
||||
The function runs remotely, so its dependencies are part of its definition:
|
||||
The function runs on the Enterprise service, not in your application, so its
|
||||
dependencies are part of its definition:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `pip` | requirements for the remote environment |
|
||||
| `pip` | requirements for the function's execution environment |
|
||||
| `conda`, `conda_channels` | conda packages instead of pip; channels in priority order |
|
||||
| `env` | environment variables |
|
||||
| `python_version` | remote Python major/minor, defaulting to yours |
|
||||
| `python_version` | Python major/minor for the execution environment, defaulting to yours |
|
||||
| `gpu` | whether every execution requires a GPU |
|
||||
|
||||
These are part of the **immutable function version**. Changing a dependency
|
||||
|
||||
@@ -78,8 +78,8 @@ That also means dropping one removes stored rows, unlike dropping a
|
||||
|
||||
## Enterprise compatibility {#deployment}
|
||||
|
||||
Enterprise supports creating and refreshing materialized views on remote
|
||||
compute. Use the SDK version supported by your deployment, or the
|
||||
Enterprise supports creating and refreshing materialized views on the
|
||||
Enterprise service. Use the SDK version supported by your deployment, or the
|
||||
[SQL interface](/sql/ddl/materialized-view). See
|
||||
[Enterprise refreshes](/feature-engineering/materialized-views#availability)
|
||||
for compatibility considerations when upgrading an existing pipeline.
|
||||
|
||||
@@ -22,7 +22,7 @@ import {
|
||||
} from '/snippets/connection.mdx';
|
||||
|
||||
LanceDB organizes Lance tables through a catalog-level abstraction defined by
|
||||
the *namespace spec*. LanceDB OSS is an embedded library for accessing those
|
||||
the *namespace spec*. LanceDB Embedded is an embedded library for accessing those
|
||||
tables, and LanceDB Enterprise is a multimodal lakehouse that provides shared
|
||||
compute and catalog services on top of the Lance format.
|
||||
If you're coming from traditional databases or lakehouses, you can think of a namespace as the catalog path that says where a table name lives.
|
||||
@@ -76,7 +76,7 @@ directory. Namespaces can nest inside it, as in this hierarchy:
|
||||
The hierarchy is logical, not a tree of folders: a directory namespace records it in a
|
||||
`__manifest` table in the root directory, and stores each table in a directory of its own there.
|
||||
|
||||
As a user of LanceDB OSS, you might never notice namespaces at first, because a plain connection
|
||||
As a user of LanceDB Embedded, you might never notice namespaces at first, because a plain connection
|
||||
uses only the root namespace, which is implicit. Connecting to this namespace is as simple as
|
||||
connecting to the catalog root:
|
||||
|
||||
@@ -165,7 +165,7 @@ println!("Created table: {}", table.name());
|
||||
</CodeGroup>
|
||||
|
||||
<Info>
|
||||
- For simple use cases in LanceDB OSS, you don't need to go too deep into namespaces.
|
||||
- For simple use cases in LanceDB Embedded, you don't need to go too deep into namespaces.
|
||||
- To integrate LanceDB with external catalogs, understand how each namespace implementation resolves table names and locations in your organization's setup.
|
||||
</Info>
|
||||
|
||||
|
||||
@@ -248,7 +248,7 @@ set to the exact version a model saw.
|
||||
<Warning>
|
||||
A shallow clone depends on the source's files continuing to exist. If the source
|
||||
is [compacted](/maintenance/compaction) past the retention window, the version
|
||||
the clone points at can be pruned. Local OSS connections support shallow clones
|
||||
the clone points at can be pruned. LanceDB Embedded connections support shallow clones
|
||||
only; `is_shallow=False` is not implemented. If the copy must outlive the
|
||||
source's history, copy its data into an independent table instead. A deployment
|
||||
that supports cloning also makes shallow clones only, and one whose catalog
|
||||
@@ -462,8 +462,9 @@ pylance. Pandas conversions that need the Lance scanner, including plain-scan
|
||||
`"lazy"` handles and legacy blobs, still require pylance. `to_arrow()`,
|
||||
`to_list()`, `to_batches()`, and the fetch methods do not require pylance.
|
||||
|
||||
`blob_mode="bytes"` requires a local table snapshot, so remote and in-memory
|
||||
tables raise `NotImplementedError`. On a remote table, `Table.to_pandas()` and
|
||||
`blob_mode="bytes"` requires a LanceDB Embedded table snapshot, so Enterprise
|
||||
and in-memory tables raise `NotImplementedError`. On an Enterprise table,
|
||||
`Table.to_pandas()` and
|
||||
`Table.to_arrow()` aren't supported, and a query's `to_pandas()` also raises in
|
||||
the default `"lazy"` mode because it would open `BlobFile` handles. Query with
|
||||
`blob_mode="descriptions"`, then read bytes with `fetch_blobs` or
|
||||
@@ -495,7 +496,7 @@ descriptors. Both `"bytes"` and `"descriptions"` hide automatic row IDs;
|
||||
explicitly requested row IDs remain visible.
|
||||
|
||||
Bytes mode has the same restrictions as `to_arrow(blob_mode="bytes")`,
|
||||
including no legacy blobs, blobs inside lists, remote tables, or namespace
|
||||
including no legacy blobs, blobs inside lists, Enterprise tables, or namespace
|
||||
`QueryTable` pushdown. If concurrent writers can overwrite the table, use a
|
||||
separate reader checked out to a fixed version before querying. Otherwise,
|
||||
an overwrite can invalidate the row IDs used to fetch bytes and raise
|
||||
@@ -686,7 +687,7 @@ between inline values and Managed objects.
|
||||
|
||||
Enterprise Functions can consume blob columns and
|
||||
return new blobs. Their output payloads are managed by the destination table.
|
||||
Local tables can also derive blob columns with
|
||||
LanceDB Embedded tables can also derive blob columns with
|
||||
[SQL computed columns](/feature-engineering/add-computed-column).
|
||||
|
||||
Deleting a row or replacing its blob does not immediately remove the old
|
||||
@@ -741,8 +742,8 @@ Write raw bytes to a V1 blob column:
|
||||
|
||||
#### 3\. Convert blob tables to pandas {#multimodal-3-convert-blob-tables-to-pandas}
|
||||
|
||||
On local tables with V1 blob columns, `to_pandas()` in Python accepts `blob_mode`
|
||||
(`"lazy"`, `"bytes"`, or `"descriptions"`). Remote tables raise
|
||||
On LanceDB Embedded tables with V1 blob columns, `to_pandas()` in Python accepts
|
||||
`blob_mode` (`"lazy"`, `"bytes"`, or `"descriptions"`). Enterprise tables raise
|
||||
`NotImplementedError`, and `"bytes"` is not supported on in-memory tables.
|
||||
Extra keyword arguments such as `split_blocks` or `self_destruct` are forwarded
|
||||
to the PyArrow / Lance pandas conversion:
|
||||
@@ -1537,7 +1538,7 @@ mixing in lower-level Lance file operations.
|
||||
**System Operations**
|
||||
|
||||
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
|
||||
In LanceDB OSS, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
|
||||
In LanceDB Embedded, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
|
||||
which is when old-version files are removed and disk space is reclaimed. A deployment prunes them in the background
|
||||
when its configuration enables [cleanup](/maintenance/cleanup); `optimize()` is not available there.
|
||||
|
||||
|
||||
@@ -7,7 +7,8 @@ icon: rocket
|
||||
import { PyConnect, TsConnect, RsConnect, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
|
||||
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartVectorSearch1, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1 } from '/snippets/quickstart.mdx';
|
||||
|
||||
This quickstart uses a local database and works without an account or API key.
|
||||
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
|
||||
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
|
||||
|
||||
Follow the steps below, or jump to the [complete examples](#code) to copy a runnable program.
|
||||
|
||||
@@ -154,7 +155,7 @@ Use the database URI, API key, and endpoint provided by your administrator or th
|
||||
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/connect#enterprise) for an example.
|
||||
</Note>
|
||||
|
||||
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
|
||||
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between Embedded and Distributed execution, see [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Connect directly to object storage instead">
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Reranking is available with OSS and Enterprise.
|
||||
Reranking is available with LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import {
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Query plans are available in OSS and Enterprise.
|
||||
Query plans are available in LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import { PyQueryPlan, TsQueryPlanBody, TsQueryPlanTail } from '/snippets/search_examples.mdx';
|
||||
@@ -26,7 +26,7 @@ the query and adds measurements, including execution time and rows processed.
|
||||
|
||||
Use the first to inspect query structure and the second to investigate work
|
||||
actually performed. Operator names and available measurements can differ
|
||||
between OSS and Enterprise. SQL provides [EXPLAIN and EXPLAIN ANALYZE](/sql/dql/explain).
|
||||
between LanceDB Embedded and LanceDB Enterprise. SQL provides [EXPLAIN and EXPLAIN ANALYZE](/sql/dql/explain).
|
||||
|
||||
<span id="try-both-methods-on-a-local-table"></span>
|
||||
|
||||
@@ -161,7 +161,7 @@ CoalesceBatchesExec: target_batch_size=1024
|
||||
|
||||
#### 5\. Data Retrieval {#5-data-retrieval}
|
||||
|
||||
- `RemoteTake` is the deployment's operator for this step; a local plan shows Lance's `Take` or a `LanceRead` over the selected row IDs, depending on the library version
|
||||
- `RemoteTake` is the deployment's operator for this step; a LanceDB Embedded plan shows Lance's `Take` or a `LanceRead` over the selected row IDs, depending on the library version
|
||||
- Handles efficient data retrieval from remote storage locations
|
||||
- Fetches specific rows and columns needed for the final output
|
||||
- Optimizes network bandwidth by only retrieving required data
|
||||
@@ -328,25 +328,26 @@ ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier
|
||||
|
||||
</Accordion>
|
||||
|
||||
### Distributed metrics on remote tables {#distributed-metrics-on-remote-tables}
|
||||
### Distributed metrics on LanceDB Enterprise {#distributed-metrics-on-remote-tables}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
When you call `analyze_plan` against a LanceDB Enterprise deployment that distributes its queries across a pool of workers, the query runs on those workers. By default the returned plan aggregates each operator's metrics into a single value, matching the local single-node output. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
|
||||
When you call `analyze_plan` against a LanceDB Enterprise deployment that distributes its queries across a pool of workers, the query runs on those workers. By default the returned plan aggregates each operator's metrics into a single value, matching the single-node output of LanceDB Embedded. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
|
||||
|
||||
| Mode | What it shows | When to use it |
|
||||
| :------------ | :---------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
|
||||
| `aggregate` | One summary value per operator (default). Same shape as local `analyze_plan`. | Everyday performance checks and comparisons against a local plan. |
|
||||
| `aggregate` | One summary value per operator (default). Same shape as Embedded `analyze_plan`. | Everyday performance checks and comparisons against an Embedded plan. |
|
||||
| `per_worker` | Metrics reported separately for each worker that participated in the query. | Diagnosing stragglers or uneven work distribution across the cluster. |
|
||||
| `full` | Both the aggregate summary and the per-worker breakdown. | Deep investigations where you need the totals *and* the per-worker detail in a single output. |
|
||||
|
||||
Python exposes this option through `distributed_metrics`. In TypeScript, pass
|
||||
the same mode to `analyzePlan()` as its `distributedMetrics` argument. Local
|
||||
Python queries ignore the distributed mode and return local measurements.
|
||||
the same mode to `analyzePlan()` as its `distributedMetrics` argument. Python
|
||||
queries on LanceDB Embedded ignore the distributed mode and return single-node
|
||||
measurements.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
# Show a per-worker breakdown for a remote query
|
||||
# Show a per-worker breakdown for a query on LanceDB Enterprise
|
||||
plan = (
|
||||
table.search(query_embed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Metadata filtering is available in OSS and Enterprise.
|
||||
Metadata filtering is available in LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import { PyFilterSearch, TsFilterSearchBody, TsFilterSearchTail } from '/snippets/search_examples.mdx';
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Full-text search is available in OSS and Enterprise.
|
||||
Full-text search is available in LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -131,7 +131,7 @@ tbl
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
OSS builds the index before the call returns. Enterprise builds it in the
|
||||
LanceDB Embedded builds the index before the call returns. Enterprise builds it in the
|
||||
background. Wait for completion with Python’s `table.wait_for_index(["text_idx"])`
|
||||
or TypeScript’s `await tbl.waitForIndex(["text_idx"], 300)` before querying it.
|
||||
Use your actual index name if you chose a different one.
|
||||
@@ -187,7 +187,7 @@ values rather than probabilities.
|
||||
|
||||
### Keeping the index up to date {#keeping-the-index-up-to-date}
|
||||
|
||||
In OSS, new rows can remain outside the FTS index until you optimize the table.
|
||||
In LanceDB Embedded, new rows can remain outside the FTS index until you optimize the table.
|
||||
Search includes those rows by scanning them, but that work can increase query
|
||||
latency. Call `optimize()` to update the index:
|
||||
|
||||
@@ -207,7 +207,7 @@ tbl.optimize(OptimizeAction::All).await?;
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Choose an OSS maintenance schedule based on write volume and query latency.
|
||||
Choose a LanceDB Embedded maintenance schedule based on write volume and query latency.
|
||||
Inspect index coverage with `table.index_stats(...)` and optimize when the
|
||||
unindexed data adds too much work. `.fast_search()` skips unindexed rows, so
|
||||
use it only when omitting recent records is acceptable.
|
||||
@@ -289,7 +289,7 @@ export LANCE_LANGUAGE_MODEL_HOME=/path/to/lance/language_models
|
||||
For example, `jieba/default` is resolved under `<model-home>/jieba/default/...`, `lindera/ipadic` under `<model-home>/lindera/ipadic/...`, and `lindera/ko-dic` under `<model-home>/lindera/ko-dic/...`.
|
||||
|
||||
Model files must be installed where indexing and queries run: on your machine
|
||||
for OSS, or on the Enterprise cluster for a remote table.
|
||||
for LanceDB Embedded, or on the Enterprise cluster for an Enterprise table.
|
||||
|
||||
<Note>
|
||||
Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. If you use another stemming language, such as Arabic, Greek, Romanian, Tamil, or Turkish, set `remove_stop_words=False` or pass `custom_stop_words`.
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Hybrid search is available in OSS and Enterprise.
|
||||
Hybrid search is available in LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
|
||||
@@ -84,9 +84,9 @@ import { Utf8 } from "apache-arrow";
|
||||
</CodeGroup>
|
||||
|
||||
### 2\. Connect to LanceDB {#2-connect-to-lancedb}
|
||||
Choose a local OSS connection or use your Enterprise connection details.
|
||||
Choose a LanceDB Embedded connection or use your Enterprise connection details.
|
||||
|
||||
<Badge color="green">OSS</Badge>
|
||||
<Badge color="green">Embedded</Badge>
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
@@ -389,7 +389,7 @@ which relevant rows reach the candidate set before evaluating how a reranker
|
||||
orders them. Record latency as well as relevance: increasing candidate counts
|
||||
or applying a model-based reranker adds work.
|
||||
|
||||
Reranking runs in the application for both OSS and Enterprise connections.
|
||||
Reranking runs in the application for both Embedded and Enterprise connections.
|
||||
See [Reranking](/reranking) for model-free fusion, model-based rerankers,
|
||||
custom implementations and score interpretation.
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Vector, full-text and hybrid search are available in OSS and Enterprise.
|
||||
Vector, full-text and hybrid search are available in LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import { PyVectorSearch, TsVectorSearchBody, TsVectorSearchTail } from '/snippets/search_examples.mdx';
|
||||
@@ -106,9 +106,9 @@ in [Data loading](/training/data-loading). The SDK buffers Enterprise
|
||||
query responses before exposing batches. Set a query limit rather than relying
|
||||
on batch iteration alone to bound memory.
|
||||
|
||||
Direct Lance dataset access through Python's `table.to_lance()` is an OSS
|
||||
interface. Use the SDK query builders for reads that need to work with both
|
||||
OSS and Enterprise.
|
||||
Direct Lance dataset access through Python's `table.to_lance()` is a LanceDB
|
||||
Embedded interface. Use the SDK query builders for reads that need to work with
|
||||
both LanceDB Embedded and LanceDB Enterprise.
|
||||
|
||||
## Investigate a slow query {#diagnostics}
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Vector search is available in OSS and Enterprise.
|
||||
Vector search is available in LanceDB Embedded and LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -131,7 +131,7 @@ Without a vector index, searches use `l2` by default. You can specify `cosine` o
|
||||
|
||||
When a vector index exists, the search metric must match the metric used to build
|
||||
the index. If you explicitly request a different metric, LanceDB cannot use that
|
||||
index and falls back to an exhaustive (brute-force) search. On LanceDB Cloud, the
|
||||
index and falls back to an exhaustive (brute-force) search. On LanceDB Cloud (legacy hosted service), the
|
||||
brute-force KNN safety limit can reject that search on a large table instead of
|
||||
executing it. LanceDB Enterprise doesn't have the limit.
|
||||
|
||||
@@ -623,7 +623,7 @@ tbl.create_index(metric="cosine", vector_column_name="vector")
|
||||
**Indexing matters more for multivector tables than for single-vector ones.**
|
||||
|
||||
A brute-force scan over a multivector column has to compare every query vector to every document vector in every row, so the cost grows with both the row count and the number of vectors per row.
|
||||
In LanceDB OSS, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.
|
||||
In LanceDB Embedded, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.
|
||||
|
||||
On LanceDB Cloud, the brute-force KNN safety check applies a stricter row threshold to multivector columns; roughly 10× lower than for single-vector columns. So an unindexed multivector table can be rejected with a "vector search would use brute-force KNN" error well before a comparable single-vector table would. LanceDB Enterprise doesn't have the limit. Build the index before you start hitting it from production traffic, even if the dataset is small enough that you'd skip indexing for a single-vector workload.
|
||||
</Info>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="800" height="676" viewBox="0 0 800 676" role="img" aria-labelledby="title desc">
|
||||
<title id="title">LanceDB OSS and Enterprise architecture</title>
|
||||
<desc id="desc">OSS runs the application and embedded library in one process with local disk or object storage. Enterprise separates query serving, read execution and caching, and indexing and background workers. The control plane governs the service. Enterprise persists tables in object storage. Both offerings use the Lance format, without automatically sharing storage.</desc>
|
||||
<title id="title">LanceDB Embedded and Enterprise architecture</title>
|
||||
<desc id="desc">LanceDB Embedded runs the application and embedded library in one process with local disk or object storage. Enterprise separates query serving, read execution and caching, and indexing and background workers. The control plane governs the service. Enterprise persists tables in object storage. Both offerings use the Lance format, without automatically sharing storage.</desc>
|
||||
<defs>
|
||||
<marker id="arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto" markerUnits="userSpaceOnUse"><path d="M1 1 7 4 1 7" fill="none" stroke="#596776" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"/></marker>
|
||||
<style>
|
||||
@@ -15,7 +15,7 @@
|
||||
<rect width="800" height="676" rx="20" fill="#ffffff"/>
|
||||
|
||||
<rect x="12" y="12" width="380" height="580" rx="16" fill="#f3f6f9" stroke="#d4dce6"/>
|
||||
<text class="title" x="36" y="53">LanceDB OSS</text>
|
||||
<text class="title" x="36" y="53">LanceDB Embedded</text>
|
||||
<text class="note" x="36" y="83">Embedded library</text>
|
||||
<rect x="36" y="110" width="332" height="296" rx="10" fill="#ffffff" stroke="#c4cfdd"/>
|
||||
<text class="label" x="202" y="164" text-anchor="middle">Application +</text>
|
||||
|
||||
|
Before Width: | Height: | Size: 4.3 KiB After Width: | Height: | Size: 4.3 KiB |
@@ -1,6 +1,6 @@
|
||||
<svg width="2048" height="540" viewBox="0 0 2048 540" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="title desc">
|
||||
<title id="title">LanceDB suite</title>
|
||||
<desc id="desc">LanceDB OSS supports search. LanceDB Enterprise is a multimodal lakehouse for curation, feature engineering, search and retrieval, and training. Both are built on the Lance open lakehouse format for multimodal AI.</desc>
|
||||
<desc id="desc">LanceDB Embedded supports search. LanceDB Enterprise is a multimodal lakehouse for curation, feature engineering, search and retrieval, and training. Both are built on the Lance open lakehouse format for multimodal AI.</desc>
|
||||
<defs>
|
||||
<linearGradient id="enterpriseGradient" x1="512" y1="90" x2="2020" y2="90" gradientUnits="userSpaceOnUse">
|
||||
<stop offset="0" stop-color="#E9C9F6"/>
|
||||
@@ -54,7 +54,7 @@
|
||||
<circle cx="94" cy="126" r="9"/>
|
||||
<circle cx="126" cy="126" r="9"/>
|
||||
</g>
|
||||
<text x="162" y="117" style="fill:#241712 !important;" font-family="Inter, Arial, sans-serif" font-size="44" font-weight="760">LanceDB OSS</text>
|
||||
<text x="162" y="113" style="fill:#241712 !important;" font-family="Inter, Arial, sans-serif" font-size="34" font-weight="760">LanceDB Embedded</text>
|
||||
<text x="260" y="291" text-anchor="middle" style="fill:#241712 !important;" font-family="Inter, Arial, sans-serif" font-size="42" font-weight="500">Search</text>
|
||||
|
||||
<g style="fill:#241712 !important;">
|
||||
|
||||
|
Before Width: | Height: | Size: 5.8 KiB After Width: | Height: | Size: 5.9 KiB |
@@ -63,18 +63,18 @@ scoped to that branch. The [comparison table](#branches-vs-tags-vs-versions) at
|
||||
the end of this page lays out when to reach for each.
|
||||
|
||||
<Note>
|
||||
Branches are supported on local and namespace-backed tables in LanceDB OSS, as
|
||||
well as on LanceDB Enterprise (remote) tables.
|
||||
Branches are supported on local and namespace-backed tables in LanceDB Embedded,
|
||||
as well as on LanceDB Enterprise tables.
|
||||
</Note>
|
||||
|
||||
## Connect to a table {#connect-to-a-table}
|
||||
|
||||
The branch API is identical no matter how you connect: only the connection
|
||||
itself differs between OSS and Enterprise. Establish a connection (`db`) and open
|
||||
itself differs between LanceDB Embedded and LanceDB Enterprise. Establish a connection (`db`) and open
|
||||
a `table` (see [Ingesting data](/ingestion)), then use the same branch calls
|
||||
in every example that follows.
|
||||
|
||||
### LanceDB OSS {#lancedb-oss}
|
||||
### LanceDB Embedded {#lancedb-oss}
|
||||
|
||||
Point LanceDB at a local directory (or an object-storage URI) to use it as an
|
||||
embedded library.
|
||||
@@ -99,7 +99,7 @@ embedded library.
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
Branching on LanceDB Enterprise works the same way as on OSS, apart from the differences
|
||||
Branching on LanceDB Enterprise works the same way as on LanceDB Embedded, apart from the differences
|
||||
noted below, but you connect to your
|
||||
Enterprise deployment with a `db://` URI, an API key, and your
|
||||
region and endpoint. Replace the placeholders below, including `us-east-1` if your
|
||||
@@ -282,7 +282,7 @@ inspects what a branch has changed relative to `main`, and `cherry_pick`
|
||||
place, without reissuing the branch's writes.
|
||||
|
||||
<Info>
|
||||
`diff` and `cherry_pick` work on Enterprise (remote) tables only. On a local
|
||||
`diff` and `cherry_pick` work on LanceDB Enterprise tables only. On an Embedded
|
||||
table they fail as not supported: Python raises `NotImplementedError`, and
|
||||
TypeScript and Rust return an error. A cherry-pick promotes added columns; use
|
||||
the [upsert](#upsert-selected-branch-rows-into-main) or rerun patterns above for
|
||||
|
||||
@@ -7,8 +7,9 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Configure read freshness in OSS and Enterprise. OSS checks for table updates;
|
||||
Enterprise uses the connection setting to tighten its configured freshness bound.
|
||||
Configure read freshness in LanceDB Embedded and LanceDB Enterprise. Embedded
|
||||
checks for table updates; Enterprise uses the connection setting to tighten its
|
||||
configured freshness bound.
|
||||
---
|
||||
|
||||
import { PyReadConsistency } from '/snippets/read_consistency.mdx';
|
||||
@@ -26,11 +27,12 @@ for the experiment.
|
||||
|
||||
## Visibility of writes {#visibility}
|
||||
|
||||
In **OSS**, a table handle sees its own committed writes. By default, it does
|
||||
In **LanceDB Embedded**, a table handle sees its own committed writes. By default, it does
|
||||
not automatically check for changes made through other handles or processes.
|
||||
Use a read consistency interval or `checkout_latest()` to pick up those changes.
|
||||
|
||||
In **Enterprise**, readers use the service's configured freshness policy.
|
||||
In **LanceDB Distributed** (provided by LanceDB Enterprise), readers use the
|
||||
service's configured freshness policy.
|
||||
A connection can request fresher data. When a read must include a completed
|
||||
write, use a zero interval or call `checkout_latest()` after the write, with a
|
||||
client and server that support these freshness controls.
|
||||
@@ -40,13 +42,13 @@ client and server that support these freshness controls.
|
||||
Set `read_consistency_interval` when connecting. This controls when reads
|
||||
pick up newer table versions; it does not change how writes commit.
|
||||
|
||||
| Setting | OSS | Enterprise |
|
||||
| Setting | Embedded | Distributed |
|
||||
| --- | --- | --- |
|
||||
| **Unset** | Keep the open table version until explicitly refreshed or changed by that handle. | Use the service's configured freshness policy. |
|
||||
| **Zero** | Check for the latest committed version on every read. | Request the latest committed version on every read. |
|
||||
| **Non-zero interval** | Check for updates on a read once the interval since the last check has elapsed. | Request data no older than the interval allows; the service may return fresher data. |
|
||||
|
||||
`checkout_latest()` explicitly refreshes an OSS table handle. On Enterprise it
|
||||
`checkout_latest()` explicitly refreshes an Embedded table handle. On Distributed it
|
||||
asks subsequent reads on that handle to reflect data at least as recent as the
|
||||
refresh request. To read a fixed historical version instead, use
|
||||
[table versioning](/management/table#versioning).
|
||||
|
||||
@@ -59,7 +59,7 @@ for batch in dataloader:
|
||||
The `StreamingDataset` wraps a LanceDB `Table` and yields plain Python dictionaries by default. Whether iterating directly or through a PyTorch `DataLoader`, only one iterator can be simultaneously active on a `StreamingDataset` instance.
|
||||
|
||||
<Note>
|
||||
`StreamingDataset` is built on the `permutation` API and works with both LanceDB OSS and LanceDB Enterprise. The underlying table data can live on local disk or object storage.
|
||||
`StreamingDataset` is built on the `permutation` API and works with both LanceDB Embedded and LanceDB Enterprise. The underlying table data can live on local disk or object storage.
|
||||
</Note>
|
||||
|
||||
To understand the wide range of scenarios handled by `StreamingDataset`, consider a model trained with stochastic gradient descent (SGD) and distributed data parallelism (DDP). In this example, we need to load batches onto multiple GPUs
|
||||
|
||||
@@ -238,7 +238,7 @@ described above.
|
||||
|
||||
## Using multiple DataLoader workers {#using-multiple-dataloader-workers}
|
||||
|
||||
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables in LanceDB Enterprise are picklable, so each worker reopens the table after it starts. A local table is not, so wrap it in a `Permutation`, as below, to read it from several workers.
|
||||
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables in LanceDB Enterprise are picklable, so each worker reopens the table after it starts. A LanceDB Embedded table is not, so wrap it in a `Permutation`, as below, to read it from several workers.
|
||||
|
||||
Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and became the default on Linux and other supported POSIX platforms in Python
|
||||
3.14. macOS and Windows default to `spawn`. See
|
||||
@@ -263,9 +263,9 @@ dataloader = torch.utils.data.DataLoader(
|
||||
)
|
||||
```
|
||||
|
||||
### Remote tables in DataLoader workers {#remote-tables-in-dataloader-workers}
|
||||
### Enterprise tables in DataLoader workers {#remote-tables-in-dataloader-workers}
|
||||
|
||||
Remote LanceDB Enterprise tables (`db://...`) work the same way: workers reopen the table from the pickled connection state. Replace the URI, API key, region and endpoint below with your deployment details. In application code, load credentials from environment variables or a secrets manager.
|
||||
LanceDB Enterprise tables (`db://...`) work the same way: workers reopen the table from the pickled connection state. Replace the URI, API key, region and endpoint below with your deployment details. In application code, load credentials from environment variables or a secrets manager.
|
||||
|
||||
```py Python icon=Python
|
||||
import lancedb
|
||||
|
||||
@@ -6,7 +6,7 @@ description: "Use LanceDB as the retrieval layer for NVIDIA RAG Blueprint with a
|
||||
|
||||
## What this tutorial shows {#what-this-tutorial-shows}
|
||||
|
||||
If you are using [NVIDIA RAG Blueprints](https://build.nvidia.com/blueprints) and want to evaluate LanceDB in that stack, this tutorial gives you a concrete starting point. It shows how to use LanceDB as the retrieval layer for a Docker-based NVIDIA RAG deployment with a small, script-driven reference integration where LanceDB OSS is embedded directly in the NVIDIA containers, the collection is prepared ahead of time, and the RAG server retrieves from it for search and generation. The example is intentionally retrieval-only, but it also includes hybrid search and reranker selection so you can see how LanceDB fits into a realistic NVIDIA retrieval workflow.
|
||||
If you are using [NVIDIA RAG Blueprints](https://build.nvidia.com/blueprints) and want to evaluate LanceDB in that stack, this tutorial gives you a concrete starting point. It shows how to use LanceDB as the retrieval layer for a Docker-based NVIDIA RAG deployment with a small, script-driven reference integration where LanceDB Embedded runs directly in the NVIDIA containers, the collection is prepared ahead of time, and the RAG server retrieves from it for search and generation. The example is intentionally retrieval-only, but it also includes hybrid search and reranker selection so you can see how LanceDB fits into a realistic NVIDIA retrieval workflow.
|
||||
|
||||
<Tip>
|
||||
The runnable example for this tutorial lives in the
|
||||
@@ -31,7 +31,7 @@ yet teach NVIDIA's ingestor how to write new documents into LanceDB automaticall
|
||||
|
||||
## Deployment model {#deployment-model}
|
||||
|
||||
This reference integration uses **LanceDB OSS as an embedded retrieval library**, not as a separate
|
||||
This reference integration uses **LanceDB Embedded as an embedded retrieval library**, not as a separate
|
||||
database service. In practice, `APP_VECTORSTORE_NAME` is set to `lancedb`, `APP_VECTORSTORE_URL`
|
||||
points to a local filesystem path inside the NVIDIA containers, the LanceDB collection is prepared
|
||||
ahead of time, and the NVIDIA RAG server loads the LanceDB adapter to retrieve directly from that
|
||||
|
||||
+3
-3
@@ -2,9 +2,9 @@
|
||||
|
||||
## Configuration and Initialization
|
||||
|
||||
### LanceDB Cloud
|
||||
### LanceDB Cloud (legacy)
|
||||
|
||||
For LanceDB Cloud, use the simplified builder API:
|
||||
For LanceDB Cloud, the legacy hosted service, use the simplified builder API:
|
||||
|
||||
```java
|
||||
import com.lancedb.LanceDbNamespaceClientBuilder;
|
||||
@@ -19,7 +19,7 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
|
||||
### LanceDB Enterprise
|
||||
|
||||
For Enterprise deployments, use your custom endpoint:
|
||||
For LanceDB Enterprise deployments, use your custom endpoint:
|
||||
|
||||
```java
|
||||
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
|
||||
@@ -22,7 +22,7 @@ import java.util.Optional;
|
||||
/**
|
||||
* Util class to help construct a {@link LanceNamespace} for LanceDB.
|
||||
*
|
||||
* <p>For LanceDB Cloud, use the simplified builder API:
|
||||
* <p>For LanceDB Cloud (legacy hosted service), use the simplified builder API:
|
||||
*
|
||||
* <pre>{@code
|
||||
* import org.lance.namespace.LanceNamespace;
|
||||
|
||||
@@ -27,7 +27,7 @@ import java.io.IOException;
|
||||
import java.io.UncheckedIOException;
|
||||
|
||||
/**
|
||||
* Minimal HTTP client for LanceDB Cloud and Enterprise routes that the Lance Namespace
|
||||
* Minimal HTTP client for LanceDB Enterprise routes that the Lance Namespace
|
||||
* specification does not cover.
|
||||
*
|
||||
* <p>Most table operations reach LanceDB through {@link org.lance.namespace.LanceNamespace}, which
|
||||
|
||||
@@ -22,7 +22,7 @@ import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/**
|
||||
* The RTE write path for one LanceDB Cloud or Enterprise table.
|
||||
* The RTE write path for one LanceDB Enterprise table.
|
||||
*
|
||||
* <p>Installing an {@link RTEWriteSpec} routes {@code mergeInsert} upserts through Lance's RTE — an
|
||||
* LSM-style append — instead of the standard merge path. Rows land in an in-memory memtable, freeze
|
||||
|
||||
@@ -115,14 +115,14 @@ describe("remote connection", () => {
|
||||
...options,
|
||||
mode: "overwrite",
|
||||
}),
|
||||
).rejects.toThrow(/write_options.*remote/);
|
||||
).rejects.toThrow(/write_options.*LanceDB Distributed/);
|
||||
await expect(
|
||||
db.createEmptyTable(
|
||||
"test",
|
||||
new Schema([new Field("value", new Int64(), true)]),
|
||||
options,
|
||||
),
|
||||
).rejects.toThrow(/write_options.*remote/);
|
||||
).rejects.toThrow(/write_options.*LanceDB Distributed/);
|
||||
},
|
||||
);
|
||||
expect(requests).toEqual([]);
|
||||
@@ -140,7 +140,9 @@ describe("remote connection", () => {
|
||||
async (db) => {
|
||||
await expect(
|
||||
db.openTable("test", undefined, options),
|
||||
).rejects.toThrow(/not supported for remote/);
|
||||
).rejects.toThrow(
|
||||
/not supported for table access in LanceDB Distributed/,
|
||||
);
|
||||
},
|
||||
);
|
||||
expect(requests).toEqual([]);
|
||||
@@ -316,7 +318,7 @@ describe("remote connection", () => {
|
||||
],
|
||||
];
|
||||
for (const [run, operation] of cases) {
|
||||
let message = `LanceDBError: not supported: ${operation} is not supported for remote tables.`;
|
||||
let message = `LanceDBError: not supported: ${operation} is not supported in LanceDB Distributed.`;
|
||||
if (operation === "IVF_HNSW_PQ") {
|
||||
message += " Please use IVF_HNSW_SQ instead.";
|
||||
}
|
||||
@@ -463,7 +465,7 @@ describe("remote connection", () => {
|
||||
table.add([{ id: 1n }], {
|
||||
allowExternalBlobOutsideBases: true,
|
||||
}),
|
||||
).rejects.toThrow("only supported on local tables");
|
||||
).rejects.toThrow("only supported in LanceDB Embedded");
|
||||
expect(insertRequests).toBe(0);
|
||||
|
||||
await expect(table.add([{ id: 2n }])).resolves.toMatchObject({
|
||||
|
||||
@@ -4526,14 +4526,14 @@ describe.each(arrowVersions)(
|
||||
];
|
||||
const table = await db.createTable("prewarm_data_test", data);
|
||||
|
||||
// prewarmData is only supported on remote tables. We verify the call
|
||||
// prewarmData is only supported in LanceDB Distributed. We verify the call
|
||||
// is wired through napi and surfaces the expected error for both
|
||||
// arg shapes (undefined and string[]).
|
||||
await expect(table.prewarmData()).rejects.toThrow(
|
||||
"prewarm_data is currently only supported on remote tables",
|
||||
"prewarm_data is currently only supported in LanceDB Distributed",
|
||||
);
|
||||
await expect(table.prewarmData(["text"])).rejects.toThrow(
|
||||
"prewarm_data is currently only supported on remote tables",
|
||||
"prewarm_data is currently only supported in LanceDB Distributed",
|
||||
);
|
||||
});
|
||||
|
||||
@@ -5506,7 +5506,7 @@ describe("computed columns", () => {
|
||||
const db = await connect(tmpDir.name);
|
||||
const table = await db.createTable("errors_local", [{ x: 1 }]);
|
||||
await expect(table.functionErrors()).rejects.toThrow(
|
||||
"LanceDB Cloud and Enterprise",
|
||||
"recorded only in LanceDB Distributed",
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -274,7 +274,7 @@ export class BlobFile {
|
||||
export type BlobReadOptions = {
|
||||
/**
|
||||
* Cancels the read. The call rejects with `signal.reason`, and the native
|
||||
* read stops, including in-flight requests to a remote table.
|
||||
* read stops, including in-flight requests to a LanceDB Distributed table.
|
||||
*/
|
||||
signal?: AbortSignal;
|
||||
};
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user