diff --git a/docs/web/basics/guarantees.mdx b/docs/web/basics/guarantees.mdx
index 47c7c8d7c..f830597f7 100644
--- a/docs/web/basics/guarantees.mdx
+++ b/docs/web/basics/guarantees.mdx
@@ -48,6 +48,10 @@ A long-lived connection **does not observe other processes' writes** until it
reopens the table or its `read_consistency_interval` elapses. The default is
right for a batch job reading a stable snapshot and wrong for a service
expected to see fresh data.
+
+This describes the embedded library. A table on a deployment is not pinned to
+the version it opened: other writers' commits become visible as the
+deployment's freshness setting allows.
Set `read_consistency_interval` when several processes write the same table and
@@ -68,8 +72,9 @@ file. The data is recoverable for as long as the version that contains it
exists.
-Retention is finite. [Compaction](/maintenance/compaction) prunes versions
-older than the retention window — **7 days by default** — and a tag or branch
+Retention is finite. [Cleanup](/maintenance/cleanup) prunes versions older
+than the retention window — **7 days by default** for `optimize()` in the
+embedded library, while a deployment's configuration sets its own — and a tag or branch
pointing at a pruned version no longer resolves. If you rely on reading old
versions, widen the window before running it, not after.
@@ -124,11 +129,11 @@ Worth stating plainly, because each is occasionally assumed:
| | |
| --- | --- |
| Multi-table transactions | A commit covers one table. There is no cross-table atomic write. |
-| Read-your-writes across processes | Governed by `read_consistency_interval`, not automatic. |
+| Read-your-writes across processes | Governed by `read_consistency_interval` and, on a deployment, its freshness setting; not automatic. |
| Unenforced primary keys | `SET UNENFORCED PRIMARY KEY` records intent; it does not enforce uniqueness. |
| Ordering of concurrent appends | Two appends both succeed; their relative row order is not defined. |
-| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction). |
-| Automatic compaction | Nothing runs in the background. `optimize()` runs when you call it. |
+| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction) in the embedded library, or until a deployment's background maintenance adds them. |
+| Automatic compaction | In the embedded library nothing runs in the background; `optimize()` runs when you call it. A deployment compacts automatically. |
The last two are the most common source of "why is this query slow" — see
[Limits](/basics/limits) for the bounds that pair with them.
diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx
index f135de8d4..fd9d3878e 100644
--- a/docs/web/basics/limits.mdx
+++ b/docs/web/basics/limits.mdx
@@ -10,8 +10,9 @@ Limits worth designing around, collected so you do not have to find them by
hitting them. Each one is enforced -- where a value is refused rather than
clamped, that is stated, because the two fail very differently.
-This page covers bounds that are the same for everyone. Deployment-specific
-quotas -- storage, throughput, concurrency -- come from your deployment.
+Some of these bounds apply only on a deployment, as their sections say. Quotas
+that a deployment sets -- storage, throughput, concurrency -- come from your
+deployment.
## Names {#names}
@@ -24,10 +25,14 @@ quotas -- storage, throughput, concurrency -- come from your deployment.
A name must not be empty, and must not contain `$` or `:` -- both are reserved
as path delimiters, so a name holding one could not be addressed unambiguously.
-Length is counted in characters, not bytes.
+Length is counted in characters, not bytes. These lengths are a deployment's
+limits; the embedded library does not limit name length.
## Jobs {#jobs}
+Enterprise These bounds belong to a deployment's job
+service. Jobs on a local database run in-process and keep no event history.
+
| | |
| --- | --- |
| `job.events()` default | 1,000 rows |
@@ -90,14 +95,15 @@ not another.
| | |
| --- | --- |
| SQL on a local database | unavailable; `execute_query` raises |
+| SQL on a deployment | needs its Flight SQL endpoint enabled and reachable |
| SQL methods in the TypeScript SDK | none; use a Flight SQL client |
| Function methods in the TypeScript SDK | none; use Python or SQL |
-| Vector search in SQL | no SQL form; use a query builder |
+| Vector search in SQL | [depends on the deployment](/basics/sql#no-vector-sql); the query builder always has it |
| Geospatial `st_*` functions | behind a build feature LanceDB does not ship |
| `tableExists` / `namespaceExists` in TypeScript | not present |
| Server-side jobs on a local database | run in-process; `events()` raises |
-See [SQL](/basics/sql) for why the first two hold.
+See [SQL](/basics/sql) for why the first three hold.
## Nothing here is a quota {#not-quotas}
diff --git a/docs/web/ingestion/create-table.mdx b/docs/web/ingestion/create-table.mdx
index fce23ac65..c7b6d0960 100644
--- a/docs/web/ingestion/create-table.mdx
+++ b/docs/web/ingestion/create-table.mdx
@@ -4,6 +4,13 @@ sidebarTitle: "Create Table"
description: "Start here: turn the data you already have into a LanceDB table."
icon: table
keywords: ["create table", "ingest", "arrow", "pandas", "polars", "pydantic", "schema"]
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Tables are created, opened and dropped in the embedded library and on a
+ deployment. On a deployment, reusing an existing table does not check its
+ schema against your data.
---
import {
@@ -94,8 +101,10 @@ You can change this behavior with two parameters that resolve the conflict in di
- **Idempotent open**: return the existing table without modifying it. Use when your
code may run more than once (notebooks, retries, init scripts) and you want to reuse
- the table on subsequent runs. The provided data is ignored, but the schema is
- validated against the existing table and a mismatch raises an error.
+ the table on subsequent runs. The provided data is ignored. In the embedded
+ library the schema is validated against the existing table and a mismatch
+ raises an error; a deployment returns the existing table without comparing
+ schemas.
- **Overwrite**: drop the existing table and create a new one with the provided data.
Use this for test fixtures or when you intentionally want to replace prior contents.
This permanently discards the old table's data.
@@ -343,5 +352,5 @@ Use the `drop_table()` method on the database to remove a table.
This permanently removes the table and is not recoverable, unlike deleting rows.
-By default, if the table does not exist an exception is raised. To suppress this,
-you can pass in `ignore_missing=True`.
+By default, if the table does not exist an exception is raised. To suppress this
+in Python on a local database, you can pass in `ignore_missing=True`.
diff --git a/docs/web/ingestion/delete.mdx b/docs/web/ingestion/delete.mdx
index 0e2a4fdfa..00518457c 100644
--- a/docs/web/ingestion/delete.mdx
+++ b/docs/web/ingestion/delete.mdx
@@ -3,6 +3,13 @@ title: "Delete rows"
sidebarTitle: "Delete"
description: "Remove rows by predicate, and reclaim their space."
icon: trash
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Rows are deleted in the embedded library and on a deployment. In the
+ embedded library you reclaim their space with `optimize()`; a deployment
+ compacts automatically and removes old versions as its configuration sets.
---
These examples use the table set up in
@@ -18,7 +25,7 @@ import {
} from '/snippets/tables.mdx';
-In SQL this is [`DELETE`](/sql/dml/delete).
+In SQL on a deployment, this is [`DELETE`](/sql/dml/delete).
## Delete rows {#delete-rows}
@@ -55,12 +62,12 @@ Expected table contents:
When rows are deleted, those rows are also excluded from the index segments, so indexed queries will not return them either. If ALL the rows are deleted (i.e., the table is emptied), ensure that you recreate the index after ingesting new data.
-To permanently remove deleted rows, you can optimize the table, which will run compaction and cleans up the soft-deleted rows, which frees up storage space.
+To permanently remove deleted rows, the table has to be compacted and its old versions cleaned up, which frees up storage space.
- In LanceDB OSS, compaction and cleanup are manual. Run `table.optimize()` regularly to free up disk space.
-- In LanceDB Enterprise, files aren't cleaned up by default. You can configure automatic compaction and cleanup behavior at cluster setup time to suit your organization's retention policy.
+- In LanceDB Enterprise, compaction runs automatically, and old versions are removed only when the deployment's configuration enables cleanup, with the retention period it sets. `optimize()` is not available there; see [Cleanup](/maintenance/cleanup).
-By default, table cleanup removes data up to 7 days ago. If you need to reclaim space from deleted rows more aggressively, manually call `table.optimize()` use a shorter retention window as follows:
+In OSS, `optimize()` removes versions older than 7 days by default. If you need to reclaim space from deleted rows more aggressively, call `table.optimize()` with a shorter retention window as follows:
diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx
index 2a8de6759..ad04657d7 100644
--- a/docs/web/ingestion/index.mdx
+++ b/docs/web/ingestion/index.mdx
@@ -4,6 +4,12 @@ sidebarTitle: "Overview"
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
icon: inbox
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Tables are created and written to in the embedded library and on a
+ deployment. The SQL forms of these writes need a deployment.
---
import {
@@ -30,7 +36,8 @@ writing to a table that exists. This page covers the first and introduces the
second.
-Every operation here has a SQL form -- see [DML](/sql#dml).
+On a deployment, every operation here except merge insert also has a SQL form --
+see [DML](/sql#dml).
Start by [creating a table](/ingestion/create-table); the rest of this
@@ -89,6 +96,9 @@ Or, connect to LanceDB Enterprise:
+For a deployment with its own endpoint, also pass that endpoint as
+`host_override` -- see [Connect to LanceDB](/basics/connect#enterprise).
+
In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.
diff --git a/docs/web/ingestion/insert.mdx b/docs/web/ingestion/insert.mdx
index 2d411eb3b..bc78e5509 100644
--- a/docs/web/ingestion/insert.mdx
+++ b/docs/web/ingestion/insert.mdx
@@ -3,6 +3,13 @@ title: "Insert data"
sidebarTitle: "Insert"
description: "Append rows to a table, including datasets too large to hold in memory."
icon: plus
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Rows are appended in the embedded library and on a deployment. Large writes
+ to a deployment are uploaded in parts, which the section on bulk ingests
+ explains how to tune.
---
Rows are appended with `table.add()`. It accepts the same shapes
@@ -20,7 +27,7 @@ import {
} from '/snippets/tables.mdx';
-In SQL this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side.
+In SQL on a deployment, this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side.
## Ingesting large datasets {#large-datasets}
diff --git a/docs/web/ingestion/merge-insert.mdx b/docs/web/ingestion/merge-insert.mdx
index 8bfb6084a..730ae885a 100644
--- a/docs/web/ingestion/merge-insert.mdx
+++ b/docs/web/ingestion/merge-insert.mdx
@@ -3,6 +3,13 @@ title: "Merge insert"
sidebarTitle: "Merge Insert"
description: "Reconcile incoming rows against a key: update what matches, insert what does not."
icon: code-merge
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Merge insert runs in the embedded library and on a deployment. A deployment
+ can be configured to refuse a merge that would scan too many unindexed
+ rows.
---
These examples use the table set up in
@@ -54,7 +61,7 @@ Read more about scalar indices in the [Scalar Index](/indexing/scalar-index/) gu
-If you see this HTTP 400 error from `merge_insert`: `Bad request: Merge insert cannot be performed because the number of unindexed rows exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying.
+A deployment can be configured to refuse a merge insert that would scan too many unindexed rows, with an HTTP 400 error such as `Merge insert cannot be performed because the number of un-indexed rows (...) exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying.
diff --git a/docs/web/ingestion/update.mdx b/docs/web/ingestion/update.mdx
index e77587ea8..5e6d33482 100644
--- a/docs/web/ingestion/update.mdx
+++ b/docs/web/ingestion/update.mdx
@@ -3,6 +3,11 @@ title: "Update rows"
sidebarTitle: "Update"
description: "Change values in rows that already exist, by predicate or SQL expression."
icon: pen
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Rows are updated by predicate in the embedded library and on a deployment.
---
These examples use the table set up in
@@ -18,7 +23,7 @@ import {
} from '/snippets/tables.mdx';
-In SQL this is [`UPDATE`](/sql/dml/update).
+In SQL on a deployment, this is [`UPDATE`](/sql/dml/update).
## Update rows {#update-rows}
@@ -79,4 +84,4 @@ Expected table contents:
See the [SQL queries](/sql#connecting) page for more information on the supported SQL syntax.
-When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards.
+When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows in the embedded library, consider triggering an index rebuild afterwards; a deployment adds updated rows back to its indexes in the background once enough have accumulated.
diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx
index 7f84b9131..05dd6f448 100644
--- a/docs/web/management/table.mdx
+++ b/docs/web/management/table.mdx
@@ -4,6 +4,13 @@ sidebarTitle: "Table"
description: "The object most operations act on: creating, opening, renaming, cloning and dropping one."
icon: table
keywords: ["table", "create_table", "open_table", "clone", "rename", "drop"]
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Tables are created, evolved, versioned and dropped in the embedded library
+ and on a deployment. Renaming, cloning, dropping every table and pruning old
+ versions differ between the two, as their sections say.
---
import {
@@ -121,7 +128,7 @@ This page covers the table as a catalog object. Getting data into one is
[Schema evolution](/management/table#schema).
-Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table).
+On a deployment, tables are also created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table).
## What a table carries {#carries}
@@ -156,10 +163,10 @@ frame, or Pydantic models. [Ingesting data](/ingestion) covers each, plus
`exist_ok` and `mode="overwrite"` for re-runnable scripts.
-An explicit schema is where nullability is decided. Creation is also the only
-point at which the SDK can give a table **stable row ids**, which a deployment
-may require on a [materialized view](/management/materialized-view) source. See
-[Limits](/basics/limits#schema).
+An explicit schema is where nullability is decided. In the embedded library,
+creation is also the only point at which a table can be given **stable row
+ids**; on a deployment, whether a new table gets them is the deployment's
+setting. See [Limits](/basics/limits#schema).
## Opening and listing {#open}
@@ -168,7 +175,6 @@ may require on a [materialized view](/management/materialized-view) source. See
```python Python icon="python"
tbl = db.open_table("documents")
db.table_names()
-db.table_exists("documents")
```
```typescript TypeScript icon="square-js"
@@ -179,7 +185,10 @@ await db.tableNames();
Each accepts `namespace_path` to act inside a [namespace](/management/namespace)
-rather than at the database root. TypeScript has no `tableExists`.
+rather than at the database root. Python's `table_exists` works only on a
+namespace connection, where it takes the table's full path as a list; local
+directory and deployment connections raise `NotImplementedError`. TypeScript
+has no `tableExists`.
## Renaming and cloning {#rename-clone}
@@ -194,6 +203,10 @@ db.clone_table(
)
```
+Renaming is not supported on a local directory connection. On a deployment,
+whether a table can be renamed depends on how the deployment stores its
+catalog.
+
A **shallow** clone -- the default -- references the source's data files rather
than copying them, so it is fast and cheap regardless of table size. It is the
practical way to branch an experiment off a known version, or to pin a training
@@ -204,13 +217,14 @@ A shallow clone depends on the source's files continuing to exist. If the source
is [compacted](/maintenance/compaction) past the retention window, the version
the clone points at can be pruned. Local OSS connections support shallow clones
only; `is_shallow=False` is not implemented. If the copy must outlive the
-source's history, copy its data into an independent table instead. For remote
-connections, confirm which clone modes your deployment supports.
+source's history, copy its data into an independent table instead. A deployment
+also makes shallow clones only.
-**Enterprise job type:** `drop_table`. Fences the table's directory with a
-marker and deletes its content in the background -- see
+**Enterprise job type:** `drop_table`. On a deployment that drops tables in the
+background, fences the table's directory with a marker and deletes its content
+as a job -- see
[Background jobs](/maintenance/background-jobs#teardown) for the namespace,
view, and function equivalent.
@@ -238,11 +252,14 @@ DROP TABLE documents
Dropping removes the table and **every version of it**.
[Versioning](/management/table#versioning) does not survive a drop, and neither
-`drop_all_tables` nor `drop_database` asks for confirmation.
+`drop_all_tables` nor `drop_database` asks for confirmation. Both work only on
+a local database; on a deployment connection the SDK raises instead.
On a large table the drop can take a while, so `drop_table_async` returns a
-[job](/basics/jobs) rather than blocking.
+[job](/basics/jobs) rather than blocking when the deployment drops tables in the
+background. In the embedded library, and on a deployment that drops inline, the
+returned job has already finished.
## The full surface {#surface}
@@ -1137,8 +1154,9 @@ mixing in lower-level Lance file operations.
**System Operations**
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
-In LanceDB OSS and Enterprise, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
-which is when old-version files are removed and disk space is reclaimed.
+In LanceDB OSS, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
+which is when old-version files are removed and disk space is reclaimed. A deployment prunes them in the background
+when its configuration enables [cleanup](/maintenance/cleanup); `optimize()` is not available there.
**Tagged versions are exempt from cleanup.** A version with a tag pointing at it is
retained regardless of age, and its files are not removed by `optimize()`. To make
diff --git a/docs/web/tables/branching.mdx b/docs/web/tables/branching.mdx
index d03ee8212..7af1faaa6 100644
--- a/docs/web/tables/branching.mdx
+++ b/docs/web/tables/branching.mdx
@@ -3,6 +3,13 @@ title: "Branches"
sidebarTitle: "Branch"
description: "Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main."
icon: "code-branch"
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Branches work in the embedded library and on a deployment. Comparing a
+ branch with `main` and promoting its new columns onto `main` is available
+ only on a deployment.
---
import {
PyConnect,
@@ -91,7 +98,8 @@ embedded library.
Enterprise
-Branching on LanceDB Enterprise works the same way as on OSS, but you connect to your
+Branching on LanceDB Enterprise works the same way as on OSS, apart from the differences
+noted below, but you connect to your
Enterprise deployment with a `db://` URI, an API key, and your
region. Once you have a connection, open a table and use the same branch calls in
every example that follows.
@@ -179,6 +187,8 @@ table. Both routes give you a writable handle tracking the branch's latest state
### Delete a branch {#delete-a-branch}
Deleting a branch removes it and its branch-local history; `main` is untouched.
+A deployment refuses to delete a branch whose columns have been promoted onto
+`main`, because `main` still depends on it.
Before deleting a branch, make sure you've retained any results you need — see
[Apply branch-tested changes to `main`](#apply-branch-tested-changes-to-main)
below.
@@ -244,6 +254,10 @@ your key column — `id` in this example. Here, `candidate` is the branch handle
+Against a deployment, the synchronous Python table raises for `to_arrow()`; read
+the branch's rows with a query instead -- see
+[Avoid materializing the whole table](/search#avoid-materializing-the-whole-table).
+
This operation transfers only the inserted or updated rows in its input. It does
not transfer branch history, schema changes, indexes, or branch-local deletions,
and it does not determine which rows changed after the branch was created.
@@ -363,9 +377,10 @@ deployment workflow. Because the indexes live on the branch, queries against
`main` never see a half-built index and are never slowed down by the branch's
build.
-This pattern is especially valuable on Enterprise
-deployments, where `main` is typically serving production traffic while you tune
-an index configuration on the side.
+This pattern is especially valuable on Enterprise deployments, where `main` is
+typically serving production traffic while you tune an index configuration on
+the side. A deployment builds the branch's indexes in the background, so wait
+for them before measuring.
diff --git a/docs/web/tables/consistency.mdx b/docs/web/tables/consistency.mdx
index dcf1a9ac6..66adfe854 100644
--- a/docs/web/tables/consistency.mdx
+++ b/docs/web/tables/consistency.mdx
@@ -3,6 +3,13 @@ title: Consistency
sidebarTitle: "Consistency"
description: Learn about consistency settings and versioning in LanceDB tables.
icon: "water"
+availability:
+ oss: available
+ enterprise: available
+ summary: >-
+ Read freshness can be set per connection in the embedded library and on a
+ deployment. On a deployment, the default freshness comes from its
+ configuration, and the connection setting can only make reads fresher.
---
import {
PyConsistencyStrong as ConsistencyStrong,
@@ -40,14 +47,14 @@ write operations are always consistent.
For remote tables (`db://` connections), `read_consistency_interval` is also
respected by the client. The interval is sent to the server as a freshness bound on each read:
-- **Unset (default)**: no freshness header is sent; reads use the server's cached view of the table.
+- **Unset (default)**: no freshness bound is sent; reads are as fresh as the deployment's own setting, described below.
- **Zero seconds**: every read asks the server for the latest committed version.
- **Non-zero interval**: reads accept data at least as fresh as `now - interval`.
In addition, after any write or after a `checkout_latest` / `restore` on a table handle, subsequent
reads on that same handle carry a freshness floor so you read your own writes without extra
configuration. The floor is the later of the configured interval and the moment of the last
-write or refresh, and it is shared across handles to the same table on the same connection.
+write or refresh. Each opened table handle keeps its own floor.
Each remote table handle also tracks the highest dataset version it has observed in a read
response and sends it back with every subsequent read, so successive reads on the same handle
@@ -59,7 +66,8 @@ of refreshing against storage, which raises per-read latency and cost.
In Enterprise deployments, the server-side default freshness is still
controlled by the cluster-level `weak_read_consistency_interval_seconds` parameter; the client setting
-tightens that bound on a per-connection basis.
+tightens that bound on a per-connection basis. When the parameter is unset, every read is strongly
+consistent; deployments installed from the Helm chart default it to 600 seconds.
30-60 seconds is a reasonable starting point for
@@ -112,7 +120,7 @@ For eventual consistency, use a non-zero interval:
-With the default unset interval, tables do not auto-refresh from other writers.
+In the embedded library, with the default unset interval, tables do not auto-refresh from other writers.
To manually check for updates, use `checkout_latest` / `checkoutLatest`: