docs: declare where ingestion and table pages apply

Creating, writing, updating, merging and deleting rows, managing tables,
branches and read consistency work in the embedded library and on a
deployment, and those pages now say so. The SQL forms they mention need
a deployment.

Where the offerings differ, the section says how: a deployment reuses an
existing table without comparing schemas, a merge insert can be refused
past a configured number of unindexed rows, space is reclaimed by
optimize() locally and by the deployment's compaction and configured
cleanup otherwise, renaming depends on the catalog, a deployment clones
shallowly only, drops all tables only locally, keeps freshness state per
table handle and defaults freshness to its own setting, and refuses to
delete a branch main depends on. table_exists works only on a namespace
connection.

Guarantees and limits span both offerings, so they carry section notes
rather than a label: reads, retention, index freshness and compaction
differ on a deployment, and the name and job limits are a deployment's.
This commit is contained in:
Brendan Clement
2026-10-08 21:31:42 -07:00
parent 027597d36d
commit 6a1645116e
11 changed files with 143 additions and 46 deletions
+10 -5
View File
@@ -48,6 +48,10 @@ A long-lived connection **does not observe other processes' writes** until it
reopens the table or its `read_consistency_interval` elapses. The default is
right for a batch job reading a stable snapshot and wrong for a service
expected to see fresh data.
This describes the embedded library. A table on a deployment is not pinned to
the version it opened: other writers' commits become visible as the
deployment's freshness setting allows.
</Warning>
Set `read_consistency_interval` when several processes write the same table and
@@ -68,8 +72,9 @@ file. The data is recoverable for as long as the version that contains it
exists.
<Warning>
Retention is finite. [Compaction](/maintenance/compaction) prunes versions
older than the retention window — **7 days by default** — and a tag or branch
Retention is finite. [Cleanup](/maintenance/cleanup) prunes versions older
than the retention window — **7 days by default** for `optimize()` in the
embedded library, while a deployment's configuration sets its own — and a tag or branch
pointing at a pruned version no longer resolves. If you rely on reading old
versions, widen the window before running it, not after.
</Warning>
@@ -124,11 +129,11 @@ Worth stating plainly, because each is occasionally assumed:
| | |
| --- | --- |
| Multi-table transactions | A commit covers one table. There is no cross-table atomic write. |
| Read-your-writes across processes | Governed by `read_consistency_interval`, not automatic. |
| Read-your-writes across processes | Governed by `read_consistency_interval` and, on a deployment, its freshness setting; not automatic. |
| Unenforced primary keys | `SET UNENFORCED PRIMARY KEY` records intent; it does not enforce uniqueness. |
| Ordering of concurrent appends | Two appends both succeed; their relative row order is not defined. |
| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction). |
| Automatic compaction | Nothing runs in the background. `optimize()` runs when you call it. |
| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction) in the embedded library, or until a deployment's background maintenance adds them. |
| Automatic compaction | In the embedded library nothing runs in the background; `optimize()` runs when you call it. A deployment compacts automatically. |
The last two are the most common source of "why is this query slow" — see
[Limits](/basics/limits) for the bounds that pair with them.
+11 -5
View File
@@ -10,8 +10,9 @@ Limits worth designing around, collected so you do not have to find them by
hitting them. Each one is enforced -- where a value is refused rather than
clamped, that is stated, because the two fail very differently.
This page covers bounds that are the same for everyone. Deployment-specific
quotas -- storage, throughput, concurrency -- come from your deployment.
Some of these bounds apply only on a deployment, as their sections say. Quotas
that a deployment sets -- storage, throughput, concurrency -- come from your
deployment.
## Names {#names}
@@ -24,10 +25,14 @@ quotas -- storage, throughput, concurrency -- come from your deployment.
A name must not be empty, and must not contain `$` or `:` -- both are reserved
as path delimiters, so a name holding one could not be addressed unambiguously.
Length is counted in characters, not bytes.
Length is counted in characters, not bytes. These lengths are a deployment's
limits; the embedded library does not limit name length.
## Jobs {#jobs}
<Badge color="red">Enterprise</Badge> These bounds belong to a deployment's job
service. Jobs on a local database run in-process and keep no event history.
| | |
| --- | --- |
| `job.events()` default | 1,000 rows |
@@ -90,14 +95,15 @@ not another.
| | |
| --- | --- |
| SQL on a local database | unavailable; `execute_query` raises |
| SQL on a deployment | needs its Flight SQL endpoint enabled and reachable |
| SQL methods in the TypeScript SDK | none; use a Flight SQL client |
| Function methods in the TypeScript SDK | none; use Python or SQL |
| Vector search in SQL | no SQL form; use a query builder |
| Vector search in SQL | [depends on the deployment](/basics/sql#no-vector-sql); the query builder always has it |
| Geospatial `st_*` functions | behind a build feature LanceDB does not ship |
| `tableExists` / `namespaceExists` in TypeScript | not present |
| Server-side jobs on a local database | run in-process; `events()` raises |
See [SQL](/basics/sql) for why the first two hold.
See [SQL](/basics/sql) for why the first three hold.
## Nothing here is a quota {#not-quotas}
+13 -4
View File
@@ -4,6 +4,13 @@ sidebarTitle: "Create Table"
description: "Start here: turn the data you already have into a LanceDB table."
icon: table
keywords: ["create table", "ingest", "arrow", "pandas", "polars", "pydantic", "schema"]
availability:
oss: available
enterprise: available
summary: >-
Tables are created, opened and dropped in the embedded library and on a
deployment. On a deployment, reusing an existing table does not check its
schema against your data.
---
import {
@@ -94,8 +101,10 @@ You can change this behavior with two parameters that resolve the conflict in di
- **Idempotent open**: return the existing table without modifying it. Use when your
code may run more than once (notebooks, retries, init scripts) and you want to reuse
the table on subsequent runs. The provided data is ignored, but the schema is
validated against the existing table and a mismatch raises an error.
the table on subsequent runs. The provided data is ignored. In the embedded
library the schema is validated against the existing table and a mismatch
raises an error; a deployment returns the existing table without comparing
schemas.
- **Overwrite**: drop the existing table and create a new one with the provided data.
Use this for test fixtures or when you intentionally want to replace prior contents.
This permanently discards the old table's data.
@@ -343,5 +352,5 @@ Use the `drop_table()` method on the database to remove a table.
</CodeGroup>
This permanently removes the table and is not recoverable, unlike deleting rows.
By default, if the table does not exist an exception is raised. To suppress this,
you can pass in `ignore_missing=True`.
By default, if the table does not exist an exception is raised. To suppress this
in Python on a local database, you can pass in `ignore_missing=True`.
+11 -4
View File
@@ -3,6 +3,13 @@ title: "Delete rows"
sidebarTitle: "Delete"
description: "Remove rows by predicate, and reclaim their space."
icon: trash
availability:
oss: available
enterprise: available
summary: >-
Rows are deleted in the embedded library and on a deployment. In the
embedded library you reclaim their space with `optimize()`; a deployment
compacts automatically and removes old versions as its configuration sets.
---
These examples use the table set up in
@@ -18,7 +25,7 @@ import {
} from '/snippets/tables.mdx';
<Note>
In SQL this is [`DELETE`](/sql/dml/delete).
In SQL on a deployment, this is [`DELETE`](/sql/dml/delete).
</Note>
## Delete rows {#delete-rows}
@@ -55,12 +62,12 @@ Expected table contents:
When rows are deleted, those rows are also excluded from the index segments, so indexed queries will not return them either. If ALL the rows are deleted (i.e., the table is emptied), ensure that you recreate the index after ingesting new data.
</Warning>
To permanently remove deleted rows, you can optimize the table, which will run compaction and cleans up the soft-deleted rows, which frees up storage space.
To permanently remove deleted rows, the table has to be compacted and its old versions cleaned up, which frees up storage space.
- In LanceDB OSS, compaction and cleanup are manual. Run `table.optimize()` regularly to free up disk space.
- In LanceDB Enterprise, files aren't cleaned up by default. You can configure automatic compaction and cleanup behavior at cluster setup time to suit your organization's retention policy.
- In LanceDB Enterprise, compaction runs automatically, and old versions are removed only when the deployment's configuration enables cleanup, with the retention period it sets. `optimize()` is not available there; see [Cleanup](/maintenance/cleanup).
By default, table cleanup removes data up to 7 days ago. If you need to reclaim space from deleted rows more aggressively, manually call `table.optimize()` use a shorter retention window as follows:
In OSS, `optimize()` removes versions older than 7 days by default. If you need to reclaim space from deleted rows more aggressively, call `table.optimize()` with a shorter retention window as follows:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
+11 -1
View File
@@ -4,6 +4,12 @@ sidebarTitle: "Overview"
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
icon: inbox
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
availability:
oss: available
enterprise: available
summary: >-
Tables are created and written to in the embedded library and on a
deployment. The SQL forms of these writes need a deployment.
---
import {
@@ -30,7 +36,8 @@ writing to a table that exists. This page covers the first and introduces the
second.
<Note>
Every operation here has a SQL form -- see [DML](/sql#dml).
On a deployment, every operation here except merge insert also has a SQL form --
see [DML](/sql#dml).
</Note>
Start by [creating a table](/ingestion/create-table); the rest of this
@@ -89,6 +96,9 @@ Or, connect to LanceDB Enterprise:
</CodeBlock>
</CodeGroup>
For a deployment with its own endpoint, also pass that endpoint as
`host_override` -- see [Connect to LanceDB](/basics/connect#enterprise).
In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.
<CodeGroup>
+8 -1
View File
@@ -3,6 +3,13 @@ title: "Insert data"
sidebarTitle: "Insert"
description: "Append rows to a table, including datasets too large to hold in memory."
icon: plus
availability:
oss: available
enterprise: available
summary: >-
Rows are appended in the embedded library and on a deployment. Large writes
to a deployment are uploaded in parts, which the section on bulk ingests
explains how to tune.
---
Rows are appended with `table.add()`. It accepts the same shapes
@@ -20,7 +27,7 @@ import {
} from '/snippets/tables.mdx';
<Note>
In SQL this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side.
In SQL on a deployment, this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side.
</Note>
## Ingesting large datasets {#large-datasets}
+8 -1
View File
@@ -3,6 +3,13 @@ title: "Merge insert"
sidebarTitle: "Merge Insert"
description: "Reconcile incoming rows against a key: update what matches, insert what does not."
icon: code-merge
availability:
oss: available
enterprise: available
summary: >-
Merge insert runs in the embedded library and on a deployment. A deployment
can be configured to refuse a merge that would scan too many unindexed
rows.
---
These examples use the table set up in
@@ -54,7 +61,7 @@ Read more about scalar indices in the [Scalar Index](/indexing/scalar-index/) gu
</Tip>
<Info>
If you see this HTTP 400 error from `merge_insert`: `Bad request: Merge insert cannot be performed because the number of unindexed rows exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying.
A deployment can be configured to refuse a merge insert that would scan too many unindexed rows, with an HTTP 400 error such as `Merge insert cannot be performed because the number of un-indexed rows (...) exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying.
</Info>
<Note>
+7 -2
View File
@@ -3,6 +3,11 @@ title: "Update rows"
sidebarTitle: "Update"
description: "Change values in rows that already exist, by predicate or SQL expression."
icon: pen
availability:
oss: available
enterprise: available
summary: >-
Rows are updated by predicate in the embedded library and on a deployment.
---
These examples use the table set up in
@@ -18,7 +23,7 @@ import {
} from '/snippets/tables.mdx';
<Note>
In SQL this is [`UPDATE`](/sql/dml/update).
In SQL on a deployment, this is [`UPDATE`](/sql/dml/update).
</Note>
## Update rows {#update-rows}
@@ -79,4 +84,4 @@ Expected table contents:
See the [SQL queries](/sql#connecting) page for more information on the supported SQL syntax.
</Note>
When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards.
When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows in the embedded library, consider triggering an index rebuild afterwards; a deployment adds updated rows back to its indexes in the background once enough have accumulated.
+33 -15
View File
@@ -4,6 +4,13 @@ sidebarTitle: "Table"
description: "The object most operations act on: creating, opening, renaming, cloning and dropping one."
icon: table
keywords: ["table", "create_table", "open_table", "clone", "rename", "drop"]
availability:
oss: available
enterprise: available
summary: >-
Tables are created, evolved, versioned and dropped in the embedded library
and on a deployment. Renaming, cloning, dropping every table and pruning old
versions differ between the two, as their sections say.
---
import {
@@ -121,7 +128,7 @@ This page covers the table as a catalog object. Getting data into one is
[Schema evolution](/management/table#schema).
<Note>
Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table).
On a deployment, tables are also created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table).
</Note>
## What a table carries {#carries}
@@ -156,10 +163,10 @@ frame, or Pydantic models. [Ingesting data](/ingestion) covers each, plus
`exist_ok` and `mode="overwrite"` for re-runnable scripts.
<Warning>
An explicit schema is where nullability is decided. Creation is also the only
point at which the SDK can give a table **stable row ids**, which a deployment
may require on a [materialized view](/management/materialized-view) source. See
[Limits](/basics/limits#schema).
An explicit schema is where nullability is decided. In the embedded library,
creation is also the only point at which a table can be given **stable row
ids**; on a deployment, whether a new table gets them is the deployment's
setting. See [Limits](/basics/limits#schema).
</Warning>
## Opening and listing {#open}
@@ -168,7 +175,6 @@ may require on a [materialized view](/management/materialized-view) source. See
```python Python icon="python"
tbl = db.open_table("documents")
db.table_names()
db.table_exists("documents")
```
```typescript TypeScript icon="square-js"
@@ -179,7 +185,10 @@ await db.tableNames();
</CodeGroup>
Each accepts `namespace_path` to act inside a [namespace](/management/namespace)
rather than at the database root. TypeScript has no `tableExists`.
rather than at the database root. Python's `table_exists` works only on a
namespace connection, where it takes the table's full path as a list; local
directory and deployment connections raise `NotImplementedError`. TypeScript
has no `tableExists`.
## Renaming and cloning {#rename-clone}
@@ -194,6 +203,10 @@ db.clone_table(
)
```
Renaming is not supported on a local directory connection. On a deployment,
whether a table can be renamed depends on how the deployment stores its
catalog.
A **shallow** clone -- the default -- references the source's data files rather
than copying them, so it is fast and cheap regardless of table size. It is the
practical way to branch an experiment off a known version, or to pin a training
@@ -204,13 +217,14 @@ A shallow clone depends on the source's files continuing to exist. If the source
is [compacted](/maintenance/compaction) past the retention window, the version
the clone points at can be pruned. Local OSS connections support shallow clones
only; `is_shallow=False` is not implemented. If the copy must outlive the
source's history, copy its data into an independent table instead. For remote
connections, confirm which clone modes your deployment supports.
source's history, copy its data into an independent table instead. A deployment
also makes shallow clones only.
</Warning>
<Note>
**Enterprise job type:** `drop_table`. Fences the table's directory with a
marker and deletes its content in the background -- see
**Enterprise job type:** `drop_table`. On a deployment that drops tables in the
background, fences the table's directory with a marker and deletes its content
as a job -- see
[Background jobs](/maintenance/background-jobs#teardown) for the namespace,
view, and function equivalent.
</Note>
@@ -238,11 +252,14 @@ DROP TABLE documents
<Warning>
Dropping removes the table and **every version of it**.
[Versioning](/management/table#versioning) does not survive a drop, and neither
`drop_all_tables` nor `drop_database` asks for confirmation.
`drop_all_tables` nor `drop_database` asks for confirmation. Both work only on
a local database; on a deployment connection the SDK raises instead.
</Warning>
On a large table the drop can take a while, so `drop_table_async` returns a
[job](/basics/jobs) rather than blocking.
[job](/basics/jobs) rather than blocking when the deployment drops tables in the
background. In the embedded library, and on a deployment that drops inline, the
returned job has already finished.
## The full surface {#surface}
@@ -1137,8 +1154,9 @@ mixing in lower-level Lance file operations.
**System Operations**
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
In LanceDB OSS and Enterprise, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
which is when old-version files are removed and disk space is reclaimed.
In LanceDB OSS, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
which is when old-version files are removed and disk space is reclaimed. A deployment prunes them in the background
when its configuration enables [cleanup](/maintenance/cleanup); `optimize()` is not available there.
**Tagged versions are exempt from cleanup.** A version with a tag pointing at it is
retained regardless of age, and its files are not removed by `optimize()`. To make
+19 -4
View File
@@ -3,6 +3,13 @@ title: "Branches"
sidebarTitle: "Branch"
description: "Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main."
icon: "code-branch"
availability:
oss: available
enterprise: available
summary: >-
Branches work in the embedded library and on a deployment. Comparing a
branch with `main` and promoting its new columns onto `main` is available
only on a deployment.
---
import {
PyConnect,
@@ -91,7 +98,8 @@ embedded library.
<Badge color="red">Enterprise</Badge>
Branching on LanceDB Enterprise works the same way as on OSS, but you connect to your
Branching on LanceDB Enterprise works the same way as on OSS, apart from the differences
noted below, but you connect to your
Enterprise deployment with a `db://` URI, an API key, and your
region. Once you have a connection, open a table and use the same branch calls in
every example that follows.
@@ -179,6 +187,8 @@ table. Both routes give you a writable handle tracking the branch's latest state
### Delete a branch {#delete-a-branch}
Deleting a branch removes it and its branch-local history; `main` is untouched.
A deployment refuses to delete a branch whose columns have been promoted onto
`main`, because `main` still depends on it.
Before deleting a branch, make sure you've retained any results you need — see
[Apply branch-tested changes to `main`](#apply-branch-tested-changes-to-main)
below.
@@ -244,6 +254,10 @@ your key column — `id` in this example. Here, `candidate` is the branch handle
</CodeBlock>
</CodeGroup>
Against a deployment, the synchronous Python table raises for `to_arrow()`; read
the branch's rows with a query instead -- see
[Avoid materializing the whole table](/search#avoid-materializing-the-whole-table).
This operation transfers only the inserted or updated rows in its input. It does
not transfer branch history, schema changes, indexes, or branch-local deletions,
and it does not determine which rows changed after the branch was created.
@@ -363,9 +377,10 @@ deployment workflow. Because the indexes live on the branch, queries against
`main` never see a half-built index and are never slowed down by the branch's
build.
This pattern is especially valuable on <Badge color="red">Enterprise</Badge>
deployments, where `main` is typically serving production traffic while you tune
an index configuration on the side.
This pattern is especially valuable on Enterprise deployments, where `main` is
typically serving production traffic while you tune an index configuration on
the side. A deployment builds the branch's indexes in the background, so wait
for them before measuring.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
+12 -4
View File
@@ -3,6 +3,13 @@ title: Consistency
sidebarTitle: "Consistency"
description: Learn about consistency settings and versioning in LanceDB tables.
icon: "water"
availability:
oss: available
enterprise: available
summary: >-
Read freshness can be set per connection in the embedded library and on a
deployment. On a deployment, the default freshness comes from its
configuration, and the connection setting can only make reads fresher.
---
import {
PyConsistencyStrong as ConsistencyStrong,
@@ -40,14 +47,14 @@ write operations are always consistent.
For remote tables (`db://` connections), `read_consistency_interval` is also
respected by the client. The interval is sent to the server as a freshness bound on each read:
- **Unset (default)**: no freshness header is sent; reads use the server's cached view of the table.
- **Unset (default)**: no freshness bound is sent; reads are as fresh as the deployment's own setting, described below.
- **Zero seconds**: every read asks the server for the latest committed version.
- **Non-zero interval**: reads accept data at least as fresh as `now - interval`.
In addition, after any write or after a `checkout_latest` / `restore` on a table handle, subsequent
reads on that same handle carry a freshness floor so you read your own writes without extra
configuration. The floor is the later of the configured interval and the moment of the last
write or refresh, and it is shared across handles to the same table on the same connection.
write or refresh. Each opened table handle keeps its own floor.
Each remote table handle also tracks the highest dataset version it has observed in a read
response and sends it back with every subsequent read, so successive reads on the same handle
@@ -59,7 +66,8 @@ of refreshing against storage, which raises per-read latency and cost.
In <Badge color="red">Enterprise</Badge> deployments, the server-side default freshness is still
controlled by the cluster-level `weak_read_consistency_interval_seconds` parameter; the client setting
tightens that bound on a per-connection basis.
tightens that bound on a per-connection basis. When the parameter is unset, every read is strongly
consistent; deployments installed from the Helm chart default it to 600 seconds.
<Tip>
30-60 seconds is a reasonable starting point for
@@ -112,7 +120,7 @@ For eventual consistency, use a non-zero interval:
</CodeGroup>
With the default unset interval, tables do not auto-refresh from other writers.
In the embedded library, with the default unset interval, tables do not auto-refresh from other writers.
To manually check for updates, use `checkout_latest` / `checkoutLatest`:
<CodeGroup>