mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 14:52:26 +00:00
docs: declare where ingestion and table pages apply
Creating, writing, updating, merging and deleting rows, managing tables, branches and read consistency work in the embedded library and on a deployment, and those pages now say so. The SQL forms they mention need a deployment. Where the offerings differ, the section says how: a deployment reuses an existing table without comparing schemas, a merge insert can be refused past a configured number of unindexed rows, space is reclaimed by optimize() locally and by the deployment's compaction and configured cleanup otherwise, renaming depends on the catalog, a deployment clones shallowly only, drops all tables only locally, keeps freshness state per table handle and defaults freshness to its own setting, and refuses to delete a branch main depends on. table_exists works only on a namespace connection. Guarantees and limits span both offerings, so they carry section notes rather than a label: reads, retention, index freshness and compaction differ on a deployment, and the name and job limits are a deployment's.
This commit is contained in:
@@ -48,6 +48,10 @@ A long-lived connection **does not observe other processes' writes** until it
|
||||
reopens the table or its `read_consistency_interval` elapses. The default is
|
||||
right for a batch job reading a stable snapshot and wrong for a service
|
||||
expected to see fresh data.
|
||||
|
||||
This describes the embedded library. A table on a deployment is not pinned to
|
||||
the version it opened: other writers' commits become visible as the
|
||||
deployment's freshness setting allows.
|
||||
</Warning>
|
||||
|
||||
Set `read_consistency_interval` when several processes write the same table and
|
||||
@@ -68,8 +72,9 @@ file. The data is recoverable for as long as the version that contains it
|
||||
exists.
|
||||
|
||||
<Warning>
|
||||
Retention is finite. [Compaction](/maintenance/compaction) prunes versions
|
||||
older than the retention window — **7 days by default** — and a tag or branch
|
||||
Retention is finite. [Cleanup](/maintenance/cleanup) prunes versions older
|
||||
than the retention window — **7 days by default** for `optimize()` in the
|
||||
embedded library, while a deployment's configuration sets its own — and a tag or branch
|
||||
pointing at a pruned version no longer resolves. If you rely on reading old
|
||||
versions, widen the window before running it, not after.
|
||||
</Warning>
|
||||
@@ -124,11 +129,11 @@ Worth stating plainly, because each is occasionally assumed:
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Multi-table transactions | A commit covers one table. There is no cross-table atomic write. |
|
||||
| Read-your-writes across processes | Governed by `read_consistency_interval`, not automatic. |
|
||||
| Read-your-writes across processes | Governed by `read_consistency_interval` and, on a deployment, its freshness setting; not automatic. |
|
||||
| Unenforced primary keys | `SET UNENFORCED PRIMARY KEY` records intent; it does not enforce uniqueness. |
|
||||
| Ordering of concurrent appends | Two appends both succeed; their relative row order is not defined. |
|
||||
| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction). |
|
||||
| Automatic compaction | Nothing runs in the background. `optimize()` runs when you call it. |
|
||||
| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction) in the embedded library, or until a deployment's background maintenance adds them. |
|
||||
| Automatic compaction | In the embedded library nothing runs in the background; `optimize()` runs when you call it. A deployment compacts automatically. |
|
||||
|
||||
The last two are the most common source of "why is this query slow" — see
|
||||
[Limits](/basics/limits) for the bounds that pair with them.
|
||||
|
||||
@@ -10,8 +10,9 @@ Limits worth designing around, collected so you do not have to find them by
|
||||
hitting them. Each one is enforced -- where a value is refused rather than
|
||||
clamped, that is stated, because the two fail very differently.
|
||||
|
||||
This page covers bounds that are the same for everyone. Deployment-specific
|
||||
quotas -- storage, throughput, concurrency -- come from your deployment.
|
||||
Some of these bounds apply only on a deployment, as their sections say. Quotas
|
||||
that a deployment sets -- storage, throughput, concurrency -- come from your
|
||||
deployment.
|
||||
|
||||
## Names {#names}
|
||||
|
||||
@@ -24,10 +25,14 @@ quotas -- storage, throughput, concurrency -- come from your deployment.
|
||||
|
||||
A name must not be empty, and must not contain `$` or `:` -- both are reserved
|
||||
as path delimiters, so a name holding one could not be addressed unambiguously.
|
||||
Length is counted in characters, not bytes.
|
||||
Length is counted in characters, not bytes. These lengths are a deployment's
|
||||
limits; the embedded library does not limit name length.
|
||||
|
||||
## Jobs {#jobs}
|
||||
|
||||
<Badge color="red">Enterprise</Badge> These bounds belong to a deployment's job
|
||||
service. Jobs on a local database run in-process and keep no event history.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `job.events()` default | 1,000 rows |
|
||||
@@ -90,14 +95,15 @@ not another.
|
||||
| | |
|
||||
| --- | --- |
|
||||
| SQL on a local database | unavailable; `execute_query` raises |
|
||||
| SQL on a deployment | needs its Flight SQL endpoint enabled and reachable |
|
||||
| SQL methods in the TypeScript SDK | none; use a Flight SQL client |
|
||||
| Function methods in the TypeScript SDK | none; use Python or SQL |
|
||||
| Vector search in SQL | no SQL form; use a query builder |
|
||||
| Vector search in SQL | [depends on the deployment](/basics/sql#no-vector-sql); the query builder always has it |
|
||||
| Geospatial `st_*` functions | behind a build feature LanceDB does not ship |
|
||||
| `tableExists` / `namespaceExists` in TypeScript | not present |
|
||||
| Server-side jobs on a local database | run in-process; `events()` raises |
|
||||
|
||||
See [SQL](/basics/sql) for why the first two hold.
|
||||
See [SQL](/basics/sql) for why the first three hold.
|
||||
|
||||
## Nothing here is a quota {#not-quotas}
|
||||
|
||||
|
||||
@@ -4,6 +4,13 @@ sidebarTitle: "Create Table"
|
||||
description: "Start here: turn the data you already have into a LanceDB table."
|
||||
icon: table
|
||||
keywords: ["create table", "ingest", "arrow", "pandas", "polars", "pydantic", "schema"]
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Tables are created, opened and dropped in the embedded library and on a
|
||||
deployment. On a deployment, reusing an existing table does not check its
|
||||
schema against your data.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -94,8 +101,10 @@ You can change this behavior with two parameters that resolve the conflict in di
|
||||
|
||||
- **Idempotent open**: return the existing table without modifying it. Use when your
|
||||
code may run more than once (notebooks, retries, init scripts) and you want to reuse
|
||||
the table on subsequent runs. The provided data is ignored, but the schema is
|
||||
validated against the existing table and a mismatch raises an error.
|
||||
the table on subsequent runs. The provided data is ignored. In the embedded
|
||||
library the schema is validated against the existing table and a mismatch
|
||||
raises an error; a deployment returns the existing table without comparing
|
||||
schemas.
|
||||
- **Overwrite**: drop the existing table and create a new one with the provided data.
|
||||
Use this for test fixtures or when you intentionally want to replace prior contents.
|
||||
This permanently discards the old table's data.
|
||||
@@ -343,5 +352,5 @@ Use the `drop_table()` method on the database to remove a table.
|
||||
</CodeGroup>
|
||||
|
||||
This permanently removes the table and is not recoverable, unlike deleting rows.
|
||||
By default, if the table does not exist an exception is raised. To suppress this,
|
||||
you can pass in `ignore_missing=True`.
|
||||
By default, if the table does not exist an exception is raised. To suppress this
|
||||
in Python on a local database, you can pass in `ignore_missing=True`.
|
||||
|
||||
@@ -3,6 +3,13 @@ title: "Delete rows"
|
||||
sidebarTitle: "Delete"
|
||||
description: "Remove rows by predicate, and reclaim their space."
|
||||
icon: trash
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Rows are deleted in the embedded library and on a deployment. In the
|
||||
embedded library you reclaim their space with `optimize()`; a deployment
|
||||
compacts automatically and removes old versions as its configuration sets.
|
||||
---
|
||||
|
||||
These examples use the table set up in
|
||||
@@ -18,7 +25,7 @@ import {
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
<Note>
|
||||
In SQL this is [`DELETE`](/sql/dml/delete).
|
||||
In SQL on a deployment, this is [`DELETE`](/sql/dml/delete).
|
||||
</Note>
|
||||
|
||||
## Delete rows {#delete-rows}
|
||||
@@ -55,12 +62,12 @@ Expected table contents:
|
||||
When rows are deleted, those rows are also excluded from the index segments, so indexed queries will not return them either. If ALL the rows are deleted (i.e., the table is emptied), ensure that you recreate the index after ingesting new data.
|
||||
</Warning>
|
||||
|
||||
To permanently remove deleted rows, you can optimize the table, which will run compaction and cleans up the soft-deleted rows, which frees up storage space.
|
||||
To permanently remove deleted rows, the table has to be compacted and its old versions cleaned up, which frees up storage space.
|
||||
|
||||
- In LanceDB OSS, compaction and cleanup are manual. Run `table.optimize()` regularly to free up disk space.
|
||||
- In LanceDB Enterprise, files aren't cleaned up by default. You can configure automatic compaction and cleanup behavior at cluster setup time to suit your organization's retention policy.
|
||||
- In LanceDB Enterprise, compaction runs automatically, and old versions are removed only when the deployment's configuration enables cleanup, with the retention period it sets. `optimize()` is not available there; see [Cleanup](/maintenance/cleanup).
|
||||
|
||||
By default, table cleanup removes data up to 7 days ago. If you need to reclaim space from deleted rows more aggressively, manually call `table.optimize()` use a shorter retention window as follows:
|
||||
In OSS, `optimize()` removes versions older than 7 days by default. If you need to reclaim space from deleted rows more aggressively, call `table.optimize()` with a shorter retention window as follows:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
|
||||
@@ -4,6 +4,12 @@ sidebarTitle: "Overview"
|
||||
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
|
||||
icon: inbox
|
||||
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Tables are created and written to in the embedded library and on a
|
||||
deployment. The SQL forms of these writes need a deployment.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -30,7 +36,8 @@ writing to a table that exists. This page covers the first and introduces the
|
||||
second.
|
||||
|
||||
<Note>
|
||||
Every operation here has a SQL form -- see [DML](/sql#dml).
|
||||
On a deployment, every operation here except merge insert also has a SQL form --
|
||||
see [DML](/sql#dml).
|
||||
</Note>
|
||||
|
||||
Start by [creating a table](/ingestion/create-table); the rest of this
|
||||
@@ -89,6 +96,9 @@ Or, connect to LanceDB Enterprise:
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
For a deployment with its own endpoint, also pass that endpoint as
|
||||
`host_override` -- see [Connect to LanceDB](/basics/connect#enterprise).
|
||||
|
||||
In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.
|
||||
|
||||
<CodeGroup>
|
||||
|
||||
@@ -3,6 +3,13 @@ title: "Insert data"
|
||||
sidebarTitle: "Insert"
|
||||
description: "Append rows to a table, including datasets too large to hold in memory."
|
||||
icon: plus
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Rows are appended in the embedded library and on a deployment. Large writes
|
||||
to a deployment are uploaded in parts, which the section on bulk ingests
|
||||
explains how to tune.
|
||||
---
|
||||
|
||||
Rows are appended with `table.add()`. It accepts the same shapes
|
||||
@@ -20,7 +27,7 @@ import {
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
<Note>
|
||||
In SQL this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side.
|
||||
In SQL on a deployment, this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side.
|
||||
</Note>
|
||||
|
||||
## Ingesting large datasets {#large-datasets}
|
||||
|
||||
@@ -3,6 +3,13 @@ title: "Merge insert"
|
||||
sidebarTitle: "Merge Insert"
|
||||
description: "Reconcile incoming rows against a key: update what matches, insert what does not."
|
||||
icon: code-merge
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Merge insert runs in the embedded library and on a deployment. A deployment
|
||||
can be configured to refuse a merge that would scan too many unindexed
|
||||
rows.
|
||||
---
|
||||
|
||||
These examples use the table set up in
|
||||
@@ -54,7 +61,7 @@ Read more about scalar indices in the [Scalar Index](/indexing/scalar-index/) gu
|
||||
</Tip>
|
||||
|
||||
<Info>
|
||||
If you see this HTTP 400 error from `merge_insert`: `Bad request: Merge insert cannot be performed because the number of unindexed rows exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying.
|
||||
A deployment can be configured to refuse a merge insert that would scan too many unindexed rows, with an HTTP 400 error such as `Merge insert cannot be performed because the number of un-indexed rows (...) exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying.
|
||||
</Info>
|
||||
|
||||
<Note>
|
||||
|
||||
@@ -3,6 +3,11 @@ title: "Update rows"
|
||||
sidebarTitle: "Update"
|
||||
description: "Change values in rows that already exist, by predicate or SQL expression."
|
||||
icon: pen
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Rows are updated by predicate in the embedded library and on a deployment.
|
||||
---
|
||||
|
||||
These examples use the table set up in
|
||||
@@ -18,7 +23,7 @@ import {
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
<Note>
|
||||
In SQL this is [`UPDATE`](/sql/dml/update).
|
||||
In SQL on a deployment, this is [`UPDATE`](/sql/dml/update).
|
||||
</Note>
|
||||
|
||||
## Update rows {#update-rows}
|
||||
@@ -79,4 +84,4 @@ Expected table contents:
|
||||
See the [SQL queries](/sql#connecting) page for more information on the supported SQL syntax.
|
||||
</Note>
|
||||
|
||||
When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards.
|
||||
When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows in the embedded library, consider triggering an index rebuild afterwards; a deployment adds updated rows back to its indexes in the background once enough have accumulated.
|
||||
|
||||
@@ -4,6 +4,13 @@ sidebarTitle: "Table"
|
||||
description: "The object most operations act on: creating, opening, renaming, cloning and dropping one."
|
||||
icon: table
|
||||
keywords: ["table", "create_table", "open_table", "clone", "rename", "drop"]
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Tables are created, evolved, versioned and dropped in the embedded library
|
||||
and on a deployment. Renaming, cloning, dropping every table and pruning old
|
||||
versions differ between the two, as their sections say.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -121,7 +128,7 @@ This page covers the table as a catalog object. Getting data into one is
|
||||
[Schema evolution](/management/table#schema).
|
||||
|
||||
<Note>
|
||||
Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table).
|
||||
On a deployment, tables are also created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table).
|
||||
</Note>
|
||||
|
||||
## What a table carries {#carries}
|
||||
@@ -156,10 +163,10 @@ frame, or Pydantic models. [Ingesting data](/ingestion) covers each, plus
|
||||
`exist_ok` and `mode="overwrite"` for re-runnable scripts.
|
||||
|
||||
<Warning>
|
||||
An explicit schema is where nullability is decided. Creation is also the only
|
||||
point at which the SDK can give a table **stable row ids**, which a deployment
|
||||
may require on a [materialized view](/management/materialized-view) source. See
|
||||
[Limits](/basics/limits#schema).
|
||||
An explicit schema is where nullability is decided. In the embedded library,
|
||||
creation is also the only point at which a table can be given **stable row
|
||||
ids**; on a deployment, whether a new table gets them is the deployment's
|
||||
setting. See [Limits](/basics/limits#schema).
|
||||
</Warning>
|
||||
|
||||
## Opening and listing {#open}
|
||||
@@ -168,7 +175,6 @@ may require on a [materialized view](/management/materialized-view) source. See
|
||||
```python Python icon="python"
|
||||
tbl = db.open_table("documents")
|
||||
db.table_names()
|
||||
db.table_exists("documents")
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
@@ -179,7 +185,10 @@ await db.tableNames();
|
||||
</CodeGroup>
|
||||
|
||||
Each accepts `namespace_path` to act inside a [namespace](/management/namespace)
|
||||
rather than at the database root. TypeScript has no `tableExists`.
|
||||
rather than at the database root. Python's `table_exists` works only on a
|
||||
namespace connection, where it takes the table's full path as a list; local
|
||||
directory and deployment connections raise `NotImplementedError`. TypeScript
|
||||
has no `tableExists`.
|
||||
|
||||
## Renaming and cloning {#rename-clone}
|
||||
|
||||
@@ -194,6 +203,10 @@ db.clone_table(
|
||||
)
|
||||
```
|
||||
|
||||
Renaming is not supported on a local directory connection. On a deployment,
|
||||
whether a table can be renamed depends on how the deployment stores its
|
||||
catalog.
|
||||
|
||||
A **shallow** clone -- the default -- references the source's data files rather
|
||||
than copying them, so it is fast and cheap regardless of table size. It is the
|
||||
practical way to branch an experiment off a known version, or to pin a training
|
||||
@@ -204,13 +217,14 @@ A shallow clone depends on the source's files continuing to exist. If the source
|
||||
is [compacted](/maintenance/compaction) past the retention window, the version
|
||||
the clone points at can be pruned. Local OSS connections support shallow clones
|
||||
only; `is_shallow=False` is not implemented. If the copy must outlive the
|
||||
source's history, copy its data into an independent table instead. For remote
|
||||
connections, confirm which clone modes your deployment supports.
|
||||
source's history, copy its data into an independent table instead. A deployment
|
||||
also makes shallow clones only.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `drop_table`. Fences the table's directory with a
|
||||
marker and deletes its content in the background -- see
|
||||
**Enterprise job type:** `drop_table`. On a deployment that drops tables in the
|
||||
background, fences the table's directory with a marker and deletes its content
|
||||
as a job -- see
|
||||
[Background jobs](/maintenance/background-jobs#teardown) for the namespace,
|
||||
view, and function equivalent.
|
||||
</Note>
|
||||
@@ -238,11 +252,14 @@ DROP TABLE documents
|
||||
<Warning>
|
||||
Dropping removes the table and **every version of it**.
|
||||
[Versioning](/management/table#versioning) does not survive a drop, and neither
|
||||
`drop_all_tables` nor `drop_database` asks for confirmation.
|
||||
`drop_all_tables` nor `drop_database` asks for confirmation. Both work only on
|
||||
a local database; on a deployment connection the SDK raises instead.
|
||||
</Warning>
|
||||
|
||||
On a large table the drop can take a while, so `drop_table_async` returns a
|
||||
[job](/basics/jobs) rather than blocking.
|
||||
[job](/basics/jobs) rather than blocking when the deployment drops tables in the
|
||||
background. In the embedded library, and on a deployment that drops inline, the
|
||||
returned job has already finished.
|
||||
|
||||
## The full surface {#surface}
|
||||
|
||||
@@ -1137,8 +1154,9 @@ mixing in lower-level Lance file operations.
|
||||
**System Operations**
|
||||
|
||||
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
|
||||
In LanceDB OSS and Enterprise, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
|
||||
which is when old-version files are removed and disk space is reclaimed.
|
||||
In LanceDB OSS, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
|
||||
which is when old-version files are removed and disk space is reclaimed. A deployment prunes them in the background
|
||||
when its configuration enables [cleanup](/maintenance/cleanup); `optimize()` is not available there.
|
||||
|
||||
**Tagged versions are exempt from cleanup.** A version with a tag pointing at it is
|
||||
retained regardless of age, and its files are not removed by `optimize()`. To make
|
||||
|
||||
@@ -3,6 +3,13 @@ title: "Branches"
|
||||
sidebarTitle: "Branch"
|
||||
description: "Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main."
|
||||
icon: "code-branch"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Branches work in the embedded library and on a deployment. Comparing a
|
||||
branch with `main` and promoting its new columns onto `main` is available
|
||||
only on a deployment.
|
||||
---
|
||||
import {
|
||||
PyConnect,
|
||||
@@ -91,7 +98,8 @@ embedded library.
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
Branching on LanceDB Enterprise works the same way as on OSS, but you connect to your
|
||||
Branching on LanceDB Enterprise works the same way as on OSS, apart from the differences
|
||||
noted below, but you connect to your
|
||||
Enterprise deployment with a `db://` URI, an API key, and your
|
||||
region. Once you have a connection, open a table and use the same branch calls in
|
||||
every example that follows.
|
||||
@@ -179,6 +187,8 @@ table. Both routes give you a writable handle tracking the branch's latest state
|
||||
### Delete a branch {#delete-a-branch}
|
||||
|
||||
Deleting a branch removes it and its branch-local history; `main` is untouched.
|
||||
A deployment refuses to delete a branch whose columns have been promoted onto
|
||||
`main`, because `main` still depends on it.
|
||||
Before deleting a branch, make sure you've retained any results you need — see
|
||||
[Apply branch-tested changes to `main`](#apply-branch-tested-changes-to-main)
|
||||
below.
|
||||
@@ -244,6 +254,10 @@ your key column — `id` in this example. Here, `candidate` is the branch handle
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Against a deployment, the synchronous Python table raises for `to_arrow()`; read
|
||||
the branch's rows with a query instead -- see
|
||||
[Avoid materializing the whole table](/search#avoid-materializing-the-whole-table).
|
||||
|
||||
This operation transfers only the inserted or updated rows in its input. It does
|
||||
not transfer branch history, schema changes, indexes, or branch-local deletions,
|
||||
and it does not determine which rows changed after the branch was created.
|
||||
@@ -363,9 +377,10 @@ deployment workflow. Because the indexes live on the branch, queries against
|
||||
`main` never see a half-built index and are never slowed down by the branch's
|
||||
build.
|
||||
|
||||
This pattern is especially valuable on <Badge color="red">Enterprise</Badge>
|
||||
deployments, where `main` is typically serving production traffic while you tune
|
||||
an index configuration on the side.
|
||||
This pattern is especially valuable on Enterprise deployments, where `main` is
|
||||
typically serving production traffic while you tune an index configuration on
|
||||
the side. A deployment builds the branch's indexes in the background, so wait
|
||||
for them before measuring.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
|
||||
@@ -3,6 +3,13 @@ title: Consistency
|
||||
sidebarTitle: "Consistency"
|
||||
description: Learn about consistency settings and versioning in LanceDB tables.
|
||||
icon: "water"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Read freshness can be set per connection in the embedded library and on a
|
||||
deployment. On a deployment, the default freshness comes from its
|
||||
configuration, and the connection setting can only make reads fresher.
|
||||
---
|
||||
import {
|
||||
PyConsistencyStrong as ConsistencyStrong,
|
||||
@@ -40,14 +47,14 @@ write operations are always consistent.
|
||||
For remote tables (`db://` connections), `read_consistency_interval` is also
|
||||
respected by the client. The interval is sent to the server as a freshness bound on each read:
|
||||
|
||||
- **Unset (default)**: no freshness header is sent; reads use the server's cached view of the table.
|
||||
- **Unset (default)**: no freshness bound is sent; reads are as fresh as the deployment's own setting, described below.
|
||||
- **Zero seconds**: every read asks the server for the latest committed version.
|
||||
- **Non-zero interval**: reads accept data at least as fresh as `now - interval`.
|
||||
|
||||
In addition, after any write or after a `checkout_latest` / `restore` on a table handle, subsequent
|
||||
reads on that same handle carry a freshness floor so you read your own writes without extra
|
||||
configuration. The floor is the later of the configured interval and the moment of the last
|
||||
write or refresh, and it is shared across handles to the same table on the same connection.
|
||||
write or refresh. Each opened table handle keeps its own floor.
|
||||
|
||||
Each remote table handle also tracks the highest dataset version it has observed in a read
|
||||
response and sends it back with every subsequent read, so successive reads on the same handle
|
||||
@@ -59,7 +66,8 @@ of refreshing against storage, which raises per-read latency and cost.
|
||||
|
||||
In <Badge color="red">Enterprise</Badge> deployments, the server-side default freshness is still
|
||||
controlled by the cluster-level `weak_read_consistency_interval_seconds` parameter; the client setting
|
||||
tightens that bound on a per-connection basis.
|
||||
tightens that bound on a per-connection basis. When the parameter is unset, every read is strongly
|
||||
consistent; deployments installed from the Helm chart default it to 600 seconds.
|
||||
|
||||
<Tip>
|
||||
30-60 seconds is a reasonable starting point for
|
||||
@@ -112,7 +120,7 @@ For eventual consistency, use a non-zero interval:
|
||||
</CodeGroup>
|
||||
|
||||
|
||||
With the default unset interval, tables do not auto-refresh from other writers.
|
||||
In the embedded library, with the default unset interval, tables do not auto-refresh from other writers.
|
||||
To manually check for updates, use `checkout_latest` / `checkoutLatest`:
|
||||
|
||||
<CodeGroup>
|
||||
|
||||
Reference in New Issue
Block a user