mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Brendan Clement
6a1645116e
Creating, writing, updating, merging and deleting rows, managing tables, branches and read consistency work in the embedded library and on a deployment, and those pages now say so. The SQL forms they mention need a deployment. Where the offerings differ, the section says how: a deployment reuses an existing table without comparing schemas, a merge insert can be refused past a configured number of unindexed rows, space is reclaimed by optimize() locally and by the deployment's compaction and configured cleanup otherwise, renaming depends on the catalog, a deployment clones shallowly only, drops all tables only locally, keeps freshness state per table handle and defaults freshness to its own setting, and refuses to delete a branch main depends on. table_exists works only on a namespace connection. Guarantees and limits span both offerings, so they carry section notes rather than a label: reads, retention, index freshness and compaction differ on a deployment, and the name and job limits are a deployment's.
140 lines
6.2 KiB
Plaintext
140 lines
6.2 KiB
Plaintext
---
|
|
title: "Guarantees"
|
|
sidebarTitle: "Guarantees"
|
|
description: "What a write promises when it returns, what a reader sees, and what survives a failure."
|
|
icon: shield-check
|
|
keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency", "acid"]
|
|
---
|
|
|
|
What the system promises, and — just as usefully — what it does not. Most
|
|
surprises in practice come from assuming one of the guarantees below is
|
|
stronger or weaker than it is.
|
|
|
|
## Writes {#writes}
|
|
|
|
**A write is atomic.** Every write commits a new table version. Either the
|
|
version exists and holds all of the change, or it does not exist and holds none
|
|
of it. There is no state in which a reader sees half an `add()`.
|
|
|
|
**A write returns only after the commit.** When the call returns, the version
|
|
is durable in object storage. Nothing is buffered in your process waiting to be
|
|
flushed.
|
|
|
|
**Every write advances the version**, including one that affects zero rows. A
|
|
delete whose predicate matches nothing still commits. Version numbers count
|
|
commits, not changes.
|
|
|
|
**Concurrent writes are resolved optimistically.** A transaction records the
|
|
change and the version it was based on. At commit time, a conflicting
|
|
concurrent operation is detected rather than silently overwritten, and the
|
|
commit is retried against the newer version. Operations that do not collide —
|
|
appending while another process appends — both succeed.
|
|
|
|
<Note>
|
|
`merge_insert` reports its retry attempts in the result. A high count means
|
|
contention on the same keys, not an error.
|
|
</Note>
|
|
|
|
## Reads {#reads}
|
|
|
|
**A read sees a consistent snapshot.** A query reads one table version
|
|
throughout; it never sees a mixture of two.
|
|
|
|
**Which version, by default, is the one open at the time the table was opened.**
|
|
This is the guarantee most often assumed to be stronger than it is:
|
|
|
|
<Warning>
|
|
A long-lived connection **does not observe other processes' writes** until it
|
|
reopens the table or its `read_consistency_interval` elapses. The default is
|
|
right for a batch job reading a stable snapshot and wrong for a service
|
|
expected to see fresh data.
|
|
|
|
This describes the embedded library. A table on a deployment is not pinned to
|
|
the version it opened: other writers' commits become visible as the
|
|
deployment's freshness setting allows.
|
|
</Warning>
|
|
|
|
Set `read_consistency_interval` when several processes write the same table and
|
|
readers must observe those writes. The cost is a metadata check on that
|
|
interval. See [Consistency](/tables/consistency).
|
|
|
|
**Your own writes are always visible to you.** The interval governs other
|
|
writers, not yourself.
|
|
|
|
## Durability and history {#durability}
|
|
|
|
**Old versions are retained, not overwritten.** A new version does not destroy
|
|
the one before it, which is what makes [time travel](/management/table#versioning) work
|
|
and what lets a reader finish a scan that started on an older version.
|
|
|
|
**Deletion is a mark, not an erase.** A deleted row is recorded in a deletion
|
|
file. The data is recoverable for as long as the version that contains it
|
|
exists.
|
|
|
|
<Warning>
|
|
Retention is finite. [Cleanup](/maintenance/cleanup) prunes versions older
|
|
than the retention window — **7 days by default** for `optimize()` in the
|
|
embedded library, while a deployment's configuration sets its own — and a tag or branch
|
|
pointing at a pruned version no longer resolves. If you rely on reading old
|
|
versions, widen the window before running it, not after.
|
|
</Warning>
|
|
|
|
**Dropping a table ends its history.** Versioning does not survive
|
|
`drop_table`; there is no undrop.
|
|
|
|
## Jobs {#jobs}
|
|
|
|
**A refresh recomputes only what changed.** It gives values to rows that lack
|
|
them and recomputes rows whose inputs changed since they were computed; it
|
|
leaves the rest alone. Two consequences, both relied on elsewhere:
|
|
|
|
- **Retrying is safe.** Re-running a refresh fills what is still empty or stale.
|
|
It is not guaranteed to cost only the rows that did not complete: a retry can
|
|
recompute values that were already written.
|
|
- **A filled value catches up at the next refresh.** If a column derives from
|
|
`price` and a row's `price` later changes, the derived value keeps its old
|
|
value until the next refresh recomputes it.
|
|
|
|
**What a failure keeps depends on where the refresh ran.** `refresh_column` in
|
|
your process commits the column once, so a failure keeps none of its values. A
|
|
refresh job on a deployment commits finished batches as it goes, so some values
|
|
can remain.
|
|
|
|
**Cancellation is a request.** `cancel()` asks a job to stop; it does not
|
|
guarantee it stops before doing more work, and cancelling a finished job does
|
|
nothing.
|
|
|
|
**A job id is opaque.** It correlates with server logs and the jobs API.
|
|
Parsing it, or storing it to resume the job later, is not supported.
|
|
|
|
See [Jobs](/basics/jobs).
|
|
|
|
## Schema {#schema}
|
|
|
|
**Schema changes are versioned like data changes.** Adding, renaming, retyping
|
|
or dropping a column commits a version, so a schema change can be travelled back
|
|
through.
|
|
|
|
**A computed column's inputs are protected.** While a declaration references a
|
|
column, that column cannot be renamed, retyped, or dropped — the declaration is
|
|
part of the schema and will not be invalidated out from under itself.
|
|
|
|
**Most schema changes do not rewrite data.** Adding a derived column writes only
|
|
the new column. Type changes are the exception and can be substantial.
|
|
|
|
## What is not guaranteed {#not-guaranteed}
|
|
|
|
Worth stating plainly, because each is occasionally assumed:
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| Multi-table transactions | A commit covers one table. There is no cross-table atomic write. |
|
|
| Read-your-writes across processes | Governed by `read_consistency_interval` and, on a deployment, its freshness setting; not automatic. |
|
|
| Unenforced primary keys | `SET UNENFORCED PRIMARY KEY` records intent; it does not enforce uniqueness. |
|
|
| Ordering of concurrent appends | Two appends both succeed; their relative row order is not defined. |
|
|
| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction) in the embedded library, or until a deployment's background maintenance adds them. |
|
|
| Automatic compaction | In the embedded library nothing runs in the background; `optimize()` runs when you call it. A deployment compacts automatically. |
|
|
|
|
The last two are the most common source of "why is this query slow" — see
|
|
[Limits](/basics/limits) for the bounds that pair with them.
|