mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
Jack YeandClaude Sonnet 5
de95ed3f6d
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
92 lines
3.9 KiB
Plaintext
92 lines
3.9 KiB
Plaintext
---
|
||
title: "Guarantees"
|
||
sidebarTitle: "Guarantees"
|
||
description: "Understand table commits, concurrent writes and the boundaries of recovery."
|
||
icon: shield-check
|
||
keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency"]
|
||
---
|
||
|
||
## Atomic table writes {#writes}
|
||
|
||
A table transaction commits its changes atomically: readers see the committed
|
||
table version, not a partially applied transaction. For example, a committed
|
||
append adds its rows together.
|
||
|
||
This boundary is one table transaction. A workflow that updates several tables
|
||
or runs a background job can contain multiple commits. Do not assume the whole
|
||
workflow is atomic.
|
||
|
||
<span id="schema" />
|
||
|
||
Schema changes are also part of table history. Changing a column's type can
|
||
rewrite data; adding a derived column can write the new column independently.
|
||
|
||
## Consistent reads {#reads}
|
||
|
||
A read of a table uses a consistent snapshot. A consistent snapshot can still
|
||
be older than another writer's latest commit. [Consistency](/tables/consistency)
|
||
explains how LanceDB Embedded and LanceDB Enterprise choose the version a reader sees.
|
||
|
||
For an experiment that must keep using the same data, select a specific retained
|
||
table version rather than relying on a connection's default freshness.
|
||
|
||
## Concurrent writes {#concurrency}
|
||
|
||
Lance uses optimistic concurrency control. Writers prepare changes, then check
|
||
compatibility with intervening commits. Compatible changes can be committed;
|
||
conflicts may be retried or returned as errors.
|
||
|
||
Applications must handle failed writes. Concurrent appends are compatible, but
|
||
that does not promise success through storage failures or exhausted retries.
|
||
Do not rely on a particular ordering between independent writers.
|
||
|
||
The [Lance transaction specification](https://lance.org/format/table/transaction/)
|
||
describes commit behavior and compatibility between operations.
|
||
|
||
## Storage durability {#durability}
|
||
|
||
A committed table version is stored in the table’s configured storage. LanceDB
|
||
Embedded uses your local filesystem or object store; LanceDB Enterprise uses the
|
||
cluster’s configured storage.
|
||
|
||
Table version history does not replace backups. Configure storage redundancy
|
||
and backups to meet your recovery requirements.
|
||
|
||
## Background job completion {#jobs}
|
||
|
||
In LanceDB Enterprise, operations such as index builds and column refreshes
|
||
can run as background jobs. Receiving a job handle confirms submission. Wait
|
||
for the job to succeed before relying on its result.
|
||
|
||
Some operations save results in batches. If a job fails or is cancelled,
|
||
batches already committed can remain in the table. Each operation’s guide
|
||
explains whether it can leave partial results.
|
||
|
||
## Writes after a client timeout {#timeouts}
|
||
|
||
If a client times out while waiting for a write, the write may already have
|
||
committed. Retrying an append can therefore insert the same records again.
|
||
|
||
Check whether the intended records were written before retrying. For an
|
||
operation submitted as a background job, check its status using the job handle.
|
||
|
||
## Version history and recovery {#history}
|
||
|
||
Retained table versions let you inspect earlier data or restore a table to a
|
||
previous version. Cleanup can remove those versions and their underlying files,
|
||
so configure retention to cover the history you need.
|
||
|
||
Dropping a table is different from deleting records. Table version history
|
||
cannot undo `drop_table`.
|
||
|
||
<span id="not-guaranteed" />
|
||
|
||
## Learn more {#next}
|
||
|
||
- [Consistency](/tables/consistency): control when reads observe updates.
|
||
- [Table versioning](/management/table#versioning): inspect and restore history.
|
||
- [Garbage Collection](/maintenance/cleanup): configure retention and understand protected versions.
|
||
- [Job Framework](/basics/jobs): distinguish submission, completion and cancellation.
|
||
- [Refresh a computed column](/feature-engineering/refresh-column): understand partial progress.
|
||
- [Architecture](/index#architecture): understand storage and background maintenance.
|