docs: clarify concepts and guarantees

2026-10-15 08:12:26 +00:00 · 2026-10-14 22:07:45 -07:00
parent cf355f14dd
commit eea28c8d35
9 changed files with 96 additions and 82 deletions
@@ -78,18 +78,18 @@ Size and monitor each tier for your workload. Read the
 [Enterprise architecture](/enterprise/architecture) for the component details
 and request flows.

-## Familiar APIs, a different operating model {#differences}
+## Application changes for Enterprise {#differences}

 Core operations remain familiar: create a table, add data, and run a search.
 With OSS, the application executes those operations. With Enterprise, the SDK
 sends requests to the service, and the service manages execution and configured
 background maintenance.

-Moving an application involves more than changing a connection string. Plan
-for data migration, connection settings, credentials, and SDK methods that
-behave differently on remote tables. See
-[Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise) before adapting
-an existing application. If you plan to use materialized views, check the
+To use Enterprise, configure the cluster endpoint, database URI and credentials,
+and make your data available in the Enterprise database. Review SDK methods
+that behave differently on remote tables, including result retrieval and
+background jobs. [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise)
+describes these application changes. If you plan to use materialized views, check the
 [stable row ID requirements](/basics/limits#schema) before creating source tables.

 ## Choose for the workload you need to run {#choosing}
@@ -6,13 +6,13 @@ icon: sitemap
 keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view"]
 ---

-## What is a catalog? {#catalog}
+## Catalog overview {#catalog}

 A catalog keeps track of named data objects, such as tables and views, so
 applications can discover and access them. In LanceDB, it organizes those
 objects into databases and namespaces.

-## How is data organized? {#objects}
+## Databases, namespaces and tables {#objects}

 A database contains namespaces that group related tables and views. Namespaces
 can also contain other namespaces, allowing teams to organize data by project,
@@ -42,7 +42,7 @@ flowchart TD

 <span id="resolution" />

-## Why does the organization matter? {#governance}
+## Organization and access control {#governance}

 - **Separate projects and environments.** Keep production and staging tables
  in distinct namespaces, even when they use the same table names.
@@ -6,7 +6,7 @@ icon: shield-check
 keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency"]
 ---

-## What commits together? {#writes}
+## Atomic table writes {#writes}

 A table transaction commits its changes atomically: readers see the committed
 table version, not a partially applied transaction. For example, a committed
@@ -21,7 +21,7 @@ workflow is atomic.
 Schema changes are also part of table history. Changing a column's type can
 rewrite data; adding a derived column can write the new column independently.

-## What does a read observe? {#reads}
+## Consistent reads {#reads}

 A read of a table uses a consistent snapshot. A consistent snapshot can still
 be older than another writer's latest commit. [Consistency](/tables/consistency)
@@ -30,7 +30,7 @@ explains how OSS and Enterprise choose the version a reader sees.
 For an experiment that must keep using the same data, select a specific retained
 table version rather than relying on a connection's default freshness.

-## What happens with concurrent writes? {#concurrency}
+## Concurrent writes {#concurrency}

 Lance uses optimistic concurrency control. Writers prepare changes, then check
 compatibility with intervening commits. Compatible changes can be committed;
@@ -43,31 +43,41 @@ Do not rely on a particular ordering between independent writers.
 The [Lance transaction specification](https://lance.org/format/table/transaction/)
 describes commit behavior and compatibility between operations.

-<span id="jobs" />
+## Storage durability {#durability}

-## What survives completion or failure? {#durability}
+A committed table version is stored in the table’s configured storage. LanceDB
+OSS uses your local filesystem or object store; LanceDB Enterprise uses the
+cluster’s configured storage.

-A successful commit records the change in the table's storage. With OSS, that
-storage can be local disk or an object store; Enterprise uses its configured
-storage infrastructure. Durability depends on that backend and its configuration.
+Table version history does not replace backups. Configure storage redundancy
+and backups to meet your recovery requirements.

-A job being accepted is not the same as its work completing. Wait for its
-successful outcome before depending on the result. Some background operations
-commit batches as they progress, so failure or cancellation can leave completed
-work in place. See the operation's guide for its commit behavior.
+## Background job completion {#jobs}

-A client timeout can leave the outcome unknown. Check the job or resulting data
-before repeating work that could create duplicates.
+In LanceDB Enterprise, operations such as index builds and column refreshes
+can run as background jobs. Receiving a job handle confirms submission. Wait
+for the job to succeed before relying on its result.

-## What can history recover? {#history}
+Some operations save results in batches. If a job fails or is cancelled,
+batches already committed can remain in the table. Each operation’s guide
+explains whether it can leave partial results.

-Retained table versions let you inspect earlier data and restore a prior state.
-Cleanup can remove old versions and the files they need. Set retention for your
-recovery and reproducibility requirements before cleanup runs.
+## Writes after a client timeout {#timeouts}

-Dropping a table is different from deleting rows: table versioning does not
-provide an undo for `drop_table`. Keep backups appropriate to your storage and
-recovery requirements.
+If a client times out while waiting for a write, the write may already have
+committed. Retrying an append can therefore insert the same records again.
+
+Check whether the intended records were written before retrying. For an
+operation submitted as a background job, check its status using the job handle.
+
+## Version history and recovery {#history}
+
+Retained table versions let you inspect earlier data or restore a table to a
+previous version. Cleanup can remove those versions and their underlying files,
+so configure retention to cover the history you need.
+
+Dropping a table is different from deleting records. Table version history
+cannot undo `drop_table`.

 <span id="not-guaranteed" />

@@ -12,32 +12,36 @@ availability:
    supports listing server jobs, reopening them by ID and reading their events.
 ---

-## What is a job? {#convention}
+## Job handles {#convention}

-A job is a handle for tracking work, such as building an index or refreshing a
-computed column. It lets you inspect progress and wait for the operation's
-outcome after submission.
+A job handle lets you track an operation, such as an index build or a computed
+column refresh. Use it to check progress, wait for completion and inspect
+failures.

-Job-returning methods include Python's `create_index_async` and TypeScript's
-`createIndexAsync`. Check the operation's reference for submission and waiting
-behavior: a method's name alone does not tell you when the work is complete.
+Methods such as Python's `create_index_async` and TypeScript's
+`createIndexAsync` return job handles. In OSS, the work runs in your application
+process; in Enterprise, it runs on server workers. A returned handle can
+represent an operation that has already finished, so check its status before
+waiting or requesting cancellation. See the operation's API reference for when
+the method returns.

-## What is its lifecycle? {#waiting}
+## Submit and wait for a job {#waiting}

 1. **Submit the operation.** Keep the returned job handle. For an Enterprise
   job, also keep its ID if another process needs to look it up later.
 2. **Check progress.** Read the current status while the operation runs.
-3. **Wait for an outcome.** A job can finish, fail or be cancelled. Inspect its
-   result or failure information before continuing dependent work.
+3. **Wait for completion.** Check that the job succeeded before using its
+   result. If it failed or was cancelled, inspect the failure details and any
+   results it committed before stopping.

 In Python, `job.status()` reports the current state and `job.wait()` waits for
-completion, raising if the job fails or is cancelled. A wait timeout ends the
-wait; it does not request cancellation of the server job.
+completion, raising if the job fails or is cancelled. If `job.wait()` times out, the client stops waiting.
+The timeout does not cancel the server job.

 <span id="inspecting" />
 <span id="events" />

-## How do you manage a job? {#managing}
+## Inspect and cancel jobs {#managing}

 | Task | What to do |
 | --- | --- |
@@ -53,7 +57,7 @@ Event responses have [row limits](/basics/limits#jobs). An explicit limit can
 request more than the default, but it does not guarantee a complete event
 history.

-## Where does the work run? {#in-process}
+## OSS and Enterprise execution {#in-process}

 | | OSS | Enterprise |
 | --- | --- | --- |
@@ -62,9 +66,6 @@ history.
 | Event history | No server event history | Available through the jobs API |
 | Reattaching after a client restart | No server job to reopen | Reopen by ID while the record is retained |

-A handle can represent work that has already completed. Check its state rather
-than assuming every returned job is still running.
-
 ## Learn more {#next}

 - [Build and repair indexes](/indexing/reindexing): submit index work.
@@ -6,7 +6,7 @@ icon: layer-group
 keywords: ["lakehouse", "multimodal", "search", "compute", "training", "feature engineering"]
 ---

-## What is the LanceDB lakehouse? {#multimodal}
+## Multimodal storage and processing {#multimodal}

 LanceDB brings storage, search and data processing together for AI workloads.
 You can store multimodal data, derive features, search your records and prepare
@@ -15,7 +15,7 @@ training datasets using tables built on the open [Lance format](/basics/lance).
 The format provides the storage foundation. The [catalog](/basics/catalog)
 organizes your tables. LanceDB provides the tools to work with that data.

-## How does the data flow? {#capabilities}
+## Example: a video search workflow {#capabilities}

 Consider a team building a video search application:

@@ -42,7 +42,7 @@ The clips, metadata and derived features stay associated in the same table,
 so retrieval and training workflows can use the same dataset. Your training
 framework runs the model; LanceDB provides access to the training data.

-## What does Enterprise add? {#where}
+## Shared compute and maintenance in Enterprise {#where}

 LanceDB OSS runs as an embedded library in your application. LanceDB Enterprise
 provides shared infrastructure for applications and teams working with larger
@@ -6,7 +6,7 @@ icon: "/static/assets/logo/lance-logo-gray.svg"
 keywords: ["lance", "format", "multimodal", "versioning", "column evolution"]
 ---

-## What is Lance? {#standard}
+## Lance format overview {#standard}

 [Lance](https://lance.org/) is an open-source, columnar format designed for
 multimodal AI. It defines how data is stored and organized into versioned tables.
@@ -18,14 +18,14 @@ LanceDB OSS and LanceDB Enterprise both use Lance as their storage foundation.
 The format is developed as an independent open-source project, with its own
 specification and tools.

-## What can it store? {#storage}
+## Multimodal data {#storage}

 A Lance table can hold images, video, audio, text, embeddings and structured
 metadata together. For example, a video dataset can keep each clip alongside
 its caption, labels and embeddings, so applications can search and work with
 those fields together.

-## What does the format enable? {#capabilities}
+## Data access, versioning and column updates {#capabilities}

 - **Efficient data access.** Read individual rows for retrieval and training,
  or scan columns for analytics.
@@ -6,11 +6,11 @@ icon: ruler
 keywords: ["limits", "maximum", "quota", "constraints", "bounds"]
 ---

-## Which limits apply? {#scope}
+<span id="scope" />

-Some limits come from an API or data type; others come from Enterprise
-configuration. Use the scope and linked reference below to identify which
-applies to your operation. SQL and server job limits apply to Enterprise.
+This page lists limits for job events, SQL error reports, sampling and
+full-text indexing. Each table identifies the affected API and what happens
+when a request reaches its limit. SQL and server job limits apply to Enterprise.

 <span id="names" />

@@ -21,7 +21,7 @@ enforce.
 <span id="jobs" />
 <span id="sql" />

-## What are the fixed limits? {#fixed-limits}
+## Fixed API limits {#fixed-limits}

 | Operation | Scope | Limit and behavior |
 | --- | --- | --- |
@@ -42,7 +42,7 @@ before choosing a schema or index. For materialized views, check
 [stable row ID requirements](/feature-engineering/materialized-views#stable-row-ids)
 before creating source tables.

-## What can you configure? {#not-quotas}
+## Configurable limits {#not-quotas}

 | Setting | What it controls |
 | --- | --- |
@@ -50,7 +50,7 @@ before creating source tables.
 | FTS `max_token_length` | The maximum indexed token length. The default is 40; longer tokens are excluded from the index. |
 | Enterprise capacity and quotas | Storage, request rate and concurrency are governed by your deployment's resources and configuration. Confirm these with your administrator or LanceDB support. |

-## What happens at a limit? {#behavior}
+## Handling rejected or incomplete results {#behavior}

 - **Rejected requests:** correct the input before retrying, such as a sampling
  percentage outside the accepted range.
@@ -6,14 +6,14 @@ icon: magnifying-glass
 keywords: ["query", "search", "SQL", "async", "streaming", "results"]
 ---

-## What is a query? {#queries}
+## Querying tables {#queries}

 A query selects data from your tables. You can filter rows, choose columns,
 search for similar content or combine those operations. For example, a video
 application might find clips similar to a sample and return only those recorded
 in the last month.

-## How can you query? {#searches}
+## SDK searches and SQL queries {#searches}

 | Interface | When to use it |
 | --- | --- |
@@ -24,26 +24,27 @@ Both interfaces work with LanceDB tables. Choose the one that supports your
 operation and fits your application. See [SQL](/basics/sql) for connection
 instructions and SQL search support, and [Search](/search) for the query builders.

-## How do results arrive? {#results}
+## Collect or stream results {#results}

 - **Collect the results** when you want a list, Arrow table or DataFrame that
  fits in memory. Use a result limit and select only the columns you need.
 - **Read batches** when you want to process a large result incrementally.
-  Streaming avoids collecting the entire result in application memory, although
-  some query operations still need to do substantial work before producing rows.
+  Reading batches avoids loading the entire result into application memory.
+  It does not guarantee that the query can return its first batch immediately.

-<span id="async-client" />
+### Asynchronous clients {#async-client}

 An async client lets your application do other work while a request is waiting.
 That is separate from whether you collect results or read them in batches.
 Python offers synchronous and asynchronous clients; TypeScript uses async APIs.
 Check the SDK reference for the methods available in your client.

-<span id="jobs" />
+### Query handles and background jobs {#jobs}

-Enterprise SQL can also return a query handle for inspecting execution and
-requesting cancellation. A [job](/basics/jobs) tracks an operation such as an
-index build. Use the result API for the operation you submitted.
+An Enterprise SQL query handle lets you inspect query execution and request
+cancellation. Background operations such as index builds return a
+[job](/basics/jobs) handle instead. Use the API for the returned handle to
+check status and retrieve results.

 ## Learn more {#next}

@@ -13,16 +13,18 @@ availability:

 import { PyReadConsistency } from '/snippets/read_consistency.mdx';

-## What does consistency mean here? {#consistency}
+## Table snapshots and freshness {#consistency}

-A table read uses a consistent snapshot of the data. **Freshness** describes how
-recent that snapshot is. A reader can see a coherent earlier version while
-another process has already committed newer data.
+Each table read uses one committed version of the data. **Freshness** describes
+whether that version includes recent writes. For example, after another
+application appends records, an already-open table handle can still read the
+version from before the append.

 For example, a dashboard may need the latest updates, while an experiment may
-need to keep reading a specific table version. These are different requirements.
+need to keep reading a specific table version. Use read freshness settings for the dashboard and an explicit table version
+for the experiment.

-## When do changes become visible? {#visibility}
+## Visibility of writes {#visibility}

 In **OSS**, a table handle sees its own committed writes. By default, it does
 not automatically check for changes made through other handles or processes.
@@ -33,10 +35,10 @@ A connection can request fresher data. When a read must include a completed
 write, use a zero interval or call `checkout_latest()` after the write, with a
 client and server that support these freshness controls.

-## How do you control freshness? {#configure-consistency-parameters}
+## Configure read freshness {#configure-consistency-parameters}

-Set `read_consistency_interval` when connecting. The setting applies to reads;
-it does not change the transaction boundary of a write.
+Set `read_consistency_interval` when connecting. This controls when reads
+pick up newer table versions; it does not change how writes commit.

 | Setting | OSS | Enterprise |
 | --- | --- | --- |
@@ -64,7 +66,7 @@ interval sees the new row on its next read.

 </Accordion>

-## How should you choose? {#choosing}
+## Choose a freshness interval {#choosing}

 - **Reads that must include recent writes:** use a zero interval. Checking more
  frequently can increase read latency and storage requests.
@@ -73,8 +75,8 @@ interval sees the new row on its next read.
 - **Reproducible experiments:** read an explicit table version and retain it for
  as long as the experiment needs it.

-There is no single interval that suits every application. Choose based on how
-stale a result your users can accept.
+Choose the interval based on the delay your application can tolerate between
+a write completing and a reader seeing it.

 ## Learn more {#next}