diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx index a1d0e46e5..997bb95b0 100644 --- a/docs/web/architecture.mdx +++ b/docs/web/architecture.mdx @@ -78,18 +78,18 @@ Size and monitor each tier for your workload. Read the [Enterprise architecture](/enterprise/architecture) for the component details and request flows. -## Familiar APIs, a different operating model {#differences} +## Application changes for Enterprise {#differences} Core operations remain familiar: create a table, add data, and run a search. With OSS, the application executes those operations. With Enterprise, the SDK sends requests to the service, and the service manages execution and configured background maintenance. -Moving an application involves more than changing a connection string. Plan -for data migration, connection settings, credentials, and SDK methods that -behave differently on remote tables. See -[Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise) before adapting -an existing application. If you plan to use materialized views, check the +To use Enterprise, configure the cluster endpoint, database URI and credentials, +and make your data available in the Enterprise database. Review SDK methods +that behave differently on remote tables, including result retrieval and +background jobs. [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise) +describes these application changes. If you plan to use materialized views, check the [stable row ID requirements](/basics/limits#schema) before creating source tables. ## Choose for the workload you need to run {#choosing} diff --git a/docs/web/basics/catalog.mdx b/docs/web/basics/catalog.mdx index 21a965999..88d373061 100644 --- a/docs/web/basics/catalog.mdx +++ b/docs/web/basics/catalog.mdx @@ -6,13 +6,13 @@ icon: sitemap keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view"] --- -## What is a catalog? {#catalog} +## Catalog overview {#catalog} A catalog keeps track of named data objects, such as tables and views, so applications can discover and access them. In LanceDB, it organizes those objects into databases and namespaces. -## How is data organized? {#objects} +## Databases, namespaces and tables {#objects} A database contains namespaces that group related tables and views. Namespaces can also contain other namespaces, allowing teams to organize data by project, @@ -42,7 +42,7 @@ flowchart TD -## Why does the organization matter? {#governance} +## Organization and access control {#governance} - **Separate projects and environments.** Keep production and staging tables in distinct namespaces, even when they use the same table names. diff --git a/docs/web/basics/guarantees.mdx b/docs/web/basics/guarantees.mdx index 1afd374f9..840ea2d50 100644 --- a/docs/web/basics/guarantees.mdx +++ b/docs/web/basics/guarantees.mdx @@ -6,7 +6,7 @@ icon: shield-check keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency"] --- -## What commits together? {#writes} +## Atomic table writes {#writes} A table transaction commits its changes atomically: readers see the committed table version, not a partially applied transaction. For example, a committed @@ -21,7 +21,7 @@ workflow is atomic. Schema changes are also part of table history. Changing a column's type can rewrite data; adding a derived column can write the new column independently. -## What does a read observe? {#reads} +## Consistent reads {#reads} A read of a table uses a consistent snapshot. A consistent snapshot can still be older than another writer's latest commit. [Consistency](/tables/consistency) @@ -30,7 +30,7 @@ explains how OSS and Enterprise choose the version a reader sees. For an experiment that must keep using the same data, select a specific retained table version rather than relying on a connection's default freshness. -## What happens with concurrent writes? {#concurrency} +## Concurrent writes {#concurrency} Lance uses optimistic concurrency control. Writers prepare changes, then check compatibility with intervening commits. Compatible changes can be committed; @@ -43,31 +43,41 @@ Do not rely on a particular ordering between independent writers. The [Lance transaction specification](https://lance.org/format/table/transaction/) describes commit behavior and compatibility between operations. - +## Storage durability {#durability} -## What survives completion or failure? {#durability} +A committed table version is stored in the table’s configured storage. LanceDB +OSS uses your local filesystem or object store; LanceDB Enterprise uses the +cluster’s configured storage. -A successful commit records the change in the table's storage. With OSS, that -storage can be local disk or an object store; Enterprise uses its configured -storage infrastructure. Durability depends on that backend and its configuration. +Table version history does not replace backups. Configure storage redundancy +and backups to meet your recovery requirements. -A job being accepted is not the same as its work completing. Wait for its -successful outcome before depending on the result. Some background operations -commit batches as they progress, so failure or cancellation can leave completed -work in place. See the operation's guide for its commit behavior. +## Background job completion {#jobs} -A client timeout can leave the outcome unknown. Check the job or resulting data -before repeating work that could create duplicates. +In LanceDB Enterprise, operations such as index builds and column refreshes +can run as background jobs. Receiving a job handle confirms submission. Wait +for the job to succeed before relying on its result. -## What can history recover? {#history} +Some operations save results in batches. If a job fails or is cancelled, +batches already committed can remain in the table. Each operation’s guide +explains whether it can leave partial results. -Retained table versions let you inspect earlier data and restore a prior state. -Cleanup can remove old versions and the files they need. Set retention for your -recovery and reproducibility requirements before cleanup runs. +## Writes after a client timeout {#timeouts} -Dropping a table is different from deleting rows: table versioning does not -provide an undo for `drop_table`. Keep backups appropriate to your storage and -recovery requirements. +If a client times out while waiting for a write, the write may already have +committed. Retrying an append can therefore insert the same records again. + +Check whether the intended records were written before retrying. For an +operation submitted as a background job, check its status using the job handle. + +## Version history and recovery {#history} + +Retained table versions let you inspect earlier data or restore a table to a +previous version. Cleanup can remove those versions and their underlying files, +so configure retention to cover the history you need. + +Dropping a table is different from deleting records. Table version history +cannot undo `drop_table`. diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index 41e7a48ab..0577cd120 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -12,32 +12,36 @@ availability: supports listing server jobs, reopening them by ID and reading their events. --- -## What is a job? {#convention} +## Job handles {#convention} -A job is a handle for tracking work, such as building an index or refreshing a -computed column. It lets you inspect progress and wait for the operation's -outcome after submission. +A job handle lets you track an operation, such as an index build or a computed +column refresh. Use it to check progress, wait for completion and inspect +failures. -Job-returning methods include Python's `create_index_async` and TypeScript's -`createIndexAsync`. Check the operation's reference for submission and waiting -behavior: a method's name alone does not tell you when the work is complete. +Methods such as Python's `create_index_async` and TypeScript's +`createIndexAsync` return job handles. In OSS, the work runs in your application +process; in Enterprise, it runs on server workers. A returned handle can +represent an operation that has already finished, so check its status before +waiting or requesting cancellation. See the operation's API reference for when +the method returns. -## What is its lifecycle? {#waiting} +## Submit and wait for a job {#waiting} 1. **Submit the operation.** Keep the returned job handle. For an Enterprise job, also keep its ID if another process needs to look it up later. 2. **Check progress.** Read the current status while the operation runs. -3. **Wait for an outcome.** A job can finish, fail or be cancelled. Inspect its - result or failure information before continuing dependent work. +3. **Wait for completion.** Check that the job succeeded before using its + result. If it failed or was cancelled, inspect the failure details and any + results it committed before stopping. In Python, `job.status()` reports the current state and `job.wait()` waits for -completion, raising if the job fails or is cancelled. A wait timeout ends the -wait; it does not request cancellation of the server job. +completion, raising if the job fails or is cancelled. If `job.wait()` times out, the client stops waiting. +The timeout does not cancel the server job. -## How do you manage a job? {#managing} +## Inspect and cancel jobs {#managing} | Task | What to do | | --- | --- | @@ -53,7 +57,7 @@ Event responses have [row limits](/basics/limits#jobs). An explicit limit can request more than the default, but it does not guarantee a complete event history. -## Where does the work run? {#in-process} +## OSS and Enterprise execution {#in-process} | | OSS | Enterprise | | --- | --- | --- | @@ -62,9 +66,6 @@ history. | Event history | No server event history | Available through the jobs API | | Reattaching after a client restart | No server job to reopen | Reopen by ID while the record is retained | -A handle can represent work that has already completed. Check its state rather -than assuming every returned job is still running. - ## Learn more {#next} - [Build and repair indexes](/indexing/reindexing): submit index work. diff --git a/docs/web/basics/lakehouse.mdx b/docs/web/basics/lakehouse.mdx index 76432a849..1d2deb1fb 100644 --- a/docs/web/basics/lakehouse.mdx +++ b/docs/web/basics/lakehouse.mdx @@ -6,7 +6,7 @@ icon: layer-group keywords: ["lakehouse", "multimodal", "search", "compute", "training", "feature engineering"] --- -## What is the LanceDB lakehouse? {#multimodal} +## Multimodal storage and processing {#multimodal} LanceDB brings storage, search and data processing together for AI workloads. You can store multimodal data, derive features, search your records and prepare @@ -15,7 +15,7 @@ training datasets using tables built on the open [Lance format](/basics/lance). The format provides the storage foundation. The [catalog](/basics/catalog) organizes your tables. LanceDB provides the tools to work with that data. -## How does the data flow? {#capabilities} +## Example: a video search workflow {#capabilities} Consider a team building a video search application: @@ -42,7 +42,7 @@ The clips, metadata and derived features stay associated in the same table, so retrieval and training workflows can use the same dataset. Your training framework runs the model; LanceDB provides access to the training data. -## What does Enterprise add? {#where} +## Shared compute and maintenance in Enterprise {#where} LanceDB OSS runs as an embedded library in your application. LanceDB Enterprise provides shared infrastructure for applications and teams working with larger diff --git a/docs/web/basics/lance.mdx b/docs/web/basics/lance.mdx index 5552807ff..f2dd0ecfa 100644 --- a/docs/web/basics/lance.mdx +++ b/docs/web/basics/lance.mdx @@ -6,7 +6,7 @@ icon: "/static/assets/logo/lance-logo-gray.svg" keywords: ["lance", "format", "multimodal", "versioning", "column evolution"] --- -## What is Lance? {#standard} +## Lance format overview {#standard} [Lance](https://lance.org/) is an open-source, columnar format designed for multimodal AI. It defines how data is stored and organized into versioned tables. @@ -18,14 +18,14 @@ LanceDB OSS and LanceDB Enterprise both use Lance as their storage foundation. The format is developed as an independent open-source project, with its own specification and tools. -## What can it store? {#storage} +## Multimodal data {#storage} A Lance table can hold images, video, audio, text, embeddings and structured metadata together. For example, a video dataset can keep each clip alongside its caption, labels and embeddings, so applications can search and work with those fields together. -## What does the format enable? {#capabilities} +## Data access, versioning and column updates {#capabilities} - **Efficient data access.** Read individual rows for retrieval and training, or scan columns for analytics. diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx index a3b646cb1..0d4f1b028 100644 --- a/docs/web/basics/limits.mdx +++ b/docs/web/basics/limits.mdx @@ -6,11 +6,11 @@ icon: ruler keywords: ["limits", "maximum", "quota", "constraints", "bounds"] --- -## Which limits apply? {#scope} + -Some limits come from an API or data type; others come from Enterprise -configuration. Use the scope and linked reference below to identify which -applies to your operation. SQL and server job limits apply to Enterprise. +This page lists limits for job events, SQL error reports, sampling and +full-text indexing. Each table identifies the affected API and what happens +when a request reaches its limit. SQL and server job limits apply to Enterprise. @@ -21,7 +21,7 @@ enforce. -## What are the fixed limits? {#fixed-limits} +## Fixed API limits {#fixed-limits} | Operation | Scope | Limit and behavior | | --- | --- | --- | @@ -42,7 +42,7 @@ before choosing a schema or index. For materialized views, check [stable row ID requirements](/feature-engineering/materialized-views#stable-row-ids) before creating source tables. -## What can you configure? {#not-quotas} +## Configurable limits {#not-quotas} | Setting | What it controls | | --- | --- | @@ -50,7 +50,7 @@ before creating source tables. | FTS `max_token_length` | The maximum indexed token length. The default is 40; longer tokens are excluded from the index. | | Enterprise capacity and quotas | Storage, request rate and concurrency are governed by your deployment's resources and configuration. Confirm these with your administrator or LanceDB support. | -## What happens at a limit? {#behavior} +## Handling rejected or incomplete results {#behavior} - **Rejected requests:** correct the input before retrying, such as a sampling percentage outside the accepted range. diff --git a/docs/web/basics/query.mdx b/docs/web/basics/query.mdx index b8a70a330..624617918 100644 --- a/docs/web/basics/query.mdx +++ b/docs/web/basics/query.mdx @@ -6,14 +6,14 @@ icon: magnifying-glass keywords: ["query", "search", "SQL", "async", "streaming", "results"] --- -## What is a query? {#queries} +## Querying tables {#queries} A query selects data from your tables. You can filter rows, choose columns, search for similar content or combine those operations. For example, a video application might find clips similar to a sample and return only those recorded in the last month. -## How can you query? {#searches} +## SDK searches and SQL queries {#searches} | Interface | When to use it | | --- | --- | @@ -24,26 +24,27 @@ Both interfaces work with LanceDB tables. Choose the one that supports your operation and fits your application. See [SQL](/basics/sql) for connection instructions and SQL search support, and [Search](/search) for the query builders. -## How do results arrive? {#results} +## Collect or stream results {#results} - **Collect the results** when you want a list, Arrow table or DataFrame that fits in memory. Use a result limit and select only the columns you need. - **Read batches** when you want to process a large result incrementally. - Streaming avoids collecting the entire result in application memory, although - some query operations still need to do substantial work before producing rows. + Reading batches avoids loading the entire result into application memory. + It does not guarantee that the query can return its first batch immediately. - +### Asynchronous clients {#async-client} An async client lets your application do other work while a request is waiting. That is separate from whether you collect results or read them in batches. Python offers synchronous and asynchronous clients; TypeScript uses async APIs. Check the SDK reference for the methods available in your client. - +### Query handles and background jobs {#jobs} -Enterprise SQL can also return a query handle for inspecting execution and -requesting cancellation. A [job](/basics/jobs) tracks an operation such as an -index build. Use the result API for the operation you submitted. +An Enterprise SQL query handle lets you inspect query execution and request +cancellation. Background operations such as index builds return a +[job](/basics/jobs) handle instead. Use the API for the returned handle to +check status and retrieve results. ## Learn more {#next} diff --git a/docs/web/tables/consistency.mdx b/docs/web/tables/consistency.mdx index 229835280..388655ff8 100644 --- a/docs/web/tables/consistency.mdx +++ b/docs/web/tables/consistency.mdx @@ -13,16 +13,18 @@ availability: import { PyReadConsistency } from '/snippets/read_consistency.mdx'; -## What does consistency mean here? {#consistency} +## Table snapshots and freshness {#consistency} -A table read uses a consistent snapshot of the data. **Freshness** describes how -recent that snapshot is. A reader can see a coherent earlier version while -another process has already committed newer data. +Each table read uses one committed version of the data. **Freshness** describes +whether that version includes recent writes. For example, after another +application appends records, an already-open table handle can still read the +version from before the append. For example, a dashboard may need the latest updates, while an experiment may -need to keep reading a specific table version. These are different requirements. +need to keep reading a specific table version. Use read freshness settings for the dashboard and an explicit table version +for the experiment. -## When do changes become visible? {#visibility} +## Visibility of writes {#visibility} In **OSS**, a table handle sees its own committed writes. By default, it does not automatically check for changes made through other handles or processes. @@ -33,10 +35,10 @@ A connection can request fresher data. When a read must include a completed write, use a zero interval or call `checkout_latest()` after the write, with a client and server that support these freshness controls. -## How do you control freshness? {#configure-consistency-parameters} +## Configure read freshness {#configure-consistency-parameters} -Set `read_consistency_interval` when connecting. The setting applies to reads; -it does not change the transaction boundary of a write. +Set `read_consistency_interval` when connecting. This controls when reads +pick up newer table versions; it does not change how writes commit. | Setting | OSS | Enterprise | | --- | --- | --- | @@ -64,7 +66,7 @@ interval sees the new row on its next read. -## How should you choose? {#choosing} +## Choose a freshness interval {#choosing} - **Reads that must include recent writes:** use a zero interval. Checking more frequently can increase read latency and storage requests. @@ -73,8 +75,8 @@ interval sees the new row on its next read. - **Reproducible experiments:** read an explicit table version and retain it for as long as the experiment needs it. -There is no single interval that suits every application. Choose based on how -stale a result your users can accept. +Choose the interval based on the delay your application can tolerate between +a write completing and a reader seeing it. ## Learn more {#next}