diff --git a/docs/web/catalog/database.mdx b/docs/web/catalog/database.mdx new file mode 100644 index 000000000..f9c78173f --- /dev/null +++ b/docs/web/catalog/database.mdx @@ -0,0 +1,94 @@ +--- +title: "Database" +sidebarTitle: "Database" +description: "The root of a catalog. Connecting to one, and the operations it holds." +icon: database +--- + +A database is the root of a catalog. A connection opens exactly one, and every +other object is reached through it. + +## Connecting {#connecting} + +```python +import lancedb + +db = lancedb.connect("./.lancedb") # a local directory +db = lancedb.connect("s3://bucket/path") # object storage +db = lancedb.connect("db://my-database", api_key="...") # Cloud or Enterprise +``` + +The URI decides what kind of database you get, and that decision reaches +further than it looks. A local or object-store URI opens the data directly from +your process. A `db://` URI talks to a deployment, which is what makes +server-side execution — [functions](/feature-engineering/functions), remote refreshes, +[SQL](/sql) — available, and what makes a few local-only features unavailable. + +`connect_async` returns an `AsyncConnection` with the same surface as +coroutines. + +### Storage options {#storage-options} + +`storage_options` configures the underlying store — credentials, endpoints, and +per-table defaults for tables created through this connection: + +```python +db = lancedb.connect( + "s3://bucket/path", + storage_options={"new_table_enable_stable_row_ids": "true"}, +) +``` + + +`new_table_enable_stable_row_ids` is worth setting deliberately. Stable row ids +cannot be enabled after a table exists, and a +[materialized view](/feature-engineering/materialized-views) requires its source to have +them. A table created without them cannot later become a view's source without +being recreated. + + +See [Storage](/storage) for the full set. + +## Operations {#operations} + +| | | +| --- | --- | +| `db.table_names()` | list the tables in the database | +| `db.create_table(name, data)` | create a table — see [Table](/tables) | +| `db.open_table(name)` | open an existing table | +| `db.drop_table(name)` | drop one table | +| `db.drop_database()` | drop everything in the database | + +Each accepts `namespace_path` to act inside a [namespace](/namespaces) +rather than at the root. + +```python +db.table_names() +tbl = db.open_table("documents") +db.drop_table("documents") +``` + + +`drop_database()` removes every table in the database. There is no confirmation +and no undo — table [versioning](/tables/versioning) does not survive the +table being dropped. + + +## Read consistency {#consistency} + +`read_consistency_interval` controls how often a connection checks for versions +committed by other writers. The default reads a snapshot taken when the table +was opened, so a long-lived connection will not see other processes' writes +until it reopens. + +Set it when several processes write the same table and readers must observe +those writes: + +```python +from datetime import timedelta + +db = lancedb.connect(uri, read_consistency_interval=timedelta(seconds=5)) +``` + +The cost is a metadata check on that interval. See +[Consistency](/tables/consistency). diff --git a/docs/web/catalog/index.mdx b/docs/web/catalog/index.mdx new file mode 100644 index 000000000..a993a2bf5 --- /dev/null +++ b/docs/web/catalog/index.mdx @@ -0,0 +1,58 @@ +--- +title: "Catalog" +sidebarTitle: "Overview" +description: "What lives in a LanceDB catalog: databases, namespaces, tables, views, and functions." +icon: sitemap +--- + +A catalog is what LanceDB knows about. Everything you create lives in one and is +addressed through it — the objects below, arranged in a hierarchy. + +This section covers what each object is and the basic operations on it. Where an +object has depth — how to search a table, how to refresh a view — the page links +to the section that covers it rather than repeating it. + +## The objects {#objects} + +| Object | What it is | +| --- | --- | +| [Database](/catalog/database) | The root. A connection opens one. | +| [Namespace](/namespaces) | A named grouping of tables inside a database. | +| [Table](/tables) | Rows and columns, versioned. The thing most operations act on. | +| [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. | +| [View](/catalog/view) | A saved query. Computed when read, not stored. | +| [Function](/feature-engineering/functions) | Registered code, applied to produce column values. | + +## How they nest {#hierarchy} + +``` +database +└── namespace + ├── table + ├── materialized view (a table, defined by a query) + ├── view (a query, not stored) + └── function +``` + +A namespace can contain further namespaces, so the hierarchy is as deep as your +organisation needs. A table is reached by its path through that hierarchy. + +## Views and materialized views are different objects {#views} + +The names are close enough to mislead. Both are defined by a query over other +tables; the difference is when that query runs. + +- A **materialized view** stores its rows. A refresh computes them, and reads + are reads of stored data — fast, and as current as the last refresh. +- A **view** stores only the query. Every read runs it, so a view is never stale + and never faster than the query it wraps. + +Reach for a materialized view when the query is expensive and the data changes +less often than it is read. Reach for a view when correctness matters more than +latency, or when the query is cheap. + +## What is not here {#elsewhere} + +Indexes belong to a table but are covered under [Maintenance](/indexing), +because creating one is a decision about performance rather than about +structure. diff --git a/docs/web/docs.json b/docs/web/docs.json index 0293bdb36..37c874110 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -26,90 +26,75 @@ } }, "favicon": "/static/favicon.ico", + "redirects": [ + { + "source": "/tables/update", + "destination": "/ingestion/update" + } + ], "navigation": { "tabs": [ { "tab": "Documentation", "groups": [ { - "group": "Get started", + "group": "Introduction", "pages": [ - "quickstart", - { - "group": "What is LanceDB?", - "pages": [ - "index", - "lance", - "tables-and-namespaces" - ] - }, - "performance" + "index", + "quickstart" ] }, { - "group": "Model training", - "pages": [ - "training/why-lancedb", - "training/index", - "training/torch", - "training/object-detection", - "training/vlm-finetuning" - ] - }, - { - "group": "Guides", + "group": "Catalog", "pages": [ + "catalog/index", + "catalog/database", { - "group": "Table operations", - "pages": [ - "tables/index", - "tables/create", - "tables/multimodal", - "tables/schema", - "tables/update", - "tables/versioning", - "tables/branching", - "tables/consistency" - ] - }, - { - "group": "Namespaces", + "group": "Namespace", "pages": [ "namespaces/index", "namespaces/usage" ] }, { - "group": "Embeddings", + "group": "Table", "pages": [ - "embedding/index", - "embedding/quickstart" - ] - }, - { - "group": "Indexing", - "pages": [ - "indexing/index", - "indexing/vector-index", - "indexing/fts-index", - "indexing/scalar-index", - "indexing/quantization", - "indexing/reindexing" - ] - }, - { - "group": "Search", - "pages": [ - "search/index", - "search/vector-search", - "search/multivector-search", - "search/full-text-search", - "search/fts-examples", - "search/hybrid-search", - "search/filtering", - "search/optimize-queries" + "tables/index", + "tables/create", + "tables/schema", + "tables/multimodal" ] }, + "feature-engineering/materialized-views", + "feature-engineering/functions", + "lance", + "tables-and-namespaces" + ] + }, + { + "group": "Ingestion", + "pages": [ + "ingestion/index", + "ingestion/insert", + "ingestion/merge-insert", + "ingestion/update", + "ingestion/delete", + "tables/versioning", + "tables/branching", + "tables/consistency" + ] + }, + { + "group": "Search", + "pages": [ + "search/index", + "search/vector-search", + "search/multivector-search", + "search/full-text-search", + "search/fts-examples", + "search/hybrid-search", + "search/filtering", + "search/optimize-queries", { "group": "Reranking", "pages": [ @@ -123,19 +108,10 @@ ] }, { - "group": "Storage", + "group": "Embedding", "pages": [ - "storage/index", - "storage/configuration", - "storage/monitoring" - ] - }, - { - "group": "Build with AI agents", - "expanded": true, - "pages": [ - "build-with-ai-agents", - "agent-branch-experiments" + "embedding/index", + "embedding/quickstart" ] } ] @@ -144,9 +120,36 @@ "group": "Feature engineering", "pages": [ "feature-engineering/index", - "feature-engineering/computed-columns", - "feature-engineering/materialized-views", - "feature-engineering/functions" + "feature-engineering/computed-columns" + ] + }, + { + "group": "Training", + "pages": [ + "training/why-lancedb", + "training/index", + "training/torch" + ] + }, + { + "group": "Maintenance", + "pages": [ + "indexing/index", + "indexing/vector-index", + "indexing/fts-index", + "indexing/scalar-index", + "indexing/quantization", + "indexing/reindexing", + "maintenance/compaction", + { + "group": "Storage", + "pages": [ + "storage/index", + "storage/configuration", + "storage/monitoring" + ] + }, + "performance" ] }, { @@ -266,7 +269,9 @@ "tutorials/agents/multimodal-agent/index" ] }, - "tutorials/feature-engineering/index" + "tutorials/feature-engineering/index", + "training/object-detection", + "training/vlm-finetuning" ] } ] @@ -316,6 +321,18 @@ ] } ] + }, + { + "tab": "Agentic Skills", + "groups": [ + { + "group": "Agentic Skills", + "pages": [ + "build-with-ai-agents", + "agent-branch-experiments" + ] + } + ] } ] }, diff --git a/docs/web/ingestion/delete.mdx b/docs/web/ingestion/delete.mdx new file mode 100644 index 000000000..08281b9fe --- /dev/null +++ b/docs/web/ingestion/delete.mdx @@ -0,0 +1,73 @@ +--- +title: "Delete rows" +sidebarTitle: "Delete" +description: "Remove rows by predicate, and reclaim their space." +icon: trash +--- + +These examples use the table set up in +[Ingesting data](/ingestion), and the connection made there. + +import { + PyDeleteOperation as DeleteOperation, + PyUpdateOptimizeCleanup as UpdateOptimizeCleanup, + TsDeleteOperation, + TsUpdateOptimizeCleanup, + RsDeleteOperation, + RsUpdateOptimizeCleanup, +} from '/snippets/tables.mdx'; + +## Delete rows {#delete-rows} + +Delete operations **soft delete** rows that match a given condition. +The underlying data is not immediately removed, but is marked +for deletion (in the [deletion files](https://lance.org/format/table/#deletion-files) at the Lance format level) and excluded from query results. + + + + {DeleteOperation} + + + + {TsDeleteOperation} + + + + {RsDeleteOperation} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bob | 20 | + + + +**Deleting rows removes them from the index** + +When rows are deleted, those rows are also excluded from the index segments, so indexed queries will not return them either. If ALL the rows are deleted (i.e., the table is emptied), ensure that you recreate the index after ingesting new data. + + +To permanently remove deleted rows, you can optimize the table, which will run compaction and cleans up the soft-deleted rows, which frees up storage space. + +- In LanceDB OSS, compaction and cleanup are manual. Run `table.optimize()` regularly to free up disk space. +- In LanceDB Enterprise, files aren't cleaned up by default. You can configure automatic compaction and cleanup behavior at cluster setup time to suit your organization's retention policy. + +By default, table cleanup removes data up to 7 days ago. If you need to reclaim space from deleted rows more aggressively, manually call `table.optimize()` use a shorter retention window as follows: + + + + {UpdateOptimizeCleanup} + + + + {TsUpdateOptimizeCleanup} + + + + {RsUpdateOptimizeCleanup} + + diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx new file mode 100644 index 000000000..7b79ae323 --- /dev/null +++ b/docs/web/ingestion/index.mdx @@ -0,0 +1,107 @@ +--- +title: "Ingesting data" +sidebarTitle: "Overview" +description: "The four write operations, and the example table the rest of this section uses." +icon: inbox +--- + +import { + PyUpdateConnectEnterprise as UpdateConnectEnterprise, + PyUpdateConnectLocal as UpdateConnectLocal, + PyUpdateExampleTableSetup as UpdateExampleTableSetup, + TsUpdateConnectEnterprise, + TsUpdateConnectLocal, + TsUpdateExampleTableSetup, + RsUpdateConnectEnterprise, + RsUpdateConnectLocal, + RsUpdateExampleTableSetup, + RsUpdateMakeUsersReader, +} from '/snippets/tables.mdx'; + +## Choose a write method {#choose-a-write-method} + +| Family | Method | Use this when... | +| --------------- | --------------- | ---------------- | +| [`add`](/ingestion/insert) | `add(data)` | You are appending new rows and do not need to match against existing ones. | +| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. | +| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. | +| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. | +| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). | +| `merge_insert` | `.when_not_matched_by_source_delete(...)` | You want to remove target rows that are missing from the incoming source set. | + +Write operations return status metadata. For example, `update` reports updated row count and +committed `version`, `delete` reports deleted row count and `version`, and `merge_insert` reports +inserted, updated, deleted, retry-attempt, and `version` fields. Writes require a mutable table +handle; after checking out an older version for reads, call `checkout_latest` / `checkoutLatest` +before modifying data. +The committed `version` advances even for writes that affect zero rows, such as a delete predicate +that matches nothing. + +## Connect to LanceDB {#connect-to-lancedb} + +Connect to your local LanceDB instance: + + + + {UpdateConnectLocal} + + + + {TsUpdateConnectLocal} + + + + {RsUpdateConnectLocal} + + + +Or, connect to LanceDB Enterprise: + + + + {UpdateConnectEnterprise} + + + + {TsUpdateConnectEnterprise} + + + + {RsUpdateConnectEnterprise} + + + +In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data. + + + + {RsUpdateMakeUsersReader} + + + +## Create the example table {#create-the-example-table} + +We'll start by creating a simple table with `id`, `name`, and `login_count` columns. All examples below use the same table. + + + {UpdateExampleTableSetup} + + + + {TsUpdateExampleTableSetup} + + + + {RsUpdateExampleTableSetup} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bob | 20 | + +The example above shows a PyArrow schema. You can just as well create the table using other +table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [Creating a table](/tables/create) for more details. diff --git a/docs/web/ingestion/insert.mdx b/docs/web/ingestion/insert.mdx new file mode 100644 index 000000000..cc49b354c --- /dev/null +++ b/docs/web/ingestion/insert.mdx @@ -0,0 +1,120 @@ +--- +title: "Insert data" +sidebarTitle: "Insert" +description: "Append rows to a table, including datasets too large to hold in memory." +icon: plus +--- + +Rows are appended with `table.add()`. It accepts the same shapes +[table creation](/tables/create) does — lists of objects, Arrow tables, pandas +and polars frames, Pydantic models — and commits a new table version. + +These examples use the table set up in [Ingesting data](/ingestion). + +import { + PyAddFromDataset as AddFromDataset, + PyCreateTableFromIterator as CreateTableFromIterator, + TsCreateTableFromIterator as TsCreateTableFromIterator, + RsCreateTableFromIterator as RsCreateTableFromIterator, + TsAddProgress, +} from '/snippets/tables.mdx'; + +## Ingesting large datasets {#large-datasets} + +When ingesting large datasets, use `table.add()` on an existing table rather than +passing all data to `create_table()`. The `add()` method auto-parallelizes large +writes, while `create_table(name, data)` does not. + + +For best performance with large datasets, create an empty table first and then call +`table.add()`. This enables automatic write parallelism for materialized data sources. + + +### From files (Parquet, CSV, etc.) {#from-files-parquet-csv-etc} +Python Only + +For file-based data, pass a `pyarrow.dataset.Dataset` to `table.add()`. This streams +data from disk without loading the entire dataset into memory. + + + + {AddFromDataset} + + + + +`pa.dataset()` input is currently Python-only. TypeScript and Rust support for +file-based dataset ingestion is tracked in +[lancedb#3173](https://github.com/lancedb/lancedb/issues/3173). + + +### From iterators (custom batch generation) {#from-iterators-custom-batch-generation} + +When you need custom batch logic — generating embeddings on the fly, transforming +rows from an external source, etc. — use an iterator of `RecordBatch` objects. + + + + {CreateTableFromIterator} + + + + {TsCreateTableFromIterator} + + + + {RsCreateTableFromIterator} + + + +Use this pattern when: + +- Your source data already arrives in Arrow batches, readers, datasets, or streams. +- Materializing the entire ingest as one giant in-memory list or array would be too expensive. +- You want to control chunk size explicitly during ingestion. + +Python can also consume iterators of other supported types like Pandas DataFrames or Python lists. + +### Write parallelism {#write-parallelism} + + +For materialized data (`pa.Table`, `pd.DataFrame`, `pa.dataset()`), LanceDB +automatically parallelizes large writes — no configuration needed. Auto-parallelism +targets approximately 1M rows or 2GB per write partition. + +For streaming sources (iterators, `RecordBatchReader`), LanceDB cannot determine +total size upfront. A `parallelism` parameter to control this manually is planned +but not yet exposed in Python or TypeScript +([tracking issue](https://github.com/lancedb/lancedb/issues/3173)). + + +### Tracking ingestion progress {#tracking-ingestion-progress} +TypeScript Only + +For long-running writes, pass a `progress` callback to `table.add()` to surface +per-batch progress in your UI, logs, or metrics pipeline. The callback fires +once per batch written and once more with `done: true` when the write completes. + +Each invocation receives a `WriteProgress` object: + +| Field | Description | +|:------|:------------| +| `outputRows` | Rows written so far. | +| `outputBytes` | Bytes written so far. | +| `totalRows` | Expected total rows when the input source reports one. Always set on the final callback. | +| `elapsedSeconds` | Wall-clock seconds since the write started. | +| `activeTasks` | Parallel write tasks currently in flight. | +| `totalTasks` | Total parallel write tasks (the write parallelism). | +| `done` | `true` only for the final callback. | + + + + {TsAddProgress} + + + +A few things to know before you wire this up: + +- Back-pressures the writer: callback invocations are serialized and run inline with each batch, so a slow callback will slow the write rather than drop updates. Every batch update is delivered, and the final `done: true` callback always fires (even on error or cancellation). Keep the callback cheap — offload heavy work to a queue you drain elsewhere. +- Errors swallowed: anything your callback throws is logged with `console.warn` and won't abort the write, so keep the callback side-effect-only and don't rely on it for control flow. +- Row totals: `totalRows` is only populated when the input source can report it up front (for example, a materialized `arrow.Table`). For streaming sources it stays `undefined` until the final callback, where it falls back to the actual rows written. diff --git a/docs/web/ingestion/merge-insert.mdx b/docs/web/ingestion/merge-insert.mdx new file mode 100644 index 000000000..1db146574 --- /dev/null +++ b/docs/web/ingestion/merge-insert.mdx @@ -0,0 +1,210 @@ +--- +title: "Merge insert" +sidebarTitle: "Merge insert" +description: "Reconcile incoming rows against a key: update what matches, insert what does not." +icon: code-merge +--- + +These examples use the table set up in +[Ingesting data](/ingestion), and the connection made there. + +import { + PyMergeMatchedUpdateOnly as MergeMatchedUpdateOnly, + PyInsertIfNotExists as InsertIfNotExists, + PyMergeUpdateInsert as MergeUpdateInsert, + PyMergeDeleteMissingBySource as MergeDeleteMissingBySource, + PyMergePartialColumns as MergePartialColumns, + TsMergeMatchedUpdateOnly, + TsInsertIfNotExists, + TsMergeUpdateInsert, + TsMergeDeleteMissingBySource, + TsMergePartialColumns, + RsMergeMatchedUpdateOnly, + RsInsertIfNotExists, + RsMergeUpdateInsert, + RsMergeDeleteMissingBySource, + RsMergePartialColumns, +} from '/snippets/tables.mdx'; + +## Merge incoming rows by key {#merge-incoming-rows-by-key} + +Merging is different from updating because it involves comparing incoming rows to existing rows by key, and then choosing what to do based on whether the key exists in the target table or not. +The `merge_insert(""..."")` method lets you do this. + +In merge operations, rows are split into three groups: + +- **Matched**: key exists in both source and target. +- **Not matched**: key exists only in source. +- **Not matched by source**: key exists only in target. + +Conditional merge clauses can compare old and new values. Use the `target.` prefix for the +existing table row and `source.` for the incoming row, for example +`target.last_update < source.last_update`. + + +**Use scalar indexes to speed up merge insert** + +The merge insert command performs a join between the input data and the target table `on` the key you provide. This requires scanning that entire column, which can be expensive for large tables. To speed up this operation, create a scalar index on the join column, which will allow LanceDB to find matches without scanning the whole table. + +Read more about scalar indices in the [Scalar Index](/indexing/scalar-index/) guide. + + + +If you see this HTTP 400 error from `merge_insert`: `Bad request: Merge insert cannot be performed because the number of unindexed rows exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying. + + + +**Rust: build merge predicates with DataFusion expressions** + +In the Rust SDK, you can pass a `datafusion_expr::Expr` directly instead of a SQL string by using +`when_matched_update_all_expr` and `when_not_matched_by_source_delete_expr` on `MergeInsertBuilder`. +This is useful when you are constructing predicates programmatically and want to avoid building a SQL string. + +These methods are only supported on local tables. Calling them against a remote table returns a +`NotSupported` error — use the SQL string variants (`when_matched_update_all` / +`when_not_matched_by_source_delete`) for remote tables. + + +Like the create table and add APIs, the merge insert API will automatically compute embeddings based on the [embedding registry](/embedding/index#embedding-registry) if the table has an embedding definition in its schema. + +During `merge_insert`, if the input data doesn't contain the source column (i.e., the original field used to generate embeddings, such as text for a text embedding model or `image_uri` for an image model), or if a vector value is already provided, LanceDB skips embedding generation for that row. Embeddings are only auto-generated when that source field is present in the incoming data, **and** the vector field is empty. + +Primary keys in LanceDB are metadata used by operations such as `merge_insert`; they are not +enforced as uniqueness constraints on ordinary writes. Keep using `merge_insert` or your own +deduplication logic when you need key-based upsert semantics. + +### Update matched rows only {#update-matched-rows-only} + +This updates keys that already exist in the target table. Source rows with new keys are ignored. + + + + {MergeMatchedUpdateOnly} + + + + {TsMergeMatchedUpdateOnly} + + + + {RsMergeMatchedUpdateOnly} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bobby | 21 | + +### Insert unmatched rows only {#insert-unmatched-rows-only} + +This inserts only brand-new keys from the source. Existing keys are left unchanged. + + + + {InsertIfNotExists} + + + + {TsInsertIfNotExists} + + + + {RsInsertIfNotExists} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bob | 20 | +| 3 | Charlie | 5 | + +### Update matched rows and insert unmatched rows {#update-matched-rows-and-insert-unmatched-rows} + +Use both `when_matched_update_all()` and `when_not_matched_insert_all()` when you want to update existing keys and insert missing keys in one operation. + + +This is a conventional **upsert**. + + + + + {MergeUpdateInsert} + + + + {TsMergeUpdateInsert} + + + + {RsMergeUpdateInsert} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bobby | 21 | +| 3 | Charlie | 5 | + +### Delete target rows that are missing from source {#delete-target-rows-that-are-missing-from-source} + +Use `when_not_matched_by_source_delete()` when you want to remove any target row that does not appear in the incoming source data. + + + + {MergeDeleteMissingBySource} + + + + {TsMergeDeleteMissingBySource} + + + + {RsMergeDeleteMissingBySource} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 2 | Bobby | 21 | +| 3 | Charlie | 5 | + +In the example above, LanceDB matches rows by `id`. Rows with `id=2` and `id=3` exist in both the table and incoming data, so they are updated. Row `id=1` exists only in the target, so it is deleted. + +### Use partial columns in merge updates {#use-partial-columns-in-merge-updates} + +Merge updates do not require you to provide values for all columns. You can provide only a subset of columns in source rows. For matched rows, only the provided columns are updated. + + + + {MergePartialColumns} + + + + {TsMergePartialColumns} + + + + {RsMergePartialColumns} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bobby | 20 | +| 3 | Charlie | null | + +Note that in the example above, when `merge_insert` creates a new row, any missing columns are written as `null`. If a missing column is non-nullable in your schema, the insert will fail. diff --git a/docs/web/ingestion/update.mdx b/docs/web/ingestion/update.mdx new file mode 100644 index 000000000..04a4206de --- /dev/null +++ b/docs/web/ingestion/update.mdx @@ -0,0 +1,78 @@ +--- +title: "Update rows" +sidebarTitle: "Update" +description: "Change values in rows that already exist, by predicate or SQL expression." +icon: pen +--- + +These examples use the table set up in +[Ingesting data](/ingestion), and the connection made there. + +import { + PyUpdateOperation as UpdateOperation, + PyUpdateUsingSql as UpdateUsingSql, + TsUpdateOperation, + TsUpdateUsingSql, + RsUpdateOperation, + RsUpdateUsingSql, +} from '/snippets/tables.mdx'; + +## Update rows {#update-rows} + +Use `update` when you already know which target rows to modify and you do not need to compare against an incoming dataset. + + + + {UpdateOperation} + + + + {TsUpdateOperation} + + + + {RsUpdateOperation} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bobby | 20 | + + +Updating nested columns is not yet supported. + + +## Update rows with SQL expressions {#update-rows-with-sql-expressions} + +Use `values_sql` when you want to use SQL-like expressions to update rows. This is useful for operations like incrementing a counter, or setting a column value based on another column. + + + + {UpdateUsingSql} + + + + {TsUpdateUsingSql} + + + + {RsUpdateUsingSql} + + + +Expected table contents: + +| id | name | login_count | +| --- | --- | --- | +| 1 | Alice | 10 | +| 2 | Bob | 21 | + + +See the [SQL queries](/search/sql/) page for more information on the supported SQL syntax. + + +When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards. diff --git a/docs/web/maintenance/compaction.mdx b/docs/web/maintenance/compaction.mdx new file mode 100644 index 000000000..13fdda503 --- /dev/null +++ b/docs/web/maintenance/compaction.mdx @@ -0,0 +1,101 @@ +--- +title: "Compaction and cleanup" +sidebarTitle: "Compaction" +description: "Merge small files, prune old versions, and fold new rows into existing indexes." +icon: broom +--- + +Every write commits a new table version, and versions accumulate: small files +from frequent appends, superseded files from updates and deletes, and rows added +since an index was built. `optimize()` addresses all three. + +It is modelled after `VACUUM` in PostgreSQL — routine housekeeping that a table +needs periodically rather than a repair for something broken. + +## What it does {#what-it-does} + +```python +table.optimize() +``` + +Three operations, in one call: + +| | | +| --- | --- | +| **Compaction** | merges small files into larger ones | +| **Prune** | removes versions older than the retention window | +| **Index** | folds newly added rows into existing indexes | + +The third is the one people miss. An [index](/indexing) does not +automatically cover rows appended after it was built — those rows are still +searchable, but by scanning rather than through the index. Running `optimize()` +is what brings them under it. + +## Retention {#retention} + +```python +from datetime import timedelta + +table.optimize(cleanup_older_than=timedelta(days=30)) +``` + +`cleanup_older_than` defaults to **7 days**. Versions older than that are +removed, along with the files only they referenced. + +The latest version is never removed, whatever you pass — `timedelta(0)` prunes +everything except it. + + +Pruning is what makes [time travel](/tables/versioning) finite. A version +outside the retention window is gone, and a tag or branch pointing at it no +longer resolves. + +If you rely on reading old versions, set the window to cover the range you need +before running this, not after. + + +## Files from failed transactions {#unverified} + +A write that fails partway can leave files that look like an operation still in +progress. `optimize()` will not remove them until they are at least seven days +old, because it cannot tell them apart from a live write. + +```python +table.optimize(delete_unverified=True) +``` + + +`delete_unverified=True` removes them regardless of age. Only set it when you +can guarantee **no other process is writing to the table** — otherwise it can +delete files belonging to a write that is genuinely in progress, and corrupt the +table. + + +## Retraining indexes {#retrain} + +```python +table.optimize(retrain=True) +``` + +Folding new rows into an index preserves the structure the index was trained on. +When the data has shifted enough that the structure itself is a poor fit — a +vector index whose clusters no longer reflect the distribution — `retrain=True` +rebuilds it rather than extending it. + +More expensive than a normal optimize, and worth it after a large ingest that +changed the shape of the data rather than just its volume. + +## When to run it {#when} + +There is no background compactor: `optimize()` runs when you call it. + +Frequent small appends are what make it necessary, because each one commits +files. A table written in a few large batches needs it rarely; one taking +continuous small writes needs it on a schedule. + + +Compaction rewrites files, which conflicts with a concurrent +[computed-column refresh](/feature-engineering/computed-columns) or +[view refresh](/feature-engineering/materialized-views) planned against the +version it replaces. Sequence them rather than running both at once. + diff --git a/docs/web/tables/branching.mdx b/docs/web/tables/branching.mdx index 2f4f0c336..9e0d06339 100644 --- a/docs/web/tables/branching.mdx +++ b/docs/web/tables/branching.mdx @@ -223,7 +223,7 @@ How you apply a validated change depends on the type of work: ### Upsert selected branch rows into `main` {#upsert-selected-branch-rows-into-main} If the result you want to retain is a set of inserted or updated rows, use -[`merge_insert`](/tables/update#merge-incoming-rows-by-key) to write them to +[`merge_insert`](/ingestion/merge-insert#merge-incoming-rows-by-key) to write them to `main`. Despite its name, `merge_insert` is a row-ingestion operation: it matches incoming rows by key and does not merge branch histories. diff --git a/docs/web/tables/create.mdx b/docs/web/tables/create.mdx index 167a0d402..735b1ceaf 100644 --- a/docs/web/tables/create.mdx +++ b/docs/web/tables/create.mdx @@ -117,7 +117,7 @@ You can change this behavior with two parameters that resolve the conflict in di `exist_ok` / `existOk` does not append the provided data to an existing table. Use -[`table.add()`](/tables/update) for that. If you need to ensure a table exists *and* +[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and* contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table). @@ -398,7 +398,7 @@ that has been extended to support LanceDB specific types like `Vector`. Once the empty table has been created, you can append to it or modify its contents, -as explained in the [updating and modifying tables](/tables/update) section. +as explained in [Ingesting data](/ingestion). ## Open an existing table {#open-an-existing-table} diff --git a/docs/web/tables/index.mdx b/docs/web/tables/index.mdx index de34d4e78..3c2f5e689 100644 --- a/docs/web/tables/index.mdx +++ b/docs/web/tables/index.mdx @@ -649,11 +649,11 @@ advanced features of LanceDB. Below are some suggested next pages. Learn the different approaches to creating and ingesting data into LanceDB tables from various sources. - Learn how to update and modify existing LanceDB tables and their data. + Insert, merge, update and delete rows in an existing table. - - {UpdateConnectLocal} - - - - {TsUpdateConnectLocal} - - - - {RsUpdateConnectLocal} - - - -Or, connect to LanceDB Enterprise: - - - - {UpdateConnectEnterprise} - - - - {TsUpdateConnectEnterprise} - - - - {RsUpdateConnectEnterprise} - - - -In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data. - - - - {RsUpdateMakeUsersReader} - - - -## Create the example table {#create-the-example-table} - -We'll start by creating a simple table with `id`, `name`, and `login_count` columns. All examples below use the same table. - - - {UpdateExampleTableSetup} - - - - {TsUpdateExampleTableSetup} - - - - {RsUpdateExampleTableSetup} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bob | 20 | - -The example above shows a PyArrow schema. You can just as well create the table using other -table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see the [ingestion](/tables/create/) guide for more details. - -## Choose a write method {#choose-a-write-method} - -| Family | Method | Use this when... | -| --------------- | --------------- | ---------------- | -| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. | -| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. | -| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. | -| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). | -| `merge_insert` | `.when_not_matched_by_source_delete(...)` | You want to remove target rows that are missing from the incoming source set. | - -Write operations return status metadata. For example, `update` reports updated row count and -committed `version`, `delete` reports deleted row count and `version`, and `merge_insert` reports -inserted, updated, deleted, retry-attempt, and `version` fields. Writes require a mutable table -handle; after checking out an older version for reads, call `checkout_latest` / `checkoutLatest` -before modifying data. -The committed `version` advances even for writes that affect zero rows, such as a delete predicate -that matches nothing. - -## Update rows {#update-rows} - -Use `update` when you already know which target rows to modify and you do not need to compare against an incoming dataset. - - - - {UpdateOperation} - - - - {TsUpdateOperation} - - - - {RsUpdateOperation} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bobby | 20 | - - -Updating nested columns is not yet supported. - - -## Update rows with SQL expressions {#update-rows-with-sql-expressions} - -Use `values_sql` when you want to use SQL-like expressions to update rows. This is useful for operations like incrementing a counter, or setting a column value based on another column. - - - - {UpdateUsingSql} - - - - {TsUpdateUsingSql} - - - - {RsUpdateUsingSql} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bob | 21 | - - -See the [SQL queries](/search/sql/) page for more information on the supported SQL syntax. - - -When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards. - -## Merge incoming rows by key {#merge-incoming-rows-by-key} - -Merging is different from updating because it involves comparing incoming rows to existing rows by key, and then choosing what to do based on whether the key exists in the target table or not. -The `merge_insert(""..."")` method lets you do this. - -In merge operations, rows are split into three groups: - -- **Matched**: key exists in both source and target. -- **Not matched**: key exists only in source. -- **Not matched by source**: key exists only in target. - -Conditional merge clauses can compare old and new values. Use the `target.` prefix for the -existing table row and `source.` for the incoming row, for example -`target.last_update < source.last_update`. - - -**Use scalar indexes to speed up merge insert** - -The merge insert command performs a join between the input data and the target table `on` the key you provide. This requires scanning that entire column, which can be expensive for large tables. To speed up this operation, create a scalar index on the join column, which will allow LanceDB to find matches without scanning the whole table. - -Read more about scalar indices in the [Scalar Index](/indexing/scalar-index/) guide. - - - -If you see this HTTP 400 error from `merge_insert`: `Bad request: Merge insert cannot be performed because the number of unindexed rows exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying. - - - -**Rust: build merge predicates with DataFusion expressions** - -In the Rust SDK, you can pass a `datafusion_expr::Expr` directly instead of a SQL string by using -`when_matched_update_all_expr` and `when_not_matched_by_source_delete_expr` on `MergeInsertBuilder`. -This is useful when you are constructing predicates programmatically and want to avoid building a SQL string. - -These methods are only supported on local tables. Calling them against a remote table returns a -`NotSupported` error — use the SQL string variants (`when_matched_update_all` / -`when_not_matched_by_source_delete`) for remote tables. - - -Like the create table and add APIs, the merge insert API will automatically compute embeddings based on the [embedding registry](/embedding/index#embedding-registry) if the table has an embedding definition in its schema. - -During `merge_insert`, if the input data doesn't contain the source column (i.e., the original field used to generate embeddings, such as text for a text embedding model or `image_uri` for an image model), or if a vector value is already provided, LanceDB skips embedding generation for that row. Embeddings are only auto-generated when that source field is present in the incoming data, **and** the vector field is empty. - -Primary keys in LanceDB are metadata used by operations such as `merge_insert`; they are not -enforced as uniqueness constraints on ordinary writes. Keep using `merge_insert` or your own -deduplication logic when you need key-based upsert semantics. - -### Update matched rows only {#update-matched-rows-only} - -This updates keys that already exist in the target table. Source rows with new keys are ignored. - - - - {MergeMatchedUpdateOnly} - - - - {TsMergeMatchedUpdateOnly} - - - - {RsMergeMatchedUpdateOnly} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bobby | 21 | - -### Insert unmatched rows only {#insert-unmatched-rows-only} - -This inserts only brand-new keys from the source. Existing keys are left unchanged. - - - - {InsertIfNotExists} - - - - {TsInsertIfNotExists} - - - - {RsInsertIfNotExists} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bob | 20 | -| 3 | Charlie | 5 | - -### Update matched rows and insert unmatched rows {#update-matched-rows-and-insert-unmatched-rows} - -Use both `when_matched_update_all()` and `when_not_matched_insert_all()` when you want to update existing keys and insert missing keys in one operation. - - -This is a conventional **upsert**. - - - - - {MergeUpdateInsert} - - - - {TsMergeUpdateInsert} - - - - {RsMergeUpdateInsert} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bobby | 21 | -| 3 | Charlie | 5 | - -### Delete target rows that are missing from source {#delete-target-rows-that-are-missing-from-source} - -Use `when_not_matched_by_source_delete()` when you want to remove any target row that does not appear in the incoming source data. - - - - {MergeDeleteMissingBySource} - - - - {TsMergeDeleteMissingBySource} - - - - {RsMergeDeleteMissingBySource} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 2 | Bobby | 21 | -| 3 | Charlie | 5 | - -In the example above, LanceDB matches rows by `id`. Rows with `id=2` and `id=3` exist in both the table and incoming data, so they are updated. Row `id=1` exists only in the target, so it is deleted. - -### Use partial columns in merge updates {#use-partial-columns-in-merge-updates} - -Merge updates do not require you to provide values for all columns. You can provide only a subset of columns in source rows. For matched rows, only the provided columns are updated. - - - - {MergePartialColumns} - - - - {TsMergePartialColumns} - - - - {RsMergePartialColumns} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bobby | 20 | -| 3 | Charlie | null | - -Note that in the example above, when `merge_insert` creates a new row, any missing columns are written as `null`. If a missing column is non-nullable in your schema, the insert will fail. - -## Delete rows {#delete-rows} - -Delete operations **soft delete** rows that match a given condition. -The underlying data is not immediately removed, but is marked -for deletion (in the [deletion files](https://lance.org/format/table/#deletion-files) at the Lance format level) and excluded from query results. - - - - {DeleteOperation} - - - - {TsDeleteOperation} - - - - {RsDeleteOperation} - - - -Expected table contents: - -| id | name | login_count | -| --- | --- | --- | -| 1 | Alice | 10 | -| 2 | Bob | 20 | - - - -**Deleting rows removes them from the index** - -When rows are deleted, those rows are also excluded from the index segments, so indexed queries will not return them either. If ALL the rows are deleted (i.e., the table is emptied), ensure that you recreate the index after ingesting new data. - - -To permanently remove deleted rows, you can optimize the table, which will run compaction and cleans up the soft-deleted rows, which frees up storage space. - -- In LanceDB OSS, compaction and cleanup are manual. Run `table.optimize()` regularly to free up disk space. -- In LanceDB Enterprise, files aren't cleaned up by default. You can configure automatic compaction and cleanup behavior at cluster setup time to suit your organization's retention policy. - -By default, table cleanup removes data up to 7 days ago. If you need to reclaim space from deleted rows more aggressively, manually call `table.optimize()` use a shorter retention window as follows: - - - - {UpdateOptimizeCleanup} - - - - {TsUpdateOptimizeCleanup} - - - - {RsUpdateOptimizeCleanup} - -