mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Brendan Clement
027597d36d
Indexes are built in the embedded library and on a deployment, which builds them in the background, keeps most of them current, and builds only IVF_FLAT, IVF_PQ, IVF_RQ and IVF_HNSW_SQ vector indexes. Compaction runs on both, by optimize() locally and in the background on a deployment; cleanup on a deployment runs only where its configuration enables it, so it varies; background jobs exist only on a deployment. Sections that described one offering as if it were both now name it: optimize() is embedded only, the brute-force limit and automatic IVF_PQ are deployment settings, FM indexes get no background updates, a remainder below the deployment's threshold can keep wait_for_index from finishing, GPU builds on a deployment are IVF_PQ with a configured GPU count, list_indices on a deployment omits num_unindexed_rows, and the background jobs page no longer says every other job is one you start.
107 lines
4.5 KiB
Plaintext
107 lines
4.5 KiB
Plaintext
---
|
|
title: "Compaction and cleanup"
|
|
sidebarTitle: "Compaction"
|
|
description: "Merge small files, prune old versions, and fold new rows into existing indexes."
|
|
icon: broom
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Tables are compacted in the embedded library and on a deployment. In the
|
|
embedded library you call `optimize()`; a deployment compacts tables in the
|
|
background, and `optimize()` is not available there.
|
|
---
|
|
|
|
import { PyStorageRebuildVectorIndex } from '/snippets/doc_storage_examples.mdx';
|
|
|
|
Every write commits a new table version, and versions accumulate: small files
|
|
from frequent appends, superseded files from updates and deletes, and rows added
|
|
since an index was built. In the embedded library, `optimize()` addresses all
|
|
three; a deployment handles them in the background and does not support
|
|
`optimize()`.
|
|
|
|
It is modelled after `VACUUM` in PostgreSQL — routine housekeeping that a table
|
|
needs periodically rather than a repair for something broken.
|
|
|
|
## What it does {#what-it-does}
|
|
|
|
```python
|
|
table.optimize()
|
|
```
|
|
|
|
Three operations, in one call:
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Compaction** | merges small files into larger ones |
|
|
| **Prune** | removes versions older than the retention window -- see [Cleanup](/maintenance/cleanup) |
|
|
| **Index** | folds newly added rows into existing indexes -- see [Build & Repair](/indexing/reindexing) |
|
|
|
|
Compaction itself is only the first of the three: merging small fragments back
|
|
into larger ones, which is what keeps a scan from opening more files than it
|
|
needs to.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `compact`. Runs automatically as data accumulates,
|
|
separately from [`cleanup`](/maintenance/cleanup) -- the two are distinct jobs
|
|
even though `optimize()` bundles both in one OSS call.
|
|
</Note>
|
|
|
|
## Retraining indexes {#retrain}
|
|
|
|
Folding new rows into an index preserves the structure the index was trained on.
|
|
When the data distribution has shifted enough that the index's clusters are a
|
|
poor fit, recreate the index with `replace=True` to train it on the current data.
|
|
The deprecated `retrain` argument to `optimize()` does not rebuild indexes.
|
|
|
|
This example assumes a populated table whose `vector` column has an IVF_FLAT
|
|
index configured for L2 distance and four partitions:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageRebuildVectorIndex}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Use your index's column, type, distance metric, and training settings when
|
|
rebuilding it. Recreating an index is more expensive than a normal optimize,
|
|
and can help after a large ingest that changed the data distribution.
|
|
|
|
## When to run it {#when}
|
|
|
|
In the embedded library there is no background compactor: `optimize()` runs
|
|
when you call it. A deployment compacts a table automatically once enough small
|
|
fragments accumulate, unless its operator turns this off.
|
|
|
|
Frequent small appends are what make it necessary, because each one commits
|
|
files. A table written in a few large batches needs it rarely; one taking
|
|
continuous small writes needs it on a schedule.
|
|
|
|
<Note>
|
|
Compaction rewrites files, which conflicts with a concurrent
|
|
[computed-column refresh](/feature-engineering/add-computed-column) or
|
|
[view refresh](/feature-engineering/materialized-views) planned against the
|
|
version it replaces. Sequence them rather than running both at once.
|
|
</Note>
|
|
|
|
See [Cleanup](/maintenance/cleanup) for pruning old versions, and
|
|
[Build & Repair](/indexing/reindexing) for the index-folding half of
|
|
`optimize()`.
|
|
|
|
## Performance {#performance}
|
|
|
|
Two things accumulate on a long-lived table as more and more data gets added to it:
|
|
|
|
- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
|
|
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
|
|
|
|
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
|
|
|
|
```python Python icon="python"
|
|
from datetime import timedelta
|
|
table.optimize() # 7-day default retention
|
|
table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner
|
|
```
|
|
|
|
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Build & Repair](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
|