mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Jack Ye
19f8f396ec
Ten merges and renames, all in one direction: fewer pages, each covering a whole subject. Table absorbs schema evolution and versioning. Vector search absorbs multi-vector. Explain absorbs explain analyze. Reranking was seven pages -- an overview and six rerankers of a few dozen lines each -- and is now one. The vector index page absorbs quantization, which was only ever a parameter of it. Indexing moves to Manage, where the other catalog objects are. The operational half stays in Operate as one Indexing page, absorbing GPU indexing -- which was filed in sophon despite importing lancedb's snippets and being mostly open-source content with one Enterprise-badged section. Performance was a single page aggregating tips for four unrelated operations, which is hard to keep current. Each section now sits with the operation it is about: ingestion tips on Insert, index tips on Indexing, compaction tips on Compaction, query tips on Search. Data Types leaves Concepts. The type reference belongs in the SQL reference, which already described the same types from the query side; Table now carries a short summary that points there. The multimodal material it had absorbed is its own page again, since storing blobs is a task rather than a type definition. Query becomes Semantic Search, Branches becomes Branch, and the Enterprise Security page becomes Compliance. Redirects cover every moved URL, and 30 files were repointed at the merged anchors.
121 lines
4.5 KiB
Plaintext
121 lines
4.5 KiB
Plaintext
---
|
|
title: "Compaction and cleanup"
|
|
sidebarTitle: "Compaction"
|
|
description: "Merge small files, prune old versions, and fold new rows into existing indexes."
|
|
icon: broom
|
|
---
|
|
|
|
Every write commits a new table version, and versions accumulate: small files
|
|
from frequent appends, superseded files from updates and deletes, and rows added
|
|
since an index was built. `optimize()` addresses all three.
|
|
|
|
It is modelled after `VACUUM` in PostgreSQL — routine housekeeping that a table
|
|
needs periodically rather than a repair for something broken.
|
|
|
|
## What it does {#what-it-does}
|
|
|
|
```python
|
|
table.optimize()
|
|
```
|
|
|
|
Three operations, in one call:
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Compaction** | merges small files into larger ones |
|
|
| **Prune** | removes versions older than the retention window |
|
|
| **Index** | folds newly added rows into existing indexes |
|
|
|
|
The third is the one people miss. An [index](/indexing) does not
|
|
automatically cover rows appended after it was built — those rows are still
|
|
searchable, but by scanning rather than through the index. Running `optimize()`
|
|
is what brings them under it.
|
|
|
|
## Retention {#retention}
|
|
|
|
```python
|
|
from datetime import timedelta
|
|
|
|
table.optimize(cleanup_older_than=timedelta(days=30))
|
|
```
|
|
|
|
`cleanup_older_than` defaults to **7 days**. Versions older than that are
|
|
removed, along with the files only they referenced.
|
|
|
|
The latest version is never removed, whatever you pass — `timedelta(0)` prunes
|
|
everything except it.
|
|
|
|
<Warning>
|
|
Pruning is what makes [time travel](/management/table#versioning) finite. A version
|
|
outside the retention window is gone, and a tag or branch pointing at it no
|
|
longer resolves.
|
|
|
|
If you rely on reading old versions, set the window to cover the range you need
|
|
before running this, not after.
|
|
</Warning>
|
|
|
|
## Files from failed transactions {#unverified}
|
|
|
|
A write that fails partway can leave files that look like an operation still in
|
|
progress. `optimize()` will not remove them until they are at least seven days
|
|
old, because it cannot tell them apart from a live write.
|
|
|
|
```python
|
|
table.optimize(delete_unverified=True)
|
|
```
|
|
|
|
<Warning>
|
|
`delete_unverified=True` removes them regardless of age. Only set it when you
|
|
can guarantee **no other process is writing to the table** — otherwise it can
|
|
delete files belonging to a write that is genuinely in progress, and corrupt the
|
|
table.
|
|
</Warning>
|
|
|
|
## Retraining indexes {#retrain}
|
|
|
|
```python
|
|
table.optimize(retrain=True)
|
|
```
|
|
|
|
Folding new rows into an index preserves the structure the index was trained on.
|
|
When the data has shifted enough that the structure itself is a poor fit — a
|
|
vector index whose clusters no longer reflect the distribution — `retrain=True`
|
|
rebuilds it rather than extending it.
|
|
|
|
More expensive than a normal optimize, and worth it after a large ingest that
|
|
changed the shape of the data rather than just its volume.
|
|
|
|
## When to run it {#when}
|
|
|
|
There is no background compactor: `optimize()` runs when you call it.
|
|
|
|
Frequent small appends are what make it necessary, because each one commits
|
|
files. A table written in a few large batches needs it rarely; one taking
|
|
continuous small writes needs it on a schedule.
|
|
|
|
<Note>
|
|
Compaction rewrites files, which conflicts with a concurrent
|
|
[computed-column refresh](/feature-engineering/add-computed-column) or
|
|
[view refresh](/feature-engineering/materialized-views) planned against the
|
|
version it replaces. Sequence them rather than running both at once.
|
|
</Note>
|
|
|
|
## Performance {#performance}
|
|
|
|
Two things accumulate on a long-lived table as more and more data gets added to it:
|
|
|
|
- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
|
|
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
|
|
|
|
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
|
|
|
|
```python Python icon="python"
|
|
from datetime import timedelta
|
|
table.optimize() # 7-day default retention
|
|
table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner
|
|
```
|
|
|
|
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Reindexing](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
|
|
|
|
LanceDB Enterprise handles both compaction and cleanup automatically.
|