mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Jack Ye
cb45b67c4c
Indexing loses its wrapper group -- Overview, Vector Index, FTS Index, Scalar Index and Build & Repair sit directly in Maintenance now, followed by Compaction, Cleanup and Background Jobs at the same level. Cleanup is new, split out of Compaction: pruning old versions is a distinct Enterprise job (cleanup) from merging files (compact), even though optimize() in OSS bundles both into one call. The retention, disk-usage and failed-transaction material that used to live under Compaction's umbrella moves with it. Background Jobs is new and covers what nothing else does: the job types a deployment runs on its own -- WAL/SPFresh maintenance, cache prewarming, registry archiving and metering export, and the object-teardown job that follows a namespace or view drop. Every entry is grounded against the job-coordinator's public job_type constants, not the internal two-letter wire codes used for registry storage. Five existing pages (Build & Repair, Table, Function, Column Refresh, Materialized View Refresh) gain a one-line job-type note rather than duplicating a page for job types they already document. Storage folds into Deployment: storage/index (choosing a backend) and storage/configuration (configuring one) merge into a single Storage page -- they never overlapped, one is the decision and one is the how-to -- and move under Deployment along with Monitoring. Cloud Providers loses its nesting; its two pages plus Azure sit flat, and the former Overview is retitled Deployment Models so it doesn't collide with the group's own Overview entry. Redirects cover every moved URL.
88 lines
3.6 KiB
Plaintext
88 lines
3.6 KiB
Plaintext
---
|
|
title: "Compaction and cleanup"
|
|
sidebarTitle: "Compaction"
|
|
description: "Merge small files, prune old versions, and fold new rows into existing indexes."
|
|
icon: broom
|
|
---
|
|
|
|
Every write commits a new table version, and versions accumulate: small files
|
|
from frequent appends, superseded files from updates and deletes, and rows added
|
|
since an index was built. `optimize()` addresses all three.
|
|
|
|
It is modelled after `VACUUM` in PostgreSQL — routine housekeeping that a table
|
|
needs periodically rather than a repair for something broken.
|
|
|
|
## What it does {#what-it-does}
|
|
|
|
```python
|
|
table.optimize()
|
|
```
|
|
|
|
Three operations, in one call:
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Compaction** | merges small files into larger ones |
|
|
| **Prune** | removes versions older than the retention window -- see [Cleanup](/maintenance/cleanup) |
|
|
| **Index** | folds newly added rows into existing indexes -- see [Build & Repair](/indexing/reindexing) |
|
|
|
|
Compaction itself is only the first of the three: merging small fragments back
|
|
into larger ones, which is what keeps a scan from opening more files than it
|
|
needs to.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `compact`. Runs automatically as data accumulates,
|
|
separately from [`cleanup`](/maintenance/cleanup) -- the two are distinct jobs
|
|
even though `optimize()` bundles both in one OSS call.
|
|
</Note>
|
|
|
|
## Retraining indexes {#retrain}
|
|
|
|
```python
|
|
table.optimize(retrain=True)
|
|
```
|
|
|
|
Folding new rows into an index preserves the structure the index was trained on.
|
|
When the data has shifted enough that the structure itself is a poor fit — a
|
|
vector index whose clusters no longer reflect the distribution — `retrain=True`
|
|
rebuilds it rather than extending it.
|
|
|
|
More expensive than a normal optimize, and worth it after a large ingest that
|
|
changed the shape of the data rather than just its volume.
|
|
|
|
## When to run it {#when}
|
|
|
|
There is no background compactor: `optimize()` runs when you call it.
|
|
|
|
Frequent small appends are what make it necessary, because each one commits
|
|
files. A table written in a few large batches needs it rarely; one taking
|
|
continuous small writes needs it on a schedule.
|
|
|
|
<Note>
|
|
Compaction rewrites files, which conflicts with a concurrent
|
|
[computed-column refresh](/feature-engineering/add-computed-column) or
|
|
[view refresh](/feature-engineering/materialized-views) planned against the
|
|
version it replaces. Sequence them rather than running both at once.
|
|
</Note>
|
|
|
|
See [Cleanup](/maintenance/cleanup) for pruning old versions, and
|
|
[Build & Repair](/indexing/reindexing) for the index-folding half of
|
|
`optimize()`.
|
|
|
|
## Performance {#performance}
|
|
|
|
Two things accumulate on a long-lived table as more and more data gets added to it:
|
|
|
|
- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
|
|
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
|
|
|
|
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
|
|
|
|
```python Python icon="python"
|
|
from datetime import timedelta
|
|
table.optimize() # 7-day default retention
|
|
table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner
|
|
```
|
|
|
|
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Build & Repair](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
|