lancedb/docs/web/maintenance/compaction.mdx

---
title: "Compaction and cleanup"
sidebarTitle: "Compaction"
description: "Merge small files, prune old versions, and fold new rows into existing indexes."
icon: broom
availability:
  embedded: available
  enterprise: available
---

import { PyStorageCompactionTuning, PyStorageRebuildVectorIndex, RsStorageCompactionTuning } from '/snippets/doc_storage_examples.mdx';

Every write commits a new table version, and versions accumulate: small files
from frequent appends, superseded files from updates and deletes, and rows added
since an index was built. In the embedded library, `optimize()` addresses all
three; a deployment handles them in the background and does not support
`optimize()`.

Run optimization as regular table maintenance, particularly after many small
writes or a large ingest.

{/*
Draft for the LanceDB Embedded compaction SQ partition check release. Keep this section
unpublished until that behavior ships.

### Recover from an oversized SQ partition {#oversized-sq-partition}

Compacting a table with an IVF scalar-quantized (SQ) vector index can stop
before rewriting files if an affected partition is too large to remap. The
error identifies the index, segment, partition size, and vector dimension
instead of letting the remap fail at Arrow's fixed-size-list `u32`
child-array limit.

Rebuild the affected vector index with enough IVF partitions to keep each
partition at or below `floor(u32::MAX / dimension)` vectors, then retry the
optimize. Rebuilding with the default IVF partitioning can also choose enough
partitions.
*/}

## Optimize a table {#what-it-does}

<CodeGroup>
```python Python icon="python"
table.optimize()
```
</CodeGroup>

Three operations, in one call:

| | |
| --- | --- |
| **Compaction** | merges small files into larger ones |
| **Prune** | removes versions older than the retention window (see [Garbage Collection](/maintenance/cleanup)) |
| **Index** | folds newly added rows into existing indexes (see [Incremental Indexing](/indexing/incremental-indexing)) |

Compaction itself is only the first of the three: merging small fragments back
into larger ones, which is what keeps a scan from opening more files than it
needs to.

<Note>
**Enterprise job type:** `compact`. Runs automatically as data accumulates,
separately from [`cleanup`](/maintenance/cleanup). The two are distinct jobs
even though `optimize()` bundles both in one LanceDB Embedded call.
</Note>

## Retraining indexes {#retrain}

Folding new rows into an index preserves the structure the index was trained on.
When the data distribution has shifted enough that the index's clusters are a
poor fit, recreate the index with `replace=True` to train it on the current data.
The deprecated `retrain` argument to `optimize()` does not rebuild indexes.

This example assumes a populated table whose `vector` column has an IVF_FLAT
index configured for L2 distance and four partitions:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {PyStorageRebuildVectorIndex}
    </CodeBlock>
</CodeGroup>

Use your index's column, type, distance metric, and training settings when
rebuilding it. Recreating an index is more expensive than a normal optimize,
and can help after a large ingest that changed the data distribution.

## When to run it {#when}

In the embedded library there is no background compactor: `optimize()` runs
when you call it. A deployment compacts a table automatically once enough small
fragments accumulate, unless its operator turns this off.

Frequent small appends are what make it necessary, because each one commits
files. A table written in a few large batches needs it rarely; one taking
continuous small writes needs it on a schedule.

Opening a table in LanceDB Embedded logs a warning at most once per table per
process when at least 100 fragments have fewer than 100,000 rows each. Set
`LANCEDB_LOG=warn` to surface this log if your logger hides warnings by default.

<Note>
Compaction rewrites files, which conflicts with a concurrent
[computed-column refresh](/feature-engineering/add-computed-column) or
[view refresh](/feature-engineering/refresh-materialized-view) planned against the
version it replaces. Sequence them rather than running both at once.
</Note>

See [Garbage Collection](/maintenance/cleanup) for pruning old versions, and
[Incremental Indexing](/indexing/incremental-indexing) for the index-folding half of
`optimize()`.

## Performance {#performance}

Two things accumulate on a long-lived table as more and more data gets added to it:

- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.

In LanceDB Embedded, you run them yourself via `optimize()`, which bundles both into a single call:

<CodeGroup>
```python Python icon="python"
from datetime import timedelta
table.optimize()                                       # 7-day default retention
table.optimize(cleanup_older_than=timedelta(days=1))   # reclaim space sooner
```
</CodeGroup>

The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates. See [Incremental Indexing](/indexing/incremental-indexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed. Update or rebuild the index after large update batches.

{/*
Draft for the Python compaction_options release. Keep this section unpublished
until the first supported release is confirmed.

### Tune compaction for wide rows {#tune-wide-rows}

For an existing LanceDB Embedded table, derive a fragment row target from the expected
row size. These examples use rows averaging 1 KiB and a 64 KiB fragment
target; choose sizes for your workload. Smaller scan batches and one compaction
task at a time can also reduce memory use.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {PyStorageCompactionTuning}
    </CodeBlock>
    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsStorageCompactionTuning}
    </CodeBlock>
</CodeGroup>

The Rust example runs compaction only. Run pruning and index optimization
separately when needed; Python's `optimize()` includes those steps.

TypeScript's `optimize()` does not currently expose the fragment row target,
scan batch size, or compaction thread count. These tuning examples apply to
Python and Rust tables in LanceDB Embedded.

Planning remains row based. The row target bounds later passes after existing
oversized fragments have been rewritten; the first pass rewrites those
fragments whole, so peak memory can exceed the target. `max_bytes_per_file`
limits output file size rather than compaction memory.

The `max_source_rows` and `max_source_bytes` budgets admit whole planned tasks.
If the first task exceeds a budget, that run performs no compaction. See the
[CompactionOptions reference](https://lancedb.github.io/lancedb/python/python/#lancedb.table.CompactionOptions)
for all options and their limits.
*/}