lancedb/docs/web/maintenance/compaction.mdx

---
title: "Compaction and cleanup"
sidebarTitle: "Compaction"
description: "Merge small files, prune old versions, and fold new rows into existing indexes."
icon: broom
availability:
  oss: available
  enterprise: available
  summary: >-
    Tables are compacted in the embedded library and in LanceDB Enterprise. In the
    embedded library you call `optimize()`; a deployment compacts tables in the
    background, and `optimize()` is not available there.
---

import { PyStorageCompactionTuning, PyStorageRebuildVectorIndex } from '/snippets/doc_storage_examples.mdx';

Every write commits a new table version, and versions accumulate: small files
from frequent appends, superseded files from updates and deletes, and rows added
since an index was built. In the embedded library, `optimize()` addresses all
three; a deployment handles them in the background and does not support
`optimize()`.

Run optimization as regular table maintenance, particularly after many small
writes or a large ingest.

## Optimize a table {#what-it-does}

```python
table.optimize()
```

Three operations, in one call:

| | |
| --- | --- |
| **Compaction** | merges small files into larger ones |
| **Prune** | removes versions older than the retention window (see [Cleanup](/maintenance/cleanup)) |
| **Index** | folds newly added rows into existing indexes (see [Build & Repair](/indexing/reindexing)) |

Compaction itself is only the first of the three: merging small fragments back
into larger ones, which is what keeps a scan from opening more files than it
needs to.

<Note>
**Enterprise job type:** `compact`. Runs automatically as data accumulates,
separately from [`cleanup`](/maintenance/cleanup). The two are distinct jobs
even though `optimize()` bundles both in one OSS call.
</Note>

## Retraining indexes {#retrain}

Folding new rows into an index preserves the structure the index was trained on.
When the data distribution has shifted enough that the index's clusters are a
poor fit, recreate the index with `replace=True` to train it on the current data.
The deprecated `retrain` argument to `optimize()` does not rebuild indexes.

This example assumes a populated table whose `vector` column has an IVF_FLAT
index configured for L2 distance and four partitions:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {PyStorageRebuildVectorIndex}
    </CodeBlock>
</CodeGroup>

Use your index's column, type, distance metric, and training settings when
rebuilding it. Recreating an index is more expensive than a normal optimize,
and can help after a large ingest that changed the data distribution.

## When to run it {#when}

In the embedded library there is no background compactor: `optimize()` runs
when you call it. A deployment compacts a table automatically once enough small
fragments accumulate, unless its operator turns this off.

Frequent small appends are what make it necessary, because each one commits
files. A table written in a few large batches needs it rarely; one taking
continuous small writes needs it on a schedule.

<Note>
Compaction rewrites files, which conflicts with a concurrent
[computed-column refresh](/feature-engineering/add-computed-column) or
[view refresh](/feature-engineering/materialized-views) planned against the
version it replaces. Sequence them rather than running both at once.
</Note>

See [Cleanup](/maintenance/cleanup) for pruning old versions, and
[Build & Repair](/indexing/reindexing) for the index-folding half of
`optimize()`.

## Performance {#performance}

Two things accumulate on a long-lived table as more and more data gets added to it:

- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.

In OSS, you run them yourself via `optimize()`, which bundles both into a single call:

```python Python icon="python"
from datetime import timedelta
table.optimize()                                       # 7-day default retention
table.optimize(cleanup_older_than=timedelta(days=1))   # reclaim space sooner
```

The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates. See [Build & Repair](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed. Update or rebuild the index after large update batches.

{/*
Draft for the Python compaction_options release. Keep this section unpublished
until the first supported release is confirmed.

### Tune compaction for wide rows {#tune-wide-rows}

For an existing local Python table, derive a fragment row target from the
expected row size. This example uses rows averaging 1 KiB and a 64 KiB fragment
target; choose sizes for your workload. Smaller scan batches and one compaction
task at a time can also reduce memory use.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {PyStorageCompactionTuning}
    </CodeBlock>
</CodeGroup>

Planning remains row based. The row target bounds later passes after existing
oversized fragments have been rewritten; the first pass rewrites those
fragments whole, so peak memory can exceed the target. `max_bytes_per_file`
limits output file size rather than compaction memory.

The `max_source_rows` and `max_source_bytes` budgets admit whole planned tasks.
If the first task exceeds a budget, that run performs no compaction. See the
[CompactionOptions reference](https://lancedb.github.io/lancedb/python/python/#lancedb.table.CompactionOptions)
for all options and their limits.
*/}