Files
lancedb/docs/web/maintenance/cleanup.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

144 lines
5.8 KiB
Plaintext

---
title: "Garbage Collection"
sidebarTitle: "Garbage Collection"
description: "Pruning old versions and the files only they referenced."
icon: broom
keywords: ["cleanup", "garbage collection", "retention", "prune", "delete_unverified", "vacuum"]
availability:
embedded: available
enterprise: available
---
Every write keeps the version before it, which is what makes
[time travel](/management/table#versioning) possible. Cleanup is what ends
that: pruning versions older than a retention window, and the files that only
those versions referenced.
<Note>
**Enterprise job type:** `cleanup`. Distinct from `compact`: compaction
merges files, cleanup removes them. `optimize()` in LanceDB Embedded runs both in one call;
Enterprise schedules them as separate jobs, and runs cleanup only when the
deployment's configuration enables it.
</Note>
## Retention {#retention}
<CodeGroup>
```python Python icon="python"
from datetime import timedelta
table.optimize(cleanup_older_than=timedelta(days=30))
```
</CodeGroup>
`cleanup_older_than` defaults to **7 days**. Versions older than that are
removed, along with the files only they referenced. In LanceDB Enterprise, the
retention period comes from its configuration instead.
The latest version is never removed, whatever you pass. `timedelta(0)` makes
every older version eligible for removal, but cleanup still keeps tagged
versions and the files branches still read, as the warning below describes.
<Warning>
Pruning is what makes [time travel](/management/table#versioning) finite: an
eligible version outside the retention window is gone. Cleanup keeps a version,
whatever its age, when it is:
- the **latest** version;
- a **tagged** version.
Cleanup also keeps the files a **branch** still reads, and the version the
branch was forked from while the branch's remaining versions read the parent's
files. That does not keep the fork for good: once the branch has replaced the
data it inherited and its own older versions are pruned, cleanup prunes the
fork version like any other. To keep a version recoverable, tag it, subject to
the policy below, or keep the window wide enough to cover it.
Under the default policy, a tag can also stop cleanup. When a tagged version is
older than the window and is not the latest, cleanup refuses to run: it raises
an error naming the tag and prunes nothing. To let it proceed, move the tag to a
version inside the window, delete the tag, or widen the window.
A tag on the latest version does not stop cleanup. But `optimize()` compacts
before it prunes, and compacting commits a new version whenever it merges
files, so a tag on what was the latest version can become an older one by the
time cleanup runs. `optimize()` then raises after compacting, and its index
update does not run.
Enterprise background cleanup keeps tagged snapshots while pruning unrelated
expired versions. If you rely on reading untagged old versions, set the window
to cover the range you
need before running this, not after.
</Warning>
## Managed blob objects {#managed-blobs}
[Managed blobs](/management/table#managed-blobs) store out-of-line payloads
under the table's `_blobs/` directory. Cleanup follows the references from
retained snapshots, including protected tags and branches, before selecting
objects for deletion. Replacing a blob or deleting a row makes its old payload
eligible only after no retained snapshot references it.
Cleanup removes whole objects. If a packed object contains several blobs, it
remains while any of them is live. Compaction preserves Managed payload objects
rather than copying their bytes, so compaction alone does not reclaim the
unused portion of a packed object.
Use table maintenance to remove Managed objects; deleting `_blobs/` files
directly can corrupt current or historical snapshots. Upgrade all clients and
maintenance workers to a version that understands Managed blobs before using
them, and reopen handles created before that upgrade.
For LanceDB Embedded tables, `optimize()` performs compaction and cleanup.
Enterprise runs these as background jobs; Python tables connected to Enterprise
do not support `optimize()`.
See [Intelligent Auto-Maintenance](/enterprise/intelligent-auto-maintenance#cleanup) for the Enterprise path.
## Disk usage does not drop immediately {#disk}
Compaction writes new, merged files before the old ones are removed, so disk
usage can rise before it falls. Space is reclaimed only when cleanup prunes the
versions that referenced the superseded files.
If you need to reclaim space aggressively in the embedded library, shorten the
retention window rather than waiting for the default:
<CodeGroup>
```python Python icon="python"
table.optimize(cleanup_older_than=timedelta(days=1))
```
</CodeGroup>
## Files from failed transactions {#unverified}
A write that fails partway can leave files that look like an operation still
in progress. Cleanup will not remove them until they are at least seven days
old, because it cannot tell them apart from a live write.
<CodeGroup>
```python Python icon="python"
table.optimize(delete_unverified=True)
```
</CodeGroup>
This option belongs to `optimize()` in the embedded library. A deployment's
cleanup jobs never set it.
<Warning>
`delete_unverified=True` removes them regardless of age. Only set it when you
can guarantee **no other process is writing to the table**. Otherwise it can
delete files belonging to a write that is genuinely in progress, and corrupt
the table.
</Warning>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Compaction" icon="broom" href="/maintenance/compaction">
Merging small files, the other half of `optimize()`.
</Card>
<Card title="Intelligent Auto-Maintenance" icon="brain" href="/enterprise/intelligent-auto-maintenance#cleanup">
How Enterprise decides to run cleanup without a call from you.
</Card>
</CardGroup>