mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-17 09:12:28 +00:00
Brendan Clement
06aa526ae1
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`.
118 lines
4.6 KiB
Plaintext
118 lines
4.6 KiB
Plaintext
---
|
|
title: "Cleanup"
|
|
sidebarTitle: "Cleanup"
|
|
description: "Pruning old versions and the files only they referenced."
|
|
icon: broom
|
|
keywords: ["cleanup", "retention", "prune", "delete_unverified", "vacuum"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Both offerings support cleanup. Embedded `optimize()` uses a retention window of 7 days by default; Enterprise uses configured background cleanup and retention, and does not support `optimize()`.
|
|
---
|
|
|
|
Every write keeps the version before it, which is what makes
|
|
[time travel](/management/table#versioning) possible. Cleanup is what ends
|
|
that -- pruning versions older than a retention window, and the files that only
|
|
those versions referenced.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `cleanup`. Distinct from `compact` -- compaction
|
|
merges files, cleanup removes them. `optimize()` in OSS runs both in one call;
|
|
Enterprise schedules them as separate jobs, and runs cleanup only when the
|
|
deployment's configuration enables it.
|
|
</Note>
|
|
|
|
## Retention {#retention}
|
|
|
|
```python
|
|
from datetime import timedelta
|
|
|
|
table.optimize(cleanup_older_than=timedelta(days=30))
|
|
```
|
|
|
|
`cleanup_older_than` defaults to **7 days**. Versions older than that are
|
|
removed, along with the files only they referenced. In LanceDB Enterprise, the
|
|
retention period comes from its configuration instead.
|
|
|
|
The latest version is never removed, whatever you pass. `timedelta(0)` makes
|
|
every older version eligible for removal, but cleanup still keeps tagged
|
|
versions and the files branches still read, as the warning below describes.
|
|
|
|
<Warning>
|
|
Pruning is what makes [time travel](/management/table#versioning) finite: an
|
|
eligible version outside the retention window is gone. Cleanup keeps a version,
|
|
whatever its age, when it is:
|
|
|
|
- the **latest** version;
|
|
- a **tagged** version.
|
|
|
|
Cleanup also keeps the files a **branch** still reads, and the version the
|
|
branch was forked from while the branch's remaining versions read the parent's
|
|
files. That does not keep the fork for good: once the branch has replaced the
|
|
data it inherited and its own older versions are pruned, cleanup prunes the
|
|
fork version like any other. To keep a version recoverable, tag it, subject to
|
|
the policy below, or keep the window wide enough to cover it.
|
|
|
|
Under the default policy, a tag can also stop cleanup. When a tagged version is
|
|
older than the window and is not the latest, cleanup refuses to run: it raises
|
|
an error naming the tag and prunes nothing. To let it proceed, move the tag to a
|
|
version inside the window, delete the tag, or widen the window.
|
|
|
|
A tag on the latest version does not stop cleanup. But `optimize()` compacts
|
|
before it prunes, and compacting commits a new version whenever it merges
|
|
files, so a tag on what was the latest version can become an older one by the
|
|
time cleanup runs. `optimize()` then raises after compacting, and its index
|
|
update does not run.
|
|
|
|
A deployment's background cleanup follows the same rules. Its own background
|
|
jobs, such as compaction and index updates, also commit new versions, so a tag
|
|
on what is now the latest version can become an older one without a call from
|
|
you. If you rely on reading old versions, set the window to cover the range you
|
|
need before running this, not after.
|
|
</Warning>
|
|
|
|
## Disk usage does not drop immediately {#disk}
|
|
|
|
Compaction writes new, merged files before the old ones are removed, so disk
|
|
usage can rise before it falls. Space is reclaimed only when cleanup prunes the
|
|
versions that referenced the superseded files.
|
|
|
|
If you need to reclaim space aggressively in the embedded library, shorten the
|
|
retention window rather than waiting for the default:
|
|
|
|
```python
|
|
table.optimize(cleanup_older_than=timedelta(days=1))
|
|
```
|
|
|
|
## Files from failed transactions {#unverified}
|
|
|
|
A write that fails partway can leave files that look like an operation still
|
|
in progress. Cleanup will not remove them until they are at least seven days
|
|
old, because it cannot tell them apart from a live write.
|
|
|
|
```python
|
|
table.optimize(delete_unverified=True)
|
|
```
|
|
|
|
This option belongs to `optimize()` in the embedded library. A deployment's
|
|
cleanup jobs never set it.
|
|
|
|
<Warning>
|
|
`delete_unverified=True` removes them regardless of age. Only set it when you
|
|
can guarantee **no other process is writing to the table** -- otherwise it can
|
|
delete files belonging to a write that is genuinely in progress, and corrupt
|
|
the table.
|
|
</Warning>
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Compaction" icon="broom" href="/maintenance/compaction">
|
|
Merging small files -- the other half of `optimize()`.
|
|
</Card>
|
|
<Card title="Background jobs" icon="gears" href="/maintenance/background-jobs">
|
|
The Enterprise jobs that run without a call from you.
|
|
</Card>
|
|
</CardGroup>
|