mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Brendan Clement
058dca70b0
The cleanup and guarantees pages said a tag or branch pointing at a pruned version stops resolving, while the table page said tagged versions are exempt and quietly kept. The pinned Lance (ae2903729f02, the commit in Cargo.lock and in Sophon's Lance submodule) keeps a tagged version whatever its age, but with its default policy refuses to clean up while a tag points at a version older than the window, raising an error that names the tag. optimize() passes that default, so it raises after compacting, prunes nothing and skips its index update until the tag is deleted. Sophon's cleanup job uses the same default. Branches are retained separately: cleaning the parent keeps the version a branch was forked from and the files the branch reads, while the parent's other old versions are pruned. The pages now say both, checked locally against the baseline SDK.
107 lines
3.8 KiB
Plaintext
107 lines
3.8 KiB
Plaintext
---
|
|
title: "Cleanup"
|
|
sidebarTitle: "Cleanup"
|
|
description: "Pruning old versions and the files only they referenced."
|
|
icon: broom
|
|
keywords: ["cleanup", "retention", "prune", "delete_unverified", "vacuum"]
|
|
availability:
|
|
oss: available
|
|
enterprise: varies
|
|
summary: >-
|
|
In the embedded library, `optimize()` removes versions older than a
|
|
retention window, 7 days by default. A deployment removes old versions in
|
|
the background only when its configuration enables cleanup, and that
|
|
configuration sets the retention period; `optimize()` is not available
|
|
there.
|
|
---
|
|
|
|
Every write keeps the version before it, which is what makes
|
|
[time travel](/management/table#versioning) possible. Cleanup is what ends
|
|
that -- pruning versions older than a retention window, and the files that only
|
|
those versions referenced.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `cleanup`. Distinct from `compact` -- compaction
|
|
merges files, cleanup removes them. `optimize()` in OSS runs both in one call;
|
|
Enterprise schedules them as separate jobs, and runs cleanup only when the
|
|
deployment's configuration enables it.
|
|
</Note>
|
|
|
|
## Retention {#retention}
|
|
|
|
```python
|
|
from datetime import timedelta
|
|
|
|
table.optimize(cleanup_older_than=timedelta(days=30))
|
|
```
|
|
|
|
`cleanup_older_than` defaults to **7 days**. Versions older than that are
|
|
removed, along with the files only they referenced. On a deployment, the
|
|
retention period comes from its configuration instead.
|
|
|
|
The latest version is never removed, whatever you pass -- `timedelta(0)` prunes
|
|
everything except it.
|
|
|
|
<Warning>
|
|
Pruning is what makes [time travel](/management/table#versioning) finite: a
|
|
version outside the retention window is gone, unless something still refers to
|
|
it.
|
|
|
|
- A **tag** keeps its version, whatever its age. While a tag points at a
|
|
version older than the window, cleanup refuses to run and raises an error
|
|
naming the tag; `optimize()` raises it after compacting, so nothing is
|
|
pruned and its index update does not run. Delete the tag to let cleanup
|
|
proceed.
|
|
- A **branch** keeps the version it was forked from, and the files it reads,
|
|
when its parent is cleaned up; the parent's other old versions are pruned as
|
|
usual.
|
|
|
|
A deployment's background cleanup follows the same rules. If you rely on
|
|
reading old versions, set the window to cover the range you need before
|
|
running this, not after.
|
|
</Warning>
|
|
|
|
## Disk usage does not drop immediately {#disk}
|
|
|
|
Compaction writes new, merged files before the old ones are removed, so disk
|
|
usage can rise before it falls. Space is reclaimed only when cleanup prunes the
|
|
versions that referenced the superseded files.
|
|
|
|
If you need to reclaim space aggressively in the embedded library, shorten the
|
|
retention window rather than waiting for the default:
|
|
|
|
```python
|
|
table.optimize(cleanup_older_than=timedelta(days=1))
|
|
```
|
|
|
|
## Files from failed transactions {#unverified}
|
|
|
|
A write that fails partway can leave files that look like an operation still
|
|
in progress. Cleanup will not remove them until they are at least seven days
|
|
old, because it cannot tell them apart from a live write.
|
|
|
|
```python
|
|
table.optimize(delete_unverified=True)
|
|
```
|
|
|
|
This option belongs to `optimize()` in the embedded library. A deployment's
|
|
cleanup jobs never set it.
|
|
|
|
<Warning>
|
|
`delete_unverified=True` removes them regardless of age. Only set it when you
|
|
can guarantee **no other process is writing to the table** -- otherwise it can
|
|
delete files belonging to a write that is genuinely in progress, and corrupt
|
|
the table.
|
|
</Warning>
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Compaction" icon="broom" href="/maintenance/compaction">
|
|
Merging small files -- the other half of `optimize()`.
|
|
</Card>
|
|
<Card title="Background jobs" icon="gears" href="/maintenance/background-jobs">
|
|
The Enterprise jobs that run without a call from you.
|
|
</Card>
|
|
</CardGroup>
|