mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 15:46:55 +00:00
Labels early access features and hides features we aren't documenting yet. **Early access label:** Functions (4 pages, plus the GPU refresh, checkpointing and adaptive execution pages), Secrets (3 pages), Tiered Index Cache, Blob V2, and merge insert, which is labelled for Enterprise only. Two sidebar titles are shorter so the tag fits. **Hidden:** branching, JSON columns and geospatial. - The JSON functions page and the agent branch-experiments tutorial go in `.mintignore`. - `tables/branching` stays reachable but out of the nav, as #4859 left it, because SDK error messages link to it. - Other pages' mentions of these features become draft comments, so they're easy to restore. **Secrets page:** drops references to manual KMS setup, which the control plane handles. It keeps the encryption-at-rest facts, states the Azure value-size limit inline, and points a 501 error to support.
150 lines
5.9 KiB
Plaintext
150 lines
5.9 KiB
Plaintext
---
|
|
title: "Garbage Collection"
|
|
sidebarTitle: "Garbage Collection"
|
|
description: "Pruning old versions and the files only they referenced."
|
|
icon: broom
|
|
keywords: ["cleanup", "garbage collection", "retention", "prune", "delete_unverified", "vacuum"]
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
Every write keeps the version before it, which is what makes
|
|
[time travel](/management/table#versioning) possible. Cleanup is what ends
|
|
that: pruning versions older than a retention window, and the files that only
|
|
those versions referenced.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `cleanup`. Distinct from `compact`: compaction
|
|
merges files, cleanup removes them. `optimize()` in LanceDB Embedded runs both in one call;
|
|
Enterprise schedules them as separate jobs, and runs cleanup only when the
|
|
deployment's configuration enables it.
|
|
</Note>
|
|
|
|
## Retention {#retention}
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from datetime import timedelta
|
|
|
|
table.optimize(cleanup_older_than=timedelta(days=30))
|
|
```
|
|
</CodeGroup>
|
|
|
|
`cleanup_older_than` defaults to **7 days**. Versions older than that are
|
|
removed, along with the files only they referenced. In LanceDB Enterprise, the
|
|
retention period comes from its configuration instead.
|
|
|
|
The latest version is never removed, whatever you pass. `timedelta(0)` makes
|
|
every older version eligible for removal, but cleanup still keeps tagged
|
|
versions{/* Draft until branching reaches early access: restore " and the files branches still read" */}, as the warning below describes.
|
|
|
|
<Warning>
|
|
Pruning is what makes [time travel](/management/table#versioning) finite: an
|
|
eligible version outside the retention window is gone. Cleanup keeps a version,
|
|
whatever its age, when it is:
|
|
|
|
- the **latest** version;
|
|
- a **tagged** version.
|
|
|
|
{/*
|
|
Draft until branching reaches early access.
|
|
|
|
Cleanup also keeps the files a **branch** still reads, and the version the
|
|
branch was forked from while the branch's remaining versions read the parent's
|
|
files. That does not keep the fork for good: once the branch has replaced the
|
|
data it inherited and its own older versions are pruned, cleanup prunes the
|
|
fork version like any other.
|
|
*/}
|
|
|
|
To keep a version recoverable, tag it, subject to the policy below, or keep
|
|
the window wide enough to cover it.
|
|
|
|
Under the default policy, a tag can also stop cleanup. When a tagged version is
|
|
older than the window and is not the latest, cleanup refuses to run: it raises
|
|
an error naming the tag and prunes nothing. To let it proceed, move the tag to a
|
|
version inside the window, delete the tag, or widen the window.
|
|
|
|
A tag on the latest version does not stop cleanup. But `optimize()` compacts
|
|
before it prunes, and compacting commits a new version whenever it merges
|
|
files, so a tag on what was the latest version can become an older one by the
|
|
time cleanup runs. `optimize()` then raises after compacting, and its index
|
|
update does not run.
|
|
|
|
Enterprise background cleanup keeps tagged snapshots while pruning unrelated
|
|
expired versions. If you rely on reading untagged old versions, set the window
|
|
to cover the range you
|
|
need before running this, not after.
|
|
</Warning>
|
|
|
|
## Managed blob objects {#managed-blobs}
|
|
|
|
[Managed blobs](/management/table#managed-blobs) store out-of-line payloads
|
|
under the table's `_blobs/` directory. Cleanup follows the references from
|
|
retained snapshots, including protected tags{/* Draft until branching reaches early access: restore " and branches" */}, before selecting
|
|
objects for deletion. Replacing a blob or deleting a row makes its old payload
|
|
eligible only after no retained snapshot references it.
|
|
|
|
Cleanup removes whole objects. If a packed object contains several blobs, it
|
|
remains while any of them is live. Compaction preserves Managed payload objects
|
|
rather than copying their bytes, so compaction alone does not reclaim the
|
|
unused portion of a packed object.
|
|
|
|
Use table maintenance to remove Managed objects; deleting `_blobs/` files
|
|
directly can corrupt current or historical snapshots. Upgrade all clients and
|
|
maintenance workers to a version that understands Managed blobs before using
|
|
them, and reopen handles created before that upgrade.
|
|
|
|
For LanceDB Embedded tables, `optimize()` performs compaction and cleanup.
|
|
Enterprise runs these as background jobs; Python tables connected to Enterprise
|
|
do not support `optimize()`.
|
|
See [Intelligent Auto-Maintenance](/enterprise/intelligent-auto-maintenance#cleanup) for the Enterprise path.
|
|
|
|
## Disk usage does not drop immediately {#disk}
|
|
|
|
Compaction writes new, merged files before the old ones are removed, so disk
|
|
usage can rise before it falls. Space is reclaimed only when cleanup prunes the
|
|
versions that referenced the superseded files.
|
|
|
|
If you need to reclaim space aggressively in the embedded library, shorten the
|
|
retention window rather than waiting for the default:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table.optimize(cleanup_older_than=timedelta(days=1))
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Files from failed transactions {#unverified}
|
|
|
|
A write that fails partway can leave files that look like an operation still
|
|
in progress. Cleanup will not remove them until they are at least seven days
|
|
old, because it cannot tell them apart from a live write.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table.optimize(delete_unverified=True)
|
|
```
|
|
</CodeGroup>
|
|
|
|
This option belongs to `optimize()` in the embedded library. A deployment's
|
|
cleanup jobs never set it.
|
|
|
|
<Warning>
|
|
`delete_unverified=True` removes them regardless of age. Only set it when you
|
|
can guarantee **no other process is writing to the table**. Otherwise it can
|
|
delete files belonging to a write that is genuinely in progress, and corrupt
|
|
the table.
|
|
</Warning>
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Compaction" icon="broom" href="/maintenance/compaction">
|
|
Merging small files, the other half of `optimize()`.
|
|
</Card>
|
|
<Card title="Intelligent Auto-Maintenance" icon="brain" href="/enterprise/intelligent-auto-maintenance#cleanup">
|
|
How Enterprise decides to run cleanup without a call from you.
|
|
</Card>
|
|
</CardGroup>
|