mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-18 08:34:00 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
144 lines
5.8 KiB
Plaintext
144 lines
5.8 KiB
Plaintext
---
|
|
title: "Garbage Collection"
|
|
sidebarTitle: "Garbage Collection"
|
|
description: "Pruning old versions and the files only they referenced."
|
|
icon: broom
|
|
keywords: ["cleanup", "garbage collection", "retention", "prune", "delete_unverified", "vacuum"]
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
Every write keeps the version before it, which is what makes
|
|
[time travel](/management/table#versioning) possible. Cleanup is what ends
|
|
that: pruning versions older than a retention window, and the files that only
|
|
those versions referenced.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `cleanup`. Distinct from `compact`: compaction
|
|
merges files, cleanup removes them. `optimize()` in LanceDB Embedded runs both in one call;
|
|
Enterprise schedules them as separate jobs, and runs cleanup only when the
|
|
deployment's configuration enables it.
|
|
</Note>
|
|
|
|
## Retention {#retention}
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from datetime import timedelta
|
|
|
|
table.optimize(cleanup_older_than=timedelta(days=30))
|
|
```
|
|
</CodeGroup>
|
|
|
|
`cleanup_older_than` defaults to **7 days**. Versions older than that are
|
|
removed, along with the files only they referenced. In LanceDB Enterprise, the
|
|
retention period comes from its configuration instead.
|
|
|
|
The latest version is never removed, whatever you pass. `timedelta(0)` makes
|
|
every older version eligible for removal, but cleanup still keeps tagged
|
|
versions and the files branches still read, as the warning below describes.
|
|
|
|
<Warning>
|
|
Pruning is what makes [time travel](/management/table#versioning) finite: an
|
|
eligible version outside the retention window is gone. Cleanup keeps a version,
|
|
whatever its age, when it is:
|
|
|
|
- the **latest** version;
|
|
- a **tagged** version.
|
|
|
|
Cleanup also keeps the files a **branch** still reads, and the version the
|
|
branch was forked from while the branch's remaining versions read the parent's
|
|
files. That does not keep the fork for good: once the branch has replaced the
|
|
data it inherited and its own older versions are pruned, cleanup prunes the
|
|
fork version like any other. To keep a version recoverable, tag it, subject to
|
|
the policy below, or keep the window wide enough to cover it.
|
|
|
|
Under the default policy, a tag can also stop cleanup. When a tagged version is
|
|
older than the window and is not the latest, cleanup refuses to run: it raises
|
|
an error naming the tag and prunes nothing. To let it proceed, move the tag to a
|
|
version inside the window, delete the tag, or widen the window.
|
|
|
|
A tag on the latest version does not stop cleanup. But `optimize()` compacts
|
|
before it prunes, and compacting commits a new version whenever it merges
|
|
files, so a tag on what was the latest version can become an older one by the
|
|
time cleanup runs. `optimize()` then raises after compacting, and its index
|
|
update does not run.
|
|
|
|
Enterprise background cleanup keeps tagged snapshots while pruning unrelated
|
|
expired versions. If you rely on reading untagged old versions, set the window
|
|
to cover the range you
|
|
need before running this, not after.
|
|
</Warning>
|
|
|
|
## Managed blob objects {#managed-blobs}
|
|
|
|
[Managed blobs](/management/table#managed-blobs) store out-of-line payloads
|
|
under the table's `_blobs/` directory. Cleanup follows the references from
|
|
retained snapshots, including protected tags and branches, before selecting
|
|
objects for deletion. Replacing a blob or deleting a row makes its old payload
|
|
eligible only after no retained snapshot references it.
|
|
|
|
Cleanup removes whole objects. If a packed object contains several blobs, it
|
|
remains while any of them is live. Compaction preserves Managed payload objects
|
|
rather than copying their bytes, so compaction alone does not reclaim the
|
|
unused portion of a packed object.
|
|
|
|
Use table maintenance to remove Managed objects; deleting `_blobs/` files
|
|
directly can corrupt current or historical snapshots. Upgrade all clients and
|
|
maintenance workers to a version that understands Managed blobs before using
|
|
them, and reopen handles created before that upgrade.
|
|
|
|
For LanceDB Embedded tables, `optimize()` performs compaction and cleanup.
|
|
Enterprise runs these as background jobs; Python tables connected to Enterprise
|
|
do not support `optimize()`.
|
|
See [Intelligent Auto-Maintenance](/enterprise/intelligent-auto-maintenance#cleanup) for the Enterprise path.
|
|
|
|
## Disk usage does not drop immediately {#disk}
|
|
|
|
Compaction writes new, merged files before the old ones are removed, so disk
|
|
usage can rise before it falls. Space is reclaimed only when cleanup prunes the
|
|
versions that referenced the superseded files.
|
|
|
|
If you need to reclaim space aggressively in the embedded library, shorten the
|
|
retention window rather than waiting for the default:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table.optimize(cleanup_older_than=timedelta(days=1))
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Files from failed transactions {#unverified}
|
|
|
|
A write that fails partway can leave files that look like an operation still
|
|
in progress. Cleanup will not remove them until they are at least seven days
|
|
old, because it cannot tell them apart from a live write.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table.optimize(delete_unverified=True)
|
|
```
|
|
</CodeGroup>
|
|
|
|
This option belongs to `optimize()` in the embedded library. A deployment's
|
|
cleanup jobs never set it.
|
|
|
|
<Warning>
|
|
`delete_unverified=True` removes them regardless of age. Only set it when you
|
|
can guarantee **no other process is writing to the table**. Otherwise it can
|
|
delete files belonging to a write that is genuinely in progress, and corrupt
|
|
the table.
|
|
</Warning>
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Compaction" icon="broom" href="/maintenance/compaction">
|
|
Merging small files, the other half of `optimize()`.
|
|
</Card>
|
|
<Card title="Intelligent Auto-Maintenance" icon="brain" href="/enterprise/intelligent-auto-maintenance#cleanup">
|
|
How Enterprise decides to run cleanup without a call from you.
|
|
</Card>
|
|
</CardGroup>
|