mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-20 09:33:58 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
149 lines
7.8 KiB
Plaintext
149 lines
7.8 KiB
Plaintext
---
|
|
title: "Real Time Engine compaction"
|
|
sidebarTitle: "RTE Compaction"
|
|
description: "How Enterprise automatically folds a Real Time Engine shard's SSTables into the base table and catches its indexes up."
|
|
icon: "compress"
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
[Distributed compaction](/enterprise/distributed-compaction) merges small
|
|
base-table fragments and reclaims deleted rows -- the same operation
|
|
[Compaction](/maintenance/compaction) describes for LanceDB Embedded. A
|
|
table with the [real time engine](/tables/realtime-engine) enabled has a
|
|
second, unrelated compaction: folding a shard's accumulated SSTables into
|
|
the base table and catching the base table's indexes up over the rows that
|
|
just landed. This produces two job types, `memwal_compaction` and
|
|
`memwal_catchup_<type>`, already distinct from ordinary `compact` in
|
|
`SHOW JOBS` and `list_jobs`. This page covers how Enterprise decides when
|
|
to run them, not just what they do.
|
|
|
|
## What triggers a compaction pass {#trigger}
|
|
|
|
A per-pod loop, ticking every `WAL_COMPACTION_INTERVAL_SECS` (5s by
|
|
default), walks every table this pod owns and reads each shard's manifest.
|
|
It dispatches a compaction job for a shard once any threshold crosses:
|
|
|
|
| Trigger | Default | Checks |
|
|
| --- | --- | --- |
|
|
| `WAL_COMPACT_TRIGGER_COUNT` | 4 | The shard's SSTable count |
|
|
| `WAL_COMPACT_TRIGGER_BYTES` | 1 GiB | The shard's total flushed SSTable bytes |
|
|
| Oldest generation's age | 7 days | How long the head SSTable generation has sat, whatever the file count -- a file that will never gather enough siblings to cross the count trigger still gets compacted eventually |
|
|
|
|
Whichever crosses first wins. A shard under write pressure does not wait
|
|
for the next tick either: the same thresholds are re-checked on every
|
|
backpressure relief pass, so a hot shard that already qualifies compacts
|
|
sooner than `WAL_COMPACTION_INTERVAL_SECS` would otherwise allow. Setting
|
|
`WAL_COMPACTION_INTERVAL_SECS` to `0` disables the periodic walk; an
|
|
operator can still force a pass with `POST /v1/compact`, and
|
|
`Table::checkpoint_rte` drives the same flush-then-compact cycle from the
|
|
client when you need a table to converge on demand rather than wait for
|
|
the trigger. `checkpoint_rte` has no liveness bound of its own -- it is
|
|
idempotent and abandonable at any point, but the caller owns the deadline
|
|
for how long to wait on it.
|
|
|
|
This trigger decision is internal to the WAL pod -- it is not the general
|
|
job scheduler deciding to run `memwal_compaction`, the pod decides to
|
|
enqueue one. Once enqueued, the work is still tracked as an ordinary job
|
|
(`memwal_compaction`, then `memwal_catchup_<type>` once it lands), so
|
|
`SHOW JOBS` and `list_jobs` surface it the same way they surface any other
|
|
background job.
|
|
|
|
## What one pass does {#pass}
|
|
|
|
By default (`wal.remoteMaintenance` is on), a tick's passes are offloaded
|
|
as ordinary Job Registry worker jobs instead of running in the WAL pod
|
|
itself, up to `WAL_COMPACT_CONCURRENCY` at a time (32 by default). With
|
|
remote maintenance off, the pod runs them itself, in a per-pod compactor
|
|
pool shared across every table it owns, bounded to 2 concurrent passes
|
|
instead. Either way, each pass:
|
|
|
|
1. Snapshots the shard's current list of SSTables.
|
|
2. Takes a bounded prefix of them, oldest first (`WAL_COMPACT_PREFIX_MAX`,
|
|
32 by default), further capped at `COMPACT_PASS_TARGET_BYTES` (2 GiB)
|
|
-- a budget every in-pod pass shares, but that an offloaded pass gets
|
|
in full to itself.
|
|
3. Opens each as a Lance file and scans its rows.
|
|
4. Deduplicates across the prefix by primary key -- a later SSTable's row
|
|
wins over an earlier one with the same key.
|
|
5. Merges the deduplicated rows into the base table with `merge_insert`.
|
|
6. Commits a manifest update that drops exactly the SSTables just folded
|
|
in, identified individually rather than by position, so a retried
|
|
commit against a newer manifest can't drop the wrong ones.
|
|
7. `memwal_catchup_<type>` then extends each of the base table's indexes
|
|
-- one job per index type (`memwal_catchup_btree`, `memwal_catchup_fts`,
|
|
`memwal_catchup_ivfpq`, and so on) -- over the rows step 5 just added,
|
|
and records how far that index has caught up.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A["Shard crosses a trigger<br/>(count, bytes, or generation age)"] --> B["Per-pod trigger loop<br/>or backpressure relief pass"]
|
|
B --> C["Compactor pool<br/>dedup by PK, merge_insert into base"]
|
|
C --> D["Manifest drops<br/>the folded SSTables"]
|
|
D --> E["memwal_catchup_<type><br/>extends each base index"]
|
|
```
|
|
|
|
Only one compaction pass runs against a given shard at a time. A second,
|
|
concurrent pass against the same shard would corrupt it: a stale pass can
|
|
overwrite rows a newer generation already replaced, or resurrect a row a
|
|
newer generation deleted, since the merge step has no way to tell a
|
|
late-arriving write from a value that was already superseded. A per-shard
|
|
latch enforces this -- the periodic trigger skips a shard that's already
|
|
compacting, a manual `POST /v1/compact` is refused outright (never queued)
|
|
on the same collision, and node drain waits for the shard's current pass
|
|
instead of skipping it, since a skipped table is exactly what drain exists
|
|
to flush.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
Shard["Shard's per-shard latch"] --> Held{"Already held by<br/>an in-progress pass?"}
|
|
Held -->|no| Grant["Grant the latch,<br/>run the pass"]
|
|
Held -->|yes, periodic trigger| Skip["Skip this tick --<br/>retried next tick"]
|
|
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,<br/>never queued"]
|
|
Held -->|yes, node drain| Wait["Wait for the current<br/>pass to finish, don't skip"]
|
|
```
|
|
|
|
## Reading while compaction runs {#reads}
|
|
|
|
A query against an RTE-enabled table always reads the shard's current
|
|
memtable and SSTables unioned with the base table, regardless of where a
|
|
compaction pass is in its cycle -- compaction only ever changes which side
|
|
of that union a row's latest value lives on, never which value is
|
|
current. There's no window where a row is visible in neither side: the
|
|
manifest update in step 6 is the single point where a row's SSTable copy
|
|
stops being listed, and it only commits after `merge_insert` has already
|
|
written that row into the base table.
|
|
|
|
A table queried *without* the real time engine's live path only sees data
|
|
once it has been folded into the base table, so its staleness bound tracks
|
|
whichever trigger fires -- wider either way than the real time engine's
|
|
own millisecond-scale freshness. A shard under steady write pressure
|
|
compacts on the count or bytes trigger, within `WAL_COMPACTION_INTERVAL_SECS`
|
|
plus the compaction job's own runtime. A shard that never gathers enough
|
|
siblings or bytes instead waits on the age trigger, up to 7 days by
|
|
default, before it's compacted at all. `WAL_MAX_DATA_AGE_SECS` bounds two
|
|
independent stages -- the memtable freeze and this compaction age trigger
|
|
-- so a slow table's worst-case staleness without the live path is up to
|
|
roughly twice that default, not one interval plus one job's runtime.
|
|
Either way, the bound also assumes a free compaction slot when
|
|
the shard becomes eligible; an eligible shard can wait in queue behind
|
|
`WAL_COMPACT_CONCURRENCY` other passes already running, so the actual
|
|
delay can run longer than the trigger alone suggests. See
|
|
[Real Time reads](/search/realtime-read)
|
|
for what a live RTE read sees instead.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Real Time Engine" icon="bolt" href="/tables/realtime-engine">
|
|
The MemWAL format, shards, and WAL nodes this page assumes.
|
|
</Card>
|
|
<Card title="Distributed compaction" icon="compress" href="/enterprise/distributed-compaction">
|
|
The unrelated, ordinary compaction every table -- WAL-enabled or not -- gets.
|
|
</Card>
|
|
<Card title="Job Framework" icon="list-check" href="/basics/jobs">
|
|
Job handles, waiting, and inspecting jobs -- memwal_compaction included.
|
|
</Card>
|
|
</CardGroup>
|