Files
lancedb/docs/web/enterprise/realtime-engine-compaction.mdx
T
Taylor 3a980054a1 docs: standardize Realtime Engine naming (#4969)
The Enterprise feature name was inconsistently written as "Real Time
Engine" and "real time engine". Standardize it to **Realtime Engine** in
eight website guides, including page and sidebar titles, descriptions,
prose, link labels, cards, and search keywords.

The remaining MemWAL references describe Lance's open source format or
literal API/job identifiers, including `memwal_compaction` and
`memwal_catchup_<type>`, and are preserved. Only `docs/web/*.mdx` pages
change; SDK source, generated API reference, code examples, URL
destinations, and section anchors are unchanged.

Validation:
- `git diff --check` passes.
- Reviewed all MemWAL and Realtime Engine references under `docs/`.
- Verified each changed file differs only by the intended naming
substitutions and preserves code blocks, inline code, links, anchors,
and MemWAL identifiers.
- Confirmed no split or hyphenated "real time engine" names remain in
website guides.
- No snippets needed regeneration because code examples are unchanged.
Vale and Mintlify are not installed locally, so their checks were not
run locally.
2026-11-13 11:24:55 -08:00

210 lines
10 KiB
Plaintext

---
title: "Realtime Engine compaction"
sidebarTitle: "RTE Compaction"
description: "How Enterprise automatically folds a Realtime Engine shard's SSTables into the base table and catches its indexes up."
icon: "compress"
availability:
embedded: unavailable
enterprise: available
---
[Distributed Compaction](/enterprise/distributed-compaction) merges small
base-table fragments and reclaims deleted rows -- the same operation
[Compaction](/maintenance/compaction) describes for LanceDB Embedded. A
table with the [Realtime Engine](/tables/realtime-engine) enabled has a
second, unrelated compaction: folding a shard's accumulated SSTables into
the base table and catching the base table's indexes up over the rows that
just landed. This produces two job types, `memwal_compaction` and
`memwal_catchup_<type>`, already distinct from ordinary `compact` in
`SHOW JOBS` and `list_jobs`. This page covers how Enterprise decides when
to run them, not just what they do.
## What triggers a compaction pass {#trigger}
A per-pod loop, ticking every `WAL_COMPACTION_INTERVAL_SECS` (5s by
default), walks every table this pod owns and reads each shard's manifest.
It dispatches a compaction job for a shard once any threshold crosses:
| Trigger | Default | Checks |
| --- | --- | --- |
| `WAL_COMPACT_TRIGGER_COUNT` | 4 | The shard's SSTable count |
| `WAL_COMPACT_TRIGGER_BYTES` | 1 GiB | The shard's total flushed SSTable bytes |
| Oldest generation's age | 7 days | How long the head SSTable generation has sat, whatever the file count -- a file that will never gather enough siblings to cross the count trigger still gets compacted eventually |
Whichever crosses first wins. A shard under write pressure does not wait
for the next tick either: the same thresholds are re-checked on every
backpressure relief pass, so a hot shard that already qualifies compacts
sooner than `WAL_COMPACTION_INTERVAL_SECS` would otherwise allow. Setting
`WAL_COMPACTION_INTERVAL_SECS` to `0` disables the periodic walk; an
operator can still force a pass with `POST /v1/compact`, and
`Table::checkpoint_rte` drives the same flush-then-compact cycle from the
client when you need a table to converge on demand rather than wait for
the trigger. `checkpoint_rte` has no liveness bound of its own -- it is
idempotent and abandonable at any point, but the caller owns the deadline
for how long to wait on it.
This trigger decision is internal to the WAL pod -- it is not the general
job scheduler deciding to run `memwal_compaction`, the pod decides to
enqueue one. Once enqueued, the work is still tracked as an ordinary job
(`memwal_compaction`, then `memwal_catchup_<type>` once it lands), so
`SHOW JOBS` and `list_jobs` surface it the same way they surface any other
background job.
## What one pass does {#pass}
By default (`wal.remoteMaintenance` is on), a tick's passes are offloaded
as ordinary Job Registry worker jobs instead of running in the WAL pod
itself, up to `WAL_COMPACT_CONCURRENCY` at a time (32 by default). With
remote maintenance off, the pod runs them itself, in a per-pod compactor
pool shared across every table it owns, bounded to 2 concurrent passes
instead. Either way, each pass:
1. Snapshots the shard's current list of SSTables.
2. Takes a bounded prefix of them, oldest first (`WAL_COMPACT_PREFIX_MAX`,
32 by default), further capped at `COMPACT_PASS_TARGET_BYTES` (2 GiB)
-- a budget every in-pod pass shares, but that an offloaded pass gets
in full to itself.
3. Opens each as a Lance file and scans its rows.
4. Deduplicates across the prefix by primary key -- a later SSTable's row
wins over an earlier one with the same key.
5. Merges the deduplicated rows into the base table with `merge_insert`.
6. Commits a manifest update that drops exactly the SSTables just folded
in, identified individually rather than by position, so a retried
commit against a newer manifest can't drop the wrong ones.
7. `memwal_catchup_<type>` then extends each of the base table's indexes
-- one job per index type (`memwal_catchup_btree`, `memwal_catchup_fts`,
`memwal_catchup_ivfpq`, and so on) -- over the rows step 5 just added,
and records how far that index has caught up.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
A["Shard crosses a trigger<br/>(count, bytes, or generation age)"] --> B["Per-pod trigger loop<br/>or backpressure relief pass"]
B --> C["Compactor pool<br/>dedup by PK, merge_insert into base"]
C --> D["Manifest drops<br/>the folded SSTables"]
D --> E["memwal_catchup_&lt;type&gt;<br/>extends each base index"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
A["Shard crosses a trigger<br/>(count, bytes, or generation age)"] --> B["Per-pod trigger loop<br/>or backpressure relief pass"]
B --> C["Compactor pool<br/>dedup by PK, merge_insert into base"]
C --> D["Manifest drops<br/>the folded SSTables"]
D --> E["memwal_catchup_&lt;type&gt;<br/>extends each base index"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
Only one compaction pass runs against a given shard at a time. A second,
concurrent pass against the same shard would corrupt it: a stale pass can
overwrite rows a newer generation already replaced, or resurrect a row a
newer generation deleted, since the merge step has no way to tell a
late-arriving write from a value that was already superseded. A per-shard
latch enforces this -- the periodic trigger skips a shard that's already
compacting, a manual `POST /v1/compact` is refused outright (never queued)
on the same collision, and node drain waits for the shard's current pass
instead of skipping it, since a skipped table is exactly what drain exists
to flush.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart TD
Shard["Shard's per-shard latch"] --> Held{"Already held by<br/>an in-progress pass?"}
Held -->|no| Grant["Grant the latch,<br/>run the pass"]
Held -->|yes, periodic trigger| Skip["Skip this tick --<br/>retried next tick"]
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,<br/>never queued"]
Held -->|yes, node drain| Wait["Wait for the current<br/>pass to finish, don't skip"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class Shard accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart TD
Shard["Shard's per-shard latch"] --> Held{"Already held by<br/>an in-progress pass?"}
Held -->|no| Grant["Grant the latch,<br/>run the pass"]
Held -->|yes, periodic trigger| Skip["Skip this tick --<br/>retried next tick"]
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,<br/>never queued"]
Held -->|yes, node drain| Wait["Wait for the current<br/>pass to finish, don't skip"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Shard accent
```
</div>
## Reading while compaction runs {#reads}
A query against an RTE-enabled table always reads the shard's current
memtable and SSTables unioned with the base table, regardless of where a
compaction pass is in its cycle -- compaction only ever changes which side
of that union a row's latest value lives on, never which value is
current. There's no window where a row is visible in neither side: the
manifest update in step 6 is the single point where a row's SSTable copy
stops being listed, and it only commits after `merge_insert` has already
written that row into the base table.
A table queried *without* the Realtime Engine's live path only sees data
once it has been folded into the base table, so its staleness bound tracks
whichever trigger fires -- wider either way than the Realtime Engine's
own millisecond-scale freshness. A shard under steady write pressure
compacts on the count or bytes trigger, within `WAL_COMPACTION_INTERVAL_SECS`
plus the compaction job's own runtime. A shard that never gathers enough
siblings or bytes instead waits on the age trigger, up to 7 days by
default, before it's compacted at all. `WAL_MAX_DATA_AGE_SECS` bounds two
independent stages -- the memtable freeze and this compaction age trigger
-- so a slow table's worst-case staleness without the live path is up to
roughly twice that default, not one interval plus one job's runtime.
Either way, the bound also assumes a free compaction slot when
the shard becomes eligible; an eligible shard can wait in queue behind
`WAL_COMPACT_CONCURRENCY` other passes already running, so the actual
delay can run longer than the trigger alone suggests. See
[RTE reads](/search/realtime-read)
for what a live RTE read sees instead.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Realtime Engine" icon="bolt" href="/tables/realtime-engine">
The MemWAL format, shards, and WAL nodes this page assumes.
</Card>
<Card title="Distributed Compaction" icon="compress" href="/enterprise/distributed-compaction">
The unrelated, ordinary compaction every table -- WAL-enabled or not -- gets.
</Card>
<Card title="Job Framework" icon="list-check" href="/basics/jobs">
Job handles, waiting, and inspecting jobs -- memwal_compaction included.
</Card>
</CardGroup>