mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-28 06:26:15 +00:00
951ae51ef6
Adds a user-facing realtime engine (RTE) page at `docs/web/tables/realtime-engine.mdx`, listed under **Management** after Branches. It is adapted from the internal WAL field guide (lancedb/sophon#6213), with the deployment and operations material left out. The page covers: - what RTE is and how it works (memtable → SSTables → compaction; reads merge the base table with the fresh tier) - getting started: `set_unenforced_primary_key`, indexes, `enable_rte` / `ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial updates, deletes, and `checkpoint_rte` / `disable_rte` - performance: write throughput and shard scaling, and read latency with and without RTE - a note on operations not yet supported (time travel, branching, `count_rows`, row IDs, `TABLESAMPLE`) Also aligns related docs with current RTE naming and behavior: - `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` / `UNSET RTE` - `maintenance/background-jobs.mdx`: job names are `memwal_compaction` and `memwal_catchup_<type>` - `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine - Python `AsyncTable.enable_rte` / `RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes created after enabling RTE are maintained too The examples are written directly in the page rather than generated from `docs/web-tests`, because RTE needs an Enterprise server to run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: XY Zhan <zhaner08@hotmail.com>
74 lines
3.8 KiB
Plaintext
74 lines
3.8 KiB
Plaintext
---
|
|
title: "Background jobs"
|
|
sidebarTitle: "Background Jobs"
|
|
description: "Understand the Enterprise background jobs that maintain tables, indexes and caches."
|
|
icon: gears
|
|
keywords: ["background jobs", "job type", "wal", "spfresh", "prewarm", "archive"]
|
|
availability:
|
|
oss: unavailable
|
|
enterprise: available
|
|
summary: >-
|
|
LanceDB Enterprise schedules background maintenance jobs according to your
|
|
cluster configuration.
|
|
---
|
|
|
|
The jobs documented elsewhere in these docs ([Build & Repair](/indexing/reindexing),
|
|
[Compaction](/maintenance/compaction), [Cleanup](/maintenance/cleanup),
|
|
[Column Refresh](/feature-engineering/refresh-column)) start from a call you
|
|
make or, for compaction, cleanup and index updates, from the deployment's own
|
|
maintenance. This page is the rest: jobs the deployment runs to keep itself
|
|
correct and fast, which you may see named in [`SHOW JOBS`](/sql/ddl/job#show)
|
|
without having asked for any of them.
|
|
|
|
These jobs are configured by the deployment's operators rather than through
|
|
the table SDK. If a maintenance job repeatedly fails or stops making progress,
|
|
share its job ID and errors with your Enterprise support team. Operators can
|
|
also start cache-warming jobs through the admin API.
|
|
|
|
## Write-ahead log maintenance {#wal}
|
|
|
|
For a table with the [realtime engine](/tables/realtime-engine) enabled, two jobs keep the
|
|
log from growing without bound:
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `memwal_compaction` | Folds a table shard's SSTables into the base table by primary key. Distinct from `compact`, which merges base-table fragments. The two are unrelated. |
|
|
| `memwal_catchup_<type>` | Extends a base-table index over the rows `memwal_compaction` just folded in, and records how far the index has caught up. Nothing else advances that record. The suffix names the index type, for example `memwal_catchup_btree`, `memwal_catchup_fts`, or `memwal_catchup_ivfpq`. |
|
|
|
|
## Index maintenance {#index}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `index_remap` | Reassigns index entries after row ids shift underneath an index (the kind of remap a delta compaction can require). |
|
|
| `spfresh_merge` | Merges SPFresh-managed segments for indexes that use it. |
|
|
|
|
## Cache warming {#cache}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `prewarm_page_cache` | Loads table data into a query node's page cache ahead of a request needing it. |
|
|
| `prewarm_index_cache` | The same, for index structures. |
|
|
| `startup_prewarm_idx` | Runs cache prewarming as part of a plan executor's own startup, rather than on a schedule, when the deployment enables it. |
|
|
|
|
## Registry housekeeping {#registry}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `job_registry_archive` | Sweeps terminal jobs out of the active registry and into archival storage, on a per-database schedule. |
|
|
| `job_events_export` | Copies terminal jobs' event histories into a curated system table for metering. Registry-scoped, like the archive sweep. |
|
|
|
|
These two are why an old job you completed weeks ago no longer shows in
|
|
`SHOW JOBS`: it was archived, not deleted. Where the deployment enables the
|
|
export, its events also survive in the exported table.
|
|
|
|
## Object teardown {#teardown}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Whether a dropped table runs as this job or as [`drop_table`](/management/table#drop), which fences the table's directory with its own marker instead, depends on how the deployment stores its catalog. |
|
|
|
|
You trigger it indirectly, by dropping a table, a [view](/sql/ddl/view#drop) or
|
|
a [namespace](/management/namespace); the teardown runs as this job rather than
|
|
inline with the call.
|
|
{/* Draft for the Functions release: Dropping a [function](/management/function#drop) starts it the same way. */}
|