mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-19 10:12:28 +00:00
Dan RammerandClaude Opus 5
fe34eb3549
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
74 lines
3.6 KiB
Plaintext
74 lines
3.6 KiB
Plaintext
---
|
|
title: "Background jobs"
|
|
sidebarTitle: "Background Jobs"
|
|
description: "Understand the Enterprise background jobs that maintain tables, indexes and caches."
|
|
icon: gears
|
|
keywords: ["background jobs", "job type", "wal", "spfresh", "prewarm", "archive"]
|
|
availability:
|
|
oss: unavailable
|
|
enterprise: available
|
|
summary: >-
|
|
LanceDB Enterprise schedules background maintenance jobs according to your
|
|
cluster configuration.
|
|
---
|
|
|
|
The jobs documented elsewhere in these docs -- [Build & Repair](/indexing/reindexing),
|
|
[Compaction](/maintenance/compaction), [Cleanup](/maintenance/cleanup),
|
|
[Column Refresh](/feature-engineering/refresh-column) -- start from a call you
|
|
make or, for compaction, cleanup and index updates, from the deployment's own
|
|
maintenance. This page is the rest: jobs the deployment runs to keep itself
|
|
correct and fast, which you may see named in [`SHOW JOBS`](/sql/ddl/job#show)
|
|
without having asked for any of them.
|
|
|
|
These jobs are configured by the deployment's operators rather than through
|
|
the table SDK. If a maintenance job repeatedly fails or stops making progress,
|
|
share its job ID and errors with your Enterprise support team. Operators can
|
|
also start cache-warming jobs through the admin API.
|
|
|
|
## Write-ahead log maintenance {#wal}
|
|
|
|
For a table with a [write-ahead log](/sql/ddl/table#properties) enabled, two jobs keep the
|
|
log from growing without bound:
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `sstable_compaction` | Folds a table shard's SSTables into the base table by primary key. Distinct from `compact`, which merges base-table fragments -- the two are unrelated. |
|
|
| `wal_index_catchup` | Extends a base-table index over the rows `sstable_compaction` just folded in, and records how far the index has caught up. Nothing else advances that record. |
|
|
|
|
## Index maintenance {#index}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `index_remap` | Reassigns index entries after row ids shift underneath an index -- the kind of remap a delta compaction can require. |
|
|
| `spfresh_merge` | Merges SPFresh-managed segments for indexes that use it. |
|
|
|
|
## Cache warming {#cache}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `prewarm_page_cache` | Loads table data into a query node's page cache ahead of a request needing it. |
|
|
| `prewarm_index_cache` | The same, for index structures. |
|
|
| `startup_prewarm_idx` | Runs cache prewarming as part of a plan executor's own startup, rather than on a schedule, when the deployment enables it. |
|
|
|
|
## Registry housekeeping {#registry}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `job_registry_archive` | Sweeps terminal jobs out of the active registry and into archival storage, on a per-database schedule. |
|
|
| `job_events_export` | Copies terminal jobs' event histories into a curated system table for metering. Registry-scoped, like the archive sweep. |
|
|
|
|
These two are why an old job you completed weeks ago no longer shows in
|
|
`SHOW JOBS` -- it was archived, not deleted. Where the deployment enables the
|
|
export, its events also survive in the exported table.
|
|
|
|
## Object teardown {#teardown}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Whether a dropped table runs as this job or as [`drop_table`](/management/table#drop), which fences the table's directory with its own marker instead, depends on how the deployment stores its catalog. |
|
|
|
|
You trigger it indirectly, by dropping a table, a [view](/sql/ddl/view#drop) or
|
|
a [namespace](/management/namespace); the teardown runs as this job rather than
|
|
inline with the call.
|
|
{/* Draft for the Functions release: Dropping a [function](/management/function#drop) starts it the same way. */}
|