mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack YeandClaude Sonnet 5
5e38120bdc
## Summary - **Define Storage Integration**: un-gates the Python (and Rust, for S3) `create_storage_integration` examples, drafted against #4618 and held back until it shipped -- it's now in v0.42.0-beta.0+, and the signature matches the drafts exactly. - **Naming rules**: drops the `Branch` row and its "Learn more" link, consistent with de-emphasizing branches elsewhere in this nav. - **Distributed RTE Write**: removes the REST path, the `__lance_mem_wal` index name, and "WAL node" terminology in favor of an abstract framing -- a query node routes writes to RTE nodes, each of which can host shards from many different tables. Both diagram variants are updated to match. - **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop Helm-specific and exact-percentage detail (disk-budget math, `startup_prewarm_block_query_ready`/`healthPort`/failure-policy table) in favor of a short feature → architecture → why-it-matters shape, each closing with a contact-sales pointer. Data Page Cache's on-demand prewarm section is removed outright. ## Note I wasn't able to get a clean visual render of the updated Distributed RTE Write diagram in local preview -- Mermaid isn't rendering in headless Chrome in this environment for *any* page right now, confirmed against an unmodified reference page showing the identical symptom. The diagram source follows the exact pattern already shipped and working in the 21 diagrams from the earlier brand-theming pass, and `mint validate`/`mint broken-links` both pass clean. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
84 lines
3.6 KiB
Plaintext
84 lines
3.6 KiB
Plaintext
---
|
|
title: "Distributed Compaction"
|
|
sidebarTitle: "Distributed Compaction"
|
|
description: "How LanceDB Enterprise distributes compaction across a fleet of job workers."
|
|
icon: "compress"
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Distributed compaction splits a table's fragments into independent,
|
|
fragment-disjoint tasks and fans them out across job workers in LanceDB
|
|
Enterprise. Unlike an index build, a compaction plan needs no bin-packing:
|
|
Lance's own planner already produces disjoint tasks, so workers claim and
|
|
execute them directly, with no coordination needed to decide who does
|
|
what. See [Architecture](/index#architecture).
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"]
|
|
L --> W1["Worker<br/>claims fragments 1-2"]
|
|
L --> W2["Worker<br/>claims fragments 3-4"]
|
|
L --> W3["Worker<br/>claims fragments 5-6"]
|
|
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
|
W2 --> C
|
|
W3 --> C
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
|
class C accent
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"]
|
|
L --> W1["Worker<br/>claims fragments 1-2"]
|
|
L --> W2["Worker<br/>claims fragments 3-4"]
|
|
L --> W3["Worker<br/>claims fragments 5-6"]
|
|
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
|
W2 --> C
|
|
W3 --> C
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
|
class C accent
|
|
```
|
|
|
|
</div>
|
|
|
|
No task depends on another's result, so a worker that finishes early
|
|
simply claims another task rather than waiting on its peers. Committing is
|
|
where the fleet does coordinate: finished tasks go through a shared
|
|
eligibility gate, and only the one worker that claims the commit actually
|
|
writes it, rather than each worker committing its own task independently.
|
|
|
|
Whether a failed task can still let the rest commit depends on the remap
|
|
mode. Inline remap is all-or-nothing: one failed task fails the whole job,
|
|
since the commit updates every affected index inline and can't skip a
|
|
task's share of that. Deferred remap can commit a subset -- a failed
|
|
task's fragments are abandoned and replanned in a later round, while the
|
|
rest of the job's tasks commit without waiting on it. See
|
|
[Advanced Conflict Resolution](/enterprise/advanced-conflict-resolution)
|
|
for when compaction defers a remap at all.
|
|
|
|
## Compaction interfaces {#compaction-interfaces}
|
|
|
|
- [Compaction](/maintenance/compaction) for triggering and configuring a compaction.
|
|
- [Garbage Collection](/maintenance/cleanup) for removing files compaction and other operations leave behind.
|
|
- [Job Framework](/basics/jobs) for the job types a deployment runs on its own schedule.
|