mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack YeandClaude Sonnet 5
d0b0efcd47
## Summary Follow-up to the Mermaid brand-theming PR, addressing review feedback: - Diagram text was hard to read at the previous 15px font size; bumped to 18px, light and dark alike. - The dark-mode orange accent border was brighter and more saturated than its fill or the default border, reading as too loud against the dark background. It now reuses the same muted orange already used for the light-mode accent border, which was fine as is. ## Companion change lancedb/sophon#9719 makes the same two changes across its own diagrams. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
84 lines
3.6 KiB
Plaintext
84 lines
3.6 KiB
Plaintext
---
|
|
title: "Distributed compaction"
|
|
sidebarTitle: "Distributed compaction"
|
|
description: "How LanceDB Enterprise distributes compaction across a fleet of job workers."
|
|
icon: "compress"
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Distributed compaction splits a table's fragments into independent,
|
|
fragment-disjoint tasks and fans them out across job workers in LanceDB
|
|
Enterprise. Unlike an index build, a compaction plan needs no bin-packing:
|
|
Lance's own planner already produces disjoint tasks, so workers claim and
|
|
execute them directly through the job registry, with no coordination
|
|
needed to decide who does what. See [Architecture](/index#architecture).
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
|
|
R --> W1["Worker<br/>claims fragments 1-2"]
|
|
R --> W2["Worker<br/>claims fragments 3-4"]
|
|
R --> W3["Worker<br/>claims fragments 5-6"]
|
|
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
|
W2 --> C
|
|
W3 --> C
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
|
class R,C accent
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
|
|
R --> W1["Worker<br/>claims fragments 1-2"]
|
|
R --> W2["Worker<br/>claims fragments 3-4"]
|
|
R --> W3["Worker<br/>claims fragments 5-6"]
|
|
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
|
W2 --> C
|
|
W3 --> C
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
|
class R,C accent
|
|
```
|
|
|
|
</div>
|
|
|
|
No task depends on another's result, so a worker that finishes early
|
|
simply claims another task rather than waiting on its peers. Committing is
|
|
where the fleet does coordinate: finished tasks go through a shared
|
|
eligibility gate, and only the one worker that claims the commit actually
|
|
writes it, rather than each worker committing its own task independently.
|
|
|
|
Whether a failed task can still let the rest commit depends on the remap
|
|
mode. Inline remap is all-or-nothing: one failed task fails the whole job,
|
|
since the commit updates every affected index inline and can't skip a
|
|
task's share of that. Deferred remap can commit a subset -- a failed
|
|
task's fragments are abandoned and replanned in a later round, while the
|
|
rest of the job's tasks commit without waiting on it. See
|
|
[Advanced Conflict Resolution](/enterprise/advanced-conflict-resolution)
|
|
for when compaction defers a remap at all.
|
|
|
|
## Compaction interfaces {#compaction-interfaces}
|
|
|
|
- [Compaction](/maintenance/compaction) for triggering and configuring a compaction.
|
|
- [Garbage Collection](/maintenance/cleanup) for removing files compaction and other operations leave behind.
|
|
- [Job Framework](/basics/jobs) for the job types a deployment runs on its own schedule.
|