mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
style: re-theme Mermaid diagrams to the brand palette, light and dark (#4822)
## Summary Every Mermaid diagram now renders in the brand palette, with separate light and dark variants that switch with the site's theme toggle. Default node styling stays neutral; one accent color marks each diagram's actual subject, chosen per diagram. A couple of diagrams changed orientation where their old layout was too wide to read comfortably. Companion change in lancedb/sophon uses the same colors and pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
b09735529b
commit
dfbf682cb5
@@ -47,7 +47,14 @@ For example, a search application can keep production and staging data in
|
||||
separate namespaces within the same database, with `prod` further split into
|
||||
a nested `search` namespace:
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
catalog["root catalog"] --> database["ai_data · database"]
|
||||
database --> prod["prod · namespace"]
|
||||
@@ -57,8 +64,33 @@ flowchart TD
|
||||
search --> search_view["recent_documents · view"]
|
||||
search --> search_fn["embed · function"]
|
||||
staging --> staging_docs["documents · table"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
catalog["root catalog"] --> database["ai_data · database"]
|
||||
database --> prod["prod · namespace"]
|
||||
database --> staging["staging · namespace"]
|
||||
prod --> search["search · namespace"]
|
||||
search --> search_docs["documents · table"]
|
||||
search --> search_view["recent_documents · view"]
|
||||
search --> search_fn["embed · function"]
|
||||
staging --> staging_docs["documents · table"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
| Object | What it represents |
|
||||
| --- | --- |
|
||||
| [Catalog](/management/catalog) | The unnamed root of an Enterprise deployment, identified by its endpoint; lists, creates and drops databases. Not present in Embedded. |
|
||||
|
||||
@@ -19,7 +19,15 @@ the join has already settled which rows survive.
|
||||
|
||||
## The two plan shapes {#plan-shapes}
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TB
|
||||
subgraph Naive["Naive: wide column read before filtering"]
|
||||
direction TB
|
||||
@@ -32,8 +40,41 @@ flowchart TB
|
||||
D1["Scan only the join key<br/>(+ early columns)"] --> D2["Join on the key"]
|
||||
D2 --> D3["RemoteTake:<br/>fetch the wide column,<br/>once per surviving row"]
|
||||
end
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class D3 accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TB
|
||||
subgraph Naive["Naive: wide column read before filtering"]
|
||||
direction TB
|
||||
N1["Scan wide column<br/>for every row"] --> N2["Join"]
|
||||
N2 --> N3["Rows that didn't match<br/>are discarded --<br/>their wide read was wasted"]
|
||||
end
|
||||
|
||||
subgraph Deferred["Late materialization: wide column read after filtering"]
|
||||
direction TB
|
||||
D1["Scan only the join key<br/>(+ early columns)"] --> D2["Join on the key"]
|
||||
D2 --> D3["RemoteTake:<br/>fetch the wide column,<br/>once per surviving row"]
|
||||
end
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class D3 accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
In the naive plan, the wide column is read once per **candidate** row --
|
||||
every row either side of the join could possibly produce -- before the
|
||||
join has thrown any of them away. In the rewritten plan, the join
|
||||
|
||||
@@ -18,7 +18,15 @@ plan needs data repartitioned across machines -- a hash join or a grouped
|
||||
aggregate, for example, where rows with the same key must land on the
|
||||
same worker regardless of which plan executor originally read them.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
subgraph Stage1["Stage: scan"]
|
||||
direction TB
|
||||
@@ -38,8 +46,48 @@ flowchart LR
|
||||
PE2 --> EX
|
||||
EX --> PE3
|
||||
EX --> PE4
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class EX accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
subgraph Stage1["Stage: scan"]
|
||||
direction TB
|
||||
PE1["Plan executor<br/>reads its partition"]
|
||||
PE2["Plan executor<br/>reads its partition"]
|
||||
end
|
||||
|
||||
EX["Exchange<br/>streaming gRPC,<br/>repartitions by join/group key,<br/>backpressure via a byte budget"]
|
||||
|
||||
subgraph Stage2["Stage: join / aggregate"]
|
||||
direction TB
|
||||
PE3["Plan executor<br/>probes its share of keys"]
|
||||
PE4["Plan executor<br/>probes its share of keys"]
|
||||
end
|
||||
|
||||
PE1 --> EX
|
||||
PE2 --> EX
|
||||
EX --> PE3
|
||||
EX --> PE4
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class EX accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Pipelined, not batched {#pipelined}
|
||||
|
||||
The exchange moves data as a gRPC stream of row batches, not a file a
|
||||
|
||||
@@ -38,7 +38,14 @@ memory, out-of-memory events, and for a GPU Function, compute and memory
|
||||
utilization per GPU -- and that measurement decides whether the *next*
|
||||
invocation runs with more resources, fewer, or the same:
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
I["Invocation runs<br/>wall time, CPU, memory, GPU measured"] --> D{"Saturated?"}
|
||||
D -->|"OOM, or memory/CPU/GPU<br/>over target utilization"| P["Promote:<br/>next size up"]
|
||||
@@ -48,8 +55,33 @@ flowchart LR
|
||||
Dm --> N
|
||||
S --> N
|
||||
N --> I
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
I["Invocation runs<br/>wall time, CPU, memory, GPU measured"] --> D{"Saturated?"}
|
||||
D -->|"OOM, or memory/CPU/GPU<br/>over target utilization"| P["Promote:<br/>next size up"]
|
||||
D -->|"fast task, low CPU,<br/>known memory"| Dm["Demote:<br/>one size down,<br/>never past a floor"]
|
||||
D -->|neither| S["Stay:<br/>same class"]
|
||||
P --> N["Next invocation"]
|
||||
Dm --> N
|
||||
S --> N
|
||||
N --> I
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
**Promotion** happens on an out-of-memory event, memory utilization above
|
||||
70% of the class's limit, or CPU pegged at 85%+ for a sample long enough
|
||||
(5 seconds) that one fast spike isn't mistaken for sustained saturation. A
|
||||
|
||||
@@ -65,7 +65,14 @@ Once a table is tagged, Lance refuses a compaction that does not set
|
||||
Enterprise-tagged table has to keep deferring the remap, or be compacted
|
||||
by a writer that can remap the tagged history eagerly.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
Commit["Compaction commit<br/>(defer_index_remap only)"] --> Tagged{"Table already<br/>tagged with an FRI?"}
|
||||
Tagged -->|no| First{"Stable row IDs, or does<br/>any index cover the<br/>rewritten fragments?"}
|
||||
@@ -76,8 +83,38 @@ flowchart TD
|
||||
Classify -->|yes| Append["Must append a tagged<br/>transition before the<br/>commit is admitted"]
|
||||
V0 --> FRI[("Fragment reuse index")]
|
||||
Append --> FRI
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class FRI accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
Commit["Compaction commit<br/>(defer_index_remap only)"] --> Tagged{"Table already<br/>tagged with an FRI?"}
|
||||
Tagged -->|no| First{"Stable row IDs, or does<br/>any index cover the<br/>rewritten fragments?"}
|
||||
First -->|no| Admit["Admitted as<br/>ordinary business --<br/>nothing appended"]
|
||||
First -->|yes| V0["Records a legacy v0 entry --<br/>table stays untagged"]
|
||||
Tagged -->|yes| Classify{"Does any index<br/>cover the rewritten<br/>fragments?"}
|
||||
Classify -->|no| Admit
|
||||
Classify -->|yes| Append["Must append a tagged<br/>transition before the<br/>commit is admitted"]
|
||||
V0 --> FRI[("Fragment reuse index")]
|
||||
Append --> FRI
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class FRI accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Enterprise defers and batches the catch-up {#catch-up}
|
||||
|
||||
Translating addresses through a long chain of transitions at every read adds
|
||||
@@ -102,15 +139,48 @@ rather than one blocking on the other: a handful of compactions can land
|
||||
before their indexes are physically remapped, and the catch-up job amortizes
|
||||
that remap across all of them in one job instead of one per compaction.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index<br/>(ledger of transitions)"]
|
||||
C2["Compaction job"] -->|appends transition| FRI
|
||||
C3["Compaction job"] -->|appends transition| FRI
|
||||
FRI -->|5 transitions or 1 day old| R["Remap catch-up job"]
|
||||
R -->|applies accumulated transitions| IDX["Affected indexes,<br/>physically remapped"]
|
||||
FRI -->|5 transitions or<br/>1 day old| R["Remap catch-up job"]
|
||||
R -->|applies accumulated<br/>transitions| IDX["Affected indexes,<br/>physically remapped"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class FRI,R accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index<br/>(ledger of transitions)"]
|
||||
C2["Compaction job"] -->|appends transition| FRI
|
||||
C3["Compaction job"] -->|appends transition| FRI
|
||||
FRI -->|5 transitions or<br/>1 day old| R["Remap catch-up job"]
|
||||
R -->|applies accumulated<br/>transitions| IDX["Affected indexes,<br/>physically remapped"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class FRI,R accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
|
||||
@@ -24,7 +24,14 @@ that started but never recorded completion is redone in full on the next
|
||||
attempt, so a Function with external side effects (a remote API call, for
|
||||
example) can have those side effects repeated for that part's rows.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Part started<br/>event recorded"] --> E["Function executes"]
|
||||
E --> C["Part completed<br/>event recorded"]
|
||||
@@ -34,8 +41,37 @@ flowchart LR
|
||||
|
||||
S -. "worker dies here" .-> R["Next attempt reads the log,<br/>sees no completion,<br/>redoes only this part"]
|
||||
C -. "worker dies here" .-> R2["Next attempt reads the log,<br/>sees this part done,<br/>inherits it, moves on"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class R,R2 accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Part started<br/>event recorded"] --> E["Function executes"]
|
||||
E --> C["Part completed<br/>event recorded"]
|
||||
C --> M{"More parts?"}
|
||||
M -->|yes| S
|
||||
M -->|no| F["Finalized:<br/>digest-verified assembly<br/>into the real data file"]
|
||||
|
||||
S -. "worker dies here" .-> R["Next attempt reads the log,<br/>sees no completion,<br/>redoes only this part"]
|
||||
C -. "worker dies here" .-> R2["Next attempt reads the log,<br/>sees this part done,<br/>inherits it, moves on"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class R,R2 accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## What gets recorded, and when {#events}
|
||||
|
||||
Each part contributes two events to its data file's log: one when a
|
||||
|
||||
@@ -16,7 +16,14 @@ the pages they serve. A background prewarm job can populate a cache ahead
|
||||
of a request that needs it, rather than filling it lazily on first read.
|
||||
See [Architecture](/index#architecture).
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
R["Read request<br/>for a data page"] --> M{"In memory?"}
|
||||
M -->|hit| Return["Served from memory"]
|
||||
@@ -24,8 +31,31 @@ flowchart LR
|
||||
D -->|hit| Promote["Loaded into memory,<br/>served"]
|
||||
D -->|miss| O[("Object storage")]
|
||||
O --> Fill["Fills memory and disk,<br/>served"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
R["Read request<br/>for a data page"] --> M{"In memory?"}
|
||||
M -->|hit| Return["Served from memory"]
|
||||
M -->|miss| D{"In local disk cache?"}
|
||||
D -->|hit| Promote["Loaded into memory,<br/>served"]
|
||||
D -->|miss| O[("Object storage")]
|
||||
O --> Fill["Fills memory and disk,<br/>served"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
A table's local disk cache shares one budget with the
|
||||
[tiered index cache](/enterprise/tiered-index-cache#disk-budget)'s disk
|
||||
tier where both are enabled on the same executor -- raising one's share of
|
||||
|
||||
@@ -15,7 +15,14 @@ Lance's own planner already produces disjoint tasks, so workers claim and
|
||||
execute them directly through the job registry, with no coordination
|
||||
needed to decide who does what. See [Architecture](/index#architecture).
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
|
||||
R --> W1["Worker<br/>claims fragments 1-2"]
|
||||
@@ -24,8 +31,36 @@ flowchart LR
|
||||
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class R,C accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
|
||||
R --> W1["Worker<br/>claims fragments 1-2"]
|
||||
R --> W2["Worker<br/>claims fragments 3-4"]
|
||||
R --> W3["Worker<br/>claims fragments 5-6"]
|
||||
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class R,C accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
No task depends on another's result, so a worker that finishes early
|
||||
simply claims another task rather than waiting on its peers. Committing is
|
||||
where the fleet does coordinate: finished tasks go through a shared
|
||||
|
||||
@@ -15,7 +15,14 @@ registry and commit the finished index together. A vector index build first
|
||||
trains a shared IVF/PQ model from sampled data, then workers build storage
|
||||
from that model in parallel. See [Architecture](/index#architecture).
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
|
||||
T --> P["Bin-pack build plan<br/>into claimable tasks"]
|
||||
@@ -26,8 +33,38 @@ flowchart LR
|
||||
W1 --> C["Commit the<br/>finished index"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class R accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
|
||||
T --> P["Bin-pack build plan<br/>into claimable tasks"]
|
||||
P --> R[("Job registry")]
|
||||
R --> W1["Worker<br/>claims a task"]
|
||||
R --> W2["Worker<br/>claims a task"]
|
||||
R --> W3["Worker<br/>claims a task"]
|
||||
W1 --> C["Commit the<br/>finished index"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class R accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Indexing interfaces {#indexing-interfaces}
|
||||
|
||||
The shared indexing guides describe how to build and maintain each index
|
||||
|
||||
@@ -17,7 +17,15 @@ usually served by different machines -- plan executors for the base table,
|
||||
a [WAL node](/tables/realtime-engine#wal-nodes) for the table's shards --
|
||||
so a query has to combine their results across the network instead.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>plans the query"]
|
||||
|
||||
@@ -41,8 +49,52 @@ flowchart LR
|
||||
W1 --> U
|
||||
W2 --> U
|
||||
U --> R["Merged result"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class U accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>plans the query"]
|
||||
|
||||
subgraph PE["Plan executors"]
|
||||
direction TB
|
||||
B1["Base-table arm<br/>index-covered scan or ANN search"]
|
||||
end
|
||||
|
||||
subgraph WN["WAL node (owns this table's shards)"]
|
||||
direction TB
|
||||
W1["SSTable arm<br/>index-covered"]
|
||||
W2["Active memtable arm<br/>vector: in-memory HNSW*<br/>FTS/scalar: indexed"]
|
||||
end
|
||||
|
||||
U["Union exec<br/>PK-keyed last-write-wins dedup,<br/>top-K cut on distance"]
|
||||
|
||||
Q --> B1
|
||||
Q --> W1
|
||||
Q --> W2
|
||||
B1 --> U
|
||||
W1 --> U
|
||||
W2 --> U
|
||||
U --> R["Merged result"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class U accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## An extra exec node, not a parallel system {#union-exec}
|
||||
|
||||
The distributed plan for an RTE-enabled table is an ordinary plan executor
|
||||
|
||||
@@ -26,7 +26,15 @@ single owning **WAL node**, a stateful pod separate from the query nodes and
|
||||
plan executors that serve ordinary queries. WAL node pods share nothing but
|
||||
the underlying object storage; no pod talks to another one directly.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>hashes (db, table) to one WAL node"]
|
||||
|
||||
@@ -44,8 +52,42 @@ flowchart LR
|
||||
|
||||
Q -- "writes + reads for 'frames'" --> W0
|
||||
Q -- "writes + reads for 'events'" --> W1
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>hashes (db, table) to one WAL node"]
|
||||
|
||||
subgraph W0["WAL node A — owns table 'frames'"]
|
||||
direction TB
|
||||
S0["Shard 0<br/>ShardWriter"]
|
||||
S1["Shard 1<br/>ShardWriter"]
|
||||
S2["Shard 2<br/>ShardWriter"]
|
||||
end
|
||||
|
||||
subgraph W1["WAL node B — owns table 'events'"]
|
||||
direction TB
|
||||
S3["Shard 0<br/>ShardWriter"]
|
||||
end
|
||||
|
||||
Q -- "writes + reads for 'frames'" --> W0
|
||||
Q -- "writes + reads for 'events'" --> W1
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
|
||||
|
||||
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
|
||||
|
||||
@@ -17,7 +17,14 @@ reports back. The query node collects those results and commits them
|
||||
together as the new table's first version; no separate coordinator
|
||||
service is involved. See [Architecture](/index#architecture).
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
||||
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
||||
@@ -32,8 +39,42 @@ flowchart LR
|
||||
PE1 --> Q
|
||||
PE2 --> Q
|
||||
PE3 --> Q
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class Q accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
||||
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
||||
PE1["Plan executor<br/>writes uncommitted files"]
|
||||
PE2["Plan executor<br/>writes uncommitted files"]
|
||||
PE3["Plan executor<br/>writes uncommitted files"]
|
||||
|
||||
S --> Q
|
||||
Q --> PE1
|
||||
Q --> PE2
|
||||
Q --> PE3
|
||||
PE1 --> Q
|
||||
PE2 --> Q
|
||||
PE3 --> Q
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class Q accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
An ordinary `INSERT` or `merge_insert`, from SQL or the SDK, does not
|
||||
distribute this way -- it runs as a single write, on whichever node or
|
||||
process executes it. Only a CTAS import fans the write itself out across
|
||||
|
||||
@@ -17,15 +17,48 @@ through the exact same placement path as
|
||||
Curation -- refreshing a column is not a separate scheduling code path
|
||||
from calling the Function directly, just a different caller.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
|
||||
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
|
||||
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
|
||||
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
|
||||
R --> P
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class P accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
|
||||
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
|
||||
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
|
||||
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
|
||||
R --> P
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class P accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
The Function's `gpu_class` (or the deployment's `default`) resolves to the
|
||||
same node selector, tolerations and resource name either way, and a
|
||||
refresh that needs more GPU capacity than is currently free waits the
|
||||
|
||||
@@ -30,7 +30,15 @@ different tables can share a group, use entirely different groups, or
|
||||
overlap partially, and each picks its own RF independently within every
|
||||
group it uses:
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TB
|
||||
subgraph G0["Replica group 0"]
|
||||
direction LR
|
||||
@@ -53,8 +61,51 @@ flowchart TB
|
||||
TA["Table A<br/>groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0
|
||||
TA -.-> G1
|
||||
TB["Table B<br/>group 2 only · RF 3"] -.-> G2
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class TA,TB accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TB
|
||||
subgraph G0["Replica group 0"]
|
||||
direction LR
|
||||
G0A["Plan executor"]
|
||||
G0B["Plan executor"]
|
||||
end
|
||||
|
||||
subgraph G1["Replica group 1"]
|
||||
direction LR
|
||||
G1A["Plan executor"]
|
||||
end
|
||||
|
||||
subgraph G2["Replica group 2"]
|
||||
direction LR
|
||||
G2A["Plan executor"]
|
||||
G2B["Plan executor"]
|
||||
G2C["Plan executor"]
|
||||
end
|
||||
|
||||
TA["Table A<br/>groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0
|
||||
TA -.-> G1
|
||||
TB["Table B<br/>group 2 only · RF 3"] -.-> G2
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class TA,TB accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
Table A is assigned to groups 0 and 1, at RF 2 and RF 1 respectively --
|
||||
three plan executors total hold a copy of Table A. Table B never touches
|
||||
groups 0 or 1 at all; it is assigned only to group 2, at RF 3. Group 2's
|
||||
@@ -81,7 +132,15 @@ no healthy plan executor. Keeping one query's reads co-located protects the
|
||||
locality of whichever group's cache is already warm for it, rather than
|
||||
letting each table's read drift to a different group.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>routes to the lead group"]
|
||||
|
||||
@@ -101,8 +160,48 @@ flowchart LR
|
||||
Q --> A1
|
||||
Q -. "only if group 0 has no healthy PE" .-> B0
|
||||
Q -. "only if group 0 has no healthy PE" .-> B1
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class Q accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>routes to the lead group"]
|
||||
|
||||
subgraph G0["Replica group 0 (lead)"]
|
||||
direction LR
|
||||
A0["Plan executor<br/>copy 1"]
|
||||
A1["Plan executor<br/>copy 2"]
|
||||
end
|
||||
|
||||
subgraph G1["Replica group 1 (fallback)"]
|
||||
direction LR
|
||||
B0["Plan executor<br/>copy 1"]
|
||||
B1["Plan executor<br/>copy 2"]
|
||||
end
|
||||
|
||||
Q --> A0
|
||||
Q --> A1
|
||||
Q -. "only if group 0 has no healthy PE" .-> B0
|
||||
Q -. "only if group 0 has no healthy PE" .-> B1
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class Q accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
This table is assigned to two replica groups, each at replication factor
|
||||
2: four plan executors in total, each holding a full copy of the table's
|
||||
data. Adding a third group, or raising RF within a group, adds more
|
||||
|
||||
@@ -17,7 +17,14 @@ needs to run, continuously, per table, in the background across your
|
||||
cluster -- nothing to schedule, trigger, or monitor by hand unless you want
|
||||
to.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
E["Table event notifications"] --> P
|
||||
S["Periodic sweep<br/>every table, on a timer"] --> P
|
||||
@@ -27,8 +34,37 @@ flowchart LR
|
||||
P --> C["Possible next step:<br/>Compaction"]
|
||||
P --> CL["Possible next step:<br/>Cleanup"]
|
||||
P --> R["Possible next step:<br/>Remap catch-up"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class P accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
E["Table event notifications"] --> P
|
||||
S["Periodic sweep<br/>every table, on a timer"] --> P
|
||||
P["Planner<br/>evaluates one table"]
|
||||
P --> W["Possible next step:<br/>WAL index catch-up"]
|
||||
P --> I["Possible next step:<br/>Index operations"]
|
||||
P --> C["Possible next step:<br/>Compaction"]
|
||||
P --> CL["Possible next step:<br/>Cleanup"]
|
||||
P --> R["Possible next step:<br/>Remap catch-up"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class P accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Tracking activity {#tracking}
|
||||
|
||||
An agent's long-running process watches two independent sources for a
|
||||
|
||||
@@ -76,14 +76,42 @@ instead. Either way, each pass:
|
||||
`memwal_catchup_ivfpq`, and so on) -- over the rows step 5 just added,
|
||||
and records how far that index has caught up.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
A["Shard crosses a trigger<br/>(count, bytes, or generation age)"] --> B["Per-pod trigger loop<br/>or backpressure relief pass"]
|
||||
B --> C["Compactor pool<br/>dedup by PK, merge_insert into base"]
|
||||
C --> D["Manifest drops<br/>the folded SSTables"]
|
||||
D --> E["memwal_catchup_<type><br/>extends each base index"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
A["Shard crosses a trigger<br/>(count, bytes, or generation age)"] --> B["Per-pod trigger loop<br/>or backpressure relief pass"]
|
||||
B --> C["Compactor pool<br/>dedup by PK, merge_insert into base"]
|
||||
C --> D["Manifest drops<br/>the folded SSTables"]
|
||||
D --> E["memwal_catchup_<type><br/>extends each base index"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
Only one compaction pass runs against a given shard at a time. A second,
|
||||
concurrent pass against the same shard would corrupt it: a stale pass can
|
||||
overwrite rows a newer generation already replaced, or resurrect a row a
|
||||
@@ -95,15 +123,48 @@ on the same collision, and node drain waits for the shard's current pass
|
||||
instead of skipping it, since a skipped table is exactly what drain exists
|
||||
to flush.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
Shard["Shard's per-shard latch"] --> Held{"Already held by<br/>an in-progress pass?"}
|
||||
Held -->|no| Grant["Grant the latch,<br/>run the pass"]
|
||||
Held -->|yes, periodic trigger| Skip["Skip this tick --<br/>retried next tick"]
|
||||
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,<br/>never queued"]
|
||||
Held -->|yes, node drain| Wait["Wait for the current<br/>pass to finish, don't skip"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class Shard accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart TD
|
||||
Shard["Shard's per-shard latch"] --> Held{"Already held by<br/>an in-progress pass?"}
|
||||
Held -->|no| Grant["Grant the latch,<br/>run the pass"]
|
||||
Held -->|yes, periodic trigger| Skip["Skip this tick --<br/>retried next tick"]
|
||||
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,<br/>never queued"]
|
||||
Held -->|yes, node drain| Wait["Wait for the current<br/>pass to finish, don't skip"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class Shard accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Reading while compaction runs {#reads}
|
||||
|
||||
A query against an RTE-enabled table always reads the shard's current
|
||||
|
||||
@@ -16,7 +16,14 @@ into memory. This lets a plan executor serve an index larger than its
|
||||
available memory without re-reading and re-deserializing it from object
|
||||
storage on every eviction. See [Architecture](/index#architecture).
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
R["Index lookup"] --> M{"In memory?"}
|
||||
M -->|hit| Return["Served from memory"]
|
||||
@@ -26,8 +33,33 @@ flowchart LR
|
||||
O --> Fill["Deserialized,<br/>loads into memory,<br/>served"]
|
||||
|
||||
Evict["Memory eviction"] -.->|serialize, don't drop| D
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
R["Index lookup"] --> M{"In memory?"}
|
||||
M -->|hit| Return["Served from memory"]
|
||||
M -->|miss| D{"Spilled to disk?"}
|
||||
D -->|hit| Promote["Deserialized,<br/>promoted into memory,<br/>served"]
|
||||
D -->|miss| O[("Object storage")]
|
||||
O --> Fill["Deserialized,<br/>loads into memory,<br/>served"]
|
||||
|
||||
Evict["Memory eviction"] -.->|serialize, don't drop| D
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Sharing the disk budget {#disk-budget}
|
||||
|
||||
Where enabled, the index disk tier and the
|
||||
|
||||
@@ -14,7 +14,14 @@ privilege that was checked, and the object it was checked against -- can
|
||||
be recorded as a durable audit log entry. On a failed check, the first
|
||||
failure is logged; on success, the final successful check is logged.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
R["Request arrives"] --> C["Authorization check"]
|
||||
C -->|denied| L1["Log the first failed check"]
|
||||
@@ -22,8 +29,35 @@ flowchart LR
|
||||
L1 --> T[("__audit_log table<br/>system database")]
|
||||
L2 --> T
|
||||
T --> Q["SELECT ... FROM system.public.__audit_log"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class T accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
R["Request arrives"] --> C["Authorization check"]
|
||||
C -->|denied| L1["Log the first failed check"]
|
||||
C -->|granted| L2["Log the final successful check"]
|
||||
L1 --> T[("__audit_log table<br/>system database")]
|
||||
L2 --> T
|
||||
T --> Q["SELECT ... FROM system.public.__audit_log"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class T accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
A deployment chooses how audit records are kept: discarded, written
|
||||
alongside ordinary application logs, or stored durably as rows in a Lance
|
||||
table. Only the durable Lance-backed sink produces something you can
|
||||
|
||||
@@ -18,7 +18,14 @@ serve every other query, with the same data page cache in front of object
|
||||
storage. No training-specific configuration turns this on; it's a
|
||||
consequence of connecting to Enterprise at all.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
DL["PyTorch DataLoader<br/>/ StreamingDataset"] --> T["Enterprise-connected table"]
|
||||
T --> PE["Plan executor"]
|
||||
@@ -26,8 +33,35 @@ flowchart LR
|
||||
C -->|hit| Fast["Served from memory<br/>or local disk"]
|
||||
C -->|miss| O[("Object storage")]
|
||||
O --> Fill["Fills the cache,<br/>served"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
||||
class Fast accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
DL["PyTorch DataLoader<br/>/ StreamingDataset"] --> T["Enterprise-connected table"]
|
||||
T --> PE["Plan executor"]
|
||||
PE --> C{"In data page cache?"}
|
||||
C -->|hit| Fast["Served from memory<br/>or local disk"]
|
||||
C -->|miss| O[("Object storage")]
|
||||
O --> Fill["Fills the cache,<br/>served"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class Fast accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
This covers the ordinary tabular columns a query reads -- the data page
|
||||
cache's unit is a fixed-size page of table data. A [Blob v2](/management/table#managed-blobs)
|
||||
column's payload is fetched through a separate range-request path keyed
|
||||
|
||||
Reference in New Issue
Block a user