style: re-theme Mermaid diagrams to the brand palette, light and dark (#4822)

## Summary

Every Mermaid diagram now renders in the brand palette, with separate
light and dark variants that switch with the site's theme toggle.
Default node styling stays neutral; one accent color marks each
diagram's actual subject, chosen per diagram. A couple of diagrams
changed orientation where their old layout was too wide to read
comfortably.

Companion change in lancedb/sophon uses the same colors and pattern.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Jack Ye
2026-11-03 21:02:24 -08:00
committed by GitHub
co-authored by Claude Sonnet 5
parent b09735529b
commit dfbf682cb5
19 changed files with 828 additions and 3 deletions
+32
View File
@@ -47,7 +47,14 @@ For example, a search application can keep production and staging data in
separate namespaces within the same database, with `prod` further split into
a nested `search` namespace:
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
catalog["root catalog"] --> database["ai_data · database"]
database --> prod["prod · namespace"]
@@ -57,8 +64,33 @@ flowchart TD
search --> search_view["recent_documents · view"]
search --> search_fn["embed · function"]
staging --> staging_docs["documents · table"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
catalog["root catalog"] --> database["ai_data · database"]
database --> prod["prod · namespace"]
database --> staging["staging · namespace"]
prod --> search["search · namespace"]
search --> search_docs["documents · table"]
search --> search_view["recent_documents · view"]
search --> search_fn["embed · function"]
staging --> staging_docs["documents · table"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
| Object | What it represents |
| --- | --- |
| [Catalog](/management/catalog) | The unnamed root of an Enterprise deployment, identified by its endpoint; lists, creates and drops databases. Not present in Embedded. |
@@ -19,7 +19,15 @@ the join has already settled which rows survive.
## The two plan shapes {#plan-shapes}
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TB
subgraph Naive["Naive: wide column read before filtering"]
direction TB
@@ -32,8 +40,41 @@ flowchart TB
D1["Scan only the join key<br/>(+ early columns)"] --> D2["Join on the key"]
D2 --> D3["RemoteTake:<br/>fetch the wide column,<br/>once per surviving row"]
end
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class D3 accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TB
subgraph Naive["Naive: wide column read before filtering"]
direction TB
N1["Scan wide column<br/>for every row"] --> N2["Join"]
N2 --> N3["Rows that didn't match<br/>are discarded --<br/>their wide read was wasted"]
end
subgraph Deferred["Late materialization: wide column read after filtering"]
direction TB
D1["Scan only the join key<br/>(+ early columns)"] --> D2["Join on the key"]
D2 --> D3["RemoteTake:<br/>fetch the wide column,<br/>once per surviving row"]
end
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class D3 accent
```
</div>
In the naive plan, the wide column is read once per **candidate** row --
every row either side of the join could possibly produce -- before the
join has thrown any of them away. In the rewritten plan, the join
+48
View File
@@ -18,7 +18,15 @@ plan needs data repartitioned across machines -- a hash join or a grouped
aggregate, for example, where rows with the same key must land on the
same worker regardless of which plan executor originally read them.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
subgraph Stage1["Stage: scan"]
direction TB
@@ -38,8 +46,48 @@ flowchart LR
PE2 --> EX
EX --> PE3
EX --> PE4
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class EX accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
subgraph Stage1["Stage: scan"]
direction TB
PE1["Plan executor<br/>reads its partition"]
PE2["Plan executor<br/>reads its partition"]
end
EX["Exchange<br/>streaming gRPC,<br/>repartitions by join/group key,<br/>backpressure via a byte budget"]
subgraph Stage2["Stage: join / aggregate"]
direction TB
PE3["Plan executor<br/>probes its share of keys"]
PE4["Plan executor<br/>probes its share of keys"]
end
PE1 --> EX
PE2 --> EX
EX --> PE3
EX --> PE4
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class EX accent
```
</div>
## Pipelined, not batched {#pipelined}
The exchange moves data as a gRPC stream of row batches, not a file a
@@ -38,7 +38,14 @@ memory, out-of-memory events, and for a GPU Function, compute and memory
utilization per GPU -- and that measurement decides whether the *next*
invocation runs with more resources, fewer, or the same:
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
I["Invocation runs<br/>wall time, CPU, memory, GPU measured"] --> D{"Saturated?"}
D -->|"OOM, or memory/CPU/GPU<br/>over target utilization"| P["Promote:<br/>next size up"]
@@ -48,8 +55,33 @@ flowchart LR
Dm --> N
S --> N
N --> I
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
I["Invocation runs<br/>wall time, CPU, memory, GPU measured"] --> D{"Saturated?"}
D -->|"OOM, or memory/CPU/GPU<br/>over target utilization"| P["Promote:<br/>next size up"]
D -->|"fast task, low CPU,<br/>known memory"| Dm["Demote:<br/>one size down,<br/>never past a floor"]
D -->|neither| S["Stay:<br/>same class"]
P --> N["Next invocation"]
Dm --> N
S --> N
N --> I
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
**Promotion** happens on an out-of-memory event, memory utilization above
70% of the class's limit, or CPU pegged at 85%+ for a sample long enough
(5 seconds) that one fast spike isn't mistaken for sustained saturation. A
@@ -65,7 +65,14 @@ Once a table is tagged, Lance refuses a compaction that does not set
Enterprise-tagged table has to keep deferring the remap, or be compacted
by a writer that can remap the tagged history eagerly.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
Commit["Compaction commit<br/>(defer_index_remap only)"] --> Tagged{"Table already<br/>tagged with an FRI?"}
Tagged -->|no| First{"Stable row IDs, or does<br/>any index cover the<br/>rewritten fragments?"}
@@ -76,8 +83,38 @@ flowchart TD
Classify -->|yes| Append["Must append a tagged<br/>transition before the<br/>commit is admitted"]
V0 --> FRI[("Fragment reuse index")]
Append --> FRI
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class FRI accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
Commit["Compaction commit<br/>(defer_index_remap only)"] --> Tagged{"Table already<br/>tagged with an FRI?"}
Tagged -->|no| First{"Stable row IDs, or does<br/>any index cover the<br/>rewritten fragments?"}
First -->|no| Admit["Admitted as<br/>ordinary business --<br/>nothing appended"]
First -->|yes| V0["Records a legacy v0 entry --<br/>table stays untagged"]
Tagged -->|yes| Classify{"Does any index<br/>cover the rewritten<br/>fragments?"}
Classify -->|no| Admit
Classify -->|yes| Append["Must append a tagged<br/>transition before the<br/>commit is admitted"]
V0 --> FRI[("Fragment reuse index")]
Append --> FRI
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class FRI accent
```
</div>
## Enterprise defers and batches the catch-up {#catch-up}
Translating addresses through a long chain of transitions at every read adds
@@ -102,15 +139,48 @@ rather than one blocking on the other: a handful of compactions can land
before their indexes are physically remapped, and the catch-up job amortizes
that remap across all of them in one job instead of one per compaction.
<div className="hidden dark:block">
```mermaid
flowchart LR
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index<br/>(ledger of transitions)"]
C2["Compaction job"] -->|appends transition| FRI
C3["Compaction job"] -->|appends transition| FRI
FRI -->|5 transitions or 1 day old| R["Remap catch-up job"]
R -->|applies accumulated transitions| IDX["Affected indexes,<br/>physically remapped"]
FRI -->|5 transitions or<br/>1 day old| R["Remap catch-up job"]
R -->|applies accumulated<br/>transitions| IDX["Affected indexes,<br/>physically remapped"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class FRI,R accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index<br/>(ledger of transitions)"]
C2["Compaction job"] -->|appends transition| FRI
C3["Compaction job"] -->|appends transition| FRI
FRI -->|5 transitions or<br/>1 day old| R["Remap catch-up job"]
R -->|applies accumulated<br/>transitions| IDX["Affected indexes,<br/>physically remapped"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class FRI,R accent
```
</div>
## What to read next {#next}
<CardGroup cols={2}>
+36
View File
@@ -24,7 +24,14 @@ that started but never recorded completion is redone in full on the next
attempt, so a Function with external side effects (a remote API call, for
example) can have those side effects repeated for that part's rows.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
S["Part started<br/>event recorded"] --> E["Function executes"]
E --> C["Part completed<br/>event recorded"]
@@ -34,8 +41,37 @@ flowchart LR
S -. "worker dies here" .-> R["Next attempt reads the log,<br/>sees no completion,<br/>redoes only this part"]
C -. "worker dies here" .-> R2["Next attempt reads the log,<br/>sees this part done,<br/>inherits it, moves on"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class R,R2 accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
S["Part started<br/>event recorded"] --> E["Function executes"]
E --> C["Part completed<br/>event recorded"]
C --> M{"More parts?"}
M -->|yes| S
M -->|no| F["Finalized:<br/>digest-verified assembly<br/>into the real data file"]
S -. "worker dies here" .-> R["Next attempt reads the log,<br/>sees no completion,<br/>redoes only this part"]
C -. "worker dies here" .-> R2["Next attempt reads the log,<br/>sees this part done,<br/>inherits it, moves on"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class R,R2 accent
```
</div>
## What gets recorded, and when {#events}
Each part contributes two events to its data file's log: one when a
+30
View File
@@ -16,7 +16,14 @@ the pages they serve. A background prewarm job can populate a cache ahead
of a request that needs it, rather than filling it lazily on first read.
See [Architecture](/index#architecture).
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
R["Read request<br/>for a data page"] --> M{"In memory?"}
M -->|hit| Return["Served from memory"]
@@ -24,8 +31,31 @@ flowchart LR
D -->|hit| Promote["Loaded into memory,<br/>served"]
D -->|miss| O[("Object storage")]
O --> Fill["Fills memory and disk,<br/>served"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
R["Read request<br/>for a data page"] --> M{"In memory?"}
M -->|hit| Return["Served from memory"]
M -->|miss| D{"In local disk cache?"}
D -->|hit| Promote["Loaded into memory,<br/>served"]
D -->|miss| O[("Object storage")]
O --> Fill["Fills memory and disk,<br/>served"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
A table's local disk cache shares one budget with the
[tiered index cache](/enterprise/tiered-index-cache#disk-budget)'s disk
tier where both are enabled on the same executor -- raising one's share of
@@ -15,7 +15,14 @@ Lance's own planner already produces disjoint tasks, so workers claim and
execute them directly through the job registry, with no coordination
needed to decide who does what. See [Architecture](/index#architecture).
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
R --> W1["Worker<br/>claims fragments 1-2"]
@@ -24,8 +31,36 @@ flowchart LR
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
W2 --> C
W3 --> C
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class R,C accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
R --> W1["Worker<br/>claims fragments 1-2"]
R --> W2["Worker<br/>claims fragments 3-4"]
R --> W3["Worker<br/>claims fragments 5-6"]
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
W2 --> C
W3 --> C
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class R,C accent
```
</div>
No task depends on another's result, so a worker that finishes early
simply claims another task rather than waiting on its peers. Committing is
where the fleet does coordinate: finished tasks go through a shared
@@ -15,7 +15,14 @@ registry and commit the finished index together. A vector index build first
trains a shared IVF/PQ model from sampled data, then workers build storage
from that model in parallel. See [Architecture](/index#architecture).
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
T --> P["Bin-pack build plan<br/>into claimable tasks"]
@@ -26,8 +33,38 @@ flowchart LR
W1 --> C["Commit the<br/>finished index"]
W2 --> C
W3 --> C
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class R accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
T --> P["Bin-pack build plan<br/>into claimable tasks"]
P --> R[("Job registry")]
R --> W1["Worker<br/>claims a task"]
R --> W2["Worker<br/>claims a task"]
R --> W3["Worker<br/>claims a task"]
W1 --> C["Commit the<br/>finished index"]
W2 --> C
W3 --> C
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class R accent
```
</div>
## Indexing interfaces {#indexing-interfaces}
The shared indexing guides describe how to build and maintain each index
@@ -17,7 +17,15 @@ usually served by different machines -- plan executors for the base table,
a [WAL node](/tables/realtime-engine#wal-nodes) for the table's shards --
so a query has to combine their results across the network instead.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
Q["Query node<br/>plans the query"]
@@ -41,8 +49,52 @@ flowchart LR
W1 --> U
W2 --> U
U --> R["Merged result"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class U accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
Q["Query node<br/>plans the query"]
subgraph PE["Plan executors"]
direction TB
B1["Base-table arm<br/>index-covered scan or ANN search"]
end
subgraph WN["WAL node (owns this table's shards)"]
direction TB
W1["SSTable arm<br/>index-covered"]
W2["Active memtable arm<br/>vector: in-memory HNSW*<br/>FTS/scalar: indexed"]
end
U["Union exec<br/>PK-keyed last-write-wins dedup,<br/>top-K cut on distance"]
Q --> B1
Q --> W1
Q --> W2
B1 --> U
W1 --> U
W2 --> U
U --> R["Merged result"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class U accent
```
</div>
## An extra exec node, not a parallel system {#union-exec}
The distributed plan for an RTE-enabled table is an ordinary plan executor
@@ -26,7 +26,15 @@ single owning **WAL node**, a stateful pod separate from the query nodes and
plan executors that serve ordinary queries. WAL node pods share nothing but
the underlying object storage; no pod talks to another one directly.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
Q["Query node<br/>hashes (db, table) to one WAL node"]
@@ -44,8 +52,42 @@ flowchart LR
Q -- "writes + reads for 'frames'" --> W0
Q -- "writes + reads for 'events'" --> W1
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
Q["Query node<br/>hashes (db, table) to one WAL node"]
subgraph W0["WAL node A — owns table 'frames'"]
direction TB
S0["Shard 0<br/>ShardWriter"]
S1["Shard 1<br/>ShardWriter"]
S2["Shard 2<br/>ShardWriter"]
end
subgraph W1["WAL node B — owns table 'events'"]
direction TB
S3["Shard 0<br/>ShardWriter"]
end
Q -- "writes + reads for 'frames'" --> W0
Q -- "writes + reads for 'events'" --> W1
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
+41
View File
@@ -17,7 +17,14 @@ reports back. The query node collects those results and commits them
together as the new table's first version; no separate coordinator
service is involved. See [Architecture](/index#architecture).
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
@@ -32,8 +39,42 @@ flowchart LR
PE1 --> Q
PE2 --> Q
PE3 --> Q
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class Q accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
PE1["Plan executor<br/>writes uncommitted files"]
PE2["Plan executor<br/>writes uncommitted files"]
PE3["Plan executor<br/>writes uncommitted files"]
S --> Q
Q --> PE1
Q --> PE2
Q --> PE3
PE1 --> Q
PE2 --> Q
PE3 --> Q
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Q accent
```
</div>
An ordinary `INSERT` or `merge_insert`, from SQL or the SDK, does not
distribute this way -- it runs as a single write, on whichever node or
process executes it. Only a CTAS import fans the write itself out across
@@ -17,15 +17,48 @@ through the exact same placement path as
Curation -- refreshing a column is not a separate scheduling code path
from calling the Function directly, just a different caller.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
R --> P
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class P accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
R --> P
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class P accent
```
</div>
The Function's `gpu_class` (or the deployment's `default`) resolves to the
same node selector, tolerations and resource name either way, and a
refresh that needs more GPU capacity than is currently free waits the
@@ -30,7 +30,15 @@ different tables can share a group, use entirely different groups, or
overlap partially, and each picks its own RF independently within every
group it uses:
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TB
subgraph G0["Replica group 0"]
direction LR
@@ -53,8 +61,51 @@ flowchart TB
TA["Table A<br/>groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0
TA -.-> G1
TB["Table B<br/>group 2 only · RF 3"] -.-> G2
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class TA,TB accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TB
subgraph G0["Replica group 0"]
direction LR
G0A["Plan executor"]
G0B["Plan executor"]
end
subgraph G1["Replica group 1"]
direction LR
G1A["Plan executor"]
end
subgraph G2["Replica group 2"]
direction LR
G2A["Plan executor"]
G2B["Plan executor"]
G2C["Plan executor"]
end
TA["Table A<br/>groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0
TA -.-> G1
TB["Table B<br/>group 2 only · RF 3"] -.-> G2
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class TA,TB accent
```
</div>
Table A is assigned to groups 0 and 1, at RF 2 and RF 1 respectively --
three plan executors total hold a copy of Table A. Table B never touches
groups 0 or 1 at all; it is assigned only to group 2, at RF 3. Group 2's
@@ -81,7 +132,15 @@ no healthy plan executor. Keeping one query's reads co-located protects the
locality of whichever group's cache is already warm for it, rather than
letting each table's read drift to a different group.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
Q["Query node<br/>routes to the lead group"]
@@ -101,8 +160,48 @@ flowchart LR
Q --> A1
Q -. "only if group 0 has no healthy PE" .-> B0
Q -. "only if group 0 has no healthy PE" .-> B1
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class Q accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
Q["Query node<br/>routes to the lead group"]
subgraph G0["Replica group 0 (lead)"]
direction LR
A0["Plan executor<br/>copy 1"]
A1["Plan executor<br/>copy 2"]
end
subgraph G1["Replica group 1 (fallback)"]
direction LR
B0["Plan executor<br/>copy 1"]
B1["Plan executor<br/>copy 2"]
end
Q --> A0
Q --> A1
Q -. "only if group 0 has no healthy PE" .-> B0
Q -. "only if group 0 has no healthy PE" .-> B1
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Q accent
```
</div>
This table is assigned to two replica groups, each at replication factor
2: four plan executors in total, each holding a full copy of the table's
data. Adding a third group, or raising RF within a group, adds more
@@ -17,7 +17,14 @@ needs to run, continuously, per table, in the background across your
cluster -- nothing to schedule, trigger, or monitor by hand unless you want
to.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
E["Table event notifications"] --> P
S["Periodic sweep<br/>every table, on a timer"] --> P
@@ -27,8 +34,37 @@ flowchart LR
P --> C["Possible next step:<br/>Compaction"]
P --> CL["Possible next step:<br/>Cleanup"]
P --> R["Possible next step:<br/>Remap catch-up"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class P accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
E["Table event notifications"] --> P
S["Periodic sweep<br/>every table, on a timer"] --> P
P["Planner<br/>evaluates one table"]
P --> W["Possible next step:<br/>WAL index catch-up"]
P --> I["Possible next step:<br/>Index operations"]
P --> C["Possible next step:<br/>Compaction"]
P --> CL["Possible next step:<br/>Cleanup"]
P --> R["Possible next step:<br/>Remap catch-up"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class P accent
```
</div>
## Tracking activity {#tracking}
An agent's long-running process watches two independent sources for a
@@ -76,14 +76,42 @@ instead. Either way, each pass:
`memwal_catchup_ivfpq`, and so on) -- over the rows step 5 just added,
and records how far that index has caught up.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
A["Shard crosses a trigger<br/>(count, bytes, or generation age)"] --> B["Per-pod trigger loop<br/>or backpressure relief pass"]
B --> C["Compactor pool<br/>dedup by PK, merge_insert into base"]
C --> D["Manifest drops<br/>the folded SSTables"]
D --> E["memwal_catchup_&lt;type&gt;<br/>extends each base index"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
A["Shard crosses a trigger<br/>(count, bytes, or generation age)"] --> B["Per-pod trigger loop<br/>or backpressure relief pass"]
B --> C["Compactor pool<br/>dedup by PK, merge_insert into base"]
C --> D["Manifest drops<br/>the folded SSTables"]
D --> E["memwal_catchup_&lt;type&gt;<br/>extends each base index"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
Only one compaction pass runs against a given shard at a time. A second,
concurrent pass against the same shard would corrupt it: a stale pass can
overwrite rows a newer generation already replaced, or resurrect a row a
@@ -95,15 +123,48 @@ on the same collision, and node drain waits for the shard's current pass
instead of skipping it, since a skipped table is exactly what drain exists
to flush.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
Shard["Shard's per-shard latch"] --> Held{"Already held by<br/>an in-progress pass?"}
Held -->|no| Grant["Grant the latch,<br/>run the pass"]
Held -->|yes, periodic trigger| Skip["Skip this tick --<br/>retried next tick"]
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,<br/>never queued"]
Held -->|yes, node drain| Wait["Wait for the current<br/>pass to finish, don't skip"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class Shard accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart TD
Shard["Shard's per-shard latch"] --> Held{"Already held by<br/>an in-progress pass?"}
Held -->|no| Grant["Grant the latch,<br/>run the pass"]
Held -->|yes, periodic trigger| Skip["Skip this tick --<br/>retried next tick"]
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,<br/>never queued"]
Held -->|yes, node drain| Wait["Wait for the current<br/>pass to finish, don't skip"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Shard accent
```
</div>
## Reading while compaction runs {#reads}
A query against an RTE-enabled table always reads the shard's current
@@ -16,7 +16,14 @@ into memory. This lets a plan executor serve an index larger than its
available memory without re-reading and re-deserializing it from object
storage on every eviction. See [Architecture](/index#architecture).
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
R["Index lookup"] --> M{"In memory?"}
M -->|hit| Return["Served from memory"]
@@ -26,8 +33,33 @@ flowchart LR
O --> Fill["Deserialized,<br/>loads into memory,<br/>served"]
Evict["Memory eviction"] -.->|serialize, don't drop| D
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
R["Index lookup"] --> M{"In memory?"}
M -->|hit| Return["Served from memory"]
M -->|miss| D{"Spilled to disk?"}
D -->|hit| Promote["Deserialized,<br/>promoted into memory,<br/>served"]
D -->|miss| O[("Object storage")]
O --> Fill["Deserialized,<br/>loads into memory,<br/>served"]
Evict["Memory eviction"] -.->|serialize, don't drop| D
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
## Sharing the disk budget {#disk-budget}
Where enabled, the index disk tier and the
+34
View File
@@ -14,7 +14,14 @@ privilege that was checked, and the object it was checked against -- can
be recorded as a durable audit log entry. On a failed check, the first
failure is logged; on success, the final successful check is logged.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
R["Request arrives"] --> C["Authorization check"]
C -->|denied| L1["Log the first failed check"]
@@ -22,8 +29,35 @@ flowchart LR
L1 --> T[("__audit_log table<br/>system database")]
L2 --> T
T --> Q["SELECT ... FROM system.public.__audit_log"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class T accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
R["Request arrives"] --> C["Authorization check"]
C -->|denied| L1["Log the first failed check"]
C -->|granted| L2["Log the final successful check"]
L1 --> T[("__audit_log table<br/>system database")]
L2 --> T
T --> Q["SELECT ... FROM system.public.__audit_log"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class T accent
```
</div>
A deployment chooses how audit records are kept: discarded, written
alongside ordinary application logs, or stored durably as rows in a Lance
table. Only the durable Lance-backed sink produces something you can
+34
View File
@@ -18,7 +18,14 @@ serve every other query, with the same data page cache in front of object
storage. No training-specific configuration turns this on; it's a
consequence of connecting to Enterprise at all.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
DL["PyTorch DataLoader<br/>/ StreamingDataset"] --> T["Enterprise-connected table"]
T --> PE["Plan executor"]
@@ -26,8 +33,35 @@ flowchart LR
C -->|hit| Fast["Served from memory<br/>or local disk"]
C -->|miss| O[("Object storage")]
O --> Fill["Fills the cache,<br/>served"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
class Fast accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
}}}%%
flowchart LR
DL["PyTorch DataLoader<br/>/ StreamingDataset"] --> T["Enterprise-connected table"]
T --> PE["Plan executor"]
PE --> C{"In data page cache?"}
C -->|hit| Fast["Served from memory<br/>or local disk"]
C -->|miss| O[("Object storage")]
O --> Fill["Fills the cache,<br/>served"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Fast accent
```
</div>
This covers the ordinary tabular columns a query reads -- the data page
cache's unit is a fixed-size page of table data. A [Blob v2](/management/table#managed-blobs)
column's payload is fetched through a separate range-request path keyed