+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart TD
catalog["root catalog"] --> database["ai_data · database"]
database --> prod["prod · namespace"]
@@ -57,8 +64,33 @@ flowchart TD
search --> search_view["recent_documents · view"]
search --> search_fn["embed · function"]
staging --> staging_docs["documents · table"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart TD
+ catalog["root catalog"] --> database["ai_data · database"]
+ database --> prod["prod · namespace"]
+ database --> staging["staging · namespace"]
+ prod --> search["search · namespace"]
+ search --> search_docs["documents · table"]
+ search --> search_view["recent_documents · view"]
+ search --> search_fn["embed · function"]
+ staging --> staging_docs["documents · table"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+```
+
+
+
| Object | What it represents |
| --- | --- |
| [Catalog](/management/catalog) | The unnamed root of an Enterprise deployment, identified by its endpoint; lists, creates and drops databases. Not present in Embedded. |
diff --git a/docs/web/curation/blob-late-materialization.mdx b/docs/web/curation/blob-late-materialization.mdx
index d1213b64a..86aff0e54 100644
--- a/docs/web/curation/blob-late-materialization.mdx
+++ b/docs/web/curation/blob-late-materialization.mdx
@@ -19,7 +19,15 @@ the join has already settled which rows survive.
## The two plan shapes {#plan-shapes}
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart TB
subgraph Naive["Naive: wide column read before filtering"]
direction TB
@@ -32,8 +40,41 @@ flowchart TB
D1["Scan only the join key
(+ early columns)"] --> D2["Join on the key"]
D2 --> D3["RemoteTake:
fetch the wide column,
once per surviving row"]
end
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class D3 accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart TB
+ subgraph Naive["Naive: wide column read before filtering"]
+ direction TB
+ N1["Scan wide column
for every row"] --> N2["Join"]
+ N2 --> N3["Rows that didn't match
are discarded --
their wide read was wasted"]
+ end
+
+ subgraph Deferred["Late materialization: wide column read after filtering"]
+ direction TB
+ D1["Scan only the join key
(+ early columns)"] --> D2["Join on the key"]
+ D2 --> D3["RemoteTake:
fetch the wide column,
once per surviving row"]
+ end
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class D3 accent
+```
+
+
+
In the naive plan, the wide column is read once per **candidate** row --
every row either side of the join could possibly produce -- before the
join has thrown any of them away. In the rewritten plan, the join
diff --git a/docs/web/curation/distributed-shuffle.mdx b/docs/web/curation/distributed-shuffle.mdx
index c8f7d1dd2..1a889f0d3 100644
--- a/docs/web/curation/distributed-shuffle.mdx
+++ b/docs/web/curation/distributed-shuffle.mdx
@@ -18,7 +18,15 @@ plan needs data repartitioned across machines -- a hash join or a grouped
aggregate, for example, where rows with the same key must land on the
same worker regardless of which plan executor originally read them.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
subgraph Stage1["Stage: scan"]
direction TB
@@ -38,8 +46,48 @@ flowchart LR
PE2 --> EX
EX --> PE3
EX --> PE4
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class EX accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ subgraph Stage1["Stage: scan"]
+ direction TB
+ PE1["Plan executor
reads its partition"]
+ PE2["Plan executor
reads its partition"]
+ end
+
+ EX["Exchange
streaming gRPC,
repartitions by join/group key,
backpressure via a byte budget"]
+
+ subgraph Stage2["Stage: join / aggregate"]
+ direction TB
+ PE3["Plan executor
probes its share of keys"]
+ PE4["Plan executor
probes its share of keys"]
+ end
+
+ PE1 --> EX
+ PE2 --> EX
+ EX --> PE3
+ EX --> PE4
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class EX accent
+```
+
+
+
## Pipelined, not batched {#pipelined}
The exchange moves data as a gRPC stream of row batches, not a file a
diff --git a/docs/web/enterprise/adaptive-execution.mdx b/docs/web/enterprise/adaptive-execution.mdx
index d83af371f..4b51bf59c 100644
--- a/docs/web/enterprise/adaptive-execution.mdx
+++ b/docs/web/enterprise/adaptive-execution.mdx
@@ -38,7 +38,14 @@ memory, out-of-memory events, and for a GPU Function, compute and memory
utilization per GPU -- and that measurement decides whether the *next*
invocation runs with more resources, fewer, or the same:
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
I["Invocation runs
wall time, CPU, memory, GPU measured"] --> D{"Saturated?"}
D -->|"OOM, or memory/CPU/GPU
over target utilization"| P["Promote:
next size up"]
@@ -48,8 +55,33 @@ flowchart LR
Dm --> N
S --> N
N --> I
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ I["Invocation runs
wall time, CPU, memory, GPU measured"] --> D{"Saturated?"}
+ D -->|"OOM, or memory/CPU/GPU
over target utilization"| P["Promote:
next size up"]
+ D -->|"fast task, low CPU,
known memory"| Dm["Demote:
one size down,
never past a floor"]
+ D -->|neither| S["Stay:
same class"]
+ P --> N["Next invocation"]
+ Dm --> N
+ S --> N
+ N --> I
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+```
+
+
+
**Promotion** happens on an out-of-memory event, memory utilization above
70% of the class's limit, or CPU pegged at 85%+ for a sample long enough
(5 seconds) that one fast spike isn't mistaken for sustained saturation. A
diff --git a/docs/web/enterprise/advanced-conflict-resolution.mdx b/docs/web/enterprise/advanced-conflict-resolution.mdx
index 7139423b9..3b3964bb5 100644
--- a/docs/web/enterprise/advanced-conflict-resolution.mdx
+++ b/docs/web/enterprise/advanced-conflict-resolution.mdx
@@ -65,7 +65,14 @@ Once a table is tagged, Lance refuses a compaction that does not set
Enterprise-tagged table has to keep deferring the remap, or be compacted
by a writer that can remap the tagged history eagerly.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart TD
Commit["Compaction commit
(defer_index_remap only)"] --> Tagged{"Table already
tagged with an FRI?"}
Tagged -->|no| First{"Stable row IDs, or does
any index cover the
rewritten fragments?"}
@@ -76,8 +83,38 @@ flowchart TD
Classify -->|yes| Append["Must append a tagged
transition before the
commit is admitted"]
V0 --> FRI[("Fragment reuse index")]
Append --> FRI
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class FRI accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart TD
+ Commit["Compaction commit
(defer_index_remap only)"] --> Tagged{"Table already
tagged with an FRI?"}
+ Tagged -->|no| First{"Stable row IDs, or does
any index cover the
rewritten fragments?"}
+ First -->|no| Admit["Admitted as
ordinary business --
nothing appended"]
+ First -->|yes| V0["Records a legacy v0 entry --
table stays untagged"]
+ Tagged -->|yes| Classify{"Does any index
cover the rewritten
fragments?"}
+ Classify -->|no| Admit
+ Classify -->|yes| Append["Must append a tagged
transition before the
commit is admitted"]
+ V0 --> FRI[("Fragment reuse index")]
+ Append --> FRI
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class FRI accent
+```
+
+
+
## Enterprise defers and batches the catch-up {#catch-up}
Translating addresses through a long chain of transitions at every read adds
@@ -102,15 +139,48 @@ rather than one blocking on the other: a handful of compactions can land
before their indexes are physically remapped, and the catch-up job amortizes
that remap across all of them in one job instead of one per compaction.
+
+
```mermaid
-flowchart LR
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart TD
C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index
(ledger of transitions)"]
C2["Compaction job"] -->|appends transition| FRI
C3["Compaction job"] -->|appends transition| FRI
- FRI -->|5 transitions or 1 day old| R["Remap catch-up job"]
- R -->|applies accumulated transitions| IDX["Affected indexes,
physically remapped"]
+ FRI -->|5 transitions or
1 day old| R["Remap catch-up job"]
+ R -->|applies accumulated
transitions| IDX["Affected indexes,
physically remapped"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class FRI,R accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart TD
+ C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index
(ledger of transitions)"]
+ C2["Compaction job"] -->|appends transition| FRI
+ C3["Compaction job"] -->|appends transition| FRI
+ FRI -->|5 transitions or
1 day old| R["Remap catch-up job"]
+ R -->|applies accumulated
transitions| IDX["Affected indexes,
physically remapped"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class FRI,R accent
+```
+
+
+
## What to read next {#next}
diff --git a/docs/web/enterprise/checkpointing.mdx b/docs/web/enterprise/checkpointing.mdx
index c4cd62893..5fe82bbda 100644
--- a/docs/web/enterprise/checkpointing.mdx
+++ b/docs/web/enterprise/checkpointing.mdx
@@ -24,7 +24,14 @@ that started but never recorded completion is redone in full on the next
attempt, so a Function with external side effects (a remote API call, for
example) can have those side effects repeated for that part's rows.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
S["Part started
event recorded"] --> E["Function executes"]
E --> C["Part completed
event recorded"]
@@ -34,8 +41,37 @@ flowchart LR
S -. "worker dies here" .-> R["Next attempt reads the log,
sees no completion,
redoes only this part"]
C -. "worker dies here" .-> R2["Next attempt reads the log,
sees this part done,
inherits it, moves on"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class R,R2 accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ S["Part started
event recorded"] --> E["Function executes"]
+ E --> C["Part completed
event recorded"]
+ C --> M{"More parts?"}
+ M -->|yes| S
+ M -->|no| F["Finalized:
digest-verified assembly
into the real data file"]
+
+ S -. "worker dies here" .-> R["Next attempt reads the log,
sees no completion,
redoes only this part"]
+ C -. "worker dies here" .-> R2["Next attempt reads the log,
sees this part done,
inherits it, moves on"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class R,R2 accent
+```
+
+
+
## What gets recorded, and when {#events}
Each part contributes two events to its data file's log: one when a
diff --git a/docs/web/enterprise/data-page-cache.mdx b/docs/web/enterprise/data-page-cache.mdx
index 7c385f1bd..a4bfd61ae 100644
--- a/docs/web/enterprise/data-page-cache.mdx
+++ b/docs/web/enterprise/data-page-cache.mdx
@@ -16,7 +16,14 @@ the pages they serve. A background prewarm job can populate a cache ahead
of a request that needs it, rather than filling it lazily on first read.
See [Architecture](/index#architecture).
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
R["Read request
for a data page"] --> M{"In memory?"}
M -->|hit| Return["Served from memory"]
@@ -24,8 +31,31 @@ flowchart LR
D -->|hit| Promote["Loaded into memory,
served"]
D -->|miss| O[("Object storage")]
O --> Fill["Fills memory and disk,
served"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ R["Read request
for a data page"] --> M{"In memory?"}
+ M -->|hit| Return["Served from memory"]
+ M -->|miss| D{"In local disk cache?"}
+ D -->|hit| Promote["Loaded into memory,
served"]
+ D -->|miss| O[("Object storage")]
+ O --> Fill["Fills memory and disk,
served"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+```
+
+
+
A table's local disk cache shares one budget with the
[tiered index cache](/enterprise/tiered-index-cache#disk-budget)'s disk
tier where both are enabled on the same executor -- raising one's share of
diff --git a/docs/web/enterprise/distributed-compaction.mdx b/docs/web/enterprise/distributed-compaction.mdx
index c23da4859..0c483865c 100644
--- a/docs/web/enterprise/distributed-compaction.mdx
+++ b/docs/web/enterprise/distributed-compaction.mdx
@@ -15,7 +15,14 @@ Lance's own planner already produces disjoint tasks, so workers claim and
execute them directly through the job registry, with no coordination
needed to decide who does what. See [Architecture](/index#architecture).
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
L["Lance's planner
splits fragments into
disjoint tasks"] --> R[("Job registry")]
R --> W1["Worker
claims fragments 1-2"]
@@ -24,8 +31,36 @@ flowchart LR
W1 --> C["Fleet-wide eligibility gate:
one worker claims the commit"]
W2 --> C
W3 --> C
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class R,C accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ L["Lance's planner
splits fragments into
disjoint tasks"] --> R[("Job registry")]
+ R --> W1["Worker
claims fragments 1-2"]
+ R --> W2["Worker
claims fragments 3-4"]
+ R --> W3["Worker
claims fragments 5-6"]
+ W1 --> C["Fleet-wide eligibility gate:
one worker claims the commit"]
+ W2 --> C
+ W3 --> C
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class R,C accent
+```
+
+
+
No task depends on another's result, so a worker that finishes early
simply claims another task rather than waiting on its peers. Committing is
where the fleet does coordinate: finished tasks go through a shared
diff --git a/docs/web/enterprise/distributed-indexing.mdx b/docs/web/enterprise/distributed-indexing.mdx
index fdf4715c0..b40e185df 100644
--- a/docs/web/enterprise/distributed-indexing.mdx
+++ b/docs/web/enterprise/distributed-indexing.mdx
@@ -15,7 +15,14 @@ registry and commit the finished index together. A vector index build first
trains a shared IVF/PQ model from sampled data, then workers build storage
from that model in parallel. See [Architecture](/index#architecture).
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
S["Sample data"] --> T["Train shared IVF/PQ model
(vector index only)"]
T --> P["Bin-pack build plan
into claimable tasks"]
@@ -26,8 +33,38 @@ flowchart LR
W1 --> C["Commit the
finished index"]
W2 --> C
W3 --> C
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class R accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ S["Sample data"] --> T["Train shared IVF/PQ model
(vector index only)"]
+ T --> P["Bin-pack build plan
into claimable tasks"]
+ P --> R[("Job registry")]
+ R --> W1["Worker
claims a task"]
+ R --> W2["Worker
claims a task"]
+ R --> W3["Worker
claims a task"]
+ W1 --> C["Commit the
finished index"]
+ W2 --> C
+ W3 --> C
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class R accent
+```
+
+
+
## Indexing interfaces {#indexing-interfaces}
The shared indexing guides describe how to build and maintain each index
diff --git a/docs/web/enterprise/distributed-real-time-read.mdx b/docs/web/enterprise/distributed-real-time-read.mdx
index 697679f16..2adc267e8 100644
--- a/docs/web/enterprise/distributed-real-time-read.mdx
+++ b/docs/web/enterprise/distributed-real-time-read.mdx
@@ -17,7 +17,15 @@ usually served by different machines -- plan executors for the base table,
a [WAL node](/tables/realtime-engine#wal-nodes) for the table's shards --
so a query has to combine their results across the network instead.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
Q["Query node
plans the query"]
@@ -41,8 +49,52 @@ flowchart LR
W1 --> U
W2 --> U
U --> R["Merged result"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class U accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ Q["Query node
plans the query"]
+
+ subgraph PE["Plan executors"]
+ direction TB
+ B1["Base-table arm
index-covered scan or ANN search"]
+ end
+
+ subgraph WN["WAL node (owns this table's shards)"]
+ direction TB
+ W1["SSTable arm
index-covered"]
+ W2["Active memtable arm
vector: in-memory HNSW*
FTS/scalar: indexed"]
+ end
+
+ U["Union exec
PK-keyed last-write-wins dedup,
top-K cut on distance"]
+
+ Q --> B1
+ Q --> W1
+ Q --> W2
+ B1 --> U
+ W1 --> U
+ W2 --> U
+ U --> R["Merged result"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class U accent
+```
+
+
+
## An extra exec node, not a parallel system {#union-exec}
The distributed plan for an RTE-enabled table is an ordinary plan executor
diff --git a/docs/web/enterprise/distributed-real-time-write.mdx b/docs/web/enterprise/distributed-real-time-write.mdx
index 8778ef616..b07d6a64d 100644
--- a/docs/web/enterprise/distributed-real-time-write.mdx
+++ b/docs/web/enterprise/distributed-real-time-write.mdx
@@ -26,7 +26,15 @@ single owning **WAL node**, a stateful pod separate from the query nodes and
plan executors that serve ordinary queries. WAL node pods share nothing but
the underlying object storage; no pod talks to another one directly.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
Q["Query node
hashes (db, table) to one WAL node"]
@@ -44,8 +52,42 @@ flowchart LR
Q -- "writes + reads for 'frames'" --> W0
Q -- "writes + reads for 'events'" --> W1
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
+ "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ Q["Query node
hashes (db, table) to one WAL node"]
+
+ subgraph W0["WAL node A — owns table 'frames'"]
+ direction TB
+ S0["Shard 0
ShardWriter"]
+ S1["Shard 1
ShardWriter"]
+ S2["Shard 2
ShardWriter"]
+ end
+
+ subgraph W1["WAL node B — owns table 'events'"]
+ direction TB
+ S3["Shard 0
ShardWriter"]
+ end
+
+ Q -- "writes + reads for 'frames'" --> W0
+ Q -- "writes + reads for 'events'" --> W1
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+```
+
+
+
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
diff --git a/docs/web/enterprise/distributed-write.mdx b/docs/web/enterprise/distributed-write.mdx
index 890d520b9..feda93a57 100644
--- a/docs/web/enterprise/distributed-write.mdx
+++ b/docs/web/enterprise/distributed-write.mdx
@@ -17,7 +17,14 @@ reports back. The query node collects those results and commits them
together as the new table's first version; no separate coordinator
service is involved. See [Architecture](/index#architecture).
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
S["Source
HuggingFace Parquet, or an
existing Lance dataset in
S3 / GCS / Azure"]
Q["Query node
plans the CTAS,
commits the result"]
@@ -32,8 +39,42 @@ flowchart LR
PE1 --> Q
PE2 --> Q
PE3 --> Q
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class Q accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ S["Source
HuggingFace Parquet, or an
existing Lance dataset in
S3 / GCS / Azure"]
+ Q["Query node
plans the CTAS,
commits the result"]
+ PE1["Plan executor
writes uncommitted files"]
+ PE2["Plan executor
writes uncommitted files"]
+ PE3["Plan executor
writes uncommitted files"]
+
+ S --> Q
+ Q --> PE1
+ Q --> PE2
+ Q --> PE3
+ PE1 --> Q
+ PE2 --> Q
+ PE3 --> Q
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class Q accent
+```
+
+
+
An ordinary `INSERT` or `merge_insert`, from SQL or the SDK, does not
distribute this way -- it runs as a single write, on whichever node or
process executes it. Only a CTAS import fans the write itself out across
diff --git a/docs/web/enterprise/gpu-refresh-execution.mdx b/docs/web/enterprise/gpu-refresh-execution.mdx
index 0749210f1..2c61c34f6 100644
--- a/docs/web/enterprise/gpu-refresh-execution.mdx
+++ b/docs/web/enterprise/gpu-refresh-execution.mdx
@@ -17,15 +17,48 @@ through the exact same placement path as
Curation -- refreshing a column is not a separate scheduling code path
from calling the Function directly, just a different caller.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
F["Function registered with
gpu=True, gpu_class"] --> A{"Who requests
a worker?"}
A -->|"You, directly"| D["Data Curation:
GPU Function execution"]
A -->|"A refresh job"| R["Feature Engineering:
computed column / MV refresh"]
D --> P["Same GPU-class placement,
same capacity-wait behavior"]
R --> P
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class P accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ F["Function registered with
gpu=True, gpu_class"] --> A{"Who requests
a worker?"}
+ A -->|"You, directly"| D["Data Curation:
GPU Function execution"]
+ A -->|"A refresh job"| R["Feature Engineering:
computed column / MV refresh"]
+ D --> P["Same GPU-class placement,
same capacity-wait behavior"]
+ R --> P
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class P accent
+```
+
+
+
The Function's `gpu_class` (or the deployment's `default`) resolves to the
same node selector, tolerations and resource name either way, and a
refresh that needs more GPU capacity than is currently free waits the
diff --git a/docs/web/enterprise/horizontal-scaling.mdx b/docs/web/enterprise/horizontal-scaling.mdx
index 8bab76ece..315dd350c 100644
--- a/docs/web/enterprise/horizontal-scaling.mdx
+++ b/docs/web/enterprise/horizontal-scaling.mdx
@@ -30,7 +30,15 @@ different tables can share a group, use entirely different groups, or
overlap partially, and each picks its own RF independently within every
group it uses:
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "clusterBkg": "#211d1a", "clusterBorder": "#4c443f",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart TB
subgraph G0["Replica group 0"]
direction LR
@@ -53,8 +61,51 @@ flowchart TB
TA["Table A
groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0
TA -.-> G1
TB["Table B
group 2 only · RF 3"] -.-> G2
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class TA,TB accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart TB
+ subgraph G0["Replica group 0"]
+ direction LR
+ G0A["Plan executor"]
+ G0B["Plan executor"]
+ end
+
+ subgraph G1["Replica group 1"]
+ direction LR
+ G1A["Plan executor"]
+ end
+
+ subgraph G2["Replica group 2"]
+ direction LR
+ G2A["Plan executor"]
+ G2B["Plan executor"]
+ G2C["Plan executor"]
+ end
+
+ TA["Table A
groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0
+ TA -.-> G1
+ TB["Table B
group 2 only · RF 3"] -.-> G2
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class TA,TB accent
+```
+
+
+
Table A is assigned to groups 0 and 1, at RF 2 and RF 1 respectively --
three plan executors total hold a copy of Table A. Table B never touches
groups 0 or 1 at all; it is assigned only to group 2, at RF 3. Group 2's
@@ -81,7 +132,15 @@ no healthy plan executor. Keeping one query's reads co-located protects the
locality of whichever group's cache is already warm for it, rather than
letting each table's read drift to a different group.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "clusterBkg": "#211d1a", "clusterBorder": "#4c443f",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
Q["Query node
routes to the lead group"]
@@ -101,8 +160,48 @@ flowchart LR
Q --> A1
Q -. "only if group 0 has no healthy PE" .-> B0
Q -. "only if group 0 has no healthy PE" .-> B1
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class Q accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ Q["Query node
routes to the lead group"]
+
+ subgraph G0["Replica group 0 (lead)"]
+ direction LR
+ A0["Plan executor
copy 1"]
+ A1["Plan executor
copy 2"]
+ end
+
+ subgraph G1["Replica group 1 (fallback)"]
+ direction LR
+ B0["Plan executor
copy 1"]
+ B1["Plan executor
copy 2"]
+ end
+
+ Q --> A0
+ Q --> A1
+ Q -. "only if group 0 has no healthy PE" .-> B0
+ Q -. "only if group 0 has no healthy PE" .-> B1
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class Q accent
+```
+
+
+
This table is assigned to two replica groups, each at replication factor
2: four plan executors in total, each holding a full copy of the table's
data. Adding a third group, or raising RF within a group, adds more
diff --git a/docs/web/enterprise/intelligent-auto-maintenance.mdx b/docs/web/enterprise/intelligent-auto-maintenance.mdx
index 719081f9e..800b867f6 100644
--- a/docs/web/enterprise/intelligent-auto-maintenance.mdx
+++ b/docs/web/enterprise/intelligent-auto-maintenance.mdx
@@ -17,7 +17,14 @@ needs to run, continuously, per table, in the background across your
cluster -- nothing to schedule, trigger, or monitor by hand unless you want
to.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
E["Table event notifications"] --> P
S["Periodic sweep
every table, on a timer"] --> P
@@ -27,8 +34,37 @@ flowchart LR
P --> C["Possible next step:
Compaction"]
P --> CL["Possible next step:
Cleanup"]
P --> R["Possible next step:
Remap catch-up"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class P accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ E["Table event notifications"] --> P
+ S["Periodic sweep
every table, on a timer"] --> P
+ P["Planner
evaluates one table"]
+ P --> W["Possible next step:
WAL index catch-up"]
+ P --> I["Possible next step:
Index operations"]
+ P --> C["Possible next step:
Compaction"]
+ P --> CL["Possible next step:
Cleanup"]
+ P --> R["Possible next step:
Remap catch-up"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class P accent
+```
+
+
+
## Tracking activity {#tracking}
An agent's long-running process watches two independent sources for a
diff --git a/docs/web/enterprise/realtime-engine-compaction.mdx b/docs/web/enterprise/realtime-engine-compaction.mdx
index e63a43c52..e42d113dc 100644
--- a/docs/web/enterprise/realtime-engine-compaction.mdx
+++ b/docs/web/enterprise/realtime-engine-compaction.mdx
@@ -76,14 +76,42 @@ instead. Either way, each pass:
`memwal_catchup_ivfpq`, and so on) -- over the rows step 5 just added,
and records how far that index has caught up.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
A["Shard crosses a trigger
(count, bytes, or generation age)"] --> B["Per-pod trigger loop
or backpressure relief pass"]
B --> C["Compactor pool
dedup by PK, merge_insert into base"]
C --> D["Manifest drops
the folded SSTables"]
D --> E["memwal_catchup_<type>
extends each base index"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ A["Shard crosses a trigger
(count, bytes, or generation age)"] --> B["Per-pod trigger loop
or backpressure relief pass"]
+ B --> C["Compactor pool
dedup by PK, merge_insert into base"]
+ C --> D["Manifest drops
the folded SSTables"]
+ D --> E["memwal_catchup_<type>
extends each base index"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+```
+
+
+
Only one compaction pass runs against a given shard at a time. A second,
concurrent pass against the same shard would corrupt it: a stale pass can
overwrite rows a newer generation already replaced, or resurrect a row a
@@ -95,15 +123,48 @@ on the same collision, and node drain waits for the shard's current pass
instead of skipping it, since a skipped table is exactly what drain exists
to flush.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart TD
Shard["Shard's per-shard latch"] --> Held{"Already held by
an in-progress pass?"}
Held -->|no| Grant["Grant the latch,
run the pass"]
Held -->|yes, periodic trigger| Skip["Skip this tick --
retried next tick"]
Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,
never queued"]
Held -->|yes, node drain| Wait["Wait for the current
pass to finish, don't skip"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class Shard accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart TD
+ Shard["Shard's per-shard latch"] --> Held{"Already held by
an in-progress pass?"}
+ Held -->|no| Grant["Grant the latch,
run the pass"]
+ Held -->|yes, periodic trigger| Skip["Skip this tick --
retried next tick"]
+ Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,
never queued"]
+ Held -->|yes, node drain| Wait["Wait for the current
pass to finish, don't skip"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class Shard accent
+```
+
+
+
## Reading while compaction runs {#reads}
A query against an RTE-enabled table always reads the shard's current
diff --git a/docs/web/enterprise/tiered-index-cache.mdx b/docs/web/enterprise/tiered-index-cache.mdx
index 1c244beec..32737e25b 100644
--- a/docs/web/enterprise/tiered-index-cache.mdx
+++ b/docs/web/enterprise/tiered-index-cache.mdx
@@ -16,7 +16,14 @@ into memory. This lets a plan executor serve an index larger than its
available memory without re-reading and re-deserializing it from object
storage on every eviction. See [Architecture](/index#architecture).
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
R["Index lookup"] --> M{"In memory?"}
M -->|hit| Return["Served from memory"]
@@ -26,8 +33,33 @@ flowchart LR
O --> Fill["Deserialized,
loads into memory,
served"]
Evict["Memory eviction"] -.->|serialize, don't drop| D
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ R["Index lookup"] --> M{"In memory?"}
+ M -->|hit| Return["Served from memory"]
+ M -->|miss| D{"Spilled to disk?"}
+ D -->|hit| Promote["Deserialized,
promoted into memory,
served"]
+ D -->|miss| O[("Object storage")]
+ O --> Fill["Deserialized,
loads into memory,
served"]
+
+ Evict["Memory eviction"] -.->|serialize, don't drop| D
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+```
+
+
+
## Sharing the disk budget {#disk-budget}
Where enabled, the index disk tier and the
diff --git a/docs/web/governance/audit-log.mdx b/docs/web/governance/audit-log.mdx
index d29f680b2..9533e346a 100644
--- a/docs/web/governance/audit-log.mdx
+++ b/docs/web/governance/audit-log.mdx
@@ -14,7 +14,14 @@ privilege that was checked, and the object it was checked against -- can
be recorded as a durable audit log entry. On a failed check, the first
failure is logged; on success, the final successful check is logged.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
R["Request arrives"] --> C["Authorization check"]
C -->|denied| L1["Log the first failed check"]
@@ -22,8 +29,35 @@ flowchart LR
L1 --> T[("__audit_log table
system database")]
L2 --> T
T --> Q["SELECT ... FROM system.public.__audit_log"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class T accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ R["Request arrives"] --> C["Authorization check"]
+ C -->|denied| L1["Log the first failed check"]
+ C -->|granted| L2["Log the final successful check"]
+ L1 --> T[("__audit_log table
system database")]
+ L2 --> T
+ T --> Q["SELECT ... FROM system.public.__audit_log"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class T accent
+```
+
+
+
A deployment chooses how audit records are kept: discarded, written
alongside ordinary application logs, or stored durably as rows in a Lance
table. Only the durable Lance-backed sink produces something you can
diff --git a/docs/web/training/training-cache.mdx b/docs/web/training/training-cache.mdx
index 727f091c7..14769d2f0 100644
--- a/docs/web/training/training-cache.mdx
+++ b/docs/web/training/training-cache.mdx
@@ -18,7 +18,14 @@ serve every other query, with the same data page cache in front of object
storage. No training-specific configuration turns this on; it's a
consequence of connecting to Enterprise at all.
+
+
```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
+ "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
+ "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
flowchart LR
DL["PyTorch DataLoader
/ StreamingDataset"] --> T["Enterprise-connected table"]
T --> PE["Plan executor"]
@@ -26,8 +33,35 @@ flowchart LR
C -->|hit| Fast["Served from memory
or local disk"]
C -->|miss| O[("Object storage")]
O --> Fill["Fills the cache,
served"]
+ classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
+ classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
+ class Fast accent
```
+
+
+
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {
+ "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
+ "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
+ "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
+}}}%%
+flowchart LR
+ DL["PyTorch DataLoader
/ StreamingDataset"] --> T["Enterprise-connected table"]
+ T --> PE["Plan executor"]
+ PE --> C{"In data page cache?"}
+ C -->|hit| Fast["Served from memory
or local disk"]
+ C -->|miss| O[("Object storage")]
+ O --> Fill["Fills the cache,
served"]
+ classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
+ classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
+ class Fast accent
+```
+
+
+
This covers the ordinary tabular columns a query reads -- the data page
cache's unit is a fixed-size page of table data. A [Blob v2](/management/table#managed-blobs)
column's payload is fetched through a separate range-request path keyed