From dfbf682cb55e42f0dd8d38738a6a1396ec24591e Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Tue, 3 Nov 2026 21:02:24 -0800 Subject: [PATCH] style: re-theme Mermaid diagrams to the brand palette, light and dark (#4822) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Every Mermaid diagram now renders in the brand palette, with separate light and dark variants that switch with the site's theme toggle. Default node styling stays neutral; one accent color marks each diagram's actual subject, chosen per diagram. A couple of diagrams changed orientation where their old layout was too wide to read comfortably. Companion change in lancedb/sophon uses the same colors and pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT Co-authored-by: Claude Sonnet 5 --- docs/web/basics/catalog.mdx | 32 ++++++ .../curation/blob-late-materialization.mdx | 41 ++++++++ docs/web/curation/distributed-shuffle.mdx | 48 +++++++++ docs/web/enterprise/adaptive-execution.mdx | 32 ++++++ .../advanced-conflict-resolution.mdx | 76 +++++++++++++- docs/web/enterprise/checkpointing.mdx | 36 +++++++ docs/web/enterprise/data-page-cache.mdx | 30 ++++++ .../web/enterprise/distributed-compaction.mdx | 35 +++++++ docs/web/enterprise/distributed-indexing.mdx | 37 +++++++ .../enterprise/distributed-real-time-read.mdx | 52 ++++++++++ .../distributed-real-time-write.mdx | 42 ++++++++ docs/web/enterprise/distributed-write.mdx | 41 ++++++++ docs/web/enterprise/gpu-refresh-execution.mdx | 33 +++++++ docs/web/enterprise/horizontal-scaling.mdx | 99 +++++++++++++++++++ .../intelligent-auto-maintenance.mdx | 36 +++++++ .../enterprise/realtime-engine-compaction.mdx | 61 ++++++++++++ docs/web/enterprise/tiered-index-cache.mdx | 32 ++++++ docs/web/governance/audit-log.mdx | 34 +++++++ docs/web/training/training-cache.mdx | 34 +++++++ 19 files changed, 828 insertions(+), 3 deletions(-) diff --git a/docs/web/basics/catalog.mdx b/docs/web/basics/catalog.mdx index f8ee0d455..f00060f25 100644 --- a/docs/web/basics/catalog.mdx +++ b/docs/web/basics/catalog.mdx @@ -47,7 +47,14 @@ For example, a search application can keep production and staging data in separate namespaces within the same database, with `prod` further split into a nested `search` namespace: +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart TD catalog["root catalog"] --> database["ai_data · database"] database --> prod["prod · namespace"] @@ -57,8 +64,33 @@ flowchart TD search --> search_view["recent_documents · view"] search --> search_fn["embed · function"] staging --> staging_docs["documents · table"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart TD + catalog["root catalog"] --> database["ai_data · database"] + database --> prod["prod · namespace"] + database --> staging["staging · namespace"] + prod --> search["search · namespace"] + search --> search_docs["documents · table"] + search --> search_view["recent_documents · view"] + search --> search_fn["embed · function"] + staging --> staging_docs["documents · table"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 +``` + +
+ | Object | What it represents | | --- | --- | | [Catalog](/management/catalog) | The unnamed root of an Enterprise deployment, identified by its endpoint; lists, creates and drops databases. Not present in Embedded. | diff --git a/docs/web/curation/blob-late-materialization.mdx b/docs/web/curation/blob-late-materialization.mdx index d1213b64a..86aff0e54 100644 --- a/docs/web/curation/blob-late-materialization.mdx +++ b/docs/web/curation/blob-late-materialization.mdx @@ -19,7 +19,15 @@ the join has already settled which rows survive. ## The two plan shapes {#plan-shapes} +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart TB subgraph Naive["Naive: wide column read before filtering"] direction TB @@ -32,8 +40,41 @@ flowchart TB D1["Scan only the join key
(+ early columns)"] --> D2["Join on the key"] D2 --> D3["RemoteTake:
fetch the wide column,
once per surviving row"] end + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class D3 accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart TB + subgraph Naive["Naive: wide column read before filtering"] + direction TB + N1["Scan wide column
for every row"] --> N2["Join"] + N2 --> N3["Rows that didn't match
are discarded --
their wide read was wasted"] + end + + subgraph Deferred["Late materialization: wide column read after filtering"] + direction TB + D1["Scan only the join key
(+ early columns)"] --> D2["Join on the key"] + D2 --> D3["RemoteTake:
fetch the wide column,
once per surviving row"] + end + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class D3 accent +``` + +
+ In the naive plan, the wide column is read once per **candidate** row -- every row either side of the join could possibly produce -- before the join has thrown any of them away. In the rewritten plan, the join diff --git a/docs/web/curation/distributed-shuffle.mdx b/docs/web/curation/distributed-shuffle.mdx index c8f7d1dd2..1a889f0d3 100644 --- a/docs/web/curation/distributed-shuffle.mdx +++ b/docs/web/curation/distributed-shuffle.mdx @@ -18,7 +18,15 @@ plan needs data repartitioned across machines -- a hash join or a grouped aggregate, for example, where rows with the same key must land on the same worker regardless of which plan executor originally read them. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR subgraph Stage1["Stage: scan"] direction TB @@ -38,8 +46,48 @@ flowchart LR PE2 --> EX EX --> PE3 EX --> PE4 + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class EX accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + subgraph Stage1["Stage: scan"] + direction TB + PE1["Plan executor
reads its partition"] + PE2["Plan executor
reads its partition"] + end + + EX["Exchange
streaming gRPC,
repartitions by join/group key,
backpressure via a byte budget"] + + subgraph Stage2["Stage: join / aggregate"] + direction TB + PE3["Plan executor
probes its share of keys"] + PE4["Plan executor
probes its share of keys"] + end + + PE1 --> EX + PE2 --> EX + EX --> PE3 + EX --> PE4 + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class EX accent +``` + +
+ ## Pipelined, not batched {#pipelined} The exchange moves data as a gRPC stream of row batches, not a file a diff --git a/docs/web/enterprise/adaptive-execution.mdx b/docs/web/enterprise/adaptive-execution.mdx index d83af371f..4b51bf59c 100644 --- a/docs/web/enterprise/adaptive-execution.mdx +++ b/docs/web/enterprise/adaptive-execution.mdx @@ -38,7 +38,14 @@ memory, out-of-memory events, and for a GPU Function, compute and memory utilization per GPU -- and that measurement decides whether the *next* invocation runs with more resources, fewer, or the same: +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR I["Invocation runs
wall time, CPU, memory, GPU measured"] --> D{"Saturated?"} D -->|"OOM, or memory/CPU/GPU
over target utilization"| P["Promote:
next size up"] @@ -48,8 +55,33 @@ flowchart LR Dm --> N S --> N N --> I + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + I["Invocation runs
wall time, CPU, memory, GPU measured"] --> D{"Saturated?"} + D -->|"OOM, or memory/CPU/GPU
over target utilization"| P["Promote:
next size up"] + D -->|"fast task, low CPU,
known memory"| Dm["Demote:
one size down,
never past a floor"] + D -->|neither| S["Stay:
same class"] + P --> N["Next invocation"] + Dm --> N + S --> N + N --> I + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 +``` + +
+ **Promotion** happens on an out-of-memory event, memory utilization above 70% of the class's limit, or CPU pegged at 85%+ for a sample long enough (5 seconds) that one fast spike isn't mistaken for sustained saturation. A diff --git a/docs/web/enterprise/advanced-conflict-resolution.mdx b/docs/web/enterprise/advanced-conflict-resolution.mdx index 7139423b9..3b3964bb5 100644 --- a/docs/web/enterprise/advanced-conflict-resolution.mdx +++ b/docs/web/enterprise/advanced-conflict-resolution.mdx @@ -65,7 +65,14 @@ Once a table is tagged, Lance refuses a compaction that does not set Enterprise-tagged table has to keep deferring the remap, or be compacted by a writer that can remap the tagged history eagerly. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart TD Commit["Compaction commit
(defer_index_remap only)"] --> Tagged{"Table already
tagged with an FRI?"} Tagged -->|no| First{"Stable row IDs, or does
any index cover the
rewritten fragments?"} @@ -76,8 +83,38 @@ flowchart TD Classify -->|yes| Append["Must append a tagged
transition before the
commit is admitted"] V0 --> FRI[("Fragment reuse index")] Append --> FRI + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class FRI accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart TD + Commit["Compaction commit
(defer_index_remap only)"] --> Tagged{"Table already
tagged with an FRI?"} + Tagged -->|no| First{"Stable row IDs, or does
any index cover the
rewritten fragments?"} + First -->|no| Admit["Admitted as
ordinary business --
nothing appended"] + First -->|yes| V0["Records a legacy v0 entry --
table stays untagged"] + Tagged -->|yes| Classify{"Does any index
cover the rewritten
fragments?"} + Classify -->|no| Admit + Classify -->|yes| Append["Must append a tagged
transition before the
commit is admitted"] + V0 --> FRI[("Fragment reuse index")] + Append --> FRI + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class FRI accent +``` + +
+ ## Enterprise defers and batches the catch-up {#catch-up} Translating addresses through a long chain of transitions at every read adds @@ -102,15 +139,48 @@ rather than one blocking on the other: a handful of compactions can land before their indexes are physically remapped, and the catch-up job amortizes that remap across all of them in one job instead of one per compaction. +
+ ```mermaid -flowchart LR +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart TD C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index
(ledger of transitions)"] C2["Compaction job"] -->|appends transition| FRI C3["Compaction job"] -->|appends transition| FRI - FRI -->|5 transitions or 1 day old| R["Remap catch-up job"] - R -->|applies accumulated transitions| IDX["Affected indexes,
physically remapped"] + FRI -->|5 transitions or
1 day old| R["Remap catch-up job"] + R -->|applies accumulated
transitions| IDX["Affected indexes,
physically remapped"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class FRI,R accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart TD + C1["Compaction job"] -->|appends transition| FRI["Fragment reuse index
(ledger of transitions)"] + C2["Compaction job"] -->|appends transition| FRI + C3["Compaction job"] -->|appends transition| FRI + FRI -->|5 transitions or
1 day old| R["Remap catch-up job"] + R -->|applies accumulated
transitions| IDX["Affected indexes,
physically remapped"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class FRI,R accent +``` + +
+ ## What to read next {#next} diff --git a/docs/web/enterprise/checkpointing.mdx b/docs/web/enterprise/checkpointing.mdx index c4cd62893..5fe82bbda 100644 --- a/docs/web/enterprise/checkpointing.mdx +++ b/docs/web/enterprise/checkpointing.mdx @@ -24,7 +24,14 @@ that started but never recorded completion is redone in full on the next attempt, so a Function with external side effects (a remote API call, for example) can have those side effects repeated for that part's rows. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR S["Part started
event recorded"] --> E["Function executes"] E --> C["Part completed
event recorded"] @@ -34,8 +41,37 @@ flowchart LR S -. "worker dies here" .-> R["Next attempt reads the log,
sees no completion,
redoes only this part"] C -. "worker dies here" .-> R2["Next attempt reads the log,
sees this part done,
inherits it, moves on"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class R,R2 accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + S["Part started
event recorded"] --> E["Function executes"] + E --> C["Part completed
event recorded"] + C --> M{"More parts?"} + M -->|yes| S + M -->|no| F["Finalized:
digest-verified assembly
into the real data file"] + + S -. "worker dies here" .-> R["Next attempt reads the log,
sees no completion,
redoes only this part"] + C -. "worker dies here" .-> R2["Next attempt reads the log,
sees this part done,
inherits it, moves on"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class R,R2 accent +``` + +
+ ## What gets recorded, and when {#events} Each part contributes two events to its data file's log: one when a diff --git a/docs/web/enterprise/data-page-cache.mdx b/docs/web/enterprise/data-page-cache.mdx index 7c385f1bd..a4bfd61ae 100644 --- a/docs/web/enterprise/data-page-cache.mdx +++ b/docs/web/enterprise/data-page-cache.mdx @@ -16,7 +16,14 @@ the pages they serve. A background prewarm job can populate a cache ahead of a request that needs it, rather than filling it lazily on first read. See [Architecture](/index#architecture). +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR R["Read request
for a data page"] --> M{"In memory?"} M -->|hit| Return["Served from memory"] @@ -24,8 +31,31 @@ flowchart LR D -->|hit| Promote["Loaded into memory,
served"] D -->|miss| O[("Object storage")] O --> Fill["Fills memory and disk,
served"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + R["Read request
for a data page"] --> M{"In memory?"} + M -->|hit| Return["Served from memory"] + M -->|miss| D{"In local disk cache?"} + D -->|hit| Promote["Loaded into memory,
served"] + D -->|miss| O[("Object storage")] + O --> Fill["Fills memory and disk,
served"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 +``` + +
+ A table's local disk cache shares one budget with the [tiered index cache](/enterprise/tiered-index-cache#disk-budget)'s disk tier where both are enabled on the same executor -- raising one's share of diff --git a/docs/web/enterprise/distributed-compaction.mdx b/docs/web/enterprise/distributed-compaction.mdx index c23da4859..0c483865c 100644 --- a/docs/web/enterprise/distributed-compaction.mdx +++ b/docs/web/enterprise/distributed-compaction.mdx @@ -15,7 +15,14 @@ Lance's own planner already produces disjoint tasks, so workers claim and execute them directly through the job registry, with no coordination needed to decide who does what. See [Architecture](/index#architecture). +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR L["Lance's planner
splits fragments into
disjoint tasks"] --> R[("Job registry")] R --> W1["Worker
claims fragments 1-2"] @@ -24,8 +31,36 @@ flowchart LR W1 --> C["Fleet-wide eligibility gate:
one worker claims the commit"] W2 --> C W3 --> C + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class R,C accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + L["Lance's planner
splits fragments into
disjoint tasks"] --> R[("Job registry")] + R --> W1["Worker
claims fragments 1-2"] + R --> W2["Worker
claims fragments 3-4"] + R --> W3["Worker
claims fragments 5-6"] + W1 --> C["Fleet-wide eligibility gate:
one worker claims the commit"] + W2 --> C + W3 --> C + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class R,C accent +``` + +
+ No task depends on another's result, so a worker that finishes early simply claims another task rather than waiting on its peers. Committing is where the fleet does coordinate: finished tasks go through a shared diff --git a/docs/web/enterprise/distributed-indexing.mdx b/docs/web/enterprise/distributed-indexing.mdx index fdf4715c0..b40e185df 100644 --- a/docs/web/enterprise/distributed-indexing.mdx +++ b/docs/web/enterprise/distributed-indexing.mdx @@ -15,7 +15,14 @@ registry and commit the finished index together. A vector index build first trains a shared IVF/PQ model from sampled data, then workers build storage from that model in parallel. See [Architecture](/index#architecture). +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR S["Sample data"] --> T["Train shared IVF/PQ model
(vector index only)"] T --> P["Bin-pack build plan
into claimable tasks"] @@ -26,8 +33,38 @@ flowchart LR W1 --> C["Commit the
finished index"] W2 --> C W3 --> C + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class R accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + S["Sample data"] --> T["Train shared IVF/PQ model
(vector index only)"] + T --> P["Bin-pack build plan
into claimable tasks"] + P --> R[("Job registry")] + R --> W1["Worker
claims a task"] + R --> W2["Worker
claims a task"] + R --> W3["Worker
claims a task"] + W1 --> C["Commit the
finished index"] + W2 --> C + W3 --> C + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class R accent +``` + +
+ ## Indexing interfaces {#indexing-interfaces} The shared indexing guides describe how to build and maintain each index diff --git a/docs/web/enterprise/distributed-real-time-read.mdx b/docs/web/enterprise/distributed-real-time-read.mdx index 697679f16..2adc267e8 100644 --- a/docs/web/enterprise/distributed-real-time-read.mdx +++ b/docs/web/enterprise/distributed-real-time-read.mdx @@ -17,7 +17,15 @@ usually served by different machines -- plan executors for the base table, a [WAL node](/tables/realtime-engine#wal-nodes) for the table's shards -- so a query has to combine their results across the network instead. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR Q["Query node
plans the query"] @@ -41,8 +49,52 @@ flowchart LR W1 --> U W2 --> U U --> R["Merged result"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class U accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + Q["Query node
plans the query"] + + subgraph PE["Plan executors"] + direction TB + B1["Base-table arm
index-covered scan or ANN search"] + end + + subgraph WN["WAL node (owns this table's shards)"] + direction TB + W1["SSTable arm
index-covered"] + W2["Active memtable arm
vector: in-memory HNSW*
FTS/scalar: indexed"] + end + + U["Union exec
PK-keyed last-write-wins dedup,
top-K cut on distance"] + + Q --> B1 + Q --> W1 + Q --> W2 + B1 --> U + W1 --> U + W2 --> U + U --> R["Merged result"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class U accent +``` + +
+ ## An extra exec node, not a parallel system {#union-exec} The distributed plan for an RTE-enabled table is an ordinary plan executor diff --git a/docs/web/enterprise/distributed-real-time-write.mdx b/docs/web/enterprise/distributed-real-time-write.mdx index 8778ef616..b07d6a64d 100644 --- a/docs/web/enterprise/distributed-real-time-write.mdx +++ b/docs/web/enterprise/distributed-real-time-write.mdx @@ -26,7 +26,15 @@ single owning **WAL node**, a stateful pod separate from the query nodes and plan executors that serve ordinary queries. WAL node pods share nothing but the underlying object storage; no pod talks to another one directly. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR Q["Query node
hashes (db, table) to one WAL node"] @@ -44,8 +52,42 @@ flowchart LR Q -- "writes + reads for 'frames'" --> W0 Q -- "writes + reads for 'events'" --> W1 + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff", + "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + Q["Query node
hashes (db, table) to one WAL node"] + + subgraph W0["WAL node A — owns table 'frames'"] + direction TB + S0["Shard 0
ShardWriter"] + S1["Shard 1
ShardWriter"] + S2["Shard 2
ShardWriter"] + end + + subgraph W1["WAL node B — owns table 'events'"] + direction TB + S3["Shard 0
ShardWriter"] + end + + Q -- "writes + reads for 'frames'" --> W0 + Q -- "writes + reads for 'events'" --> W1 + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 +``` + +
+ ## Shards scale one table, replicas scale the cluster {#shards-vs-replicas} `RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets` diff --git a/docs/web/enterprise/distributed-write.mdx b/docs/web/enterprise/distributed-write.mdx index 890d520b9..feda93a57 100644 --- a/docs/web/enterprise/distributed-write.mdx +++ b/docs/web/enterprise/distributed-write.mdx @@ -17,7 +17,14 @@ reports back. The query node collects those results and commits them together as the new table's first version; no separate coordinator service is involved. See [Architecture](/index#architecture). +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR S["Source
HuggingFace Parquet, or an
existing Lance dataset in
S3 / GCS / Azure"] Q["Query node
plans the CTAS,
commits the result"] @@ -32,8 +39,42 @@ flowchart LR PE1 --> Q PE2 --> Q PE3 --> Q + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class Q accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + S["Source
HuggingFace Parquet, or an
existing Lance dataset in
S3 / GCS / Azure"] + Q["Query node
plans the CTAS,
commits the result"] + PE1["Plan executor
writes uncommitted files"] + PE2["Plan executor
writes uncommitted files"] + PE3["Plan executor
writes uncommitted files"] + + S --> Q + Q --> PE1 + Q --> PE2 + Q --> PE3 + PE1 --> Q + PE2 --> Q + PE3 --> Q + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class Q accent +``` + +
+ An ordinary `INSERT` or `merge_insert`, from SQL or the SDK, does not distribute this way -- it runs as a single write, on whichever node or process executes it. Only a CTAS import fans the write itself out across diff --git a/docs/web/enterprise/gpu-refresh-execution.mdx b/docs/web/enterprise/gpu-refresh-execution.mdx index 0749210f1..2c61c34f6 100644 --- a/docs/web/enterprise/gpu-refresh-execution.mdx +++ b/docs/web/enterprise/gpu-refresh-execution.mdx @@ -17,15 +17,48 @@ through the exact same placement path as Curation -- refreshing a column is not a separate scheduling code path from calling the Function directly, just a different caller. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR F["Function registered with
gpu=True, gpu_class"] --> A{"Who requests
a worker?"} A -->|"You, directly"| D["Data Curation:
GPU Function execution"] A -->|"A refresh job"| R["Feature Engineering:
computed column / MV refresh"] D --> P["Same GPU-class placement,
same capacity-wait behavior"] R --> P + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class P accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + F["Function registered with
gpu=True, gpu_class"] --> A{"Who requests
a worker?"} + A -->|"You, directly"| D["Data Curation:
GPU Function execution"] + A -->|"A refresh job"| R["Feature Engineering:
computed column / MV refresh"] + D --> P["Same GPU-class placement,
same capacity-wait behavior"] + R --> P + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class P accent +``` + +
+ The Function's `gpu_class` (or the deployment's `default`) resolves to the same node selector, tolerations and resource name either way, and a refresh that needs more GPU capacity than is currently free waits the diff --git a/docs/web/enterprise/horizontal-scaling.mdx b/docs/web/enterprise/horizontal-scaling.mdx index 8bab76ece..315dd350c 100644 --- a/docs/web/enterprise/horizontal-scaling.mdx +++ b/docs/web/enterprise/horizontal-scaling.mdx @@ -30,7 +30,15 @@ different tables can share a group, use entirely different groups, or overlap partially, and each picks its own RF independently within every group it uses: +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart TB subgraph G0["Replica group 0"] direction LR @@ -53,8 +61,51 @@ flowchart TB TA["Table A
groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0 TA -.-> G1 TB["Table B
group 2 only · RF 3"] -.-> G2 + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class TA,TB accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart TB + subgraph G0["Replica group 0"] + direction LR + G0A["Plan executor"] + G0B["Plan executor"] + end + + subgraph G1["Replica group 1"] + direction LR + G1A["Plan executor"] + end + + subgraph G2["Replica group 2"] + direction LR + G2A["Plan executor"] + G2B["Plan executor"] + G2C["Plan executor"] + end + + TA["Table A
groups 0, 1 · RF 2 in group 0, RF 1 in group 1"] -.-> G0 + TA -.-> G1 + TB["Table B
group 2 only · RF 3"] -.-> G2 + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class TA,TB accent +``` + +
+ Table A is assigned to groups 0 and 1, at RF 2 and RF 1 respectively -- three plan executors total hold a copy of Table A. Table B never touches groups 0 or 1 at all; it is assigned only to group 2, at RF 3. Group 2's @@ -81,7 +132,15 @@ no healthy plan executor. Keeping one query's reads co-located protects the locality of whichever group's cache is already warm for it, rather than letting each table's read drift to a different group. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "clusterBkg": "#211d1a", "clusterBorder": "#4c443f", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR Q["Query node
routes to the lead group"] @@ -101,8 +160,48 @@ flowchart LR Q --> A1 Q -. "only if group 0 has no healthy PE" .-> B0 Q -. "only if group 0 has no healthy PE" .-> B1 + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class Q accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + Q["Query node
routes to the lead group"] + + subgraph G0["Replica group 0 (lead)"] + direction LR + A0["Plan executor
copy 1"] + A1["Plan executor
copy 2"] + end + + subgraph G1["Replica group 1 (fallback)"] + direction LR + B0["Plan executor
copy 1"] + B1["Plan executor
copy 2"] + end + + Q --> A0 + Q --> A1 + Q -. "only if group 0 has no healthy PE" .-> B0 + Q -. "only if group 0 has no healthy PE" .-> B1 + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class Q accent +``` + +
+ This table is assigned to two replica groups, each at replication factor 2: four plan executors in total, each holding a full copy of the table's data. Adding a third group, or raising RF within a group, adds more diff --git a/docs/web/enterprise/intelligent-auto-maintenance.mdx b/docs/web/enterprise/intelligent-auto-maintenance.mdx index 719081f9e..800b867f6 100644 --- a/docs/web/enterprise/intelligent-auto-maintenance.mdx +++ b/docs/web/enterprise/intelligent-auto-maintenance.mdx @@ -17,7 +17,14 @@ needs to run, continuously, per table, in the background across your cluster -- nothing to schedule, trigger, or monitor by hand unless you want to. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR E["Table event notifications"] --> P S["Periodic sweep
every table, on a timer"] --> P @@ -27,8 +34,37 @@ flowchart LR P --> C["Possible next step:
Compaction"] P --> CL["Possible next step:
Cleanup"] P --> R["Possible next step:
Remap catch-up"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class P accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + E["Table event notifications"] --> P + S["Periodic sweep
every table, on a timer"] --> P + P["Planner
evaluates one table"] + P --> W["Possible next step:
WAL index catch-up"] + P --> I["Possible next step:
Index operations"] + P --> C["Possible next step:
Compaction"] + P --> CL["Possible next step:
Cleanup"] + P --> R["Possible next step:
Remap catch-up"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class P accent +``` + +
+ ## Tracking activity {#tracking} An agent's long-running process watches two independent sources for a diff --git a/docs/web/enterprise/realtime-engine-compaction.mdx b/docs/web/enterprise/realtime-engine-compaction.mdx index e63a43c52..e42d113dc 100644 --- a/docs/web/enterprise/realtime-engine-compaction.mdx +++ b/docs/web/enterprise/realtime-engine-compaction.mdx @@ -76,14 +76,42 @@ instead. Either way, each pass: `memwal_catchup_ivfpq`, and so on) -- over the rows step 5 just added, and records how far that index has caught up. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR A["Shard crosses a trigger
(count, bytes, or generation age)"] --> B["Per-pod trigger loop
or backpressure relief pass"] B --> C["Compactor pool
dedup by PK, merge_insert into base"] C --> D["Manifest drops
the folded SSTables"] D --> E["memwal_catchup_<type>
extends each base index"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + A["Shard crosses a trigger
(count, bytes, or generation age)"] --> B["Per-pod trigger loop
or backpressure relief pass"] + B --> C["Compactor pool
dedup by PK, merge_insert into base"] + C --> D["Manifest drops
the folded SSTables"] + D --> E["memwal_catchup_<type>
extends each base index"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 +``` + +
+ Only one compaction pass runs against a given shard at a time. A second, concurrent pass against the same shard would corrupt it: a stale pass can overwrite rows a newer generation already replaced, or resurrect a row a @@ -95,15 +123,48 @@ on the same collision, and node drain waits for the shard's current pass instead of skipping it, since a skipped table is exactly what drain exists to flush. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart TD Shard["Shard's per-shard latch"] --> Held{"Already held by
an in-progress pass?"} Held -->|no| Grant["Grant the latch,
run the pass"] Held -->|yes, periodic trigger| Skip["Skip this tick --
retried next tick"] Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,
never queued"] Held -->|yes, node drain| Wait["Wait for the current
pass to finish, don't skip"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class Shard accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart TD + Shard["Shard's per-shard latch"] --> Held{"Already held by
an in-progress pass?"} + Held -->|no| Grant["Grant the latch,
run the pass"] + Held -->|yes, periodic trigger| Skip["Skip this tick --
retried next tick"] + Held -->|yes, manual POST /v1/compact| Refuse["Refused outright,
never queued"] + Held -->|yes, node drain| Wait["Wait for the current
pass to finish, don't skip"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class Shard accent +``` + +
+ ## Reading while compaction runs {#reads} A query against an RTE-enabled table always reads the shard's current diff --git a/docs/web/enterprise/tiered-index-cache.mdx b/docs/web/enterprise/tiered-index-cache.mdx index 1c244beec..32737e25b 100644 --- a/docs/web/enterprise/tiered-index-cache.mdx +++ b/docs/web/enterprise/tiered-index-cache.mdx @@ -16,7 +16,14 @@ into memory. This lets a plan executor serve an index larger than its available memory without re-reading and re-deserializing it from object storage on every eviction. See [Architecture](/index#architecture). +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR R["Index lookup"] --> M{"In memory?"} M -->|hit| Return["Served from memory"] @@ -26,8 +33,33 @@ flowchart LR O --> Fill["Deserialized,
loads into memory,
served"] Evict["Memory eviction"] -.->|serialize, don't drop| D + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + R["Index lookup"] --> M{"In memory?"} + M -->|hit| Return["Served from memory"] + M -->|miss| D{"Spilled to disk?"} + D -->|hit| Promote["Deserialized,
promoted into memory,
served"] + D -->|miss| O[("Object storage")] + O --> Fill["Deserialized,
loads into memory,
served"] + + Evict["Memory eviction"] -.->|serialize, don't drop| D + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 +``` + +
+ ## Sharing the disk budget {#disk-budget} Where enabled, the index disk tier and the diff --git a/docs/web/governance/audit-log.mdx b/docs/web/governance/audit-log.mdx index d29f680b2..9533e346a 100644 --- a/docs/web/governance/audit-log.mdx +++ b/docs/web/governance/audit-log.mdx @@ -14,7 +14,14 @@ privilege that was checked, and the object it was checked against -- can be recorded as a durable audit log entry. On a failed check, the first failure is logged; on success, the final successful check is logged. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR R["Request arrives"] --> C["Authorization check"] C -->|denied| L1["Log the first failed check"] @@ -22,8 +29,35 @@ flowchart LR L1 --> T[("__audit_log table
system database")] L2 --> T T --> Q["SELECT ... FROM system.public.__audit_log"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class T accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + R["Request arrives"] --> C["Authorization check"] + C -->|denied| L1["Log the first failed check"] + C -->|granted| L2["Log the final successful check"] + L1 --> T[("__audit_log table
system database")] + L2 --> T + T --> Q["SELECT ... FROM system.public.__audit_log"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class T accent +``` + +
+ A deployment chooses how audit records are kept: discarded, written alongside ordinary application logs, or stored durably as rows in a Lance table. Only the durable Lance-backed sink produces something you can diff --git a/docs/web/training/training-cache.mdx b/docs/web/training/training-cache.mdx index 727f091c7..14769d2f0 100644 --- a/docs/web/training/training-cache.mdx +++ b/docs/web/training/training-cache.mdx @@ -18,7 +18,14 @@ serve every other query, with the same data page cache in front of object storage. No training-specific configuration turns this on; it's a consequence of connecting to Enterprise at all. +
+ ```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8", + "primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8", + "edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% flowchart LR DL["PyTorch DataLoader
/ StreamingDataset"] --> T["Enterprise-connected table"] T --> PE["Plan executor"] @@ -26,8 +33,35 @@ flowchart LR C -->|hit| Fast["Served from memory
or local disk"] C -->|miss| O[("Object storage")] O --> Fill["Fills the cache,
served"] + classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8 + classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8 + class Fast accent ``` +
+ +
+ +```mermaid +%%{init: {"theme": "base", "themeVariables": { + "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19", + "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19", + "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px" +}}}%% +flowchart LR + DL["PyTorch DataLoader
/ StreamingDataset"] --> T["Enterprise-connected table"] + T --> PE["Plan executor"] + PE --> C{"In data page cache?"} + C -->|hit| Fast["Served from memory
or local disk"] + C -->|miss| O[("Object storage")] + O --> Fill["Fills the cache,
served"] + classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19 + classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19 + class Fast accent +``` + +
+ This covers the ordinary tabular columns a query reads -- the data page cache's unit is a fixed-size page of table data. A [Blob v2](/management/table#managed-blobs) column's payload is fetched through a separate range-request path keyed