mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack YeandClaude Sonnet 5
5e38120bdc
## Summary - **Define Storage Integration**: un-gates the Python (and Rust, for S3) `create_storage_integration` examples, drafted against #4618 and held back until it shipped -- it's now in v0.42.0-beta.0+, and the signature matches the drafts exactly. - **Naming rules**: drops the `Branch` row and its "Learn more" link, consistent with de-emphasizing branches elsewhere in this nav. - **Distributed RTE Write**: removes the REST path, the `__lance_mem_wal` index name, and "WAL node" terminology in favor of an abstract framing -- a query node routes writes to RTE nodes, each of which can host shards from many different tables. Both diagram variants are updated to match. - **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop Helm-specific and exact-percentage detail (disk-budget math, `startup_prewarm_block_query_ready`/`healthPort`/failure-policy table) in favor of a short feature → architecture → why-it-matters shape, each closing with a contact-sales pointer. Data Page Cache's on-demand prewarm section is removed outright. ## Note I wasn't able to get a clean visual render of the updated Distributed RTE Write diagram in local preview -- Mermaid isn't rendering in headless Chrome in this environment for *any* page right now, confirmed against an unmodified reference page showing the identical symptom. The diagram source follows the exact pattern already shipped and working in the 21 diagrams from the earlier brand-theming pass, and `mint validate`/`mint broken-links` both pass clean. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
112 lines
4.0 KiB
Plaintext
112 lines
4.0 KiB
Plaintext
---
|
|
title: "Distributed Shuffle"
|
|
sidebarTitle: "Distributed Shuffle"
|
|
description: "How a SQL query's intermediate results move between plan executors for a join, aggregate or sort."
|
|
icon: shuffle
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A distributed SQL query plans as a sequence of **stages**, each with its
|
|
own set of partitions, connected by an **exchange**: a streaming gRPC
|
|
service that carries one stage's output to the next stage's plan
|
|
executors. [Distributed Search](/enterprise/distributed-search) covers
|
|
how a query gets placed on the fleet in the first place; this page covers
|
|
what happens between placement and the final result, for a query whose
|
|
plan needs data repartitioned across machines -- a hash join or a grouped
|
|
aggregate, for example, where rows with the same key must land on the
|
|
same worker regardless of which plan executor originally read them.
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
|
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
subgraph Stage1["Stage: scan"]
|
|
direction TB
|
|
PE1["Plan executor<br/>reads its partition"]
|
|
PE2["Plan executor<br/>reads its partition"]
|
|
end
|
|
|
|
EX["Exchange<br/>streaming gRPC,<br/>repartitions by join/group key,<br/>backpressure via a byte budget"]
|
|
|
|
subgraph Stage2["Stage: join / aggregate"]
|
|
direction TB
|
|
PE3["Plan executor<br/>probes its share of keys"]
|
|
PE4["Plan executor<br/>probes its share of keys"]
|
|
end
|
|
|
|
PE1 --> EX
|
|
PE2 --> EX
|
|
EX --> PE3
|
|
EX --> PE4
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
|
class EX accent
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
|
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
subgraph Stage1["Stage: scan"]
|
|
direction TB
|
|
PE1["Plan executor<br/>reads its partition"]
|
|
PE2["Plan executor<br/>reads its partition"]
|
|
end
|
|
|
|
EX["Exchange<br/>streaming gRPC,<br/>repartitions by join/group key,<br/>backpressure via a byte budget"]
|
|
|
|
subgraph Stage2["Stage: join / aggregate"]
|
|
direction TB
|
|
PE3["Plan executor<br/>probes its share of keys"]
|
|
PE4["Plan executor<br/>probes its share of keys"]
|
|
end
|
|
|
|
PE1 --> EX
|
|
PE2 --> EX
|
|
EX --> PE3
|
|
EX --> PE4
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
|
class EX accent
|
|
```
|
|
|
|
</div>
|
|
|
|
## Pipelined, not batched {#pipelined}
|
|
|
|
The exchange moves data as a gRPC stream of row batches, not a file a
|
|
later stage waits for: a consuming plan executor pulls batches as they
|
|
become available, and a slow consumer throttles its producer through a
|
|
byte budget on unread messages rather than letting memory grow without
|
|
bound. A partition that stops being read -- for example the probe side of
|
|
a hash join whose build side turned out empty -- is drained rather than
|
|
left to stall the budget for every other partition sharing the same
|
|
connection.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Distributed Search" icon="network-wired" href="/enterprise/distributed-search">
|
|
How a query gets placed on the query-serving fleet in the first place.
|
|
</Card>
|
|
<Card title="Query plans" icon="map" href="/basics/sql#query-plans">
|
|
Inspect a specific query's plan and its actual execution metrics.
|
|
</Card>
|
|
</CardGroup>
|