mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack YeandClaude Sonnet 5
dfbf682cb5
## Summary Every Mermaid diagram now renders in the brand palette, with separate light and dark variants that switch with the site's theme toggle. Default node styling stays neutral; one accent color marks each diagram's actual subject, chosen per diagram. A couple of diagrams changed orientation where their old layout was too wide to read comfortably. Companion change in lancedb/sophon uses the same colors and pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
91 lines
3.3 KiB
Plaintext
91 lines
3.3 KiB
Plaintext
---
|
|
title: "Distributed write"
|
|
sidebarTitle: "Distributed write"
|
|
description: "How LanceDB Enterprise distributes a CTAS import's writes across plan executors."
|
|
icon: upload
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Distributed write applies specifically to importing data with `CREATE
|
|
TABLE ... AS SELECT`: the query node that planned the statement partitions
|
|
the source -- Parquet files from a HuggingFace dataset, or row-group
|
|
ranges of an existing Lance dataset in object storage -- across plan
|
|
executors, each of which writes its own uncommitted Lance data files and
|
|
reports back. The query node collects those results and commits them
|
|
together as the new table's first version; no separate coordinator
|
|
service is involved. See [Architecture](/index#architecture).
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
|
}}}%%
|
|
flowchart LR
|
|
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
|
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
|
PE1["Plan executor<br/>writes uncommitted files"]
|
|
PE2["Plan executor<br/>writes uncommitted files"]
|
|
PE3["Plan executor<br/>writes uncommitted files"]
|
|
|
|
S --> Q
|
|
Q --> PE1
|
|
Q --> PE2
|
|
Q --> PE3
|
|
PE1 --> Q
|
|
PE2 --> Q
|
|
PE3 --> Q
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
classDef accent fill:#2a1810,stroke:#ff734a,stroke-width:1.5px,color:#f4ebe8
|
|
class Q accent
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "15px"
|
|
}}}%%
|
|
flowchart LR
|
|
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
|
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
|
PE1["Plan executor<br/>writes uncommitted files"]
|
|
PE2["Plan executor<br/>writes uncommitted files"]
|
|
PE3["Plan executor<br/>writes uncommitted files"]
|
|
|
|
S --> Q
|
|
Q --> PE1
|
|
Q --> PE2
|
|
Q --> PE3
|
|
PE1 --> Q
|
|
PE2 --> Q
|
|
PE3 --> Q
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
|
class Q accent
|
|
```
|
|
|
|
</div>
|
|
|
|
An ordinary `INSERT` or `merge_insert`, from SQL or the SDK, does not
|
|
distribute this way -- it runs as a single write, on whichever node or
|
|
process executes it. Only a CTAS import fans the write itself out across
|
|
machines.
|
|
|
|
## Write interfaces {#write-interfaces}
|
|
|
|
- [Import Datasets](/ingestion/import) for the CTAS syntax this
|
|
distributes.
|
|
- [Insert data](/ingestion/insert) and [Merge](/ingestion/merge-insert)
|
|
for the ordinary, non-distributed write paths.
|
|
- [Storage Integration](/sql/ddl/storage-integration) for configuring
|
|
the source a CTAS import reads from.
|