mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack YeandClaude Sonnet 5
d0b0efcd47
## Summary Follow-up to the Mermaid brand-theming PR, addressing review feedback: - Diagram text was hard to read at the previous 15px font size; bumped to 18px, light and dark alike. - The dark-mode orange accent border was brighter and more saturated than its fill or the default border, reading as too loud against the dark background. It now reuses the same muted orange already used for the light-mode accent border, which was fine as is. ## Companion change lancedb/sophon#9719 makes the same two changes across its own diagrams. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
91 lines
3.3 KiB
Plaintext
91 lines
3.3 KiB
Plaintext
---
|
|
title: "Distributed write"
|
|
sidebarTitle: "Distributed write"
|
|
description: "How LanceDB Enterprise distributes a CTAS import's writes across plan executors."
|
|
icon: upload
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Distributed write applies specifically to importing data with `CREATE
|
|
TABLE ... AS SELECT`: the query node that planned the statement partitions
|
|
the source -- Parquet files from a HuggingFace dataset, or row-group
|
|
ranges of an existing Lance dataset in object storage -- across plan
|
|
executors, each of which writes its own uncommitted Lance data files and
|
|
reports back. The query node collects those results and commits them
|
|
together as the new table's first version; no separate coordinator
|
|
service is involved. See [Architecture](/index#architecture).
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
|
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
|
PE1["Plan executor<br/>writes uncommitted files"]
|
|
PE2["Plan executor<br/>writes uncommitted files"]
|
|
PE3["Plan executor<br/>writes uncommitted files"]
|
|
|
|
S --> Q
|
|
Q --> PE1
|
|
Q --> PE2
|
|
Q --> PE3
|
|
PE1 --> Q
|
|
PE2 --> Q
|
|
PE3 --> Q
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
|
class Q accent
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
|
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
|
PE1["Plan executor<br/>writes uncommitted files"]
|
|
PE2["Plan executor<br/>writes uncommitted files"]
|
|
PE3["Plan executor<br/>writes uncommitted files"]
|
|
|
|
S --> Q
|
|
Q --> PE1
|
|
Q --> PE2
|
|
Q --> PE3
|
|
PE1 --> Q
|
|
PE2 --> Q
|
|
PE3 --> Q
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
|
class Q accent
|
|
```
|
|
|
|
</div>
|
|
|
|
An ordinary `INSERT` or `merge_insert`, from SQL or the SDK, does not
|
|
distribute this way -- it runs as a single write, on whichever node or
|
|
process executes it. Only a CTAS import fans the write itself out across
|
|
machines.
|
|
|
|
## Write interfaces {#write-interfaces}
|
|
|
|
- [Import Datasets](/ingestion/import) for the CTAS syntax this
|
|
distributes.
|
|
- [Insert data](/ingestion/insert) and [Merge](/ingestion/merge-insert)
|
|
for the ordinary, non-distributed write paths.
|
|
- [Storage Integration](/sql/ddl/storage-integration) for configuring
|
|
the source a CTAS import reads from.
|