mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
Jack YeandClaude Sonnet 5
5e38120bdc
## Summary - **Define Storage Integration**: un-gates the Python (and Rust, for S3) `create_storage_integration` examples, drafted against #4618 and held back until it shipped -- it's now in v0.42.0-beta.0+, and the signature matches the drafts exactly. - **Naming rules**: drops the `Branch` row and its "Learn more" link, consistent with de-emphasizing branches elsewhere in this nav. - **Distributed RTE Write**: removes the REST path, the `__lance_mem_wal` index name, and "WAL node" terminology in favor of an abstract framing -- a query node routes writes to RTE nodes, each of which can host shards from many different tables. Both diagram variants are updated to match. - **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop Helm-specific and exact-percentage detail (disk-budget math, `startup_prewarm_block_query_ready`/`healthPort`/failure-policy table) in favor of a short feature → architecture → why-it-matters shape, each closing with a contact-sales pointer. Data Page Cache's on-demand prewarm section is removed outright. ## Note I wasn't able to get a clean visual render of the updated Distributed RTE Write diagram in local preview -- Mermaid isn't rendering in headless Chrome in this environment for *any* page right now, confirmed against an unmodified reference page showing the identical symptom. The diagram source follows the exact pattern already shipped and working in the 21 diagrams from the earlier brand-theming pass, and `mint validate`/`mint broken-links` both pass clean. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
91 lines
3.3 KiB
Plaintext
91 lines
3.3 KiB
Plaintext
---
|
|
title: "Distributed Write"
|
|
sidebarTitle: "Distributed Write"
|
|
description: "How LanceDB Enterprise distributes a CTAS import's writes across plan executors."
|
|
icon: upload
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Distributed write applies specifically to importing data with `CREATE
|
|
TABLE ... AS SELECT`: the query node that planned the statement partitions
|
|
the source -- Parquet files from a HuggingFace dataset, or row-group
|
|
ranges of an existing Lance dataset in object storage -- across plan
|
|
executors, each of which writes its own uncommitted Lance data files and
|
|
reports back. The query node collects those results and commits them
|
|
together as the new table's first version; no separate coordinator
|
|
service is involved. See [Architecture](/index#architecture).
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
|
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
|
PE1["Plan executor<br/>writes uncommitted files"]
|
|
PE2["Plan executor<br/>writes uncommitted files"]
|
|
PE3["Plan executor<br/>writes uncommitted files"]
|
|
|
|
S --> Q
|
|
Q --> PE1
|
|
Q --> PE2
|
|
Q --> PE3
|
|
PE1 --> Q
|
|
PE2 --> Q
|
|
PE3 --> Q
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
|
class Q accent
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
S["Source<br/>HuggingFace Parquet, or an<br/>existing Lance dataset in<br/>S3 / GCS / Azure"]
|
|
Q["Query node<br/>plans the CTAS,<br/>commits the result"]
|
|
PE1["Plan executor<br/>writes uncommitted files"]
|
|
PE2["Plan executor<br/>writes uncommitted files"]
|
|
PE3["Plan executor<br/>writes uncommitted files"]
|
|
|
|
S --> Q
|
|
Q --> PE1
|
|
Q --> PE2
|
|
Q --> PE3
|
|
PE1 --> Q
|
|
PE2 --> Q
|
|
PE3 --> Q
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
|
class Q accent
|
|
```
|
|
|
|
</div>
|
|
|
|
An ordinary `INSERT` or `merge_insert`, from SQL or the SDK, does not
|
|
distribute this way -- it runs as a single write, on whichever node or
|
|
process executes it. Only a CTAS import fans the write itself out across
|
|
machines.
|
|
|
|
## Write interfaces {#write-interfaces}
|
|
|
|
- [Import Datasets](/ingestion/import) for the CTAS syntax this
|
|
distributes.
|
|
- [Insert data](/ingestion/insert) and [Merge](/ingestion/merge-insert)
|
|
for the ordinary, non-distributed write paths.
|
|
- [Storage Integration](/sql/ddl/storage-integration) for configuring
|
|
the source a CTAS import reads from.
|