mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-18 16:43:59 +00:00
- Removes the stale `/enterprise` redirect that hijacked the real LanceDB Enterprise page, sending it to the Embedded vs Enterprise comparison page instead. - Removes the Merge page from the Data Ingestion nav (confusing parity story between Embedded and Enterprise); moves RTE Write up to right below Delete and renames it **RTE Write (Upsert)** to take over that slot. - Removes the Early access badge from Tiered Index Cache and restores its full sidebar title (was shortened to "Index Cache" to fit the badge). - Moves Changes below Dedupe in Data Curation. - Matches Advanced Conflict Resolution's sidebar title to its page title (previously just "Conflict Resolution" in the sidebar). - Marks Job Framework as Enterprise-only. - Distributed RTE Write diagram now shows the `metrics` table split across multiple shards, illustrating the per-table sharding described in the page's own text. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
138 lines
5.5 KiB
Plaintext
138 lines
5.5 KiB
Plaintext
---
|
|
title: "Distributed RTE Write"
|
|
sidebarTitle: "Distributed RTE Write"
|
|
description: "How Enterprise scales RTE writes horizontally across a dedicated serving tier, and the throughput limits that come with it."
|
|
icon: "bolt"
|
|
keywords: ["rte", "real time engine", "shard", "sharding", "horizontal scaling", "write throughput"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
[RTE writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
|
|
as one call that works the same way on either connection. On Enterprise, that
|
|
call does not run locally -- it posts to the server, which installs the write
|
|
path on a dedicated, horizontally scalable serving tier instead of running it
|
|
in your process. This page covers that tier: how it shards a table's writes,
|
|
the one real limit sharding does not remove, and the throughput you can expect.
|
|
|
|
## How writes are routed {#routing}
|
|
|
|
Enabling RTE on a remote table sends the spec to the server, which installs
|
|
the write path on the dataset instead of running it in your process. From
|
|
then on, the query node routes every request for that table -- reads and
|
|
writes alike -- to a single owning **RTE node**, a stateful component
|
|
separate from the query nodes and plan executors that serve ordinary
|
|
queries. A query node routes different tables to different RTE nodes, and
|
|
a single RTE node can host shards from many different tables at once. RTE
|
|
nodes share nothing but the underlying object storage; no node talks to
|
|
another one directly.
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
|
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
Q["Query node<br/>routes each table's shards to an RTE node"]
|
|
|
|
subgraph W0["RTE node A"]
|
|
direction TB
|
|
S0["Shard 0 — table 'frames'"]
|
|
S1["Shard 0 — table 'events'"]
|
|
end
|
|
|
|
subgraph W1["RTE node B"]
|
|
direction TB
|
|
S2["Shard 0 — table 'metrics'"]
|
|
S3["Shard 1 — table 'metrics'"]
|
|
end
|
|
|
|
Q -- "'frames'" --> W0
|
|
Q -- "'events'" --> W0
|
|
Q -- "'metrics'" --> W1
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
|
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
Q["Query node<br/>routes each table's shards to an RTE node"]
|
|
|
|
subgraph W0["RTE node A"]
|
|
direction TB
|
|
S0["Shard 0 — table 'frames'"]
|
|
S1["Shard 0 — table 'events'"]
|
|
end
|
|
|
|
subgraph W1["RTE node B"]
|
|
direction TB
|
|
S2["Shard 0 — table 'metrics'"]
|
|
S3["Shard 1 — table 'metrics'"]
|
|
end
|
|
|
|
Q -- "'frames'" --> W0
|
|
Q -- "'events'" --> W0
|
|
Q -- "'metrics'" --> W1
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
```
|
|
|
|
</div>
|
|
|
|
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
|
|
|
|
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
|
|
**table shards**, each a separate, independently-appending stream, giving the
|
|
table that many parallel write paths instead of one. A shard's identity is
|
|
derived from the table itself, not assigned by an operator, so a shard
|
|
survives the table being claimed by a different RTE node later.
|
|
|
|
This is where the horizontal scaling actually happens: splitting `frames`
|
|
into 8 shards gives it 8 independent append streams instead of 1, raising
|
|
that table's own write ceiling. Adding RTE node replicas spreads shard
|
|
ownership across more nodes, so the cluster can host more tables -- and more
|
|
shards -- without any one node becoming a bottleneck for everything it serves.
|
|
|
|
`RTEWriteSpec::identity(column)` and `RTEWriteSpec::unsharded()` both resolve
|
|
to a single table shard -- `bucket` is the sharding mode that actually
|
|
parallelizes write throughput; the other two are about routing semantics, not
|
|
scaling.
|
|
|
|
## Throughput {#throughput}
|
|
|
|
A table shard is the unit of capacity: measured runs show it sustaining
|
|
roughly 85-90 MB/s of durable append before acknowledgement, and throughput
|
|
scaling nearly linearly with shard count up to at least 8 shards on one
|
|
node -- the bottleneck is the durable write itself, not CPU or memory. This
|
|
is observed capacity, not an enforced quota; nothing in the deployment caps
|
|
a shard's or a table's throughput below it. [Contact
|
|
support](https://www.lancedb.com/contact) if your workload needs more write
|
|
throughput than this provides.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="RTE Write (Upsert)" icon="bolt" href="/ingestion/realtime-write">
|
|
Enable RTE and write to a table; the same call on Embedded or Enterprise.
|
|
</Card>
|
|
<Card title="How the real time engine works" icon="database" href="/tables/realtime-engine">
|
|
MemWAL, shards, memtables and SSTables -- the shared architecture.
|
|
</Card>
|
|
<Card title="Distributed RTE Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
|
|
How reads combine the WAL tier's fresh rows with the base table across the fleet.
|
|
</Card>
|
|
</CardGroup>
|