Files
lancedb/docs/web/enterprise/distributed-real-time-write.mdx
T
Taylor 3a980054a1 docs: standardize Realtime Engine naming (#4969)
The Enterprise feature name was inconsistently written as "Real Time
Engine" and "real time engine". Standardize it to **Realtime Engine** in
eight website guides, including page and sidebar titles, descriptions,
prose, link labels, cards, and search keywords.

The remaining MemWAL references describe Lance's open source format or
literal API/job identifiers, including `memwal_compaction` and
`memwal_catchup_<type>`, and are preserved. Only `docs/web/*.mdx` pages
change; SDK source, generated API reference, code examples, URL
destinations, and section anchors are unchanged.

Validation:
- `git diff --check` passes.
- Reviewed all MemWAL and Realtime Engine references under `docs/`.
- Verified each changed file differs only by the intended naming
substitutions and preserves code blocks, inline code, links, anchors,
and MemWAL identifiers.
- Confirmed no split or hyphenated "real time engine" names remain in
website guides.
- No snippets needed regeneration because code examples are unchanged.
Vale and Mintlify are not installed locally, so their checks were not
run locally.
2026-11-13 11:24:55 -08:00

138 lines
5.5 KiB
Plaintext

---
title: "Distributed RTE Write"
sidebarTitle: "Distributed RTE Write"
description: "How Enterprise scales RTE writes horizontally across a dedicated serving tier, and the throughput limits that come with it."
icon: "bolt"
keywords: ["rte", "realtime engine", "shard", "sharding", "horizontal scaling", "write throughput"]
availability:
embedded: unavailable
enterprise: available
---
[RTE writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
as one call that works the same way on either connection. On Enterprise, that
call does not run locally -- it posts to the server, which installs the write
path on a dedicated, horizontally scalable serving tier instead of running it
in your process. This page covers that tier: how it shards a table's writes,
the one real limit sharding does not remove, and the throughput you can expect.
## How writes are routed {#routing}
Enabling RTE on a remote table sends the spec to the server, which installs
the write path on the dataset instead of running it in your process. From
then on, the query node routes every request for that table -- reads and
writes alike -- to a single owning **RTE node**, a stateful component
separate from the query nodes and plan executors that serve ordinary
queries. A query node routes different tables to different RTE nodes, and
a single RTE node can host shards from many different tables at once. RTE
nodes share nothing but the underlying object storage; no node talks to
another one directly.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
Q["Query node<br/>routes each table's shards to an RTE node"]
subgraph W0["RTE node A"]
direction TB
S0["Shard 0 — table 'frames'"]
S1["Shard 0 — table 'events'"]
end
subgraph W1["RTE node B"]
direction TB
S2["Shard 0 — table 'metrics'"]
S3["Shard 1 — table 'metrics'"]
end
Q -- "'frames'" --> W0
Q -- "'events'" --> W0
Q -- "'metrics'" --> W1
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
Q["Query node<br/>routes each table's shards to an RTE node"]
subgraph W0["RTE node A"]
direction TB
S0["Shard 0 — table 'frames'"]
S1["Shard 0 — table 'events'"]
end
subgraph W1["RTE node B"]
direction TB
S2["Shard 0 — table 'metrics'"]
S3["Shard 1 — table 'metrics'"]
end
Q -- "'frames'" --> W0
Q -- "'events'" --> W0
Q -- "'metrics'" --> W1
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
</div>
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
**table shards**, each a separate, independently-appending stream, giving the
table that many parallel write paths instead of one. A shard's identity is
derived from the table itself, not assigned by an operator, so a shard
survives the table being claimed by a different RTE node later.
This is where the horizontal scaling actually happens: splitting `frames`
into 8 shards gives it 8 independent append streams instead of 1, raising
that table's own write ceiling. Adding RTE node replicas spreads shard
ownership across more nodes, so the cluster can host more tables -- and more
shards -- without any one node becoming a bottleneck for everything it serves.
`RTEWriteSpec::identity(column)` and `RTEWriteSpec::unsharded()` both resolve
to a single table shard -- `bucket` is the sharding mode that actually
parallelizes write throughput; the other two are about routing semantics, not
scaling.
## Throughput {#throughput}
A table shard is the unit of capacity: measured runs show it sustaining
roughly 85-90 MB/s of durable append before acknowledgement, and throughput
scaling nearly linearly with shard count up to at least 8 shards on one
node -- the bottleneck is the durable write itself, not CPU or memory. This
is observed capacity, not an enforced quota; nothing in the deployment caps
a shard's or a table's throughput below it. [Contact
support](https://www.lancedb.com/contact) if your workload needs more write
throughput than this provides.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="RTE Write (Upsert)" icon="bolt" href="/ingestion/realtime-write">
Enable RTE and write to a table; the same call on Embedded or Enterprise.
</Card>
<Card title="How the Realtime Engine works" icon="database" href="/tables/realtime-engine">
MemWAL, shards, memtables and SSTables -- the shared architecture.
</Card>
<Card title="Distributed RTE Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
How reads combine the WAL tier's fresh rows with the base table across the fleet.
</Card>
</CardGroup>