mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 15:46:55 +00:00
The Enterprise feature name was inconsistently written as "Real Time Engine" and "real time engine". Standardize it to **Realtime Engine** in eight website guides, including page and sidebar titles, descriptions, prose, link labels, cards, and search keywords. The remaining MemWAL references describe Lance's open source format or literal API/job identifiers, including `memwal_compaction` and `memwal_catchup_<type>`, and are preserved. Only `docs/web/*.mdx` pages change; SDK source, generated API reference, code examples, URL destinations, and section anchors are unchanged. Validation: - `git diff --check` passes. - Reviewed all MemWAL and Realtime Engine references under `docs/`. - Verified each changed file differs only by the intended naming substitutions and preserves code blocks, inline code, links, anchors, and MemWAL identifiers. - Confirmed no split or hyphenated "real time engine" names remain in website guides. - No snippets needed regeneration because code examples are unchanged. Vale and Mintlify are not installed locally, so their checks were not run locally.
138 lines
5.5 KiB
Plaintext
138 lines
5.5 KiB
Plaintext
---
|
|
title: "Distributed RTE Write"
|
|
sidebarTitle: "Distributed RTE Write"
|
|
description: "How Enterprise scales RTE writes horizontally across a dedicated serving tier, and the throughput limits that come with it."
|
|
icon: "bolt"
|
|
keywords: ["rte", "realtime engine", "shard", "sharding", "horizontal scaling", "write throughput"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
[RTE writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
|
|
as one call that works the same way on either connection. On Enterprise, that
|
|
call does not run locally -- it posts to the server, which installs the write
|
|
path on a dedicated, horizontally scalable serving tier instead of running it
|
|
in your process. This page covers that tier: how it shards a table's writes,
|
|
the one real limit sharding does not remove, and the throughput you can expect.
|
|
|
|
## How writes are routed {#routing}
|
|
|
|
Enabling RTE on a remote table sends the spec to the server, which installs
|
|
the write path on the dataset instead of running it in your process. From
|
|
then on, the query node routes every request for that table -- reads and
|
|
writes alike -- to a single owning **RTE node**, a stateful component
|
|
separate from the query nodes and plan executors that serve ordinary
|
|
queries. A query node routes different tables to different RTE nodes, and
|
|
a single RTE node can host shards from many different tables at once. RTE
|
|
nodes share nothing but the underlying object storage; no node talks to
|
|
another one directly.
|
|
|
|
<div className="hidden dark:block">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
|
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
|
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
|
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
Q["Query node<br/>routes each table's shards to an RTE node"]
|
|
|
|
subgraph W0["RTE node A"]
|
|
direction TB
|
|
S0["Shard 0 — table 'frames'"]
|
|
S1["Shard 0 — table 'events'"]
|
|
end
|
|
|
|
subgraph W1["RTE node B"]
|
|
direction TB
|
|
S2["Shard 0 — table 'metrics'"]
|
|
S3["Shard 1 — table 'metrics'"]
|
|
end
|
|
|
|
Q -- "'frames'" --> W0
|
|
Q -- "'events'" --> W0
|
|
Q -- "'metrics'" --> W1
|
|
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
|
```
|
|
|
|
</div>
|
|
|
|
<div className="block dark:hidden">
|
|
|
|
```mermaid
|
|
%%{init: {"theme": "base", "themeVariables": {
|
|
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
|
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
|
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
|
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
|
}}}%%
|
|
flowchart LR
|
|
Q["Query node<br/>routes each table's shards to an RTE node"]
|
|
|
|
subgraph W0["RTE node A"]
|
|
direction TB
|
|
S0["Shard 0 — table 'frames'"]
|
|
S1["Shard 0 — table 'events'"]
|
|
end
|
|
|
|
subgraph W1["RTE node B"]
|
|
direction TB
|
|
S2["Shard 0 — table 'metrics'"]
|
|
S3["Shard 1 — table 'metrics'"]
|
|
end
|
|
|
|
Q -- "'frames'" --> W0
|
|
Q -- "'events'" --> W0
|
|
Q -- "'metrics'" --> W1
|
|
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
|
```
|
|
|
|
</div>
|
|
|
|
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
|
|
|
|
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
|
|
**table shards**, each a separate, independently-appending stream, giving the
|
|
table that many parallel write paths instead of one. A shard's identity is
|
|
derived from the table itself, not assigned by an operator, so a shard
|
|
survives the table being claimed by a different RTE node later.
|
|
|
|
This is where the horizontal scaling actually happens: splitting `frames`
|
|
into 8 shards gives it 8 independent append streams instead of 1, raising
|
|
that table's own write ceiling. Adding RTE node replicas spreads shard
|
|
ownership across more nodes, so the cluster can host more tables -- and more
|
|
shards -- without any one node becoming a bottleneck for everything it serves.
|
|
|
|
`RTEWriteSpec::identity(column)` and `RTEWriteSpec::unsharded()` both resolve
|
|
to a single table shard -- `bucket` is the sharding mode that actually
|
|
parallelizes write throughput; the other two are about routing semantics, not
|
|
scaling.
|
|
|
|
## Throughput {#throughput}
|
|
|
|
A table shard is the unit of capacity: measured runs show it sustaining
|
|
roughly 85-90 MB/s of durable append before acknowledgement, and throughput
|
|
scaling nearly linearly with shard count up to at least 8 shards on one
|
|
node -- the bottleneck is the durable write itself, not CPU or memory. This
|
|
is observed capacity, not an enforced quota; nothing in the deployment caps
|
|
a shard's or a table's throughput below it. [Contact
|
|
support](https://www.lancedb.com/contact) if your workload needs more write
|
|
throughput than this provides.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="RTE Write (Upsert)" icon="bolt" href="/ingestion/realtime-write">
|
|
Enable RTE and write to a table; the same call on Embedded or Enterprise.
|
|
</Card>
|
|
<Card title="How the Realtime Engine works" icon="database" href="/tables/realtime-engine">
|
|
MemWAL, shards, memtables and SSTables -- the shared architecture.
|
|
</Card>
|
|
<Card title="Distributed RTE Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
|
|
How reads combine the WAL tier's fresh rows with the base table across the fleet.
|
|
</Card>
|
|
</CardGroup>
|