Files
lancedb/docs/web/enterprise/distributed-real-time-write.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

101 lines
4.5 KiB
Plaintext

---
title: "Distributed Real Time Write"
sidebarTitle: "Distributed RTE Write"
description: "How Enterprise scales real time writes horizontally across a dedicated WAL-node tier, and the throughput limits that come with it."
icon: "bolt"
keywords: ["rte", "real time engine", "wal node", "shard", "sharding", "horizontal scaling", "write throughput", "memwal"]
availability:
embedded: unavailable
enterprise: available
---
[Real Time writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
as one call that works the same way on either connection. On Enterprise, that
call does not run locally -- it posts to the server, which installs the write
path on a dedicated, horizontally scalable serving tier instead of running it
in your process. This page covers that tier: how it shards a table's writes,
the one real limit sharding does not remove, and the throughput you can expect.
## The WAL-node tier {#wal-nodes}
Enabling RTE on a remote table sends the spec to the server
(`POST /v1/table/<name>/enable_rte/`), which installs Lance's `__lance_mem_wal`
index on the dataset. From then on, every request for that table -- reads and
writes alike -- is consistent-hashed on the database and table name to a
single owning **WAL node**, a stateful pod separate from the query nodes and
plan executors that serve ordinary queries. WAL node pods share nothing but
the underlying object storage; no pod talks to another one directly.
```mermaid
flowchart LR
Q["Query node<br/>hashes (db, table) to one WAL node"]
subgraph W0["WAL node A — owns table 'frames'"]
direction TB
S0["Shard 0<br/>ShardWriter"]
S1["Shard 1<br/>ShardWriter"]
S2["Shard 2<br/>ShardWriter"]
end
subgraph W1["WAL node B — owns table 'events'"]
direction TB
S3["Shard 0<br/>ShardWriter"]
end
Q -- "writes + reads for 'frames'" --> W0
Q -- "writes + reads for 'events'" --> W1
```
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
**table shards**, each a separate, independently-appending `ShardWriter`. A
shard's identity is derived from the table's own URI
(`uuid_v5(WAL_SHARD_NS, "{table_uri}#{i}")`), not assigned by an operator, so
a shard survives the table being claimed by a different pod later.
This is where the horizontal scaling actually happens, and where its limit
is: **one WAL node owns every shard of a given table.** Splitting `frames`
into 8 shards gives it 8 independent append streams, all still served by the
one pod that owns `frames` -- it raises that table's own ceiling, not the
number of pods working on it. Adding WAL node replicas instead spreads
*other* tables' ownership across more pods, raising how many tables the
cluster can serve, not how fast any single one of them can write.
<Note>
A single table's write throughput is capped by one pod's capacity, however
many shards you give it. A workload that needs to exceed one pod's ceiling on
one table is not solved by adding shards -- there is no path to spreading one
table's shards across multiple WAL nodes today.
</Note>
`RTEWriteSpec::identity(column)` and `RTEWriteSpec::unsharded()` both resolve
to a single table shard -- `bucket` is the sharding mode that actually
parallelizes write throughput; the other two are about routing semantics, not
scaling.
## Throughput {#throughput}
A table shard is the unit of capacity: measured runs show it sustaining
roughly 85-90 MB/s of durable append before acknowledgement, and throughput
scaling nearly linearly with shard count up to at least 8 shards on one
pod -- the bottleneck is the durable write itself, not CPU or memory. This
is observed capacity, not an enforced quota; nothing in the deployment caps
a shard's or a table's throughput below it. [Contact
support](https://www.lancedb.com/contact) if your workload needs more than
one pod's worth of capacity for a single table.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Real Time writes" icon="bolt" href="/ingestion/realtime-write">
Enable RTE and write to a table; the same call on Embedded or Enterprise.
</Card>
<Card title="How the real time engine works" icon="database" href="/tables/realtime-engine">
MemWAL, shards, memtables and SSTables -- the shared architecture.
</Card>
<Card title="Distributed Real Time Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
How reads combine the WAL tier's fresh rows with the base table across the fleet.
</Card>
</CardGroup>