mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-18 16:43:59 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
101 lines
4.5 KiB
Plaintext
101 lines
4.5 KiB
Plaintext
---
|
|
title: "Distributed Real Time Write"
|
|
sidebarTitle: "Distributed RTE Write"
|
|
description: "How Enterprise scales real time writes horizontally across a dedicated WAL-node tier, and the throughput limits that come with it."
|
|
icon: "bolt"
|
|
keywords: ["rte", "real time engine", "wal node", "shard", "sharding", "horizontal scaling", "write throughput", "memwal"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
[Real Time writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
|
|
as one call that works the same way on either connection. On Enterprise, that
|
|
call does not run locally -- it posts to the server, which installs the write
|
|
path on a dedicated, horizontally scalable serving tier instead of running it
|
|
in your process. This page covers that tier: how it shards a table's writes,
|
|
the one real limit sharding does not remove, and the throughput you can expect.
|
|
|
|
## The WAL-node tier {#wal-nodes}
|
|
|
|
Enabling RTE on a remote table sends the spec to the server
|
|
(`POST /v1/table/<name>/enable_rte/`), which installs Lance's `__lance_mem_wal`
|
|
index on the dataset. From then on, every request for that table -- reads and
|
|
writes alike -- is consistent-hashed on the database and table name to a
|
|
single owning **WAL node**, a stateful pod separate from the query nodes and
|
|
plan executors that serve ordinary queries. WAL node pods share nothing but
|
|
the underlying object storage; no pod talks to another one directly.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Q["Query node<br/>hashes (db, table) to one WAL node"]
|
|
|
|
subgraph W0["WAL node A — owns table 'frames'"]
|
|
direction TB
|
|
S0["Shard 0<br/>ShardWriter"]
|
|
S1["Shard 1<br/>ShardWriter"]
|
|
S2["Shard 2<br/>ShardWriter"]
|
|
end
|
|
|
|
subgraph W1["WAL node B — owns table 'events'"]
|
|
direction TB
|
|
S3["Shard 0<br/>ShardWriter"]
|
|
end
|
|
|
|
Q -- "writes + reads for 'frames'" --> W0
|
|
Q -- "writes + reads for 'events'" --> W1
|
|
```
|
|
|
|
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
|
|
|
|
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
|
|
**table shards**, each a separate, independently-appending `ShardWriter`. A
|
|
shard's identity is derived from the table's own URI
|
|
(`uuid_v5(WAL_SHARD_NS, "{table_uri}#{i}")`), not assigned by an operator, so
|
|
a shard survives the table being claimed by a different pod later.
|
|
|
|
This is where the horizontal scaling actually happens, and where its limit
|
|
is: **one WAL node owns every shard of a given table.** Splitting `frames`
|
|
into 8 shards gives it 8 independent append streams, all still served by the
|
|
one pod that owns `frames` -- it raises that table's own ceiling, not the
|
|
number of pods working on it. Adding WAL node replicas instead spreads
|
|
*other* tables' ownership across more pods, raising how many tables the
|
|
cluster can serve, not how fast any single one of them can write.
|
|
|
|
<Note>
|
|
A single table's write throughput is capped by one pod's capacity, however
|
|
many shards you give it. A workload that needs to exceed one pod's ceiling on
|
|
one table is not solved by adding shards -- there is no path to spreading one
|
|
table's shards across multiple WAL nodes today.
|
|
</Note>
|
|
|
|
`RTEWriteSpec::identity(column)` and `RTEWriteSpec::unsharded()` both resolve
|
|
to a single table shard -- `bucket` is the sharding mode that actually
|
|
parallelizes write throughput; the other two are about routing semantics, not
|
|
scaling.
|
|
|
|
## Throughput {#throughput}
|
|
|
|
A table shard is the unit of capacity: measured runs show it sustaining
|
|
roughly 85-90 MB/s of durable append before acknowledgement, and throughput
|
|
scaling nearly linearly with shard count up to at least 8 shards on one
|
|
pod -- the bottleneck is the durable write itself, not CPU or memory. This
|
|
is observed capacity, not an enforced quota; nothing in the deployment caps
|
|
a shard's or a table's throughput below it. [Contact
|
|
support](https://www.lancedb.com/contact) if your workload needs more than
|
|
one pod's worth of capacity for a single table.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Real Time writes" icon="bolt" href="/ingestion/realtime-write">
|
|
Enable RTE and write to a table; the same call on Embedded or Enterprise.
|
|
</Card>
|
|
<Card title="How the real time engine works" icon="database" href="/tables/realtime-engine">
|
|
MemWAL, shards, memtables and SSTables -- the shared architecture.
|
|
</Card>
|
|
<Card title="Distributed Real Time Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
|
|
How reads combine the WAL tier's fresh rows with the base table across the fleet.
|
|
</Card>
|
|
</CardGroup>
|