Files
lancedb/docs/web/tables/realtime-engine.mdx
T
Taylor 3a980054a1 docs: standardize Realtime Engine naming (#4969)
The Enterprise feature name was inconsistently written as "Real Time
Engine" and "real time engine". Standardize it to **Realtime Engine** in
eight website guides, including page and sidebar titles, descriptions,
prose, link labels, cards, and search keywords.

The remaining MemWAL references describe Lance's open source format or
literal API/job identifiers, including `memwal_compaction` and
`memwal_catchup_<type>`, and are preserved. Only `docs/web/*.mdx` pages
change; SDK source, generated API reference, code examples, URL
destinations, and section anchors are unchanged.

Validation:
- `git diff --check` passes.
- Reviewed all MemWAL and Realtime Engine references under `docs/`.
- Verified each changed file differs only by the intended naming
substitutions and preserves code blocks, inline code, links, anchors,
and MemWAL identifiers.
- Confirmed no split or hyphenated "real time engine" names remain in
website guides.
- No snippets needed regeneration because code examples are unchanged.
Vale and Mintlify are not installed locally, so their checks were not
run locally.
2026-11-13 11:24:55 -08:00

94 lines
4.7 KiB
Plaintext

---
title: "Realtime Engine (RTE)"
sidebarTitle: "Realtime Engine (RTE)"
description: "Absorb small, frequent writes cheaply and make them queryable within milliseconds."
icon: "bolt"
keywords: ["rte", "realtime engine", "wal", "memwal", "wal node", "lsm", "write-ahead log", "streaming writes", "upsert", "sharding"]
availability:
embedded: available
enterprise: available
---
## What is the Realtime Engine? {#what-is-rte}
Every write to a LanceDB table is normally a commit to the underlying Lance
dataset. That is ideal for bulk loads, but a stream of small upserts becomes a
stream of small commits, and each write is only visible once its commit lands.
The Realtime Engine (RTE) puts an LSM-style write tier in front of the table.
It turns many small writes into a few large commits, and makes each write
**searchable within milliseconds** of being acknowledged. It is enabled per
table, and nothing about how you query that table changes.
## The Lance MemWAL format {#memwal-format}
RTE is built on **MemWAL**, a log-structured-merge (LSM) format that Lance
itself defines -- the structure Enterprise serves is part of the open Lance
format, not a layer bolted on beside it. A table with RTE enabled is divided
into one or more **shards** (see [RTE writes](/ingestion/realtime-write#primary-key)
for enabling and sizing them), and each shard holds:
- A **memtable**: the active, in-memory table for that shard's newest rows,
queryable as soon as a write lands in it.
- A **WAL buffer**: the same writes, batched and flushed to object storage
before a write is acknowledged, so an unflushed memtable is never the only
copy of a row.
- Its own **in-memory indexes** (vector, full-text, BTree, and other scalar
types), kept current as rows land, independently of the base table's
indexes.
A write commits in two stages, both before it is acknowledged: the WAL buffer
append makes it durable, then it lands in the shard's memtable. When a
memtable crosses a size or row-count threshold, it freezes and flushes as an
immutable **SSTable** -- an ordinary Lance data file -- and a fresh memtable
takes over. [RTE Compaction](/enterprise/realtime-engine-compaction)
later folds SSTables into the base table, catching its indexes up as it goes.
## WAL nodes serve MemWAL shards {#wal-nodes}
This section describes LanceDB Enterprise's serving layer. LanceDB Embedded
runs the same MemWAL mechanics directly against the local dataset, in one
process -- there is no separate node to route to or scale; see
[RTE writes](/ingestion/realtime-write#getting-started) and
[RTE reads](/search/realtime-read#embedded) for what that looks like.
RTE's Enterprise serving layer is a dedicated component, separate from the
[query nodes and plan executors](/enterprise/distributed-search)
that serve ordinary queries:
| Component | Responsibility |
| --- | --- |
| WAL node | Own one or more tables' shards; append their WAL, maintain their memtables and in-memory indexes, and serve reads and writes against them. |
A query node routes every request for an RTE-enabled table -- reads and
writes alike -- to the WAL node that owns that table, by a consistent hash of
the database and table name. WAL node pods share nothing but the underlying
object storage; no pod talks to another one directly.
One WAL node owns every shard of a given table. Splitting a table into more
shards parallelizes that table's own writes into more independent append
streams on the node that already owns it; adding more WAL node replicas
instead spreads ownership of more *tables* across more pods. Shard a large
table to raise its own ingest ceiling, and add WAL node replicas to raise the
cluster's overall capacity.
If a WAL node pod is replaced, its successor must claim ownership of a shard,
by a version number that only ever increases, before writing to it -- so two
pods are never both appending to the same shard's WAL at once.
## Reading a real time table {#reads}
Every query runs against the base table and the fresh tier -- every shard's
memtable and not-yet-compacted SSTables -- in parallel. Fresh rows supersede
base rows with the same primary key, and the two are merged into one result.
See [RTE reads](/search/realtime-read) for what this means for query
results.
## Next steps {#next-steps}
- [RTE writes](/ingestion/realtime-write) — enable RTE and write to a table
- [RTE reads](/search/realtime-read) — query an RTE table
- [Distributed RTE Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
- [Distributed RTE Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
- [RTE Compaction](/enterprise/realtime-engine-compaction) — how Enterprise folds shards into the base table