mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 15:46:55 +00:00
The Enterprise feature name was inconsistently written as "Real Time Engine" and "real time engine". Standardize it to **Realtime Engine** in eight website guides, including page and sidebar titles, descriptions, prose, link labels, cards, and search keywords. The remaining MemWAL references describe Lance's open source format or literal API/job identifiers, including `memwal_compaction` and `memwal_catchup_<type>`, and are preserved. Only `docs/web/*.mdx` pages change; SDK source, generated API reference, code examples, URL destinations, and section anchors are unchanged. Validation: - `git diff --check` passes. - Reviewed all MemWAL and Realtime Engine references under `docs/`. - Verified each changed file differs only by the intended naming substitutions and preserves code blocks, inline code, links, anchors, and MemWAL identifiers. - Confirmed no split or hyphenated "real time engine" names remain in website guides. - No snippets needed regeneration because code examples are unchanged. Vale and Mintlify are not installed locally, so their checks were not run locally.
94 lines
4.7 KiB
Plaintext
94 lines
4.7 KiB
Plaintext
---
|
|
title: "Realtime Engine (RTE)"
|
|
sidebarTitle: "Realtime Engine (RTE)"
|
|
description: "Absorb small, frequent writes cheaply and make them queryable within milliseconds."
|
|
icon: "bolt"
|
|
keywords: ["rte", "realtime engine", "wal", "memwal", "wal node", "lsm", "write-ahead log", "streaming writes", "upsert", "sharding"]
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
## What is the Realtime Engine? {#what-is-rte}
|
|
|
|
Every write to a LanceDB table is normally a commit to the underlying Lance
|
|
dataset. That is ideal for bulk loads, but a stream of small upserts becomes a
|
|
stream of small commits, and each write is only visible once its commit lands.
|
|
|
|
The Realtime Engine (RTE) puts an LSM-style write tier in front of the table.
|
|
It turns many small writes into a few large commits, and makes each write
|
|
**searchable within milliseconds** of being acknowledged. It is enabled per
|
|
table, and nothing about how you query that table changes.
|
|
|
|
## The Lance MemWAL format {#memwal-format}
|
|
|
|
RTE is built on **MemWAL**, a log-structured-merge (LSM) format that Lance
|
|
itself defines -- the structure Enterprise serves is part of the open Lance
|
|
format, not a layer bolted on beside it. A table with RTE enabled is divided
|
|
into one or more **shards** (see [RTE writes](/ingestion/realtime-write#primary-key)
|
|
for enabling and sizing them), and each shard holds:
|
|
|
|
- A **memtable**: the active, in-memory table for that shard's newest rows,
|
|
queryable as soon as a write lands in it.
|
|
- A **WAL buffer**: the same writes, batched and flushed to object storage
|
|
before a write is acknowledged, so an unflushed memtable is never the only
|
|
copy of a row.
|
|
- Its own **in-memory indexes** (vector, full-text, BTree, and other scalar
|
|
types), kept current as rows land, independently of the base table's
|
|
indexes.
|
|
|
|
A write commits in two stages, both before it is acknowledged: the WAL buffer
|
|
append makes it durable, then it lands in the shard's memtable. When a
|
|
memtable crosses a size or row-count threshold, it freezes and flushes as an
|
|
immutable **SSTable** -- an ordinary Lance data file -- and a fresh memtable
|
|
takes over. [RTE Compaction](/enterprise/realtime-engine-compaction)
|
|
later folds SSTables into the base table, catching its indexes up as it goes.
|
|
|
|
## WAL nodes serve MemWAL shards {#wal-nodes}
|
|
|
|
This section describes LanceDB Enterprise's serving layer. LanceDB Embedded
|
|
runs the same MemWAL mechanics directly against the local dataset, in one
|
|
process -- there is no separate node to route to or scale; see
|
|
[RTE writes](/ingestion/realtime-write#getting-started) and
|
|
[RTE reads](/search/realtime-read#embedded) for what that looks like.
|
|
|
|
RTE's Enterprise serving layer is a dedicated component, separate from the
|
|
[query nodes and plan executors](/enterprise/distributed-search)
|
|
that serve ordinary queries:
|
|
|
|
| Component | Responsibility |
|
|
| --- | --- |
|
|
| WAL node | Own one or more tables' shards; append their WAL, maintain their memtables and in-memory indexes, and serve reads and writes against them. |
|
|
|
|
A query node routes every request for an RTE-enabled table -- reads and
|
|
writes alike -- to the WAL node that owns that table, by a consistent hash of
|
|
the database and table name. WAL node pods share nothing but the underlying
|
|
object storage; no pod talks to another one directly.
|
|
|
|
One WAL node owns every shard of a given table. Splitting a table into more
|
|
shards parallelizes that table's own writes into more independent append
|
|
streams on the node that already owns it; adding more WAL node replicas
|
|
instead spreads ownership of more *tables* across more pods. Shard a large
|
|
table to raise its own ingest ceiling, and add WAL node replicas to raise the
|
|
cluster's overall capacity.
|
|
|
|
If a WAL node pod is replaced, its successor must claim ownership of a shard,
|
|
by a version number that only ever increases, before writing to it -- so two
|
|
pods are never both appending to the same shard's WAL at once.
|
|
|
|
## Reading a real time table {#reads}
|
|
|
|
Every query runs against the base table and the fresh tier -- every shard's
|
|
memtable and not-yet-compacted SSTables -- in parallel. Fresh rows supersede
|
|
base rows with the same primary key, and the two are merged into one result.
|
|
See [RTE reads](/search/realtime-read) for what this means for query
|
|
results.
|
|
|
|
## Next steps {#next-steps}
|
|
|
|
- [RTE writes](/ingestion/realtime-write) — enable RTE and write to a table
|
|
- [RTE reads](/search/realtime-read) — query an RTE table
|
|
- [Distributed RTE Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
|
|
- [Distributed RTE Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
|
|
- [RTE Compaction](/enterprise/realtime-engine-compaction) — how Enterprise folds shards into the base table
|