lancedb/docs/web/tables/realtime-engine.mdx

---
title: "Real Time engine (RTE)"
sidebarTitle: "Real Time Engine (RTE)"
description: "Absorb small, frequent writes cheaply and make them queryable within milliseconds."
icon: "bolt"
keywords: ["rte", "real time engine", "wal", "memwal", "wal node", "lsm", "write-ahead log", "streaming writes", "upsert", "sharding"]
availability:
  embedded: available
  enterprise: available
---

## What is the real time engine? {#what-is-rte}

Every write to a LanceDB table is normally a commit to the underlying Lance
dataset. That is ideal for bulk loads, but a stream of small upserts becomes a
stream of small commits, and each write is only visible once its commit lands.

The real time engine (RTE) puts an LSM-style write tier in front of the table.
It turns many small writes into a few large commits, and makes each write
**searchable within milliseconds** of being acknowledged. It is enabled per
table, and nothing about how you query that table changes.

## The Lance MemWAL format {#memwal-format}

RTE is built on **MemWAL**, a log-structured-merge (LSM) format that Lance
itself defines -- the structure Enterprise serves is part of the open Lance
format, not a layer bolted on beside it. A table with RTE enabled is divided
into one or more **shards** (see [Real Time writes](/ingestion/realtime-write#primary-key)
for enabling and sizing them), and each shard holds:

- A **memtable**: the active, in-memory table for that shard's newest rows,
  queryable as soon as a write lands in it.
- A **WAL buffer**: the same writes, batched and flushed to object storage
  before a write is acknowledged, so an unflushed memtable is never the only
  copy of a row.
- Its own **in-memory indexes** (vector, full-text, BTree, and other scalar
  types), kept current as rows land, independently of the base table's
  indexes.

A write commits in two stages, both before it is acknowledged: the WAL buffer
append makes it durable, then it lands in the shard's memtable. When a
memtable crosses a size or row-count threshold, it freezes and flushes as an
immutable **SSTable** -- an ordinary Lance data file -- and a fresh memtable
takes over. [RTE Compaction](/enterprise/realtime-engine-compaction)
later folds SSTables into the base table, catching its indexes up as it goes.

## WAL nodes serve MemWAL shards {#wal-nodes}

This section describes LanceDB Enterprise's serving layer. LanceDB Embedded
runs the same MemWAL mechanics directly against the local dataset, in one
process -- there is no separate node to route to or scale; see
[Real Time writes](/ingestion/realtime-write#getting-started) and
[Real Time reads](/search/realtime-read#embedded) for what that looks like.

RTE's Enterprise serving layer is a dedicated component, separate from the
[query nodes and plan executors](/enterprise/distributed-search)
that serve ordinary queries:

| Component | Responsibility |
| --- | --- |
| WAL node | Own one or more tables' shards; append their WAL, maintain their memtables and in-memory indexes, and serve reads and writes against them. |

A query node routes every request for an RTE-enabled table -- reads and
writes alike -- to the WAL node that owns that table, by a consistent hash of
the database and table name. WAL node pods share nothing but the underlying
object storage; no pod talks to another one directly.

One WAL node owns every shard of a given table. Splitting a table into more
shards parallelizes that table's own writes into more independent append
streams on the node that already owns it; adding more WAL node replicas
instead spreads ownership of more *tables* across more pods. Shard a large
table to raise its own ingest ceiling, and add WAL node replicas to raise the
cluster's overall capacity.

If a WAL node pod is replaced, its successor must claim ownership of a shard,
by a version number that only ever increases, before writing to it -- so two
pods are never both appending to the same shard's WAL at once.

## Reading a real time table {#reads}

Every query runs against the base table and the fresh tier -- every shard's
memtable and not-yet-compacted SSTables -- in parallel. Fresh rows supersede
base rows with the same primary key, and the two are merged into one result.
See [Real Time reads](/search/realtime-read) for what this means for query
results.

## Next steps {#next-steps}

- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table
- [Real Time reads](/search/realtime-read) — query an RTE table
- [Distributed Real Time Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
- [Distributed Real Time Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
- [RTE Compaction](/enterprise/realtime-engine-compaction) — how Enterprise folds shards into the base table