mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
Jack YeandClaude Sonnet 5
5e38120bdc
## Summary - **Define Storage Integration**: un-gates the Python (and Rust, for S3) `create_storage_integration` examples, drafted against #4618 and held back until it shipped -- it's now in v0.42.0-beta.0+, and the signature matches the drafts exactly. - **Naming rules**: drops the `Branch` row and its "Learn more" link, consistent with de-emphasizing branches elsewhere in this nav. - **Distributed RTE Write**: removes the REST path, the `__lance_mem_wal` index name, and "WAL node" terminology in favor of an abstract framing -- a query node routes writes to RTE nodes, each of which can host shards from many different tables. Both diagram variants are updated to match. - **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop Helm-specific and exact-percentage detail (disk-budget math, `startup_prewarm_block_query_ready`/`healthPort`/failure-policy table) in favor of a short feature → architecture → why-it-matters shape, each closing with a contact-sales pointer. Data Page Cache's on-demand prewarm section is removed outright. ## Note I wasn't able to get a clean visual render of the updated Distributed RTE Write diagram in local preview -- Mermaid isn't rendering in headless Chrome in this environment for *any* page right now, confirmed against an unmodified reference page showing the identical symptom. The diagram source follows the exact pattern already shipped and working in the 21 diagrams from the earlier brand-theming pass, and `mint validate`/`mint broken-links` both pass clean. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
94 lines
4.7 KiB
Plaintext
94 lines
4.7 KiB
Plaintext
---
|
|
title: "Real Time engine (RTE)"
|
|
sidebarTitle: "Real Time Engine (RTE)"
|
|
description: "Absorb small, frequent writes cheaply and make them queryable within milliseconds."
|
|
icon: "bolt"
|
|
keywords: ["rte", "real time engine", "wal", "memwal", "wal node", "lsm", "write-ahead log", "streaming writes", "upsert", "sharding"]
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
## What is the real time engine? {#what-is-rte}
|
|
|
|
Every write to a LanceDB table is normally a commit to the underlying Lance
|
|
dataset. That is ideal for bulk loads, but a stream of small upserts becomes a
|
|
stream of small commits, and each write is only visible once its commit lands.
|
|
|
|
The real time engine (RTE) puts an LSM-style write tier in front of the table.
|
|
It turns many small writes into a few large commits, and makes each write
|
|
**searchable within milliseconds** of being acknowledged. It is enabled per
|
|
table, and nothing about how you query that table changes.
|
|
|
|
## The Lance MemWAL format {#memwal-format}
|
|
|
|
RTE is built on **MemWAL**, a log-structured-merge (LSM) format that Lance
|
|
itself defines -- the structure Enterprise serves is part of the open Lance
|
|
format, not a layer bolted on beside it. A table with RTE enabled is divided
|
|
into one or more **shards** (see [RTE writes](/ingestion/realtime-write#primary-key)
|
|
for enabling and sizing them), and each shard holds:
|
|
|
|
- A **memtable**: the active, in-memory table for that shard's newest rows,
|
|
queryable as soon as a write lands in it.
|
|
- A **WAL buffer**: the same writes, batched and flushed to object storage
|
|
before a write is acknowledged, so an unflushed memtable is never the only
|
|
copy of a row.
|
|
- Its own **in-memory indexes** (vector, full-text, BTree, and other scalar
|
|
types), kept current as rows land, independently of the base table's
|
|
indexes.
|
|
|
|
A write commits in two stages, both before it is acknowledged: the WAL buffer
|
|
append makes it durable, then it lands in the shard's memtable. When a
|
|
memtable crosses a size or row-count threshold, it freezes and flushes as an
|
|
immutable **SSTable** -- an ordinary Lance data file -- and a fresh memtable
|
|
takes over. [RTE Compaction](/enterprise/realtime-engine-compaction)
|
|
later folds SSTables into the base table, catching its indexes up as it goes.
|
|
|
|
## WAL nodes serve MemWAL shards {#wal-nodes}
|
|
|
|
This section describes LanceDB Enterprise's serving layer. LanceDB Embedded
|
|
runs the same MemWAL mechanics directly against the local dataset, in one
|
|
process -- there is no separate node to route to or scale; see
|
|
[RTE writes](/ingestion/realtime-write#getting-started) and
|
|
[RTE reads](/search/realtime-read#embedded) for what that looks like.
|
|
|
|
RTE's Enterprise serving layer is a dedicated component, separate from the
|
|
[query nodes and plan executors](/enterprise/distributed-search)
|
|
that serve ordinary queries:
|
|
|
|
| Component | Responsibility |
|
|
| --- | --- |
|
|
| WAL node | Own one or more tables' shards; append their WAL, maintain their memtables and in-memory indexes, and serve reads and writes against them. |
|
|
|
|
A query node routes every request for an RTE-enabled table -- reads and
|
|
writes alike -- to the WAL node that owns that table, by a consistent hash of
|
|
the database and table name. WAL node pods share nothing but the underlying
|
|
object storage; no pod talks to another one directly.
|
|
|
|
One WAL node owns every shard of a given table. Splitting a table into more
|
|
shards parallelizes that table's own writes into more independent append
|
|
streams on the node that already owns it; adding more WAL node replicas
|
|
instead spreads ownership of more *tables* across more pods. Shard a large
|
|
table to raise its own ingest ceiling, and add WAL node replicas to raise the
|
|
cluster's overall capacity.
|
|
|
|
If a WAL node pod is replaced, its successor must claim ownership of a shard,
|
|
by a version number that only ever increases, before writing to it -- so two
|
|
pods are never both appending to the same shard's WAL at once.
|
|
|
|
## Reading a real time table {#reads}
|
|
|
|
Every query runs against the base table and the fresh tier -- every shard's
|
|
memtable and not-yet-compacted SSTables -- in parallel. Fresh rows supersede
|
|
base rows with the same primary key, and the two are merged into one result.
|
|
See [RTE reads](/search/realtime-read) for what this means for query
|
|
results.
|
|
|
|
## Next steps {#next-steps}
|
|
|
|
- [RTE writes](/ingestion/realtime-write) — enable RTE and write to a table
|
|
- [RTE reads](/search/realtime-read) — query an RTE table
|
|
- [Distributed RTE Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
|
|
- [Distributed RTE Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
|
|
- [RTE Compaction](/enterprise/realtime-engine-compaction) — how Enterprise folds shards into the base table
|