docs: add storage integration SDK examples, abstract RTE write, simplify cache pages (#4865)

## Summary

- **Define Storage Integration**: un-gates the Python (and Rust, for S3)
`create_storage_integration` examples, drafted against #4618 and held
back until it shipped -- it's now in v0.42.0-beta.0+, and the signature
matches the drafts exactly.
- **Naming rules**: drops the `Branch` row and its "Learn more" link,
consistent with de-emphasizing branches elsewhere in this nav.
- **Distributed RTE Write**: removes the REST path, the
`__lance_mem_wal` index name, and "WAL node" terminology in favor of an
abstract framing -- a query node routes writes to RTE nodes, each of
which can host shards from many different tables. Both diagram variants
are updated to match.
- **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop
Helm-specific and exact-percentage detail (disk-budget math,
`startup_prewarm_block_query_ready`/`healthPort`/failure-policy table)
in favor of a short feature → architecture → why-it-matters shape, each
closing with a contact-sales pointer. Data Page Cache's on-demand
prewarm section is removed outright.

## Note

I wasn't able to get a clean visual render of the updated Distributed
RTE Write diagram in local preview -- Mermaid isn't rendering in
headless Chrome in this environment for *any* page right now, confirmed
against an unmodified reference page showing the identical symptom. The
diagram source follows the exact pattern already shipped and working in
the 21 diagrams from the earlier brand-theming pass, and `mint
validate`/`mint broken-links` both pass clean.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Jack Ye
2026-11-05 01:40:54 -08:00
committed by GitHub
co-authored by Claude Sonnet 5
parent 69ac2158a1
commit 5e38120bdc
35 changed files with 597 additions and 622 deletions
+5 -7
View File
@@ -42,7 +42,7 @@ file format, so moving data from one edition to the other does not require a dat
| | [Hybrid search](/search/hybrid-search) | ✅ | ✅ |
| | [Filtering](/search/filtering) | ✅ | ✅ |
| | [Reranking](/reranking) | ✅ | ✅ |
| | [Distributed search](/enterprise/distributed-search) | ❌ | ✅ |
| | [Distributed Search](/enterprise/distributed-search) | ❌ | ✅ |
| | [RTE Read](/search/realtime-read) | ✅ | ✅ |
| | [Distributed RTE Read](/enterprise/distributed-real-time-read) | ❌ | ✅ |
| | [Cache Prewarm](/enterprise/cache-prewarm) | ❌ | ✅ |
@@ -52,19 +52,15 @@ file format, so moving data from one edition to the other does not require a dat
| | [Aggregation and profiling](/curation/aggregate) | ❌ | ✅ |
| | [Joins](/curation/join) | ❌ | ✅ |
| | [Blob Late Materialization](/curation/blob-late-materialization) | ❌ | ✅ |
| | [Ranking and deduplication](/curation/windowing) | ❌ | ✅ |
| | [Ranking](/curation/windowing) | ❌ | ✅ |
| | [Near-duplicate detection](/curation/dedupe) | ❌ | ✅ |
| | [Chunking](/curation/chunking) | ❌ | ✅ |
| | [Changes since a version](/curation/changes) | ❌ | ✅ |
| | [GPU Function Execution](/curation/gpu-function-execution) | ❌ | ✅ |
| | [Slurm Scheduling](/curation/slurm-scheduling) | ❌ | ✅ |
| **Feature Engineering** | [Computed columns](/feature-engineering/add-computed-column) | ❌ | ✅ |
| | [Incremental Refresh](/enterprise/incremental-refresh) | ❌ | ✅ |
| | [Auto Refresh](/enterprise/auto-refresh) | ❌ | ✅ |
| | [Checkpointing](/enterprise/checkpointing) | ❌ | ✅ |
| | [Adaptive Execution](/enterprise/adaptive-execution) | ❌ | ✅ |
| | [GPU Refresh Execution](/enterprise/gpu-refresh-execution) | ❌ | ✅ |
| | [Slurm Scheduling](/enterprise/slurm-refresh-scheduling) | ❌ | ✅ |
| | [Refresh Materialized View](/feature-engineering/refresh-materialized-view) | ❌ | ✅ |
| | [Embedding registry](/embedding) | ✅ | ✅ |
| **Training** | [PyTorch data loader](/training/data-loading) | ✅ | ✅ |
@@ -94,7 +90,7 @@ file format, so moving data from one edition to the other does not require a dat
| | [Compaction](/maintenance/compaction) | ✅ | ✅ |
| | [RTE Compaction](/enterprise/realtime-engine-compaction) | ❌ | ✅ |
| | [Garbage Collection](/maintenance/cleanup) | ✅ | ✅ |
| | [Distributed compaction](/enterprise/distributed-compaction) | ❌ | ✅ |
| | [Distributed Compaction](/enterprise/distributed-compaction) | ❌ | ✅ |
| | [Advanced Conflict Resolution](/enterprise/advanced-conflict-resolution) | ❌ | ✅ |
| **Security** | [Authentication](/enterprise/authentication) | ❌ | ✅ |
| | [Authorization](/governance/authorization) | ❌ | ✅ |
@@ -102,6 +98,8 @@ file format, so moving data from one edition to the other does not require a dat
| **Deployment** | [Horizontal Scaling](/enterprise/horizontal-scaling) | ❌ | ✅ |
| | [OpenTelemetry](/deployment/opentelemetry) | ✅ | ✅ |
| | [Kubernetes](/enterprise/deployment/kubernetes) | ❌ | ✅ |
| | [Multicloud](/deployment/multicloud) | ❌ | ✅ |
| | [Slurm](/deployment/slurm) | ❌ | ✅ |
{/*
Draft until branching reaches early access; restore to the Management rows.
+13 -37
View File
@@ -227,45 +227,21 @@ described below.
### SQL-backed job handles
On Enterprise, materialized-view CREATE, REFRESH, and DROP use the SQL query backend.
Python's `create_materialized_view_async()`, `mv.refresh_async()`, and
`drop_materialized_view_async()` return job handles with `status()`, `wait()`,
and `cancel()`. Synchronous creation, refresh, and drop wait on the same SQL
execution before returning. A completed DROP means the catalog binding has been
removed; physical cleanup continues in the background.
On Enterprise, several catalog operations return a job handle backed by the
SQL query engine rather than the job registry:
Remote function registration also returns a SQL query job. The SQL operation
submits the durable image build and waits for catalog publication before returning
the immutable FunctionVersion. `drop_function_async()` returns the catalog-removal
boolean together with a query job; waiting on that handle confirms removal while
physical cleanup continues separately.
| Operation | SDK methods | Behavior |
| --- | --- | --- |
| Materialized views | `create_materialized_view_async()`, `refresh_async()`, `drop_materialized_view_async()` | Synchronous variants wait on the same SQL execution. Drop removes the catalog binding immediately; physical cleanup continues in the background. |
| Functions | Registration, `drop_function_async()` | Registration waits for catalog publication before returning the immutable FunctionVersion. Drop confirms catalog removal while cleanup continues separately. |
| Secrets | Create, alter, list, describe, drop | Credential values are bound as SQL parameters, never embedded in SQL text or returned in results. `DESCRIBE SECRET` additionally exposes rotation history. |
| Views | Create, describe, list, drop | Create returns the stored definition and schema. Drop confirms catalog removal while cleanup continues separately. |
Remote Secret create, alter, list, describe, and drop use the SQL query backend.
The synchronous APIs wait on query completion internally. Create and alter bind
the credential as a parameter to
`CREATE SECRET name TYPE = GENERIC_STRING SECRET_STRING = $1` or
`ALTER SECRET name SET SECRET_STRING = $1`, keeping it out of SQL text. Results contain only
completion status or Secret names and timestamps; they never return credential
values. SQL `DESCRIBE SECRET` additionally exposes rotation history.
Remote view creation, description, listing, and drop also use SQL. Creation
returns the stored definition and schema from the same SQL execution.
`drop_view_async()` returns a query job: waiting confirms catalog removal while
physical cleanup continues in the background.
The existing `job.job_type` identifies query-backed handles as `"query"` from
submission, and `job.id` contains the connection-scoped SQL query UUID string.
Use `describe_query(job.id)` on the submitting connection to inspect the query.
Registry handles retain their server-reported job types (available after
`status()` or `refresh()`) and registry IDs. Local handles have no ID or job type.
SQL-backed handles do not expose registry records or event history and cannot
be reopened through `open_job()` or reattached after a client restart. Their IDs
are not durable resume tokens. Cancellation follows the SQL server's
capabilities and can be unsupported.
The server decides whether SQL executes on its serving fleet or starts SQL job
workers. That placement does not change the handle's query-backed identity.
All of these report `job.job_type` as `"query"`, with `job.id` holding the
connection-scoped SQL query UUID -- use `describe_query(job.id)` to inspect
one. Unlike registry handles, they don't expose event history, can't be
reopened through `open_job()` or reattached after a client restart, and
cancellation depends on the SQL server's support for it.
### Object teardown {#teardown}
@@ -91,7 +91,7 @@ qualifies. It applies to a plain inner equi-join (a single `ON left_col
`LIMIT` fused into it) where one side -- and only one side -- bottoms out
at a single, otherwise-unfiltered table scan: no existing row filter,
scan range, fragment subset, deleted-row handling, or
[real-time write](/ingestion/realtime-write) tier. If both sides would
[RTE write](/ingestion/realtime-write) tier. If both sides would
otherwise qualify, the side with a BlobV2 column is preferred over the
other; if neither or both sides have one, the rewrite declines rather
than guessing which to narrow.
+24 -11
View File
@@ -77,15 +77,19 @@ def split_into_passages(text):
## Explode it in a query {#explode}
Call the function in the `FROM` clause, after the source table, with an
alias: a Function there yields one row per element of the list it returns,
with every other column repeated. Reference the function's output fields
through the alias:
Wrap the function call in `UNNEST` in the `SELECT` list, with an alias: it
yields one row per element of the list the function returns, with every
other selected column repeated. Reference the exploded struct's fields
through that alias in a query over this one:
<CodeGroup>
```sql SQL icon="database"
WITH exploded AS (
SELECT doc_id, title, UNNEST(split_into_passages(text)) AS p
FROM documents
)
SELECT doc_id, title, p.chunk_index, p.chunk_text
FROM documents, split_into_passages(text) AS p
FROM exploded
```
</CodeGroup>
@@ -160,8 +164,12 @@ timestamp, then call `extract_frame` on each row to produce its image:
<CodeGroup>
```sql SQL icon="database"
WITH sampled AS (
SELECT video_id, video, UNNEST(sample_frame_timestamps(duration_seconds)) AS f
FROM videos
)
SELECT video_id, f.frame_index, f.ts, extract_frame(video, f.ts) AS jpeg
FROM videos, sample_frame_timestamps(duration_seconds) AS f
FROM sampled
```
</CodeGroup>
@@ -179,8 +187,12 @@ recomputing the explosion on every query:
<CodeGroup>
```sql SQL icon="database"
CREATE MATERIALIZED VIEW chunks AS
WITH exploded AS (
SELECT doc_id, title, UNNEST(split_into_passages(text)) AS p
FROM documents
)
SELECT doc_id, title, p.chunk_index, p.chunk_text
FROM documents, split_into_passages(text) AS p
FROM exploded
```
</CodeGroup>
@@ -194,10 +206,11 @@ FROM documents, split_into_passages(text) AS p
The view refreshes like any other materialized view: new source rows are
chunked and inserted, deleted source rows drop their children, and an
updated row's old children are replaced by freshly chunked ones on the next
[refresh](/feature-engineering/refresh-column). A single Function call or
`UNNEST(column)` is the only lateral source a materialized view can carry --
not both, and not combined with `GROUP BY` -- so keep aggregation or further
curation in a second step over the chunked table.
[refresh](/feature-engineering/refresh-column). A single `UNNEST(...)` --
whether unnesting a Function's list output directly, as above, or an
existing list column -- is the only lateral expansion a materialized view
can carry, and it can't be combined with `GROUP BY`, so keep aggregation or
further curation in a second step over the chunked table.
## Chunk, then curate {#next}
+1 -1
View File
@@ -9,7 +9,7 @@ availability:
enterprise: available
---
[Ranking and deduplication](/curation/windowing#dedup) removes records by an
[Ranking](/curation/windowing#dedup) removes records by an
explicit rule, such as keeping the latest version of a known key. This page
covers the other kind of duplicate: records with no shared key that are still
nearly identical, judged by distance between their embeddings -- a stopped
+2 -2
View File
@@ -11,7 +11,7 @@ availability:
A distributed SQL query plans as a sequence of **stages**, each with its
own set of partitions, connected by an **exchange**: a streaming gRPC
service that carries one stage's output to the next stage's plan
executors. [Distributed search](/enterprise/distributed-search) covers
executors. [Distributed Search](/enterprise/distributed-search) covers
how a query gets placed on the fleet in the first place; this page covers
what happens between placement and the final result, for a query whose
plan needs data repartitioned across machines -- a hash join or a grouped
@@ -102,7 +102,7 @@ connection.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Distributed search" icon="network-wired" href="/enterprise/distributed-search">
<Card title="Distributed Search" icon="network-wired" href="/enterprise/distributed-search">
How a query gets placed on the query-serving fleet in the first place.
</Card>
<Card title="Query plans" icon="map" href="/basics/sql#query-plans">
@@ -1,79 +0,0 @@
---
title: "GPU Function Execution"
tag: "Early access"
sidebarTitle: "GPU Functions"
description: "How a GPU Function's worker gets scheduled onto a GPU-capable node, and what happens when none is free."
icon: microchip
keywords: ["gpu", "gpu class", "scheduling", "function", "udf", "kubernetes", "node selector"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="purple">Early access</Badge>
[Registering a GPU Function](/curation/create-function#gpu) sets `gpu=True`
and an optional `gpu_class`; this page covers what happens after that, when
a job actually needs a worker. An operator configures one or more named
**GPU classes** for the deployment -- each one a Kubernetes node selector,
toleration set, and resource name (such as `nvidia.com/gpu`) pointing at a
specific pool of GPU nodes, plus a hardware description (GPU model, GPU
memory, how many GPUs one worker may request). A Function with no
`gpu_class` uses the deployment's `default` class.
## Placement {#placement}
When a job needs to run a GPU Function, the agent resolves its `gpu_class`
(or `default`) to that class's node selector, tolerations, and resource
name, and requests a worker pod shaped accordingly -- this is a separate
placement path from the ordinary CPU worker ladder used by non-GPU
Functions and other background jobs (see
the [Job Framework](/basics/jobs) for that general job
machinery). The resulting pod can only schedule onto a node matching the
class's selector and tolerations, so a class is really naming a pool of
nodes, not just a resource count.
## Waiting for capacity {#capacity}
A GPU class describes the deployment's hardware, not live availability --
listing classes never tells you whether a GPU is free right now. If no
node in the class has room, the worker pod stays pending rather than
failing the job outright or falling back to a CPU worker: the job reports
that it is waiting on that GPU class's capacity, and starts running as soon
as a matching node has room. A job stuck waiting indefinitely usually means
the class's node pool needs more capacity, not that anything is
misconfigured.
{/*
Draft for the list_gpu_classes release. Keep this section unpublished
until that API ships -- v0.41.0 (the latest release) does not have it,
and there is no SQL equivalent; discovery is a REST call
(`GET /v1/gpu/classes`), not a SQL statement.
## Inspecting GPU classes {#inspecting}
<CodeGroup>
```python Python icon="python"
for gpu in db.list_gpu_classes():
print(gpu.name, gpu.gpu_model, gpu.gpu_memory_bytes, gpu.max_gpu_count_per_worker)
```
</CodeGroup>
Returns the operator-configured classes: name, resource name, GPU model
and memory, and the largest per-worker GPU count the class permits.
Listing requires the System Usage privilege.
*/}
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Create a Function" icon="code" href="/curation/create-function#gpu">
Register a GPU Function and pin it to a specific class.
</Card>
<Card title="Job Framework" icon="list-check" href="/basics/jobs">
The job types a deployment runs on its own schedule.
</Card>
<Card title="GPU Refresh Execution" icon="microchip" href="/enterprise/gpu-refresh-execution">
The same placement path, for a computed column or MV refresh instead of a direct call.
</Card>
</CardGroup>
-70
View File
@@ -1,70 +0,0 @@
---
title: "Slurm Scheduling"
sidebarTitle: "Slurm Scheduling"
description: "Run LanceDB Enterprise worker pods on a Slurm-on-Kubernetes cluster, accounted by Slurm."
icon: "server"
availability:
embedded: unavailable
enterprise: available
---
[Slurm](https://slurm.schedmd.com/) is the workload manager most GPU
clusters run to schedule and account for jobs -- queueing work across
shared nodes, enforcing fair-share and priority policies between teams,
and tracking GPU-hour usage per job. That makes it the de facto standard
in AI training infrastructure, where expensive GPU nodes are shared
across many training and data jobs instead of dedicated to one team.
CoreWeave, a major GPU cloud provider, runs Slurm this way through SUNK
(Slurm on Kubernetes) -- Slurm's scheduling and accounting layered onto
an otherwise ordinary Kubernetes cluster.
LanceDB Enterprise's job workers -- indexing, compaction, cache prewarm,
registration, and most other background work -- all launch as ordinary
Kubernetes pods through one shared scheduling configuration. Today, that
configuration integrates with CoreWeave's SUNK specifically; the rest of
this page describes its effect. On a Neocloud running SUNK, pointing
that configuration at SUNK's pod scheduler puts every one of those
worker pods under Slurm's scheduling and accounting, instead of plain
Kubernetes bin-packing. See [Architecture](/index#architecture).
## What SUNK changes {#what-changes}
Most agent-launched worker pods -- a one-pod job worker or a multi-worker
distributed indexer Job -- share one scheduling configuration: node
selector, tolerations, scheduler name, and termination grace period.
Pointing the scheduler name at the cluster's SUNK pod scheduler routes
pod placement through SUNK instead of the default Kubernetes scheduler,
so Slurm sees and accounts for the workload. SUNK also expects a short
termination grace period, kept below Slurm's own kill-wait threshold --
deployments commonly use 5 to 20 seconds.
<Warning>
Function refresh's own child workers -- including GPU Function execution
-- get the scheduler name but not this shared termination-grace setting;
they currently launch with Kubernetes' 30-second default, above most
SUNK deployments' kill-wait bound. Confirm with your deployment operator
before running Function refresh on a SUNK-scheduled cluster. See
[GPU Function execution](/curation/gpu-function-execution) for the
placement mechanism this affects.
</Warning>
## Configuring a SUNK-backed deployment {#config}
This scheduling configuration is set by a deployment operator, not by a
SQL statement, table property, or SDK call -- there is no per-query or
per-table knob for it. [Contact support](https://www.lancedb.com/contact)
to configure or size a SUNK-backed deployment.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="GPU Function Execution" icon="microchip" href="/curation/gpu-function-execution">
How a GPU-bound Function gets scheduled onto GPU-equipped workers.
</Card>
<Card title="Architecture" icon="sitemap" href="/index#architecture">
The query, executor, cache and jobs fleets SUNK schedules onto.
</Card>
<Card title="Slurm Scheduling for refresh" icon="server" href="/enterprise/slurm-refresh-scheduling">
What does and doesn't carry over for a computed column or MV refresh.
</Card>
</CardGroup>
+2 -2
View File
@@ -1,6 +1,6 @@
---
title: "Ranking and deduplication"
sidebarTitle: "Rank and Deduplicate"
title: "Ranking"
sidebarTitle: "Ranking"
description: "Choose records with explicit ranking rules and deterministic tie handling."
icon: chart-simple
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
+111
View File
@@ -0,0 +1,111 @@
---
title: "Multicloud"
sidebarTitle: "Multicloud"
description: "Keep deployment orchestration in your cloud while dispatching compute-heavy work, like GPU UDF execution, to run in a different cloud."
icon: "cloud"
keywords: ["multicloud", "cross-cloud", "gpu", "deployment"]
availability:
embedded: unavailable
enterprise: available
---
A LanceDB Enterprise deployment doesn't have to keep all of its compute in
one cloud. Orchestration -- query nodes, job coordination, the catalog --
stays with your deployment in its own cloud account and region. Specific,
compute-heavy work, such as a GPU-bound Function's execution during a
refresh, can instead dispatch to run in a **different** cloud account,
region, or provider entirely, and send its results back to complete the job.
## How it fits together {#architecture}
Your deployment's job orchestration plans the work and tracks it to
completion the same way it would locally. The part that actually needs the
remote capacity -- a GPU UDF call, for example -- runs across a network
boundary instead of on a worker in your own cloud, and reports its result
back the same way a local worker would.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
subgraph P["Your cloud"]
direction TB
QN["Query node"]
Job["Job orchestration"]
end
subgraph R["Remote cloud"]
direction TB
Exec["Compute execution<br/>(e.g. GPU UDFs)"]
end
QN --> Job
Job -- "dispatches work" --> Exec
Exec -- "results" --> Job
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class Exec accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
subgraph P["Your cloud"]
direction TB
QN["Query node"]
Job["Job orchestration"]
end
subgraph R["Remote cloud"]
direction TB
Exec["Compute execution<br/>(e.g. GPU UDFs)"]
end
QN --> Job
Job -- "dispatches work" --> Exec
Exec -- "results" --> Job
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Exec accent
```
</div>
## Why it matters {#why-it-matters}
The cloud that's right for your primary deployment -- where your data and
other infrastructure already live -- isn't always the cloud with the best
GPU availability or pricing. Splitting orchestration from execution means
you don't have to choose: keep your deployment where it already makes
sense, and reach compute capacity elsewhere only for the work that
actually needs it, without migrating the deployment or operating that
remote capacity yourself.
[Contact us](https://www.lancedb.com/contact) to learn more about setting
up cross-cloud execution for your deployment.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Slurm" icon="server" href="/deployment/slurm">
A common way the remote cloud's compute gets scheduled: as Slurm jobs.
</Card>
<Card title="Adaptive Execution" icon="gauge-high" href="/enterprise/adaptive-execution">
How a refresh's resource class, part size and worker count adjust themselves.
</Card>
</CardGroup>
+131
View File
@@ -0,0 +1,131 @@
---
title: "Slurm"
sidebarTitle: "Slurm"
description: "Run LanceDB Enterprise's own pods as Slurm-scheduled jobs, and reach a Neocloud's GPU capacity through Slurm in the multicloud case."
icon: "server"
keywords: ["slurm", "scheduling", "gpu", "neocloud", "preemption", "multicloud"]
availability:
embedded: unavailable
enterprise: available
---
[Slurm](https://slurm.schedmd.com/) is the workload manager most GPU
clusters run to schedule and account for jobs -- queueing work across
shared nodes, enforcing fair-share and priority policies between teams,
and tracking GPU-hour usage per job. That makes it the de facto standard
in AI training infrastructure, where expensive GPU nodes are shared
across many training and data jobs instead of dedicated to one team.
GPU-focused cloud providers ("Neoclouds") such as CoreWeave and Nebius run
Slurm-based scheduling across their Kubernetes clusters for exactly this
reason.
Slurm shows up for LanceDB Enterprise in two related ways. Its own pods --
query nodes, plan executors, and background job workers alike -- can run
as Slurm-scheduled jobs rather than plain Kubernetes pods, under the same
scheduling and accounting as everything else on the cluster. And in the
[multicloud](/deployment/multicloud) case, when the remote cloud is a
Neocloud, Slurm is commonly how the dispatched GPU work actually gets
scheduled there.
## How it fits together {#architecture}
Scheduling LanceDB's pods through Slurm puts them under the same
scheduler as GPU training jobs, so the cluster's capacity is accounted
for consistently regardless of which one is using it -- Slurm's
fair-share and priority policies apply to LanceDB the same way they
apply to a training job, including preemption when training needs the
capacity back.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
subgraph P["Your cloud"]
direction TB
Job["Job orchestration"]
end
subgraph N["Neocloud (e.g. CoreWeave, Nebius)"]
direction TB
Slurm["Slurm scheduler"]
Train["GPU training jobs"]
LDB["LanceDB Enterprise pods"]
Slurm --> Train
Slurm --> LDB
Train -.->|"can preempt"| LDB
end
Job -- "dispatches GPU work" --> Slurm
LDB -- "results" --> Job
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class Slurm accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
subgraph P["Your cloud"]
direction TB
Job["Job orchestration"]
end
subgraph N["Neocloud (e.g. CoreWeave, Nebius)"]
direction TB
Slurm["Slurm scheduler"]
Train["GPU training jobs"]
LDB["LanceDB Enterprise pods"]
Slurm --> Train
Slurm --> LDB
Train -.->|"can preempt"| LDB
end
Job -- "dispatches GPU work" --> Slurm
LDB -- "results" --> Job
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Slurm accent
```
</div>
## Why it matters {#why-it-matters}
Most GPU infrastructure already runs Slurm for training. Deploying LanceDB
Enterprise into that same infrastructure, under the same scheduler, means
it can share capacity with training workloads safely -- queued, accounted,
and preemptible like everything else -- instead of requiring a separate,
dedicated pool of nodes carved out just for LanceDB. Combined with
[multicloud](/deployment/multicloud) dispatch, that makes it practical to
keep your primary deployment wherever it already lives and still reach
GPU capacity on a Neocloud built around Slurm, only for the work that
needs it.
[Contact us](https://www.lancedb.com/contact) to learn more about running
LanceDB Enterprise on a Slurm-scheduled cluster.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Multicloud" icon="cloud" href="/deployment/multicloud">
The cross-cloud dispatch model this page's Neocloud case builds on.
</Card>
<Card title="Architecture" icon="sitemap" href="/index#architecture">
The query, executor, cache and jobs fleets this scheduling applies to.
</Card>
</CardGroup>
+15 -9
View File
@@ -121,11 +121,11 @@
},
{
"source": "/curation/sunk-integration",
"destination": "/curation/slurm-scheduling"
"destination": "/deployment/slurm"
},
{
"source": "/curation/slurm-integration",
"destination": "/curation/slurm-scheduling"
"destination": "/deployment/slurm"
},
{
"source": "/curation/sampling",
@@ -362,6 +362,14 @@
{
"source": "/enterprise/distributed-column-refresh",
"destination": "/enterprise/adaptive-execution"
},
{
"source": "/curation/slurm-scheduling",
"destination": "/deployment/slurm"
},
{
"source": "/enterprise/slurm-refresh-scheduling",
"destination": "/deployment/slurm"
}
],
"navigation": {
@@ -452,9 +460,7 @@
"curation/clustering",
"curation/dedupe",
"feature-engineering/view",
"curation/vend-secret",
"curation/gpu-function-execution",
"curation/slurm-scheduling"
"curation/vend-secret"
]
},
{
@@ -468,9 +474,7 @@
"enterprise/incremental-refresh",
"enterprise/auto-refresh",
"enterprise/checkpointing",
"enterprise/adaptive-execution",
"enterprise/gpu-refresh-execution",
"enterprise/slurm-refresh-scheduling"
"enterprise/adaptive-execution"
]
},
{
@@ -534,7 +538,9 @@
"pages": [
"enterprise/horizontal-scaling",
"deployment/opentelemetry",
"enterprise/deployment/kubernetes"
"enterprise/deployment/kubernetes",
"deployment/multicloud",
"deployment/slurm"
]
},
{
@@ -136,7 +136,4 @@ part sizing described above still decides how many rows each part covers.
<Card title="Checkpointing" icon="floppy-disk" href="/enterprise/checkpointing">
How a part's progress survives a failure or restart.
</Card>
<Card title="GPU Refresh Execution" icon="microchip" href="/enterprise/gpu-refresh-execution">
How a GPU-bound Function's worker gets placed, for a refresh specifically.
</Card>
</CardGroup>
@@ -190,10 +190,10 @@ flowchart TD
<Card title="Incremental Indexing" icon="arrows-rotate" href="/indexing/incremental-indexing#index">
The `index_remap` job type, and keeping an index current as the table changes.
</Card>
<Card title="Distributed compaction" icon="layer-group" href="/enterprise/distributed-compaction">
<Card title="Distributed Compaction" icon="layer-group" href="/enterprise/distributed-compaction">
How compaction itself scales across the jobs fleet.
</Card>
<Card title="Distributed indexing" icon="diagram-project" href="/enterprise/distributed-indexing">
<Card title="Distributed Indexing" icon="diagram-project" href="/enterprise/distributed-indexing">
How an index build scales across the jobs fleet.
</Card>
</CardGroup>
+59 -37
View File
@@ -3,61 +3,83 @@ title: "Cache Prewarm"
sidebarTitle: "Cache Prewarm"
description: "Populate the tiered index cache ahead of a request, instead of filling it lazily on first read."
icon: "fire"
keywords: ["prewarm", "cache", "page cache", "index cache", "startup"]
keywords: ["prewarm", "cache", "index cache", "startup"]
availability:
embedded: unavailable
enterprise: available
---
This page is about `startup_prewarm_idx`, which warms the distributed
[tiered index cache](/enterprise/tiered-index-cache) a plan executor keeps
as part of Enterprise's serving layer -- LanceDB Embedded has no such
cache, so there is nothing for it to warm on Embedded. Embedded does have
its own, separate `prewarm_index()` on a table, an idempotent hint that
loads a built index into memory ahead of a query; it has no equivalent for
raw column data, where only Enterprise supports prewarming. On Enterprise,
the index cache fills lazily on first read by default; `startup_prewarm_idx`
loads it ahead of that first request instead. The
[data page cache](/enterprise/data-page-cache) is a separate cache with its
own, on-demand prewarm -- `startup_prewarm_idx` does not touch it.
Cache Prewarm loads the [tiered index cache](/enterprise/tiered-index-cache)
a plan executor keeps as part of Enterprise's serving layer, instead of
leaving it to fill lazily on first read. You configure which tables to
prewarm; on startup, each plan executor loads those tables' indexes into
cache before it joins the fleet. The executor does not accept queries until
prewarming finishes, so it never serves a request against a cold cache.
## At plan executor startup {#startup}
LanceDB Embedded has no such cache, so there is nothing for it to warm
there. Embedded does have its own, separate `prewarm_index()` on a table --
an idempotent hint that loads a built index into memory ahead of a query.
`startup_prewarm_idx` runs as part of a plan executor's own boot sequence,
before it joins the fleet, rather than on a schedule. A deployment configures
it with `startup_prewarm_enabled`, an index selection (which columns' indexes
to warm, up to 1024 entries), and a concurrency limit for how many it loads
at once.
<div className="hidden dark:block">
Whether this delays the executor from serving queries is a separate setting,
`startup_prewarm_block_query_ready`, off by default in the binary itself but
**on by default in the Helm chart** -- most deployments block on it unless
the chart value is set otherwise:
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
List[("Configured list<br/>of tables to prewarm")]
Start["Plan executor starts<br/>(not ready — no queries served)"] --> Warm["Loads the configured<br/>tables' indexes into cache"]
List -.-> Warm
Warm --> Ready["Ready — joins the fleet,<br/>starts serving queries"]
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class Warm,List accent
```
- **Unset:** the executor becomes query-ready immediately and
prewarm runs concurrently in the background. Early queries may still miss
the cache; later ones benefit once prewarm catches up.
- **Set (the Helm chart's default):** query readiness waits for prewarm to
finish, up to `startup_prewarm_timeout_secs`. Blocking query readiness
this way requires `healthPort` to be set (9096 in the chart's default
values), so process health stays reachable while readiness is gated.
</div>
If prewarm fails or times out, `startup_prewarm_failure_policy` decides what
happens next:
<div className="block dark:hidden">
| Policy | Behavior |
| --- | --- |
| `mark_ready_cold` (default) | The executor becomes query-ready anyway, just without a warm cache. |
| `stay_not_ready` | Combined with blocking query readiness, the executor never becomes ready -- a failed prewarm keeps it out of the fleet rather than serving cold. |
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
List[("Configured list<br/>of tables to prewarm")]
Start["Plan executor starts<br/>(not ready — no queries served)"] --> Warm["Loads the configured<br/>tables' indexes into cache"]
List -.-> Warm
Warm --> Ready["Ready — joins the fleet,<br/>starts serving queries"]
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class Warm,List accent
```
</div>
## Why it matters {#why-it-matters}
Without prewarming, a fresh or restarted executor would fill its indexes
lazily, so the first queries it serves pay the cost of loading and
deserializing them from object storage. Holding the executor out of the
fleet until its configured tables are warm means every query it ever
serves gets a warm cache, so performance stays consistent through restarts
and rollouts instead of dipping right after each one.
[Contact us](https://www.lancedb.com/contact) to learn more about how
prewarming is configured for your deployment.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Data Page Cache" icon="database" href="/enterprise/data-page-cache">
The raw-data-page cache -- warmed on demand, not by startup prewarm.
The raw-data-page cache -- a separate cache from the one prewarm fills.
</Card>
<Card title="Tiered Index Cache" icon="layer-group" href="/enterprise/tiered-index-cache">
The deserialized-index cache startup prewarm fills.
The deserialized-index cache prewarm fills.
</Card>
<Card title="Architecture" icon="diagram-project" href="/index#architecture">
Where plan executors and query nodes sit in the serving fleet.
+11 -26
View File
@@ -56,36 +56,21 @@ flowchart LR
</div>
A table's local disk cache shares one budget with the
A table's local disk cache shares one configurable budget with the
[tiered index cache](/enterprise/tiered-index-cache#disk-budget)'s disk
tier where both are enabled on the same executor -- raising one's share of
that budget lowers the other's.
tier where both are enabled on the same executor.
## Prewarming on demand {#prewarm}
## Why it matters {#why-it-matters}
[Cache Prewarm](/enterprise/cache-prewarm)'s `startup_prewarm_idx` only
warms the [tiered index cache](/enterprise/tiered-index-cache), not this
one. The data page cache has its own, separate on-demand prewarm, run by a
deployment operator rather than called from application code, optionally
scoped to specific columns:
Object storage is durable and cheap, but every round trip to it adds
latency and egress cost. For workloads that read the same ranges
repeatedly -- a hot subset of embeddings, a frequently-filtered column --
the data page cache turns most of those reads into local memory or disk
hits instead, so performance stays predictable as traffic grows and you
pay object storage once per page, not once per read.
<CodeGroup>
```bash CLI icon="terminal"
lancedb admin cache prewarm data --table embedding_table --column embedding
```
</CodeGroup>
The command returns once the request is accepted, not once prewarming
finishes; it is idempotent and safe to run concurrently from multiple
clients. Prewarming has a large upfront cost, so it pays off for workloads
that repeatedly query the same columns -- large columns such as embeddings
or binary data may not be practical to prewarm this way.
<Warning>
The OSS SDK's `table.prewarm_data()` targets LanceDB Cloud, a different
product; calling it against an Enterprise deployment 404s. Use the admin
command above instead.
</Warning>
[Contact us](https://www.lancedb.com/contact) to learn more about how
caching is sized and tuned for your workload.
## Cache interfaces {#interfaces}
+14 -14
View File
@@ -1,6 +1,6 @@
---
title: "Distributed compaction"
sidebarTitle: "Distributed compaction"
title: "Distributed Compaction"
sidebarTitle: "Distributed Compaction"
description: "How LanceDB Enterprise distributes compaction across a fleet of job workers."
icon: "compress"
availability:
@@ -12,8 +12,8 @@ Distributed compaction splits a table's fragments into independent,
fragment-disjoint tasks and fans them out across job workers in LanceDB
Enterprise. Unlike an index build, a compaction plan needs no bin-packing:
Lance's own planner already produces disjoint tasks, so workers claim and
execute them directly through the job registry, with no coordination
needed to decide who does what. See [Architecture](/index#architecture).
execute them directly, with no coordination needed to decide who does
what. See [Architecture](/index#architecture).
<div className="hidden dark:block">
@@ -24,16 +24,16 @@ needed to decide who does what. See [Architecture](/index#architecture).
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
R --> W1["Worker<br/>claims fragments 1-2"]
R --> W2["Worker<br/>claims fragments 3-4"]
R --> W3["Worker<br/>claims fragments 5-6"]
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"]
L --> W1["Worker<br/>claims fragments 1-2"]
L --> W2["Worker<br/>claims fragments 3-4"]
L --> W3["Worker<br/>claims fragments 5-6"]
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
W2 --> C
W3 --> C
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class R,C accent
class C accent
```
</div>
@@ -47,16 +47,16 @@ flowchart LR
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
R --> W1["Worker<br/>claims fragments 1-2"]
R --> W2["Worker<br/>claims fragments 3-4"]
R --> W3["Worker<br/>claims fragments 5-6"]
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"]
L --> W1["Worker<br/>claims fragments 1-2"]
L --> W2["Worker<br/>claims fragments 3-4"]
L --> W3["Worker<br/>claims fragments 5-6"]
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
W2 --> C
W3 --> C
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class R,C accent
class C accent
```
</div>
+41 -29
View File
@@ -1,19 +1,33 @@
---
title: "Distributed indexing"
sidebarTitle: "Distributed indexing"
description: "How LanceDB Enterprise distributes an index build across a fleet of job workers."
title: "Distributed Indexing"
sidebarTitle: "Distributed Indexing"
description: "How LanceDB Enterprise distributes a vector index build across a fleet of workers, stage by stage."
icon: "layer-group"
availability:
embedded: unavailable
enterprise: available
---
Distributed indexing bin-packs a vector, full-text, or scalar index build into
a plan of claimable tasks, then fans the plan out across job workers in
LanceDB Enterprise. Workers claim tasks independently through the job
registry and commit the finished index together. A vector index build first
trains a shared IVF/PQ model from sampled data, then workers build storage
from that model in parallel. See [Architecture](/index#architecture).
A vector index build splits into stages, and the data in scope narrows at
each one: training works from a small sample, shuffling touches every row
once, and merging works entirely within one partition at a time. Full-text
and scalar index builds skip training and shuffling -- they bin-pack
directly into independent tasks workers claim and build in parallel,
closer to how [distributed compaction](/enterprise/distributed-compaction)
splits work. See [Architecture](/index#architecture).
## Stages of a vector index build {#stages}
1. **Train.** A sample of the table's vectors -- not the full dataset --
trains the shared model every worker will use: IVF partition centroids,
and for IVF-PQ, a PQ codebook on top of them.
2. **Shuffle.** Every row in the table gets assigned to its nearest
trained partition and regrouped so that all of one partition's rows end
up together, wherever in the table they originally lived.
3. **Merge.** Each partition is now self-contained, so workers build and
merge partitions in parallel with no cross-partition coordination --
one worker's output never depends on another's.
4. **Commit.** The finished index is committed to the table.
<div className="hidden dark:block">
@@ -24,18 +38,17 @@ from that model in parallel. See [Architecture](/index#architecture).
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
T --> P["Bin-pack build plan<br/>into claimable tasks"]
P --> R[("Job registry")]
R --> W1["Worker<br/>claims a task"]
R --> W2["Worker<br/>claims a task"]
R --> W3["Worker<br/>claims a task"]
W1 --> C["Commit the<br/>finished index"]
W2 --> C
W3 --> C
S["Sample"] --> T["Train IVF centroids<br/>+ PQ codebook"]
T --> Sh["Shuffle: assign every row<br/>to its nearest partition"]
Sh --> M1["Merge<br/>partition A"]
Sh --> M2["Merge<br/>partition B"]
Sh --> M3["Merge<br/>partition C"]
M1 --> C["Commit the<br/>finished index"]
M2 --> C
M3 --> C
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class R accent
class Sh accent
```
</div>
@@ -49,18 +62,17 @@ flowchart LR
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
T --> P["Bin-pack build plan<br/>into claimable tasks"]
P --> R[("Job registry")]
R --> W1["Worker<br/>claims a task"]
R --> W2["Worker<br/>claims a task"]
R --> W3["Worker<br/>claims a task"]
W1 --> C["Commit the<br/>finished index"]
W2 --> C
W3 --> C
S["Sample"] --> T["Train IVF centroids<br/>+ PQ codebook"]
T --> Sh["Shuffle: assign every row<br/>to its nearest partition"]
Sh --> M1["Merge<br/>partition A"]
Sh --> M2["Merge<br/>partition B"]
Sh --> M3["Merge<br/>partition C"]
M1 --> C["Commit the<br/>finished index"]
M2 --> C
M3 --> C
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class R accent
class Sh accent
```
</div>
@@ -1,5 +1,5 @@
---
title: "Distributed Real Time Read"
title: "Distributed RTE Read"
sidebarTitle: "Distributed RTE Read"
description: "How a query combines the real time engine's fresh tier with the ordinary distributed query-serving path, machine to machine."
icon: "network-wired"
@@ -8,13 +8,13 @@ availability:
enterprise: available
---
[Real Time Read](/search/realtime-read) covers what a query against an
[RTE Read](/search/realtime-read) covers what a query against an
RTE-enabled table returns: base-table rows merged with the fresh tier,
fresh rows superseding base rows with the same primary key. On a single
Embedded process that merge is one function call against one Lance
dataset. In LanceDB Enterprise, the base table and the fresh tier are
usually served by different machines -- plan executors for the base table,
a [WAL node](/tables/realtime-engine#wal-nodes) for the table's shards --
an [RTE node](/tables/realtime-engine#wal-nodes) for the table's shards --
so a query has to combine their results across the network instead.
<div className="hidden dark:block">
@@ -34,7 +34,7 @@ flowchart LR
B1["Base-table arm<br/>index-covered scan or ANN search"]
end
subgraph WN["WAL node (owns this table's shards)"]
subgraph WN["RTE node (owns this table's shards)"]
direction TB
W1["SSTable arm<br/>index-covered"]
W2["Active memtable arm<br/>vector: in-memory HNSW*<br/>FTS/scalar: indexed"]
@@ -73,7 +73,7 @@ flowchart LR
B1["Base-table arm<br/>index-covered scan or ANN search"]
end
subgraph WN["WAL node (owns this table's shards)"]
subgraph WN["RTE node (owns this table's shards)"]
direction TB
W1["SSTable arm<br/>index-covered"]
W2["Active memtable arm<br/>vector: in-memory HNSW*<br/>FTS/scalar: indexed"]
@@ -99,24 +99,76 @@ flowchart LR
The distributed plan for an RTE-enabled table is an ordinary plan executor
query plan with one additional node spliced in: a union exec that queries
the owning WAL node's shard and merges its rows into the base arm's stream
the owning RTE node's shard and merges its rows into the base arm's stream
by primary key, keeping the newest generation on a collision and cutting
back to the requested top-K once both sides have contributed. An optimizer
pass inserts this node only when the query actually touches a WAL-enabled
pass inserts this node only when the query actually touches an RTE-enabled
table; a query that touches none produces a plan identical to one with no
RTE tables involved at all, so RTE adds no overhead to queries that never
use it.
Because the base arm already read through the plan-executor fleet's ordinary
caches and index structures, the only new network hop this adds is the one
to the WAL node that owns the table's shards -- the same WAL node [Real Time
to the RTE node that owns the table's shards -- the same RTE node [Real Time
writes](/ingestion/realtime-write) land on, found by the query node through
the same consistent hash of database and table name.
## Semantic search stays index-covered {#index-coverage}
## Search stays index-covered {#index-coverage}
Every tier the union exec reads from -- the base table, the RTE node's
SSTables, and its currently-active memtable -- is covered by a maintained
index, for vector, full-text, and scalar search alike:
| Layer | Vector search | Full-text / scalar search |
| --- | --- | --- |
| Base table | ANN index | Maintained index |
| SSTables | ANN index | Maintained index |
| Active memtable | In-memory HNSW graph\* | Maintained index |
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
B["Base table"] --> I1["Vector, full-text, and<br/>scalar index coverage"]
S["SSTables"] --> I1
M["Active memtable"] --> I2["Full-text and scalar index<br/>coverage; vector via<br/>in-memory HNSW*"]
I1 --> R["No unindexed data<br/>left to scan in full"]
I2 --> R
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class R accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
B["Base table"] --> I1["Vector, full-text, and<br/>scalar index coverage"]
S["SSTables"] --> I1
M["Active memtable"] --> I2["Full-text and scalar index<br/>coverage; vector via<br/>in-memory HNSW*"]
I1 --> R["No unindexed data<br/>left to scan in full"]
I2 --> R
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class R accent
```
</div>
For full-text and scalar search, a maintained index covers every tier the
union exec reads from -- the base table, the WAL node's SSTables, and its
union exec reads from -- the base table, the RTE node's SSTables, and its
currently-active memtable -- so a row written a moment ago is just as
index-searchable as one written yesterday. If a resident memtable predates
an index being added to the maintained set, the query is rejected outright
@@ -147,13 +199,13 @@ table alone.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Real Time Read" icon="magnifying-glass" href="/search/realtime-read">
<Card title="RTE Read" icon="magnifying-glass" href="/search/realtime-read">
What a query returns, and how Embedded does this same merge in-process.
</Card>
<Card title="Distributed Real Time Write" icon="bolt" href="/enterprise/distributed-real-time-write">
How writes reach the WAL node this page's queries read from.
<Card title="Distributed RTE Write" icon="bolt" href="/enterprise/distributed-real-time-write">
How writes reach the RTE node this page's queries read from.
</Card>
<Card title="Distributed search" icon="network-wired" href="/enterprise/distributed-search">
<Card title="Distributed Search" icon="network-wired" href="/enterprise/distributed-search">
The ordinary plan-executor path this layers on top of.
</Card>
</CardGroup>
@@ -1,30 +1,32 @@
---
title: "Distributed Real Time Write"
title: "Distributed RTE Write"
sidebarTitle: "Distributed RTE Write"
description: "How Enterprise scales real time writes horizontally across a dedicated WAL-node tier, and the throughput limits that come with it."
description: "How Enterprise scales RTE writes horizontally across a dedicated serving tier, and the throughput limits that come with it."
icon: "bolt"
keywords: ["rte", "real time engine", "wal node", "shard", "sharding", "horizontal scaling", "write throughput", "memwal"]
keywords: ["rte", "real time engine", "shard", "sharding", "horizontal scaling", "write throughput"]
availability:
embedded: unavailable
enterprise: available
---
[Real Time writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
[RTE writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
as one call that works the same way on either connection. On Enterprise, that
call does not run locally -- it posts to the server, which installs the write
path on a dedicated, horizontally scalable serving tier instead of running it
in your process. This page covers that tier: how it shards a table's writes,
the one real limit sharding does not remove, and the throughput you can expect.
## The WAL-node tier {#wal-nodes}
## How writes are routed {#routing}
Enabling RTE on a remote table sends the spec to the server
(`POST /v1/table/<name>/enable_rte/`), which installs Lance's `__lance_mem_wal`
index on the dataset. From then on, every request for that table -- reads and
writes alike -- is consistent-hashed on the database and table name to a
single owning **WAL node**, a stateful pod separate from the query nodes and
plan executors that serve ordinary queries. WAL node pods share nothing but
the underlying object storage; no pod talks to another one directly.
Enabling RTE on a remote table sends the spec to the server, which installs
the write path on the dataset instead of running it in your process. From
then on, the query node routes every request for that table -- reads and
writes alike -- to a single owning **RTE node**, a stateful component
separate from the query nodes and plan executors that serve ordinary
queries. A query node routes different tables to different RTE nodes, and
a single RTE node can host shards from many different tables at once. RTE
nodes share nothing but the underlying object storage; no node talks to
another one directly.
<div className="hidden dark:block">
@@ -36,22 +38,22 @@ the underlying object storage; no pod talks to another one directly.
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
Q["Query node<br/>hashes (db, table) to one WAL node"]
Q["Query node<br/>routes each table's shards to an RTE node"]
subgraph W0["WAL node A — owns table 'frames'"]
subgraph W0["RTE node A"]
direction TB
S0["Shard 0<br/>ShardWriter"]
S1["Shard 1<br/>ShardWriter"]
S2["Shard 2<br/>ShardWriter"]
S0["Shard 0 — table 'frames'"]
S1["Shard 0 — table 'events'"]
end
subgraph W1["WAL node B — owns table 'events'"]
subgraph W1["RTE node B"]
direction TB
S3["Shard 0<br/>ShardWriter"]
S2["Shard 0 — table 'metrics'"]
end
Q -- "writes + reads for 'frames'" --> W0
Q -- "writes + reads for 'events'" --> W1
Q -- "'frames'" --> W0
Q -- "'events'" --> W0
Q -- "'metrics'" --> W1
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
```
@@ -67,22 +69,22 @@ flowchart LR
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
Q["Query node<br/>hashes (db, table) to one WAL node"]
Q["Query node<br/>routes each table's shards to an RTE node"]
subgraph W0["WAL node A — owns table 'frames'"]
subgraph W0["RTE node A"]
direction TB
S0["Shard 0<br/>ShardWriter"]
S1["Shard 1<br/>ShardWriter"]
S2["Shard 2<br/>ShardWriter"]
S0["Shard 0 — table 'frames'"]
S1["Shard 0 — table 'events'"]
end
subgraph W1["WAL node B — owns table 'events'"]
subgraph W1["RTE node B"]
direction TB
S3["Shard 0<br/>ShardWriter"]
S2["Shard 0 — table 'metrics'"]
end
Q -- "writes + reads for 'frames'" --> W0
Q -- "writes + reads for 'events'" --> W1
Q -- "'frames'" --> W0
Q -- "'events'" --> W0
Q -- "'metrics'" --> W1
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
```
@@ -91,25 +93,16 @@ flowchart LR
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
**table shards**, each a separate, independently-appending `ShardWriter`. A
shard's identity is derived from the table's own URI
(`uuid_v5(WAL_SHARD_NS, "{table_uri}#{i}")`), not assigned by an operator, so
a shard survives the table being claimed by a different pod later.
**table shards**, each a separate, independently-appending stream, giving the
table that many parallel write paths instead of one. A shard's identity is
derived from the table itself, not assigned by an operator, so a shard
survives the table being claimed by a different RTE node later.
This is where the horizontal scaling actually happens, and where its limit
is: **one WAL node owns every shard of a given table.** Splitting `frames`
into 8 shards gives it 8 independent append streams, all still served by the
one pod that owns `frames` -- it raises that table's own ceiling, not the
number of pods working on it. Adding WAL node replicas instead spreads
*other* tables' ownership across more pods, raising how many tables the
cluster can serve, not how fast any single one of them can write.
<Note>
A single table's write throughput is capped by one pod's capacity, however
many shards you give it. A workload that needs to exceed one pod's ceiling on
one table is not solved by adding shards -- there is no path to spreading one
table's shards across multiple WAL nodes today.
</Note>
This is where the horizontal scaling actually happens: splitting `frames`
into 8 shards gives it 8 independent append streams instead of 1, raising
that table's own write ceiling. Adding RTE node replicas spreads shard
ownership across more nodes, so the cluster can host more tables -- and more
shards -- without any one node becoming a bottleneck for everything it serves.
`RTEWriteSpec::identity(column)` and `RTEWriteSpec::unsharded()` both resolve
to a single table shard -- `bucket` is the sharding mode that actually
@@ -121,22 +114,22 @@ scaling.
A table shard is the unit of capacity: measured runs show it sustaining
roughly 85-90 MB/s of durable append before acknowledgement, and throughput
scaling nearly linearly with shard count up to at least 8 shards on one
pod -- the bottleneck is the durable write itself, not CPU or memory. This
node -- the bottleneck is the durable write itself, not CPU or memory. This
is observed capacity, not an enforced quota; nothing in the deployment caps
a shard's or a table's throughput below it. [Contact
support](https://www.lancedb.com/contact) if your workload needs more than
one pod's worth of capacity for a single table.
support](https://www.lancedb.com/contact) if your workload needs more write
throughput than this provides.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Real Time writes" icon="bolt" href="/ingestion/realtime-write">
<Card title="RTE writes" icon="bolt" href="/ingestion/realtime-write">
Enable RTE and write to a table; the same call on Embedded or Enterprise.
</Card>
<Card title="How the real time engine works" icon="database" href="/tables/realtime-engine">
MemWAL, shards, memtables and SSTables -- the shared architecture.
</Card>
<Card title="Distributed Real Time Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
<Card title="Distributed RTE Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
How reads combine the WAL tier's fresh rows with the base table across the fleet.
</Card>
</CardGroup>
+2 -2
View File
@@ -1,6 +1,6 @@
---
title: "Distributed search"
sidebarTitle: "Distributed search"
title: "Distributed Search"
sidebarTitle: "Distributed Search"
description: "How LanceDB Enterprise distributes index execution and table-data reads."
icon: "network-wired"
availability:
+2 -2
View File
@@ -1,6 +1,6 @@
---
title: "Distributed write"
sidebarTitle: "Distributed write"
title: "Distributed Write"
sidebarTitle: "Distributed Write"
description: "How LanceDB Enterprise distributes a CTAS import's writes across plan executors."
icon: upload
availability:
@@ -1,93 +0,0 @@
---
title: "GPU Refresh Execution"
tag: "Early access"
sidebarTitle: "GPU Refresh Execution"
description: "A computed column or materialized view filled by a GPU Function schedules its refresh workers the same way GPU Function execution does in Data Curation."
icon: microchip
keywords: ["gpu", "gpu class", "scheduling", "refresh", "computed column", "function"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="purple">Early access</Badge>
A [computed column](/feature-engineering/add-computed-column) or
[materialized view](/feature-engineering/materialized-views) filled by a
registered [Function](/curation/create-function) with `gpu=True` refreshes
through the exact same placement path as
[GPU Function execution](/curation/gpu-function-execution) in Data
Curation -- refreshing a column is not a separate scheduling code path
from calling the Function directly, just a different caller.
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
R --> P
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class P accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
R --> P
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class P accent
```
</div>
The Function's `gpu_class` (or the deployment's `default`) resolves to the
same node selector, tolerations and resource name either way, and a
refresh that needs more GPU capacity than is currently free waits the
same way a direct call would -- see
[Waiting for capacity](/curation/gpu-function-execution#capacity) for that
behavior in full. [Adaptive Execution](/enterprise/adaptive-execution)'s
resource-consumption-based sizing applies here too: a refresh's GPU
utilization is measured and fed back into sizing the *next* part the same
way CPU and memory are.
A single refresh can also override the Function's registered placement for
that run -- see
[Configure workers and resources](/feature-engineering/refresh-column#resources)
for the `gpu_class`/`gpu_count` options on `refresh_column_async` and the
equivalent SQL `WITH (...)` clause.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="GPU Function Execution" icon="microchip" href="/curation/gpu-function-execution">
The full placement and capacity-wait mechanism this page reuses.
</Card>
<Card title="Adaptive Execution" icon="gauge-high" href="/enterprise/adaptive-execution">
How a refresh's resource class, part size and worker count adjust themselves.
</Card>
<Card title="Configure workers and resources" icon="sliders" href="/feature-engineering/refresh-column#resources">
Override GPU class and count for one refresh call.
</Card>
</CardGroup>
+1 -1
View File
@@ -211,7 +211,7 @@ placement.
## Scaling interfaces {#scaling-interfaces}
- [Distributed search](/enterprise/distributed-search) for how a query
- [Distributed Search](/enterprise/distributed-search) for how a query
divides across query nodes and plan executors generally.
- [Architecture](/index#architecture) for the components a replica group
is made of.
@@ -11,7 +11,7 @@ availability:
Keeping a table in good condition takes several different kinds of work --
compacting files, catching indexes up, cleaning old versions, folding a
real-time write tier into the base table -- each with its own trigger. In
RTE write tier into the base table -- each with its own trigger. In
LanceDB Enterprise, a fleet of agent processes decides when each of these
needs to run, continuously, per table, in the background across your
cluster -- nothing to schedule, trigger, or monitor by hand unless you want
@@ -92,7 +92,7 @@ whichever ones are due:
1. **WAL index catch-up.** Checked first and gated by nothing else, because
it is correctness-driven rather than discretionary: while an index lags
behind the compacted generation, the real-time write tier cannot trim its
behind the compacted generation, the RTE write tier cannot trim its
log. See [RTE Compaction](/enterprise/realtime-engine-compaction).
2. **Index operations.** Reindexing a column, unless a compaction that
cannot defer its remap is about to run this same tick -- deferred-remap
@@ -100,7 +100,7 @@ whichever ones are due:
[Incremental Indexing](/indexing/incremental-indexing).
3. **Compaction.** Whether enough small fragments or deleted rows have
accumulated to merge. See [Compaction](/maintenance/compaction) and
[Distributed compaction](/enterprise/distributed-compaction).
[Distributed Compaction](/enterprise/distributed-compaction).
4. **Cleanup.** Independent of the other four -- it can always run in
parallel with them. [See below](#cleanup) for exactly when it's due.
5. **Deferred-remap catch-up.** The batched, amortized follow-up for index
@@ -8,7 +8,7 @@ availability:
enterprise: available
---
[Distributed compaction](/enterprise/distributed-compaction) merges small
[Distributed Compaction](/enterprise/distributed-compaction) merges small
base-table fragments and reclaims deleted rows -- the same operation
[Compaction](/maintenance/compaction) describes for LanceDB Embedded. A
table with the [real time engine](/tables/realtime-engine) enabled has a
@@ -191,7 +191,7 @@ Either way, the bound also assumes a free compaction slot when
the shard becomes eligible; an eligible shard can wait in queue behind
`WAL_COMPACT_CONCURRENCY` other passes already running, so the actual
delay can run longer than the trigger alone suggests. See
[Real Time reads](/search/realtime-read)
[RTE reads](/search/realtime-read)
for what a live RTE read sees instead.
## What to read next {#next}
@@ -200,7 +200,7 @@ for what a live RTE read sees instead.
<Card title="Real Time Engine" icon="bolt" href="/tables/realtime-engine">
The MemWAL format, shards, and WAL nodes this page assumes.
</Card>
<Card title="Distributed compaction" icon="compress" href="/enterprise/distributed-compaction">
<Card title="Distributed Compaction" icon="compress" href="/enterprise/distributed-compaction">
The unrelated, ordinary compaction every table -- WAL-enabled or not -- gets.
</Card>
<Card title="Job Framework" icon="list-check" href="/basics/jobs">
@@ -1,39 +0,0 @@
---
title: "Slurm Scheduling"
sidebarTitle: "Slurm Scheduling"
description: "A computed column or materialized view refresh launches worker pods through the same SUNK-backed scheduling configuration as any other job worker."
icon: "server"
keywords: ["slurm", "sunk", "scheduling", "refresh", "computed column"]
availability:
embedded: unavailable
enterprise: available
---
This page is about [Slurm scheduling](/curation/slurm-scheduling) as it
applies to a computed column or materialized view refresh specifically --
the mechanism itself is covered in depth on that Data Curation page. A
refresh's top-level job -- the one that shows up in `SHOW JOBS` -- is an
agent-launched job worker like any other, and shares the full scheduling
configuration: node selector, tolerations, scheduler name and
termination grace period alike.
<Warning>
That job's own *child* workers -- the ones that actually plan, finalize
and run a Function during the refresh -- are different: they inherit the
scheduler name, but not the termination grace period, so they launch
with Kubernetes' 30-second default regardless of the shared
configuration. See [What SUNK changes](/curation/slurm-scheduling#what-changes)
for why that matters on most SUNK deployments, and confirm with your
deployment operator before running a Function-backed refresh there.
</Warning>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Slurm Scheduling" icon="server" href="/curation/slurm-scheduling">
The full scheduling configuration this page reuses.
</Card>
<Card title="GPU Refresh Execution" icon="microchip" href="/enterprise/gpu-refresh-execution">
GPU-specific placement for a refresh's Function workers.
</Card>
</CardGroup>
+16 -12
View File
@@ -67,21 +67,25 @@ flowchart LR
Where enabled, the index disk tier and the
[data page cache](/enterprise/data-page-cache)'s disk tier draw from one
combined disk allocation per executor: a configurable percentage of the
provisioned volume (90% by default, leaving headroom for filesystem
metadata and kubelet usage), out of which the index disk tier reserves its
own configurable share of the *full* provisioned volume. On a 200 GiB
volume, the 90% default leaves 180 GiB combined; reserving 25% of the full
200 GiB (50 GiB) for the index disk tier leaves the remaining 130 GiB for
data pages. An exact byte split overrides the percentage split when you
need one tier sized precisely rather than proportionally. Setting the
index disk tier's share to zero disables it while leaving the in-memory
index cache and the data page cache's disk tier untouched.
combined, configurable disk allocation per executor, split between the two
tiers so neither one needs its own dedicated volume.
## Why it matters {#why-it-matters}
Vector, full-text, and scalar indexes can easily outgrow available memory
on large tables. Without this cache, an eviction means re-reading and
re-deserializing that index from object storage on the next request --
slow, and repeated on every eviction. Spilling to local disk instead keeps
index lookups fast even when the working set is larger than memory, without
giving every executor enough memory to hold every index at once.
[Contact us](https://www.lancedb.com/contact) to learn more about how
caching is sized and tuned for your workload.
## Index cache interfaces {#interfaces}
- [Cache Prewarm](/enterprise/cache-prewarm) for how `startup_prewarm_idx`
warms this cache ahead of the first request.
- [Cache Prewarm](/enterprise/cache-prewarm) for warming this cache ahead
of the first request.
- [Data Page Cache](/enterprise/data-page-cache) for the related cache
that holds raw table data pages instead of deserialized index structures.
- [Vector indexes](/indexing/vector-index), [full-text indexes](/indexing/fts-index),
+2 -2
View File
@@ -82,7 +82,7 @@ you need to [install PyTorch with CUDA support](https://pytorch.org/get-started/
<Card title="Incremental Indexing" icon="refresh" href="/indexing/incremental-indexing">
Keeping an index up-to-date as new data arrives.
</Card>
<Card title="GPU Function Execution" icon="microchip" href="/curation/gpu-function-execution">
How a GPU-bound Function's worker gets scheduled -- the same GPU-placement story for Functions instead of index builds.
<Card title="Slurm" icon="server" href="/deployment/slurm">
Running these GPU workers alongside training on shared Slurm-scheduled infrastructure.
</Card>
</CardGroup>
@@ -48,14 +48,6 @@ CREATE STORAGE INTEGRATION raw
STORAGE_AWS_ACCESS_KEY_ID_SECRET = aws_key_id
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = aws_key_secret
```
</CodeGroup>
{/*
Draft for the storage-integration SDK release (#4618). Keep this section
unpublished until create_storage_integration ships in a release; it is not
in v0.41.0.
<CodeGroup>
```python Python icon="python"
db.create_storage_integration(
"raw", "s3://customer-data/raw/",
@@ -70,7 +62,6 @@ db.create_storage_integration(
aws_secret_access_key_secret="aws_key_secret",
)
```
```rust
use lancedb::storage_integration::CreateStorageIntegrationRequest;
@@ -84,7 +75,6 @@ let info = db.create_storage_integration(
).await?;
```
</CodeGroup>
*/}
### Google Cloud Storage {#gcp}
@@ -95,14 +85,6 @@ CREATE STORAGE INTEGRATION gcs
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = gcp_service_account
```
</CodeGroup>
{/*
Draft for the storage-integration SDK release (#4618). Keep this section
unpublished until create_storage_integration ships in a release; it is not
in v0.41.0.
<CodeGroup>
```python Python icon="python"
db.create_storage_integration(
"gcs", "gs://customer-data/raw/",
@@ -111,7 +93,6 @@ db.create_storage_integration(
)
```
</CodeGroup>
*/}
The Secret holds the full service-account JSON key. GCP has no
role-based alternative.
@@ -128,14 +109,6 @@ CREATE STORAGE INTEGRATION blobs
STORAGE_AZURE_CLIENT_ID_SECRET = azure_client
STORAGE_AZURE_CLIENT_SECRET_SECRET = azure_credential
```
</CodeGroup>
{/*
Draft for the storage-integration SDK release (#4618). Keep this section
unpublished until create_storage_integration ships in a release; it is not
in v0.41.0.
<CodeGroup>
```python Python icon="python"
db.create_storage_integration(
"blobs", "az://customer-data/raw/",
@@ -147,7 +120,6 @@ db.create_storage_integration(
)
```
</CodeGroup>
*/}
All four options are required together; there is no role-based
alternative for Azure.
@@ -161,14 +133,6 @@ CREATE STORAGE INTEGRATION hub
STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
HUGGINGFACE_TOKEN_SECRET = hf_token
```
</CodeGroup>
{/*
Draft for the storage-integration SDK release (#4618, #4775). Keep this
section unpublished until create_storage_integration ships in a release;
it is not in v0.41.0.
<CodeGroup>
```python Python icon="python"
db.create_storage_integration(
"hub", "hf://datasets/owner/private/data/",
@@ -177,7 +141,6 @@ db.create_storage_integration(
)
```
</CodeGroup>
*/}
`HUGGINGFACE_TOKEN_SECRET` is required to create this integration; omitting
it is rejected, not treated as anonymous. For anonymous access at
+2 -2
View File
@@ -45,7 +45,7 @@ Azure examples.
## How the import runs {#distributed}
See [Distributed write](/enterprise/distributed-write) for how an import's
See [Distributed Write](/enterprise/distributed-write) for how an import's
writes divide across plan executors, and when a small import instead runs
as one local task.
@@ -65,7 +65,7 @@ omits the destination's unflushed data.
<Card title="Inspect Datasets" icon="database" href="/curation/inspect-datasets">
Query and import HuggingFace or object-storage datasets in full.
</Card>
<Card title="Distributed write" icon="arrows-split-up-and-left" href="/enterprise/distributed-write">
<Card title="Distributed Write" icon="arrows-split-up-and-left" href="/enterprise/distributed-write">
How an import's writes divide across plan executors.
</Card>
<Card title="Storage Integration" icon="cloud" href="/sql/ddl/storage-integration">
+7 -7
View File
@@ -1,5 +1,5 @@
---
title: "Real Time writes"
title: "RTE writes"
sidebarTitle: "RTE Write"
description: "Enable the real time engine on a table and absorb small, frequent writes cheaply, each one committed in milliseconds."
icon: "bolt"
@@ -12,7 +12,7 @@ availability:
The [real time engine](/tables/realtime-engine) puts an LSM-style write tier in
front of a table, turning many small writes into a few large commits. This
page covers enabling it and writing to a table; for querying an RTE table, see
[Real Time reads](/search/realtime-read).
[RTE reads](/search/realtime-read).
`table.enable_rte(spec)` is the same call on both connections, but what it
does underneath differs:
@@ -22,7 +22,7 @@ does underneath differs:
there is no separate serving layer to scale or configure.
- **Enterprise** forwards the call to the server, which runs it on a
dedicated, horizontally scalable WAL-node tier instead of your table's
query-serving fleet. See [Distributed Real Time Write](/enterprise/distributed-real-time-write)
query-serving fleet. See [Distributed RTE Write](/enterprise/distributed-real-time-write)
for how that tier shards a table's writes and the throughput limits that come with it.
## Getting started {#getting-started}
@@ -162,7 +162,7 @@ conditional updates or deleting rows missing from the source, are rejected.
</Warning>
Writes go through `merge_insert` on the primary key, and are visible to queries
immediately — see [Real Time reads](/search/realtime-read) for querying them
immediately — see [RTE reads](/search/realtime-read) for querying them
back:
<CodeGroup>
@@ -232,13 +232,13 @@ your own machine's resources.
An Enterprise table can be split into shards that write independently, with
each shard's durable-append throughput bounded by its WAL node's own
capacity -- measured, not an enforced cap. See
[Distributed Real Time Write](/enterprise/distributed-real-time-write#throughput)
[Distributed RTE Write](/enterprise/distributed-real-time-write#throughput)
for the numbers.
## Next steps {#next-steps}
- [Real Time reads](/search/realtime-read) — query an RTE table
- [Distributed Real Time Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales this
- [RTE reads](/search/realtime-read) — query an RTE table
- [Distributed RTE Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales this
- [How the real time engine works](/tables/realtime-engine#memwal-format)
- [Merge insert](/ingestion/merge-insert)
- [`ALTER TABLE ... SET RTE`](/sql/ddl/table#properties)
-7
View File
@@ -10,13 +10,6 @@ import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuick
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
<Tip>
Using LanceDB Enterprise? Only the connection changes. In step 2, use the
[Enterprise](#connect-to-lancedb-enterprise) connection with your deployment's database URI, API key, and endpoint
in place of the local directory. The remaining steps use the same API, except
steps 5 and 6, which need Enterprise either way.
</Tip>
## 1\. Install LanceDB {#1-install-lancedb}
Choose your language. Python uses Pandas to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
+7 -7
View File
@@ -1,5 +1,5 @@
---
title: "Real Time reads"
title: "RTE reads"
sidebarTitle: "RTE Read"
description: "Query a table with the real time engine enabled and get writes back within milliseconds of being written, merged transparently with the base table."
icon: "magnifying-glass"
@@ -12,9 +12,9 @@ availability:
A table with the [real time engine](/tables/realtime-engine) enabled queries
the same way as any other table — nothing about `search`, filtering, or SQL
changes. This page covers what you get back; for enabling RTE and writing to
a table, see [Real Time writes](/ingestion/realtime-write). For how Enterprise
a table, see [RTE writes](/ingestion/realtime-write). For how Enterprise
combines these results across machines, see
[Distributed Real Time Read](/enterprise/distributed-real-time-read).
[Distributed RTE Read](/enterprise/distributed-real-time-read).
## Query a real time table {#query}
@@ -25,7 +25,7 @@ moment ago is already reflected:
<CodeGroup>
```python Python icon="python"
# continuing the `frames` table connected and written to in Real Time writes
# continuing the `frames` table connected and written to in RTE writes
import numpy as np
query_vector = np.random.rand(frames.schema.field("embedding").type.list_size).astype("float32")
@@ -57,7 +57,7 @@ maintained-index rules for vector, full-text, and scalar search, just with no
network hop to a separate serving tier. There is no WAL node to route to and
nothing to combine across machines: a query against an RTE-enabled Embedded
table already reads the fresh tier together with the base table in one call.
See [Distributed Real Time Read](/enterprise/distributed-real-time-read) for
See [Distributed RTE Read](/enterprise/distributed-real-time-read) for
what Enterprise adds on top when that fresh tier is served by a separate
machine from the one answering the rest of the query.
@@ -77,8 +77,8 @@ available.
## Next steps {#next-steps}
- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table
- [Distributed Real Time Read](/enterprise/distributed-real-time-read) — combining results across machines in Enterprise
- [RTE writes](/ingestion/realtime-write) — enable RTE and write to a table
- [Distributed RTE Read](/enterprise/distributed-real-time-read) — combining results across machines in Enterprise
- [How the real time engine works](/tables/realtime-engine#memwal-format)
- [Hybrid search](/search/hybrid-search)
- [Vector search](/search/vector-search)
+8 -8
View File
@@ -25,7 +25,7 @@ table, and nothing about how you query that table changes.
RTE is built on **MemWAL**, a log-structured-merge (LSM) format that Lance
itself defines -- the structure Enterprise serves is part of the open Lance
format, not a layer bolted on beside it. A table with RTE enabled is divided
into one or more **shards** (see [Real Time writes](/ingestion/realtime-write#primary-key)
into one or more **shards** (see [RTE writes](/ingestion/realtime-write#primary-key)
for enabling and sizing them), and each shard holds:
- A **memtable**: the active, in-memory table for that shard's newest rows,
@@ -49,8 +49,8 @@ later folds SSTables into the base table, catching its indexes up as it goes.
This section describes LanceDB Enterprise's serving layer. LanceDB Embedded
runs the same MemWAL mechanics directly against the local dataset, in one
process -- there is no separate node to route to or scale; see
[Real Time writes](/ingestion/realtime-write#getting-started) and
[Real Time reads](/search/realtime-read#embedded) for what that looks like.
[RTE writes](/ingestion/realtime-write#getting-started) and
[RTE reads](/search/realtime-read#embedded) for what that looks like.
RTE's Enterprise serving layer is a dedicated component, separate from the
[query nodes and plan executors](/enterprise/distributed-search)
@@ -81,13 +81,13 @@ pods are never both appending to the same shard's WAL at once.
Every query runs against the base table and the fresh tier -- every shard's
memtable and not-yet-compacted SSTables -- in parallel. Fresh rows supersede
base rows with the same primary key, and the two are merged into one result.
See [Real Time reads](/search/realtime-read) for what this means for query
See [RTE reads](/search/realtime-read) for what this means for query
results.
## Next steps {#next-steps}
- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table
- [Real Time reads](/search/realtime-read) — query an RTE table
- [Distributed Real Time Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
- [Distributed Real Time Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
- [RTE writes](/ingestion/realtime-write) — enable RTE and write to a table
- [RTE reads](/search/realtime-read) — query an RTE table
- [Distributed RTE Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
- [Distributed RTE Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
- [RTE Compaction](/enterprise/realtime-engine-compaction) — how Enterprise folds shards into the base table