mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 23:56:52 +00:00
docs: add storage integration SDK examples, abstract RTE write, simplify cache pages (#4865)
## Summary - **Define Storage Integration**: un-gates the Python (and Rust, for S3) `create_storage_integration` examples, drafted against #4618 and held back until it shipped -- it's now in v0.42.0-beta.0+, and the signature matches the drafts exactly. - **Naming rules**: drops the `Branch` row and its "Learn more" link, consistent with de-emphasizing branches elsewhere in this nav. - **Distributed RTE Write**: removes the REST path, the `__lance_mem_wal` index name, and "WAL node" terminology in favor of an abstract framing -- a query node routes writes to RTE nodes, each of which can host shards from many different tables. Both diagram variants are updated to match. - **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop Helm-specific and exact-percentage detail (disk-budget math, `startup_prewarm_block_query_ready`/`healthPort`/failure-policy table) in favor of a short feature → architecture → why-it-matters shape, each closing with a contact-sales pointer. Data Page Cache's on-demand prewarm section is removed outright. ## Note I wasn't able to get a clean visual render of the updated Distributed RTE Write diagram in local preview -- Mermaid isn't rendering in headless Chrome in this environment for *any* page right now, confirmed against an unmodified reference page showing the identical symptom. The diagram source follows the exact pattern already shipped and working in the 21 diagrams from the earlier brand-theming pass, and `mint validate`/`mint broken-links` both pass clean. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
69ac2158a1
commit
5e38120bdc
@@ -42,7 +42,7 @@ file format, so moving data from one edition to the other does not require a dat
|
||||
| | [Hybrid search](/search/hybrid-search) | ✅ | ✅ |
|
||||
| | [Filtering](/search/filtering) | ✅ | ✅ |
|
||||
| | [Reranking](/reranking) | ✅ | ✅ |
|
||||
| | [Distributed search](/enterprise/distributed-search) | ❌ | ✅ |
|
||||
| | [Distributed Search](/enterprise/distributed-search) | ❌ | ✅ |
|
||||
| | [RTE Read](/search/realtime-read) | ✅ | ✅ |
|
||||
| | [Distributed RTE Read](/enterprise/distributed-real-time-read) | ❌ | ✅ |
|
||||
| | [Cache Prewarm](/enterprise/cache-prewarm) | ❌ | ✅ |
|
||||
@@ -52,19 +52,15 @@ file format, so moving data from one edition to the other does not require a dat
|
||||
| | [Aggregation and profiling](/curation/aggregate) | ❌ | ✅ |
|
||||
| | [Joins](/curation/join) | ❌ | ✅ |
|
||||
| | [Blob Late Materialization](/curation/blob-late-materialization) | ❌ | ✅ |
|
||||
| | [Ranking and deduplication](/curation/windowing) | ❌ | ✅ |
|
||||
| | [Ranking](/curation/windowing) | ❌ | ✅ |
|
||||
| | [Near-duplicate detection](/curation/dedupe) | ❌ | ✅ |
|
||||
| | [Chunking](/curation/chunking) | ❌ | ✅ |
|
||||
| | [Changes since a version](/curation/changes) | ❌ | ✅ |
|
||||
| | [GPU Function Execution](/curation/gpu-function-execution) | ❌ | ✅ |
|
||||
| | [Slurm Scheduling](/curation/slurm-scheduling) | ❌ | ✅ |
|
||||
| **Feature Engineering** | [Computed columns](/feature-engineering/add-computed-column) | ❌ | ✅ |
|
||||
| | [Incremental Refresh](/enterprise/incremental-refresh) | ❌ | ✅ |
|
||||
| | [Auto Refresh](/enterprise/auto-refresh) | ❌ | ✅ |
|
||||
| | [Checkpointing](/enterprise/checkpointing) | ❌ | ✅ |
|
||||
| | [Adaptive Execution](/enterprise/adaptive-execution) | ❌ | ✅ |
|
||||
| | [GPU Refresh Execution](/enterprise/gpu-refresh-execution) | ❌ | ✅ |
|
||||
| | [Slurm Scheduling](/enterprise/slurm-refresh-scheduling) | ❌ | ✅ |
|
||||
| | [Refresh Materialized View](/feature-engineering/refresh-materialized-view) | ❌ | ✅ |
|
||||
| | [Embedding registry](/embedding) | ✅ | ✅ |
|
||||
| **Training** | [PyTorch data loader](/training/data-loading) | ✅ | ✅ |
|
||||
@@ -94,7 +90,7 @@ file format, so moving data from one edition to the other does not require a dat
|
||||
| | [Compaction](/maintenance/compaction) | ✅ | ✅ |
|
||||
| | [RTE Compaction](/enterprise/realtime-engine-compaction) | ❌ | ✅ |
|
||||
| | [Garbage Collection](/maintenance/cleanup) | ✅ | ✅ |
|
||||
| | [Distributed compaction](/enterprise/distributed-compaction) | ❌ | ✅ |
|
||||
| | [Distributed Compaction](/enterprise/distributed-compaction) | ❌ | ✅ |
|
||||
| | [Advanced Conflict Resolution](/enterprise/advanced-conflict-resolution) | ❌ | ✅ |
|
||||
| **Security** | [Authentication](/enterprise/authentication) | ❌ | ✅ |
|
||||
| | [Authorization](/governance/authorization) | ❌ | ✅ |
|
||||
@@ -102,6 +98,8 @@ file format, so moving data from one edition to the other does not require a dat
|
||||
| **Deployment** | [Horizontal Scaling](/enterprise/horizontal-scaling) | ❌ | ✅ |
|
||||
| | [OpenTelemetry](/deployment/opentelemetry) | ✅ | ✅ |
|
||||
| | [Kubernetes](/enterprise/deployment/kubernetes) | ❌ | ✅ |
|
||||
| | [Multicloud](/deployment/multicloud) | ❌ | ✅ |
|
||||
| | [Slurm](/deployment/slurm) | ❌ | ✅ |
|
||||
|
||||
{/*
|
||||
Draft until branching reaches early access; restore to the Management rows.
|
||||
|
||||
+13
-37
@@ -227,45 +227,21 @@ described below.
|
||||
|
||||
### SQL-backed job handles
|
||||
|
||||
On Enterprise, materialized-view CREATE, REFRESH, and DROP use the SQL query backend.
|
||||
Python's `create_materialized_view_async()`, `mv.refresh_async()`, and
|
||||
`drop_materialized_view_async()` return job handles with `status()`, `wait()`,
|
||||
and `cancel()`. Synchronous creation, refresh, and drop wait on the same SQL
|
||||
execution before returning. A completed DROP means the catalog binding has been
|
||||
removed; physical cleanup continues in the background.
|
||||
On Enterprise, several catalog operations return a job handle backed by the
|
||||
SQL query engine rather than the job registry:
|
||||
|
||||
Remote function registration also returns a SQL query job. The SQL operation
|
||||
submits the durable image build and waits for catalog publication before returning
|
||||
the immutable FunctionVersion. `drop_function_async()` returns the catalog-removal
|
||||
boolean together with a query job; waiting on that handle confirms removal while
|
||||
physical cleanup continues separately.
|
||||
| Operation | SDK methods | Behavior |
|
||||
| --- | --- | --- |
|
||||
| Materialized views | `create_materialized_view_async()`, `refresh_async()`, `drop_materialized_view_async()` | Synchronous variants wait on the same SQL execution. Drop removes the catalog binding immediately; physical cleanup continues in the background. |
|
||||
| Functions | Registration, `drop_function_async()` | Registration waits for catalog publication before returning the immutable FunctionVersion. Drop confirms catalog removal while cleanup continues separately. |
|
||||
| Secrets | Create, alter, list, describe, drop | Credential values are bound as SQL parameters, never embedded in SQL text or returned in results. `DESCRIBE SECRET` additionally exposes rotation history. |
|
||||
| Views | Create, describe, list, drop | Create returns the stored definition and schema. Drop confirms catalog removal while cleanup continues separately. |
|
||||
|
||||
Remote Secret create, alter, list, describe, and drop use the SQL query backend.
|
||||
The synchronous APIs wait on query completion internally. Create and alter bind
|
||||
the credential as a parameter to
|
||||
`CREATE SECRET name TYPE = GENERIC_STRING SECRET_STRING = $1` or
|
||||
`ALTER SECRET name SET SECRET_STRING = $1`, keeping it out of SQL text. Results contain only
|
||||
completion status or Secret names and timestamps; they never return credential
|
||||
values. SQL `DESCRIBE SECRET` additionally exposes rotation history.
|
||||
|
||||
Remote view creation, description, listing, and drop also use SQL. Creation
|
||||
returns the stored definition and schema from the same SQL execution.
|
||||
`drop_view_async()` returns a query job: waiting confirms catalog removal while
|
||||
physical cleanup continues in the background.
|
||||
|
||||
The existing `job.job_type` identifies query-backed handles as `"query"` from
|
||||
submission, and `job.id` contains the connection-scoped SQL query UUID string.
|
||||
Use `describe_query(job.id)` on the submitting connection to inspect the query.
|
||||
Registry handles retain their server-reported job types (available after
|
||||
`status()` or `refresh()`) and registry IDs. Local handles have no ID or job type.
|
||||
|
||||
SQL-backed handles do not expose registry records or event history and cannot
|
||||
be reopened through `open_job()` or reattached after a client restart. Their IDs
|
||||
are not durable resume tokens. Cancellation follows the SQL server's
|
||||
capabilities and can be unsupported.
|
||||
|
||||
The server decides whether SQL executes on its serving fleet or starts SQL job
|
||||
workers. That placement does not change the handle's query-backed identity.
|
||||
All of these report `job.job_type` as `"query"`, with `job.id` holding the
|
||||
connection-scoped SQL query UUID -- use `describe_query(job.id)` to inspect
|
||||
one. Unlike registry handles, they don't expose event history, can't be
|
||||
reopened through `open_job()` or reattached after a client restart, and
|
||||
cancellation depends on the SQL server's support for it.
|
||||
|
||||
### Object teardown {#teardown}
|
||||
|
||||
|
||||
@@ -91,7 +91,7 @@ qualifies. It applies to a plain inner equi-join (a single `ON left_col
|
||||
`LIMIT` fused into it) where one side -- and only one side -- bottoms out
|
||||
at a single, otherwise-unfiltered table scan: no existing row filter,
|
||||
scan range, fragment subset, deleted-row handling, or
|
||||
[real-time write](/ingestion/realtime-write) tier. If both sides would
|
||||
[RTE write](/ingestion/realtime-write) tier. If both sides would
|
||||
otherwise qualify, the side with a BlobV2 column is preferred over the
|
||||
other; if neither or both sides have one, the rewrite declines rather
|
||||
than guessing which to narrow.
|
||||
|
||||
@@ -77,15 +77,19 @@ def split_into_passages(text):
|
||||
|
||||
## Explode it in a query {#explode}
|
||||
|
||||
Call the function in the `FROM` clause, after the source table, with an
|
||||
alias: a Function there yields one row per element of the list it returns,
|
||||
with every other column repeated. Reference the function's output fields
|
||||
through the alias:
|
||||
Wrap the function call in `UNNEST` in the `SELECT` list, with an alias: it
|
||||
yields one row per element of the list the function returns, with every
|
||||
other selected column repeated. Reference the exploded struct's fields
|
||||
through that alias in a query over this one:
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
WITH exploded AS (
|
||||
SELECT doc_id, title, UNNEST(split_into_passages(text)) AS p
|
||||
FROM documents
|
||||
)
|
||||
SELECT doc_id, title, p.chunk_index, p.chunk_text
|
||||
FROM documents, split_into_passages(text) AS p
|
||||
FROM exploded
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
@@ -160,8 +164,12 @@ timestamp, then call `extract_frame` on each row to produce its image:
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
WITH sampled AS (
|
||||
SELECT video_id, video, UNNEST(sample_frame_timestamps(duration_seconds)) AS f
|
||||
FROM videos
|
||||
)
|
||||
SELECT video_id, f.frame_index, f.ts, extract_frame(video, f.ts) AS jpeg
|
||||
FROM videos, sample_frame_timestamps(duration_seconds) AS f
|
||||
FROM sampled
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
@@ -179,8 +187,12 @@ recomputing the explosion on every query:
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
CREATE MATERIALIZED VIEW chunks AS
|
||||
WITH exploded AS (
|
||||
SELECT doc_id, title, UNNEST(split_into_passages(text)) AS p
|
||||
FROM documents
|
||||
)
|
||||
SELECT doc_id, title, p.chunk_index, p.chunk_text
|
||||
FROM documents, split_into_passages(text) AS p
|
||||
FROM exploded
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
@@ -194,10 +206,11 @@ FROM documents, split_into_passages(text) AS p
|
||||
The view refreshes like any other materialized view: new source rows are
|
||||
chunked and inserted, deleted source rows drop their children, and an
|
||||
updated row's old children are replaced by freshly chunked ones on the next
|
||||
[refresh](/feature-engineering/refresh-column). A single Function call or
|
||||
`UNNEST(column)` is the only lateral source a materialized view can carry --
|
||||
not both, and not combined with `GROUP BY` -- so keep aggregation or further
|
||||
curation in a second step over the chunked table.
|
||||
[refresh](/feature-engineering/refresh-column). A single `UNNEST(...)` --
|
||||
whether unnesting a Function's list output directly, as above, or an
|
||||
existing list column -- is the only lateral expansion a materialized view
|
||||
can carry, and it can't be combined with `GROUP BY`, so keep aggregation or
|
||||
further curation in a second step over the chunked table.
|
||||
|
||||
## Chunk, then curate {#next}
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ availability:
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
[Ranking and deduplication](/curation/windowing#dedup) removes records by an
|
||||
[Ranking](/curation/windowing#dedup) removes records by an
|
||||
explicit rule, such as keeping the latest version of a known key. This page
|
||||
covers the other kind of duplicate: records with no shared key that are still
|
||||
nearly identical, judged by distance between their embeddings -- a stopped
|
||||
|
||||
@@ -11,7 +11,7 @@ availability:
|
||||
A distributed SQL query plans as a sequence of **stages**, each with its
|
||||
own set of partitions, connected by an **exchange**: a streaming gRPC
|
||||
service that carries one stage's output to the next stage's plan
|
||||
executors. [Distributed search](/enterprise/distributed-search) covers
|
||||
executors. [Distributed Search](/enterprise/distributed-search) covers
|
||||
how a query gets placed on the fleet in the first place; this page covers
|
||||
what happens between placement and the final result, for a query whose
|
||||
plan needs data repartitioned across machines -- a hash join or a grouped
|
||||
@@ -102,7 +102,7 @@ connection.
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Distributed search" icon="network-wired" href="/enterprise/distributed-search">
|
||||
<Card title="Distributed Search" icon="network-wired" href="/enterprise/distributed-search">
|
||||
How a query gets placed on the query-serving fleet in the first place.
|
||||
</Card>
|
||||
<Card title="Query plans" icon="map" href="/basics/sql#query-plans">
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
---
|
||||
title: "GPU Function Execution"
|
||||
tag: "Early access"
|
||||
sidebarTitle: "GPU Functions"
|
||||
description: "How a GPU Function's worker gets scheduled onto a GPU-capable node, and what happens when none is free."
|
||||
icon: microchip
|
||||
keywords: ["gpu", "gpu class", "scheduling", "function", "udf", "kubernetes", "node selector"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="purple">Early access</Badge>
|
||||
|
||||
[Registering a GPU Function](/curation/create-function#gpu) sets `gpu=True`
|
||||
and an optional `gpu_class`; this page covers what happens after that, when
|
||||
a job actually needs a worker. An operator configures one or more named
|
||||
**GPU classes** for the deployment -- each one a Kubernetes node selector,
|
||||
toleration set, and resource name (such as `nvidia.com/gpu`) pointing at a
|
||||
specific pool of GPU nodes, plus a hardware description (GPU model, GPU
|
||||
memory, how many GPUs one worker may request). A Function with no
|
||||
`gpu_class` uses the deployment's `default` class.
|
||||
|
||||
## Placement {#placement}
|
||||
|
||||
When a job needs to run a GPU Function, the agent resolves its `gpu_class`
|
||||
(or `default`) to that class's node selector, tolerations, and resource
|
||||
name, and requests a worker pod shaped accordingly -- this is a separate
|
||||
placement path from the ordinary CPU worker ladder used by non-GPU
|
||||
Functions and other background jobs (see
|
||||
the [Job Framework](/basics/jobs) for that general job
|
||||
machinery). The resulting pod can only schedule onto a node matching the
|
||||
class's selector and tolerations, so a class is really naming a pool of
|
||||
nodes, not just a resource count.
|
||||
|
||||
## Waiting for capacity {#capacity}
|
||||
|
||||
A GPU class describes the deployment's hardware, not live availability --
|
||||
listing classes never tells you whether a GPU is free right now. If no
|
||||
node in the class has room, the worker pod stays pending rather than
|
||||
failing the job outright or falling back to a CPU worker: the job reports
|
||||
that it is waiting on that GPU class's capacity, and starts running as soon
|
||||
as a matching node has room. A job stuck waiting indefinitely usually means
|
||||
the class's node pool needs more capacity, not that anything is
|
||||
misconfigured.
|
||||
|
||||
{/*
|
||||
Draft for the list_gpu_classes release. Keep this section unpublished
|
||||
until that API ships -- v0.41.0 (the latest release) does not have it,
|
||||
and there is no SQL equivalent; discovery is a REST call
|
||||
(`GET /v1/gpu/classes`), not a SQL statement.
|
||||
|
||||
## Inspecting GPU classes {#inspecting}
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
for gpu in db.list_gpu_classes():
|
||||
print(gpu.name, gpu.gpu_model, gpu.gpu_memory_bytes, gpu.max_gpu_count_per_worker)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Returns the operator-configured classes: name, resource name, GPU model
|
||||
and memory, and the largest per-worker GPU count the class permits.
|
||||
Listing requires the System Usage privilege.
|
||||
*/}
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Create a Function" icon="code" href="/curation/create-function#gpu">
|
||||
Register a GPU Function and pin it to a specific class.
|
||||
</Card>
|
||||
<Card title="Job Framework" icon="list-check" href="/basics/jobs">
|
||||
The job types a deployment runs on its own schedule.
|
||||
</Card>
|
||||
<Card title="GPU Refresh Execution" icon="microchip" href="/enterprise/gpu-refresh-execution">
|
||||
The same placement path, for a computed column or MV refresh instead of a direct call.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
title: "Slurm Scheduling"
|
||||
sidebarTitle: "Slurm Scheduling"
|
||||
description: "Run LanceDB Enterprise worker pods on a Slurm-on-Kubernetes cluster, accounted by Slurm."
|
||||
icon: "server"
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
[Slurm](https://slurm.schedmd.com/) is the workload manager most GPU
|
||||
clusters run to schedule and account for jobs -- queueing work across
|
||||
shared nodes, enforcing fair-share and priority policies between teams,
|
||||
and tracking GPU-hour usage per job. That makes it the de facto standard
|
||||
in AI training infrastructure, where expensive GPU nodes are shared
|
||||
across many training and data jobs instead of dedicated to one team.
|
||||
CoreWeave, a major GPU cloud provider, runs Slurm this way through SUNK
|
||||
(Slurm on Kubernetes) -- Slurm's scheduling and accounting layered onto
|
||||
an otherwise ordinary Kubernetes cluster.
|
||||
|
||||
LanceDB Enterprise's job workers -- indexing, compaction, cache prewarm,
|
||||
registration, and most other background work -- all launch as ordinary
|
||||
Kubernetes pods through one shared scheduling configuration. Today, that
|
||||
configuration integrates with CoreWeave's SUNK specifically; the rest of
|
||||
this page describes its effect. On a Neocloud running SUNK, pointing
|
||||
that configuration at SUNK's pod scheduler puts every one of those
|
||||
worker pods under Slurm's scheduling and accounting, instead of plain
|
||||
Kubernetes bin-packing. See [Architecture](/index#architecture).
|
||||
|
||||
## What SUNK changes {#what-changes}
|
||||
|
||||
Most agent-launched worker pods -- a one-pod job worker or a multi-worker
|
||||
distributed indexer Job -- share one scheduling configuration: node
|
||||
selector, tolerations, scheduler name, and termination grace period.
|
||||
Pointing the scheduler name at the cluster's SUNK pod scheduler routes
|
||||
pod placement through SUNK instead of the default Kubernetes scheduler,
|
||||
so Slurm sees and accounts for the workload. SUNK also expects a short
|
||||
termination grace period, kept below Slurm's own kill-wait threshold --
|
||||
deployments commonly use 5 to 20 seconds.
|
||||
|
||||
<Warning>
|
||||
Function refresh's own child workers -- including GPU Function execution
|
||||
-- get the scheduler name but not this shared termination-grace setting;
|
||||
they currently launch with Kubernetes' 30-second default, above most
|
||||
SUNK deployments' kill-wait bound. Confirm with your deployment operator
|
||||
before running Function refresh on a SUNK-scheduled cluster. See
|
||||
[GPU Function execution](/curation/gpu-function-execution) for the
|
||||
placement mechanism this affects.
|
||||
</Warning>
|
||||
|
||||
## Configuring a SUNK-backed deployment {#config}
|
||||
|
||||
This scheduling configuration is set by a deployment operator, not by a
|
||||
SQL statement, table property, or SDK call -- there is no per-query or
|
||||
per-table knob for it. [Contact support](https://www.lancedb.com/contact)
|
||||
to configure or size a SUNK-backed deployment.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="GPU Function Execution" icon="microchip" href="/curation/gpu-function-execution">
|
||||
How a GPU-bound Function gets scheduled onto GPU-equipped workers.
|
||||
</Card>
|
||||
<Card title="Architecture" icon="sitemap" href="/index#architecture">
|
||||
The query, executor, cache and jobs fleets SUNK schedules onto.
|
||||
</Card>
|
||||
<Card title="Slurm Scheduling for refresh" icon="server" href="/enterprise/slurm-refresh-scheduling">
|
||||
What does and doesn't carry over for a computed column or MV refresh.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Ranking and deduplication"
|
||||
sidebarTitle: "Rank and Deduplicate"
|
||||
title: "Ranking"
|
||||
sidebarTitle: "Ranking"
|
||||
description: "Choose records with explicit ranking rules and deterministic tie handling."
|
||||
icon: chart-simple
|
||||
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: "Multicloud"
|
||||
sidebarTitle: "Multicloud"
|
||||
description: "Keep deployment orchestration in your cloud while dispatching compute-heavy work, like GPU UDF execution, to run in a different cloud."
|
||||
icon: "cloud"
|
||||
keywords: ["multicloud", "cross-cloud", "gpu", "deployment"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
A LanceDB Enterprise deployment doesn't have to keep all of its compute in
|
||||
one cloud. Orchestration -- query nodes, job coordination, the catalog --
|
||||
stays with your deployment in its own cloud account and region. Specific,
|
||||
compute-heavy work, such as a GPU-bound Function's execution during a
|
||||
refresh, can instead dispatch to run in a **different** cloud account,
|
||||
region, or provider entirely, and send its results back to complete the job.
|
||||
|
||||
## How it fits together {#architecture}
|
||||
|
||||
Your deployment's job orchestration plans the work and tracks it to
|
||||
completion the same way it would locally. The part that actually needs the
|
||||
remote capacity -- a GPU UDF call, for example -- runs across a network
|
||||
boundary instead of on a worker in your own cloud, and reports its result
|
||||
back the same way a local worker would.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
subgraph P["Your cloud"]
|
||||
direction TB
|
||||
QN["Query node"]
|
||||
Job["Job orchestration"]
|
||||
end
|
||||
|
||||
subgraph R["Remote cloud"]
|
||||
direction TB
|
||||
Exec["Compute execution<br/>(e.g. GPU UDFs)"]
|
||||
end
|
||||
|
||||
QN --> Job
|
||||
Job -- "dispatches work" --> Exec
|
||||
Exec -- "results" --> Job
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
||||
class Exec accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
subgraph P["Your cloud"]
|
||||
direction TB
|
||||
QN["Query node"]
|
||||
Job["Job orchestration"]
|
||||
end
|
||||
|
||||
subgraph R["Remote cloud"]
|
||||
direction TB
|
||||
Exec["Compute execution<br/>(e.g. GPU UDFs)"]
|
||||
end
|
||||
|
||||
QN --> Job
|
||||
Job -- "dispatches work" --> Exec
|
||||
Exec -- "results" --> Job
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class Exec accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Why it matters {#why-it-matters}
|
||||
|
||||
The cloud that's right for your primary deployment -- where your data and
|
||||
other infrastructure already live -- isn't always the cloud with the best
|
||||
GPU availability or pricing. Splitting orchestration from execution means
|
||||
you don't have to choose: keep your deployment where it already makes
|
||||
sense, and reach compute capacity elsewhere only for the work that
|
||||
actually needs it, without migrating the deployment or operating that
|
||||
remote capacity yourself.
|
||||
|
||||
[Contact us](https://www.lancedb.com/contact) to learn more about setting
|
||||
up cross-cloud execution for your deployment.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Slurm" icon="server" href="/deployment/slurm">
|
||||
A common way the remote cloud's compute gets scheduled: as Slurm jobs.
|
||||
</Card>
|
||||
<Card title="Adaptive Execution" icon="gauge-high" href="/enterprise/adaptive-execution">
|
||||
How a refresh's resource class, part size and worker count adjust themselves.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
title: "Slurm"
|
||||
sidebarTitle: "Slurm"
|
||||
description: "Run LanceDB Enterprise's own pods as Slurm-scheduled jobs, and reach a Neocloud's GPU capacity through Slurm in the multicloud case."
|
||||
icon: "server"
|
||||
keywords: ["slurm", "scheduling", "gpu", "neocloud", "preemption", "multicloud"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
[Slurm](https://slurm.schedmd.com/) is the workload manager most GPU
|
||||
clusters run to schedule and account for jobs -- queueing work across
|
||||
shared nodes, enforcing fair-share and priority policies between teams,
|
||||
and tracking GPU-hour usage per job. That makes it the de facto standard
|
||||
in AI training infrastructure, where expensive GPU nodes are shared
|
||||
across many training and data jobs instead of dedicated to one team.
|
||||
GPU-focused cloud providers ("Neoclouds") such as CoreWeave and Nebius run
|
||||
Slurm-based scheduling across their Kubernetes clusters for exactly this
|
||||
reason.
|
||||
|
||||
Slurm shows up for LanceDB Enterprise in two related ways. Its own pods --
|
||||
query nodes, plan executors, and background job workers alike -- can run
|
||||
as Slurm-scheduled jobs rather than plain Kubernetes pods, under the same
|
||||
scheduling and accounting as everything else on the cluster. And in the
|
||||
[multicloud](/deployment/multicloud) case, when the remote cloud is a
|
||||
Neocloud, Slurm is commonly how the dispatched GPU work actually gets
|
||||
scheduled there.
|
||||
|
||||
## How it fits together {#architecture}
|
||||
|
||||
Scheduling LanceDB's pods through Slurm puts them under the same
|
||||
scheduler as GPU training jobs, so the cluster's capacity is accounted
|
||||
for consistently regardless of which one is using it -- Slurm's
|
||||
fair-share and priority policies apply to LanceDB the same way they
|
||||
apply to a training job, including preemption when training needs the
|
||||
capacity back.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
subgraph P["Your cloud"]
|
||||
direction TB
|
||||
Job["Job orchestration"]
|
||||
end
|
||||
|
||||
subgraph N["Neocloud (e.g. CoreWeave, Nebius)"]
|
||||
direction TB
|
||||
Slurm["Slurm scheduler"]
|
||||
Train["GPU training jobs"]
|
||||
LDB["LanceDB Enterprise pods"]
|
||||
Slurm --> Train
|
||||
Slurm --> LDB
|
||||
Train -.->|"can preempt"| LDB
|
||||
end
|
||||
|
||||
Job -- "dispatches GPU work" --> Slurm
|
||||
LDB -- "results" --> Job
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
||||
class Slurm accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
subgraph P["Your cloud"]
|
||||
direction TB
|
||||
Job["Job orchestration"]
|
||||
end
|
||||
|
||||
subgraph N["Neocloud (e.g. CoreWeave, Nebius)"]
|
||||
direction TB
|
||||
Slurm["Slurm scheduler"]
|
||||
Train["GPU training jobs"]
|
||||
LDB["LanceDB Enterprise pods"]
|
||||
Slurm --> Train
|
||||
Slurm --> LDB
|
||||
Train -.->|"can preempt"| LDB
|
||||
end
|
||||
|
||||
Job -- "dispatches GPU work" --> Slurm
|
||||
LDB -- "results" --> Job
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class Slurm accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Why it matters {#why-it-matters}
|
||||
|
||||
Most GPU infrastructure already runs Slurm for training. Deploying LanceDB
|
||||
Enterprise into that same infrastructure, under the same scheduler, means
|
||||
it can share capacity with training workloads safely -- queued, accounted,
|
||||
and preemptible like everything else -- instead of requiring a separate,
|
||||
dedicated pool of nodes carved out just for LanceDB. Combined with
|
||||
[multicloud](/deployment/multicloud) dispatch, that makes it practical to
|
||||
keep your primary deployment wherever it already lives and still reach
|
||||
GPU capacity on a Neocloud built around Slurm, only for the work that
|
||||
needs it.
|
||||
|
||||
[Contact us](https://www.lancedb.com/contact) to learn more about running
|
||||
LanceDB Enterprise on a Slurm-scheduled cluster.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Multicloud" icon="cloud" href="/deployment/multicloud">
|
||||
The cross-cloud dispatch model this page's Neocloud case builds on.
|
||||
</Card>
|
||||
<Card title="Architecture" icon="sitemap" href="/index#architecture">
|
||||
The query, executor, cache and jobs fleets this scheduling applies to.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
+15
-9
@@ -121,11 +121,11 @@
|
||||
},
|
||||
{
|
||||
"source": "/curation/sunk-integration",
|
||||
"destination": "/curation/slurm-scheduling"
|
||||
"destination": "/deployment/slurm"
|
||||
},
|
||||
{
|
||||
"source": "/curation/slurm-integration",
|
||||
"destination": "/curation/slurm-scheduling"
|
||||
"destination": "/deployment/slurm"
|
||||
},
|
||||
{
|
||||
"source": "/curation/sampling",
|
||||
@@ -362,6 +362,14 @@
|
||||
{
|
||||
"source": "/enterprise/distributed-column-refresh",
|
||||
"destination": "/enterprise/adaptive-execution"
|
||||
},
|
||||
{
|
||||
"source": "/curation/slurm-scheduling",
|
||||
"destination": "/deployment/slurm"
|
||||
},
|
||||
{
|
||||
"source": "/enterprise/slurm-refresh-scheduling",
|
||||
"destination": "/deployment/slurm"
|
||||
}
|
||||
],
|
||||
"navigation": {
|
||||
@@ -452,9 +460,7 @@
|
||||
"curation/clustering",
|
||||
"curation/dedupe",
|
||||
"feature-engineering/view",
|
||||
"curation/vend-secret",
|
||||
"curation/gpu-function-execution",
|
||||
"curation/slurm-scheduling"
|
||||
"curation/vend-secret"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -468,9 +474,7 @@
|
||||
"enterprise/incremental-refresh",
|
||||
"enterprise/auto-refresh",
|
||||
"enterprise/checkpointing",
|
||||
"enterprise/adaptive-execution",
|
||||
"enterprise/gpu-refresh-execution",
|
||||
"enterprise/slurm-refresh-scheduling"
|
||||
"enterprise/adaptive-execution"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -534,7 +538,9 @@
|
||||
"pages": [
|
||||
"enterprise/horizontal-scaling",
|
||||
"deployment/opentelemetry",
|
||||
"enterprise/deployment/kubernetes"
|
||||
"enterprise/deployment/kubernetes",
|
||||
"deployment/multicloud",
|
||||
"deployment/slurm"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -136,7 +136,4 @@ part sizing described above still decides how many rows each part covers.
|
||||
<Card title="Checkpointing" icon="floppy-disk" href="/enterprise/checkpointing">
|
||||
How a part's progress survives a failure or restart.
|
||||
</Card>
|
||||
<Card title="GPU Refresh Execution" icon="microchip" href="/enterprise/gpu-refresh-execution">
|
||||
How a GPU-bound Function's worker gets placed, for a refresh specifically.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -190,10 +190,10 @@ flowchart TD
|
||||
<Card title="Incremental Indexing" icon="arrows-rotate" href="/indexing/incremental-indexing#index">
|
||||
The `index_remap` job type, and keeping an index current as the table changes.
|
||||
</Card>
|
||||
<Card title="Distributed compaction" icon="layer-group" href="/enterprise/distributed-compaction">
|
||||
<Card title="Distributed Compaction" icon="layer-group" href="/enterprise/distributed-compaction">
|
||||
How compaction itself scales across the jobs fleet.
|
||||
</Card>
|
||||
<Card title="Distributed indexing" icon="diagram-project" href="/enterprise/distributed-indexing">
|
||||
<Card title="Distributed Indexing" icon="diagram-project" href="/enterprise/distributed-indexing">
|
||||
How an index build scales across the jobs fleet.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -3,61 +3,83 @@ title: "Cache Prewarm"
|
||||
sidebarTitle: "Cache Prewarm"
|
||||
description: "Populate the tiered index cache ahead of a request, instead of filling it lazily on first read."
|
||||
icon: "fire"
|
||||
keywords: ["prewarm", "cache", "page cache", "index cache", "startup"]
|
||||
keywords: ["prewarm", "cache", "index cache", "startup"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
This page is about `startup_prewarm_idx`, which warms the distributed
|
||||
[tiered index cache](/enterprise/tiered-index-cache) a plan executor keeps
|
||||
as part of Enterprise's serving layer -- LanceDB Embedded has no such
|
||||
cache, so there is nothing for it to warm on Embedded. Embedded does have
|
||||
its own, separate `prewarm_index()` on a table, an idempotent hint that
|
||||
loads a built index into memory ahead of a query; it has no equivalent for
|
||||
raw column data, where only Enterprise supports prewarming. On Enterprise,
|
||||
the index cache fills lazily on first read by default; `startup_prewarm_idx`
|
||||
loads it ahead of that first request instead. The
|
||||
[data page cache](/enterprise/data-page-cache) is a separate cache with its
|
||||
own, on-demand prewarm -- `startup_prewarm_idx` does not touch it.
|
||||
Cache Prewarm loads the [tiered index cache](/enterprise/tiered-index-cache)
|
||||
a plan executor keeps as part of Enterprise's serving layer, instead of
|
||||
leaving it to fill lazily on first read. You configure which tables to
|
||||
prewarm; on startup, each plan executor loads those tables' indexes into
|
||||
cache before it joins the fleet. The executor does not accept queries until
|
||||
prewarming finishes, so it never serves a request against a cold cache.
|
||||
|
||||
## At plan executor startup {#startup}
|
||||
LanceDB Embedded has no such cache, so there is nothing for it to warm
|
||||
there. Embedded does have its own, separate `prewarm_index()` on a table --
|
||||
an idempotent hint that loads a built index into memory ahead of a query.
|
||||
|
||||
`startup_prewarm_idx` runs as part of a plan executor's own boot sequence,
|
||||
before it joins the fleet, rather than on a schedule. A deployment configures
|
||||
it with `startup_prewarm_enabled`, an index selection (which columns' indexes
|
||||
to warm, up to 1024 entries), and a concurrency limit for how many it loads
|
||||
at once.
|
||||
<div className="hidden dark:block">
|
||||
|
||||
Whether this delays the executor from serving queries is a separate setting,
|
||||
`startup_prewarm_block_query_ready`, off by default in the binary itself but
|
||||
**on by default in the Helm chart** -- most deployments block on it unless
|
||||
the chart value is set otherwise:
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
List[("Configured list<br/>of tables to prewarm")]
|
||||
Start["Plan executor starts<br/>(not ready — no queries served)"] --> Warm["Loads the configured<br/>tables' indexes into cache"]
|
||||
List -.-> Warm
|
||||
Warm --> Ready["Ready — joins the fleet,<br/>starts serving queries"]
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
||||
class Warm,List accent
|
||||
```
|
||||
|
||||
- **Unset:** the executor becomes query-ready immediately and
|
||||
prewarm runs concurrently in the background. Early queries may still miss
|
||||
the cache; later ones benefit once prewarm catches up.
|
||||
- **Set (the Helm chart's default):** query readiness waits for prewarm to
|
||||
finish, up to `startup_prewarm_timeout_secs`. Blocking query readiness
|
||||
this way requires `healthPort` to be set (9096 in the chart's default
|
||||
values), so process health stays reachable while readiness is gated.
|
||||
</div>
|
||||
|
||||
If prewarm fails or times out, `startup_prewarm_failure_policy` decides what
|
||||
happens next:
|
||||
<div className="block dark:hidden">
|
||||
|
||||
| Policy | Behavior |
|
||||
| --- | --- |
|
||||
| `mark_ready_cold` (default) | The executor becomes query-ready anyway, just without a warm cache. |
|
||||
| `stay_not_ready` | Combined with blocking query readiness, the executor never becomes ready -- a failed prewarm keeps it out of the fleet rather than serving cold. |
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
List[("Configured list<br/>of tables to prewarm")]
|
||||
Start["Plan executor starts<br/>(not ready — no queries served)"] --> Warm["Loads the configured<br/>tables' indexes into cache"]
|
||||
List -.-> Warm
|
||||
Warm --> Ready["Ready — joins the fleet,<br/>starts serving queries"]
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class Warm,List accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
## Why it matters {#why-it-matters}
|
||||
|
||||
Without prewarming, a fresh or restarted executor would fill its indexes
|
||||
lazily, so the first queries it serves pay the cost of loading and
|
||||
deserializing them from object storage. Holding the executor out of the
|
||||
fleet until its configured tables are warm means every query it ever
|
||||
serves gets a warm cache, so performance stays consistent through restarts
|
||||
and rollouts instead of dipping right after each one.
|
||||
|
||||
[Contact us](https://www.lancedb.com/contact) to learn more about how
|
||||
prewarming is configured for your deployment.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Data Page Cache" icon="database" href="/enterprise/data-page-cache">
|
||||
The raw-data-page cache -- warmed on demand, not by startup prewarm.
|
||||
The raw-data-page cache -- a separate cache from the one prewarm fills.
|
||||
</Card>
|
||||
<Card title="Tiered Index Cache" icon="layer-group" href="/enterprise/tiered-index-cache">
|
||||
The deserialized-index cache startup prewarm fills.
|
||||
The deserialized-index cache prewarm fills.
|
||||
</Card>
|
||||
<Card title="Architecture" icon="diagram-project" href="/index#architecture">
|
||||
Where plan executors and query nodes sit in the serving fleet.
|
||||
|
||||
@@ -56,36 +56,21 @@ flowchart LR
|
||||
|
||||
</div>
|
||||
|
||||
A table's local disk cache shares one budget with the
|
||||
A table's local disk cache shares one configurable budget with the
|
||||
[tiered index cache](/enterprise/tiered-index-cache#disk-budget)'s disk
|
||||
tier where both are enabled on the same executor -- raising one's share of
|
||||
that budget lowers the other's.
|
||||
tier where both are enabled on the same executor.
|
||||
|
||||
## Prewarming on demand {#prewarm}
|
||||
## Why it matters {#why-it-matters}
|
||||
|
||||
[Cache Prewarm](/enterprise/cache-prewarm)'s `startup_prewarm_idx` only
|
||||
warms the [tiered index cache](/enterprise/tiered-index-cache), not this
|
||||
one. The data page cache has its own, separate on-demand prewarm, run by a
|
||||
deployment operator rather than called from application code, optionally
|
||||
scoped to specific columns:
|
||||
Object storage is durable and cheap, but every round trip to it adds
|
||||
latency and egress cost. For workloads that read the same ranges
|
||||
repeatedly -- a hot subset of embeddings, a frequently-filtered column --
|
||||
the data page cache turns most of those reads into local memory or disk
|
||||
hits instead, so performance stays predictable as traffic grows and you
|
||||
pay object storage once per page, not once per read.
|
||||
|
||||
<CodeGroup>
|
||||
```bash CLI icon="terminal"
|
||||
lancedb admin cache prewarm data --table embedding_table --column embedding
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The command returns once the request is accepted, not once prewarming
|
||||
finishes; it is idempotent and safe to run concurrently from multiple
|
||||
clients. Prewarming has a large upfront cost, so it pays off for workloads
|
||||
that repeatedly query the same columns -- large columns such as embeddings
|
||||
or binary data may not be practical to prewarm this way.
|
||||
|
||||
<Warning>
|
||||
The OSS SDK's `table.prewarm_data()` targets LanceDB Cloud, a different
|
||||
product; calling it against an Enterprise deployment 404s. Use the admin
|
||||
command above instead.
|
||||
</Warning>
|
||||
[Contact us](https://www.lancedb.com/contact) to learn more about how
|
||||
caching is sized and tuned for your workload.
|
||||
|
||||
## Cache interfaces {#interfaces}
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Distributed compaction"
|
||||
sidebarTitle: "Distributed compaction"
|
||||
title: "Distributed Compaction"
|
||||
sidebarTitle: "Distributed Compaction"
|
||||
description: "How LanceDB Enterprise distributes compaction across a fleet of job workers."
|
||||
icon: "compress"
|
||||
availability:
|
||||
@@ -12,8 +12,8 @@ Distributed compaction splits a table's fragments into independent,
|
||||
fragment-disjoint tasks and fans them out across job workers in LanceDB
|
||||
Enterprise. Unlike an index build, a compaction plan needs no bin-packing:
|
||||
Lance's own planner already produces disjoint tasks, so workers claim and
|
||||
execute them directly through the job registry, with no coordination
|
||||
needed to decide who does what. See [Architecture](/index#architecture).
|
||||
execute them directly, with no coordination needed to decide who does
|
||||
what. See [Architecture](/index#architecture).
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
@@ -24,16 +24,16 @@ needed to decide who does what. See [Architecture](/index#architecture).
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
|
||||
R --> W1["Worker<br/>claims fragments 1-2"]
|
||||
R --> W2["Worker<br/>claims fragments 3-4"]
|
||||
R --> W3["Worker<br/>claims fragments 5-6"]
|
||||
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"]
|
||||
L --> W1["Worker<br/>claims fragments 1-2"]
|
||||
L --> W2["Worker<br/>claims fragments 3-4"]
|
||||
L --> W3["Worker<br/>claims fragments 5-6"]
|
||||
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
||||
class R,C accent
|
||||
class C accent
|
||||
```
|
||||
|
||||
</div>
|
||||
@@ -47,16 +47,16 @@ flowchart LR
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"] --> R[("Job registry")]
|
||||
R --> W1["Worker<br/>claims fragments 1-2"]
|
||||
R --> W2["Worker<br/>claims fragments 3-4"]
|
||||
R --> W3["Worker<br/>claims fragments 5-6"]
|
||||
L["Lance's planner<br/>splits fragments into<br/>disjoint tasks"]
|
||||
L --> W1["Worker<br/>claims fragments 1-2"]
|
||||
L --> W2["Worker<br/>claims fragments 3-4"]
|
||||
L --> W3["Worker<br/>claims fragments 5-6"]
|
||||
W1 --> C["Fleet-wide eligibility gate:<br/>one worker claims the commit"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class R,C accent
|
||||
class C accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
@@ -1,19 +1,33 @@
|
||||
---
|
||||
title: "Distributed indexing"
|
||||
sidebarTitle: "Distributed indexing"
|
||||
description: "How LanceDB Enterprise distributes an index build across a fleet of job workers."
|
||||
title: "Distributed Indexing"
|
||||
sidebarTitle: "Distributed Indexing"
|
||||
description: "How LanceDB Enterprise distributes a vector index build across a fleet of workers, stage by stage."
|
||||
icon: "layer-group"
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
Distributed indexing bin-packs a vector, full-text, or scalar index build into
|
||||
a plan of claimable tasks, then fans the plan out across job workers in
|
||||
LanceDB Enterprise. Workers claim tasks independently through the job
|
||||
registry and commit the finished index together. A vector index build first
|
||||
trains a shared IVF/PQ model from sampled data, then workers build storage
|
||||
from that model in parallel. See [Architecture](/index#architecture).
|
||||
A vector index build splits into stages, and the data in scope narrows at
|
||||
each one: training works from a small sample, shuffling touches every row
|
||||
once, and merging works entirely within one partition at a time. Full-text
|
||||
and scalar index builds skip training and shuffling -- they bin-pack
|
||||
directly into independent tasks workers claim and build in parallel,
|
||||
closer to how [distributed compaction](/enterprise/distributed-compaction)
|
||||
splits work. See [Architecture](/index#architecture).
|
||||
|
||||
## Stages of a vector index build {#stages}
|
||||
|
||||
1. **Train.** A sample of the table's vectors -- not the full dataset --
|
||||
trains the shared model every worker will use: IVF partition centroids,
|
||||
and for IVF-PQ, a PQ codebook on top of them.
|
||||
2. **Shuffle.** Every row in the table gets assigned to its nearest
|
||||
trained partition and regrouped so that all of one partition's rows end
|
||||
up together, wherever in the table they originally lived.
|
||||
3. **Merge.** Each partition is now self-contained, so workers build and
|
||||
merge partitions in parallel with no cross-partition coordination --
|
||||
one worker's output never depends on another's.
|
||||
4. **Commit.** The finished index is committed to the table.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
@@ -24,18 +38,17 @@ from that model in parallel. See [Architecture](/index#architecture).
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
|
||||
T --> P["Bin-pack build plan<br/>into claimable tasks"]
|
||||
P --> R[("Job registry")]
|
||||
R --> W1["Worker<br/>claims a task"]
|
||||
R --> W2["Worker<br/>claims a task"]
|
||||
R --> W3["Worker<br/>claims a task"]
|
||||
W1 --> C["Commit the<br/>finished index"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
S["Sample"] --> T["Train IVF centroids<br/>+ PQ codebook"]
|
||||
T --> Sh["Shuffle: assign every row<br/>to its nearest partition"]
|
||||
Sh --> M1["Merge<br/>partition A"]
|
||||
Sh --> M2["Merge<br/>partition B"]
|
||||
Sh --> M3["Merge<br/>partition C"]
|
||||
M1 --> C["Commit the<br/>finished index"]
|
||||
M2 --> C
|
||||
M3 --> C
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
||||
class R accent
|
||||
class Sh accent
|
||||
```
|
||||
|
||||
</div>
|
||||
@@ -49,18 +62,17 @@ flowchart LR
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
S["Sample data"] --> T["Train shared IVF/PQ model<br/>(vector index only)"]
|
||||
T --> P["Bin-pack build plan<br/>into claimable tasks"]
|
||||
P --> R[("Job registry")]
|
||||
R --> W1["Worker<br/>claims a task"]
|
||||
R --> W2["Worker<br/>claims a task"]
|
||||
R --> W3["Worker<br/>claims a task"]
|
||||
W1 --> C["Commit the<br/>finished index"]
|
||||
W2 --> C
|
||||
W3 --> C
|
||||
S["Sample"] --> T["Train IVF centroids<br/>+ PQ codebook"]
|
||||
T --> Sh["Shuffle: assign every row<br/>to its nearest partition"]
|
||||
Sh --> M1["Merge<br/>partition A"]
|
||||
Sh --> M2["Merge<br/>partition B"]
|
||||
Sh --> M3["Merge<br/>partition C"]
|
||||
M1 --> C["Commit the<br/>finished index"]
|
||||
M2 --> C
|
||||
M3 --> C
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class R accent
|
||||
class Sh accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: "Distributed Real Time Read"
|
||||
title: "Distributed RTE Read"
|
||||
sidebarTitle: "Distributed RTE Read"
|
||||
description: "How a query combines the real time engine's fresh tier with the ordinary distributed query-serving path, machine to machine."
|
||||
icon: "network-wired"
|
||||
@@ -8,13 +8,13 @@ availability:
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
[Real Time Read](/search/realtime-read) covers what a query against an
|
||||
[RTE Read](/search/realtime-read) covers what a query against an
|
||||
RTE-enabled table returns: base-table rows merged with the fresh tier,
|
||||
fresh rows superseding base rows with the same primary key. On a single
|
||||
Embedded process that merge is one function call against one Lance
|
||||
dataset. In LanceDB Enterprise, the base table and the fresh tier are
|
||||
usually served by different machines -- plan executors for the base table,
|
||||
a [WAL node](/tables/realtime-engine#wal-nodes) for the table's shards --
|
||||
an [RTE node](/tables/realtime-engine#wal-nodes) for the table's shards --
|
||||
so a query has to combine their results across the network instead.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
@@ -34,7 +34,7 @@ flowchart LR
|
||||
B1["Base-table arm<br/>index-covered scan or ANN search"]
|
||||
end
|
||||
|
||||
subgraph WN["WAL node (owns this table's shards)"]
|
||||
subgraph WN["RTE node (owns this table's shards)"]
|
||||
direction TB
|
||||
W1["SSTable arm<br/>index-covered"]
|
||||
W2["Active memtable arm<br/>vector: in-memory HNSW*<br/>FTS/scalar: indexed"]
|
||||
@@ -73,7 +73,7 @@ flowchart LR
|
||||
B1["Base-table arm<br/>index-covered scan or ANN search"]
|
||||
end
|
||||
|
||||
subgraph WN["WAL node (owns this table's shards)"]
|
||||
subgraph WN["RTE node (owns this table's shards)"]
|
||||
direction TB
|
||||
W1["SSTable arm<br/>index-covered"]
|
||||
W2["Active memtable arm<br/>vector: in-memory HNSW*<br/>FTS/scalar: indexed"]
|
||||
@@ -99,24 +99,76 @@ flowchart LR
|
||||
|
||||
The distributed plan for an RTE-enabled table is an ordinary plan executor
|
||||
query plan with one additional node spliced in: a union exec that queries
|
||||
the owning WAL node's shard and merges its rows into the base arm's stream
|
||||
the owning RTE node's shard and merges its rows into the base arm's stream
|
||||
by primary key, keeping the newest generation on a collision and cutting
|
||||
back to the requested top-K once both sides have contributed. An optimizer
|
||||
pass inserts this node only when the query actually touches a WAL-enabled
|
||||
pass inserts this node only when the query actually touches an RTE-enabled
|
||||
table; a query that touches none produces a plan identical to one with no
|
||||
RTE tables involved at all, so RTE adds no overhead to queries that never
|
||||
use it.
|
||||
|
||||
Because the base arm already read through the plan-executor fleet's ordinary
|
||||
caches and index structures, the only new network hop this adds is the one
|
||||
to the WAL node that owns the table's shards -- the same WAL node [Real Time
|
||||
to the RTE node that owns the table's shards -- the same RTE node [Real Time
|
||||
writes](/ingestion/realtime-write) land on, found by the query node through
|
||||
the same consistent hash of database and table name.
|
||||
|
||||
## Semantic search stays index-covered {#index-coverage}
|
||||
## Search stays index-covered {#index-coverage}
|
||||
|
||||
Every tier the union exec reads from -- the base table, the RTE node's
|
||||
SSTables, and its currently-active memtable -- is covered by a maintained
|
||||
index, for vector, full-text, and scalar search alike:
|
||||
|
||||
| Layer | Vector search | Full-text / scalar search |
|
||||
| --- | --- | --- |
|
||||
| Base table | ANN index | Maintained index |
|
||||
| SSTables | ANN index | Maintained index |
|
||||
| Active memtable | In-memory HNSW graph\* | Maintained index |
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
B["Base table"] --> I1["Vector, full-text, and<br/>scalar index coverage"]
|
||||
S["SSTables"] --> I1
|
||||
M["Active memtable"] --> I2["Full-text and scalar index<br/>coverage; vector via<br/>in-memory HNSW*"]
|
||||
I1 --> R["No unindexed data<br/>left to scan in full"]
|
||||
I2 --> R
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
||||
class R accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
B["Base table"] --> I1["Vector, full-text, and<br/>scalar index coverage"]
|
||||
S["SSTables"] --> I1
|
||||
M["Active memtable"] --> I2["Full-text and scalar index<br/>coverage; vector via<br/>in-memory HNSW*"]
|
||||
I1 --> R["No unindexed data<br/>left to scan in full"]
|
||||
I2 --> R
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class R accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
For full-text and scalar search, a maintained index covers every tier the
|
||||
union exec reads from -- the base table, the WAL node's SSTables, and its
|
||||
union exec reads from -- the base table, the RTE node's SSTables, and its
|
||||
currently-active memtable -- so a row written a moment ago is just as
|
||||
index-searchable as one written yesterday. If a resident memtable predates
|
||||
an index being added to the maintained set, the query is rejected outright
|
||||
@@ -147,13 +199,13 @@ table alone.
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Real Time Read" icon="magnifying-glass" href="/search/realtime-read">
|
||||
<Card title="RTE Read" icon="magnifying-glass" href="/search/realtime-read">
|
||||
What a query returns, and how Embedded does this same merge in-process.
|
||||
</Card>
|
||||
<Card title="Distributed Real Time Write" icon="bolt" href="/enterprise/distributed-real-time-write">
|
||||
How writes reach the WAL node this page's queries read from.
|
||||
<Card title="Distributed RTE Write" icon="bolt" href="/enterprise/distributed-real-time-write">
|
||||
How writes reach the RTE node this page's queries read from.
|
||||
</Card>
|
||||
<Card title="Distributed search" icon="network-wired" href="/enterprise/distributed-search">
|
||||
<Card title="Distributed Search" icon="network-wired" href="/enterprise/distributed-search">
|
||||
The ordinary plan-executor path this layers on top of.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -1,30 +1,32 @@
|
||||
---
|
||||
title: "Distributed Real Time Write"
|
||||
title: "Distributed RTE Write"
|
||||
sidebarTitle: "Distributed RTE Write"
|
||||
description: "How Enterprise scales real time writes horizontally across a dedicated WAL-node tier, and the throughput limits that come with it."
|
||||
description: "How Enterprise scales RTE writes horizontally across a dedicated serving tier, and the throughput limits that come with it."
|
||||
icon: "bolt"
|
||||
keywords: ["rte", "real time engine", "wal node", "shard", "sharding", "horizontal scaling", "write throughput", "memwal"]
|
||||
keywords: ["rte", "real time engine", "shard", "sharding", "horizontal scaling", "write throughput"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
[Real Time writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
|
||||
[RTE writes](/ingestion/realtime-write) describes `table.enable_rte(spec)`
|
||||
as one call that works the same way on either connection. On Enterprise, that
|
||||
call does not run locally -- it posts to the server, which installs the write
|
||||
path on a dedicated, horizontally scalable serving tier instead of running it
|
||||
in your process. This page covers that tier: how it shards a table's writes,
|
||||
the one real limit sharding does not remove, and the throughput you can expect.
|
||||
|
||||
## The WAL-node tier {#wal-nodes}
|
||||
## How writes are routed {#routing}
|
||||
|
||||
Enabling RTE on a remote table sends the spec to the server
|
||||
(`POST /v1/table/<name>/enable_rte/`), which installs Lance's `__lance_mem_wal`
|
||||
index on the dataset. From then on, every request for that table -- reads and
|
||||
writes alike -- is consistent-hashed on the database and table name to a
|
||||
single owning **WAL node**, a stateful pod separate from the query nodes and
|
||||
plan executors that serve ordinary queries. WAL node pods share nothing but
|
||||
the underlying object storage; no pod talks to another one directly.
|
||||
Enabling RTE on a remote table sends the spec to the server, which installs
|
||||
the write path on the dataset instead of running it in your process. From
|
||||
then on, the query node routes every request for that table -- reads and
|
||||
writes alike -- to a single owning **RTE node**, a stateful component
|
||||
separate from the query nodes and plan executors that serve ordinary
|
||||
queries. A query node routes different tables to different RTE nodes, and
|
||||
a single RTE node can host shards from many different tables at once. RTE
|
||||
nodes share nothing but the underlying object storage; no node talks to
|
||||
another one directly.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
@@ -36,22 +38,22 @@ the underlying object storage; no pod talks to another one directly.
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>hashes (db, table) to one WAL node"]
|
||||
Q["Query node<br/>routes each table's shards to an RTE node"]
|
||||
|
||||
subgraph W0["WAL node A — owns table 'frames'"]
|
||||
subgraph W0["RTE node A"]
|
||||
direction TB
|
||||
S0["Shard 0<br/>ShardWriter"]
|
||||
S1["Shard 1<br/>ShardWriter"]
|
||||
S2["Shard 2<br/>ShardWriter"]
|
||||
S0["Shard 0 — table 'frames'"]
|
||||
S1["Shard 0 — table 'events'"]
|
||||
end
|
||||
|
||||
subgraph W1["WAL node B — owns table 'events'"]
|
||||
subgraph W1["RTE node B"]
|
||||
direction TB
|
||||
S3["Shard 0<br/>ShardWriter"]
|
||||
S2["Shard 0 — table 'metrics'"]
|
||||
end
|
||||
|
||||
Q -- "writes + reads for 'frames'" --> W0
|
||||
Q -- "writes + reads for 'events'" --> W1
|
||||
Q -- "'frames'" --> W0
|
||||
Q -- "'events'" --> W0
|
||||
Q -- "'metrics'" --> W1
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
```
|
||||
|
||||
@@ -67,22 +69,22 @@ flowchart LR
|
||||
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
Q["Query node<br/>hashes (db, table) to one WAL node"]
|
||||
Q["Query node<br/>routes each table's shards to an RTE node"]
|
||||
|
||||
subgraph W0["WAL node A — owns table 'frames'"]
|
||||
subgraph W0["RTE node A"]
|
||||
direction TB
|
||||
S0["Shard 0<br/>ShardWriter"]
|
||||
S1["Shard 1<br/>ShardWriter"]
|
||||
S2["Shard 2<br/>ShardWriter"]
|
||||
S0["Shard 0 — table 'frames'"]
|
||||
S1["Shard 0 — table 'events'"]
|
||||
end
|
||||
|
||||
subgraph W1["WAL node B — owns table 'events'"]
|
||||
subgraph W1["RTE node B"]
|
||||
direction TB
|
||||
S3["Shard 0<br/>ShardWriter"]
|
||||
S2["Shard 0 — table 'metrics'"]
|
||||
end
|
||||
|
||||
Q -- "writes + reads for 'frames'" --> W0
|
||||
Q -- "writes + reads for 'events'" --> W1
|
||||
Q -- "'frames'" --> W0
|
||||
Q -- "'events'" --> W0
|
||||
Q -- "'metrics'" --> W1
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
```
|
||||
|
||||
@@ -91,25 +93,16 @@ flowchart LR
|
||||
## Shards scale one table, replicas scale the cluster {#shards-vs-replicas}
|
||||
|
||||
`RTEWriteSpec::bucket(column, num_buckets)` splits a table into `num_buckets`
|
||||
**table shards**, each a separate, independently-appending `ShardWriter`. A
|
||||
shard's identity is derived from the table's own URI
|
||||
(`uuid_v5(WAL_SHARD_NS, "{table_uri}#{i}")`), not assigned by an operator, so
|
||||
a shard survives the table being claimed by a different pod later.
|
||||
**table shards**, each a separate, independently-appending stream, giving the
|
||||
table that many parallel write paths instead of one. A shard's identity is
|
||||
derived from the table itself, not assigned by an operator, so a shard
|
||||
survives the table being claimed by a different RTE node later.
|
||||
|
||||
This is where the horizontal scaling actually happens, and where its limit
|
||||
is: **one WAL node owns every shard of a given table.** Splitting `frames`
|
||||
into 8 shards gives it 8 independent append streams, all still served by the
|
||||
one pod that owns `frames` -- it raises that table's own ceiling, not the
|
||||
number of pods working on it. Adding WAL node replicas instead spreads
|
||||
*other* tables' ownership across more pods, raising how many tables the
|
||||
cluster can serve, not how fast any single one of them can write.
|
||||
|
||||
<Note>
|
||||
A single table's write throughput is capped by one pod's capacity, however
|
||||
many shards you give it. A workload that needs to exceed one pod's ceiling on
|
||||
one table is not solved by adding shards -- there is no path to spreading one
|
||||
table's shards across multiple WAL nodes today.
|
||||
</Note>
|
||||
This is where the horizontal scaling actually happens: splitting `frames`
|
||||
into 8 shards gives it 8 independent append streams instead of 1, raising
|
||||
that table's own write ceiling. Adding RTE node replicas spreads shard
|
||||
ownership across more nodes, so the cluster can host more tables -- and more
|
||||
shards -- without any one node becoming a bottleneck for everything it serves.
|
||||
|
||||
`RTEWriteSpec::identity(column)` and `RTEWriteSpec::unsharded()` both resolve
|
||||
to a single table shard -- `bucket` is the sharding mode that actually
|
||||
@@ -121,22 +114,22 @@ scaling.
|
||||
A table shard is the unit of capacity: measured runs show it sustaining
|
||||
roughly 85-90 MB/s of durable append before acknowledgement, and throughput
|
||||
scaling nearly linearly with shard count up to at least 8 shards on one
|
||||
pod -- the bottleneck is the durable write itself, not CPU or memory. This
|
||||
node -- the bottleneck is the durable write itself, not CPU or memory. This
|
||||
is observed capacity, not an enforced quota; nothing in the deployment caps
|
||||
a shard's or a table's throughput below it. [Contact
|
||||
support](https://www.lancedb.com/contact) if your workload needs more than
|
||||
one pod's worth of capacity for a single table.
|
||||
support](https://www.lancedb.com/contact) if your workload needs more write
|
||||
throughput than this provides.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Real Time writes" icon="bolt" href="/ingestion/realtime-write">
|
||||
<Card title="RTE writes" icon="bolt" href="/ingestion/realtime-write">
|
||||
Enable RTE and write to a table; the same call on Embedded or Enterprise.
|
||||
</Card>
|
||||
<Card title="How the real time engine works" icon="database" href="/tables/realtime-engine">
|
||||
MemWAL, shards, memtables and SSTables -- the shared architecture.
|
||||
</Card>
|
||||
<Card title="Distributed Real Time Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
|
||||
<Card title="Distributed RTE Read" icon="magnifying-glass" href="/enterprise/distributed-real-time-read">
|
||||
How reads combine the WAL tier's fresh rows with the base table across the fleet.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Distributed search"
|
||||
sidebarTitle: "Distributed search"
|
||||
title: "Distributed Search"
|
||||
sidebarTitle: "Distributed Search"
|
||||
description: "How LanceDB Enterprise distributes index execution and table-data reads."
|
||||
icon: "network-wired"
|
||||
availability:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Distributed write"
|
||||
sidebarTitle: "Distributed write"
|
||||
title: "Distributed Write"
|
||||
sidebarTitle: "Distributed Write"
|
||||
description: "How LanceDB Enterprise distributes a CTAS import's writes across plan executors."
|
||||
icon: upload
|
||||
availability:
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
title: "GPU Refresh Execution"
|
||||
tag: "Early access"
|
||||
sidebarTitle: "GPU Refresh Execution"
|
||||
description: "A computed column or materialized view filled by a GPU Function schedules its refresh workers the same way GPU Function execution does in Data Curation."
|
||||
icon: microchip
|
||||
keywords: ["gpu", "gpu class", "scheduling", "refresh", "computed column", "function"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="purple">Early access</Badge>
|
||||
|
||||
A [computed column](/feature-engineering/add-computed-column) or
|
||||
[materialized view](/feature-engineering/materialized-views) filled by a
|
||||
registered [Function](/curation/create-function) with `gpu=True` refreshes
|
||||
through the exact same placement path as
|
||||
[GPU Function execution](/curation/gpu-function-execution) in Data
|
||||
Curation -- refreshing a column is not a separate scheduling code path
|
||||
from calling the Function directly, just a different caller.
|
||||
|
||||
<div className="hidden dark:block">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
|
||||
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
|
||||
"edgeLabelBackground": "#171513", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
|
||||
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
|
||||
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
|
||||
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
|
||||
R --> P
|
||||
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
|
||||
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
|
||||
class P accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
<div className="block dark:hidden">
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {
|
||||
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
|
||||
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
|
||||
"edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
|
||||
}}}%%
|
||||
flowchart LR
|
||||
F["Function registered with<br/>gpu=True, gpu_class"] --> A{"Who requests<br/>a worker?"}
|
||||
A -->|"You, directly"| D["Data Curation:<br/>GPU Function execution"]
|
||||
A -->|"A refresh job"| R["Feature Engineering:<br/>computed column / MV refresh"]
|
||||
D --> P["Same GPU-class placement,<br/>same capacity-wait behavior"]
|
||||
R --> P
|
||||
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
|
||||
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
|
||||
class P accent
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
The Function's `gpu_class` (or the deployment's `default`) resolves to the
|
||||
same node selector, tolerations and resource name either way, and a
|
||||
refresh that needs more GPU capacity than is currently free waits the
|
||||
same way a direct call would -- see
|
||||
[Waiting for capacity](/curation/gpu-function-execution#capacity) for that
|
||||
behavior in full. [Adaptive Execution](/enterprise/adaptive-execution)'s
|
||||
resource-consumption-based sizing applies here too: a refresh's GPU
|
||||
utilization is measured and fed back into sizing the *next* part the same
|
||||
way CPU and memory are.
|
||||
|
||||
A single refresh can also override the Function's registered placement for
|
||||
that run -- see
|
||||
[Configure workers and resources](/feature-engineering/refresh-column#resources)
|
||||
for the `gpu_class`/`gpu_count` options on `refresh_column_async` and the
|
||||
equivalent SQL `WITH (...)` clause.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="GPU Function Execution" icon="microchip" href="/curation/gpu-function-execution">
|
||||
The full placement and capacity-wait mechanism this page reuses.
|
||||
</Card>
|
||||
<Card title="Adaptive Execution" icon="gauge-high" href="/enterprise/adaptive-execution">
|
||||
How a refresh's resource class, part size and worker count adjust themselves.
|
||||
</Card>
|
||||
<Card title="Configure workers and resources" icon="sliders" href="/feature-engineering/refresh-column#resources">
|
||||
Override GPU class and count for one refresh call.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -211,7 +211,7 @@ placement.
|
||||
|
||||
## Scaling interfaces {#scaling-interfaces}
|
||||
|
||||
- [Distributed search](/enterprise/distributed-search) for how a query
|
||||
- [Distributed Search](/enterprise/distributed-search) for how a query
|
||||
divides across query nodes and plan executors generally.
|
||||
- [Architecture](/index#architecture) for the components a replica group
|
||||
is made of.
|
||||
|
||||
@@ -11,7 +11,7 @@ availability:
|
||||
|
||||
Keeping a table in good condition takes several different kinds of work --
|
||||
compacting files, catching indexes up, cleaning old versions, folding a
|
||||
real-time write tier into the base table -- each with its own trigger. In
|
||||
RTE write tier into the base table -- each with its own trigger. In
|
||||
LanceDB Enterprise, a fleet of agent processes decides when each of these
|
||||
needs to run, continuously, per table, in the background across your
|
||||
cluster -- nothing to schedule, trigger, or monitor by hand unless you want
|
||||
@@ -92,7 +92,7 @@ whichever ones are due:
|
||||
|
||||
1. **WAL index catch-up.** Checked first and gated by nothing else, because
|
||||
it is correctness-driven rather than discretionary: while an index lags
|
||||
behind the compacted generation, the real-time write tier cannot trim its
|
||||
behind the compacted generation, the RTE write tier cannot trim its
|
||||
log. See [RTE Compaction](/enterprise/realtime-engine-compaction).
|
||||
2. **Index operations.** Reindexing a column, unless a compaction that
|
||||
cannot defer its remap is about to run this same tick -- deferred-remap
|
||||
@@ -100,7 +100,7 @@ whichever ones are due:
|
||||
[Incremental Indexing](/indexing/incremental-indexing).
|
||||
3. **Compaction.** Whether enough small fragments or deleted rows have
|
||||
accumulated to merge. See [Compaction](/maintenance/compaction) and
|
||||
[Distributed compaction](/enterprise/distributed-compaction).
|
||||
[Distributed Compaction](/enterprise/distributed-compaction).
|
||||
4. **Cleanup.** Independent of the other four -- it can always run in
|
||||
parallel with them. [See below](#cleanup) for exactly when it's due.
|
||||
5. **Deferred-remap catch-up.** The batched, amortized follow-up for index
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
[Distributed compaction](/enterprise/distributed-compaction) merges small
|
||||
[Distributed Compaction](/enterprise/distributed-compaction) merges small
|
||||
base-table fragments and reclaims deleted rows -- the same operation
|
||||
[Compaction](/maintenance/compaction) describes for LanceDB Embedded. A
|
||||
table with the [real time engine](/tables/realtime-engine) enabled has a
|
||||
@@ -191,7 +191,7 @@ Either way, the bound also assumes a free compaction slot when
|
||||
the shard becomes eligible; an eligible shard can wait in queue behind
|
||||
`WAL_COMPACT_CONCURRENCY` other passes already running, so the actual
|
||||
delay can run longer than the trigger alone suggests. See
|
||||
[Real Time reads](/search/realtime-read)
|
||||
[RTE reads](/search/realtime-read)
|
||||
for what a live RTE read sees instead.
|
||||
|
||||
## What to read next {#next}
|
||||
@@ -200,7 +200,7 @@ for what a live RTE read sees instead.
|
||||
<Card title="Real Time Engine" icon="bolt" href="/tables/realtime-engine">
|
||||
The MemWAL format, shards, and WAL nodes this page assumes.
|
||||
</Card>
|
||||
<Card title="Distributed compaction" icon="compress" href="/enterprise/distributed-compaction">
|
||||
<Card title="Distributed Compaction" icon="compress" href="/enterprise/distributed-compaction">
|
||||
The unrelated, ordinary compaction every table -- WAL-enabled or not -- gets.
|
||||
</Card>
|
||||
<Card title="Job Framework" icon="list-check" href="/basics/jobs">
|
||||
|
||||
@@ -1,39 +0,0 @@
|
||||
---
|
||||
title: "Slurm Scheduling"
|
||||
sidebarTitle: "Slurm Scheduling"
|
||||
description: "A computed column or materialized view refresh launches worker pods through the same SUNK-backed scheduling configuration as any other job worker."
|
||||
icon: "server"
|
||||
keywords: ["slurm", "sunk", "scheduling", "refresh", "computed column"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
This page is about [Slurm scheduling](/curation/slurm-scheduling) as it
|
||||
applies to a computed column or materialized view refresh specifically --
|
||||
the mechanism itself is covered in depth on that Data Curation page. A
|
||||
refresh's top-level job -- the one that shows up in `SHOW JOBS` -- is an
|
||||
agent-launched job worker like any other, and shares the full scheduling
|
||||
configuration: node selector, tolerations, scheduler name and
|
||||
termination grace period alike.
|
||||
|
||||
<Warning>
|
||||
That job's own *child* workers -- the ones that actually plan, finalize
|
||||
and run a Function during the refresh -- are different: they inherit the
|
||||
scheduler name, but not the termination grace period, so they launch
|
||||
with Kubernetes' 30-second default regardless of the shared
|
||||
configuration. See [What SUNK changes](/curation/slurm-scheduling#what-changes)
|
||||
for why that matters on most SUNK deployments, and confirm with your
|
||||
deployment operator before running a Function-backed refresh there.
|
||||
</Warning>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Slurm Scheduling" icon="server" href="/curation/slurm-scheduling">
|
||||
The full scheduling configuration this page reuses.
|
||||
</Card>
|
||||
<Card title="GPU Refresh Execution" icon="microchip" href="/enterprise/gpu-refresh-execution">
|
||||
GPU-specific placement for a refresh's Function workers.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -67,21 +67,25 @@ flowchart LR
|
||||
|
||||
Where enabled, the index disk tier and the
|
||||
[data page cache](/enterprise/data-page-cache)'s disk tier draw from one
|
||||
combined disk allocation per executor: a configurable percentage of the
|
||||
provisioned volume (90% by default, leaving headroom for filesystem
|
||||
metadata and kubelet usage), out of which the index disk tier reserves its
|
||||
own configurable share of the *full* provisioned volume. On a 200 GiB
|
||||
volume, the 90% default leaves 180 GiB combined; reserving 25% of the full
|
||||
200 GiB (50 GiB) for the index disk tier leaves the remaining 130 GiB for
|
||||
data pages. An exact byte split overrides the percentage split when you
|
||||
need one tier sized precisely rather than proportionally. Setting the
|
||||
index disk tier's share to zero disables it while leaving the in-memory
|
||||
index cache and the data page cache's disk tier untouched.
|
||||
combined, configurable disk allocation per executor, split between the two
|
||||
tiers so neither one needs its own dedicated volume.
|
||||
|
||||
## Why it matters {#why-it-matters}
|
||||
|
||||
Vector, full-text, and scalar indexes can easily outgrow available memory
|
||||
on large tables. Without this cache, an eviction means re-reading and
|
||||
re-deserializing that index from object storage on the next request --
|
||||
slow, and repeated on every eviction. Spilling to local disk instead keeps
|
||||
index lookups fast even when the working set is larger than memory, without
|
||||
giving every executor enough memory to hold every index at once.
|
||||
|
||||
[Contact us](https://www.lancedb.com/contact) to learn more about how
|
||||
caching is sized and tuned for your workload.
|
||||
|
||||
## Index cache interfaces {#interfaces}
|
||||
|
||||
- [Cache Prewarm](/enterprise/cache-prewarm) for how `startup_prewarm_idx`
|
||||
warms this cache ahead of the first request.
|
||||
- [Cache Prewarm](/enterprise/cache-prewarm) for warming this cache ahead
|
||||
of the first request.
|
||||
- [Data Page Cache](/enterprise/data-page-cache) for the related cache
|
||||
that holds raw table data pages instead of deserialized index structures.
|
||||
- [Vector indexes](/indexing/vector-index), [full-text indexes](/indexing/fts-index),
|
||||
|
||||
@@ -82,7 +82,7 @@ you need to [install PyTorch with CUDA support](https://pytorch.org/get-started/
|
||||
<Card title="Incremental Indexing" icon="refresh" href="/indexing/incremental-indexing">
|
||||
Keeping an index up-to-date as new data arrives.
|
||||
</Card>
|
||||
<Card title="GPU Function Execution" icon="microchip" href="/curation/gpu-function-execution">
|
||||
How a GPU-bound Function's worker gets scheduled -- the same GPU-placement story for Functions instead of index builds.
|
||||
<Card title="Slurm" icon="server" href="/deployment/slurm">
|
||||
Running these GPU workers alongside training on shared Slurm-scheduled infrastructure.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -48,14 +48,6 @@ CREATE STORAGE INTEGRATION raw
|
||||
STORAGE_AWS_ACCESS_KEY_ID_SECRET = aws_key_id
|
||||
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = aws_key_secret
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
{/*
|
||||
Draft for the storage-integration SDK release (#4618). Keep this section
|
||||
unpublished until create_storage_integration ships in a release; it is not
|
||||
in v0.41.0.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
db.create_storage_integration(
|
||||
"raw", "s3://customer-data/raw/",
|
||||
@@ -70,7 +62,6 @@ db.create_storage_integration(
|
||||
aws_secret_access_key_secret="aws_key_secret",
|
||||
)
|
||||
```
|
||||
|
||||
```rust
|
||||
use lancedb::storage_integration::CreateStorageIntegrationRequest;
|
||||
|
||||
@@ -84,7 +75,6 @@ let info = db.create_storage_integration(
|
||||
).await?;
|
||||
```
|
||||
</CodeGroup>
|
||||
*/}
|
||||
|
||||
### Google Cloud Storage {#gcp}
|
||||
|
||||
@@ -95,14 +85,6 @@ CREATE STORAGE INTEGRATION gcs
|
||||
STORAGE_PREFIX = 'gs://customer-data/raw/'
|
||||
GCP_CREDENTIALS_JSON_SECRET = gcp_service_account
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
{/*
|
||||
Draft for the storage-integration SDK release (#4618). Keep this section
|
||||
unpublished until create_storage_integration ships in a release; it is not
|
||||
in v0.41.0.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
db.create_storage_integration(
|
||||
"gcs", "gs://customer-data/raw/",
|
||||
@@ -111,7 +93,6 @@ db.create_storage_integration(
|
||||
)
|
||||
```
|
||||
</CodeGroup>
|
||||
*/}
|
||||
|
||||
The Secret holds the full service-account JSON key. GCP has no
|
||||
role-based alternative.
|
||||
@@ -128,14 +109,6 @@ CREATE STORAGE INTEGRATION blobs
|
||||
STORAGE_AZURE_CLIENT_ID_SECRET = azure_client
|
||||
STORAGE_AZURE_CLIENT_SECRET_SECRET = azure_credential
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
{/*
|
||||
Draft for the storage-integration SDK release (#4618). Keep this section
|
||||
unpublished until create_storage_integration ships in a release; it is not
|
||||
in v0.41.0.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
db.create_storage_integration(
|
||||
"blobs", "az://customer-data/raw/",
|
||||
@@ -147,7 +120,6 @@ db.create_storage_integration(
|
||||
)
|
||||
```
|
||||
</CodeGroup>
|
||||
*/}
|
||||
|
||||
All four options are required together; there is no role-based
|
||||
alternative for Azure.
|
||||
@@ -161,14 +133,6 @@ CREATE STORAGE INTEGRATION hub
|
||||
STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
|
||||
HUGGINGFACE_TOKEN_SECRET = hf_token
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
{/*
|
||||
Draft for the storage-integration SDK release (#4618, #4775). Keep this
|
||||
section unpublished until create_storage_integration ships in a release;
|
||||
it is not in v0.41.0.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
db.create_storage_integration(
|
||||
"hub", "hf://datasets/owner/private/data/",
|
||||
@@ -177,7 +141,6 @@ db.create_storage_integration(
|
||||
)
|
||||
```
|
||||
</CodeGroup>
|
||||
*/}
|
||||
|
||||
`HUGGINGFACE_TOKEN_SECRET` is required to create this integration; omitting
|
||||
it is rejected, not treated as anonymous. For anonymous access at
|
||||
|
||||
@@ -45,7 +45,7 @@ Azure examples.
|
||||
|
||||
## How the import runs {#distributed}
|
||||
|
||||
See [Distributed write](/enterprise/distributed-write) for how an import's
|
||||
See [Distributed Write](/enterprise/distributed-write) for how an import's
|
||||
writes divide across plan executors, and when a small import instead runs
|
||||
as one local task.
|
||||
|
||||
@@ -65,7 +65,7 @@ omits the destination's unflushed data.
|
||||
<Card title="Inspect Datasets" icon="database" href="/curation/inspect-datasets">
|
||||
Query and import HuggingFace or object-storage datasets in full.
|
||||
</Card>
|
||||
<Card title="Distributed write" icon="arrows-split-up-and-left" href="/enterprise/distributed-write">
|
||||
<Card title="Distributed Write" icon="arrows-split-up-and-left" href="/enterprise/distributed-write">
|
||||
How an import's writes divide across plan executors.
|
||||
</Card>
|
||||
<Card title="Storage Integration" icon="cloud" href="/sql/ddl/storage-integration">
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: "Real Time writes"
|
||||
title: "RTE writes"
|
||||
sidebarTitle: "RTE Write"
|
||||
description: "Enable the real time engine on a table and absorb small, frequent writes cheaply, each one committed in milliseconds."
|
||||
icon: "bolt"
|
||||
@@ -12,7 +12,7 @@ availability:
|
||||
The [real time engine](/tables/realtime-engine) puts an LSM-style write tier in
|
||||
front of a table, turning many small writes into a few large commits. This
|
||||
page covers enabling it and writing to a table; for querying an RTE table, see
|
||||
[Real Time reads](/search/realtime-read).
|
||||
[RTE reads](/search/realtime-read).
|
||||
|
||||
`table.enable_rte(spec)` is the same call on both connections, but what it
|
||||
does underneath differs:
|
||||
@@ -22,7 +22,7 @@ does underneath differs:
|
||||
there is no separate serving layer to scale or configure.
|
||||
- **Enterprise** forwards the call to the server, which runs it on a
|
||||
dedicated, horizontally scalable WAL-node tier instead of your table's
|
||||
query-serving fleet. See [Distributed Real Time Write](/enterprise/distributed-real-time-write)
|
||||
query-serving fleet. See [Distributed RTE Write](/enterprise/distributed-real-time-write)
|
||||
for how that tier shards a table's writes and the throughput limits that come with it.
|
||||
|
||||
## Getting started {#getting-started}
|
||||
@@ -162,7 +162,7 @@ conditional updates or deleting rows missing from the source, are rejected.
|
||||
</Warning>
|
||||
|
||||
Writes go through `merge_insert` on the primary key, and are visible to queries
|
||||
immediately — see [Real Time reads](/search/realtime-read) for querying them
|
||||
immediately — see [RTE reads](/search/realtime-read) for querying them
|
||||
back:
|
||||
|
||||
<CodeGroup>
|
||||
@@ -232,13 +232,13 @@ your own machine's resources.
|
||||
An Enterprise table can be split into shards that write independently, with
|
||||
each shard's durable-append throughput bounded by its WAL node's own
|
||||
capacity -- measured, not an enforced cap. See
|
||||
[Distributed Real Time Write](/enterprise/distributed-real-time-write#throughput)
|
||||
[Distributed RTE Write](/enterprise/distributed-real-time-write#throughput)
|
||||
for the numbers.
|
||||
|
||||
## Next steps {#next-steps}
|
||||
|
||||
- [Real Time reads](/search/realtime-read) — query an RTE table
|
||||
- [Distributed Real Time Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales this
|
||||
- [RTE reads](/search/realtime-read) — query an RTE table
|
||||
- [Distributed RTE Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales this
|
||||
- [How the real time engine works](/tables/realtime-engine#memwal-format)
|
||||
- [Merge insert](/ingestion/merge-insert)
|
||||
- [`ALTER TABLE ... SET RTE`](/sql/ddl/table#properties)
|
||||
|
||||
@@ -10,13 +10,6 @@ import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuick
|
||||
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
|
||||
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
|
||||
|
||||
<Tip>
|
||||
Using LanceDB Enterprise? Only the connection changes. In step 2, use the
|
||||
[Enterprise](#connect-to-lancedb-enterprise) connection with your deployment's database URI, API key, and endpoint
|
||||
in place of the local directory. The remaining steps use the same API, except
|
||||
steps 5 and 6, which need Enterprise either way.
|
||||
</Tip>
|
||||
|
||||
## 1\. Install LanceDB {#1-install-lancedb}
|
||||
|
||||
Choose your language. Python uses Pandas to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: "Real Time reads"
|
||||
title: "RTE reads"
|
||||
sidebarTitle: "RTE Read"
|
||||
description: "Query a table with the real time engine enabled and get writes back within milliseconds of being written, merged transparently with the base table."
|
||||
icon: "magnifying-glass"
|
||||
@@ -12,9 +12,9 @@ availability:
|
||||
A table with the [real time engine](/tables/realtime-engine) enabled queries
|
||||
the same way as any other table — nothing about `search`, filtering, or SQL
|
||||
changes. This page covers what you get back; for enabling RTE and writing to
|
||||
a table, see [Real Time writes](/ingestion/realtime-write). For how Enterprise
|
||||
a table, see [RTE writes](/ingestion/realtime-write). For how Enterprise
|
||||
combines these results across machines, see
|
||||
[Distributed Real Time Read](/enterprise/distributed-real-time-read).
|
||||
[Distributed RTE Read](/enterprise/distributed-real-time-read).
|
||||
|
||||
## Query a real time table {#query}
|
||||
|
||||
@@ -25,7 +25,7 @@ moment ago is already reflected:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
# continuing the `frames` table connected and written to in Real Time writes
|
||||
# continuing the `frames` table connected and written to in RTE writes
|
||||
import numpy as np
|
||||
|
||||
query_vector = np.random.rand(frames.schema.field("embedding").type.list_size).astype("float32")
|
||||
@@ -57,7 +57,7 @@ maintained-index rules for vector, full-text, and scalar search, just with no
|
||||
network hop to a separate serving tier. There is no WAL node to route to and
|
||||
nothing to combine across machines: a query against an RTE-enabled Embedded
|
||||
table already reads the fresh tier together with the base table in one call.
|
||||
See [Distributed Real Time Read](/enterprise/distributed-real-time-read) for
|
||||
See [Distributed RTE Read](/enterprise/distributed-real-time-read) for
|
||||
what Enterprise adds on top when that fresh tier is served by a separate
|
||||
machine from the one answering the rest of the query.
|
||||
|
||||
@@ -77,8 +77,8 @@ available.
|
||||
|
||||
## Next steps {#next-steps}
|
||||
|
||||
- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table
|
||||
- [Distributed Real Time Read](/enterprise/distributed-real-time-read) — combining results across machines in Enterprise
|
||||
- [RTE writes](/ingestion/realtime-write) — enable RTE and write to a table
|
||||
- [Distributed RTE Read](/enterprise/distributed-real-time-read) — combining results across machines in Enterprise
|
||||
- [How the real time engine works](/tables/realtime-engine#memwal-format)
|
||||
- [Hybrid search](/search/hybrid-search)
|
||||
- [Vector search](/search/vector-search)
|
||||
|
||||
@@ -25,7 +25,7 @@ table, and nothing about how you query that table changes.
|
||||
RTE is built on **MemWAL**, a log-structured-merge (LSM) format that Lance
|
||||
itself defines -- the structure Enterprise serves is part of the open Lance
|
||||
format, not a layer bolted on beside it. A table with RTE enabled is divided
|
||||
into one or more **shards** (see [Real Time writes](/ingestion/realtime-write#primary-key)
|
||||
into one or more **shards** (see [RTE writes](/ingestion/realtime-write#primary-key)
|
||||
for enabling and sizing them), and each shard holds:
|
||||
|
||||
- A **memtable**: the active, in-memory table for that shard's newest rows,
|
||||
@@ -49,8 +49,8 @@ later folds SSTables into the base table, catching its indexes up as it goes.
|
||||
This section describes LanceDB Enterprise's serving layer. LanceDB Embedded
|
||||
runs the same MemWAL mechanics directly against the local dataset, in one
|
||||
process -- there is no separate node to route to or scale; see
|
||||
[Real Time writes](/ingestion/realtime-write#getting-started) and
|
||||
[Real Time reads](/search/realtime-read#embedded) for what that looks like.
|
||||
[RTE writes](/ingestion/realtime-write#getting-started) and
|
||||
[RTE reads](/search/realtime-read#embedded) for what that looks like.
|
||||
|
||||
RTE's Enterprise serving layer is a dedicated component, separate from the
|
||||
[query nodes and plan executors](/enterprise/distributed-search)
|
||||
@@ -81,13 +81,13 @@ pods are never both appending to the same shard's WAL at once.
|
||||
Every query runs against the base table and the fresh tier -- every shard's
|
||||
memtable and not-yet-compacted SSTables -- in parallel. Fresh rows supersede
|
||||
base rows with the same primary key, and the two are merged into one result.
|
||||
See [Real Time reads](/search/realtime-read) for what this means for query
|
||||
See [RTE reads](/search/realtime-read) for what this means for query
|
||||
results.
|
||||
|
||||
## Next steps {#next-steps}
|
||||
|
||||
- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table
|
||||
- [Real Time reads](/search/realtime-read) — query an RTE table
|
||||
- [Distributed Real Time Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
|
||||
- [Distributed Real Time Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
|
||||
- [RTE writes](/ingestion/realtime-write) — enable RTE and write to a table
|
||||
- [RTE reads](/search/realtime-read) — query an RTE table
|
||||
- [Distributed RTE Write](/enterprise/distributed-real-time-write) — how Enterprise shards and scales writes
|
||||
- [Distributed RTE Read](/enterprise/distributed-real-time-read) — how Enterprise combines reads across machines
|
||||
- [RTE Compaction](/enterprise/realtime-engine-compaction) — how Enterprise folds shards into the base table
|
||||
|
||||
Reference in New Issue
Block a user