mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-14 22:56:54 +00:00
Jack YeandClaude Sonnet 5
de95ed3f6d
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
65 lines
2.5 KiB
Plaintext
65 lines
2.5 KiB
Plaintext
---
|
|
title: "Data Page Cache"
|
|
sidebarTitle: "Data Page Cache"
|
|
description: "How LanceDB Enterprise caches table data pages to avoid re-reading object storage."
|
|
icon: "database"
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
The data page cache holds fixed-size pages of raw table data read from
|
|
object storage, keyed by their location, so a later read of the same range
|
|
is served from memory or local disk instead of round-tripping to the
|
|
backing store. Plan executors and query nodes each keep their own cache for
|
|
the pages they serve. A background prewarm job can populate a cache ahead
|
|
of a request that needs it, rather than filling it lazily on first read.
|
|
See [Architecture](/index#architecture).
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
R["Read request<br/>for a data page"] --> M{"In memory?"}
|
|
M -->|hit| Return["Served from memory"]
|
|
M -->|miss| D{"In local disk cache?"}
|
|
D -->|hit| Promote["Loaded into memory,<br/>served"]
|
|
D -->|miss| O[("Object storage")]
|
|
O --> Fill["Fills memory and disk,<br/>served"]
|
|
```
|
|
|
|
A table's local disk cache shares one budget with the
|
|
[tiered index cache](/enterprise/tiered-index-cache#disk-budget)'s disk
|
|
tier where both are enabled on the same executor -- raising one's share of
|
|
that budget lowers the other's.
|
|
|
|
## Prewarming on demand {#prewarm}
|
|
|
|
[Cache Prewarm](/enterprise/cache-prewarm)'s `startup_prewarm_idx` only
|
|
warms the [tiered index cache](/enterprise/tiered-index-cache), not this
|
|
one. The data page cache has its own, separate on-demand prewarm, run by a
|
|
deployment operator rather than called from application code, optionally
|
|
scoped to specific columns:
|
|
|
|
<CodeGroup>
|
|
```bash CLI icon="terminal"
|
|
lancedb admin cache prewarm data --table embedding_table --column embedding
|
|
```
|
|
</CodeGroup>
|
|
|
|
The command returns once the request is accepted, not once prewarming
|
|
finishes; it is idempotent and safe to run concurrently from multiple
|
|
clients. Prewarming has a large upfront cost, so it pays off for workloads
|
|
that repeatedly query the same columns -- large columns such as embeddings
|
|
or binary data may not be practical to prewarm this way.
|
|
|
|
<Warning>
|
|
The OSS SDK's `table.prewarm_data()` targets LanceDB Cloud, a different
|
|
product; calling it against an Enterprise deployment 404s. Use the admin
|
|
command above instead.
|
|
</Warning>
|
|
|
|
## Cache interfaces {#interfaces}
|
|
|
|
- [Tiered Index Cache](/enterprise/tiered-index-cache) for the related
|
|
cache that holds deserialized index structures instead of raw data pages.
|
|
- [Query plans](/basics/sql#query-plans) for inspecting what a query actually read.
|