Files
lancedb/docs/web/curation/blob-late-materialization.mdx
T
Jack YeandClaude Sonnet 5 5e38120bdc docs: add storage integration SDK examples, abstract RTE write, simplify cache pages (#4865)
## Summary

- **Define Storage Integration**: un-gates the Python (and Rust, for S3)
`create_storage_integration` examples, drafted against #4618 and held
back until it shipped -- it's now in v0.42.0-beta.0+, and the signature
matches the drafts exactly.
- **Naming rules**: drops the `Branch` row and its "Learn more" link,
consistent with de-emphasizing branches elsewhere in this nav.
- **Distributed RTE Write**: removes the REST path, the
`__lance_mem_wal` index name, and "WAL node" terminology in favor of an
abstract framing -- a query node routes writes to RTE nodes, each of
which can host shards from many different tables. Both diagram variants
are updated to match.
- **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop
Helm-specific and exact-percentage detail (disk-budget math,
`startup_prewarm_block_query_ready`/`healthPort`/failure-policy table)
in favor of a short feature → architecture → why-it-matters shape, each
closing with a contact-sales pointer. Data Page Cache's on-demand
prewarm section is removed outright.

## Note

I wasn't able to get a clean visual render of the updated Distributed
RTE Write diagram in local preview -- Mermaid isn't rendering in
headless Chrome in this environment for *any* page right now, confirmed
against an unmodified reference page showing the identical symptom. The
diagram source follows the exact pattern already shipped and working in
the 21 diagrams from the earlier brand-theming pass, and `mint
validate`/`mint broken-links` both pass clean.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-05 01:40:54 -08:00

177 lines
7.8 KiB
Plaintext

---
title: "Blob Late Materialization"
sidebarTitle: "Blob Late Materialization"
description: "Deferring a wide BlobV2 or vector column's read until after a join has already narrowed down which rows survive."
icon: clock
keywords: ["late materialization", "blob", "blobv2", "join", "remotetake", "take"]
availability:
embedded: unavailable
enterprise: available
---
A join that includes a wide column -- a [BlobV2](/management/table#managed-blobs)
or an embedding -- can multiply that column's read far past the result's
own row count: a naive plan reads the wide column for every candidate row
the join considers, then throws most of that work away once the join
condition filters non-matches. Blob late materialization is the
optimizer rewrite that avoids this, deferring the wide read until after
the join has already settled which rows survive.
## The two plan shapes {#plan-shapes}
<div className="hidden dark:block">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#171513", "primaryColor": "#1e1b19", "primaryTextColor": "#f4ebe8",
"primaryBorderColor": "#4c443f", "lineColor": "#847770", "textColor": "#f4ebe8",
"clusterBkg": "#211d1a", "clusterBorder": "#4c443f", "edgeLabelBackground": "#171513",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart TB
subgraph Naive["Naive: wide column read before filtering"]
direction TB
N1["Scan wide column<br/>for every row"] --> N2["Join"]
N2 --> N3["Rows that didn't match<br/>are discarded --<br/>their wide read was wasted"]
end
subgraph Deferred["Late materialization: wide column read after filtering"]
direction TB
D1["Scan only the join key<br/>(+ early columns)"] --> D2["Join on the key"]
D2 --> D3["RemoteTake:<br/>fetch the wide column,<br/>once per surviving row"]
end
classDef default fill:#1e1b19,stroke:#4c443f,stroke-width:1px,color:#f4ebe8
classDef accent fill:#2a1810,stroke:#e8582d,stroke-width:1.5px,color:#f4ebe8
class D3 accent
```
</div>
<div className="block dark:hidden">
```mermaid
%%{init: {"theme": "base", "themeVariables": {
"background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
"primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
"clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
"fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart TB
subgraph Naive["Naive: wide column read before filtering"]
direction TB
N1["Scan wide column<br/>for every row"] --> N2["Join"]
N2 --> N3["Rows that didn't match<br/>are discarded --<br/>their wide read was wasted"]
end
subgraph Deferred["Late materialization: wide column read after filtering"]
direction TB
D1["Scan only the join key<br/>(+ early columns)"] --> D2["Join on the key"]
D2 --> D3["RemoteTake:<br/>fetch the wide column,<br/>once per surviving row"]
end
classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
class D3 accent
```
</div>
In the naive plan, the wide column is read once per **candidate** row --
every row either side of the join could possibly produce -- before the
join has thrown any of them away. In the rewritten plan, the join
operates on a narrowed schema (the key, any small "early" columns, and a
row id), and a `RemoteTake` node fetches the wide column afterward, once
per row that actually **survived** the join.
## Eligibility {#eligibility}
This is a physical-plan optimizer rewrite, not something you opt into
with special syntax -- the optimizer decides whether a given join
qualifies. It applies to a plain inner equi-join (a single `ON left_col
= right_col` condition, no additional filter fused into the join, no
`LIMIT` fused into it) where one side -- and only one side -- bottoms out
at a single, otherwise-unfiltered table scan: no existing row filter,
scan range, fragment subset, deleted-row handling, or
[RTE write](/ingestion/realtime-write) tier. If both sides would
otherwise qualify, the side with a BlobV2 column is preferred over the
other; if neither or both sides have one, the rewrite declines rather
than guessing which to narrow.
Which columns actually get deferred differs by which side of the join is
narrowed:
- **The side that builds the join's hash table:** every column beyond
the join key is deferred, regardless of size. The hash table holds
everything this side reads for the lifetime of the join, so even a
modestly-sized column adds up across every row the join holds in
memory at once.
- **The side that probes the hash table:** only columns wide enough to
cross a size heuristic are deferred; small columns stay in the initial
scan, since reading them eagerly doesn't carry the same memory risk.
A BlobV2 column's **descriptor** (small, pointer-like) is early and
stays in the scan either way -- only the raw **bytes** of the blob are
late and deferred, so narrowing doesn't lose track of which rows have a
blob at all, just postpones reading its payload.
## Where the deferred read actually happens {#remotetake}
The take itself runs through the same replica-group-aware routing as any
other distributed table read (see [Horizontal Scaling](/enterprise/horizontal-scaling)) --
it is not shipped back to the query node to be joined there directly.
Routing is resolved once per query and reused across every batch the
take processes, rather than re-resolved per batch, which is what keeps
repeated takes in the same query hitting a cache that's already warm
instead of bouncing between replica groups. Where the executing node
already holds the needed data, the take can also resolve natively
without a separate network round trip at all.
## Confirming it happened {#explain}
A join written this way is a **candidate** for the rewrite, not a
guarantee -- confirm it per query with
[`EXPLAIN`](/basics/sql#query-plans). A `Take` or `RemoteTake` node by
itself isn't proof either way -- it's the general operator that fetches
columns for selected rows from remote storage, and shows up in an
ordinary, join-free query just as it does here (see the
[recorded vector-search plan](/basics/sql#5-data-retrieval)); the
rewrite itself inserts a plain `Take`, which only a later, separate pass
turns into `RemoteTake`. What confirms the deferral specifically is a
`Take`/`RemoteTake` node sitting above a `HashJoinExec`, fetching a
column set that includes `__join_late_materialization_key` -- that key
column only exists when this rewrite fired. Join on a narrow, selective
key and project the wide columns only in the final select list, rather
than alongside the join condition:
<CodeGroup>
```sql SQL icon="database"
SELECT s.frame_id, f.image, f.embedding
FROM selected AS s
JOIN frames AS f ON f.frame_id = s.frame_id
```
</CodeGroup>
`selected` is narrow -- just the frame IDs an earlier curation step
already kept. `frames` is wide -- a BlobV2 `image` and a vector
`embedding` alongside its key. Neither a passing query nor an `EXPLAIN`
that omits `RemoteTake` tells you how much I/O a plan did or did not
save on its own -- use [`EXPLAIN ANALYZE`](/sql/dql/explain) for the
plan's actual row and byte counts, since plain `EXPLAIN` shows the
planned shape without executing the query.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Joining datasets" icon="link" href="/curation/join">
Join types, duplicate keys, and calling a Function across a join.
</Card>
<Card title="Joins" icon="link" href="/sql/dql/join">
Join syntax and semantics in the SQL reference.
</Card>
<Card title="Query plans" icon="map" href="/basics/sql#query-plans">
Reading `EXPLAIN` output in general.
</Card>
<Card title="Horizontal Scaling" icon="arrows-left-right" href="/enterprise/horizontal-scaling">
The replica-group routing a RemoteTake's read goes through.
</Card>
</CardGroup>