Files
lancedb/docs/web/search/explain.mdx
T
Jack Ye 4015560b83 docs: restructure around the three API surfaces
The Documentation tab now reads as a manual over Python, TypeScript and SQL
rather than a Python guide with SQL bolted on.

Basics replaces the Catalog section: what the catalog holds, what each API
surface can do, jobs, agentic skills, and what Lance is. Management is new and
is the CRUD and listing reference for every catalog object, with a column per
language so the gaps are visible -- TypeScript has no function or SQL methods,
SQL needs a deployment.

Search becomes Query and gains a SELECT reference, since an analytical query is
a query whether it came from a builder or a statement.

Jobs is new. Every long-running operation already returned the same handle from
an _async method; nothing documented it.

The SQL function reference gains what Lance registers on top of DataFusion:
contains_tokens, backed by the FTS index, and the JSON accessors, backed by a
JSON scalar index. Geospatial st_* functions are behind a build feature LanceDB
does not ship, which is now stated rather than left to a confusing error.

Removes the Geneva pages' orphaned snippets, merges FTS Examples into Full-Text
Search, and splits Query Optimization into Explain and Explain Analyze.
2026-09-09 22:20:19 -07:00

198 lines
6.7 KiB
Plaintext

---
title: "Explain a query plan"
sidebarTitle: Explain
description: "Read the plan LanceDB resolved for a query, before running it."
icon: "map"
---
`explain_plan` prints the plan LanceDB resolved for a query without executing
it. It answers structural questions -- did the filter push into the scan, is the
vector index being used, which columns are actually read -- and costs nothing to
run, so it is the first thing to reach for when a query behaves unexpectedly.
To measure a query rather than read it, use
[`analyze_plan`](/search/explain-analyze), which executes it and annotates every
operator with runtime metrics.
## What it shows {#inspect-a-query-plan}
Reveals the logical query plan before execution, helping you identify potential issues with query structure and index usage. This tool is useful for:
- Verifying query optimization strategies
- Validating index selection
- Understanding query execution order
- Detecting missing indices
Metadata filters are prefiltered by default, which usually shows the filter pushed into the
`LanceScan` or index scan. If you set `prefilter=False`, expect a separate `FilterExec` after
search instead; that can be useful for some expensive filters, but it changes both latency and
the number of rows available after filtering.
## Reading the Execution Plan {#reading-the-execution-plan}
To demonstrate query performance analysis, we'll use a table containing 1.2M rows sampled from the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia). Initially, the table has no indices, allowing us to observe the impact of optimization.
Let's examine a vector search query that:
- Filters rows where `identifier` is between 0 and 1,000,000
- Returns the top 100 matches
- Projects specific columns: `chunk_index`, `title`, and `identifier`
<CodeGroup>
```python Python icon="python"
# explain_plan
query_explain_plan = (
table.search(query_embed)
.where("identifier > 0 AND identifier < 1000000")
.select(["chunk_index", "title", "identifier"])
.limit(100)
.explain_plan(True)
)
```
```typescript TypeScript icon="square-js"
// explain_plan
const explainPlan = await table
.search(queryEmbed)
.where("identifier > 0 AND identifier < 1000000")
.select(["chunk_index", "title", "identifier"])
.limit(100)
.explainPlan(true);
```
</CodeGroup>
### Execution Plan Components {#execution-plan-components}
The execution plan reveals the sequence of operations performed to execute your query. Let's examine each component:
```
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance]
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title"
CoalesceBatchesExec: target_batch_size=1024
GlobalLimitExec: skip=0, fetch=100
FilterExec: _distance@3 IS NOT NULL
SortExec: TopK(fetch=100), expr=[_distance@3 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000
LanceScan: uri=***, projection=[vector, identifier], row_id=true, row_addr=false, ordered=false
```
#### 1\. Base Layer (LanceScan) {#1-base-layer-lancescan}
- Initial data scan loading only specified columns to minimize I/O
- Unordered scan enabling parallel processing
```
LanceScan:
- projection=[vector, identifier]
- row_id=true, row_addr=false, ordered=false
```
#### 2\. First Filter {#2-first-filter}
- Apply requested filter on `identifier` column
- Reduces the number of vectors that need KNN computation
```
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000
```
#### 3\. Vector Search {#3-vector-search}
- Computes L2 (Euclidean) distances between query vector and all vectors that passed the filter
```
KNNVectorDistance: metric=l2
```
#### 4\. Results Processing {#4-results-processing}
- Filters out null distance results
- Sorts by distance and takes top 100 results
- Processes in batches of 1024 for optimal memory usage
```
SortExec: TopK(fetch=100)
- expr=[_distance@3 ASC NULLS LAST]
- preserve_partitioning=[false]
FilterExec: _distance@3 IS NOT NULL
GlobalLimitExec: skip=0, fetch=100
CoalesceBatchesExec: target_batch_size=1024
```
#### 5\. Data Retrieval {#5-data-retrieval}
- `RemoteTake` is a key component of Lance's I/O cache
- Handles efficient data retrieval from remote storage locations
- Fetches specific rows and columns needed for the final output
- Optimizes network bandwidth by only retrieving required data
```
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title"
```
#### 6\. Final Output {#6-final-output}
- Returns only requested columns and maintains column ordering
```python
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance]
```
This plan demonstrates a basic search without index optimizations: it performs a full scan and filter before vector search.
## Non-vector queries {#analyzing-non-vector-queries}
`explain_plan` is not vector-specific -- it is available on every query builder,
including FTS and hybrid. The most common reason to read the plan for a
non-vector query is to confirm whether your `where` clause pushed into the scan
(good) or ran as a separate `FilterExec` on top of the search results (often
slower, and a hint that the filter column needs a scalar index).
### FTS queries {#fts-queries}
<CodeGroup>
```python Python icon="python"
plan = (
table.search("puppy", query_type="fts")
.where("category = 'animals'", prefilter=True)
.limit(10)
.explain_plan(True)
)
print(plan)
```
```typescript TypeScript icon="square-js"
const plan = await table
.query()
.fullTextSearch("puppy")
.where("category = 'animals'")
.limit(10)
.explainPlan(true);
```
</CodeGroup>
In an indexed FTS plan you should see a `MatchQuery` (or other FTS execution node) reading from the inverted index, with the metadata filter pushed down. If the plan shows a `LanceScan` followed by `FilterExec` over the entire text column, the FTS index either isn't covering the column or the filter isn't using a scalar index — both worth investigating.
### Hybrid queries {#hybrid-queries}
For hybrid queries, `explain_plan` returns the reranker label followed by the vector and FTS sub-plans, indented for readability:
<CodeGroup>
```python Python icon="python"
plan = (
table.search("flower moon", query_type="hybrid")
.where("category = 'film'", prefilter=True)
.limit(10)
.explain_plan(True)
)
print(plan)
# RRFReranker(...)
# <vector sub-plan>
# <FTS sub-plan>
```
</CodeGroup>
Reading the plan tells you the shape. To find out which half dominates latency,
[analyze it](/search/explain-analyze#hybrid-queries).