mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
The function catalogue is now real rather than a pointer. Scalar functions are listed by taxonomy -- math, string, unicode, regex, date/time, array, map, struct, crypto, type -- and the names come from what datafusion-functions 54 actually registers, not from what DataFusion documents on main. Aggregate and window pages gain the same treatment. Two things the audit corrected rather than copied. SHOW TABLES does not work. DataFusion implements it over information_schema, which this session does not enable, so it fails at plan time. It was documented as working here, on the SHOW page and in three Manage tables; all four are fixed, and the SQL overview now lists the absence alongside SHOW COLUMNS. Prepared statements are a Flight SQL protocol feature, not a PREPARE statement. A script carrying PREPARE ... AS SELECT will not run, while the same intent expressed through a client will. The new page says so rather than copying DataFusion's syntax. Every section of the Documentation tab that has a SQL counterpart now links to it -- 27 of 76 non-SQL pages, each pointing at the specific statement rather than at the reference index.
202 lines
6.8 KiB
Plaintext
202 lines
6.8 KiB
Plaintext
---
|
|
title: "Explain a query plan"
|
|
sidebarTitle: Explain
|
|
description: "Read the plan LanceDB resolved for a query, before running it."
|
|
icon: "map"
|
|
---
|
|
|
|
`explain_plan` prints the plan LanceDB resolved for a query without executing
|
|
it. It answers structural questions -- did the filter push into the scan, is the
|
|
vector index being used, which columns are actually read -- and costs nothing to
|
|
run, so it is the first thing to reach for when a query behaves unexpectedly.
|
|
|
|
To measure a query rather than read it, use
|
|
[`analyze_plan`](/search/explain-analyze), which executes it and annotates every
|
|
operator with runtime metrics.
|
|
|
|
<Note>
|
|
The SQL equivalent is [`EXPLAIN`](/sql/explain).
|
|
</Note>
|
|
|
|
## What it shows {#inspect-a-query-plan}
|
|
|
|
Reveals the logical query plan before execution, helping you identify potential issues with query structure and index usage. This tool is useful for:
|
|
|
|
- Verifying query optimization strategies
|
|
- Validating index selection
|
|
- Understanding query execution order
|
|
- Detecting missing indices
|
|
|
|
Metadata filters are prefiltered by default, which usually shows the filter pushed into the
|
|
`LanceScan` or index scan. If you set `prefilter=False`, expect a separate `FilterExec` after
|
|
search instead; that can be useful for some expensive filters, but it changes both latency and
|
|
the number of rows available after filtering.
|
|
|
|
## Reading the Execution Plan {#reading-the-execution-plan}
|
|
|
|
To demonstrate query performance analysis, we'll use a table containing 1.2M rows sampled from the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia). Initially, the table has no indices, allowing us to observe the impact of optimization.
|
|
|
|
Let's examine a vector search query that:
|
|
|
|
- Filters rows where `identifier` is between 0 and 1,000,000
|
|
- Returns the top 100 matches
|
|
- Projects specific columns: `chunk_index`, `title`, and `identifier`
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# explain_plan
|
|
query_explain_plan = (
|
|
table.search(query_embed)
|
|
.where("identifier > 0 AND identifier < 1000000")
|
|
.select(["chunk_index", "title", "identifier"])
|
|
.limit(100)
|
|
.explain_plan(True)
|
|
)
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
// explain_plan
|
|
const explainPlan = await table
|
|
.search(queryEmbed)
|
|
.where("identifier > 0 AND identifier < 1000000")
|
|
.select(["chunk_index", "title", "identifier"])
|
|
.limit(100)
|
|
.explainPlan(true);
|
|
```
|
|
</CodeGroup>
|
|
|
|
### Execution Plan Components {#execution-plan-components}
|
|
|
|
The execution plan reveals the sequence of operations performed to execute your query. Let's examine each component:
|
|
|
|
```
|
|
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance]
|
|
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title"
|
|
CoalesceBatchesExec: target_batch_size=1024
|
|
GlobalLimitExec: skip=0, fetch=100
|
|
FilterExec: _distance@3 IS NOT NULL
|
|
SortExec: TopK(fetch=100), expr=[_distance@3 ASC NULLS LAST], preserve_partitioning=[false]
|
|
KNNVectorDistance: metric=l2
|
|
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000
|
|
LanceScan: uri=***, projection=[vector, identifier], row_id=true, row_addr=false, ordered=false
|
|
```
|
|
|
|
#### 1\. Base Layer (LanceScan) {#1-base-layer-lancescan}
|
|
|
|
- Initial data scan loading only specified columns to minimize I/O
|
|
- Unordered scan enabling parallel processing
|
|
|
|
```
|
|
LanceScan:
|
|
- projection=[vector, identifier]
|
|
- row_id=true, row_addr=false, ordered=false
|
|
```
|
|
|
|
#### 2\. First Filter {#2-first-filter}
|
|
|
|
- Apply requested filter on `identifier` column
|
|
- Reduces the number of vectors that need KNN computation
|
|
|
|
```
|
|
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000
|
|
```
|
|
|
|
#### 3\. Vector Search {#3-vector-search}
|
|
|
|
- Computes L2 (Euclidean) distances between query vector and all vectors that passed the filter
|
|
|
|
```
|
|
KNNVectorDistance: metric=l2
|
|
```
|
|
|
|
#### 4\. Results Processing {#4-results-processing}
|
|
|
|
- Filters out null distance results
|
|
- Sorts by distance and takes top 100 results
|
|
- Processes in batches of 1024 for optimal memory usage
|
|
|
|
```
|
|
SortExec: TopK(fetch=100)
|
|
- expr=[_distance@3 ASC NULLS LAST]
|
|
- preserve_partitioning=[false]
|
|
FilterExec: _distance@3 IS NOT NULL
|
|
GlobalLimitExec: skip=0, fetch=100
|
|
CoalesceBatchesExec: target_batch_size=1024
|
|
```
|
|
|
|
#### 5\. Data Retrieval {#5-data-retrieval}
|
|
|
|
- `RemoteTake` is a key component of Lance's I/O cache
|
|
- Handles efficient data retrieval from remote storage locations
|
|
- Fetches specific rows and columns needed for the final output
|
|
- Optimizes network bandwidth by only retrieving required data
|
|
|
|
```
|
|
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title"
|
|
```
|
|
|
|
#### 6\. Final Output {#6-final-output}
|
|
|
|
- Returns only requested columns and maintains column ordering
|
|
|
|
```python
|
|
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance]
|
|
```
|
|
|
|
This plan demonstrates a basic search without index optimizations: it performs a full scan and filter before vector search.
|
|
|
|
## Non-vector queries {#analyzing-non-vector-queries}
|
|
|
|
`explain_plan` is not vector-specific -- it is available on every query builder,
|
|
including FTS and hybrid. The most common reason to read the plan for a
|
|
non-vector query is to confirm whether your `where` clause pushed into the scan
|
|
(good) or ran as a separate `FilterExec` on top of the search results (often
|
|
slower, and a hint that the filter column needs a scalar index).
|
|
|
|
### FTS queries {#fts-queries}
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
plan = (
|
|
table.search("puppy", query_type="fts")
|
|
.where("category = 'animals'", prefilter=True)
|
|
.limit(10)
|
|
.explain_plan(True)
|
|
)
|
|
print(plan)
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
const plan = await table
|
|
.query()
|
|
.fullTextSearch("puppy")
|
|
.where("category = 'animals'")
|
|
.limit(10)
|
|
.explainPlan(true);
|
|
```
|
|
</CodeGroup>
|
|
|
|
In an indexed FTS plan you should see a `MatchQuery` (or other FTS execution node) reading from the inverted index, with the metadata filter pushed down. If the plan shows a `LanceScan` followed by `FilterExec` over the entire text column, the FTS index either isn't covering the column or the filter isn't using a scalar index — both worth investigating.
|
|
|
|
### Hybrid queries {#hybrid-queries}
|
|
|
|
For hybrid queries, `explain_plan` returns the reranker label followed by the vector and FTS sub-plans, indented for readability:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
plan = (
|
|
table.search("flower moon", query_type="hybrid")
|
|
.where("category = 'film'", prefilter=True)
|
|
.limit(10)
|
|
.explain_plan(True)
|
|
)
|
|
print(plan)
|
|
# RRFReranker(...)
|
|
# <vector sub-plan>
|
|
# <FTS sub-plan>
|
|
```
|
|
</CodeGroup>
|
|
|
|
Reading the plan tells you the shape. To find out which half dominates latency,
|
|
[analyze it](/search/explain-analyze#hybrid-queries).
|