mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 14:52:26 +00:00
docs: declare where the search pages apply
Search, full-text search, hybrid search, filtering, query plans and reranking all work in the embedded library and against a deployment, and now say so. Where the two differ, the section says how: a deployment returns a query's whole result in one response, the synchronous Python remote table refuses to_pandas, to_arrow and to_lance, probe widening depends on the deployment's version, full-text indexes build in the background and are kept current without optimize(), distributed scoring can differ slightly, model files have to be present on the deployment, plans carry deployment operators, and hybrid explain_plan is Python only. Reranking runs in the application either way.
This commit is contained in:
Brendan Clement
@@ -4,6 +4,13 @@ sidebarTitle: "Reranking"
|
||||
description: "Use a reranker to improve search relevance."
|
||||
icon: "sort-amount-down"
|
||||
keywords: ["re-ranking", "reranker", "rerank"]
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Rerankers run in your application after the search returns, so they work
|
||||
the same with the embedded library and with a deployment. Most built-in
|
||||
rerankers are Python-only.
|
||||
---
|
||||
|
||||
import {
|
||||
|
||||
@@ -3,12 +3,19 @@ title: "Explain a query plan"
|
||||
sidebarTitle: "Query Plans"
|
||||
description: "Read the plan LanceDB resolved for a query, and measure it with analyze."
|
||||
icon: "map"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Query plans can be explained and analyzed in the embedded library and against
|
||||
a deployment. A deployment's plans include its own operators, and per-worker
|
||||
metrics apply only to a deployment that distributes its queries.
|
||||
---
|
||||
|
||||
`explain_plan` prints the plan LanceDB resolved for a query without executing
|
||||
it. It answers structural questions -- did the filter push into the scan, is the
|
||||
vector index being used, which columns are actually read -- and costs nothing to
|
||||
run, so it is the first thing to reach for when a query behaves unexpectedly.
|
||||
vector index being used, which columns are actually read -- and does not execute
|
||||
the query, so it is the first thing to reach for when a query behaves unexpectedly.
|
||||
|
||||
To measure a query rather than read it, use
|
||||
[`analyze_plan`](/search/explain), which executes it and annotates every
|
||||
@@ -34,7 +41,7 @@ the number of rows available after filtering.
|
||||
|
||||
## Reading the Execution Plan {#reading-the-execution-plan}
|
||||
|
||||
To demonstrate query performance analysis, we'll use a table containing 1.2M rows sampled from the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia). Initially, the table has no indices, allowing us to observe the impact of optimization.
|
||||
To demonstrate query performance analysis, we'll use a table containing 1.2M rows sampled from the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia). Initially, the table has no indices, allowing us to observe the impact of optimization. The table is on a deployment, so its plans include deployment operators such as `RemoteTake` and hide storage paths as `uri=***`; a local plan shows Lance's `Take` and the dataset path instead.
|
||||
|
||||
Let's examine a vector search query that:
|
||||
|
||||
@@ -126,7 +133,7 @@ CoalesceBatchesExec: target_batch_size=1024
|
||||
|
||||
#### 5\. Data Retrieval {#5-data-retrieval}
|
||||
|
||||
- `RemoteTake` is a key component of Lance's I/O cache
|
||||
- `RemoteTake` is the deployment's operator for this step; a local plan shows Lance's `Take`
|
||||
- Handles efficient data retrieval from remote storage locations
|
||||
- Fetches specific rows and columns needed for the final output
|
||||
- Optimizes network bandwidth by only retrieving required data
|
||||
@@ -148,7 +155,7 @@ This plan demonstrates a basic search without index optimizations: it performs a
|
||||
## Non-vector queries {#analyzing-non-vector-queries}
|
||||
|
||||
`explain_plan` is not vector-specific -- it is available on every query builder,
|
||||
including FTS and hybrid. The most common reason to read the plan for a
|
||||
including FTS, and on hybrid queries in Python. The most common reason to read the plan for a
|
||||
non-vector query is to confirm whether your `where` clause pushed into the scan
|
||||
(good) or ran as a separate `FilterExec` on top of the search results (often
|
||||
slower, and a hint that the filter column needs a scalar index).
|
||||
@@ -288,7 +295,7 @@ ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
When you call `analyze_plan` against a LanceDB Enterprise table, the query runs across a pool of workers. By default the returned plan aggregates each operator's metrics into a single value, matching the local single-node output. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
|
||||
When you call `analyze_plan` against a LanceDB Enterprise deployment that distributes its queries across a pool of workers, the query runs on those workers. By default the returned plan aggregates each operator's metrics into a single value, matching the local single-node output. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
|
||||
|
||||
| Mode | What it shows | When to use it |
|
||||
| :------------ | :---------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
|
||||
|
||||
@@ -3,6 +3,12 @@ title: "Metadata Filtering in LanceDB"
|
||||
sidebarTitle: Filtering
|
||||
description: Filter search results in LanceDB based on metadata fields.
|
||||
icon: "filter"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Metadata filters work in the embedded library and against a deployment,
|
||||
which evaluates them as part of the search.
|
||||
---
|
||||
|
||||
LanceDB supports filtering features of query results based on metadata fields.
|
||||
@@ -15,7 +21,7 @@ with filtering capabilities even on datasets containing billions of records.
|
||||
On hybrid queries, the same `where(...)` filter is applied to both the vector and full-text halves of the query. The prefilter or postfilter choice controls whether that happens before each subquery scores candidates or after the subquery top-k is produced.
|
||||
|
||||
<Note>
|
||||
The same predicates work in SQL -- see [SELECT](/sql/dql/select#filtering) and [Operators](/sql/operators).
|
||||
The same predicates work in SQL, which runs on a deployment -- see [SELECT](/sql/dql/select#filtering) and [Operators](/sql/operators).
|
||||
</Note>
|
||||
|
||||
## Chaining `where` clauses {#chaining-where-clauses}
|
||||
|
||||
@@ -3,6 +3,13 @@ title: Full-Text Search (FTS)
|
||||
sidebarTitle: Full-Text Search (FTS)
|
||||
description: Learn how to implement full-text search in LanceDB using BM25 for keyword-based retrieval.
|
||||
icon: "book"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Full-text search runs in the embedded library and against a deployment. On a
|
||||
deployment, indexes are built in the background and kept up to date by the
|
||||
deployment rather than by optimize().
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -15,7 +22,7 @@ import { PyFtsBooleanQueries } from '/snippets/doc_function_examples.mdx'
|
||||
LanceDB provides support for Full-Text Search via Lance, allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions.
|
||||
|
||||
<Note>
|
||||
In SQL, full-text search is the [`fts()`](/sql/functions#fts) table function, or the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate when you do not need a score.
|
||||
In SQL, full-text search is the [`fts()`](/sql/functions#fts) table function, or the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate when you do not need a score. [SQL](/basics/sql) runs on a deployment.
|
||||
</Note>
|
||||
|
||||
## Basic Usage {#basic-usage}
|
||||
@@ -95,6 +102,10 @@ tbl
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Locally the index is built before the call returns. On a deployment the call
|
||||
starts a background build and returns at once; pass `wait_timeout` or call
|
||||
`wait_for_index` before relying on the index.
|
||||
|
||||
### Full-text Search {#full-text-search}
|
||||
|
||||
Perform full-text search and retrieve results:
|
||||
@@ -137,6 +148,10 @@ LanceDB automatically searches on the existing FTS index if the input to the sea
|
||||
|
||||
If a table has more than one FTS index, specify the indexed text column in the query. In Python you can use `fts_columns` or the query builder's `nearest_to_text(..., columns=...)`; in TypeScript, use `query().nearestToText(..., columns)`. The newer Lance-native FTS does not accept legacy Tantivy-only index parameters.
|
||||
|
||||
A deployment that distributes search across its workers scores each worker's
|
||||
rows with that worker's statistics unless it is configured for global scoring,
|
||||
so scores and close rankings can differ slightly from the embedded library's.
|
||||
|
||||
### Keeping the index up to date {#keeping-the-index-up-to-date}
|
||||
|
||||
Rows you add after building an FTS index aren't part of the index until you optimize the table. Until then, queries fall back to a flat scan over the unindexed fragments to keep results complete, which slows them down as the unindexed tail grows. Call `table.optimize()` to fold new rows into the existing index — it's the same operation used for vector indexes:
|
||||
@@ -159,6 +174,8 @@ tbl.optimize(OptimizeAction::All).await?;
|
||||
|
||||
A useful rule of thumb is to call `optimize()` after roughly 100,000 row changes or 20 data-modification operations, whichever comes first. For tables with continuous ingest, schedule it on a cadence that keeps `num_unindexed_rows` (from `table.index_stats(...)`) close to zero. If you want to skip the flat scan over unindexed rows entirely — for example, on a hot read path where stale results are acceptable — call `.fast_search()` on the query so the search returns only indexed results.
|
||||
|
||||
On a deployment, `optimize()` is not available: the deployment adds new rows to the index itself, and until it has, queries scan the unindexed rows.
|
||||
|
||||
## Advanced Usage {#advanced-usage}
|
||||
|
||||
### Tokenize Table Data {#tokenize-table-data}
|
||||
@@ -215,6 +232,10 @@ export LANCE_LANGUAGE_MODEL_HOME=/path/to/lance/language_models
|
||||
|
||||
For example, `jieba/default` is resolved under `<model-home>/jieba/default/...`, `lindera/ipadic` under `<model-home>/lindera/ipadic/...`, and `lindera/ko-dic` under `<model-home>/lindera/ko-dic/...`.
|
||||
|
||||
The model files are read where the index is built and queried. For a table on a
|
||||
deployment that is the deployment, so ask its administrator whether they are
|
||||
installed there.
|
||||
|
||||
<Note>
|
||||
Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. If you use another stemming language, such as Arabic, Greek, Romanian, Tamil, or Turkish, set `remove_stop_words=False` or pass `custom_stop_words`.
|
||||
</Note>
|
||||
@@ -720,7 +741,7 @@ const multiMatchBoostingResults = await table.query()
|
||||
- Create full-text search indices on text columns that will be frequently searched
|
||||
- For hybrid search combining text and vectors, see our [hybrid search guide](/search/hybrid-search/)
|
||||
- For performance benchmarks, check our [benchmark results](/enterprise/benchmarks/)
|
||||
- For complex queries, use SQL to combine FTS with other filter conditions
|
||||
- For complex queries on a deployment, use SQL to combine FTS with other filter conditions
|
||||
</Card>
|
||||
|
||||
### Boolean Queries {#boolean-queries}
|
||||
|
||||
@@ -3,6 +3,13 @@ title: Hybrid Search
|
||||
sidebarTitle: "Hybrid Search"
|
||||
description: Learn how to perform hybrid search in LanceDB by combining vector and full-text search techniques with reranking.
|
||||
icon: "search"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Hybrid search runs in the embedded library and against a deployment. With a
|
||||
deployment, the vector and full-text searches run there, and the SDK combines
|
||||
and reranks their results in your application.
|
||||
---
|
||||
|
||||
In certain cases, you may want to retrieve documents that are semantically similar to a given query,
|
||||
@@ -341,11 +348,11 @@ await table.query()
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The choice gets baked into both sub-queries, so the vector and FTS halves see the filter applied the same way. Use [`explain_plan`](/search/explain#analyzing-non-vector-queries) on a hybrid query to see whether the filter pushed into the scan or ran as a separate `FilterExec` step.
|
||||
The choice gets baked into both sub-queries, so the vector and FTS halves see the filter applied the same way. In Python, use [`explain_plan`](/search/explain#analyzing-non-vector-queries) on a hybrid query to see whether the filter pushed into the scan or ran as a separate `FilterExec` step.
|
||||
|
||||
## More on Reranking {#more-on-reranking}
|
||||
|
||||
You can perform hybrid search in LanceDB by combining the results of semantic and full-text search via a reranking algorithm of your choice. LanceDB comes with [**built-in rerankers**](https://docs.lancedb.com/reranking) and you can implement your own **custom reranker** as well.
|
||||
You can perform hybrid search in LanceDB by combining the results of semantic and full-text search via a reranking algorithm of your choice. LanceDB comes with [**built-in rerankers**](https://docs.lancedb.com/reranking) and you can implement your own **custom reranker** as well. Reranking and score normalization run in your application, whether the table is local or on a deployment.
|
||||
|
||||
By default, LanceDB uses `RRFReranker()`, which uses reciprocal rank fusion score, to combine and rerank the results of semantic and full-text search. You can customize the hyperparameters as needed or write your own custom reranker. Here's how you can use any of the available rerankers:
|
||||
|
||||
|
||||
@@ -3,6 +3,13 @@ title: Search
|
||||
sidebarTitle: Overview
|
||||
description: "Comprehensive guide to all search capabilities in LanceDB including vector search, full-text search, hybrid search, and more."
|
||||
icon: "list"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Vector, full-text and hybrid search run in the embedded library and against
|
||||
a deployment. Reading very large results and probe tuning differ between the
|
||||
two, as the sections below note.
|
||||
---
|
||||
|
||||
| Feature | Description |
|
||||
@@ -45,7 +52,7 @@ Three knobs materially affect query latency, memory use, and recall. Be delibera
|
||||
|
||||
- Quantized indexes (PQ, RQ, SQ): `refine_factor` pulls extra candidates and re-scores them on full vectors.
|
||||
- HNSW-backed indexes: `ef` at search time. Start at `1.5 × k`, raise toward `10 × k` if recall is short.
|
||||
- IVF candidate breadth: `nprobes` is auto-tuned; override only when a selective pre-filter leaves too few neighbors.
|
||||
- IVF candidate breadth: `nprobes` is auto-tuned; override only when a selective pre-filter leaves too few neighbors. Whether a deployment widens the search this way depends on its version; one that does not searches exactly `minimum_nprobes` partitions.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
@@ -53,7 +60,7 @@ For hybrid search, the default `RRFReranker()` combines vector and FTS results i
|
||||
|
||||
#### Avoid materializing the whole table {#avoid-materializing-the-whole-table}
|
||||
|
||||
If you need every row in the table (for training, export, or migration), calling `to_pandas()` or `to_arrow()` will run you out of memory on any non-trivial dataset — both materialize the full table at once. The best practice is to iterate via `table.search(...)` or `table.query(...)`, which work the same way in both LanceDB OSS and Enterprise.
|
||||
If you need every row in the table (for training, export, or migration), calling `to_pandas()` or `to_arrow()` will run you out of memory on any non-trivial dataset — both materialize the full table at once. Read it through queries instead: `table.search(...)`, or `table.query(...)` in async Python, TypeScript and Rust. Against a deployment, each query returns its whole result in one response, which the SDK holds in memory before the first batch, so read a large table in bounded pieces, for example with `limit` and `offset` or a filter.
|
||||
|
||||
In OSS, you can also stream batches through the underlying Lance dataset directly — useful when you need filter pushdown or fragment-level parallelism via `ds.scanner(...)`:
|
||||
|
||||
@@ -66,7 +73,7 @@ for batch in ds.to_batches(columns=["id", "text"], batch_size=10000):
|
||||
<Info>
|
||||
**API disparity between OSS and Enterprise.**
|
||||
|
||||
LanceDB OSS exposes the `to_pandas()`, `to_arrow()`, and `table.to_lance()` methods for direct Lance dataset access. Enterprise's `RemoteTable` exposes none of these. Only `table.search(...)` and `table.query(...)`. In general, it's always best to go through `search()` and `query()` to keep your code portable across both OSS and Enterprise.
|
||||
A local table also gives direct Lance dataset access through `table.to_lance()`. Against a deployment, the synchronous Python table raises for `to_pandas()`, `to_arrow()` and `to_lance()`, while async Python and TypeScript run `to_arrow()` as a query over the whole table. Going through queries keeps your code portable across both OSS and Enterprise.
|
||||
</Info>
|
||||
|
||||
#### Diagnostics {#diagnostics}
|
||||
|
||||
Reference in New Issue
Block a user