mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
docs: consolidate pages that were split for no reason
Ten merges and renames, all in one direction: fewer pages, each covering a whole subject. Table absorbs schema evolution and versioning. Vector search absorbs multi-vector. Explain absorbs explain analyze. Reranking was seven pages -- an overview and six rerankers of a few dozen lines each -- and is now one. The vector index page absorbs quantization, which was only ever a parameter of it. Indexing moves to Manage, where the other catalog objects are. The operational half stays in Operate as one Indexing page, absorbing GPU indexing -- which was filed in sophon despite importing lancedb's snippets and being mostly open-source content with one Enterprise-badged section. Performance was a single page aggregating tips for four unrelated operations, which is hard to keep current. Each section now sits with the operation it is about: ingestion tips on Insert, index tips on Indexing, compaction tips on Compaction, query tips on Search. Data Types leaves Concepts. The type reference belongs in the SQL reference, which already described the same types from the query side; Table now carries a short summary that points there. The multimodal material it had absorbed is its own page again, since storing blobs is a task rather than a type definition. Query becomes Semantic Search, Branches becomes Branch, and the Enterprise Security page becomes Compliance. Redirects cover every moved URL, and 30 files were repointed at the merged anchors.
This commit is contained in:
Jack Ye
@@ -55,7 +55,7 @@ worked tutorial.
|
||||
</Card>
|
||||
</Columns>
|
||||
|
||||
The second is the more interesting pattern. [Branches](/tables/versioning)
|
||||
The second is the more interesting pattern. [Branches](/management/table#versioning)
|
||||
give an agent somewhere to be wrong safely: it can rewrite a schema, recompute a
|
||||
column, or try three embedding models without touching `main`, and you keep only
|
||||
what measured better.
|
||||
|
||||
@@ -48,7 +48,7 @@ these docs is about one of them:
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns |
|
||||
| **Versions** | every write commits one, so reads can [travel back](/tables/versioning) |
|
||||
| **Versions** | every write commits one, so reads can [travel back](/management/table#versioning) |
|
||||
| **Indexes** | vector, full-text, and scalar, built over the columns |
|
||||
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ writers, not yourself.
|
||||
## Durability and history {#durability}
|
||||
|
||||
**Old versions are retained, not overwritten.** A new version does not destroy
|
||||
the one before it, which is what makes [time travel](/tables/versioning) work
|
||||
the one before it, which is what makes [time travel](/management/table#versioning) work
|
||||
and what lets a reader finish a scan that started on an older version.
|
||||
|
||||
**Deletion is a mark, not an erase.** A deleted row is recorded in a deletion
|
||||
|
||||
@@ -60,7 +60,7 @@ without them cannot become a
|
||||
[materialized view](/management/materialized-view) source without being
|
||||
recreated -- the one schema decision here that is genuinely irreversible.
|
||||
|
||||
See [Data types](/basics/data-types).
|
||||
See [Data types](/sql/data-types).
|
||||
|
||||
## SQL {#sql}
|
||||
|
||||
|
||||
+47
-32
@@ -102,6 +102,38 @@
|
||||
{
|
||||
"source": "/namespaces/usage",
|
||||
"destination": "/management/namespace"
|
||||
},
|
||||
{
|
||||
"source": "/basics/data-types",
|
||||
"destination": "/sql/data-types"
|
||||
},
|
||||
{
|
||||
"source": "/tables/schema",
|
||||
"destination": "/management/table#schema"
|
||||
},
|
||||
{
|
||||
"source": "/tables/versioning",
|
||||
"destination": "/management/table#versioning"
|
||||
},
|
||||
{
|
||||
"source": "/search/multivector-search",
|
||||
"destination": "/search/vector-search"
|
||||
},
|
||||
{
|
||||
"source": "/search/explain-analyze",
|
||||
"destination": "/search/explain"
|
||||
},
|
||||
{
|
||||
"source": "/indexing/quantization",
|
||||
"destination": "/indexing/vector-index"
|
||||
},
|
||||
{
|
||||
"source": "/performance",
|
||||
"destination": "/search#performance"
|
||||
},
|
||||
{
|
||||
"source": "/reranking/:slug*",
|
||||
"destination": "/reranking"
|
||||
}
|
||||
],
|
||||
"navigation": {
|
||||
@@ -123,7 +155,6 @@
|
||||
"basics/lance",
|
||||
"basics/catalog",
|
||||
"basics/jobs",
|
||||
"basics/data-types",
|
||||
"tables/consistency",
|
||||
"basics/guarantees",
|
||||
"basics/limits",
|
||||
@@ -138,11 +169,19 @@
|
||||
"management/database",
|
||||
"management/namespace",
|
||||
"management/table",
|
||||
"tables/schema",
|
||||
"tables/versioning",
|
||||
"tables/multimodal",
|
||||
"tables/branching",
|
||||
"management/materialized-view",
|
||||
"management/function"
|
||||
"management/function",
|
||||
{
|
||||
"group": "Indexing",
|
||||
"pages": [
|
||||
"indexing/index",
|
||||
"indexing/vector-index",
|
||||
"indexing/fts-index",
|
||||
"indexing/scalar-index"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -156,38 +195,15 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Query",
|
||||
"group": "Semantic Search",
|
||||
"pages": [
|
||||
"search/index",
|
||||
"search/vector-search",
|
||||
"search/multivector-search",
|
||||
"search/full-text-search",
|
||||
"search/hybrid-search",
|
||||
"search/filtering",
|
||||
{
|
||||
"group": "Reranking",
|
||||
"pages": [
|
||||
"reranking/index",
|
||||
"reranking/rrf",
|
||||
"reranking/linear_combination",
|
||||
"reranking/mrr",
|
||||
"reranking/cross_encoder",
|
||||
"reranking/custom-reranker",
|
||||
"reranking/eval"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Indexing",
|
||||
"pages": [
|
||||
"indexing/index",
|
||||
"indexing/vector-index",
|
||||
"indexing/fts-index",
|
||||
"indexing/scalar-index",
|
||||
"indexing/quantization"
|
||||
]
|
||||
},
|
||||
"search/explain",
|
||||
"search/explain-analyze"
|
||||
"reranking/index",
|
||||
"search/explain"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -220,8 +236,7 @@
|
||||
"storage/configuration",
|
||||
"storage/monitoring"
|
||||
]
|
||||
},
|
||||
"performance"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -23,7 +23,7 @@ table.add_columns(computed={"doubled": "x * 2"})
|
||||
```
|
||||
|
||||
That commits a new table version with the column present and empty. It is an
|
||||
ordinary table version, so the usual [versioning](/tables/versioning) applies --
|
||||
ordinary table version, so the usual [versioning](/management/table#versioning) applies --
|
||||
including that you can travel back to before the column existed.
|
||||
|
||||
## `computed` and `transforms` are different {#computed-vs-transforms}
|
||||
|
||||
@@ -19,7 +19,7 @@ import {
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
Data evolution is adding **values** derived from what a table already holds.
|
||||
[Schema evolution](/tables/schema) changes the table's shape; this fills the
|
||||
[Schema evolution](/management/table#schema) changes the table's shape; this fills the
|
||||
shape with something new.
|
||||
|
||||
The reason it is cheap is the format. Adding a derived column writes only that
|
||||
@@ -111,7 +111,7 @@ exist when that does not fit:
|
||||
<Card title="Add a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
|
||||
Declare now, fill later.
|
||||
</Card>
|
||||
<Card title="Schema evolution" icon="boxes-stacked" href="/tables/schema">
|
||||
<Card title="Schema evolution" icon="boxes-stacked" href="/management/table#schema">
|
||||
Change the table's shape instead of its contents.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -115,7 +115,7 @@ An existing view is reopened with `db.open_materialized_view(name)`.
|
||||
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
||||
Derive a column instead of a whole table.
|
||||
</Card>
|
||||
<Card title="Versioning" icon="clock-rotate-left" href="/tables/versioning">
|
||||
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
|
||||
Every refresh commits a table version.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -26,7 +26,7 @@ REFRESH COLUMN doubled ON my_table
|
||||
</CodeGroup>
|
||||
|
||||
The refresh commits an ordinary table version, so
|
||||
[versioning](/tables/versioning) applies to it like any other write.
|
||||
[versioning](/management/table#versioning) applies to it like any other write.
|
||||
|
||||
<Note>
|
||||
The SQL form is [`REFRESH COLUMN`](/sql/refresh#refresh-column), with the same resource options.
|
||||
|
||||
@@ -163,3 +163,34 @@ Then the greedy search routine operates as follows:
|
||||
* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer.
|
||||
* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method.
|
||||
* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10).
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
#### Vector indexes {#vector-indexes}
|
||||
|
||||
If vector search latency climbs with table size (i.e., queries that ran in milliseconds on a small table take seconds as it grows to millions of rows), the cause is the default brute-force scan over every vector. That works fine below ~100K vectors, but past that you should build a dedicated vector index. Pick the type by your data shape:
|
||||
|
||||
| Index | When to use |
|
||||
|-------|-------------|
|
||||
| `IVF_PQ` | General-purpose default; what Enterprise builds automatically. Higher accuracy than RQ at small dimensions (≤ 256). |
|
||||
| `IVF_RQ` | Maximum compression on high-dim vectors, faster builds than PQ. |
|
||||
| `IVF_HNSW_SQ` | Best recall/latency for unfiltered search; higher latency variance under selective filters. |
|
||||
| `IVF_FLAT` | Required for binary vectors with `hamming`. |
|
||||
|
||||
The distance metric is fixed once the index is built. Pick the distance metric based on how the embedding model was trained: `cosine` (unnormalized), `dot` (already-normalized, best performance), `l2` (general-purpose, default), `hamming` (binary). For parameter tuning, see [Vector Indexing](/indexing/vector-index).
|
||||
|
||||
#### Scalar indexes {#scalar-indexes}
|
||||
|
||||
If filtered queries slow down as the table grows — even when the filter is selective — the cause is a full column scan: without a scalar index, LanceDB evaluates the `where(...)` predicate on every row, and the same applies to `merge_insert` join keys. The best practice is to build a scalar index on every column you filter or join on, picking the type by the column's shape:
|
||||
|
||||
| Index | Best for |
|
||||
|-------|----------|
|
||||
| `BTREE` (default) | Numeric, string, temporal columns with mostly distinct values |
|
||||
| `BITMAP` | Boolean and low-cardinality columns (< ~1,000 distinct values) |
|
||||
| `LABEL_LIST` | `List<T>` columns queried with `array_has_any` / `array_has_all` |
|
||||
|
||||
See [Scalar Indexing](/indexing/scalar-index).
|
||||
|
||||
#### Full-text search {#full-text-search}
|
||||
|
||||
If your full-text index is much larger than expected, or takes longer than expected to build, the cause is usually phrase-query flags being enabled when they aren't needed: `with_position=True` and `remove_stop_words=False` both significantly inflate index size and build time. The best practice is to keep the defaults for most workloads, and only enable those flags when you actually need to search for phrases. See [FTS Indexing](/indexing/fts-index) configuration options for the full set of options.
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
---
|
||||
title: "Quantization"
|
||||
sidebarTitle: "Quantization"
|
||||
description: "Learn about quantization when creating an index in LanceDB."
|
||||
icon: "compress"
|
||||
keywords: ["quantization", "quantize", "rabitq"]
|
||||
---
|
||||
|
||||
Quantization compresses high-dimensional float vectors into a smaller, approximate representation, where instead of storing every vector as a float32 or float64, it's stored in compressed form, without too much of a compromise in search quality.
|
||||
|
||||
Use quantization when:
|
||||
|
||||
- You have a large dataset with relatively high-dimensional vectors (512, 768, 1024+)
|
||||
- Index build time and query latency matter
|
||||
|
||||
LanceDB currently exposes multiple quantized vector index types, including:
|
||||
- `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
|
||||
- `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`.
|
||||
- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
|
||||
- `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads.
|
||||
- `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search.
|
||||
|
||||
Two axes are being combined here: whether partitions are searched flatly or via an HNSW graph (`IVF_*` vs. `IVF_HNSW_*`), and which quantizer compresses the vectors (`PQ`, `RQ`, or `SQ`). `IVF_PQ` is the default and works well in many cases. For more drastic compression, RaBitQ (`IVF_RQ`) is a reasonable option. For higher recall at low latency, the HNSW-backed variants are usually the right pick. The ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table on the vector indexing page is the canonical decision tool.
|
||||
|
||||
Use the same distance metric when training the index and running queries against it. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`.
|
||||
|
||||
## RaBitQ quantization {#rabitq-quantization}
|
||||
|
||||
RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall.
|
||||
|
||||
### How RaBitQ works {#how-rabitq-works}
|
||||
|
||||
- Embeddings are grouped around centroids (as in other IVF indexes).
|
||||
- Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere.
|
||||
- The sign pattern of that vector is stored as bits (1 bit per dimension).
|
||||
- Two small corrective factors are stored:
|
||||
1. The distance from the original vector to its centroid
|
||||
2. The dot product between the normalized vector and its quantized version
|
||||
|
||||
Compared to `IVF_PQ`, RaBitQ:
|
||||
- Avoids training expensive PQ codebooks
|
||||
- Builds indexes faster and handles updates more easily
|
||||
- Maintains or improves recall at high dimensionality under the same storage budget
|
||||
|
||||
For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/).
|
||||
|
||||
### Using RaBitQ {#using-rabitq}
|
||||
|
||||
You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`.
|
||||
|
||||
<Note title="Dimension requirement">
|
||||
When using `IVF_RQ`, vector dimensions must be divisible by `8`.
|
||||
</Note>
|
||||
|
||||
`num_bits` controls how many bits per dimension are used:
|
||||
|
||||
1 bit is the classic RaBitQ setting. You can set it to 2, 4, or 8 bits to improve fidelity for better precision or recall — the main trade-off is additional storage for the extra bits per dimension, with only a modest increase in query-time compute.
|
||||
It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to how you would do in `IVF_PQ`.
|
||||
|
||||
<Warning title="Reading multi-bit indexes across versions">
|
||||
Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions.
|
||||
</Warning>
|
||||
|
||||
## API Reference {#api-reference}
|
||||
|
||||
The full list of parameters to the algorithm are listed below.
|
||||
|
||||
- `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2"
|
||||
The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product.
|
||||
- `num_partitions`: Optional[int], defaults to None
|
||||
Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count.
|
||||
- `num_bits`: int, defaults to 1
|
||||
Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage.
|
||||
- `max_iterations`: int, defaults to 50
|
||||
Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality.
|
||||
- `sample_rate`: int, defaults to 256
|
||||
Number of samples per partition during training. Higher values may improve accuracy but increase training time.
|
||||
- `target_partition_size`: Optional[int], defaults to None
|
||||
Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence.
|
||||
@@ -1,11 +1,16 @@
|
||||
---
|
||||
title: "Keeping Indexes Up-to-Date with Reindexing"
|
||||
sidebarTitle: "Reindexing"
|
||||
title: "Building and repairing indexes"
|
||||
sidebarTitle: "Indexing"
|
||||
description: "Learn how to keep your indexes up-to-date in LanceDB using incremental indexing, including best practices for adding new records without full reindexing."
|
||||
icon: "refresh"
|
||||
keywords: ["reindexing", "re-index", "incremental indexing", "reindex"]
|
||||
---
|
||||
import { PyReindexingIncremental as ReindexingIncremental } from '/snippets/indexing.mdx';
|
||||
|
||||
import {
|
||||
PyReindexingIncremental as ReindexingIncremental,
|
||||
PyGpuIndexCuda as GpuIndexCuda,
|
||||
PyGpuIndexMps as GpuIndexMps,
|
||||
} from '/snippets/indexing.mdx';
|
||||
|
||||
As you add new data to your LanceDB tables, your indexes may become outdated.
|
||||
Reindexing is the process of updating the index to account for new data -- this applies to either a full-text search (FTS) index or a vector index. Reindexing is an important operation to run periodically as your data grows, as it has performance implications.
|
||||
@@ -57,3 +62,67 @@ If you need to reclaim space more aggressively in OSS, use a shorter retention w
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Building on GPUs {#gpu}
|
||||
|
||||
With LanceDB's GPU-powered vector indexing you can index very large datasets in far less time
|
||||
than you could with the default CPU-based indexing. In our tests, LanceDB
|
||||
is capable of indexing billions of rows in under four hours on a 1-8 GPU cluster.
|
||||
|
||||
<Info>
|
||||
**Automatic GPU indexing**
|
||||
<Badge color="red">Enterprise-only</Badge>
|
||||
|
||||
Automatic GPU Indexing is currently only available in [LanceDB Enterprise](/enterprise/).
|
||||
Please [contact us](mailto:contact@lancedb.com) to enable this feature for your deployment.
|
||||
|
||||
The vector index is created when you call `create_index`. The backend will use GPU resources
|
||||
to build either the IVF or HNSW indexes. The system automatically selects the optimal GPU
|
||||
configuration based on your data size and available hardware.
|
||||
|
||||
This process is also asynchronous by default, but you can use `wait_for_index` to convert it
|
||||
into a synchronous process by waiting until the index is built.
|
||||
|
||||
</Info>
|
||||
|
||||
`wait_for_index(...)` waits for the index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. It can time out if the table is receiving continuous writes while the build is trying to catch up.
|
||||
|
||||
GPU acceleration changes how the vector index is built, not the lifecycle after the build. Rows
|
||||
appended later still need `optimize()` before they are part of the index, and `fast_search()` only
|
||||
searches indexed rows.
|
||||
|
||||
### Manual GPU indexing in LanceDB OSS {#gpu-manual-gpu-indexing-in-lancedb-oss}
|
||||
|
||||
You can use the Python SDK to manually create the `IVF_PQ` index on a GPU. You'll need
|
||||
[PyTorch>2.0](https://pytorch.org/). Note that GPU-based indexing is currently only
|
||||
supported by the synchronous SDK in LanceDB OSS.
|
||||
|
||||
Specify the values `cuda` or `mps` (on Apple Silicon) for the `accelerator` parameter
|
||||
to enable GPU training on your device.
|
||||
|
||||
#### GPU indexing on Linux {#gpu-gpu-indexing-on-linux}
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{GpuIndexCuda}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### GPU indexing on macOS (Apple Silicon) {#gpu-gpu-indexing-on-macos-apple-silicon}
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{GpuIndexMps}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Performance considerations {#gpu-performance-considerations}
|
||||
|
||||
- GPU memory usage scales with `num_partitions` and vector dimensions
|
||||
- For optimal performance, ensure GPU memory exceeds dataset size
|
||||
- Batch size is automatically tuned based on available GPU memory
|
||||
- Indexing speed improves with larger batch sizes
|
||||
|
||||
### Troubleshooting {#gpu-troubleshooting}
|
||||
|
||||
If you encounter the error `AssertionError: Torch not compiled with CUDA enabled`,
|
||||
you need to [install PyTorch with CUDA support](https://pytorch.org/get-started/locally/).
|
||||
|
||||
@@ -4,6 +4,7 @@ sidebarTitle: "Vector Index"
|
||||
description: "Build and optimize LanceDB vector indexes, including IVF, HNSW and binary quantized indexes."
|
||||
icon: "arrow-up-right-dots"
|
||||
---
|
||||
|
||||
import {
|
||||
PyVectorIndexConfigureIvf as VectorIndexConfigureIvf,
|
||||
PyVectorIndexSetup as VectorIndexSetup,
|
||||
@@ -256,7 +257,7 @@ Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall mea
|
||||
</Warning>
|
||||
|
||||
<Note title="Multivector distance constraint">
|
||||
Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules.
|
||||
Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/vector-search) for the full rules.
|
||||
</Note>
|
||||
|
||||
## Example: Construct an HNSW Index {#example-construct-an-hnsw-index}
|
||||
@@ -377,3 +378,75 @@ The `{column}_idx` suffix is a default convention, not the only supported naming
|
||||
{VectorIndexCustomName}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Quantization compresses high-dimensional float vectors into a smaller, approximate representation, where instead of storing every vector as a float32 or float64, it's stored in compressed form, without too much of a compromise in search quality.
|
||||
|
||||
Use quantization when:
|
||||
|
||||
- You have a large dataset with relatively high-dimensional vectors (512, 768, 1024+)
|
||||
- Index build time and query latency matter
|
||||
|
||||
LanceDB currently exposes multiple quantized vector index types, including:
|
||||
- `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
|
||||
- `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`.
|
||||
- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
|
||||
- `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads.
|
||||
- `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search.
|
||||
|
||||
Two axes are being combined here: whether partitions are searched flatly or via an HNSW graph (`IVF_*` vs. `IVF_HNSW_*`), and which quantizer compresses the vectors (`PQ`, `RQ`, or `SQ`). `IVF_PQ` is the default and works well in many cases. For more drastic compression, RaBitQ (`IVF_RQ`) is a reasonable option. For higher recall at low latency, the HNSW-backed variants are usually the right pick. The ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table on the vector indexing page is the canonical decision tool.
|
||||
|
||||
Use the same distance metric when training the index and running queries against it. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`.
|
||||
|
||||
## RaBitQ quantization {#rabitq-quantization}
|
||||
|
||||
RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall.
|
||||
|
||||
### How RaBitQ works {#how-rabitq-works}
|
||||
|
||||
- Embeddings are grouped around centroids (as in other IVF indexes).
|
||||
- Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere.
|
||||
- The sign pattern of that vector is stored as bits (1 bit per dimension).
|
||||
- Two small corrective factors are stored:
|
||||
1. The distance from the original vector to its centroid
|
||||
2. The dot product between the normalized vector and its quantized version
|
||||
|
||||
Compared to `IVF_PQ`, RaBitQ:
|
||||
- Avoids training expensive PQ codebooks
|
||||
- Builds indexes faster and handles updates more easily
|
||||
- Maintains or improves recall at high dimensionality under the same storage budget
|
||||
|
||||
For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/).
|
||||
|
||||
### Using RaBitQ {#using-rabitq}
|
||||
|
||||
You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`.
|
||||
|
||||
<Note title="Dimension requirement">
|
||||
When using `IVF_RQ`, vector dimensions must be divisible by `8`.
|
||||
</Note>
|
||||
|
||||
`num_bits` controls how many bits per dimension are used:
|
||||
|
||||
1 bit is the classic RaBitQ setting. You can set it to 2, 4, or 8 bits to improve fidelity for better precision or recall — the main trade-off is additional storage for the extra bits per dimension, with only a modest increase in query-time compute.
|
||||
It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to how you would do in `IVF_PQ`.
|
||||
|
||||
<Warning title="Reading multi-bit indexes across versions">
|
||||
Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions.
|
||||
</Warning>
|
||||
|
||||
## API Reference {#api-reference}
|
||||
|
||||
The full list of parameters to the algorithm are listed below.
|
||||
|
||||
- `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2"
|
||||
The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product.
|
||||
- `num_partitions`: Optional[int], defaults to None
|
||||
Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count.
|
||||
- `num_bits`: int, defaults to 1
|
||||
Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage.
|
||||
- `max_iterations`: int, defaults to 50
|
||||
Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality.
|
||||
- `sample_rate`: int, defaults to 256
|
||||
Number of samples per partition during training. Higher values may improve accuracy but increase training time.
|
||||
- `target_partition_size`: Optional[int], defaults to None
|
||||
Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence.
|
||||
|
||||
@@ -122,3 +122,94 @@ A few things to know before you wire this up:
|
||||
- Back-pressures the writer: callback invocations are serialized and run inline with each batch, so a slow callback will slow the write rather than drop updates. Every batch update is delivered, and the final `done: true` callback always fires (even on error or cancellation). Keep the callback cheap — offload heavy work to a queue you drain elsewhere.
|
||||
- Errors swallowed: anything your callback throws is logged with `console.warn` and won't abort the write, so keep the callback side-effect-only and don't rely on it for control flow.
|
||||
- Row totals: `totalRows` is only populated when the input source can report it up front (for example, a materialized `arrow.Table`). For streaming sources it stays `undefined` until the final callback, where it falls back to the actual rows written.
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
If ingestion is taking longer than expected on a large dataset, the cause is almost always how `add()` is called: each call commits a new version and a new fragment, so a per-row loop pays that per-call overhead at every row. The best practice is to pick the ingestion mode that matches your data shape — bulk ingestion when the data is already materialized, or iterator ingestion when it's streamed or computed on the fly.
|
||||
|
||||
<Note>
|
||||
**Why `merge_insert()` is significantly slower than `add()`**
|
||||
|
||||
A merge has to scan existing data to find matches on the join key (or look them up via a scalar index, if one exists), and then delete-and-reinsert any updated rows in a single transaction; `add()` simply appends new fragments.
|
||||
|
||||
Use `add()` for pure appends, and reach for `merge_insert()` only when you need upsert or conditional-insert logic. When you do use it, build a scalar index on the join column first — otherwise the match step falls back to a full column scan, which is the dominant cost at scale.
|
||||
</Note>
|
||||
|
||||
#### Bulk ingestion: for data you already have {#bulk-ingestion-for-data-you-already-have}
|
||||
|
||||
For materialized inputs (Arrow Tables, DataFrames) and file-backed sources (`pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers, estimating the partition count from the data size — more partitions means more concurrent writes and higher throughput, up to the CPU core count.
|
||||
|
||||
```python Python icon="python"
|
||||
table.add(arrow_table) # in-memory
|
||||
table.add(df) # pandas
|
||||
table.add(ds.dataset("data/", format="parquet")) # streams from disk, still parallelized
|
||||
```
|
||||
|
||||
Pass `progress=True` to watch it happen: LanceDB shows a live tqdm bar with rows written, throughput in MB/s, and active worker count.
|
||||
|
||||
```python Python icon="python"
|
||||
table.add(ds.dataset("data/", format="parquet"), progress=True)
|
||||
```
|
||||
|
||||
```text
|
||||
71%|███████▏ | 710000/1000000 [00:12<00:05, 58.8kit/s, 42.3 MB/s | 8/8 workers]
|
||||
```
|
||||
|
||||
For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over a hand-built `pyarrow.RecordBatchReader`: a `Dataset` can be counted and rescanned, so LanceDB knows the row count upfront (better auto-parallelism) and can retry a failed write from the start — a reader can only be consumed once, so neither is possible. Reach for the iterator path below only when the data genuinely can't be backed by files, e.g. to apply custom data transformations as you ingest.
|
||||
|
||||
For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path.
|
||||
|
||||
#### Iterator ingestion: for data you transform on the fly {#iterator-ingestion-for-data-you-transform-on-the-fly}
|
||||
|
||||
If each row needs work before it can be written — applying a custom transformation as you ingest, for example — the data doesn't exist as a file you can point `ds.dataset(...)` at. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead; LanceDB consumes one batch at a time as you produce them.
|
||||
|
||||
```python Python icon="python"
|
||||
def stream():
|
||||
for raw in source:
|
||||
vectors = model.encode(raw["text"])
|
||||
yield pa.RecordBatch.from_pydict({**raw, "vector": vectors})
|
||||
|
||||
table.add(stream())
|
||||
```
|
||||
|
||||
Use decently large chunks of several thousand rows or more, rather than yielding single-row batches.
|
||||
|
||||
<Note>
|
||||
**Set `write_parallelism` yourself for large inputs**
|
||||
|
||||
A reader can't be counted or rescanned the way a `Dataset` can (see above), so LanceDB can't auto-size parallelism for it — set `write_parallelism` explicitly for large inputs:
|
||||
|
||||
```python Python icon="python"
|
||||
table.add(stream(), write_parallelism=4)
|
||||
```
|
||||
|
||||
Each partition becomes its own fragment, so don't over-allocate on a small input — budget one unit of parallelism per ~100K rows or ~1 GB of data as a rule of thumb.
|
||||
</Note>
|
||||
|
||||
#### Bulk ingests into a remote table {#bulk-ingests-into-a-remote-table}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
This section applies only to remote tables (`db://` connections). Embedded (local) writes are unaffected.
|
||||
|
||||
When you write to a remote LanceDB Enterprise table, the client splits each write partition into one or more HTTP `insert` parts and uploads them under a single upload id. Each part is streamed rather than buffered, so peak memory stays bounded. But each part still has to finish within the client read timeout while the server writes it to object storage, so on very large ingests an oversized part can run past that timeout and surface as:
|
||||
|
||||
```text
|
||||
lancedb.remote.errors.HttpError: operation timed out
|
||||
```
|
||||
|
||||
Two environment variables control how parts are cut. Both are picked up automatically by the Python and TypeScript clients, and are the only way to tune this from those SDKs:
|
||||
|
||||
| Variable | Default | What it controls |
|
||||
|----------|---------|------------------|
|
||||
| `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` | 8 GiB | Maximum size of a single insert part, in LZ4-compressed Arrow IPC bytes. Set to `0` to disable splitting and send one request per partition. |
|
||||
| `LANCE_CLIENT_MAX_REQUEST_DURATION` | Half the read timeout | Maximum wall-clock time, in whole seconds, that any one insert part may stay open. Set to `0` to disable the time-based cut. |
|
||||
|
||||
A part is cut when it hits either limit, whichever comes first. Lower the byte budget when large ingests hit the read timeout; lower the duration when uploads are slow or throttled and the byte budget isn't the limiting factor.
|
||||
|
||||
```bash
|
||||
export LANCE_CLIENT_MAX_BYTES_PER_REQUEST=1073741824 # 1 GiB
|
||||
export LANCE_CLIENT_MAX_REQUEST_DURATION=120 # 2 minutes
|
||||
```
|
||||
|
||||
Splitting into more parts does not change the final table. The server stages every part under the shared upload id and merges them atomically when the write completes.
|
||||
|
||||
@@ -46,7 +46,7 @@ The latest version is never removed, whatever you pass — `timedelta(0)` prunes
|
||||
everything except it.
|
||||
|
||||
<Warning>
|
||||
Pruning is what makes [time travel](/tables/versioning) finite. A version
|
||||
Pruning is what makes [time travel](/management/table#versioning) finite. A version
|
||||
outside the retention window is gone, and a tag or branch pointing at it no
|
||||
longer resolves.
|
||||
|
||||
@@ -99,3 +99,22 @@ Compaction rewrites files, which conflicts with a concurrent
|
||||
[view refresh](/feature-engineering/materialized-views) planned against the
|
||||
version it replaces. Sequence them rather than running both at once.
|
||||
</Note>
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
Two things accumulate on a long-lived table as more and more data gets added to it:
|
||||
|
||||
- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
|
||||
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
|
||||
|
||||
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
|
||||
|
||||
```python Python icon="python"
|
||||
from datetime import timedelta
|
||||
table.optimize() # 7-day default retention
|
||||
table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner
|
||||
```
|
||||
|
||||
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Reindexing](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
|
||||
|
||||
LanceDB Enterprise handles both compaction and cleanup automatically.
|
||||
|
||||
@@ -173,7 +173,7 @@ than at the root.
|
||||
|
||||
<Warning>
|
||||
`drop_database()` removes every table in the database. There is no confirmation
|
||||
and no undo -- table [versioning](/tables/versioning) does not survive the table
|
||||
and no undo -- table [versioning](/management/table#versioning) does not survive the table
|
||||
being dropped.
|
||||
</Warning>
|
||||
|
||||
|
||||
@@ -146,7 +146,7 @@ In SQL the path is written as dotted identifiers instead.
|
||||
|
||||
<Warning>
|
||||
`drop_table` removes the table and every version of it. Table
|
||||
[versioning](/tables/versioning) does not survive the table being dropped, and
|
||||
[versioning](/management/table#versioning) does not survive the table being dropped, and
|
||||
neither `drop_database` nor `drop_all_tables` asks for confirmation.
|
||||
</Warning>
|
||||
|
||||
|
||||
+754
-18
@@ -6,12 +6,91 @@ icon: table
|
||||
keywords: ["table", "create_table", "open_table", "clone", "rename", "drop"]
|
||||
---
|
||||
|
||||
import {
|
||||
PySchemaAddSetup as SchemaAddSetup,
|
||||
TsSchemaAddSetup as TsSchemaAddSetup,
|
||||
RsSchemaAddSetup as RsSchemaAddSetup,
|
||||
PyAddColumnsDefaultValues as AddColumnsDefaultValues,
|
||||
TsAddColumnsDefaultValues as TsAddColumnsDefaultValues,
|
||||
RsAddColumnsDefaultValues as RsAddColumnsDefaultValues,
|
||||
PyAddColumnsNullable as AddColumnsNullable,
|
||||
TsAddColumnsNullable as TsAddColumnsNullable,
|
||||
RsAddColumnsNullable as RsAddColumnsNullable,
|
||||
PySchemaAlterSetup as SchemaAlterSetup,
|
||||
TsSchemaAlterSetup as TsSchemaAlterSetup,
|
||||
RsSchemaAlterSetup as RsSchemaAlterSetup,
|
||||
PyAlterColumnsRename as AlterColumnsRename,
|
||||
TsAlterColumnsRename as TsAlterColumnsRename,
|
||||
RsAlterColumnsRename as RsAlterColumnsRename,
|
||||
PyAlterColumnsDataType as AlterColumnsDataType,
|
||||
TsAlterColumnsDataType as TsAlterColumnsDataType,
|
||||
RsAlterColumnsDataType as RsAlterColumnsDataType,
|
||||
PyAlterColumnsNullable as AlterColumnsNullable,
|
||||
TsAlterColumnsNullable as TsAlterColumnsNullable,
|
||||
RsAlterColumnsNullable as RsAlterColumnsNullable,
|
||||
PyAlterColumnsMultiple as AlterColumnsMultiple,
|
||||
TsAlterColumnsMultiple as TsAlterColumnsMultiple,
|
||||
PyAlterColumnsWithExpression as AlterColumnsWithExpression,
|
||||
TsAlterColumnsWithExpression as TsAlterColumnsWithExpression,
|
||||
RsAlterColumnsMultiple as RsAlterColumnsMultiple,
|
||||
RsAlterColumnsWithExpression as RsAlterColumnsWithExpression,
|
||||
PySchemaDropSetup as SchemaDropSetup,
|
||||
TsSchemaDropSetup as TsSchemaDropSetup,
|
||||
RsSchemaDropSetup as RsSchemaDropSetup,
|
||||
PyDropColumnsSingle as DropColumnsSingle,
|
||||
TsDropColumnsSingle as TsDropColumnsSingle,
|
||||
RsDropColumnsSingle as RsDropColumnsSingle,
|
||||
PyDropColumnsMultiple as DropColumnsMultiple,
|
||||
TsDropColumnsMultiple as TsDropColumnsMultiple,
|
||||
RsDropColumnsMultiple as RsDropColumnsMultiple,
|
||||
PyAlterVectorColumn as AlterVectorColumn,
|
||||
TsAlterVectorColumn as TsAlterVectorColumn,
|
||||
RsAlterVectorColumn as RsAlterVectorColumn,
|
||||
PySchemaFieldMetadataMerge as SchemaFieldMetadataMerge,
|
||||
TsSchemaFieldMetadataMerge as TsSchemaFieldMetadataMerge,
|
||||
RsSchemaFieldMetadataMerge as RsSchemaFieldMetadataMerge,
|
||||
PySchemaFieldMetadataReplace as SchemaFieldMetadataReplace,
|
||||
TsSchemaFieldMetadataReplace as TsSchemaFieldMetadataReplace,
|
||||
RsSchemaFieldMetadataReplace as RsSchemaFieldMetadataReplace,
|
||||
PyVersioningBasicSetup as VersioningBasicSetup,
|
||||
TsVersioningBasicSetup as TsVersioningBasicSetup,
|
||||
RsVersioningBasicSetup as RsVersioningBasicSetup,
|
||||
PyVersioningCheckInitialVersion as VersioningCheckInitialVersion,
|
||||
TsVersioningCheckInitialVersion as TsVersioningCheckInitialVersion,
|
||||
RsVersioningCheckInitialVersion as RsVersioningCheckInitialVersion,
|
||||
PyVersioningUpdateData as VersioningUpdateData,
|
||||
TsVersioningUpdateData as TsVersioningUpdateData,
|
||||
RsVersioningUpdateData as RsVersioningUpdateData,
|
||||
PyVersioningAddData as VersioningAddData,
|
||||
TsVersioningAddData as TsVersioningAddData,
|
||||
RsVersioningAddData as RsVersioningAddData,
|
||||
PyVersioningCheckVersionsAfterMod as VersioningCheckVersionsAfterMod,
|
||||
TsVersioningCheckVersionsAfterMod as TsVersioningCheckVersionsAfterMod,
|
||||
RsVersioningCheckVersionsAfterMod as RsVersioningCheckVersionsAfterMod,
|
||||
PyVersioningListAllVersions as VersioningListAllVersions,
|
||||
TsVersioningListAllVersions as TsVersioningListAllVersions,
|
||||
RsVersioningListAllVersions as RsVersioningListAllVersions,
|
||||
PyVersioningRollback as VersioningRollback,
|
||||
TsVersioningRollback as TsVersioningRollback,
|
||||
RsVersioningRollback as RsVersioningRollback,
|
||||
PyVersioningCheckoutLatest as VersioningCheckoutLatest,
|
||||
TsVersioningCheckoutLatest as TsVersioningCheckoutLatest,
|
||||
RsVersioningCheckoutLatest as RsVersioningCheckoutLatest,
|
||||
PyVersioningDeleteData as VersioningDeleteData,
|
||||
TsVersioningDeleteData as TsVersioningDeleteData,
|
||||
RsVersioningDeleteData as RsVersioningDeleteData,
|
||||
PyVersioningTags as VersioningTags,
|
||||
TsVersioningTags as TsVersioningTags,
|
||||
RsVersioningTags as RsVersioningTags,
|
||||
RsVersioningMakeQuotesReader as RsVersioningMakeQuotesReader,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
A table is rows and columns, versioned. It is what searches read, what writes
|
||||
commit to, and what indexes and branches belong to.
|
||||
|
||||
This page covers the table as a catalog object. Getting data into one is
|
||||
[Ingest](/ingestion); changing its shape is
|
||||
[Schema evolution](/tables/schema).
|
||||
[Schema evolution](/management/table#schema).
|
||||
|
||||
<Note>
|
||||
Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl#tables).
|
||||
@@ -22,7 +101,7 @@ Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sq
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Columns** | including vectors, blobs, and [computed](/feature-engineering/add-computed-column) columns |
|
||||
| **Versions** | every write commits one, so reads can [travel back](/tables/versioning) |
|
||||
| **Versions** | every write commits one, so reads can [travel back](/management/table#versioning) |
|
||||
| **Indexes** | vector, full-text and scalar, built over its columns |
|
||||
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
|
||||
|
||||
@@ -121,7 +200,7 @@ DROP TABLE documents
|
||||
|
||||
<Warning>
|
||||
Dropping removes the table and **every version of it**.
|
||||
[Versioning](/tables/versioning) does not survive a drop, and neither
|
||||
[Versioning](/management/table#versioning) does not survive a drop, and neither
|
||||
`drop_all_tables` nor `drop_database` asks for confirmation.
|
||||
</Warning>
|
||||
|
||||
@@ -142,19 +221,676 @@ On a large table the drop can take a while, so `drop_table_async` returns a
|
||||
| drop | `drop_table`, `drop_table_async` | `dropTable`, `dropTableAsync` | `DROP TABLE` |
|
||||
| drop all | `drop_all_tables` | `dropAllTables` | -- |
|
||||
|
||||
## Where to go next {#next}
|
||||
## Data types {#data-types}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Ingesting data" icon="inbox" href="/ingestion">
|
||||
Creating a table from data, and writing to one that exists.
|
||||
</Card>
|
||||
<Card title="Schema evolution" icon="boxes-stacked" href="/tables/schema">
|
||||
Adding, altering and dropping columns.
|
||||
</Card>
|
||||
<Card title="Versioning" icon="clock-rotate-left" href="/tables/versioning">
|
||||
Every write commits a version; reads can go back.
|
||||
</Card>
|
||||
<Card title="Branching" icon="code-branch" href="/tables/branching">
|
||||
Isolated lines of change, merged or discarded.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
A table's schema is an **Arrow schema** -- there is no separate LanceDB type
|
||||
system mapped onto storage, so a column is the same object to the query engine
|
||||
as it is to the format.
|
||||
|
||||
| Category | Types |
|
||||
| --- | --- |
|
||||
| Numeric | `int8`-`int64` and unsigned, `float16`, `float32`, `float64` |
|
||||
| String and binary | `utf8`, `large_utf8`, `binary`, `large_binary`, `fixed_size_binary(n)` |
|
||||
| Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp`, `duration` |
|
||||
| Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` |
|
||||
| Lance extensions | vectors, [JSON](/sql/data-types#json), [blobs](/tables/multimodal), bfloat16 |
|
||||
|
||||
A vector column is a `fixed_size_list<float32, dim>` -- an ordinary Arrow type,
|
||||
which is why it is selectable and joinable but why arithmetic over it is list
|
||||
arithmetic rather than scalar.
|
||||
|
||||
See [SQL data types](/sql/data-types) for the full reference, including the
|
||||
extension types and pseudo-columns.
|
||||
|
||||
## Schema evolution {#schema}
|
||||
|
||||
Schema evolution enables non-breaking modifications to a database table's structure — such as adding columns, altering data types, or dropping fields — to adapt to evolving data requirements without service interruptions.
|
||||
LanceDB supports ACID-compliant schema evolution through granular operations (add/alter/drop columns), allowing you to:
|
||||
|
||||
* Iterate Safely: Modify schemas in production with versioned datasets and backward compatibility
|
||||
* Scale Seamlessly: Handle ML model iterations, regulatory changes, or feature additions
|
||||
* Optimize Continuously: Remove unused fields or enforce new constraints without downtime
|
||||
|
||||
<Note>
|
||||
In SQL, schema changes are [`ALTER TABLE`](/sql/ddl#columns).
|
||||
</Note>
|
||||
|
||||
### Schema evolution operations {#schema-schema-evolution-operations}
|
||||
|
||||
LanceDB supports four primary schema evolution operations:
|
||||
|
||||
1. **Adding new columns**: Extend your table with additional attributes
|
||||
2. **Altering existing columns**: Change column names, data types, or nullability
|
||||
3. **Updating field metadata**: Attach or change per-column Arrow metadata
|
||||
4. **Dropping columns**: Remove unnecessary columns from your schema
|
||||
|
||||
|
||||
<Tip title="Schema Evolution Performance">
|
||||
Schema evolution operations are applied immediately but do not typically require rewriting all data. However, data type changes may involve more substantial operations.
|
||||
</Tip>
|
||||
|
||||
Each schema evolution operation commits a new table version and returns status metadata such as
|
||||
the committed `version`. Run these operations from a mutable table handle; if you checked out an
|
||||
older version for reads, call `checkout_latest` / `checkoutLatest` before modifying the schema.
|
||||
|
||||
### Add new columns {#schema-add-new-columns}
|
||||
|
||||
You can add new columns to a table with the [`add_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.add_columns)
|
||||
method in Python, [`addColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#addcolumns) in TypeScript/JavaScript, or `add_columns` in Rust.
|
||||
New columns are populated based on SQL expressions you provide.
|
||||
|
||||
|
||||
#### Set up the example table {#schema-set-up-the-example-table}
|
||||
|
||||
First, let's create a sample table with product data to demonstrate schema evolution:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaAddSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaAddSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaAddSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Columns derived from existing data -- SQL expressions over other columns -- are
|
||||
covered in [Data evolution](/feature-engineering/data-evolution), because
|
||||
deriving values is a feature-engineering job rather than a schema one.
|
||||
</Note>
|
||||
|
||||
#### Add columns with default values {#schema-add-columns-with-default-values}
|
||||
|
||||
Add boolean columns with default values for status tracking:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddColumnsDefaultValues}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddColumnsDefaultValues}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddColumnsDefaultValues}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Add nullable columns {#schema-add-nullable-columns}
|
||||
|
||||
Add timestamp columns that can contain NULL values:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddColumnsNullable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Warning title="NULL Values in New Columns">
|
||||
When adding columns that should contain NULL values, be sure to cast the NULL to the appropriate type, e.g., `cast(NULL as timestamp)`.
|
||||
</Warning>
|
||||
|
||||
### Alter existing columns {#schema-alter-existing-columns}
|
||||
|
||||
You can alter columns using the [`alter_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.alter_columns)
|
||||
method in Python, [`alterColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#altercolumns) in TypeScript/JavaScript, or `alter_columns` in Rust. This allows you to:
|
||||
|
||||
- Rename a column
|
||||
- Change a column's data type
|
||||
- Modify nullability (whether a column can contain NULL values)
|
||||
|
||||
|
||||
#### Set up the example table {#schema-set-up-the-example-table-2}
|
||||
|
||||
Create a table with a custom schema to demonstrate column alterations:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaAlterSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaAlterSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaAlterSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Rename columns {#schema-rename-columns}
|
||||
|
||||
Change column names to better reflect their purpose:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsRename}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsRename}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsRename}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Change data types {#schema-change-data-types}
|
||||
|
||||
Convert column data types for better performance or compatibility:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsDataType}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsDataType}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsDataType}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Make columns nullable {#schema-make-columns-nullable}
|
||||
|
||||
You can alter columns to contain NULL values:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsNullable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Changing a column to nullable affects future writes and merges too: missing values are accepted
|
||||
only when the target column is nullable.
|
||||
|
||||
#### Multiple changes at once {#schema-multiple-changes-at-once}
|
||||
|
||||
Apply several alterations in a single operation:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsMultiple}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Expression-based type changes {#schema-expression-based-type-changes}
|
||||
|
||||
For transformations that are not simple casts (for example, converting `"$100"` to an integer), use a SQL-expression column add, then drop and rename:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsWithExpression}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsWithExpression}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsWithExpression}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Alter embedding types and dimensions {#schema-alter-embedding-types-and-dimensions}
|
||||
|
||||
It's quite common to need to change an embedding column's schema, in case a new model becomes available with a different embedding dimension.
|
||||
- In Python, the example shows an in-place type update when the cast is compatible.
|
||||
- In TypeScript and Rust, the example shows a dimension change (`384 -> 1024`), which cannot be cast in-place.
|
||||
|
||||
For dimension changes, use this 3-step pattern: add a new column with the target type, drop the old column, then rename the new column to the original name.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterVectorColumn}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterVectorColumn}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterVectorColumn}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Warning>
|
||||
**`FixedSizeList` Dimension Changes in TypeScript and Rust**
|
||||
|
||||
`alterColumns` / `alter_columns` can cast between compatible types, but changing `FixedSizeList` dimensions (for example `384 -> 1024`) is not a compatible cast.
|
||||
For such cases, use `addColumns` / `add_columns` (with `arrow_cast`), then `dropColumns` / `drop_columns`, then rename the replacement column.
|
||||
</Warning>
|
||||
|
||||
<Warning title="Data Type Changes">
|
||||
Changing data types requires rewriting the column data and may be resource-intensive for large tables. Renaming columns or changing nullability is more efficient as it only updates metadata.
|
||||
</Warning>
|
||||
|
||||
### Update field metadata {#schema-update-field-metadata}
|
||||
|
||||
Each column in a LanceDB table can carry a small key/value map of Arrow field metadata — useful
|
||||
for annotating columns with units, provenance, PII flags, embedding model versions, or any other
|
||||
schema-level context your application needs.
|
||||
|
||||
Use [`update_field_metadata`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.update_field_metadata)
|
||||
in Python, [`updateFieldMetadata`](https://lancedb.github.io/lancedb/js/classes/Table/#updatefieldmetadata)
|
||||
in TypeScript/JavaScript, or `update_field_metadata` in Rust to add, change, or remove these
|
||||
key/value pairs without rewriting the column data. Each call commits a new table version and returns
|
||||
the new `version`.
|
||||
|
||||
Each update targets one field by **dot-path**: top-level columns are addressed by name (for
|
||||
example `"embedding"`), and nested fields by their full path (for example `"address.zip"`). By
|
||||
default, the keys you pass are **merged** into the field's existing metadata — keys you do not
|
||||
mention are preserved, and passing `None` (Python) or `null` (TypeScript) deletes a key. Set
|
||||
`replace: true` to swap the field's entire metadata map instead of merging.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaFieldMetadataMerge}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaFieldMetadataMerge}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaFieldMetadataMerge}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
To overwrite a field's metadata entirely instead of merging, set `replace` to `true`:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaFieldMetadataReplace}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaFieldMetadataReplace}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaFieldMetadataReplace}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Tip>
|
||||
You can pass multiple updates in a single call to change metadata on several fields at once —
|
||||
each call commits a single new table version.
|
||||
</Tip>
|
||||
|
||||
#### Canonical metadata keys {#schema-canonical-metadata-keys}
|
||||
|
||||
You can use any string as a metadata key. By convention, LanceDB treats keys prefixed with
|
||||
`lancedb:` as canonical. LanceDB Enterprise displays these keys in the UI, and agents can rely
|
||||
on finding this information in a consistent place. Use them when they fit your use case:
|
||||
|
||||
| Key | Purpose | Example |
|
||||
| --- | --- | --- |
|
||||
| `lancedb:description` | Human-readable description of the field. | `"CLIP embedding of the product image"` |
|
||||
| `lancedb:tag:<name>` | User-defined key/value tag, where the suffix names the tag category. | `"lancedb:tag:model": "clip"` |
|
||||
| `lancedb:logical-column` | Groups related columns into one logical column. For example, `feature_v1` and `feature_v2` might belong to the same logical column. | `"lancedb:logical-column": "feature"` |
|
||||
| `lancedb:status` | Life cycle state of the column: `production`, `candidate`, `deprecated`, or `archived`. | `"lancedb:status": "production"` |
|
||||
|
||||
<Note>
|
||||
These keys are conventions, not enforced constraints. LanceDB does not validate their values,
|
||||
but following the conventions keeps your tables consistent with Enterprise tooling.
|
||||
</Note>
|
||||
|
||||
### Drop columns {#schema-drop-columns}
|
||||
|
||||
You can remove columns using the [`drop_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.drop_columns)
|
||||
method in Python, [`dropColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#dropcolumns) in TypeScript/JavaScript, or `drop_columns` in Rust.
|
||||
|
||||
|
||||
#### Set Up the example table {#schema-set-up-the-example-table-3}
|
||||
|
||||
Create a table with temporary columns that we'll remove:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaDropSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaDropSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaDropSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Drop single columns {#schema-drop-single-columns}
|
||||
|
||||
Remove individual columns that are no longer needed:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DropColumnsSingle}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDropColumnsSingle}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDropColumnsSingle}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Drop multiple columns {#schema-drop-multiple-columns}
|
||||
|
||||
Remove several columns at once for efficiency:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DropColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDropColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDropColumnsMultiple}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Warning title="Irreversible Column Deletion">
|
||||
Dropping columns cannot be undone. Make sure you have backups or are certain before removing columns.
|
||||
</Warning>
|
||||
|
||||
## Versioning {#versioning}
|
||||
|
||||
This page shows the core table-versioning APIs used in the code snippets for Python, TypeScript, and Rust.
|
||||
Each operation below maps directly to methods shown in the examples.
|
||||
|
||||
### Basic Versioning Example {#versioning-basic-versioning-example}
|
||||
|
||||
Let's create a table with sample data to demonstrate LanceDB's versioning capabilities:
|
||||
|
||||
#### Set Up the Table {#versioning-set-up-the-table}
|
||||
|
||||
First, let's create a table with some sample data:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningBasicSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningBasicSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningBasicSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Expandable title="Rust helper used in this page">
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningMakeQuotesReader}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
</Expandable>
|
||||
|
||||
#### Check Initial Version {#versioning-check-initial-version}
|
||||
|
||||
After creating the table, let's check the initial version information:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningCheckInitialVersion}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningCheckInitialVersion}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningCheckInitialVersion}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Modify Data {#versioning-modify-data}
|
||||
|
||||
When you modify data through operations like update or delete, LanceDB automatically creates new versions.
|
||||
|
||||
#### Update Existing Data {#versioning-update-existing-data}
|
||||
|
||||
Let's update some existing records to see versioning in action:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningUpdateData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningUpdateData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningUpdateData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Add New Data {#versioning-add-new-data}
|
||||
|
||||
Now let's add more records to the table:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningAddData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningAddData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningAddData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Check Version Changes {#versioning-check-version-changes}
|
||||
|
||||
Let's see how the versions have changed after our modifications:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningCheckVersionsAfterMod}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningCheckVersionsAfterMod}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningCheckVersionsAfterMod}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Rollback to Previous Versions {#versioning-rollback-to-previous-versions}
|
||||
|
||||
LanceDB supports fast rollbacks to any previous version without data duplication.
|
||||
|
||||
#### View All Versions {#versioning-view-all-versions}
|
||||
|
||||
First, let's see all the versions we've created:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningListAllVersions}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningListAllVersions}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningListAllVersions}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Restore a Version Snapshot {#versioning-restore-a-version-snapshot}
|
||||
|
||||
Now let's restore a captured version snapshot:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningRollback}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningRollback}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningRollback}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Tag-Based Versioning {#versioning-tag-based-versioning}
|
||||
|
||||
Numeric table versions like `v3` or `v17` are precise but hard to remember. Tags
|
||||
let you attach human-readable labels (e.g., `"prod"`, `"baseline"`,
|
||||
`"q3-evaluation"`) to specific versions and check those out by name. They are
|
||||
conceptually similar to git tags, and unlike numeric versions, **tagged versions
|
||||
are preserved when old versions are pruned** (see the cleanup note at the bottom
|
||||
of this page).
|
||||
|
||||
The tags API supports the standard CRUD operations — create, list, update, delete —
|
||||
plus checking out by tag name.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningTags}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningTags}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningTags}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Deleting a tag only removes the label, not the version it points to. After
|
||||
deletion, the underlying table version becomes eligible for cleanup again.
|
||||
</Note>
|
||||
|
||||
### Branches {#versioning-branches}
|
||||
|
||||
Beyond linear history, LanceDB also supports **branches** — isolated, writable
|
||||
lines of history forked from `main` (or a specific version). Whereas tags and
|
||||
`checkout` give you read-only views of existing versions, a branch has its own
|
||||
writable history, making it ideal for experiments, backfills, and migrations
|
||||
that you want to keep separate from production reads on `main`.
|
||||
|
||||
Branches are covered in their own guide: see [Branches](/tables/branching).
|
||||
|
||||
### Delete Data From the Table {#versioning-delete-data-from-the-table}
|
||||
|
||||
Let's demonstrate how deletions also create new versions:
|
||||
|
||||
#### Go Back to Latest Version {#versioning-go-back-to-latest-version}
|
||||
|
||||
First, let's return to the latest version:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningCheckoutLatest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningCheckoutLatest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningCheckoutLatest}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Delete Data {#versioning-delete-data}
|
||||
|
||||
Now let's delete some data to see how it affects versioning:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningDeleteData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningDeleteData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningDeleteData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Version History and Operations {#versioning-version-history-and-operations}
|
||||
|
||||
On a fresh table, the snippets in this guide produce this version sequence:
|
||||
|
||||
1. `v1`: create table (`create_table` / `createTable` / `create_table`)
|
||||
2. `v2`: update rows (`update`)
|
||||
3. `v3`: add rows (`add`)
|
||||
4. `v4`: restore snapshot (`restore`) from `version_after_mod`/`versionAfterMod`
|
||||
5. `v5`: delete rows (`delete`)
|
||||
|
||||
Read-only and checkout operations shown here (`list_versions`/`listVersions`, `version`, `checkout`, `checkout_latest`/`checkoutLatest`) do not create new versions.
|
||||
|
||||
The version metadata fields can differ by backend. Direct table-backed version listing exposes a
|
||||
timestamp, while namespace-backed listing may expose fields such as `manifest_path`,
|
||||
`manifest_size`, `e_tag`, and `timestamp_millis`. In deployments that use managed versioning,
|
||||
prefer the table version APIs exposed by LanceDB Enterprise or the namespace service instead of
|
||||
mixing in lower-level Lance file operations.
|
||||
|
||||
<Note>
|
||||
**System Operations**
|
||||
|
||||
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
|
||||
In LanceDB OSS and Enterprise, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
|
||||
which is when old-version files are removed and disk space is reclaimed.
|
||||
|
||||
**Tagged versions are exempt from cleanup.** A version with a tag pointing at it is
|
||||
retained regardless of age, and its files are not removed by `optimize()`. To make
|
||||
a tagged version eligible for pruning, [delete the tag](#tag-based-versioning) first.
|
||||
</Note>
|
||||
|
||||
@@ -1,241 +0,0 @@
|
||||
---
|
||||
title: "Performance Tips and Best Practices"
|
||||
sidebarTitle: "Performance Tips"
|
||||
description: "Optimize LanceDB for your workload across ingestion, indexing, querying, and maintenance."
|
||||
icon: "gauge-high"
|
||||
keywords: ["performance", "tuning", "best practices", "optimization", "ingestion", "indexing", "vector search", "filtering", "compaction", "oss", "enterprise"]
|
||||
---
|
||||
|
||||
LanceDB is performant by default. This page covers performance best practices that matter when you want to ensure you get the right performance for a specific workload. Use the table below to jump to the area relevant to what you're working on.
|
||||
|
||||
| When you're working on... | Read |
|
||||
|---------------------------|------|
|
||||
| Loading data into a table | [Ingestion](#ingestion) |
|
||||
| Running filtered or vector queries at scale | [Indexing](#indexing) |
|
||||
| Iterating over large result sets (training, export, migration) | [Querying](#querying) |
|
||||
| Keeping a long-lived dataset healthy | [Maintenance](#maintenance) |
|
||||
| Inspecting query plans | [Diagnostics](#diagnostics) |
|
||||
|
||||
<Danger>
|
||||
When using Python with multiprocessing, use `spawn` rather than `fork`. LanceDB is multi-threaded internally, and `fork` plus a multi-threaded process is unsafe.
|
||||
</Danger>
|
||||
|
||||
## Ingestion {#ingestion}
|
||||
|
||||
If ingestion is taking longer than expected on a large dataset, the cause is almost always how `add()` is called: each call commits a new version and a new fragment, so a per-row loop pays that per-call overhead at every row. The best practice is to pick the ingestion mode that matches your data shape — bulk ingestion when the data is already materialized, or iterator ingestion when it's streamed or computed on the fly.
|
||||
|
||||
<Note>
|
||||
**Why `merge_insert()` is significantly slower than `add()`**
|
||||
|
||||
A merge has to scan existing data to find matches on the join key (or look them up via a scalar index, if one exists), and then delete-and-reinsert any updated rows in a single transaction; `add()` simply appends new fragments.
|
||||
|
||||
Use `add()` for pure appends, and reach for `merge_insert()` only when you need upsert or conditional-insert logic. When you do use it, build a scalar index on the join column first — otherwise the match step falls back to a full column scan, which is the dominant cost at scale.
|
||||
</Note>
|
||||
|
||||
### Bulk ingestion: for data you already have {#bulk-ingestion-for-data-you-already-have}
|
||||
|
||||
For materialized inputs (Arrow Tables, DataFrames) and file-backed sources (`pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers, estimating the partition count from the data size — more partitions means more concurrent writes and higher throughput, up to the CPU core count.
|
||||
|
||||
```python Python icon="python"
|
||||
table.add(arrow_table) # in-memory
|
||||
table.add(df) # pandas
|
||||
table.add(ds.dataset("data/", format="parquet")) # streams from disk, still parallelized
|
||||
```
|
||||
|
||||
Pass `progress=True` to watch it happen: LanceDB shows a live tqdm bar with rows written, throughput in MB/s, and active worker count.
|
||||
|
||||
```python Python icon="python"
|
||||
table.add(ds.dataset("data/", format="parquet"), progress=True)
|
||||
```
|
||||
|
||||
```text
|
||||
71%|███████▏ | 710000/1000000 [00:12<00:05, 58.8kit/s, 42.3 MB/s | 8/8 workers]
|
||||
```
|
||||
|
||||
For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over a hand-built `pyarrow.RecordBatchReader`: a `Dataset` can be counted and rescanned, so LanceDB knows the row count upfront (better auto-parallelism) and can retry a failed write from the start — a reader can only be consumed once, so neither is possible. Reach for the iterator path below only when the data genuinely can't be backed by files, e.g. to apply custom data transformations as you ingest.
|
||||
|
||||
For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path.
|
||||
|
||||
### Iterator ingestion: for data you transform on the fly {#iterator-ingestion-for-data-you-transform-on-the-fly}
|
||||
|
||||
If each row needs work before it can be written — applying a custom transformation as you ingest, for example — the data doesn't exist as a file you can point `ds.dataset(...)` at. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead; LanceDB consumes one batch at a time as you produce them.
|
||||
|
||||
```python Python icon="python"
|
||||
def stream():
|
||||
for raw in source:
|
||||
vectors = model.encode(raw["text"])
|
||||
yield pa.RecordBatch.from_pydict({**raw, "vector": vectors})
|
||||
|
||||
table.add(stream())
|
||||
```
|
||||
|
||||
Use decently large chunks of several thousand rows or more, rather than yielding single-row batches.
|
||||
|
||||
<Note>
|
||||
**Set `write_parallelism` yourself for large inputs**
|
||||
|
||||
A reader can't be counted or rescanned the way a `Dataset` can (see above), so LanceDB can't auto-size parallelism for it — set `write_parallelism` explicitly for large inputs:
|
||||
|
||||
```python Python icon="python"
|
||||
table.add(stream(), write_parallelism=4)
|
||||
```
|
||||
|
||||
Each partition becomes its own fragment, so don't over-allocate on a small input — budget one unit of parallelism per ~100K rows or ~1 GB of data as a rule of thumb.
|
||||
</Note>
|
||||
|
||||
### Bulk ingests into a remote table {#bulk-ingests-into-a-remote-table}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
This section applies only to remote tables (`db://` connections). Embedded (local) writes are unaffected.
|
||||
|
||||
When you write to a remote LanceDB Enterprise table, the client splits each write partition into one or more HTTP `insert` parts and uploads them under a single upload id. Each part is streamed rather than buffered, so peak memory stays bounded. But each part still has to finish within the client read timeout while the server writes it to object storage, so on very large ingests an oversized part can run past that timeout and surface as:
|
||||
|
||||
```text
|
||||
lancedb.remote.errors.HttpError: operation timed out
|
||||
```
|
||||
|
||||
Two environment variables control how parts are cut. Both are picked up automatically by the Python and TypeScript clients, and are the only way to tune this from those SDKs:
|
||||
|
||||
| Variable | Default | What it controls |
|
||||
|----------|---------|------------------|
|
||||
| `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` | 8 GiB | Maximum size of a single insert part, in LZ4-compressed Arrow IPC bytes. Set to `0` to disable splitting and send one request per partition. |
|
||||
| `LANCE_CLIENT_MAX_REQUEST_DURATION` | Half the read timeout | Maximum wall-clock time, in whole seconds, that any one insert part may stay open. Set to `0` to disable the time-based cut. |
|
||||
|
||||
A part is cut when it hits either limit, whichever comes first. Lower the byte budget when large ingests hit the read timeout; lower the duration when uploads are slow or throttled and the byte budget isn't the limiting factor.
|
||||
|
||||
```bash
|
||||
export LANCE_CLIENT_MAX_BYTES_PER_REQUEST=1073741824 # 1 GiB
|
||||
export LANCE_CLIENT_MAX_REQUEST_DURATION=120 # 2 minutes
|
||||
```
|
||||
|
||||
Splitting into more parts does not change the final table. The server stages every part under the shared upload id and merges them atomically when the write completes.
|
||||
|
||||
## Indexing {#indexing}
|
||||
|
||||
### Vector indexes {#vector-indexes}
|
||||
|
||||
If vector search latency climbs with table size (i.e., queries that ran in milliseconds on a small table take seconds as it grows to millions of rows), the cause is the default brute-force scan over every vector. That works fine below ~100K vectors, but past that you should build a dedicated vector index. Pick the type by your data shape:
|
||||
|
||||
| Index | When to use |
|
||||
|-------|-------------|
|
||||
| `IVF_PQ` | General-purpose default; what Enterprise builds automatically. Higher accuracy than RQ at small dimensions (≤ 256). |
|
||||
| `IVF_RQ` | Maximum compression on high-dim vectors, faster builds than PQ. |
|
||||
| `IVF_HNSW_SQ` | Best recall/latency for unfiltered search; higher latency variance under selective filters. |
|
||||
| `IVF_FLAT` | Required for binary vectors with `hamming`. |
|
||||
|
||||
The distance metric is fixed once the index is built. Pick the distance metric based on how the embedding model was trained: `cosine` (unnormalized), `dot` (already-normalized, best performance), `l2` (general-purpose, default), `hamming` (binary). For parameter tuning, see [Vector Indexing](/indexing/vector-index).
|
||||
|
||||
### Scalar indexes {#scalar-indexes}
|
||||
|
||||
If filtered queries slow down as the table grows — even when the filter is selective — the cause is a full column scan: without a scalar index, LanceDB evaluates the `where(...)` predicate on every row, and the same applies to `merge_insert` join keys. The best practice is to build a scalar index on every column you filter or join on, picking the type by the column's shape:
|
||||
|
||||
| Index | Best for |
|
||||
|-------|----------|
|
||||
| `BTREE` (default) | Numeric, string, temporal columns with mostly distinct values |
|
||||
| `BITMAP` | Boolean and low-cardinality columns (< ~1,000 distinct values) |
|
||||
| `LABEL_LIST` | `List<T>` columns queried with `array_has_any` / `array_has_all` |
|
||||
|
||||
See [Scalar Indexing](/indexing/scalar-index).
|
||||
|
||||
### Full-text search {#full-text-search}
|
||||
|
||||
If your full-text index is much larger than expected, or takes longer than expected to build, the cause is usually phrase-query flags being enabled when they aren't needed: `with_position=True` and `remove_stop_words=False` both significantly inflate index size and build time. The best practice is to keep the defaults for most workloads, and only enable those flags when you actually need to search for phrases. See [FTS Indexing](/indexing/fts-index) configuration options for the full set of options.
|
||||
|
||||
## Compaction and cleanup {#compaction-and-cleanup}
|
||||
|
||||
Two things accumulate on a long-lived table as more and more data gets added to it:
|
||||
|
||||
- **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
|
||||
- **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
|
||||
|
||||
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
|
||||
|
||||
```python Python icon="python"
|
||||
from datetime import timedelta
|
||||
table.optimize() # 7-day default retention
|
||||
table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner
|
||||
```
|
||||
|
||||
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Reindexing](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
|
||||
|
||||
LanceDB Enterprise handles both compaction and cleanup automatically.
|
||||
|
||||
## Querying {#querying}
|
||||
|
||||
Three knobs materially affect query latency, memory use, and recall. Be deliberate about each one on every query:
|
||||
|
||||
<Steps>
|
||||
<Step title="Project and limit explicitly">
|
||||
If queries return more data than you need or take longer than expected, the cause is usually projecting more columns than necessary, or letting the result count run unbounded. The best practice is to always pass both `select()` and `limit()`:
|
||||
|
||||
```python Python icon="python"
|
||||
table.search(emb).select(["id", "title"]).limit(20)
|
||||
```
|
||||
</Step>
|
||||
<Step title="Stay on pre-filter unless fewer matches are acceptable">
|
||||
If a query with `prefilter=False` returns fewer than `limit` results — sometimes zero — the cause is that post-filter applies the predicate after the top-k is selected, so only candidates that already passed the search are filtered. Pre-filter is the default and guarantees every result satisfies the predicate. Switch to `prefilter=False` only when fewer-than-`limit` results are acceptable. See [Filtering](/search/filtering).
|
||||
</Step>
|
||||
<Step title="Tune for recall one knob at a time">
|
||||
If recall is lower than expected, the cause is that the search-time knobs limit how many candidates the index considers. Adjust based on your index type:
|
||||
|
||||
- Quantized indexes (PQ, RQ, SQ): `refine_factor` pulls extra candidates and re-scores them on full vectors.
|
||||
- HNSW-backed indexes: `ef` at search time. Start at `1.5 × k`, raise toward `10 × k` if recall is short.
|
||||
- IVF candidate breadth: `nprobes` is auto-tuned; override only when a selective pre-filter leaves too few neighbors.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
For hybrid search, the default `RRFReranker()` combines vector and FTS results into a single ranking via reciprocal rank fusion. See [Hybrid Search](/search/hybrid-search).
|
||||
|
||||
### Avoid materializing the whole table {#avoid-materializing-the-whole-table}
|
||||
|
||||
If you need every row in the table (for training, export, or migration), calling `to_pandas()` or `to_arrow()` will run you out of memory on any non-trivial dataset — both materialize the full table at once. The best practice is to iterate via `table.search(...)` or `table.query(...)`, which work the same way in both LanceDB OSS and Enterprise.
|
||||
|
||||
In OSS, you can also stream batches through the underlying Lance dataset directly — useful when you need filter pushdown or fragment-level parallelism via `ds.scanner(...)`:
|
||||
|
||||
```python Python icon="python"
|
||||
ds = table.to_lance()
|
||||
for batch in ds.to_batches(columns=["id", "text"], batch_size=10000):
|
||||
process(batch)
|
||||
```
|
||||
|
||||
<Info>
|
||||
**API disparity between OSS and Enterprise.**
|
||||
|
||||
LanceDB OSS exposes the `to_pandas()`, `to_arrow()`, and `table.to_lance()` methods for direct Lance dataset access. Enterprise's `RemoteTable` exposes none of these. Only `table.search(...)` and `table.query(...)`. In general, it's always best to go through `search()` and `query()` to keep your code portable across both OSS and Enterprise.
|
||||
</Info>
|
||||
|
||||
### Diagnostics {#diagnostics}
|
||||
|
||||
To analyze a slow query, inspect what the query engine actually did and the state of the indexes it touched. These two tools surface that information — use them in this order:
|
||||
|
||||
```python Python icon="python"
|
||||
print(table.search(emb).where("year > 2000").limit(10).analyze_plan())
|
||||
print(table.index_stats("vector_idx")) # num_unindexed_rows should be ~0
|
||||
```
|
||||
|
||||
`analyze_plan()` returns the execution plan with per-stage timings, so you can see where the query actually spent its time. `index_stats()` shows how many rows are still unindexed — you want `num_unindexed_rows` to be ~0. In `analyze_plan`, look for:
|
||||
|
||||
| Plan pattern | Fix |
|
||||
|--------------|-----|
|
||||
| `LanceScan` with high `bytes_read` / `iops` | Add a missing index, project columns with `select()`, or check that the dataset has been compacted |
|
||||
| Multiple sequential filters | Reorder filter conditions |
|
||||
|
||||
[Explain Analyze](/search/explain-analyze) walks through a fully worked before/after example, including how `KNNVectorDistance` and `output_batches` change once indexes are in place.
|
||||
|
||||
## Where to go next {#where-to-go-next}
|
||||
|
||||
<Columns cols={2}>
|
||||
<Card title="Explain Analyze" icon="gauge" href="/search/explain-analyze">
|
||||
Execute a query and read where the time went.
|
||||
</Card>
|
||||
<Card title="Vector Indexing" icon="layer-group" href="/indexing/vector-index">
|
||||
Index types, parameters, and tuning in depth.
|
||||
</Card>
|
||||
<Card title="Filtering" icon="filter" href="/search/filtering">
|
||||
Pre- vs post-filter, scalar indexes, list columns.
|
||||
</Card>
|
||||
<Card title="Enterprise Benchmarks" icon="server" href="/enterprise/performance">
|
||||
Benchmark methodology and reference latency numbers.
|
||||
</Card>
|
||||
</Columns>
|
||||
@@ -1,53 +0,0 @@
|
||||
---
|
||||
title: Cross Encoder Reranker
|
||||
sidebarTitle: "Cross Encoder"
|
||||
description: Implement semantic search reranking in LanceDB using Cross Encoder models. Features configurable model selection, device optimization, and comprehensive scoring options for all search types.
|
||||
|
||||
---
|
||||
|
||||
import { PyRerankingCrossEncoderUsage } from '/snippets/integrations.mdx';
|
||||
|
||||
# Cross Encoder Reranker
|
||||
|
||||
This reranker uses Cross Encoder models from sentence-transformers to rerank the search results. You can use this reranker by passing `CrossEncoderReranker()` to the `rerank()` method.
|
||||
> **Note:** Supported query types – Hybrid, Vector, and FTS.
|
||||
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingCrossEncoderUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Accepted Arguments {#accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `model_name` | `str` | `"cross-encoder/ms-marco-TinyBERT-L-6"` | The name of the reranker model to use.|
|
||||
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
|
||||
| `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". |
|
||||
| `trust_remote_code` | `bool` | `True` | Passed to Sentence Transformers model loading. Set this to `False` when you only want to load models that do not require custom repository code. |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all" is supported, returns relevance score along with the vector and/or FTS scores depending on query type. |
|
||||
|
||||
The reranker loads the model locally through `sentence-transformers`, so install the local model
|
||||
runtime dependencies you need, such as PyTorch and any device-specific acceleration packages.
|
||||
|
||||
## Supported Scores for each query type {#supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
### Hybrid Search {#hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
### Vector Search {#vector-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
### FTS Search {#fts-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
@@ -1,134 +0,0 @@
|
||||
---
|
||||
title: Building Custom Rerankers
|
||||
sidebarTitle: "Custom rerankers"
|
||||
description: Learn how to create custom rerankers in LanceDB by extending the base Reranker class.
|
||||
icon: "code"
|
||||
---
|
||||
|
||||
You can build your own custom reranker in LanceDB by subclassing the base `Reranker` class. At a
|
||||
minimum, you need to implement `rerank_hybrid()`, which is the logic that combines vector and
|
||||
full-text search results. Beyond that, you can optionally implement `rerank_vector()` and
|
||||
`rerank_fts()` if you want your reranker to also handle pure vector or pure full-text searches.
|
||||
|
||||
Decide up front which surfaces — hybrid, pure vector, or pure full-text — your reranker should
|
||||
cover, and only override the ones you need. The base class leaves `rerank_vector()` and
|
||||
`rerank_fts()` unimplemented, so calling `.rerank(...)` on a single-modality search you haven't
|
||||
overridden raises `NotImplementedError` rather than silently returning unsorted results. That's a
|
||||
useful guard, but worth knowing about before you wire up a query path you didn't plan for.
|
||||
|
||||
The Python base class exposes hybrid, vector-only, and FTS-only rerank hooks. TypeScript and Rust
|
||||
currently expose the custom reranker interface for hybrid reranking. In Rust, a custom reranker must
|
||||
also satisfy the trait bounds `Debug + Send + Sync`.
|
||||
|
||||
## Interface {#interface}
|
||||
|
||||
The `Reranker` base interface comes with a `merge_results()` method that can be used to combine the
|
||||
results of semantic and full-text search. This is a vanilla merging algorithm that simply concatenates
|
||||
the results and removes the duplicates without taking the scores into consideration. It only keeps the
|
||||
first copy of the row encountered. This works well in cases that don't require the scores of semantic
|
||||
and full-text search to combine the results. If you want to use the scores or want to support
|
||||
`return_score="all"`, you'll need to implement your own merging algorithm. The base
|
||||
`return_score` option accepts only `"relevance"` and `"all"`.
|
||||
|
||||
Whichever methods you override, your reranker has one job on the way out: attach a
|
||||
`_relevance_score` column with the most relevant rows at the top. LanceDB will reject the result
|
||||
if that column is missing, and downstream `.limit(...)` calls trust the order you return, so
|
||||
sort descending before handing the table back.
|
||||
|
||||
For vector-only reranking in Python, pass a text query to `.rerank(..., query_string="...")`.
|
||||
The vector query itself may be numeric, but `rerank_vector(query, vector_results)` still receives
|
||||
a string query so your reranker can score each candidate against the user's text intent.
|
||||
|
||||
Below, we show the pseudocode of a custom reranker that combines the results of semantic and full-text
|
||||
search using a linear combination of the scores:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
from lancedb.rerankers import Reranker
|
||||
import pyarrow as pa
|
||||
|
||||
class MyReranker(Reranker):
|
||||
def __init__(self, param1, param2, ..., return_score="relevance"):
|
||||
super().__init__(return_score)
|
||||
self.param1 = param1
|
||||
self.param2 = param2
|
||||
|
||||
def rerank_hybrid(self, query: str, vector_results: pa.Table, fts_results: pa.Table):
|
||||
# Use the built-in merging function
|
||||
combined_result = self.merge_results(vector_results, fts_results)
|
||||
|
||||
# Do something with the combined results
|
||||
# ...
|
||||
|
||||
# Return the combined results
|
||||
return combined_result
|
||||
|
||||
def rerank_vector(self, query: str, vector_results: pa.Table):
|
||||
# Do something with the vector results
|
||||
# ...
|
||||
|
||||
# Return the vector results
|
||||
return vector_results
|
||||
|
||||
def rerank_fts(self, query: str, fts_results: pa.Table):
|
||||
# Do something with the FTS results
|
||||
# ...
|
||||
|
||||
# Return the FTS results
|
||||
return fts_results
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Example {#example}
|
||||
|
||||
As an example, let's build custom reranker that enhances the Cohere Reranker by accepting a filter
|
||||
query, and accepts any other `CohereReranker` params as `kwargs`.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
from typing import List, Union
|
||||
import pandas as pd
|
||||
from lancedb.rerankers import CohereReranker
|
||||
|
||||
class ModifiedCohereReranker(CohereReranker):
|
||||
def __init__(self, filters: Union[str, List[str]], **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
filters = filters if isinstance(filters, list) else [filters]
|
||||
self.filters = filters
|
||||
|
||||
def rerank_hybrid(self, query: str, vector_results: pa.Table, fts_results: pa.Table)-> pa.Table:
|
||||
combined_result = super().rerank_hybrid(query, vector_results, fts_results)
|
||||
df = combined_result.to_pandas()
|
||||
for filter in self.filters:
|
||||
df = df.query("not text.str.contains(@filter)")
|
||||
|
||||
return pa.Table.from_pandas(df)
|
||||
|
||||
def rerank_vector(self, query: str, vector_results: pa.Table)-> pa.Table:
|
||||
vector_results = super().rerank_vector(query, vector_results)
|
||||
df = vector_results.to_pandas()
|
||||
for filter in self.filters:
|
||||
df = df.query("not text.str.contains(@filter)")
|
||||
|
||||
return pa.Table.from_pandas(df)
|
||||
|
||||
def rerank_fts(self, query: str, fts_results: pa.Table)-> pa.Table:
|
||||
fts_results = super().rerank_fts(query, fts_results)
|
||||
df = fts_results.to_pandas()
|
||||
for filter in self.filters:
|
||||
df = df.query("not text.str.contains(@filter)")
|
||||
|
||||
return pa.Table.from_pandas(df)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Card icon="lightbulb">
|
||||
Under the hood, `vector_results` and `fts_results` are PyArrow tables. You can learn more about
|
||||
PyArrow tables [here](https://arrow.apache.org/docs/python). The advantage of PyArrow tables is their
|
||||
interoperability -- you can easily convert them to Pandas/Polars DataFrames, `PyDict`, `PyList`, etc.
|
||||
|
||||
The benefits are also bidirectional -- just as you can easily convert PyArrow tables _to_ Pandas
|
||||
DataFrames using the `to_pandas()` method -- you can perform DataFrame transformations
|
||||
and just as easily convert the DataFrame back to PyArrow tables using `pa.Table.from_pandas()` method
|
||||
as shown in the example above.
|
||||
</Card>
|
||||
@@ -1,103 +0,0 @@
|
||||
---
|
||||
title: "Evaluating Hybrid Search Performance"
|
||||
sidebarTitle: "Evaluation"
|
||||
description: Learn about evaluating hybrid search performance in LanceDB.
|
||||
icon: "chart-bar"
|
||||
---
|
||||
|
||||
Hybrid search is an often misused and/or misunderstood term. In this section, we're using
|
||||
the definition of "hybrid search" to mean using a combination of keyword-based and vector search.
|
||||
Because the vector search operates in a dense embedding space and keyword-based search operate
|
||||
in a sparse embedding space, their relevance scores cannot be directly compared.
|
||||
Combining results from multiple searches thus requires a reranking step.
|
||||
|
||||
Before evaluating hybrid search, build an FTS index on the text column and use a table with embedding
|
||||
metadata or explicit query vectors for the vector side. Otherwise the evaluation is measuring setup
|
||||
fallbacks or errors rather than the reranker.
|
||||
|
||||
## Reranking strategies {#reranking-strategies}
|
||||
|
||||
There are two common approaches for reranking search results from multiple sources.
|
||||
|
||||
- **Score-based**: Calculate final relevance scores from the individual search algorithm scores. Examples: Reciprocal Rank Fusion (the default in LanceDB), mean reciprocal rank fusion, and weighted linear combination of semantic and keyword-based search scores.
|
||||
|
||||
- **Relevance-based**: Discards the existing scores and calculates the relevance of each search result-query pair. Example: Cross Encoder models
|
||||
|
||||
<Info>
|
||||
If you call `.rerank()` on a hybrid query without passing a reranker, LanceDB defaults to
|
||||
`RRFReranker()` — a score-based reranker that uses Reciprocal Rank Fusion. This is the
|
||||
score-based path most readers encounter first; `LinearCombinationReranker` is an alternative
|
||||
score-based strategy you opt into explicitly.
|
||||
</Info>
|
||||
|
||||
By default, rerankers return `_relevance_score`. Pass `return_score="all"` when a reranker
|
||||
supports it, and you also need the original vector or FTS scores for debugging.
|
||||
Evaluation code can rely on returned rows being ordered by descending `_relevance_score`. Empty
|
||||
reranked result sets still include the `_relevance_score` column.
|
||||
|
||||
The hybrid `rerank(...)` method also accepts a `normalize` argument that controls how the raw
|
||||
vector and FTS scores are made comparable before reranking:
|
||||
|
||||
- `normalize="score"` (the default) — normalizes the raw vector and FTS scores directly.
|
||||
- `normalize="rank"` — converts each result list to ranks first, then normalizes.
|
||||
|
||||
This choice materially affects score-based rerankers (such as `LinearCombinationReranker`), so
|
||||
when you evaluate score-based strategies, treat `normalize` as a tunable hyperparameter
|
||||
alongside the reranker itself.
|
||||
|
||||
For score-based evaluation, the main built-in knobs are:
|
||||
|
||||
- `RRFReranker(K=60)`: rank fusion with a positive smoothing constant.
|
||||
- `MRRReranker(weight_vector=0.5, weight_fts=0.5)`: weighted reciprocal rank fusion where the weights must sum to `1.0`.
|
||||
- `LinearCombinationReranker(weight=0.7, fill=1.0)`: blends vector and FTS scores directly; `fill` controls how strongly to penalize a result missing from one side.
|
||||
|
||||
Even though there are many more strategies for reranking, there are no "universally best"
|
||||
ones that work well for all cases, because reranking quality is dataset and application specific.
|
||||
Evaluating whether a reranking strategy is a good fit is also a challenge. In the next
|
||||
section, we discuss an example evaluation of different reranking strategies on a sample dataset.
|
||||
|
||||
## Example evaluation {#example-evaluation}
|
||||
|
||||
The table below shows our evaluation results from an experiment comparing multiple rerankers on
|
||||
~800 hybrid search queries. This is a modified version of an evaluation script by
|
||||
[LlamaIndex](https://github.com/run-llama/finetune-embedding/blob/main/evaluate.ipynb) that measures
|
||||
hit-rate \@ top-k.
|
||||
|
||||
### Using OpenAI `text-embedding-ada-002` {#using-openai-text-embedding-ada-002}
|
||||
|
||||
Vector Search baseline: **0.64**
|
||||
|
||||
| Reranker | Top-3 | Top-5 | Top-10 |
|
||||
| --- | --- | --- | --- |
|
||||
| Linear Combination | `0.73` | `0.74` | `0.85` |
|
||||
| Cross Encoder | `0.71` | `0.70` | `0.77` |
|
||||
| Cohere | `0.81` | `0.81` | `0.85` |
|
||||
| ColBERT | `0.68` | `0.68` | `0.73` |
|
||||
|
||||
<img src="https://github.com/AyushExel/assets/assets/15766192/d57b1780-ef27-414c-a5c3-73bee7808a45" />
|
||||
|
||||
|
||||
### Using OpenAI `text-embedding-3-small` {#using-openai-text-embedding-3-small}
|
||||
|
||||
Vector Search baseline: **0.59**
|
||||
|
||||
| Reranker | Top-3 | Top-5 | Top-10 |
|
||||
| --- | --- | --- | --- |
|
||||
| Linear Combination | `0.68` | `0.70` | `0.84` |
|
||||
| Cross Encoder | `0.72` | `0.72` | `0.79` |
|
||||
| Cohere | `0.79` | `0.79` | `0.84` |
|
||||
| ColBERT | `0.70` | `0.70` | `0.76` |
|
||||
|
||||
<img src="https://github.com/AyushExel/assets/assets/15766192/259adfd2-6ec6-4df6-a77d-1456598970dd" />
|
||||
|
||||
## Conclusion {#conclusion}
|
||||
|
||||
The results show that the reranking methods can significantly improve the search relevance. However,
|
||||
the improvement we saw was not consistent across all rerankers. In reality, the choice of reranker
|
||||
likely depends on the dataset and the application.
|
||||
|
||||
It's also important to note that the reranking methods are not a
|
||||
replacement for the search methods they supplement. They are complementary and it's likely that you'd
|
||||
have to tune them together to get the best results. The latency vs. recall tradeoff is also an
|
||||
important factor to consider when choosing the reranker. Hopefully this evaluation
|
||||
gives you a starting point for your own experiments with hybrid search in LanceDB!
|
||||
@@ -6,7 +6,13 @@ icon: "sort-amount-down"
|
||||
keywords: ["re-ranking", "reranker", "rerank"]
|
||||
---
|
||||
|
||||
import { PyRerankingLinearCombinationUsage, PyRerankingRrfUsage, PyRerankingCohereUsage } from '/snippets/integrations.mdx';
|
||||
import {
|
||||
PyRerankingLinearCombinationUsage,
|
||||
PyRerankingRrfUsage,
|
||||
PyRerankingCohereUsage,
|
||||
PyRerankingMrrUsage,
|
||||
PyRerankingCrossEncoderUsage,
|
||||
} from '/snippets/integrations.mdx';
|
||||
|
||||
Reranking is the process of re-ordering search results to improve relevance, often using a
|
||||
different model than the one used for the initial search. LanceDB has built-in support for reranking
|
||||
@@ -130,4 +136,409 @@ the `deduplicate` flag.
|
||||
|
||||
LanceDB also allows you to create custom rerankers by extending the base `Reranker` class. The custom reranker
|
||||
should implement the `rerank` method that takes a list of search results and returns a reranked list of
|
||||
search results. This is covered in more detail in the [creating custom rerankers](/reranking/custom-reranker/) section.
|
||||
search results. This is covered in more detail in the [creating custom rerankers](/reranking) section.
|
||||
|
||||
## Reciprocal rank fusion {#rrf}
|
||||
|
||||
## Reciprocal Rank Fusion Reranker
|
||||
|
||||
**Reciprocal Rank Fusion (RRF)** is a model-free way to merge several ranked result lists into a
|
||||
single ordering. Rather than comparing raw similarity scores (which aren't directly comparable
|
||||
across, say, a vector search and a full-text search), RRF looks only at each document's *rank
|
||||
position* in each list. It scores every document with the formula `1 / (rank + K)`, sums those
|
||||
contributions across the lists, and re-sorts by the total. Documents that rank highly in more than
|
||||
one search rise to the top. Because there's no model to load or call, it's fast and cheap, which is
|
||||
why it's the default reranker for LanceDB hybrid search. The implementation follows the
|
||||
[Cormack et al. paper](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf).
|
||||
|
||||
> **Supported query types:** hybrid search and [multi-vector reranking](/reranking#multi-vector-reranking).
|
||||
> Because RRF fuses two or more ranked lists, it can't rerank a single vector or full-text result set
|
||||
> on its own. Calling `rerank_vector` or `rerank_fts` on an `RRFReranker` raises `NotImplementedError`.
|
||||
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingRrfUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Accepted Arguments {#rrf-accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `K` | `int` | `60` | A constant used in the RRF formula (default is 60). Experiments indicate that k = 60 was near-optimal, but that the choice is not critical. |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. |
|
||||
|
||||
`K` must be greater than `0`. In TypeScript, construct the built-in reranker with
|
||||
`await RRFReranker.create(k)` before passing it to `.rerank(...)`.
|
||||
|
||||
### Multi-vector reranking {#rrf-multi-vector-reranking}
|
||||
|
||||
`RRFReranker` can also fuse the results of several vector searches with `rerank_multivector`, applying
|
||||
the same rank-fusion algorithm across more than two lists. Every input result set must include the
|
||||
`_rowid` column, so add `.with_row_id(True)` to each `table.search(...)` call before reranking,
|
||||
otherwise the call raises a `ValueError`. See [multi-vector reranking](/reranking#multi-vector-reranking)
|
||||
for a full example.
|
||||
|
||||
### Supported Scores for each query type {#rrf-supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
#### Hybrid Search {#rrf-hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Returned rows only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Returned rows have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
## Linear combination {#linear-combination}
|
||||
|
||||
## Linear Combination Reranker
|
||||
|
||||
> **Note:** This reranker is deprecated. Use the `RRFReranker` if you need a score-based reranker.
|
||||
|
||||
The Linear Combination Reranker combines the results of semantic and full-text search using a linear combination of the scores. The weights for the linear combination can be specified, and defaults to 0.7, i.e, 70% weight for semantic search and 30% weight for full-text search.
|
||||
|
||||
> **Note:** Supported query type – Hybrid search only.
|
||||
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingLinearCombinationUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Accepted Arguments {#linear-combination-accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `weight` | `float` | `0.7` | The weight to use for the semantic search score. The weight for the full-text search score is `1 - weight`. |
|
||||
| `fill` | `float` | `1.0` | Score used when a result is missing from one side of the hybrid query. The default strongly penalizes missing vector or FTS matches. |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all", returns all scores from the vector and FTS search along with the relevance score. |
|
||||
|
||||
`weight` must be between `0` and `1`. If either the vector or FTS side returns no rows, the reranker
|
||||
returns the non-empty side with `_relevance_score` attached rather than failing.
|
||||
|
||||
|
||||
### Supported Scores for each query type {#linear-combination-supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
#### Hybrid Search {#linear-combination-hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column |
|
||||
| `all` | ✅ Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_distance`) |
|
||||
|
||||
## MRR {#mrr}
|
||||
|
||||
## MRR Reranker
|
||||
|
||||
This reranker uses the Mean Reciprocal Rank (MRR) algorithm to combine and rerank search results from vector and full-text search. You can use this reranker by passing `MRRReranker()` to the `rerank()` method. The MRR algorithm calculates the average of reciprocal ranks across different search results, providing a balanced way to merge results from multiple ranking systems.
|
||||
|
||||
> **Note:** Supported query types – Hybrid and Multivector search.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingMrrUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Accepted Arguments {#mrr-accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `weight_vector` | `float` | `0.5` | Weight for vector search results (0.0 to 1.0). |
|
||||
| `weight_fts` | `float` | `0.5` | Weight for FTS search results (0.0 to 1.0). |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. |
|
||||
|
||||
**Note:** `weight_vector` + `weight_fts` must equal 1.0.
|
||||
|
||||
For multivector reranking, input result sets need `_rowid` so LanceDB can identify the same row
|
||||
across the ranked lists. Add `.with_row_id(True)` to each vector search before passing the results
|
||||
to the reranker.
|
||||
|
||||
### Supported Scores for each query type {#mrr-supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
#### Hybrid Search {#mrr-hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
#### Multivector Search {#mrr-multivector-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have vector distances from all searches along with `_relevance_score`. |
|
||||
|
||||
## Cross encoder {#cross-encoder}
|
||||
|
||||
## Cross Encoder Reranker
|
||||
|
||||
This reranker uses Cross Encoder models from sentence-transformers to rerank the search results. You can use this reranker by passing `CrossEncoderReranker()` to the `rerank()` method.
|
||||
> **Note:** Supported query types – Hybrid, Vector, and FTS.
|
||||
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingCrossEncoderUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Accepted Arguments {#cross-encoder-accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `model_name` | `str` | `"cross-encoder/ms-marco-TinyBERT-L-6"` | The name of the reranker model to use.|
|
||||
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
|
||||
| `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". |
|
||||
| `trust_remote_code` | `bool` | `True` | Passed to Sentence Transformers model loading. Set this to `False` when you only want to load models that do not require custom repository code. |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all" is supported, returns relevance score along with the vector and/or FTS scores depending on query type. |
|
||||
|
||||
The reranker loads the model locally through `sentence-transformers`, so install the local model
|
||||
runtime dependencies you need, such as PyTorch and any device-specific acceleration packages.
|
||||
|
||||
### Supported Scores for each query type {#cross-encoder-supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
#### Hybrid Search {#cross-encoder-hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
#### Vector Search {#cross-encoder-vector-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
#### FTS Search {#cross-encoder-fts-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
## Building a custom reranker {#custom}
|
||||
|
||||
You can build your own custom reranker in LanceDB by subclassing the base `Reranker` class. At a
|
||||
minimum, you need to implement `rerank_hybrid()`, which is the logic that combines vector and
|
||||
full-text search results. Beyond that, you can optionally implement `rerank_vector()` and
|
||||
`rerank_fts()` if you want your reranker to also handle pure vector or pure full-text searches.
|
||||
|
||||
Decide up front which surfaces — hybrid, pure vector, or pure full-text — your reranker should
|
||||
cover, and only override the ones you need. The base class leaves `rerank_vector()` and
|
||||
`rerank_fts()` unimplemented, so calling `.rerank(...)` on a single-modality search you haven't
|
||||
overridden raises `NotImplementedError` rather than silently returning unsorted results. That's a
|
||||
useful guard, but worth knowing about before you wire up a query path you didn't plan for.
|
||||
|
||||
The Python base class exposes hybrid, vector-only, and FTS-only rerank hooks. TypeScript and Rust
|
||||
currently expose the custom reranker interface for hybrid reranking. In Rust, a custom reranker must
|
||||
also satisfy the trait bounds `Debug + Send + Sync`.
|
||||
|
||||
### Interface {#custom-interface}
|
||||
|
||||
The `Reranker` base interface comes with a `merge_results()` method that can be used to combine the
|
||||
results of semantic and full-text search. This is a vanilla merging algorithm that simply concatenates
|
||||
the results and removes the duplicates without taking the scores into consideration. It only keeps the
|
||||
first copy of the row encountered. This works well in cases that don't require the scores of semantic
|
||||
and full-text search to combine the results. If you want to use the scores or want to support
|
||||
`return_score="all"`, you'll need to implement your own merging algorithm. The base
|
||||
`return_score` option accepts only `"relevance"` and `"all"`.
|
||||
|
||||
Whichever methods you override, your reranker has one job on the way out: attach a
|
||||
`_relevance_score` column with the most relevant rows at the top. LanceDB will reject the result
|
||||
if that column is missing, and downstream `.limit(...)` calls trust the order you return, so
|
||||
sort descending before handing the table back.
|
||||
|
||||
For vector-only reranking in Python, pass a text query to `.rerank(..., query_string="...")`.
|
||||
The vector query itself may be numeric, but `rerank_vector(query, vector_results)` still receives
|
||||
a string query so your reranker can score each candidate against the user's text intent.
|
||||
|
||||
Below, we show the pseudocode of a custom reranker that combines the results of semantic and full-text
|
||||
search using a linear combination of the scores:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
from lancedb.rerankers import Reranker
|
||||
import pyarrow as pa
|
||||
|
||||
class MyReranker(Reranker):
|
||||
def __init__(self, param1, param2, ..., return_score="relevance"):
|
||||
super().__init__(return_score)
|
||||
self.param1 = param1
|
||||
self.param2 = param2
|
||||
|
||||
def rerank_hybrid(self, query: str, vector_results: pa.Table, fts_results: pa.Table):
|
||||
# Use the built-in merging function
|
||||
combined_result = self.merge_results(vector_results, fts_results)
|
||||
|
||||
# Do something with the combined results
|
||||
# ...
|
||||
|
||||
# Return the combined results
|
||||
return combined_result
|
||||
|
||||
def rerank_vector(self, query: str, vector_results: pa.Table):
|
||||
# Do something with the vector results
|
||||
# ...
|
||||
|
||||
# Return the vector results
|
||||
return vector_results
|
||||
|
||||
def rerank_fts(self, query: str, fts_results: pa.Table):
|
||||
# Do something with the FTS results
|
||||
# ...
|
||||
|
||||
# Return the FTS results
|
||||
return fts_results
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Example {#custom-example}
|
||||
|
||||
As an example, let's build custom reranker that enhances the Cohere Reranker by accepting a filter
|
||||
query, and accepts any other `CohereReranker` params as `kwargs`.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
from typing import List, Union
|
||||
import pandas as pd
|
||||
from lancedb.rerankers import CohereReranker
|
||||
|
||||
class ModifiedCohereReranker(CohereReranker):
|
||||
def __init__(self, filters: Union[str, List[str]], **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
filters = filters if isinstance(filters, list) else [filters]
|
||||
self.filters = filters
|
||||
|
||||
def rerank_hybrid(self, query: str, vector_results: pa.Table, fts_results: pa.Table)-> pa.Table:
|
||||
combined_result = super().rerank_hybrid(query, vector_results, fts_results)
|
||||
df = combined_result.to_pandas()
|
||||
for filter in self.filters:
|
||||
df = df.query("not text.str.contains(@filter)")
|
||||
|
||||
return pa.Table.from_pandas(df)
|
||||
|
||||
def rerank_vector(self, query: str, vector_results: pa.Table)-> pa.Table:
|
||||
vector_results = super().rerank_vector(query, vector_results)
|
||||
df = vector_results.to_pandas()
|
||||
for filter in self.filters:
|
||||
df = df.query("not text.str.contains(@filter)")
|
||||
|
||||
return pa.Table.from_pandas(df)
|
||||
|
||||
def rerank_fts(self, query: str, fts_results: pa.Table)-> pa.Table:
|
||||
fts_results = super().rerank_fts(query, fts_results)
|
||||
df = fts_results.to_pandas()
|
||||
for filter in self.filters:
|
||||
df = df.query("not text.str.contains(@filter)")
|
||||
|
||||
return pa.Table.from_pandas(df)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Card icon="lightbulb">
|
||||
Under the hood, `vector_results` and `fts_results` are PyArrow tables. You can learn more about
|
||||
PyArrow tables [here](https://arrow.apache.org/docs/python). The advantage of PyArrow tables is their
|
||||
interoperability -- you can easily convert them to Pandas/Polars DataFrames, `PyDict`, `PyList`, etc.
|
||||
|
||||
The benefits are also bidirectional -- just as you can easily convert PyArrow tables _to_ Pandas
|
||||
DataFrames using the `to_pandas()` method -- you can perform DataFrame transformations
|
||||
and just as easily convert the DataFrame back to PyArrow tables using `pa.Table.from_pandas()` method
|
||||
as shown in the example above.
|
||||
</Card>
|
||||
|
||||
## Evaluating hybrid search {#eval}
|
||||
|
||||
Hybrid search is an often misused and/or misunderstood term. In this section, we're using
|
||||
the definition of "hybrid search" to mean using a combination of keyword-based and vector search.
|
||||
Because the vector search operates in a dense embedding space and keyword-based search operate
|
||||
in a sparse embedding space, their relevance scores cannot be directly compared.
|
||||
Combining results from multiple searches thus requires a reranking step.
|
||||
|
||||
Before evaluating hybrid search, build an FTS index on the text column and use a table with embedding
|
||||
metadata or explicit query vectors for the vector side. Otherwise the evaluation is measuring setup
|
||||
fallbacks or errors rather than the reranker.
|
||||
|
||||
### Reranking strategies {#eval-reranking-strategies}
|
||||
|
||||
There are two common approaches for reranking search results from multiple sources.
|
||||
|
||||
- **Score-based**: Calculate final relevance scores from the individual search algorithm scores. Examples: Reciprocal Rank Fusion (the default in LanceDB), mean reciprocal rank fusion, and weighted linear combination of semantic and keyword-based search scores.
|
||||
|
||||
- **Relevance-based**: Discards the existing scores and calculates the relevance of each search result-query pair. Example: Cross Encoder models
|
||||
|
||||
<Info>
|
||||
If you call `.rerank()` on a hybrid query without passing a reranker, LanceDB defaults to
|
||||
`RRFReranker()` — a score-based reranker that uses Reciprocal Rank Fusion. This is the
|
||||
score-based path most readers encounter first; `LinearCombinationReranker` is an alternative
|
||||
score-based strategy you opt into explicitly.
|
||||
</Info>
|
||||
|
||||
By default, rerankers return `_relevance_score`. Pass `return_score="all"` when a reranker
|
||||
supports it, and you also need the original vector or FTS scores for debugging.
|
||||
Evaluation code can rely on returned rows being ordered by descending `_relevance_score`. Empty
|
||||
reranked result sets still include the `_relevance_score` column.
|
||||
|
||||
The hybrid `rerank(...)` method also accepts a `normalize` argument that controls how the raw
|
||||
vector and FTS scores are made comparable before reranking:
|
||||
|
||||
- `normalize="score"` (the default) — normalizes the raw vector and FTS scores directly.
|
||||
- `normalize="rank"` — converts each result list to ranks first, then normalizes.
|
||||
|
||||
This choice materially affects score-based rerankers (such as `LinearCombinationReranker`), so
|
||||
when you evaluate score-based strategies, treat `normalize` as a tunable hyperparameter
|
||||
alongside the reranker itself.
|
||||
|
||||
For score-based evaluation, the main built-in knobs are:
|
||||
|
||||
- `RRFReranker(K=60)`: rank fusion with a positive smoothing constant.
|
||||
- `MRRReranker(weight_vector=0.5, weight_fts=0.5)`: weighted reciprocal rank fusion where the weights must sum to `1.0`.
|
||||
- `LinearCombinationReranker(weight=0.7, fill=1.0)`: blends vector and FTS scores directly; `fill` controls how strongly to penalize a result missing from one side.
|
||||
|
||||
Even though there are many more strategies for reranking, there are no "universally best"
|
||||
ones that work well for all cases, because reranking quality is dataset and application specific.
|
||||
Evaluating whether a reranking strategy is a good fit is also a challenge. In the next
|
||||
section, we discuss an example evaluation of different reranking strategies on a sample dataset.
|
||||
|
||||
### Example evaluation {#eval-example-evaluation}
|
||||
|
||||
The table below shows our evaluation results from an experiment comparing multiple rerankers on
|
||||
~800 hybrid search queries. This is a modified version of an evaluation script by
|
||||
[LlamaIndex](https://github.com/run-llama/finetune-embedding/blob/main/evaluate.ipynb) that measures
|
||||
hit-rate \@ top-k.
|
||||
|
||||
#### Using OpenAI `text-embedding-ada-002` {#eval-using-openai-text-embedding-ada-002}
|
||||
|
||||
Vector Search baseline: **0.64**
|
||||
|
||||
| Reranker | Top-3 | Top-5 | Top-10 |
|
||||
| --- | --- | --- | --- |
|
||||
| Linear Combination | `0.73` | `0.74` | `0.85` |
|
||||
| Cross Encoder | `0.71` | `0.70` | `0.77` |
|
||||
| Cohere | `0.81` | `0.81` | `0.85` |
|
||||
| ColBERT | `0.68` | `0.68` | `0.73` |
|
||||
|
||||
<img src="https://github.com/AyushExel/assets/assets/15766192/d57b1780-ef27-414c-a5c3-73bee7808a45" />
|
||||
|
||||
|
||||
#### Using OpenAI `text-embedding-3-small` {#eval-using-openai-text-embedding-3-small}
|
||||
|
||||
Vector Search baseline: **0.59**
|
||||
|
||||
| Reranker | Top-3 | Top-5 | Top-10 |
|
||||
| --- | --- | --- | --- |
|
||||
| Linear Combination | `0.68` | `0.70` | `0.84` |
|
||||
| Cross Encoder | `0.72` | `0.72` | `0.79` |
|
||||
| Cohere | `0.79` | `0.79` | `0.84` |
|
||||
| ColBERT | `0.70` | `0.70` | `0.76` |
|
||||
|
||||
<img src="https://github.com/AyushExel/assets/assets/15766192/259adfd2-6ec6-4df6-a77d-1456598970dd" />
|
||||
|
||||
### Conclusion {#eval-conclusion}
|
||||
|
||||
The results show that the reranking methods can significantly improve the search relevance. However,
|
||||
the improvement we saw was not consistent across all rerankers. In reality, the choice of reranker
|
||||
likely depends on the dataset and the application.
|
||||
|
||||
It's also important to note that the reranking methods are not a
|
||||
replacement for the search methods they supplement. They are complementary and it's likely that you'd
|
||||
have to tune them together to get the best results. The latency vs. recall tradeoff is also an
|
||||
important factor to consider when choosing the reranker. Hopefully this evaluation
|
||||
gives you a starting point for your own experiments with hybrid search in LanceDB!
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
---
|
||||
title: Linear Combination Reranker
|
||||
sidebarTitle: "Linear Combination"
|
||||
description: Learn about LanceDB's deprecated Linear Combination Reranker for combining semantic and full-text search scores.
|
||||
|
||||
---
|
||||
|
||||
import { PyRerankingLinearCombinationUsage } from '/snippets/integrations.mdx';
|
||||
|
||||
# Linear Combination Reranker
|
||||
|
||||
> **Note:** This reranker is deprecated. Use the `RRFReranker` if you need a score-based reranker.
|
||||
|
||||
The Linear Combination Reranker combines the results of semantic and full-text search using a linear combination of the scores. The weights for the linear combination can be specified, and defaults to 0.7, i.e, 70% weight for semantic search and 30% weight for full-text search.
|
||||
|
||||
> **Note:** Supported query type – Hybrid search only.
|
||||
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingLinearCombinationUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Accepted Arguments {#accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `weight` | `float` | `0.7` | The weight to use for the semantic search score. The weight for the full-text search score is `1 - weight`. |
|
||||
| `fill` | `float` | `1.0` | Score used when a result is missing from one side of the hybrid query. The default strongly penalizes missing vector or FTS matches. |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all", returns all scores from the vector and FTS search along with the relevance score. |
|
||||
|
||||
`weight` must be between `0` and `1`. If either the vector or FTS side returns no rows, the reranker
|
||||
returns the non-empty side with `_relevance_score` attached rather than failing.
|
||||
|
||||
|
||||
## Supported Scores for each query type {#supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
### Hybrid Search {#hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column |
|
||||
| `all` | ✅ Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_distance`) |
|
||||
@@ -1,48 +0,0 @@
|
||||
---
|
||||
title: MRR Reranker
|
||||
sidebarTitle: "MRR Algorithm"
|
||||
description: Combine and rerank search results using Mean Reciprocal Rank (MRR) algorithm in LanceDB. Supports weighted scoring for hybrid and multivector search.
|
||||
|
||||
---
|
||||
|
||||
import { PyRerankingMrrUsage } from '/snippets/integrations.mdx';
|
||||
|
||||
# MRR Reranker
|
||||
|
||||
This reranker uses the Mean Reciprocal Rank (MRR) algorithm to combine and rerank search results from vector and full-text search. You can use this reranker by passing `MRRReranker()` to the `rerank()` method. The MRR algorithm calculates the average of reciprocal ranks across different search results, providing a balanced way to merge results from multiple ranking systems.
|
||||
|
||||
> **Note:** Supported query types – Hybrid and Multivector search.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingMrrUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Accepted Arguments {#accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `weight_vector` | `float` | `0.5` | Weight for vector search results (0.0 to 1.0). |
|
||||
| `weight_fts` | `float` | `0.5` | Weight for FTS search results (0.0 to 1.0). |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. |
|
||||
|
||||
**Note:** `weight_vector` + `weight_fts` must equal 1.0.
|
||||
|
||||
For multivector reranking, input result sets need `_rowid` so LanceDB can identify the same row
|
||||
across the ranked lists. Add `.with_row_id(True)` to each vector search before passing the results
|
||||
to the reranker.
|
||||
|
||||
## Supported Scores for each query type {#supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
### Hybrid Search {#hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
|
||||
### Multivector Search {#multivector-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Results have vector distances from all searches along with `_relevance_score`. |
|
||||
@@ -1,56 +0,0 @@
|
||||
---
|
||||
title: Reciprocal Rank Fusion Reranker
|
||||
sidebarTitle: "RRF Algorithm"
|
||||
description: Learn about LanceDB's default Reciprocal Rank Fusion (RRF) reranker for hybrid search. Implements the Cormack et al. algorithm for optimal search result ranking.
|
||||
|
||||
---
|
||||
|
||||
import { PyRerankingRrfUsage } from '/snippets/integrations.mdx';
|
||||
|
||||
# Reciprocal Rank Fusion Reranker
|
||||
|
||||
**Reciprocal Rank Fusion (RRF)** is a model-free way to merge several ranked result lists into a
|
||||
single ordering. Rather than comparing raw similarity scores (which aren't directly comparable
|
||||
across, say, a vector search and a full-text search), RRF looks only at each document's *rank
|
||||
position* in each list. It scores every document with the formula `1 / (rank + K)`, sums those
|
||||
contributions across the lists, and re-sorts by the total. Documents that rank highly in more than
|
||||
one search rise to the top. Because there's no model to load or call, it's fast and cheap, which is
|
||||
why it's the default reranker for LanceDB hybrid search. The implementation follows the
|
||||
[Cormack et al. paper](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf).
|
||||
|
||||
> **Supported query types:** hybrid search and [multi-vector reranking](/reranking#multi-vector-reranking).
|
||||
> Because RRF fuses two or more ranked lists, it can't rerank a single vector or full-text result set
|
||||
> on its own. Calling `rerank_vector` or `rerank_fts` on an `RRFReranker` raises `NotImplementedError`.
|
||||
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyRerankingRrfUsage}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Accepted Arguments {#accepted-arguments}
|
||||
| Argument | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `K` | `int` | `60` | A constant used in the RRF formula (default is 60). Experiments indicate that k = 60 was near-optimal, but that the choice is not critical. |
|
||||
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. |
|
||||
|
||||
`K` must be greater than `0`. In TypeScript, construct the built-in reranker with
|
||||
`await RRFReranker.create(k)` before passing it to `.rerank(...)`.
|
||||
|
||||
## Multi-vector reranking {#multi-vector-reranking}
|
||||
|
||||
`RRFReranker` can also fuse the results of several vector searches with `rerank_multivector`, applying
|
||||
the same rank-fusion algorithm across more than two lists. Every input result set must include the
|
||||
`_rowid` column, so add `.with_row_id(True)` to each `table.search(...)` call before reranking,
|
||||
otherwise the call raises a `ValueError`. See [multi-vector reranking](/reranking#multi-vector-reranking)
|
||||
for a full example.
|
||||
|
||||
## Supported Scores for each query type {#supported-scores-for-each-query-type}
|
||||
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
|
||||
|
||||
### Hybrid Search {#hybrid-search}
|
||||
|`return_score`| Status | Description |
|
||||
| --- | --- | --- |
|
||||
| `relevance` | ✅ Supported | Returned rows only have the `_relevance_score` column. |
|
||||
| `all` | ✅ Supported | Returned rows have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
|
||||
@@ -1,314 +0,0 @@
|
||||
---
|
||||
title: "Explain analyze a query plan"
|
||||
sidebarTitle: Explain Analyze
|
||||
description: "Execute a query and read the runtime metrics attached to every operator."
|
||||
icon: "gauge"
|
||||
---
|
||||
|
||||
`analyze_plan` executes the query and returns the physical plan annotated with
|
||||
runtime metrics -- elapsed time, rows produced, bytes read, index comparisons.
|
||||
Where [`explain_plan`](/search/explain) tells you what a query intends to do,
|
||||
this tells you what it cost.
|
||||
|
||||
<Note>
|
||||
The SQL equivalent is [`EXPLAIN ANALYZE`](/sql/explain).
|
||||
</Note>
|
||||
|
||||
## What it measures {#analyze-a-query-plan}
|
||||
|
||||
Executes the query and provides detailed runtime metrics, including:
|
||||
|
||||
- Operation duration (`_elapsed_compute_`)
|
||||
- Data processing statistics (`_output_rows_`, `_bytes_read_`)
|
||||
- Index effectiveness (`_index_comparisons_`, `_indices_loaded_`)
|
||||
- Resource utilization (`_iops_`, `_requests_`)
|
||||
|
||||
## Performance Analysis {#performance-analysis}
|
||||
|
||||
Let's use `analyze_plan` to run the query and analyze the query performance, which will help us identify potential bottlenecks:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
# analyze_plan
|
||||
query_analyze_plan = (
|
||||
table.search(query_embed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.select(["chunk_index", "title", "identifier"])
|
||||
.limit(100)
|
||||
.analyze_plan()
|
||||
)
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
// analyze_plan
|
||||
const analyzePlan = await table
|
||||
.search(queryEmbed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.select(["chunk_index", "title", "identifier"])
|
||||
.limit(100)
|
||||
.analyzePlan();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Performance Metrics Analysis {#performance-metrics-analysis}
|
||||
|
||||
```
|
||||
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance], metrics=[output_rows=100, elapsed_compute=1.424µs]
|
||||
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title", metrics=[output_rows=100, elapsed_compute=175.53097ms, output_batches=1, remote_takes=100]
|
||||
CoalesceBatchesExec: target_batch_size=1024, metrics=[output_rows=100, elapsed_compute=2.748µs]
|
||||
GlobalLimitExec: skip=0, fetch=100, metrics=[output_rows=100, elapsed_compute=1.819µs]
|
||||
FilterExec: _distance@3 IS NOT NULL, metrics=[output_rows=100, elapsed_compute=10.275µs]
|
||||
SortExec: TopK(fetch=100), expr=[_distance@3 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=100, elapsed_compute=39.259451ms, row_replacements=546]
|
||||
KNNVectorDistance: metric=l2, metrics=[output_rows=1099508, elapsed_compute=56.783526ms, output_batches=1076]
|
||||
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000, metrics=[output_rows=1099508, elapsed_compute=17.136819ms]
|
||||
LanceScan: uri=***, projection=[vector, identifier], row_id=true, row_addr=false, ordered=false, metrics=[output_rows=1200000, elapsed_compute=21.348178ms, bytes_read=1852931072, iops=78, requests=78]
|
||||
```
|
||||
|
||||
#### 1\. Data Loading (LanceScan) {#1-data-loading-lancescan}
|
||||
|
||||
- Scanned 1,200,000 rows from the LanceDB table
|
||||
- Read 1.86GB of data in 78 I/O operations
|
||||
- Only loaded necessary columns (`vector` and `identifier`)
|
||||
- Unordered scan for parallel processing
|
||||
|
||||
#### 2\. Filtering and Search {#2-filtering-and-search}
|
||||
|
||||
- Applied prefilter condition (`identifier > 0 AND identifier < 1000000`)
|
||||
- Reduced dataset from 1.2M to 1,099,508 rows
|
||||
- KNN search used L2 (Euclidean) distance metric
|
||||
- Vector comparisons processed in 1076 batches
|
||||
|
||||
#### 3\. Results Processing {#3-results-processing}
|
||||
|
||||
- KNN results sorted by distance (TopK with fetch=100)
|
||||
- Null distances filtered out
|
||||
- Batches coalesced to target size of 1024 rows
|
||||
- Additional columns fetched for final results
|
||||
- Remote take operation for 100 results
|
||||
- Final projection of required columns
|
||||
|
||||
### Distributed metrics on remote tables {#distributed-metrics-on-remote-tables}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
When you call `analyze_plan` against a LanceDB Enterprise table, the query runs across a pool of workers. By default the returned plan aggregates each operator's metrics into a single value, matching the local single-node output. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
|
||||
|
||||
| Mode | What it shows | When to use it |
|
||||
| :------------ | :---------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
|
||||
| `aggregate` | One summary value per operator (default). Same shape as local `analyze_plan`. | Everyday performance checks and comparisons against a local plan. |
|
||||
| `per_worker` | Metrics reported separately for each worker that participated in the query. | Diagnosing stragglers or uneven work distribution across the cluster. |
|
||||
| `full` | Both the aggregate summary and the per-worker breakdown. | Deep investigations where you need the totals *and* the per-worker detail in a single output. |
|
||||
|
||||
The parameter only affects remote plans — local queries always return the aggregate output regardless of what you pass.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
# Show a per-worker breakdown for a remote query
|
||||
plan = (
|
||||
table.search(query_embed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.limit(100)
|
||||
.analyze_plan(distributed_metrics="per_worker")
|
||||
)
|
||||
print(plan)
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
// Show a per-worker breakdown for a remote query
|
||||
const plan = await table
|
||||
.search(queryEmbed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.limit(100)
|
||||
.analyzePlan("per_worker");
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Use `per_worker` or `full` sparingly — the plan output grows with the size of your worker pool. Stick with the default `aggregate` mode for routine tuning.
|
||||
</Note>
|
||||
|
||||
### Key Observations {#key-observations}
|
||||
|
||||
- Vector search is the primary bottleneck (1,099,508 vector comparisons)
|
||||
- Significant I/O overhead (1.86GB data read)
|
||||
- Full table scan due to lack of indices
|
||||
- Substantial optimization potential through proper index implementation
|
||||
|
||||
## Optimized Query Execution {#optimized-query-execution}
|
||||
|
||||
After creating vector and scalar indices, the execution plan shows:
|
||||
|
||||
```
|
||||
ProjectionExec: expr=[chunk_index@3 as chunk_index, title@4 as title, identifier@2 as identifier, _distance@0 as _distance]
|
||||
RemoteTake: columns="_distance, _rowid, identifier, chunk_index, title"
|
||||
CoalesceBatchesExec: target_batch_size=1024
|
||||
GlobalLimitExec: skip=0, fetch=100
|
||||
SortExec: TopK(fetch=100), expr=[_distance@0 ASC NULLS LAST], preserve_partitioning=[false]
|
||||
ANNSubIndex: name=vector_idx, k=100, deltas=1
|
||||
ANNIvfPartition: uuid=83916fd5-fc45-4977-bad9-1f0737539bb9, nprobes=20, deltas=1
|
||||
ScalarIndexQuery: query=AND(identifier > 0,identifier < 1000000)
|
||||
```
|
||||
|
||||
### Optimized Plan Analysis {#optimized-plan-analysis}
|
||||
|
||||
#### 1\. Scalar Index Query {#1-scalar-index-query}
|
||||
|
||||
```
|
||||
ScalarIndexQuery: query=AND(identifier > 0,identifier < 1000000)
|
||||
metrics=[
|
||||
output_rows=2
|
||||
index_comparisons=2,301,824
|
||||
indices_loaded=2
|
||||
output_batches=1
|
||||
parts_loaded=562
|
||||
elapsed_compute=86.979354ms
|
||||
]
|
||||
```
|
||||
|
||||
- Range filter using scalar index
|
||||
- Only 2 index files and 562 scalar index parts loaded
|
||||
- 2.3M index comparisons for matches
|
||||
|
||||
#### 2\. Vector Search {#2-vector-search}
|
||||
|
||||
```
|
||||
ANNSubIndex: name=vector_idx, k=100, deltas=1
|
||||
metrics=[
|
||||
output_rows=2,000
|
||||
index_comparisons=25,893
|
||||
indices_loaded=0
|
||||
output_batches=20
|
||||
parts_loaded=20
|
||||
elapsed_compute=111.849043ms
|
||||
]
|
||||
```
|
||||
|
||||
- IVF index with 20 probes
|
||||
- Only 20 index parts loaded
|
||||
- 25,893 vector comparisons
|
||||
- 2,000 matching vectors
|
||||
|
||||
#### 3\. Results Processing {#3-results-processing-2}
|
||||
|
||||
```
|
||||
SortExec: TopK(fetch=100), expr=[_distance@0 ASC NULLS LAST], preserve_partitioning=[false]
|
||||
GlobalLimitExec: skip=0, fetch=100
|
||||
CoalesceBatchesExec: target_batch_size=1024
|
||||
```
|
||||
|
||||
- Sorts by distance
|
||||
- Limits to top 100 results
|
||||
- Batches into groups of 1024
|
||||
|
||||
#### 4\. Data Fetching {#4-data-fetching}
|
||||
|
||||
```
|
||||
RemoteTake: columns="_distance, _rowid, identifier, chunk_index, title"
|
||||
metrics=[output_rows=100, elapsed_compute=113.491859ms, output_batches=1, remote_takes=100]
|
||||
```
|
||||
|
||||
- Single output batch
|
||||
- One remote take per row
|
||||
|
||||
#### 5\. Final Projection {#5-final-projection}
|
||||
|
||||
```
|
||||
ProjectionExec: expr=[chunk_index@3 as chunk_index, title@4 as title, identifier@2 as identifier, _distance@0 as _distance]
|
||||
```
|
||||
|
||||
- Returns specified columns: chunk_index, title, identifier, and distance
|
||||
|
||||
### Performance Improvements {#performance-improvements}
|
||||
|
||||
#### 1\. Initial Data Access {#1-initial-data-access}
|
||||
|
||||
```
|
||||
ScalarIndexQuery metrics:
|
||||
- indices_loaded=2
|
||||
- parts_loaded=562
|
||||
- output_batches=1
|
||||
```
|
||||
|
||||
- Before: Full table scan of 1.2M rows, 1.86GB data
|
||||
- After: Only 2 indices and 562 scalar index parts loaded
|
||||
- Benefit: Eliminated table scans for prefilter
|
||||
|
||||
#### 2\. Vector Search Efficiency {#2-vector-search-efficiency}
|
||||
|
||||
```
|
||||
ANNSubIndex:
|
||||
- index_comparisons=25,893
|
||||
- indices_loaded=0
|
||||
- parts_loaded=20
|
||||
- output_batches=20
|
||||
```
|
||||
|
||||
- Before: L2 calculations on 1,099,508 vectors
|
||||
- After:
|
||||
- 99.8% reduction in vector comparisons
|
||||
- Decreased output batches from 1,076 to 20
|
||||
|
||||
#### 3\. Data Retrieval Optimization {#3-data-retrieval-optimization}
|
||||
|
||||
```
|
||||
RemoteTake:
|
||||
- remote_takes=100
|
||||
- output_batches=1
|
||||
```
|
||||
|
||||
- RemoteTake operation remains consistent
|
||||
|
||||
## Performance Optimization Guide {#performance-optimization-guide}
|
||||
|
||||
### 1\. Index Implementation {#1-index-implementation}
|
||||
|
||||
#### When to Create Indices {#when-to-create-indices}
|
||||
|
||||
- Columns used in WHERE clauses
|
||||
- Vector columns for similarity searches
|
||||
- Join columns used in `merge_insert`
|
||||
|
||||
#### Index Type Selection {#index-type-selection}
|
||||
|
||||
| Data Type | Recommended Index | Use Case |
|
||||
| ----------- | ------------------ | ---------------------------------------- |
|
||||
| Vector | IVF_PQ/IVF_HNSW_SQ/IVF_HNSW_FLAT | Approximate nearest neighbor search |
|
||||
| Scalar | B-Tree | Range queries and sorting |
|
||||
| Categorical | Bitmap | Multi-value filters and set operations |
|
||||
| `List` | Label_list | Multi-label classification and filtering |
|
||||
|
||||
<Card title="Index Coverage Monitoring" icon="lightbulb">
|
||||
Use `table.index_stats()` to monitor index coverage.
|
||||
A well-optimized table should have `num_unindexed_rows ~ 0`.
|
||||
</Card>
|
||||
|
||||
### 2\. Query Plan Optimization {#2-query-plan-optimization}
|
||||
|
||||
#### Common Patterns and Fixes {#common-patterns-and-fixes}
|
||||
|
||||
| Plan Pattern | Optimization |
|
||||
| ------------------------------------------ | -------------------------------------------- |
|
||||
| LanceScan with high _bytes_read_ or _iops_ | Add missing index |
|
||||
| | Use `select()` to limit returned columns |
|
||||
| | Check whether the dataset has been compacted |
|
||||
| Multiple sequential filters | Reorder filter conditions |
|
||||
|
||||
<Note>
|
||||
Regularly analyze your query plans to identify and address performance
|
||||
bottlenecks. The `analyze_plan` output provides detailed metrics to guide
|
||||
optimization efforts.
|
||||
</Note>
|
||||
|
||||
### 3\. Getting Started with Optimization {#3-getting-started-with-optimization}
|
||||
|
||||
For vector search performance:
|
||||
|
||||
- Create ANN index on your vector column(s) as described in the [index guide](/indexing/vector-index/)
|
||||
- If you often filter by metadata, create [scalar indices](/indexing/scalar-index/) on those columns
|
||||
|
||||
### Hybrid queries {#hybrid-queries}
|
||||
|
||||
For a hybrid query, `analyze_plan` executes both sub-queries and labels them
|
||||
`Vector Search Plan:` and `FTS Search Plan:` in the output. This is the easiest
|
||||
way to see whether the filter pushed into both halves uniformly, and which half
|
||||
is dominating latency.
|
||||
+307
-4
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: "Explain a query plan"
|
||||
sidebarTitle: Explain
|
||||
description: "Read the plan LanceDB resolved for a query, before running it."
|
||||
description: "Read the plan LanceDB resolved for a query, and measure it with analyze."
|
||||
icon: "map"
|
||||
---
|
||||
|
||||
@@ -11,7 +11,7 @@ vector index being used, which columns are actually read -- and costs nothing to
|
||||
run, so it is the first thing to reach for when a query behaves unexpectedly.
|
||||
|
||||
To measure a query rather than read it, use
|
||||
[`analyze_plan`](/search/explain-analyze), which executes it and annotates every
|
||||
[`analyze_plan`](/search/explain), which executes it and annotates every
|
||||
operator with runtime metrics.
|
||||
|
||||
<Note>
|
||||
@@ -197,5 +197,308 @@ print(plan)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Reading the plan tells you the shape. To find out which half dominates latency,
|
||||
[analyze it](/search/explain-analyze#hybrid-queries).
|
||||
Reading the plan tells you the shape. `analyze_plan` executes both sub-queries
|
||||
and labels them `Vector Search Plan:` and `FTS Search Plan:`, which is how you
|
||||
see whether the filter pushed into both halves uniformly and which half
|
||||
dominates latency.
|
||||
|
||||
`analyze_plan` executes the query and returns the physical plan annotated with
|
||||
runtime metrics -- elapsed time, rows produced, bytes read, index comparisons.
|
||||
Where [`explain_plan`](#inspect-a-query-plan) tells you what a query intends to do,
|
||||
this tells you what it cost.
|
||||
|
||||
<Note>
|
||||
The SQL equivalent is [`EXPLAIN ANALYZE`](/sql/explain).
|
||||
</Note>
|
||||
|
||||
## What it measures {#analyze-a-query-plan}
|
||||
|
||||
Executes the query and provides detailed runtime metrics, including:
|
||||
|
||||
- Operation duration (`_elapsed_compute_`)
|
||||
- Data processing statistics (`_output_rows_`, `_bytes_read_`)
|
||||
- Index effectiveness (`_index_comparisons_`, `_indices_loaded_`)
|
||||
- Resource utilization (`_iops_`, `_requests_`)
|
||||
|
||||
## Performance Analysis {#performance-analysis}
|
||||
|
||||
Let's use `analyze_plan` to run the query and analyze the query performance, which will help us identify potential bottlenecks:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
# analyze_plan
|
||||
query_analyze_plan = (
|
||||
table.search(query_embed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.select(["chunk_index", "title", "identifier"])
|
||||
.limit(100)
|
||||
.analyze_plan()
|
||||
)
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
// analyze_plan
|
||||
const analyzePlan = await table
|
||||
.search(queryEmbed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.select(["chunk_index", "title", "identifier"])
|
||||
.limit(100)
|
||||
.analyzePlan();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Performance Metrics Analysis {#performance-metrics-analysis}
|
||||
|
||||
```
|
||||
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance], metrics=[output_rows=100, elapsed_compute=1.424µs]
|
||||
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title", metrics=[output_rows=100, elapsed_compute=175.53097ms, output_batches=1, remote_takes=100]
|
||||
CoalesceBatchesExec: target_batch_size=1024, metrics=[output_rows=100, elapsed_compute=2.748µs]
|
||||
GlobalLimitExec: skip=0, fetch=100, metrics=[output_rows=100, elapsed_compute=1.819µs]
|
||||
FilterExec: _distance@3 IS NOT NULL, metrics=[output_rows=100, elapsed_compute=10.275µs]
|
||||
SortExec: TopK(fetch=100), expr=[_distance@3 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=100, elapsed_compute=39.259451ms, row_replacements=546]
|
||||
KNNVectorDistance: metric=l2, metrics=[output_rows=1099508, elapsed_compute=56.783526ms, output_batches=1076]
|
||||
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000, metrics=[output_rows=1099508, elapsed_compute=17.136819ms]
|
||||
LanceScan: uri=***, projection=[vector, identifier], row_id=true, row_addr=false, ordered=false, metrics=[output_rows=1200000, elapsed_compute=21.348178ms, bytes_read=1852931072, iops=78, requests=78]
|
||||
```
|
||||
|
||||
#### 1\. Data Loading (LanceScan) {#1-data-loading-lancescan}
|
||||
|
||||
- Scanned 1,200,000 rows from the LanceDB table
|
||||
- Read 1.86GB of data in 78 I/O operations
|
||||
- Only loaded necessary columns (`vector` and `identifier`)
|
||||
- Unordered scan for parallel processing
|
||||
|
||||
#### 2\. Filtering and Search {#2-filtering-and-search}
|
||||
|
||||
- Applied prefilter condition (`identifier > 0 AND identifier < 1000000`)
|
||||
- Reduced dataset from 1.2M to 1,099,508 rows
|
||||
- KNN search used L2 (Euclidean) distance metric
|
||||
- Vector comparisons processed in 1076 batches
|
||||
|
||||
#### 3\. Results Processing {#3-results-processing}
|
||||
|
||||
- KNN results sorted by distance (TopK with fetch=100)
|
||||
- Null distances filtered out
|
||||
- Batches coalesced to target size of 1024 rows
|
||||
- Additional columns fetched for final results
|
||||
- Remote take operation for 100 results
|
||||
- Final projection of required columns
|
||||
|
||||
### Distributed metrics on remote tables {#distributed-metrics-on-remote-tables}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
When you call `analyze_plan` against a LanceDB Enterprise table, the query runs across a pool of workers. By default the returned plan aggregates each operator's metrics into a single value, matching the local single-node output. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
|
||||
|
||||
| Mode | What it shows | When to use it |
|
||||
| :------------ | :---------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
|
||||
| `aggregate` | One summary value per operator (default). Same shape as local `analyze_plan`. | Everyday performance checks and comparisons against a local plan. |
|
||||
| `per_worker` | Metrics reported separately for each worker that participated in the query. | Diagnosing stragglers or uneven work distribution across the cluster. |
|
||||
| `full` | Both the aggregate summary and the per-worker breakdown. | Deep investigations where you need the totals *and* the per-worker detail in a single output. |
|
||||
|
||||
The parameter only affects remote plans — local queries always return the aggregate output regardless of what you pass.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
# Show a per-worker breakdown for a remote query
|
||||
plan = (
|
||||
table.search(query_embed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.limit(100)
|
||||
.analyze_plan(distributed_metrics="per_worker")
|
||||
)
|
||||
print(plan)
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
// Show a per-worker breakdown for a remote query
|
||||
const plan = await table
|
||||
.search(queryEmbed)
|
||||
.where("identifier > 0 AND identifier < 1000000")
|
||||
.limit(100)
|
||||
.analyzePlan("per_worker");
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Use `per_worker` or `full` sparingly — the plan output grows with the size of your worker pool. Stick with the default `aggregate` mode for routine tuning.
|
||||
</Note>
|
||||
|
||||
### Key Observations {#key-observations}
|
||||
|
||||
- Vector search is the primary bottleneck (1,099,508 vector comparisons)
|
||||
- Significant I/O overhead (1.86GB data read)
|
||||
- Full table scan due to lack of indices
|
||||
- Substantial optimization potential through proper index implementation
|
||||
|
||||
## Optimized Query Execution {#optimized-query-execution}
|
||||
|
||||
After creating vector and scalar indices, the execution plan shows:
|
||||
|
||||
```
|
||||
ProjectionExec: expr=[chunk_index@3 as chunk_index, title@4 as title, identifier@2 as identifier, _distance@0 as _distance]
|
||||
RemoteTake: columns="_distance, _rowid, identifier, chunk_index, title"
|
||||
CoalesceBatchesExec: target_batch_size=1024
|
||||
GlobalLimitExec: skip=0, fetch=100
|
||||
SortExec: TopK(fetch=100), expr=[_distance@0 ASC NULLS LAST], preserve_partitioning=[false]
|
||||
ANNSubIndex: name=vector_idx, k=100, deltas=1
|
||||
ANNIvfPartition: uuid=83916fd5-fc45-4977-bad9-1f0737539bb9, nprobes=20, deltas=1
|
||||
ScalarIndexQuery: query=AND(identifier > 0,identifier < 1000000)
|
||||
```
|
||||
|
||||
### Optimized Plan Analysis {#optimized-plan-analysis}
|
||||
|
||||
#### 1\. Scalar Index Query {#1-scalar-index-query}
|
||||
|
||||
```
|
||||
ScalarIndexQuery: query=AND(identifier > 0,identifier < 1000000)
|
||||
metrics=[
|
||||
output_rows=2
|
||||
index_comparisons=2,301,824
|
||||
indices_loaded=2
|
||||
output_batches=1
|
||||
parts_loaded=562
|
||||
elapsed_compute=86.979354ms
|
||||
]
|
||||
```
|
||||
|
||||
- Range filter using scalar index
|
||||
- Only 2 index files and 562 scalar index parts loaded
|
||||
- 2.3M index comparisons for matches
|
||||
|
||||
#### 2\. Vector Search {#2-vector-search}
|
||||
|
||||
```
|
||||
ANNSubIndex: name=vector_idx, k=100, deltas=1
|
||||
metrics=[
|
||||
output_rows=2,000
|
||||
index_comparisons=25,893
|
||||
indices_loaded=0
|
||||
output_batches=20
|
||||
parts_loaded=20
|
||||
elapsed_compute=111.849043ms
|
||||
]
|
||||
```
|
||||
|
||||
- IVF index with 20 probes
|
||||
- Only 20 index parts loaded
|
||||
- 25,893 vector comparisons
|
||||
- 2,000 matching vectors
|
||||
|
||||
#### 3\. Results Processing {#3-results-processing-2}
|
||||
|
||||
```
|
||||
SortExec: TopK(fetch=100), expr=[_distance@0 ASC NULLS LAST], preserve_partitioning=[false]
|
||||
GlobalLimitExec: skip=0, fetch=100
|
||||
CoalesceBatchesExec: target_batch_size=1024
|
||||
```
|
||||
|
||||
- Sorts by distance
|
||||
- Limits to top 100 results
|
||||
- Batches into groups of 1024
|
||||
|
||||
#### 4\. Data Fetching {#4-data-fetching}
|
||||
|
||||
```
|
||||
RemoteTake: columns="_distance, _rowid, identifier, chunk_index, title"
|
||||
metrics=[output_rows=100, elapsed_compute=113.491859ms, output_batches=1, remote_takes=100]
|
||||
```
|
||||
|
||||
- Single output batch
|
||||
- One remote take per row
|
||||
|
||||
#### 5\. Final Projection {#5-final-projection}
|
||||
|
||||
```
|
||||
ProjectionExec: expr=[chunk_index@3 as chunk_index, title@4 as title, identifier@2 as identifier, _distance@0 as _distance]
|
||||
```
|
||||
|
||||
- Returns specified columns: chunk_index, title, identifier, and distance
|
||||
|
||||
### Performance Improvements {#performance-improvements}
|
||||
|
||||
#### 1\. Initial Data Access {#1-initial-data-access}
|
||||
|
||||
```
|
||||
ScalarIndexQuery metrics:
|
||||
- indices_loaded=2
|
||||
- parts_loaded=562
|
||||
- output_batches=1
|
||||
```
|
||||
|
||||
- Before: Full table scan of 1.2M rows, 1.86GB data
|
||||
- After: Only 2 indices and 562 scalar index parts loaded
|
||||
- Benefit: Eliminated table scans for prefilter
|
||||
|
||||
#### 2\. Vector Search Efficiency {#2-vector-search-efficiency}
|
||||
|
||||
```
|
||||
ANNSubIndex:
|
||||
- index_comparisons=25,893
|
||||
- indices_loaded=0
|
||||
- parts_loaded=20
|
||||
- output_batches=20
|
||||
```
|
||||
|
||||
- Before: L2 calculations on 1,099,508 vectors
|
||||
- After:
|
||||
- 99.8% reduction in vector comparisons
|
||||
- Decreased output batches from 1,076 to 20
|
||||
|
||||
#### 3\. Data Retrieval Optimization {#3-data-retrieval-optimization}
|
||||
|
||||
```
|
||||
RemoteTake:
|
||||
- remote_takes=100
|
||||
- output_batches=1
|
||||
```
|
||||
|
||||
- RemoteTake operation remains consistent
|
||||
|
||||
## Performance Optimization Guide {#performance-optimization-guide}
|
||||
|
||||
### 1\. Index Implementation {#1-index-implementation}
|
||||
|
||||
#### When to Create Indices {#when-to-create-indices}
|
||||
|
||||
- Columns used in WHERE clauses
|
||||
- Vector columns for similarity searches
|
||||
- Join columns used in `merge_insert`
|
||||
|
||||
#### Index Type Selection {#index-type-selection}
|
||||
|
||||
| Data Type | Recommended Index | Use Case |
|
||||
| ----------- | ------------------ | ---------------------------------------- |
|
||||
| Vector | IVF_PQ/IVF_HNSW_SQ/IVF_HNSW_FLAT | Approximate nearest neighbor search |
|
||||
| Scalar | B-Tree | Range queries and sorting |
|
||||
| Categorical | Bitmap | Multi-value filters and set operations |
|
||||
| `List` | Label_list | Multi-label classification and filtering |
|
||||
|
||||
<Card title="Index Coverage Monitoring" icon="lightbulb">
|
||||
Use `table.index_stats()` to monitor index coverage.
|
||||
A well-optimized table should have `num_unindexed_rows ~ 0`.
|
||||
</Card>
|
||||
|
||||
### 2\. Query Plan Optimization {#2-query-plan-optimization}
|
||||
|
||||
#### Common Patterns and Fixes {#common-patterns-and-fixes}
|
||||
|
||||
| Plan Pattern | Optimization |
|
||||
| ------------------------------------------ | -------------------------------------------- |
|
||||
| LanceScan with high _bytes_read_ or _iops_ | Add missing index |
|
||||
| | Use `select()` to limit returned columns |
|
||||
| | Check whether the dataset has been compacted |
|
||||
| Multiple sequential filters | Reorder filter conditions |
|
||||
|
||||
<Note>
|
||||
Regularly analyze your query plans to identify and address performance
|
||||
bottlenecks. The `analyze_plan` output provides detailed metrics to guide
|
||||
optimization efforts.
|
||||
</Note>
|
||||
|
||||
### 3\. Getting Started with Optimization {#3-getting-started-with-optimization}
|
||||
|
||||
For vector search performance:
|
||||
|
||||
- Create ANN index on your vector column(s) as described in the [index guide](/indexing/vector-index/)
|
||||
- If you often filter by metadata, create [scalar indices](/indexing/scalar-index/) on those columns
|
||||
|
||||
@@ -24,3 +24,65 @@ icon: "list"
|
||||
but spelling it out makes latency and result-count assumptions visible. Query builders also
|
||||
support controls such as prefilter/postfilter, distance ranges, row-id inclusion, offset
|
||||
pagination, and Arrow/Pandas/list result materialization.
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
Three knobs materially affect query latency, memory use, and recall. Be deliberate about each one on every query:
|
||||
|
||||
<Steps>
|
||||
<Step title="Project and limit explicitly">
|
||||
If queries return more data than you need or take longer than expected, the cause is usually projecting more columns than necessary, or letting the result count run unbounded. The best practice is to always pass both `select()` and `limit()`:
|
||||
|
||||
```python Python icon="python"
|
||||
table.search(emb).select(["id", "title"]).limit(20)
|
||||
```
|
||||
</Step>
|
||||
<Step title="Stay on pre-filter unless fewer matches are acceptable">
|
||||
If a query with `prefilter=False` returns fewer than `limit` results — sometimes zero — the cause is that post-filter applies the predicate after the top-k is selected, so only candidates that already passed the search are filtered. Pre-filter is the default and guarantees every result satisfies the predicate. Switch to `prefilter=False` only when fewer-than-`limit` results are acceptable. See [Filtering](/search/filtering).
|
||||
</Step>
|
||||
<Step title="Tune for recall one knob at a time">
|
||||
If recall is lower than expected, the cause is that the search-time knobs limit how many candidates the index considers. Adjust based on your index type:
|
||||
|
||||
- Quantized indexes (PQ, RQ, SQ): `refine_factor` pulls extra candidates and re-scores them on full vectors.
|
||||
- HNSW-backed indexes: `ef` at search time. Start at `1.5 × k`, raise toward `10 × k` if recall is short.
|
||||
- IVF candidate breadth: `nprobes` is auto-tuned; override only when a selective pre-filter leaves too few neighbors.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
For hybrid search, the default `RRFReranker()` combines vector and FTS results into a single ranking via reciprocal rank fusion. See [Hybrid Search](/search/hybrid-search).
|
||||
|
||||
#### Avoid materializing the whole table {#avoid-materializing-the-whole-table}
|
||||
|
||||
If you need every row in the table (for training, export, or migration), calling `to_pandas()` or `to_arrow()` will run you out of memory on any non-trivial dataset — both materialize the full table at once. The best practice is to iterate via `table.search(...)` or `table.query(...)`, which work the same way in both LanceDB OSS and Enterprise.
|
||||
|
||||
In OSS, you can also stream batches through the underlying Lance dataset directly — useful when you need filter pushdown or fragment-level parallelism via `ds.scanner(...)`:
|
||||
|
||||
```python Python icon="python"
|
||||
ds = table.to_lance()
|
||||
for batch in ds.to_batches(columns=["id", "text"], batch_size=10000):
|
||||
process(batch)
|
||||
```
|
||||
|
||||
<Info>
|
||||
**API disparity between OSS and Enterprise.**
|
||||
|
||||
LanceDB OSS exposes the `to_pandas()`, `to_arrow()`, and `table.to_lance()` methods for direct Lance dataset access. Enterprise's `RemoteTable` exposes none of these. Only `table.search(...)` and `table.query(...)`. In general, it's always best to go through `search()` and `query()` to keep your code portable across both OSS and Enterprise.
|
||||
</Info>
|
||||
|
||||
#### Diagnostics {#diagnostics}
|
||||
|
||||
To analyze a slow query, inspect what the query engine actually did and the state of the indexes it touched. These two tools surface that information — use them in this order:
|
||||
|
||||
```python Python icon="python"
|
||||
print(table.search(emb).where("year > 2000").limit(10).analyze_plan())
|
||||
print(table.index_stats("vector_idx")) # num_unindexed_rows should be ~0
|
||||
```
|
||||
|
||||
`analyze_plan()` returns the execution plan with per-stage timings, so you can see where the query actually spent its time. `index_stats()` shows how many rows are still unindexed — you want `num_unindexed_rows` to be ~0. In `analyze_plan`, look for:
|
||||
|
||||
| Plan pattern | Fix |
|
||||
|--------------|-----|
|
||||
| `LanceScan` with high `bytes_read` / `iops` | Add a missing index, project columns with `select()`, or check that the dataset has been compacted |
|
||||
| Multiple sequential filters | Reorder filter conditions |
|
||||
|
||||
[Explain Analyze](/search/explain) walks through a fully worked before/after example, including how `KNNVectorDistance` and `output_batches` change once indexes are in place.
|
||||
|
||||
@@ -1,246 +0,0 @@
|
||||
---
|
||||
title: "Multivector Search"
|
||||
sidebarTitle: Multivector search
|
||||
description: Learn how to perform multivector search in LanceDB to handle multiple vector embeddings per document, which is ideal for late-interaction models like ColBERT and ColPaLi.
|
||||
icon: "braille"
|
||||
---
|
||||
|
||||
LanceDB's multivector support enables you to store and search multiple vector embeddings for a single item.
|
||||
|
||||
This capability is particularly valuable when working with late-interaction models like ColBERT and ColPaLi, which generate multiple embeddings per document.
|
||||
|
||||
In this tutorial, you'll create a table with multiple vector embeddings per document and learn how to perform multivector search. For more end-to-end examples, see the [VectorDB recipes repository](https://github.com/lancedb/vectordb-recipes/tree/main/examples).
|
||||
|
||||
## Multivector Support {#multivector-support}
|
||||
|
||||
Each item in your dataset can have a column containing multiple vectors, which LanceDB can efficiently index and search. When performing a search, you can query with either a single vector embedding or multiple vector embeddings.
|
||||
|
||||
<Warning>
|
||||
Currently, only the `cosine` metric is supported for multivector search. The vector value type can be `float16`, `float32`, or `float64`.
|
||||
</Warning>
|
||||
|
||||
Each query vector must match the inner vector dimension in the multivector column. This applies to both single-vector queries and multi-vector query matrices.
|
||||
|
||||
## Computing Similarity {#computing-similarity}
|
||||
|
||||
MaxSim (Maximum Similarity) is a key concept in late-interaction models that:
|
||||
|
||||
- Computes the maximum similarity between each query embedding and all document embeddings
|
||||
- Sums these maximum similarities to get the final relevance score
|
||||
- Effectively captures fine-grained semantic matches between query and document tokens
|
||||
|
||||
The MaxSim calculation can be expressed as:
|
||||
|
||||
$$
|
||||
\text{MaxSim}(Q, D) = \sum_{i=1}^{|Q|} \max_{j=1}^{|D|} \text{sim}(q_i, d_j)
|
||||
$$
|
||||
|
||||
Where $sim$ is the similarity function (e.g., cosine similarity).
|
||||
|
||||
$$
|
||||
Q = \{q_1, q_2, ..., q_{|Q|}\}
|
||||
$$
|
||||
|
||||
$Q$ represents the query embeddings, and $D = \{d_1, d_2, ..., d_{|D|}\}$ represents the document embeddings.
|
||||
|
||||
## Using Multivector Search {#using-multivector-search}
|
||||
|
||||
### 1\. Setup {#1-setup}
|
||||
|
||||
Connect to LanceDB and import the required libraries.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
import lancedb
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
|
||||
db = lancedb.connect(
|
||||
uri="db://your-project-slug",
|
||||
api_key="your-api-key",
|
||||
region="your-region"
|
||||
)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 2\. Define Schema {#2-define-schema}
|
||||
|
||||
Define a schema that specifies a multivector field. A multivector field is a nested list structure in which each document contains multiple vectors. In this case, we'll create a schema with:
|
||||
|
||||
1. An ID field as an integer (int64)
|
||||
2. A vector field that is a list of lists of float32 values
|
||||
- The outer list represents multiple vectors per document
|
||||
- Each inner list is a 256-dimensional vector
|
||||
- Using float32 for memory efficiency while maintaining precision
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
db = lancedb.connect("data/multivector_demo")
|
||||
schema = pa.schema(
|
||||
[
|
||||
pa.field("id", pa.int64()),
|
||||
# float16, float32, and float64 are supported
|
||||
pa.field("vector", pa.list_(pa.list_(pa.float32(), 256))),
|
||||
]
|
||||
)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 3\. Generate Multivectors {#3-generate-multivectors}
|
||||
|
||||
Generate sample data where each document contains multiple vector embeddings, which can represent different aspects or views of the same document.
|
||||
|
||||
In this example, we create **1024 documents** where each document has **2 random vectors** of **dimension 256**, simulating a real-world scenario where you might have multiple embeddings per item.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
data = [
|
||||
{
|
||||
"id": i,
|
||||
"vector": np.random.random(size=(2, 256)).tolist(), # Each document has 2 vectors
|
||||
}
|
||||
for i in range(1024)
|
||||
]
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 4\. Create a Table {#4-create-a-table}
|
||||
|
||||
Create a table with the defined schema and sample data, which will store multiple vectors per document for similarity search.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
tbl = db.create_table("multivector_example", data=data, schema=schema)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 5\. Build an Index {#5-build-an-index}
|
||||
|
||||
Only cosine similarity is supported for multivector search operations.
|
||||
For faster search, build the standard `IVF_PQ` index over your vectors:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
tbl.create_index(metric="cosine", vector_column_name="vector")
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Info>
|
||||
**Indexing matters more for multivector tables than for single-vector ones.**
|
||||
|
||||
A brute-force scan over a multivector column has to compare every query vector to every document vector in every row, so the cost grows with both the row count and the number of vectors per row.
|
||||
In LanceDB OSS, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.
|
||||
|
||||
On LanceDB Enterprise, the brute-force KNN safety check applies a stricter row threshold to multivector columns — roughly 10× lower than for single-vector columns. So an unindexed multivector table will start being rejected with a "vector search would use brute-force KNN" error well before a comparable single-vector table would. Build the index before you start hitting it from production traffic, even if the dataset is small enough that you'd skip indexing for a single-vector workload.
|
||||
</Info>
|
||||
|
||||
### 6\. Query a Single Vector {#6-query-a-single-vector}
|
||||
|
||||
When searching with a single query vector, it will be compared against all vectors in each document, and the similarity scores will be aggregated to find the most relevant documents.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
query = np.random.random(256)
|
||||
results_single = tbl.search(query).limit(5).to_pandas()
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 7\. Query Multiple Vectors {#7-query-multiple-vectors}
|
||||
|
||||
With multiple query vectors, LanceDB calculates similarity using late interaction, a late-interaction technique that computes relevance by finding the best-matching pairs between query and document vectors. This approach provides more nuanced matching while maintaining fast retrieval speeds.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
query_multi = np.random.random(size=(2, 256))
|
||||
results_multi = tbl.search(query_multi).limit(5).to_pandas()
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
|
||||
Visit the [Hugging Face embedding integration](/integrations/embedding/huggingface/) page for information on embedding models.
|
||||
|
||||
## Simple Example: ColBERT Embeddings {#simple-example-colbert-embeddings}
|
||||
|
||||
[ColBERT](https://arxiv.org/abs/2004.12832) is the most well-known late-interaction retrieval model that
|
||||
represents each document and query as multiple token embeddings and scores matches by taking the best
|
||||
token-to-token similarities (MaxSim) across them.
|
||||
|
||||
Install the dependencies before running this example:
|
||||
|
||||
```bash
|
||||
pip install pylate lancedb pandas
|
||||
```
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
import lancedb
|
||||
from pylate import models
|
||||
|
||||
# 1) Load a late-interaction model via PyLate
|
||||
# PyLate docs show ColBERT() + encode(..., is_query=...) :contentReference[oaicite:2]{index=2}
|
||||
model = models.ColBERT(model_name_or_path="lightonai/GTE-ModernColBERT-v1")
|
||||
|
||||
# You can discover dim from one embedding (avoid guessing)
|
||||
dim = model.encode(["hello"], is_query=True)[0].shape[1]
|
||||
|
||||
# 2) Create a LanceDB table with a multivector column
|
||||
db = lancedb.connect("./pylate_lancedb")
|
||||
schema = pa.schema([
|
||||
pa.field("doc_id", pa.string()),
|
||||
pa.field("text", pa.string()),
|
||||
# multivector: list<list<float32, dim>> :contentReference[oaicite:3]{index=3}
|
||||
pa.field("mv", pa.list_(pa.list_(pa.float32(), dim))),
|
||||
])
|
||||
|
||||
docs = [
|
||||
{"doc_id": "1", "text": "The train to Tokyo leaves at 5pm."},
|
||||
{"doc_id": "2", "text": "That Pho restaurant in Hanoi is highly rated."},
|
||||
{"doc_id": "3", "text": "This is a noodle bar in Osaka, Japan."},
|
||||
]
|
||||
|
||||
# 3) Encode documents with PyLate (token vectors per doc)
|
||||
doc_texts = [d["text"] for d in docs]
|
||||
doc_embs = model.encode(doc_texts, is_query=False) # list/array of (T, dim) per doc :contentReference[oaicite:4]{index=4}
|
||||
|
||||
rows = []
|
||||
for d, emb in zip(docs, doc_embs):
|
||||
emb = np.asarray(emb, dtype=np.float32)
|
||||
rows.append({**d, "mv": emb.tolist()})
|
||||
|
||||
tbl = db.create_table("docs", data=rows, schema=schema, mode="overwrite")
|
||||
|
||||
# 4) Build an index + query using a query matrix.
|
||||
# Multivector brute-force scales with rows × vectors-per-row, so build the index
|
||||
# at much smaller dataset sizes than you would for single-vector search — and
|
||||
# always before exposing the table to remote traffic.
|
||||
tbl.create_index(vector_column_name="mv", metric="cosine")
|
||||
|
||||
query = "Tell me about ramen in Japan"
|
||||
q_emb = np.asarray(model.encode([query], is_query=True)[0], dtype=np.float32) # (Tq, dim) :contentReference[oaicite:5]{index=5}
|
||||
|
||||
out = tbl.search(q_emb).limit(5).to_pandas() # multivector search accepts a matrix :contentReference[oaicite:6]{index=6}
|
||||
print(out[["doc_id", "text"]])
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Late-interaction model implementations evolve rapidly, so it's a good idea to check the latest popular models
|
||||
when trying multivector search.
|
||||
|
||||
## Advanced Example: XTR Embeddings {#advanced-example-xtr-embeddings}
|
||||
|
||||
[ConteXtualized Token Retriever (XTR)](https://arxiv.org/abs/2304.01982) is a late-interaction retrieval model that represents text as token-level vectors instead of a single embedding.
|
||||
This lets search score token-to-token matches (MaxSim), which can improve fine-grained relevance.
|
||||
|
||||
The notebook linked below shows how to integrate XTR, which prioritizes critical document
|
||||
tokens during the initial retrieval stage and removes the gathering stage to improve performance significantly.
|
||||
By focusing on the most semantically salient tokens early in the process, XTR reduces computational complexity
|
||||
while improving recall and ensuring rapid identification of candidate documents.
|
||||
|
||||
<Card
|
||||
icon="book"
|
||||
title="Multivector search with ColPali + XTR embeddings and LanceDB"
|
||||
href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multivector_xtr/main.ipynb"
|
||||
>
|
||||
</Card>
|
||||
@@ -4,6 +4,7 @@ sidebarTitle: Vector search
|
||||
description: "Learn how to run vector search queries in LanceDB. Includes best practices, tips and examples."
|
||||
icon: "arrow-up-right-dots"
|
||||
---
|
||||
|
||||
import {
|
||||
PyConfigureDistanceMetric as ConfigureDistanceMetric,
|
||||
PySelectVectorColumn as SelectVectorColumn,
|
||||
@@ -455,3 +456,243 @@ This approach is particularly useful when:
|
||||
- Evaluating ANN index quality
|
||||
- Calculating recall metrics to tune index parameters
|
||||
- Ensuring exact results for critical applications
|
||||
|
||||
LanceDB's multivector support enables you to store and search multiple vector embeddings for a single item.
|
||||
|
||||
This capability is particularly valuable when working with late-interaction models like ColBERT and ColPaLi, which generate multiple embeddings per document.
|
||||
|
||||
In this tutorial, you'll create a table with multiple vector embeddings per document and learn how to perform multivector search. For more end-to-end examples, see the [VectorDB recipes repository](https://github.com/lancedb/vectordb-recipes/tree/main/examples).
|
||||
|
||||
## Multivector Support {#multivector-support}
|
||||
|
||||
Each item in your dataset can have a column containing multiple vectors, which LanceDB can efficiently index and search. When performing a search, you can query with either a single vector embedding or multiple vector embeddings.
|
||||
|
||||
<Warning>
|
||||
Currently, only the `cosine` metric is supported for multivector search. The vector value type can be `float16`, `float32`, or `float64`.
|
||||
</Warning>
|
||||
|
||||
Each query vector must match the inner vector dimension in the multivector column. This applies to both single-vector queries and multi-vector query matrices.
|
||||
|
||||
## Computing Similarity {#computing-similarity}
|
||||
|
||||
MaxSim (Maximum Similarity) is a key concept in late-interaction models that:
|
||||
|
||||
- Computes the maximum similarity between each query embedding and all document embeddings
|
||||
- Sums these maximum similarities to get the final relevance score
|
||||
- Effectively captures fine-grained semantic matches between query and document tokens
|
||||
|
||||
The MaxSim calculation can be expressed as:
|
||||
|
||||
$$
|
||||
\text{MaxSim}(Q, D) = \sum_{i=1}^{|Q|} \max_{j=1}^{|D|} \text{sim}(q_i, d_j)
|
||||
$$
|
||||
|
||||
Where $sim$ is the similarity function (e.g., cosine similarity).
|
||||
|
||||
$$
|
||||
Q = \{q_1, q_2, ..., q_{|Q|}\}
|
||||
$$
|
||||
|
||||
$Q$ represents the query embeddings, and $D = \{d_1, d_2, ..., d_{|D|}\}$ represents the document embeddings.
|
||||
|
||||
## Using Multivector Search {#using-multivector-search}
|
||||
|
||||
### 1\. Setup {#1-setup}
|
||||
|
||||
Connect to LanceDB and import the required libraries.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
import lancedb
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
|
||||
db = lancedb.connect(
|
||||
uri="db://your-project-slug",
|
||||
api_key="your-api-key",
|
||||
region="your-region"
|
||||
)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 2\. Define Schema {#2-define-schema}
|
||||
|
||||
Define a schema that specifies a multivector field. A multivector field is a nested list structure in which each document contains multiple vectors. In this case, we'll create a schema with:
|
||||
|
||||
1. An ID field as an integer (int64)
|
||||
2. A vector field that is a list of lists of float32 values
|
||||
- The outer list represents multiple vectors per document
|
||||
- Each inner list is a 256-dimensional vector
|
||||
- Using float32 for memory efficiency while maintaining precision
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
db = lancedb.connect("data/multivector_demo")
|
||||
schema = pa.schema(
|
||||
[
|
||||
pa.field("id", pa.int64()),
|
||||
# float16, float32, and float64 are supported
|
||||
pa.field("vector", pa.list_(pa.list_(pa.float32(), 256))),
|
||||
]
|
||||
)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 3\. Generate Multivectors {#3-generate-multivectors}
|
||||
|
||||
Generate sample data where each document contains multiple vector embeddings, which can represent different aspects or views of the same document.
|
||||
|
||||
In this example, we create **1024 documents** where each document has **2 random vectors** of **dimension 256**, simulating a real-world scenario where you might have multiple embeddings per item.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
data = [
|
||||
{
|
||||
"id": i,
|
||||
"vector": np.random.random(size=(2, 256)).tolist(), # Each document has 2 vectors
|
||||
}
|
||||
for i in range(1024)
|
||||
]
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 4\. Create a Table {#4-create-a-table}
|
||||
|
||||
Create a table with the defined schema and sample data, which will store multiple vectors per document for similarity search.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
tbl = db.create_table("multivector_example", data=data, schema=schema)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 5\. Build an Index {#5-build-an-index}
|
||||
|
||||
Only cosine similarity is supported for multivector search operations.
|
||||
For faster search, build the standard `IVF_PQ` index over your vectors:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
tbl.create_index(metric="cosine", vector_column_name="vector")
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Info>
|
||||
**Indexing matters more for multivector tables than for single-vector ones.**
|
||||
|
||||
A brute-force scan over a multivector column has to compare every query vector to every document vector in every row, so the cost grows with both the row count and the number of vectors per row.
|
||||
In LanceDB OSS, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.
|
||||
|
||||
On LanceDB Enterprise, the brute-force KNN safety check applies a stricter row threshold to multivector columns — roughly 10× lower than for single-vector columns. So an unindexed multivector table will start being rejected with a "vector search would use brute-force KNN" error well before a comparable single-vector table would. Build the index before you start hitting it from production traffic, even if the dataset is small enough that you'd skip indexing for a single-vector workload.
|
||||
</Info>
|
||||
|
||||
### 6\. Query a Single Vector {#6-query-a-single-vector}
|
||||
|
||||
When searching with a single query vector, it will be compared against all vectors in each document, and the similarity scores will be aggregated to find the most relevant documents.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
query = np.random.random(256)
|
||||
results_single = tbl.search(query).limit(5).to_pandas()
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 7\. Query Multiple Vectors {#7-query-multiple-vectors}
|
||||
|
||||
With multiple query vectors, LanceDB calculates similarity using late interaction, a late-interaction technique that computes relevance by finding the best-matching pairs between query and document vectors. This approach provides more nuanced matching while maintaining fast retrieval speeds.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
query_multi = np.random.random(size=(2, 256))
|
||||
results_multi = tbl.search(query_multi).limit(5).to_pandas()
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
|
||||
Visit the [Hugging Face embedding integration](/integrations/embedding/huggingface/) page for information on embedding models.
|
||||
|
||||
## Simple Example: ColBERT Embeddings {#simple-example-colbert-embeddings}
|
||||
|
||||
[ColBERT](https://arxiv.org/abs/2004.12832) is the most well-known late-interaction retrieval model that
|
||||
represents each document and query as multiple token embeddings and scores matches by taking the best
|
||||
token-to-token similarities (MaxSim) across them.
|
||||
|
||||
Install the dependencies before running this example:
|
||||
|
||||
```bash
|
||||
pip install pylate lancedb pandas
|
||||
```
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
import lancedb
|
||||
from pylate import models
|
||||
|
||||
# 1) Load a late-interaction model via PyLate
|
||||
# PyLate docs show ColBERT() + encode(..., is_query=...) :contentReference[oaicite:2]{index=2}
|
||||
model = models.ColBERT(model_name_or_path="lightonai/GTE-ModernColBERT-v1")
|
||||
|
||||
# You can discover dim from one embedding (avoid guessing)
|
||||
dim = model.encode(["hello"], is_query=True)[0].shape[1]
|
||||
|
||||
# 2) Create a LanceDB table with a multivector column
|
||||
db = lancedb.connect("./pylate_lancedb")
|
||||
schema = pa.schema([
|
||||
pa.field("doc_id", pa.string()),
|
||||
pa.field("text", pa.string()),
|
||||
# multivector: list<list<float32, dim>> :contentReference[oaicite:3]{index=3}
|
||||
pa.field("mv", pa.list_(pa.list_(pa.float32(), dim))),
|
||||
])
|
||||
|
||||
docs = [
|
||||
{"doc_id": "1", "text": "The train to Tokyo leaves at 5pm."},
|
||||
{"doc_id": "2", "text": "That Pho restaurant in Hanoi is highly rated."},
|
||||
{"doc_id": "3", "text": "This is a noodle bar in Osaka, Japan."},
|
||||
]
|
||||
|
||||
# 3) Encode documents with PyLate (token vectors per doc)
|
||||
doc_texts = [d["text"] for d in docs]
|
||||
doc_embs = model.encode(doc_texts, is_query=False) # list/array of (T, dim) per doc :contentReference[oaicite:4]{index=4}
|
||||
|
||||
rows = []
|
||||
for d, emb in zip(docs, doc_embs):
|
||||
emb = np.asarray(emb, dtype=np.float32)
|
||||
rows.append({**d, "mv": emb.tolist()})
|
||||
|
||||
tbl = db.create_table("docs", data=rows, schema=schema, mode="overwrite")
|
||||
|
||||
# 4) Build an index + query using a query matrix.
|
||||
# Multivector brute-force scales with rows × vectors-per-row, so build the index
|
||||
# at much smaller dataset sizes than you would for single-vector search — and
|
||||
# always before exposing the table to remote traffic.
|
||||
tbl.create_index(vector_column_name="mv", metric="cosine")
|
||||
|
||||
query = "Tell me about ramen in Japan"
|
||||
q_emb = np.asarray(model.encode([query], is_query=True)[0], dtype=np.float32) # (Tq, dim) :contentReference[oaicite:5]{index=5}
|
||||
|
||||
out = tbl.search(q_emb).limit(5).to_pandas() # multivector search accepts a matrix :contentReference[oaicite:6]{index=6}
|
||||
print(out[["doc_id", "text"]])
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Late-interaction model implementations evolve rapidly, so it's a good idea to check the latest popular models
|
||||
when trying multivector search.
|
||||
|
||||
## Advanced Example: XTR Embeddings {#advanced-example-xtr-embeddings}
|
||||
|
||||
[ConteXtualized Token Retriever (XTR)](https://arxiv.org/abs/2304.01982) is a late-interaction retrieval model that represents text as token-level vectors instead of a single embedding.
|
||||
This lets search score token-to-token matches (MaxSim), which can improve fine-grained relevance.
|
||||
|
||||
The notebook linked below shows how to integrate XTR, which prioritizes critical document
|
||||
tokens during the initial retrieval stage and removes the gathering stage to improve performance significantly.
|
||||
By focusing on the most semantically salient tokens early in the process, XTR reduces computational complexity
|
||||
while improving recall and ensuring rapid identification of candidate documents.
|
||||
|
||||
<Card
|
||||
icon="book"
|
||||
title="Multivector search with ColPali + XTR embeddings and LanceDB"
|
||||
href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multivector_xtr/main.ipynb"
|
||||
>
|
||||
</Card>
|
||||
|
||||
@@ -13,9 +13,32 @@ DataFusion the dataset's Arrow schema directly, so a column is the same object
|
||||
to the query engine as it is to the format.
|
||||
|
||||
The consequence is that SQL type names here are **spellings of Arrow types**,
|
||||
not a system that maps onto them. [Data types](/basics/data-types) documents the
|
||||
not a system that maps onto them. [Data types](/sql/data-types) documents the
|
||||
storage side; this page covers what changes when you reach a column from SQL.
|
||||
|
||||
## The Arrow types a column can be {#supported}
|
||||
|
||||
| Category | Types |
|
||||
| --- | --- |
|
||||
| Null | `null` |
|
||||
| Boolean | `bool` |
|
||||
| Integer | `int8`, `int16`, `int32`, `int64` and their `uint` counterparts |
|
||||
| Floating point | `float16`, `float32`, `float64` |
|
||||
| String | `utf8`, `large_utf8` |
|
||||
| Binary | `binary`, `large_binary`, `fixed_size_binary(n)` |
|
||||
| Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp` (with or without timezone), `duration` |
|
||||
| Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` |
|
||||
| Dictionary | `dictionary(key, value)` |
|
||||
|
||||
Two limits are worth stating because they surface as schema errors rather than
|
||||
as documentation:
|
||||
|
||||
- A `map` with `keys_sorted = true` is rejected. Sorted-key maps do not survive
|
||||
the round trip through the Python bindings, so they are refused rather than
|
||||
silently unsorted.
|
||||
- Not every type is legal as a `fixed_size_list` child. The vector case is,
|
||||
which is what matters in practice.
|
||||
|
||||
## SQL names for Arrow types {#names}
|
||||
|
||||
| SQL | Arrow |
|
||||
@@ -79,6 +102,26 @@ The default reads blob columns as descriptions and other binary columns as
|
||||
binary. A query selecting a blob column and expecting bytes gets a struct.
|
||||
</Warning>
|
||||
|
||||
### bfloat16 {#bfloat16}
|
||||
|
||||
Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`.
|
||||
Useful as a vector element type when halving the memory matters more than the
|
||||
precision does.
|
||||
|
||||
## Geospatial {#geospatial}
|
||||
|
||||
<Warning>
|
||||
Lance defines geospatial types and `st_*` functions on
|
||||
[GeoArrow](https://geoarrow.org), but they sit behind an optional build feature
|
||||
that **LanceDB does not ship**. Calling `st_intersects` and friends returns an
|
||||
error naming the missing feature rather than an unknown-function error, and the
|
||||
spatial index is gated the same way.
|
||||
</Warning>
|
||||
|
||||
Geometry can still be stored -- it is binary, and binary columns work -- but
|
||||
nothing interprets it. Filter on a bounding box held in ordinary numeric
|
||||
columns if you need spatial predicates today.
|
||||
|
||||
## Pseudo-columns {#pseudo-columns}
|
||||
|
||||
Row metadata, available only when named — so it stays out of `SELECT *`:
|
||||
|
||||
@@ -11,7 +11,7 @@ keywords: ["insert", "update", "delete", "copy", "dml"]
|
||||
Statements that change rows. For the shape of the table, see [DDL](/sql/ddl).
|
||||
|
||||
Each of these commits a new table version, so
|
||||
[versioning](/tables/versioning) and the
|
||||
[versioning](/management/table#versioning) and the
|
||||
[write guarantees](/basics/guarantees#writes) apply -- including that the
|
||||
version advances even when a statement affects zero rows.
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ explain how to read them:
|
||||
<Card title="Explain" icon="map" href="/search/explain">
|
||||
Reading a plan: operators, pushdown, index usage.
|
||||
</Card>
|
||||
<Card title="Explain Analyze" icon="gauge" href="/search/explain-analyze">
|
||||
<Card title="Explain Analyze" icon="gauge" href="/search/explain">
|
||||
Reading the metrics, and a worked before/after.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -132,4 +132,4 @@ same operators the SDK methods do:
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
See [Explain](/search/explain) and [Explain Analyze](/search/explain-analyze).
|
||||
See [Explain](/search/explain) and [Explain Analyze](/search/explain).
|
||||
|
||||
@@ -96,5 +96,5 @@ read.
|
||||
<Note>
|
||||
Window functions sort within each partition, so on a large table they are among
|
||||
the more expensive things you can ask for. Filter first, and check the cost with
|
||||
[Explain Analyze](/search/explain-analyze).
|
||||
[Explain Analyze](/search/explain).
|
||||
</Note>
|
||||
|
||||
@@ -105,7 +105,7 @@ A few starting points for dashboards and alerts:
|
||||
## Where to go next {#where-to-go-next}
|
||||
|
||||
<Columns cols={2}>
|
||||
<Card title="Performance tips" icon="gauge-high" href="/performance">
|
||||
<Card title="Performance tips" icon="gauge-high" href="/search#performance">
|
||||
Tune ingestion, indexing, and query patterns once metrics highlight a hot spot.
|
||||
</Card>
|
||||
<Card title="Storage configuration" icon="wrench" href="/storage/configuration">
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Branches"
|
||||
sidebarTitle: "Branches"
|
||||
sidebarTitle: "Branch"
|
||||
description: "Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main."
|
||||
icon: "code-branch"
|
||||
---
|
||||
@@ -46,8 +46,8 @@ reads exactly as before. Branches are a natural fit when you want to:
|
||||
|
||||
## How branches relate to versions and tags {#how-branches-relate-to-versions-and-tags}
|
||||
|
||||
Every LanceDB table already tracks a linear history of [versions](/tables/versioning),
|
||||
and you can [tag](/tables/versioning#tag-based-versioning) a version or `checkout`
|
||||
Every LanceDB table already tracks a linear history of [versions](/management/table#versioning),
|
||||
and you can [tag](/management/table#versioning-tag-based-versioning) a version or `checkout`
|
||||
one to read it. Branches add the missing piece: a *separate, writable* line of
|
||||
history. Where a tag is a read-only label and `checkout` is a read-only view, a
|
||||
branch forks from a point in history and then evolves on its own. Creating or
|
||||
@@ -394,10 +394,10 @@ give you to work with table history. Choose these approaches based on whether yo
|
||||
| Feature | Writable? | Purpose |
|
||||
| --- | --- | --- |
|
||||
| **[Branch](#work-with-branches)** | ✅ Yes | Write on top of a point in history without touching `main`. |
|
||||
| **[Tag](/tables/versioning#tag-based-versioning)** | ❌ No | Attach a human-readable label to an existing version; protects it from cleanup. |
|
||||
| **[`checkout(version)`](/tables/versioning#rollback-to-previous-versions)** | ❌ No | Read a historical version of `main` without forking. Read-only until you `restore`. |
|
||||
| **[Tag](/management/table#versioning-tag-based-versioning)** | ❌ No | Attach a human-readable label to an existing version; protects it from cleanup. |
|
||||
| **[`checkout(version)`](/management/table#versioning-rollback-to-previous-versions)** | ❌ No | Read a historical version of `main` without forking. Read-only until you `restore`. |
|
||||
|
||||
<Note>
|
||||
For linear version history — creating versions, listing them, rolling back, and
|
||||
tagging — see the [Versioning and Reproducibility](/tables/versioning) guide.
|
||||
tagging — see the [Versioning and Reproducibility](/management/table#versioning) guide.
|
||||
</Note>
|
||||
|
||||
@@ -125,7 +125,7 @@ To manually check for updates, use `checkout_latest` / `checkoutLatest`:
|
||||
For reproducible reads, you can also pin a table to a specific snapshot with `checkout(...)` or
|
||||
a tag, restore a table to a prior version, then return to the live table with
|
||||
`checkout_latest` / `checkoutLatest`. See
|
||||
[Versioning](/tables/versioning/) for the full version and tag workflow.
|
||||
[Versioning](/management/table#versioning) for the full version and tag workflow.
|
||||
|
||||
## Handle bad vectors {#handle-bad-vectors}
|
||||
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
---
|
||||
title: "Schema and data types"
|
||||
sidebarTitle: "Data Types"
|
||||
description: "The Arrow type system LanceDB stores, and the extension types Lance adds on top of it."
|
||||
icon: shapes
|
||||
keywords: ["schema", "data types", "arrow", "json", "blob", "bfloat16", "vector", "geospatial"]
|
||||
title: "Multimodal data"
|
||||
sidebarTitle: "Multimodal Data"
|
||||
description: "Storing images, audio and video in a table, as binary columns or blobs."
|
||||
icon: photo-film
|
||||
keywords: ["multimodal", "blob", "binary", "images", "video", "blob api"]
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -35,115 +35,19 @@ import {
|
||||
PyQueryToPandasKwargs as QueryToPandasKwargs,
|
||||
} from '/snippets/multimodal.mdx';
|
||||
|
||||
A table's schema is an **Arrow schema**. There is no separate LanceDB type
|
||||
system mapped onto storage -- a column is the same object to the query engine,
|
||||
to the format, and to any other engine reading the files.
|
||||
Binary data -- images, audio, video, model artifacts -- lives in the table
|
||||
beside the structured columns and embeddings that describe it, and is governed
|
||||
by the same versioning and search.
|
||||
|
||||
That identity is mostly invisible and occasionally load-bearing, so it is worth
|
||||
knowing what it admits.
|
||||
Small values go in an ordinary `binary` column. Large ones go in a **blob**
|
||||
column, stored out of line so a scan over the other columns stays fast. See
|
||||
[Data types](/sql/data-types#blobs) for how the two differ to a reader.
|
||||
|
||||
## Supported types {#supported}
|
||||
|
||||
| Category | Types |
|
||||
| --- | --- |
|
||||
| Null | `null` |
|
||||
| Boolean | `bool` |
|
||||
| Integer | `int8`, `int16`, `int32`, `int64` and their `uint` counterparts |
|
||||
| Floating point | `float16`, `float32`, `float64` |
|
||||
| String | `utf8`, `large_utf8` |
|
||||
| Binary | `binary`, `large_binary`, `fixed_size_binary(n)` |
|
||||
| Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp` (with or without timezone), `duration` |
|
||||
| Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` |
|
||||
| Dictionary | `dictionary(key, value)` |
|
||||
|
||||
Two limits are worth stating because they surface as schema errors rather than
|
||||
as documentation:
|
||||
|
||||
- A `map` with `keys_sorted = true` is rejected. Sorted-key maps do not survive
|
||||
the round trip through the Python bindings, so they are refused rather than
|
||||
silently unsorted.
|
||||
- Not every type is legal as a `fixed_size_list` child. The vector case is,
|
||||
which is what matters in practice.
|
||||
|
||||
## Vectors are fixed-size lists {#vectors}
|
||||
|
||||
A vector column is a `fixed_size_list<float32, dim>` -- an ordinary Arrow type,
|
||||
not a special one:
|
||||
|
||||
```python
|
||||
import pyarrow as pa
|
||||
|
||||
schema = pa.schema([
|
||||
pa.field("id", pa.int64()),
|
||||
pa.field("vector", pa.list_(pa.float32(), 1536)),
|
||||
])
|
||||
```
|
||||
|
||||
Because it is ordinary, it is selectable, testable for nullness, and joinable
|
||||
like any other column. The consequence people trip on is the other side of that:
|
||||
arithmetic over it is **list** arithmetic, not scalar arithmetic. When the
|
||||
operation you want is a similarity comparison, use
|
||||
[vector search](/search/vector-search) rather than SQL.
|
||||
|
||||
Without an explicit schema, Python infers a list of floats as a fixed-size
|
||||
`float32` vector from the dimension it observes in the data.
|
||||
|
||||
## Extension types {#extensions}
|
||||
|
||||
Lance adds a few types as **Arrow extension types** -- an underlying storage
|
||||
type plus an `ARROW:extension:name` marker. Any Arrow reader can read the
|
||||
storage; readers that know the extension get the richer behaviour.
|
||||
|
||||
| Type | Extension name | Stored as |
|
||||
| --- | --- | --- |
|
||||
| [JSON](#json) | `lance.json` | `large_binary`, JSONB-encoded |
|
||||
| [Blob](#blob) | `lance.blob.v2` | `large_binary`, stored out of line |
|
||||
| bfloat16 | `lance.bfloat16` | `fixed_size_binary(2)` |
|
||||
|
||||
`arrow.json` is accepted as well, for interoperability with writers that use
|
||||
Arrow's own canonical JSON extension name.
|
||||
|
||||
### JSON {#json}
|
||||
|
||||
A JSON column stores documents in **JSONB**, a binary encoding, rather than as
|
||||
text. Accessors read into a document without parsing the whole thing:
|
||||
|
||||
```sql
|
||||
SELECT json_get_string(data, 'name') FROM events
|
||||
WHERE json_get_int(data, 'status') = 200
|
||||
```
|
||||
|
||||
`json_get` returns JSON, so accessors nest for a nested document. **A JSON
|
||||
scalar index accelerates `json_get`, its typed variants, and `json_extract`** --
|
||||
the rest evaluate per row. The full function list is in
|
||||
[SQL functions](/sql/scalar-functions#json).
|
||||
|
||||
This is the type to reach for when a column's shape varies per row. When it does
|
||||
not, ordinary `struct` columns are faster and typed.
|
||||
|
||||
### Blob {#blob}
|
||||
|
||||
A blob column holds large binary objects -- images, video, audio, model
|
||||
artifacts -- **out of line** from the rest of the row. That is what keeps a scan
|
||||
over the other columns fast on a table whose rows are megabytes each.
|
||||
|
||||
A blob can be read two ways, and the choice changes the column's Arrow type:
|
||||
|
||||
- as its **binary contents**, or
|
||||
- as a **description** -- a pointer plus metadata, without fetching the bytes.
|
||||
|
||||
<Warning>
|
||||
The default reads blob columns as descriptions and other binary columns as
|
||||
binary. A query that selects a blob column expecting bytes gets a struct
|
||||
instead, unless the read is configured otherwise. Worth knowing before writing a
|
||||
predicate against one.
|
||||
</Warning>
|
||||
|
||||
### Store binary data {#store-binary-data}
|
||||
## Store binary data {#store-binary-data}
|
||||
|
||||
To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust).
|
||||
|
||||
#### 1\. Setup and imports {#1-setup-and-imports}
|
||||
### 1\. Setup and imports {#1-setup-and-imports}
|
||||
|
||||
First, import the necessary libraries for LanceDB and Arrow in your SDK.
|
||||
|
||||
@@ -161,7 +65,7 @@ First, import the necessary libraries for LanceDB and Arrow in your SDK.
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 2\. Prepare data {#2-prepare-data}
|
||||
### 2\. Prepare data {#2-prepare-data}
|
||||
|
||||
For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object.
|
||||
|
||||
@@ -179,7 +83,7 @@ For this example, we'll create some dummy in-memory images. In a real applicatio
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 3\. Define the schema {#3-define-the-schema}
|
||||
### 3\. Define the schema {#3-define-the-schema}
|
||||
|
||||
When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list.
|
||||
|
||||
@@ -197,7 +101,7 @@ When creating the table, it is **highly recommended** to define the schema expli
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 4\. Ingest data {#4-ingest-data}
|
||||
### 4\. Ingest data {#4-ingest-data}
|
||||
|
||||
Now, create the table using the data and the defined schema.
|
||||
|
||||
@@ -215,7 +119,7 @@ Now, create the table using the data and the defined schema.
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Retrieve and use blobs {#retrieve-and-use-blobs}
|
||||
## Retrieve and use blobs {#retrieve-and-use-blobs}
|
||||
|
||||
When you search your LanceDB table, you can retrieve the binary column just like any other metadata.
|
||||
|
||||
@@ -233,7 +137,7 @@ When you search your LanceDB table, you can retrieve the binary column just like
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Convert bytes back to objects {#convert-bytes-back-to-objects}
|
||||
### Convert bytes back to objects {#convert-bytes-back-to-objects}
|
||||
|
||||
Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer).
|
||||
|
||||
@@ -251,11 +155,11 @@ Once you have the bytes back from the search result, you can decode them into th
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Large Blobs (Blob API) {#large-blobs-blob-api}
|
||||
## Large Blobs (Blob API) {#large-blobs-blob-api}
|
||||
|
||||
For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.
|
||||
|
||||
#### 1\. Define a blob schema {#1-define-a-blob-schema}
|
||||
### 1\. Define a blob schema {#1-define-a-blob-schema}
|
||||
|
||||
To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata.
|
||||
|
||||
@@ -273,7 +177,7 @@ To use the Blob API, you must mark the column with `{"lance-encoding:blob": "tru
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 2\. Ingest large blobs {#2-ingest-large-blobs}
|
||||
### 2\. Ingest large blobs {#2-ingest-large-blobs}
|
||||
|
||||
You can then ingest data normally, and Lance will handle the optimized storage.
|
||||
|
||||
@@ -296,7 +200,7 @@ For more advanced usage, including random access and file-like reading of blobs,
|
||||
Lance format's [blob API documentation](https://lance.org/guide/blob/).
|
||||
</Card>
|
||||
|
||||
#### 3\. Convert blob tables to pandas {#3-convert-blob-tables-to-pandas}
|
||||
### 3\. Convert blob tables to pandas {#3-convert-blob-tables-to-pandas}
|
||||
|
||||
When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`.
|
||||
|
||||
@@ -328,7 +232,7 @@ Query builders also accept `blob_mode` on their `to_pandas()` method:
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Other modalities {#other-modalities}
|
||||
## Other modalities {#other-modalities}
|
||||
|
||||
The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data:
|
||||
|
||||
@@ -336,32 +240,8 @@ The `pa.binary()` and `pa.large_binary()` types are universal. You can use this
|
||||
- **Video:** Store video transitions or full clips using the Blob API.
|
||||
- **PDFs/Documents:** Store the raw file content for document search.
|
||||
|
||||
### bfloat16 {#bfloat16}
|
||||
## bfloat16 {#bfloat16}
|
||||
|
||||
Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`.
|
||||
Useful as a vector element type when halving the memory matters more than the
|
||||
precision does.
|
||||
|
||||
## Geospatial {#geospatial}
|
||||
|
||||
<Warning>
|
||||
Lance defines geospatial types and `st_*` functions on
|
||||
[GeoArrow](https://geoarrow.org), but they sit behind an optional build feature
|
||||
that **LanceDB does not ship**. Calling `st_intersects` and friends returns an
|
||||
error naming the missing feature rather than an unknown-function error, and the
|
||||
spatial index is gated the same way.
|
||||
</Warning>
|
||||
|
||||
Geometry can still be stored -- it is binary, and binary columns work -- but
|
||||
nothing interprets it. Filter on a bounding box held in ordinary numeric
|
||||
columns if you need spatial predicates today.
|
||||
|
||||
## Changing types later {#evolution}
|
||||
|
||||
A schema is not fixed at creation. Columns can be added, renamed, retyped, made
|
||||
nullable, and dropped, usually without rewriting existing data -- see
|
||||
[Schema evolution](/tables/schema). Deriving new values from existing columns is
|
||||
[data evolution](/feature-engineering/data-evolution).
|
||||
|
||||
Nullability is worth deciding up front: if later writes omit a non-nullable
|
||||
column, or write actual nulls into it, ingestion fails.
|
||||
@@ -1,442 +0,0 @@
|
||||
---
|
||||
title: "Schema evolution"
|
||||
sidebarTitle: "Schema Evolution"
|
||||
description: Learn how to manage table schemas in LanceDB, including adding, altering, and dropping columns.
|
||||
icon: "boxes-stacked"
|
||||
---
|
||||
import {
|
||||
PySchemaAddSetup as SchemaAddSetup,
|
||||
TsSchemaAddSetup as TsSchemaAddSetup,
|
||||
RsSchemaAddSetup as RsSchemaAddSetup,
|
||||
PyAddColumnsDefaultValues as AddColumnsDefaultValues,
|
||||
TsAddColumnsDefaultValues as TsAddColumnsDefaultValues,
|
||||
RsAddColumnsDefaultValues as RsAddColumnsDefaultValues,
|
||||
PyAddColumnsNullable as AddColumnsNullable,
|
||||
TsAddColumnsNullable as TsAddColumnsNullable,
|
||||
RsAddColumnsNullable as RsAddColumnsNullable,
|
||||
PySchemaAlterSetup as SchemaAlterSetup,
|
||||
TsSchemaAlterSetup as TsSchemaAlterSetup,
|
||||
RsSchemaAlterSetup as RsSchemaAlterSetup,
|
||||
PyAlterColumnsRename as AlterColumnsRename,
|
||||
TsAlterColumnsRename as TsAlterColumnsRename,
|
||||
RsAlterColumnsRename as RsAlterColumnsRename,
|
||||
PyAlterColumnsDataType as AlterColumnsDataType,
|
||||
TsAlterColumnsDataType as TsAlterColumnsDataType,
|
||||
RsAlterColumnsDataType as RsAlterColumnsDataType,
|
||||
PyAlterColumnsNullable as AlterColumnsNullable,
|
||||
TsAlterColumnsNullable as TsAlterColumnsNullable,
|
||||
RsAlterColumnsNullable as RsAlterColumnsNullable,
|
||||
PyAlterColumnsMultiple as AlterColumnsMultiple,
|
||||
TsAlterColumnsMultiple as TsAlterColumnsMultiple,
|
||||
PyAlterColumnsWithExpression as AlterColumnsWithExpression,
|
||||
TsAlterColumnsWithExpression as TsAlterColumnsWithExpression,
|
||||
RsAlterColumnsMultiple as RsAlterColumnsMultiple,
|
||||
RsAlterColumnsWithExpression as RsAlterColumnsWithExpression,
|
||||
PySchemaDropSetup as SchemaDropSetup,
|
||||
TsSchemaDropSetup as TsSchemaDropSetup,
|
||||
RsSchemaDropSetup as RsSchemaDropSetup,
|
||||
PyDropColumnsSingle as DropColumnsSingle,
|
||||
TsDropColumnsSingle as TsDropColumnsSingle,
|
||||
RsDropColumnsSingle as RsDropColumnsSingle,
|
||||
PyDropColumnsMultiple as DropColumnsMultiple,
|
||||
TsDropColumnsMultiple as TsDropColumnsMultiple,
|
||||
RsDropColumnsMultiple as RsDropColumnsMultiple,
|
||||
PyAlterVectorColumn as AlterVectorColumn,
|
||||
TsAlterVectorColumn as TsAlterVectorColumn,
|
||||
RsAlterVectorColumn as RsAlterVectorColumn,
|
||||
PySchemaFieldMetadataMerge as SchemaFieldMetadataMerge,
|
||||
TsSchemaFieldMetadataMerge as TsSchemaFieldMetadataMerge,
|
||||
RsSchemaFieldMetadataMerge as RsSchemaFieldMetadataMerge,
|
||||
PySchemaFieldMetadataReplace as SchemaFieldMetadataReplace,
|
||||
TsSchemaFieldMetadataReplace as TsSchemaFieldMetadataReplace,
|
||||
RsSchemaFieldMetadataReplace as RsSchemaFieldMetadataReplace,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
Schema evolution enables non-breaking modifications to a database table's structure — such as adding columns, altering data types, or dropping fields — to adapt to evolving data requirements without service interruptions.
|
||||
LanceDB supports ACID-compliant schema evolution through granular operations (add/alter/drop columns), allowing you to:
|
||||
|
||||
* Iterate Safely: Modify schemas in production with versioned datasets and backward compatibility
|
||||
* Scale Seamlessly: Handle ML model iterations, regulatory changes, or feature additions
|
||||
* Optimize Continuously: Remove unused fields or enforce new constraints without downtime
|
||||
|
||||
<Note>
|
||||
In SQL, schema changes are [`ALTER TABLE`](/sql/ddl#columns).
|
||||
</Note>
|
||||
|
||||
## Schema evolution operations {#schema-evolution-operations}
|
||||
|
||||
LanceDB supports four primary schema evolution operations:
|
||||
|
||||
1. **Adding new columns**: Extend your table with additional attributes
|
||||
2. **Altering existing columns**: Change column names, data types, or nullability
|
||||
3. **Updating field metadata**: Attach or change per-column Arrow metadata
|
||||
4. **Dropping columns**: Remove unnecessary columns from your schema
|
||||
|
||||
|
||||
<Tip title="Schema Evolution Performance">
|
||||
Schema evolution operations are applied immediately but do not typically require rewriting all data. However, data type changes may involve more substantial operations.
|
||||
</Tip>
|
||||
|
||||
Each schema evolution operation commits a new table version and returns status metadata such as
|
||||
the committed `version`. Run these operations from a mutable table handle; if you checked out an
|
||||
older version for reads, call `checkout_latest` / `checkoutLatest` before modifying the schema.
|
||||
|
||||
## Add new columns {#add-new-columns}
|
||||
|
||||
You can add new columns to a table with the [`add_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.add_columns)
|
||||
method in Python, [`addColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#addcolumns) in TypeScript/JavaScript, or `add_columns` in Rust.
|
||||
New columns are populated based on SQL expressions you provide.
|
||||
|
||||
|
||||
### Set up the example table {#set-up-the-example-table}
|
||||
|
||||
First, let's create a sample table with product data to demonstrate schema evolution:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaAddSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaAddSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaAddSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Columns derived from existing data -- SQL expressions over other columns -- are
|
||||
covered in [Data evolution](/feature-engineering/data-evolution), because
|
||||
deriving values is a feature-engineering job rather than a schema one.
|
||||
</Note>
|
||||
|
||||
### Add columns with default values {#add-columns-with-default-values}
|
||||
|
||||
Add boolean columns with default values for status tracking:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddColumnsDefaultValues}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddColumnsDefaultValues}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddColumnsDefaultValues}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Add nullable columns {#add-nullable-columns}
|
||||
|
||||
Add timestamp columns that can contain NULL values:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddColumnsNullable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Warning title="NULL Values in New Columns">
|
||||
When adding columns that should contain NULL values, be sure to cast the NULL to the appropriate type, e.g., `cast(NULL as timestamp)`.
|
||||
</Warning>
|
||||
|
||||
## Alter existing columns {#alter-existing-columns}
|
||||
|
||||
You can alter columns using the [`alter_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.alter_columns)
|
||||
method in Python, [`alterColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#altercolumns) in TypeScript/JavaScript, or `alter_columns` in Rust. This allows you to:
|
||||
|
||||
- Rename a column
|
||||
- Change a column's data type
|
||||
- Modify nullability (whether a column can contain NULL values)
|
||||
|
||||
|
||||
### Set up the example table {#set-up-the-example-table-2}
|
||||
|
||||
Create a table with a custom schema to demonstrate column alterations:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaAlterSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaAlterSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaAlterSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Rename columns {#rename-columns}
|
||||
|
||||
Change column names to better reflect their purpose:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsRename}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsRename}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsRename}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Change data types {#change-data-types}
|
||||
|
||||
Convert column data types for better performance or compatibility:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsDataType}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsDataType}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsDataType}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Make columns nullable {#make-columns-nullable}
|
||||
|
||||
You can alter columns to contain NULL values:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsNullable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsNullable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Changing a column to nullable affects future writes and merges too: missing values are accepted
|
||||
only when the target column is nullable.
|
||||
|
||||
### Multiple changes at once {#multiple-changes-at-once}
|
||||
|
||||
Apply several alterations in a single operation:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsMultiple}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Expression-based type changes {#expression-based-type-changes}
|
||||
|
||||
For transformations that are not simple casts (for example, converting `"$100"` to an integer), use a SQL-expression column add, then drop and rename:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterColumnsWithExpression}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterColumnsWithExpression}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterColumnsWithExpression}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Alter embedding types and dimensions {#alter-embedding-types-and-dimensions}
|
||||
|
||||
It's quite common to need to change an embedding column's schema, in case a new model becomes available with a different embedding dimension.
|
||||
- In Python, the example shows an in-place type update when the cast is compatible.
|
||||
- In TypeScript and Rust, the example shows a dimension change (`384 -> 1024`), which cannot be cast in-place.
|
||||
|
||||
For dimension changes, use this 3-step pattern: add a new column with the target type, drop the old column, then rename the new column to the original name.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AlterVectorColumn}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAlterVectorColumn}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAlterVectorColumn}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Warning>
|
||||
**`FixedSizeList` Dimension Changes in TypeScript and Rust**
|
||||
|
||||
`alterColumns` / `alter_columns` can cast between compatible types, but changing `FixedSizeList` dimensions (for example `384 -> 1024`) is not a compatible cast.
|
||||
For such cases, use `addColumns` / `add_columns` (with `arrow_cast`), then `dropColumns` / `drop_columns`, then rename the replacement column.
|
||||
</Warning>
|
||||
|
||||
<Warning title="Data Type Changes">
|
||||
Changing data types requires rewriting the column data and may be resource-intensive for large tables. Renaming columns or changing nullability is more efficient as it only updates metadata.
|
||||
</Warning>
|
||||
|
||||
## Update field metadata {#update-field-metadata}
|
||||
|
||||
Each column in a LanceDB table can carry a small key/value map of Arrow field metadata — useful
|
||||
for annotating columns with units, provenance, PII flags, embedding model versions, or any other
|
||||
schema-level context your application needs.
|
||||
|
||||
Use [`update_field_metadata`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.update_field_metadata)
|
||||
in Python, [`updateFieldMetadata`](https://lancedb.github.io/lancedb/js/classes/Table/#updatefieldmetadata)
|
||||
in TypeScript/JavaScript, or `update_field_metadata` in Rust to add, change, or remove these
|
||||
key/value pairs without rewriting the column data. Each call commits a new table version and returns
|
||||
the new `version`.
|
||||
|
||||
Each update targets one field by **dot-path**: top-level columns are addressed by name (for
|
||||
example `"embedding"`), and nested fields by their full path (for example `"address.zip"`). By
|
||||
default, the keys you pass are **merged** into the field's existing metadata — keys you do not
|
||||
mention are preserved, and passing `None` (Python) or `null` (TypeScript) deletes a key. Set
|
||||
`replace: true` to swap the field's entire metadata map instead of merging.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaFieldMetadataMerge}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaFieldMetadataMerge}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaFieldMetadataMerge}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
To overwrite a field's metadata entirely instead of merging, set `replace` to `true`:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaFieldMetadataReplace}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaFieldMetadataReplace}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaFieldMetadataReplace}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Tip>
|
||||
You can pass multiple updates in a single call to change metadata on several fields at once —
|
||||
each call commits a single new table version.
|
||||
</Tip>
|
||||
|
||||
### Canonical metadata keys {#canonical-metadata-keys}
|
||||
|
||||
You can use any string as a metadata key. By convention, LanceDB treats keys prefixed with
|
||||
`lancedb:` as canonical. LanceDB Enterprise displays these keys in the UI, and agents can rely
|
||||
on finding this information in a consistent place. Use them when they fit your use case:
|
||||
|
||||
| Key | Purpose | Example |
|
||||
| --- | --- | --- |
|
||||
| `lancedb:description` | Human-readable description of the field. | `"CLIP embedding of the product image"` |
|
||||
| `lancedb:tag:<name>` | User-defined key/value tag, where the suffix names the tag category. | `"lancedb:tag:model": "clip"` |
|
||||
| `lancedb:logical-column` | Groups related columns into one logical column. For example, `feature_v1` and `feature_v2` might belong to the same logical column. | `"lancedb:logical-column": "feature"` |
|
||||
| `lancedb:status` | Life cycle state of the column: `production`, `candidate`, `deprecated`, or `archived`. | `"lancedb:status": "production"` |
|
||||
|
||||
<Note>
|
||||
These keys are conventions, not enforced constraints. LanceDB does not validate their values,
|
||||
but following the conventions keeps your tables consistent with Enterprise tooling.
|
||||
</Note>
|
||||
|
||||
## Drop columns {#drop-columns}
|
||||
|
||||
You can remove columns using the [`drop_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.drop_columns)
|
||||
method in Python, [`dropColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#dropcolumns) in TypeScript/JavaScript, or `drop_columns` in Rust.
|
||||
|
||||
|
||||
### Set Up the example table {#set-up-the-example-table-3}
|
||||
|
||||
Create a table with temporary columns that we'll remove:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaDropSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaDropSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaDropSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Drop single columns {#drop-single-columns}
|
||||
|
||||
Remove individual columns that are no longer needed:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DropColumnsSingle}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDropColumnsSingle}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDropColumnsSingle}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Drop multiple columns {#drop-multiple-columns}
|
||||
|
||||
Remove several columns at once for efficiency:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DropColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDropColumnsMultiple}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDropColumnsMultiple}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Warning title="Irreversible Column Deletion">
|
||||
Dropping columns cannot be undone. Make sure you have backups or are certain before removing columns.
|
||||
</Warning>
|
||||
@@ -1,299 +0,0 @@
|
||||
---
|
||||
title: "Versioning and Reproducibility"
|
||||
sidebarTitle: "Versioning"
|
||||
description: "Learn how to implement versioning and ensure reproducibility in LanceDB. Includes version control, data snapshots, and audit trails."
|
||||
icon: "clock"
|
||||
---
|
||||
import {
|
||||
PyVersioningBasicSetup as VersioningBasicSetup,
|
||||
TsVersioningBasicSetup as TsVersioningBasicSetup,
|
||||
RsVersioningBasicSetup as RsVersioningBasicSetup,
|
||||
PyVersioningCheckInitialVersion as VersioningCheckInitialVersion,
|
||||
TsVersioningCheckInitialVersion as TsVersioningCheckInitialVersion,
|
||||
RsVersioningCheckInitialVersion as RsVersioningCheckInitialVersion,
|
||||
PyVersioningUpdateData as VersioningUpdateData,
|
||||
TsVersioningUpdateData as TsVersioningUpdateData,
|
||||
RsVersioningUpdateData as RsVersioningUpdateData,
|
||||
PyVersioningAddData as VersioningAddData,
|
||||
TsVersioningAddData as TsVersioningAddData,
|
||||
RsVersioningAddData as RsVersioningAddData,
|
||||
PyVersioningCheckVersionsAfterMod as VersioningCheckVersionsAfterMod,
|
||||
TsVersioningCheckVersionsAfterMod as TsVersioningCheckVersionsAfterMod,
|
||||
RsVersioningCheckVersionsAfterMod as RsVersioningCheckVersionsAfterMod,
|
||||
PyVersioningListAllVersions as VersioningListAllVersions,
|
||||
TsVersioningListAllVersions as TsVersioningListAllVersions,
|
||||
RsVersioningListAllVersions as RsVersioningListAllVersions,
|
||||
PyVersioningRollback as VersioningRollback,
|
||||
TsVersioningRollback as TsVersioningRollback,
|
||||
RsVersioningRollback as RsVersioningRollback,
|
||||
PyVersioningCheckoutLatest as VersioningCheckoutLatest,
|
||||
TsVersioningCheckoutLatest as TsVersioningCheckoutLatest,
|
||||
RsVersioningCheckoutLatest as RsVersioningCheckoutLatest,
|
||||
PyVersioningDeleteData as VersioningDeleteData,
|
||||
TsVersioningDeleteData as TsVersioningDeleteData,
|
||||
RsVersioningDeleteData as RsVersioningDeleteData,
|
||||
PyVersioningTags as VersioningTags,
|
||||
TsVersioningTags as TsVersioningTags,
|
||||
RsVersioningTags as RsVersioningTags,
|
||||
RsVersioningMakeQuotesReader as RsVersioningMakeQuotesReader,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
This page shows the core table-versioning APIs used in the code snippets for Python, TypeScript, and Rust.
|
||||
Each operation below maps directly to methods shown in the examples.
|
||||
|
||||
## Basic Versioning Example {#basic-versioning-example}
|
||||
|
||||
Let's create a table with sample data to demonstrate LanceDB's versioning capabilities:
|
||||
|
||||
### Set Up the Table {#set-up-the-table}
|
||||
|
||||
First, let's create a table with some sample data:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningBasicSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningBasicSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningBasicSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Expandable title="Rust helper used in this page">
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningMakeQuotesReader}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
</Expandable>
|
||||
|
||||
### Check Initial Version {#check-initial-version}
|
||||
|
||||
After creating the table, let's check the initial version information:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningCheckInitialVersion}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningCheckInitialVersion}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningCheckInitialVersion}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Modify Data {#modify-data}
|
||||
|
||||
When you modify data through operations like update or delete, LanceDB automatically creates new versions.
|
||||
|
||||
### Update Existing Data {#update-existing-data}
|
||||
|
||||
Let's update some existing records to see versioning in action:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningUpdateData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningUpdateData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningUpdateData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Add New Data {#add-new-data}
|
||||
|
||||
Now let's add more records to the table:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningAddData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningAddData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningAddData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Check Version Changes {#check-version-changes}
|
||||
|
||||
Let's see how the versions have changed after our modifications:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningCheckVersionsAfterMod}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningCheckVersionsAfterMod}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningCheckVersionsAfterMod}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Rollback to Previous Versions {#rollback-to-previous-versions}
|
||||
|
||||
LanceDB supports fast rollbacks to any previous version without data duplication.
|
||||
|
||||
### View All Versions {#view-all-versions}
|
||||
|
||||
First, let's see all the versions we've created:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningListAllVersions}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningListAllVersions}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningListAllVersions}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Restore a Version Snapshot {#restore-a-version-snapshot}
|
||||
|
||||
Now let's restore a captured version snapshot:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningRollback}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningRollback}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningRollback}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Tag-Based Versioning {#tag-based-versioning}
|
||||
|
||||
Numeric table versions like `v3` or `v17` are precise but hard to remember. Tags
|
||||
let you attach human-readable labels (e.g., `"prod"`, `"baseline"`,
|
||||
`"q3-evaluation"`) to specific versions and check those out by name. They are
|
||||
conceptually similar to git tags, and unlike numeric versions, **tagged versions
|
||||
are preserved when old versions are pruned** (see the cleanup note at the bottom
|
||||
of this page).
|
||||
|
||||
The tags API supports the standard CRUD operations — create, list, update, delete —
|
||||
plus checking out by tag name.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningTags}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningTags}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningTags}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Deleting a tag only removes the label, not the version it points to. After
|
||||
deletion, the underlying table version becomes eligible for cleanup again.
|
||||
</Note>
|
||||
|
||||
## Branches {#branches}
|
||||
|
||||
Beyond linear history, LanceDB also supports **branches** — isolated, writable
|
||||
lines of history forked from `main` (or a specific version). Whereas tags and
|
||||
`checkout` give you read-only views of existing versions, a branch has its own
|
||||
writable history, making it ideal for experiments, backfills, and migrations
|
||||
that you want to keep separate from production reads on `main`.
|
||||
|
||||
Branches are covered in their own guide: see [Branches](/tables/branching).
|
||||
|
||||
## Delete Data From the Table {#delete-data-from-the-table}
|
||||
|
||||
Let's demonstrate how deletions also create new versions:
|
||||
|
||||
### Go Back to Latest Version {#go-back-to-latest-version}
|
||||
|
||||
First, let's return to the latest version:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningCheckoutLatest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningCheckoutLatest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningCheckoutLatest}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Delete Data {#delete-data}
|
||||
|
||||
Now let's delete some data to see how it affects versioning:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{VersioningDeleteData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsVersioningDeleteData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsVersioningDeleteData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Version History and Operations {#version-history-and-operations}
|
||||
|
||||
On a fresh table, the snippets in this guide produce this version sequence:
|
||||
|
||||
1. `v1`: create table (`create_table` / `createTable` / `create_table`)
|
||||
2. `v2`: update rows (`update`)
|
||||
3. `v3`: add rows (`add`)
|
||||
4. `v4`: restore snapshot (`restore`) from `version_after_mod`/`versionAfterMod`
|
||||
5. `v5`: delete rows (`delete`)
|
||||
|
||||
Read-only and checkout operations shown here (`list_versions`/`listVersions`, `version`, `checkout`, `checkout_latest`/`checkoutLatest`) do not create new versions.
|
||||
|
||||
The version metadata fields can differ by backend. Direct table-backed version listing exposes a
|
||||
timestamp, while namespace-backed listing may expose fields such as `manifest_path`,
|
||||
`manifest_size`, `e_tag`, and `timestamp_millis`. In deployments that use managed versioning,
|
||||
prefer the table version APIs exposed by LanceDB Enterprise or the namespace service instead of
|
||||
mixing in lower-level Lance file operations.
|
||||
|
||||
<Note>
|
||||
**System Operations**
|
||||
|
||||
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
|
||||
In LanceDB OSS and Enterprise, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
|
||||
which is when old-version files are removed and disk space is reclaimed.
|
||||
|
||||
**Tagged versions are exempt from cleanup.** A version with a tag pointing at it is
|
||||
retained regardless of age, and its files are not removed by `optimize()`. To make
|
||||
a tagged version eligible for pruning, [delete the tag](#tag-based-versioning) first.
|
||||
</Note>
|
||||
@@ -331,7 +331,7 @@ with open(out / "metadata.json", "w") as f:
|
||||
}, f, indent=2)
|
||||
```
|
||||
|
||||
To reproduce a run, [time-travel](/tables/versioning) the view to the recorded version before opening the loader:
|
||||
To reproduce a run, [time-travel](/management/table#versioning) the view to the recorded version before opening the loader:
|
||||
|
||||
```py Python icon=Python
|
||||
tbl = lancedb.connect("data/bdd100k/lancedb").open_table("bdd100k_rider_train")
|
||||
|
||||
@@ -108,7 +108,7 @@ for batch in dataloader:
|
||||
|
||||
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. LanceDB tables and `Permutation` objects are picklable, so each worker reopens the table after it starts.
|
||||
|
||||
Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and it is set to become the Python default. See [the performance guide](/performance) for more multiprocessing guidance.
|
||||
Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and it is set to become the Python default. See [the performance guide](/search#performance) for more multiprocessing guidance.
|
||||
|
||||
<Note>
|
||||
`forkserver` is only available on POSIX systems (Linux and macOS). On Windows, use `spawn` instead — it is the only start method available there.
|
||||
|
||||
@@ -28,7 +28,7 @@ print the resolved query plan.
|
||||
|
||||
LanceDB provides two powerful tools for query analysis and optimization: `explain_plan` and `analyze_plan`.
|
||||
|
||||
Read the full guides on [Explain](/search/explain) and [Explain Analyze](/search/explain-analyze).
|
||||
Read the full guides on [Explain](/search/explain) and [Explain Analyze](/search/explain).
|
||||
|
||||
### The Python multiprocessing module {#the-python-multiprocessing-module}
|
||||
Multiprocessing with `fork` is not supported. You should use `spawn` instead.
|
||||
|
||||
@@ -32,7 +32,7 @@ vector databases are ill-equipped to handle.
|
||||
|
||||
4. "We need to A/B test a new chunking strategy, but we can't disrupt the production system or duplicate the entire dataset." Experimentation is vital for improvement, but it can't come at the cost of production stability or a massive infrastructure bill.
|
||||
|
||||
LanceDB's [zero-cost data evolution](/tables/schema) and [time-travel capabilities](https://docs.lancedb.com/tables/versioning) directly address these critical enterprise pain points, providing the foundation for a reliable, auditable, and production-ready RAG system.
|
||||
LanceDB's [zero-cost data evolution](/management/table#schema) and [time-travel capabilities](https://docs.lancedb.com/management/table#versioning) directly address these critical enterprise pain points, providing the foundation for a reliable, auditable, and production-ready RAG system.
|
||||
|
||||
## Dataset: The U.S. Federal Register {#dataset-the-us-federal-register}
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ are runnable in Colab; the rest are written up here.
|
||||
| [Vector search](/search/vector-search) | Similarity search, distance metrics, and tuning it. |
|
||||
| [Full-text search](/search/full-text-search) | Keyword search over text columns. |
|
||||
| [Hybrid search](/search/hybrid-search) | Keyword and vector search combined. |
|
||||
| [Multi-vector search](/search/multivector-search) | Several embeddings per document. |
|
||||
| [Multi-vector search](/search/vector-search) | Several embeddings per document. |
|
||||
| [Reranking](/reranking) | Reordering results to improve the top-k. |
|
||||
| [Feature engineering](/feature-engineering) | Deriving columns and tables from data already stored. |
|
||||
| [Materialized views](/management/materialized-view) | A table defined by a query, refreshed incrementally. |
|
||||
|
||||
@@ -4,7 +4,7 @@ sidebarTitle: "Multi-vector XTR"
|
||||
description: "Represent a document with many vectors, and speed retrieval with ConteXtualized Token Retrieval."
|
||||
---
|
||||
|
||||
Covers [multi-vector search](/search/multivector-search) enhanced by XTR
|
||||
Covers [multi-vector search](/search/vector-search) enhanced by XTR
|
||||
(ConteXtualized Token Retriever): representing complex data with several vectors
|
||||
for more nuanced meaning, and using XTR to speed retrieval by prioritizing the
|
||||
most important tokens.
|
||||
|
||||
Reference in New Issue
Block a user