mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
This commit is contained in:
Jack Ye
co-authored by
Brendan Clement
Brendan Clement
parent
de005f1d61
commit
47b7babc08
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Agentic RAG"
|
||||
sidebarTitle: "Agentic RAG"
|
||||
description: "Several agents collaborating to retrieve and answer."
|
||||
---
|
||||
|
||||
Builds a RAG system where multiple agents collaborate -- retrieving
|
||||
information, checking it, and generating answers -- rather than a single
|
||||
retrieve-then-generate pass.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Agentic_RAG/main.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/Agentic_RAG">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: "Contextual RAG"
|
||||
sidebarTitle: "Contextual RAG"
|
||||
description: "Prepend LLM-generated context to each chunk before embedding it."
|
||||
---
|
||||
|
||||
Addresses the "lost in the middle" problem in retrieval. An LLM generates a
|
||||
succinct piece of context for each document chunk, which is prepended to the
|
||||
chunk before embedding -- so a chunk carries enough surrounding meaning to be
|
||||
retrieved for the right queries.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Contextual-RAG/Anthropic_Contextual_RAG.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/Contextual-RAG">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: "HyDE: hypothetical document embeddings"
|
||||
sidebarTitle: "HyDE"
|
||||
description: "Generate a hypothetical answer, then retrieve documents similar to it."
|
||||
---
|
||||
|
||||
An LLM writes a hypothetical document answering the query, and that document --
|
||||
not the query -- is embedded and used for retrieval. It works because a
|
||||
plausible answer sits nearer the real answers in embedding space than a question
|
||||
does.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Advance-RAG-with-HyDE/main.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/Advance-RAG-with-HyDE">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Late chunking"
|
||||
sidebarTitle: "Late Chunking"
|
||||
description: "Retrieve whole documents first, chunk them afterwards."
|
||||
---
|
||||
|
||||
Inverts the usual order: documents are retrieved first and chunked just before
|
||||
synthesis. Chunking late preserves context that pre-chunking discards, since a
|
||||
chunk is cut with knowledge of the query it is answering.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Late_Chunking/Late_Chunking_(Chunked_Pooling).ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/Advanced_RAG_Late_Chunking">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Matryoshka embeddings"
|
||||
sidebarTitle: "Matryoshka Embeddings"
|
||||
description: "Nested, variable-size embeddings with LanceDB and LlamaIndex."
|
||||
---
|
||||
|
||||
A RAG pipeline built on Matryoshka embeddings with LlamaIndex. The embeddings
|
||||
nest, so one model serves several dimensionalities -- store the full vector and
|
||||
retrieve with a truncated one when the cheaper comparison is good enough.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/RAG-with_MatryoshkaEmbed-Llamaindex/main.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/RAG-with_MatryoshkaEmbed-Llamaindex">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "Multimodal Agent"
|
||||
sidebarTitle: "Multimodal Agent"
|
||||
description: "Build an AI agent that understands both text and images to help users find recipes using LanceDB and PydanticAI"
|
||||
---
|
||||
|
||||
|
||||
[](https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing)
|
||||
|
||||
Ever wanted to combine the power of text and images in a single AI agent? In this tutorial,
|
||||
you'll build an agent that can understand both text and images to help users discover recipes that are relevant to them. The approach shown combines LanceDB's multimodal capabilities with [Pydantic AI](https://ai.pydantic.dev/) for the agentic workflow.
|
||||
|
||||
## Key Technologies {#key-technologies}
|
||||
|
||||
- **LanceDB**: Embedded retrieval library and multimodal lakehouse for efficient storage and retrieval
|
||||
- **PydanticAI**: Modern AI agent framework with type safety
|
||||
- **Sentence Transformers**: Text embeddings for semantic search
|
||||
- **CLIP**: Vision-language model for image understanding
|
||||
- **Streamlit**: Interactive web application framework
|
||||
|
||||
## Tutorial Overview {#tutorial-overview}
|
||||
|
||||
### Option 1: Notebook {#option-1-notebook}
|
||||
The notebook shows how to work through the steps and prepare a small sample recipe dataset, generate both text and image
|
||||
embeddings, store everything efficiently in LanceDB, and then build a PydanticAI agent with custom tools to
|
||||
query it. You'll finish by testing the agent against a few example questions to see the full multimodal flow
|
||||
end to end.
|
||||
|
||||
<Card title="Notebook" href="https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing">
|
||||
This simple tutorial provides a step-by-step workflow with a small demo dataset of 4 examples.
|
||||
No local setup required - just click and start learning about multimodal agents.
|
||||
[](https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing)
|
||||
</Card>
|
||||
|
||||
### Option 2: Demo Application (Local Setup) {#option-2-demo-application-local-setup}
|
||||
The demo application is the full codebase: you'll download and process a real recipe dataset with thousands
|
||||
of items, run a Streamlit chat interface that supports image upload, and follow a structure that includes
|
||||
production-minded touches like error handling, logging, and monitoring. Everything you need to deploy is
|
||||
included.
|
||||
|
||||
<Card icon="download" title="Download Tutorial Files" href="https://github.com/lancedb/vectordb-recipes/tree/main/applications/multimodal-recipe-agent">
|
||||
Download the files for the full demo application here.
|
||||
</Card>
|
||||
|
||||
### Dataset Information {#dataset-information}
|
||||
- **Source**: [Kaggle Recipe Dataset](https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images)
|
||||
- **Size**: Thousands of recipes with images
|
||||
- **Format**: CSV file with recipe data and image references
|
||||
|
||||
### Setup {#setup}
|
||||
|
||||
<CodeGroup>
|
||||
```bash bash icon="code"
|
||||
# 1. Extract the downloaded files to a folder
|
||||
# 2. Navigate to the folder in terminal
|
||||
cd multimodal-recipe-agent
|
||||
|
||||
# 3. Install dependencies with uv
|
||||
uv sync
|
||||
|
||||
# 4. Download the Kaggle dataset
|
||||
# Visit: https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images
|
||||
# Extract recipes.csv to the data/ folder
|
||||
|
||||
# 5. Import the dataset
|
||||
uv run python import.py
|
||||
|
||||
# 6. Run the complete Streamlit chat application
|
||||
uv run streamlit run app.py
|
||||
```
|
||||
</CodeGroup>
|
||||
@@ -0,0 +1,175 @@
|
||||
---
|
||||
title: "NVIDIA RAG Blueprint with LanceDB"
|
||||
sidebarTitle: "NVIDIA RAG Blueprint"
|
||||
description: "Use LanceDB as the retrieval layer for NVIDIA RAG Blueprint with a Docker-first, retrieval-only reference integration."
|
||||
---
|
||||
|
||||
## What this tutorial shows {#what-this-tutorial-shows}
|
||||
|
||||
If you are using [NVIDIA RAG Blueprints](https://build.nvidia.com/blueprints) and want to evaluate LanceDB in that stack, this tutorial gives you a concrete starting point. It shows how to use LanceDB as the retrieval layer for a Docker-based NVIDIA RAG deployment with a small, script-driven reference integration where LanceDB OSS is embedded directly in the NVIDIA containers, the collection is prepared ahead of time, and the RAG server retrieves from it for search and generation. The example is intentionally retrieval-only, but it also includes hybrid search and reranker selection so you can see how LanceDB fits into a realistic NVIDIA retrieval workflow.
|
||||
|
||||
<Tip>
|
||||
The runnable example for this tutorial lives in the
|
||||
[VectorDB recipes repository](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb).
|
||||
</Tip>
|
||||
|
||||
|
||||
## How NVIDIA organizes vector databases {#how-nvidia-organizes-vector-databases}
|
||||
|
||||
NVIDIA's [RAG Blueprint documentation](https://docs.nvidia.com/rag/latest/readme.html) effectively describes three different patterns for vector database support.
|
||||
1. There are built-in backends such as Milvus, where NVIDIA already owns both ingestion and
|
||||
retrieval.
|
||||
2. There are built-in alternatives such as Elasticsearch, where NVIDIA still owns
|
||||
the end-to-end flow but switches the backend through configuration.
|
||||
3. Then, there is the custom vector database path, where you implement a `VDBRag` backend yourself and register it in NVIDIA's
|
||||
factory.
|
||||
|
||||
The LanceDB example shown below fits into the third category. More specifically, it follows NVIDIA's
|
||||
**retrieval-only** custom backend path: the data is prepared in LanceDB ahead of time, and NVIDIA
|
||||
RAG Blueprint is then pointed at that existing collection for search and generation. It does not
|
||||
yet teach NVIDIA's ingestor how to write new documents into LanceDB automatically.
|
||||
|
||||
## Deployment model {#deployment-model}
|
||||
|
||||
This reference integration uses **LanceDB OSS as an embedded retrieval library**, not as a separate
|
||||
database service. In practice, `APP_VECTORSTORE_NAME` is set to `lancedb`, `APP_VECTORSTORE_URL`
|
||||
points to a local filesystem path inside the NVIDIA containers, the LanceDB collection is prepared
|
||||
ahead of time, and the NVIDIA RAG server loads the LanceDB adapter to retrieve directly from that
|
||||
local dataset.
|
||||
|
||||
## What the recipe contains {#what-the-recipe-contains}
|
||||
|
||||
The recipe at
|
||||
[`examples/nvidia-rag-blueprint-lancedb`](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb)
|
||||
is organized around a small number of practical pieces. The data-prep script builds a demo
|
||||
LanceDB collection from scratch, generates embeddings through the LanceDB embedding registry, and
|
||||
creates a full-text index so hybrid retrieval works immediately. The adapter file shows the
|
||||
retrieval-only integration point for NVIDIA RAG Blueprint, while the Docker override and NVIDIA
|
||||
change guide show the minimal configuration and source changes needed to run the example against
|
||||
NVIDIA's containers.
|
||||
|
||||
## End-to-end flow {#end-to-end-flow}
|
||||
|
||||
### 1\. Prepare the LanceDB collection {#1-prepare-the-lancedb-collection}
|
||||
|
||||
From the [recipe directory](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb):
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
uv run prepare_lancedb.py --embedder demo-keyword --reranker mrr
|
||||
```
|
||||
|
||||
That script creates:
|
||||
|
||||
- a local LanceDB dataset under `data/`
|
||||
- a collection named `nvidia_blueprint_demo`
|
||||
- automatic embeddings generated at ingest time
|
||||
- an FTS index for hybrid search
|
||||
|
||||
The default embedder is an offline demo embedder so the example stays easy to run. If you want a
|
||||
more realistic setup, the same script can switch to a sentence-transformers embedder.
|
||||
|
||||
### 2\. Patch the NVIDIA blueprint {#2-patch-the-nvidia-blueprint}
|
||||
|
||||
Follow the instructions in the recipe's
|
||||
[`nvidia_blueprint_changes.md`](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb/nvidia_blueprint_changes.md).
|
||||
The essential changes are:
|
||||
|
||||
- add LanceDB dependencies to the NVIDIA environment
|
||||
- copy `lancedb_vdb.py` into the NVIDIA source tree
|
||||
- register the `lancedb` branch in NVIDIA's VDB factory
|
||||
|
||||
NVIDIA's [RAG blueprint documentation](https://docs.nvidia.com/rag/latest/readme.html) and custom-VDB guide provide useful background if you want more context before applying the LanceDB-specific changes.
|
||||
|
||||
### 3\. Start the Docker deployment {#3-start-the-docker-deployment}
|
||||
|
||||
Set the absolute path to the recipe directory:
|
||||
|
||||
```bash
|
||||
export LANCEDB_RECIPE_DIR=/absolute/path/to/vectordb-recipes/examples/nvidia-rag-blueprint-lancedb
|
||||
```
|
||||
|
||||
Then from the NVIDIA repo root:
|
||||
|
||||
```bash
|
||||
docker compose \
|
||||
-f deploy/compose/docker-compose-rag-server.yaml \
|
||||
-f "$LANCEDB_RECIPE_DIR"/docker-compose.override.yml \
|
||||
up -d --build
|
||||
|
||||
docker compose \
|
||||
-f deploy/compose/docker-compose-ingestor-server.yaml \
|
||||
-f "$LANCEDB_RECIPE_DIR"/docker-compose.override.yml \
|
||||
up -d --build
|
||||
```
|
||||
|
||||
The key environment values are:
|
||||
|
||||
- `APP_VECTORSTORE_NAME=lancedb`
|
||||
- `APP_VECTORSTORE_URL=/opt/lancedb-recipe/data`
|
||||
- `COLLECTION_NAME=nvidia_blueprint_demo`
|
||||
- `APP_VECTORSTORE_SEARCHTYPE=hybrid`
|
||||
- `LANCEDB_RERANKER=mrr`
|
||||
|
||||
## Verifying the integration {#verifying-the-integration}
|
||||
|
||||
### Search {#search}
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8081/v1/search \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"query": "How do I replace Milvus in the NVIDIA RAG blueprint with LanceDB?",
|
||||
"use_knowledge_base": true,
|
||||
"collection_names": ["nvidia_blueprint_demo"],
|
||||
"vdb_top_k": 3,
|
||||
"reranker_top_k": 0
|
||||
}'
|
||||
```
|
||||
|
||||
### Generate {#generate}
|
||||
|
||||
```bash
|
||||
curl -N -X POST http://localhost:8081/v1/generate \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"messages": [{"role":"user","content":"Summarize the LanceDB integration approach."}],
|
||||
"use_knowledge_base": true,
|
||||
"collection_names": ["nvidia_blueprint_demo"],
|
||||
"vdb_top_k": 3,
|
||||
"reranker_top_k": 0
|
||||
}'
|
||||
```
|
||||
|
||||
## Hybrid retrieval and rerankers {#hybrid-retrieval-and-rerankers}
|
||||
|
||||
This example is meant to prove more than a trivial vector lookup.
|
||||
|
||||
- LanceDB hybrid retrieval combines vector search with full-text search
|
||||
- the recipe creates the FTS index as part of dataset prep
|
||||
- the adapter supports `RRFReranker`, `MRRReranker`, and `CrossEncoderReranker`
|
||||
- the default example uses `MRRReranker`, not a plain weighted linear combination
|
||||
|
||||
That matters for NVIDIA partner workloads because product names, storage platforms, and technical
|
||||
jargon often need exact lexical matching as well as semantic retrieval.
|
||||
|
||||
## How this can be extended {#how-this-can-be-extended}
|
||||
|
||||
The current example follows NVIDIA's **custom retrieval-only backend** path. In practice, that
|
||||
means the LanceDB collection is created ahead of time and NVIDIA RAG Blueprint is then pointed at
|
||||
that existing collection for search and generation. The sample data in `prepare_lancedb.py` exists
|
||||
only to make that flow runnable end to end: it creates a small local collection, inserts a few
|
||||
documents, generates embeddings, and builds an FTS index so the NVIDIA side has something real to
|
||||
query.
|
||||
|
||||
A fuller integration is possible. NVIDIA's custom `VDBRag` interface also supports the pattern used
|
||||
by built-in backends such as Milvus and Elasticsearch, where NVIDIA owns both ingestion and
|
||||
retrieval. To make LanceDB work that way, a complete LanceDB backend would need to implement the
|
||||
ingestion methods NVIDIA documents, especially `create_collection` and `write_to_index`, along with
|
||||
the retrieval and collection-management methods expected by the rest of the stack.
|
||||
|
||||
The open work is in defining how NVIDIA's ingestor should write
|
||||
records into LanceDB, how that storage is shared between the ingestor and the RAG server, and how
|
||||
document and collection metadata should be exposed so the broader NVIDIA APIs behave correctly.
|
||||
Until those pieces exist, this example should be read as: prepare LanceDB first, then let NVIDIA
|
||||
retrieve from it.
|
||||
@@ -0,0 +1,179 @@
|
||||
---
|
||||
title: "Time-Travel RAG with versioned data"
|
||||
sidebarTitle: "Time-travel RAG"
|
||||
description: "Learn how to build production-ready RAG systems with LanceDB's time-travel capabilities for regulatory compliance and audit trails."
|
||||
---
|
||||
|
||||
<Tip>
|
||||
All the scripts and code for this tutorial are available in the
|
||||
[vectorDB recipes](https://github.com/lancedb/vectordb-recipes/tree/main/examples/time-travel-rag) repository.
|
||||
</Tip>
|
||||
|
||||
## Use case: Financial services regulatory knowledge base {#use-case-financial-services-regulatory-knowledge-base}
|
||||
|
||||
Imagine you're a major investment bank. Your team is tasked with building a critical Retrieval-Augmented Generation (RAG) system. This system must provide instant, accurate answers to compliance officers about ever-changing financial regulations. A wrong or out-of-date answer isn't just an inconvenience—it could lead to multi-million dollar fines, reputational damage, and regulatory audits.
|
||||
|
||||
Your knowledge base is a living entity, constantly evolving with:
|
||||
|
||||
* Daily regulatory updates from government bodies.
|
||||
* New internal policy documents and interpretations.
|
||||
* A/B testing of different embedding models and text chunking strategies to improve accuracy.
|
||||
|
||||
This dynamic environment creates a series of high-stakes challenges that traditional
|
||||
vector databases are ill-equipped to handle.
|
||||
|
||||
## Pain points solved by LanceDB {#pain-points-solved-by-lancedb}
|
||||
|
||||
1. "Our RAG gave different answers yesterday versus today. Which version was used in the official compliance report?" Without versioning, you can't prove what the AI knew at a specific point in time, making audits impossible.
|
||||
|
||||
2. "The new embedding model we deployed corrupted half the vectors. Can we instantly roll back our 10TB dataset?" With traditional systems, a rollback means a painful, hours-long (or days-long) process of re-indexing from a backup, leading to significant downtime.
|
||||
|
||||
3. "Regulators want to audit an AI-assisted decision from three months ago. How can we prove what data the model had access to at that exact moment?" Reproducibility is key for compliance. You must be able to reconstruct the exact state of the knowledge base for any historical query.
|
||||
|
||||
4. "We need to A/B test a new chunking strategy, but we can't disrupt the production system or duplicate the entire dataset." Experimentation is vital for improvement, but it can't come at the cost of production stability or a massive infrastructure bill.
|
||||
|
||||
LanceDB's [zero-cost data evolution](/management/table#schema) and [time-travel capabilities](https://docs.lancedb.com/management/table#versioning) directly address these critical enterprise pain points, providing the foundation for a reliable, auditable, and production-ready RAG system.
|
||||
|
||||
## Dataset: The U.S. Federal Register {#dataset-the-us-federal-register}
|
||||
|
||||
To make this use case realistic, we'll use a perfect real-world dataset: The U.S. Federal Register, the official daily journal of the United States Government.
|
||||
|
||||
It contains all new rules, proposed rules, and notices from federal agencies. It is the canonical source for regulatory changes, and it's updated every business day. It even has a public API, allowing us to simulate the real-time ingestion of new documents.
|
||||
|
||||
An example output of the workflow defined in main.py
|
||||
is shown below.
|
||||
|
||||
```bash main.py expandable
|
||||
--- Initializing Database Environment ---
|
||||
Removed old database at ./lancedb
|
||||
Loading embedding model: all-MiniLM-L6-v2...
|
||||
|
||||
--- STEP 1: Initial Data Ingestion ---
|
||||
|
||||
Fetching 500 documents for publication date: 2024-08-19...
|
||||
Successfully fetched 86 documents.
|
||||
Embedding 86 documents...
|
||||
Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7.12it/s]
|
||||
Successfully embedded 86 documents.
|
||||
|
||||
Creating table 'federal_register'...
|
||||
✅ Table 'federal_register' created. Version: 1, Rows: 86
|
||||
|
||||
--- STEP 2: Simulating Sequential Daily Updates ---
|
||||
|
||||
Fetching 500 documents for publication date: 2024-08-20...
|
||||
Successfully fetched 102 documents.
|
||||
Embedding 102 documents...
|
||||
Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 10.66it/s]
|
||||
Successfully embedded 102 documents.
|
||||
✅ Data added to 'federal_register'. New Version: 2, Total Rows: 188
|
||||
|
||||
Fetching 500 documents for publication date: 2024-08-21...
|
||||
Successfully fetched 114 documents.
|
||||
Embedding 114 documents...
|
||||
Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 10.19it/s]
|
||||
Successfully embedded 114 documents.
|
||||
✅ Data added to 'federal_register'. New Version: 3, Total Rows: 302
|
||||
|
||||
|
||||
========================================================
|
||||
= PART 1: AUDITING KNOWLEDGE BASE ACROSS TIME =
|
||||
========================================================
|
||||
|
||||
Running audit for query: 'cybersecurity reporting requirements for public companies'
|
||||
|
||||
Attempting to open 'federal_register' and checkout version 1...
|
||||
✅ Successfully checked out Version 1 of 'federal_register'. Total rows: 86
|
||||
Querying table 'federal_register' (Version 1)...
|
||||
|
||||
--- Top Result for Version: V1 (all-MiniLM-L6-v2) ---
|
||||
📄 Title: Public Company Accounting Oversight Board; Extension of Approval Periods for Proposed Rules on a Firm's System of Quality Control and Related Amendments to PCAOB Standards, Proposed Rules on Amendments Related to Aspects of Designing and Performing Audit Procedures That Involve Technology-Assisted Analysis of Information in Electronic Form, and Proposed Rules on Amendment to PCAOB Rule 3502 Governing Contributory Liability
|
||||
🗓️ Date: 2024-08-19
|
||||
📏 Distance: 1.1667
|
||||
📝 Abstract:
|
||||
[No abstract available for this document]
|
||||
--------------------------------------
|
||||
|
||||
Attempting to open 'federal_register' and checkout version 2...
|
||||
✅ Successfully checked out Version 2 of 'federal_register'. Total rows: 188
|
||||
Querying table 'federal_register' (Version 2)...
|
||||
|
||||
--- Top Result for Version: V2 (all-MiniLM-L6-v2) ---
|
||||
📄 Title: Information Collection Being Reviewed by the Federal Communications Commission Under Delegated Authority
|
||||
🗓️ Date: 2024-08-20
|
||||
📏 Distance: 1.1436
|
||||
📝 Abstract:
|
||||
As part of its continuing effort to reduce paperwork burdens, and as required by the Paperwork
|
||||
Reduction Act (PRA) of 1995, the Federal Communications Commission (FCC or the Commission) invites
|
||||
the general public and other Federal agencies to take this opportunity to comment on the following
|
||||
information collection. Comments are requested concerning: whether the proposed collection of
|
||||
information is necessary for the proper performance of the functions of the Commission, including
|
||||
whether the information shall have practical utility; the accuracy of the Commission's burden
|
||||
estimate; ways to enhance the quality, utility, and clarity of the information collected; ways to
|
||||
minimize the burden of the collection of information on the respondents, including the use of
|
||||
automated collection techniques or other forms of information technology; and ways to further reduce
|
||||
the information collection burden on small business concerns with fewer than 25 employees. The FCC
|
||||
may not conduct or sponsor a collection of information unless it displays a currently valid control
|
||||
number. No person shall be subject to any penalty for failing to comply with a collection of
|
||||
information subject to the PRA that does not display a valid Office of Management and Budget (OMB)
|
||||
control number.
|
||||
--------------------------------------
|
||||
|
||||
Attempting to open 'federal_register' and checkout version 3...
|
||||
✅ Successfully checked out Version 3 of 'federal_register'. Total rows: 302
|
||||
Querying table 'federal_register' (Version 3)...
|
||||
|
||||
--- Top Result for Version: V3 (all-MiniLM-L6-v2) ---
|
||||
📄 Title: Equipment, Systems, and Network Information Security Protection
|
||||
🗓️ Date: 2024-08-21
|
||||
📏 Distance: 1.0942
|
||||
📝 Abstract:
|
||||
This proposed rulemaking would impose new design standards to address cybersecurity threats for
|
||||
transport category airplanes, engines, and propellers. The intended effect of this proposed action
|
||||
is to standardize the FAA's criteria for addressing cybersecurity threats, reducing certification
|
||||
costs and time while maintaining the same level of safety provided by current special conditions.
|
||||
--------------------------------------
|
||||
|
||||
✅ Date-based audit complete. Results show how knowledge evolves over time. This demonstrates LanceDB's powerful [versioning capabilities](/tutorials/tables/consistency#versioning) for maintaining audit trails.
|
||||
|
||||
|
||||
=============================================================
|
||||
= PART 2: A/B TESTING DIFFERENT EMBEDDING MODELS =
|
||||
=============================================================
|
||||
Loading embedding model: all-mpnet-base-v2...
|
||||
Embedding 302 documents...
|
||||
Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:03<00:00, 2.55it/s]
|
||||
Successfully embedded 302 documents.
|
||||
|
||||
Creating table 'federal_register_experimental'...
|
||||
✅ Table 'federal_register_experimental' created. Version: 1, Rows: 302
|
||||
|
||||
Comparing search results for the same data with different models:
|
||||
Querying table 'federal_register' (Version 3)...
|
||||
|
||||
--- Top Result for Version: Latest Prod V3 (all-MiniLM-L6-v2) ---
|
||||
📄 Title: Equipment, Systems, and Network Information Security Protection
|
||||
🗓️ Date: 2024-08-21
|
||||
📏 Distance: 1.0942
|
||||
📝 Abstract:
|
||||
This proposed rulemaking would impose new design standards to address cybersecurity threats for
|
||||
transport category airplanes, engines, and propellers. The intended effect of this proposed action
|
||||
is to standardize the FAA's criteria for addressing cybersecurity threats, reducing certification
|
||||
costs and time while maintaining the same level of safety provided by current special conditions.
|
||||
--------------------------------------
|
||||
Querying table 'federal_register_experimental' (Version 1)...
|
||||
|
||||
--- Top Result for Version: Experimental (all-mpnet-base-v2) ---
|
||||
📄 Title: Commission Information Collection Activities (FERC-725B); Comment Request; Extension
|
||||
🗓️ Date: 2024-08-20
|
||||
📏 Distance: 1.0827
|
||||
📝 Abstract:
|
||||
In compliance with the requirements of the Paperwork Reduction Act of 1995, the Federal Energy
|
||||
Regulatory Commission (Commission or FERC) is soliciting public comment on the currently approved
|
||||
information collection, FERC-725B, Mandatory Reliability Standards, Critical Infrastructure
|
||||
Protection (CIP) (Update for CIP-012-1 to version CIP-012-02) Cyber Security--Communications between
|
||||
Control Centers. The 60-day notice comment period ended on July 23, 2024, with no comments received.
|
||||
--------------------------------------
|
||||
|
||||
✅ A/B test complete. Notice the difference in relevance (distance score) between models. This showcases how LanceDB enables [experimentation with different embedding models](/docs/embeddings/) without disrupting production systems.
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Feature engineering 101"
|
||||
sidebarTitle: "Feature Engineering 101"
|
||||
description: "Add derived feature columns to an existing table."
|
||||
---
|
||||
|
||||
Adds new derived features to a table, the pattern documented under
|
||||
[data evolution](/feature-engineering/data-evolution) and
|
||||
[computed columns](/feature-engineering/add-computed-column).
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/feature-engineering/feature-engineering-101.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/feature-engineering/feature-engineering-101.ipynb">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Materialized views"
|
||||
sidebarTitle: "Materialized Views"
|
||||
description: "Persist query results as physical tables and keep them current."
|
||||
---
|
||||
|
||||
Creates [materialized views](/management/materialized-view): query results
|
||||
persisted as physical tables, brought up to date by a
|
||||
[refresh](/feature-engineering/materialized-views).
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/feature-engineering/materialized-views.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/feature-engineering/materialized-views.ipynb">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: "Tutorials"
|
||||
sidebarTitle: "Overview"
|
||||
description: "Worked examples of building with LanceDB, grouped by what they build."
|
||||
---
|
||||
|
||||
Every tutorial is one page in the sidebar, grouped by what it builds. The
|
||||
documentation explains how a feature works; these show it doing something.
|
||||
|
||||
Examples marked as notebooks live in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository and
|
||||
are runnable in Colab; the rest are written up here.
|
||||
|
||||
## Where to start {#start}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Search" icon="magnifying-glass" href="/tutorials/search/hybrid-search-beir">
|
||||
Vector, multi-vector, hybrid and full-text retrieval.
|
||||
</Card>
|
||||
<Card title="RAG" icon="robot" href="/tutorials/agents/contextual-rag">
|
||||
Retrieval-augmented generation and agent applications.
|
||||
</Card>
|
||||
<Card title="Feature engineering" icon="wand-magic-sparkles" href="/tutorials/feature-engineering/feature-engineering-101">
|
||||
Deriving columns, materialized views, experimenting on branches.
|
||||
</Card>
|
||||
<Card title="Training" icon="graduation-cap" href="/training/object-detection">
|
||||
Curating data and training models directly on a table.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## The features these use {#read-the-docs}
|
||||
|
||||
| Feature | What it covers |
|
||||
|:--------|:------------|
|
||||
| [Vector search](/search/vector-search) | Similarity search, distance metrics, and tuning it. |
|
||||
| [Full-text search](/search/full-text-search) | Keyword search over text columns. |
|
||||
| [Hybrid search](/search/hybrid-search) | Keyword and vector search combined. |
|
||||
| [Multi-vector search](/search/vector-search) | Several embeddings per document. |
|
||||
| [Reranking](/reranking) | Reordering results to improve the top-k. |
|
||||
| [Feature engineering](/feature-engineering) | Deriving columns and tables from data already stored. |
|
||||
| [Materialized views](/management/materialized-view) | A table defined by a query, brought up to date by each refresh. |
|
||||
| [Training](/training/data-loading) | Reading a table into a data loader. |
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
title: "Hybrid search and reranking on BEIR"
|
||||
sidebarTitle: "Hybrid Search on BEIR"
|
||||
description: "Combine semantic and keyword search on the BEIR benchmark, then rerank the results."
|
||||
---
|
||||
|
||||
Uses LanceDB's built-in [hybrid search](/search/hybrid-search), which combines the
|
||||
strengths of semantic and full-text search. Working against the BEIR dataset, it
|
||||
shows how searching for both the meaning of a query and the specific keywords in
|
||||
it produces more relevant results than either alone, and how a
|
||||
[reranker](/reranking) refines what comes back.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Inbuilt-Hybrid-Search/Inbuilt_Hybrid_Search_with_LanceDB.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/Inbuilt-Hybrid-Search">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,330 @@
|
||||
---
|
||||
title: "Finding the Needle in a Haystack: Comparing Multi-vector Search Strategies"
|
||||
sidebarTitle: "Needle in a Haystack"
|
||||
description: "Tutorial on token-level retrieval with LanceDB multivector search."
|
||||
---
|
||||
|
||||
|
||||
|
||||
In the development of advanced search and retrieval systems, moving from keyword matching to semantic understanding is a critical step. However, a key distinction exists between finding a relevant document and locating a specific piece of information within that document with precision. While there are techniques that perform well for retrieving documents, most of them work by extracting summarized semantic meaning of the document.
|
||||
|
||||
This can be seen as these models trying to understand the "gist" of the documents. Both single-vector search and late-interaction approaches work well for these conditions with various tradeoffs involved. But there's another type of problem where the goal is not just to understand the overall topic of a document in general, but to also specifically account for the requested detail within the document. This "needle in a haystack" problem is a significant challenge, and addressing it is essential for building high-precision retrieval systems.
|
||||
|
||||
This guide provides a technical analysis of multivector search for high-precision information retrieval. We will examine various optimization strategies and analyze their performance. This guide should be seen as complementary to resources like [this blog post by Answer.AI](https://www.answer.ai/posts/colbert-pooling.html) on ColBERT pooling, which explains how pooling strategies can be effective for document-level retrieval. Here, we will demonstrate why those same techniques can be counterproductive when precision at an intra-document, token level is the primary objective.
|
||||
|
||||
To reproduce the work below, see the code [here](https://github.com/lancedb/research/tree/main/multivector-needle-haystack-bench).
|
||||
|
||||
## The Dataset {#the-dataset}
|
||||
|
||||
This task is different from benchmarks like BEIR, which focus on text-based doc retrieval, finding the most relevant documents from a large collection. Here, we want *intra-document localization*, where the goal is to find a precise piece of information within a single, dense document, in a multimodal setting.
|
||||
|
||||
### The Task: The Document Haystack Dataset {#the-task-the-document-haystack-dataset}
|
||||
|
||||
Our benchmark is built on the **[AmazonScience/document-haystack](https://huggingface.co/datasets/AmazonScience/document-haystack)** dataset, which contains 25 visually complex source documents (e.g., financial reports, academic papers). To create a rigorous test, our evaluation follows a per-document methodology:
|
||||
|
||||
* We process each of the 25 source documents independently.
|
||||
* **Table Creation:** For a single source document (e.g., "AIG"), we ingest all pages from all of its page-length variants (from 5 to 200 pages long). This creates a temporary LanceDB table containing approximately 1,230 pages.
|
||||
* **The Task:** We then query this table using a set of "needle" questions, where the goal is to retrieve the **exact page number** containing the answer. A successful retrieval means the correct page number is within the top K results.
|
||||
* **Target metric:** We measure both retrieval accuracy (Hit@K) and the average search latency for each query against this table.
|
||||
|
||||
**Dataset example**
|
||||
The documents contain "text needles" like these
|
||||
|
||||
|
||||
|
||||
|
||||
During evaluation, the queries processed are somewhat like this:
|
||||
```
|
||||
What is the secret currency in the document?
|
||||
What is the secret object #3 in the document?
|
||||
```
|
||||
The intention of this task is to find the page which has the text needle that answers this questions
|
||||
|
||||
|
||||
## Models and Architectures {#models-and-architectures}
|
||||
|
||||
Our testbed includes a baseline single-vector model and a family of advanced multivector models.
|
||||
|
||||
### Single-Vector (Bi-Encoder) Baseline: CLIP ViT-Base-Patch32 {#single-vector-bi-encoder-baseline-clip-vit-base-patch32}
|
||||
A bi-encoder maps an entire piece of content (a query, a document page) to a *single* vector. The search process is simple: pre-compute one vector for every page, and at query time, find the page vector closest to the query vector.
|
||||
|
||||
* **Strength:** Speed and simplicity.
|
||||
* **Weakness:** This creates an **information bottleneck**. All the nuanced details, keywords, and semantic relationships on a page must be compressed into a single, fixed-size vector. For finding a needle, this is like trying to describe a specific person's face using only one word.
|
||||
|
||||
### Multi-vector (Late-Interaction) Models {#multi-vector-late-interaction-models}
|
||||
|
||||
Multi-vector models, pioneered by ColBERT, take a different approach. Instead of one vector per page, they generate a *set of vectors* for each page—one for every token (or image patch).
|
||||
|
||||
* **Mechanism (MaxSim):** The search process is more sophisticated. For each token in the query, the system finds the most similar token on the page. These maximum similarity scores are then summed up to get the final relevance score. This "late-interaction" preserves fine-grained, token-level details.
|
||||
* **The Models:** We used several vision-language models adapted for this architecture, including `ColPali`, `ColQwen2`, and `ColSmol`. While their underlying transformer backbones differ, they all share the ColBERT philosophy of representing documents as a bag of contextualized token embeddings.
|
||||
|
||||
|
||||
|
||||
## Different Retrieval Strategies Used {#different-retrieval-strategies-used}
|
||||
|
||||
A full multivector search is powerful but computationally intensive. Here are five strategies for managing it, complete with LanceDB implementation details.
|
||||
|
||||
|
||||
### 1\. `base`: The Gold Standard (Full Multi-vector Search) {#1-base-the-gold-standard-full-multi-vector-search}
|
||||
|
||||
This is the pure, baseline late-interaction search. It offers the highest potential for accuracy by considering every token.
|
||||
|
||||
**LanceDB also integrates with ConteXtualized Token Retriever (XTR)** , an advanced retrieval model that prioritizes the most semantically important document tokens during search. This integration enhances the quality of search results by focusing on the most relevant token matches.
|
||||
**LanceDB Implementation:**
|
||||
|
||||
```python Python icon="python"
|
||||
import lancedb
|
||||
import pyarrow as pa
|
||||
|
||||
# Schema for multivector data
|
||||
# Assumes embeddings are 128-dimensional
|
||||
schema = pa.schema([
|
||||
pa.field("page_num", pa.int32()),
|
||||
pa.field("vector", pa.list_(pa.list_(pa.float32(), 128)))
|
||||
])
|
||||
|
||||
db = lancedb.connect("./lancedb")
|
||||
tbl = db.create_table("document_pages_base", schema=schema)
|
||||
|
||||
# Ingesting multi-token embeddings for a page
|
||||
# multi_token_embeddings is a NumPy array of shape (num_tokens, 128)
|
||||
tbl.add([{"page_num": 1, "vector": multi_token_embeddings.tolist()}])
|
||||
|
||||
# Searching with a multi-token query
|
||||
# query_multi_vector is also shape (num_query_tokens, 128)
|
||||
results = tbl.search(query_multi_vector).limit(5).to_list()
|
||||
```
|
||||
|
||||
### 2\. `flatten`: Mean Pooling {#2-flatten-mean-pooling}
|
||||
|
||||
This strategy "flattens" the set of token vectors into a single vector by averaging them. This transforms the search into a standard, fast approximate nearest neighbor (ANN) search.
|
||||
|
||||
**LanceDB Implementation:**
|
||||
|
||||
```python Python icon="python"
|
||||
# Schema for single-vector data
|
||||
schema_flat = pa.schema([
|
||||
pa.field("page_num", pa.int32()),
|
||||
pa.field("vector", pa.list_(pa.float32(), 128))
|
||||
])
|
||||
tbl_flat = db.create_table("document_pages_flat", schema=schema_flat)
|
||||
|
||||
# Ingesting the mean-pooled vector
|
||||
mean_vector = multi_token_embeddings.mean(axis=0)
|
||||
tbl_flat.add([{"page_num": 1, "vector": mean_vector.tolist()}])
|
||||
|
||||
# Searching with a single averaged query vector
|
||||
query_mean_vector = query_multi_vector.mean(axis=0)
|
||||
results = tbl_flat.search(query_mean_vector).limit(5).to_list()
|
||||
```
|
||||
|
||||
### 3\. Max pooling {#3-max-pooling}
|
||||
|
||||
This is a variation of `flatten`. `max_pooling` takes the element-wise max across all token vectors instead of the mean. The implementation is identical to `flatten`, just with a different aggregation method (`.max(axis=0)`).
|
||||
|
||||
### 4\. `flatten and multivector rerank`: The Hybrid Optimization {#4-flatten-and-multivector-rerank-the-hybrid-optimization}
|
||||
This two-stage strategy aims for the best of both worlds. First, use a fast, pooled-vector search to find a set of promising candidates. Then, run the full, accurate multivector search on *only* those candidates.
|
||||
|
||||
**LanceDB Implementation:**
|
||||
This requires a table with two vector columns.
|
||||
|
||||
```python Python icon="python"
|
||||
# Schema with both flat and multivector columns
|
||||
schema_rerank = pa.schema([
|
||||
pa.field("page_num", pa.int32()),
|
||||
pa.field("vector_flat", pa.list_(pa.float32(), 128)),
|
||||
pa.field("vector_multi", pa.list_(pa.list_(pa.float32(), 128)))
|
||||
])
|
||||
tbl_rerank = db.create_table("document_pages_rerank", schema=schema_rerank)
|
||||
|
||||
# Ingest both vectors
|
||||
tbl_rerank.add([
|
||||
{
|
||||
"page_num": 1,
|
||||
"vector_flat": multi_token_embeddings.mean(axis=0).tolist(),
|
||||
"vector_multi": multi_token_embeddings.tolist()
|
||||
}
|
||||
])
|
||||
|
||||
# --- Two-Stage Search ---
|
||||
# Stage 1: Fast search on the flat vector
|
||||
query_flat = query_multi_vector.mean(axis=0)
|
||||
candidates = tbl_rerank.search(query_flat, vector_column_name="vector_flat") \
|
||||
.limit(100) \
|
||||
.with_row_id(True) \
|
||||
.to_pandas()
|
||||
|
||||
# Stage 2: Precise multivector search on candidates
|
||||
candidate_ids = tuple(candidates["_rowid"].to_list())
|
||||
final_results = tbl_rerank.search(query_multi_vector, vector_column_name="vector_multi") \
|
||||
.where(f"_rowid IN {candidate_ids}") \
|
||||
.limit(5) \
|
||||
.to_list()
|
||||
```
|
||||
|
||||
### 5\. `hierarchical token pooling`: Compressing the Haystack {#5-hierarchical-token-pooling-compressing-the-haystack}
|
||||
|
||||
This is an indexing-time strategy that aims to reduce the storage footprint and computational cost of multivector search by reducing the number of vectors per document. Instead of using every token vector, it clusters semantically similar tokens together and replaces them with a single, averaged vector.
|
||||
|
||||
* **Mechanism:** For each document, it computes the similarity between all token vectors, performs hierarchical clustering to group them, and then mean-pools the vectors within each cluster. This results in a smaller, more compact set of token vectors representing the document.
|
||||
* **Goal:** To reduce memory and disk usage while attempting to preserve the most important semantic information, potentially offering a middle ground between the high accuracy of `base` search and the speed of pooled methods.
|
||||
|
||||
**LanceDB Implementation:**
|
||||
The schema is identical to the `base` multivector search, but the data is pre-processed before ingestion.
|
||||
|
||||
```python Python icon="python"
|
||||
from utils import pool_embeddings_hierarchical
|
||||
import numpy as np
|
||||
|
||||
# Schema is the same as the base multivector schema
|
||||
schema = pa.schema([
|
||||
pa.field("page_num", pa.int32()),
|
||||
pa.field("vector", pa.list_(pa.list_(pa.float32(), 128)))
|
||||
])
|
||||
tbl_hierarchical = db.create_table("document_pages_hierarchical", schema=schema)
|
||||
|
||||
# Pool the embeddings before ingestion
|
||||
# multi_token_embeddings is a NumPy array of shape (num_tokens, 128)
|
||||
pooled_embeddings = pool_embeddings_hierarchical(
|
||||
multi_token_embeddings,
|
||||
pool_factor=4 # Reduce vector count by a factor of 4
|
||||
)
|
||||
|
||||
# Ingest the smaller set of multi-token embeddings
|
||||
# pooled_embeddings is now shape (approx. num_tokens / 4, 128)
|
||||
tbl_hierarchical.add([{"page_num": 1, "vector": pooled_embeddings.tolist()}])
|
||||
|
||||
# Search is identical to the base multivector search
|
||||
results = tbl_hierarchical.search(query_multi_vector).limit(5).to_list()
|
||||
```
|
||||
|
||||
## The Results {#the-results}
|
||||
|
||||
For a "needle in a haystack" task, retrieval accuracy is the primary metric of success. The benchmark results reveal a significant performance gap between the full multivector search strategy and common optimization techniques.
|
||||
|
||||
### Baseline Performance: Single-Vector Bi-Encoder {#baseline-performance-single-vector-bi-encoder}
|
||||
|
||||
First, we establish a baseline using a standard single-vector bi-encoder model, `openai/clip-vit-base-patch32`. This represents a common approach to semantic search but, as the data shows, is ill-suited for this task's precision requirements.
|
||||
|
||||
| Model | Strategy | Hit@1 | Hit@5 | Hit@20 | Avg. Latency (s) |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| `openai/clip-vit-base-patch32` | `base` | 1.6% | 4.7% | 11.8% | **0.008 s** |
|
||||
|
||||
With a Hit@20 rate of just under 12%, the baseline model struggles to reliably locate the correct page. This performance level is insufficient for applications requiring high precision.
|
||||
|
||||
### Multi-vector Model Performance {#multi-vector-model-performance}
|
||||
|
||||
We now examine the performance of multivector models using different strategies. The following table compares the `base` (full multivector), `flatten` (mean pooling), and `rerank` (hybrid) strategies across several late-interaction models.
|
||||
|
||||
| Model | Strategy | Hit@1 | Hit@5 | Hit@20 | Avg. Latency (s) |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| `vidore/colqwen2-v1.0` | `flatten` | 1.9% | 5.5% | 11.9% | 0.010 s |
|
||||
| `vidore/colqwen2-v1.0` | `flatten and multivector rerank` | 0.3% | 1.5% | 7.3% | 0.692 s |
|
||||
| **`vidore/colqwen2-v1.0`** | **`hierarchical token pooling`** | **13.7%** | **60.5%** | **91.6%** | **0.144 s** |
|
||||
| **`vidore/colqwen2-v1.0`** | **`base`** | **14.0%** | **65.4%** | **95.5%** | 0.668 s |
|
||||
| `vidore/colpali-v1.3` | `flatten` | 1.7% | 4.5% | 9.3% | 0.008 s |
|
||||
| `vidore/colpali-v1.3` | `flatten and multivector rerank` | 0.6% | 2.3% | 6.9% | 0.949 s |
|
||||
| **`vidore/colpali-v1.3`** | **`hierarchical token pooling`** | **10.8%** | **41.7%** | **64.8%** | **0.189 s** |
|
||||
| **`vidore/colpali-v1.3`** | **`base`** | **11.3%** | **42.3%** | **65.6%** | 0.936 s |
|
||||
| `vidore/colSmol-256M` | `flatten` | 1.6% | 4.7% | 10.5% | 0.008 s |
|
||||
| `vidore/colSmol-256M` | `flatten and multivector rerank` | 0.3% | 1.6% | 7.0% | 0.853 s |
|
||||
| **`vidore/colSmol-256M`** | **`base`** | **14.4%** | **64.0%** | **91.7%** | 0.848 s |
|
||||
|
||||
The data shows a consistent pattern: the `base` strategy outperforms all other techniques. The flattned pooling and reranking strategies perform no better than the single-vector baseline. However, hierarchical token pooling seems like a decent alternative to base considering speed vs accuracy tradeoff. Let's look at the numbers in detail.
|
||||
|
||||
### In-Depth Analysis of Pooling Strategies {#in-depth-analysis-of-pooling-strategies}
|
||||
|
||||
To further understand the failure of optimization techniques, we compared different methods for pooling token vectors into a single vector: `mean` (`flatten`), `max`.
|
||||
|
||||
| Model & Pooling Strategy | Hit@1 | Hit@5 | Hit@20 | Avg. Latency (s) |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| `vidore/colqwen2-v1.0` (`mean_pooling`) | 1.9% | 5.5% | 11.9% | 0.010 s |
|
||||
| `vidore/colqwen2-v1.0` (`max_pooling`) | 1.4% | 4.2% | 11.2% | 0.011 s |
|
||||
| **`vidore/colqwen2-v1.0` (`base`)** | **14.0%** | **65.4%** | **95.5%** | 0.668 s |
|
||||
|
||||
All flattened pooling methods perform poorly, confirming that the aggregation of token vectors into a single representation loses the fine-grained detail required for this task.
|
||||
|
||||
**Finding the Right Trade-Off**
|
||||
|
||||
|
||||
|
||||
1. **The Failure of Simple Pooling:** The `flatten` (mean pooling) and `max_pooling` strategies fail to improve upon the baseline. This is because their aggressive compression **destroys the essential localization signal**. The resulting single vector represents the *topic* of the page, not the *specific needle* on it.
|
||||
2. **The Failure of `flatten and multivector rerank`:** This hybrid strategy is the *worst-performing* of all. The reason is a fundamental flaw in its design for this task: the first stage uses a simple pooled vector to retrieve candidates. Since this pooling eliminates the localization signal, the initial candidate set is effectively random.
|
||||
3. **`hierarchical token pooling`:** By clustering and pooling tokens at indexing time, it reduces the number of vectors per page (in our case, by a factor of 4). This intelligently compresses the data, while preserving enough token-level detail in multivector setting. It achieves a **Hit@20 of 91.6%**, only slightly behind the `base` strategy's 95.5%, but is significantly faster.
|
||||
4. **`base` multivector Search:** The vanilla, un-optimized `base` multivector search remains the most accurate strategy. Preserving every token vector provides the highest guarantee of finding the needle, but this comes at the highest computational cost.
|
||||
|
||||
### Latency: {#latency}
|
||||
|
||||
|
||||
|
||||
|
||||
The "optimizations" are not all created equal. While simple pooling is fast, its inaccuracy makes it unusable. Hierarchical pooling, however, offers a compelling balance of speed and accuracy.
|
||||
|
||||
| Strategy (on `vidore/colqwen2-v1.0`) | Avg. Search Latency (s) | Hit@20 Accuracy |
|
||||
| :--- | :--- | :--- |
|
||||
| `flatten` (Fast but Ineffective) | **0.010 s** | 11.9% |
|
||||
| `flatten and multivector rerank` (Slower and Ineffective) | 0.692 s | 7.3% |
|
||||
| **`hierarchical token pooling` (Accurate & Fast)** | **0.144 s** | **91.6%** |
|
||||
| `base` (Most Accurate) | 0.668 s | **95.5%** |
|
||||
_Latency reported is as seen on NVIDIA H100 GPUs_
|
||||
|
||||
## Practical Considerations {#practical-considerations}
|
||||
|
||||
The accuracy of `base` multivector search is impressive, but its computational intensity has historically limited its use. `hierarchical token pooling` as a viable strategy creates a new, practical sweet spot on the accuracy-latency curve, making high-precision search accessible for a wider range of applications.
|
||||
|
||||
### Search Latency and Computational Complexity {#search-latency-and-computational-complexity}
|
||||
|
||||
|
||||
As the benchmark data shows, the search latency for `base` multivector search is orders of magnitude higher than for single-vector (or pooled-vector) search. It's important to note that the reported ~670ms latency is an average from per-document evaluations. In this benchmark, each of the 25 documents is processed independently. All pages from a single document's variants (ranging from 5 to 200 pages) are ingested into a temporary table, resulting in a table size of approximately **1,230 rows (pages)** per evaluation. The search is performed on this table, and then the table is discarded. This highlights a significant performance cost even on a relatively small, per-document scale. This stems from a fundamental difference in computational complexity:
|
||||
|
||||
* **Modern ANN Search (for single vectors):** Algorithms like HNSW (Hierarchical Navigable Small World) provide sub-linear search times, often close to `O(log N)`, where `N` is the number of items in the index. This allows them to scale to billions of vectors with millisecond-level latency.
|
||||
* **Late-Interaction Search (Multi-vector):** The search process is far more intensive. For each query, it must compute similarity scores between query tokens and the tokens of many candidate documents. The complexity is closer to `O(M * Q * D)`, where `M` is the number of candidate documents to score, `Q` is the number of query tokens, and `D` is the average number of tokens per document. `Hierarchical token pooling` directly attacks this problem by reducing `D`, leading to a significant reduction in search latency.
|
||||
|
||||
### When to Use Multi-Vector Search {#when-to-use-multi-vector-search}
|
||||
|
||||
Given these constraints, the choice of strategy depends on the specific requirements of the application.
|
||||
|
||||
* **For Maximum Precision (`base`):** In domains where the cost of missing the needle is extremely high, the full `base` search is the most reliable option.
|
||||
* **For a Balance of Precision and Performance (`hierarchical token pooling`):** This is the ideal choice for many applications. It makes high-precision search practical for larger datasets and more interactive use cases where the sub-second latency of the `base` search may be too high. It significantly lowers the barrier to entry for adopting multivector search. It should still not be seen as a drop-in replacement for ANN, as it still requires more computational resources than single-vector search.
|
||||
* **For General-Purpose Document Retrieval (`flatten` / single-vector):** For large-scale retrieval where understanding the "gist" is sufficient or where in cases where large-context text-based models suffice, single-vector search remains the most practical and scalable solution.
|
||||
|
||||
## Appendix: Full Benchmark Results {#appendix-full-benchmark-results}
|
||||
|
||||
The full benchmark results are shown below.
|
||||
|
||||
```text
|
||||
| name | _runtime | _step | _timestamp | _wandb | avg_inference_latency | avg_search_latency | hit_rates | model_name | strategy |
|
||||
|:--------------------------------------------|-----------:|--------:|--------------:|:-------------------|------------------------:|---------------------:|:------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------|:---------------------|
|
||||
| vidore/colqwen2-v0.1_base | 13410 | 0 | 1.75873e+09 | {'runtime': 13410} | 0.0418646 | 0.751151 | {'1': 0.1355151515151515, '10': 0.888, '20': 0.9597575757575758, '3': 0.3936969696969697, '5': 0.6349090909090909} | vidore/colqwen2-v0.1 | base |
|
||||
| vidore/colqwen2-v1.0_rerank | 13008 | 0 | 1.75873e+09 | {'runtime': 13008} | 0.0426252 | 0.692482 | {'1': 0.003393939393939394, '10': 0.03296969696969697, '20': 0.07296969696969698, '3': 0.010666666666666666, '5': 0.015272727272727271} | vidore/colqwen2-v1.0 | rerank |
|
||||
| vidore/colpali-v1.3_flatten | 2998 | 0 | 1.75872e+09 | {'runtime': 2998} | 0.0296511 | 0.00833224 | {'1': 0.017454545454545455, '10': 0.06448484848484848, '20': 0.09333333333333334, '3': 0.034666666666666665, '5': 0.04509090909090909} | vidore/colpali-v1.3 | flatten |
|
||||
| vidore/colqwen2-v0.1_rerank | 13512 | 0 | 1.75873e+09 | {'runtime': 13512} | 0.0418855 | 0.662372 | {'1': 0.002909090909090909, '10': 0.026424242424242423, '20': 0.05987878787878788, '3': 0.0075151515151515155, '5': 0.014545454545454544} | vidore/colqwen2-v0.1 | rerank |
|
||||
| vidore/colqwen2-v1.0_base | 12933 | 0 | 1.75873e+09 | {'runtime': 12933} | 0.0416839 | 0.667898 | {'1': 0.14012121212121212, '10': 0.8846060606060606, '20': 0.9553939393939394, '3': 0.4111515151515152, '5': 0.6538181818181819} | vidore/colqwen2-v1.0 | base |
|
||||
| vidore/colpali-v1.3_rerank | 12090 | 0 | 1.75873e+09 | {'runtime': 12090} | 0.0310783 | 0.949047 | {'1': 0.006060606060606061, '10': 0.03878787878787879, '20': 0.06933333333333333, '3': 0.014545454545454544, '5': 0.022787878787878788} | vidore/colpali-v1.3 | rerank |
|
||||
| vidore/colqwen2-v1.0_flatten | 4846 | 0 | 1.75874e+09 | {'runtime': 4846} | 0.0504031 | 0.010443 | {'1': 0.018666666666666668, '10': 0.07854545454545454, '20': 0.11903030303030304, '3': 0.041212121212121214, '5': 0.055030303030303034} | vidore/colqwen2-v1.0 | flatten |
|
||||
| vidore/colqwen2-v0.1_base | 10838 | 0 | 1.75874e+09 | {'runtime': 10838} | 0.04627 | 0.692836 | {'1': 0.13575757575757577, '10': 0.8870303030303031, '20': 0.96, '3': 0.3941818181818182, '5': 0.6351515151515151} | vidore/colqwen2-v0.1 | base |
|
||||
| vidore/colqwen2-v0.1_flatten | 4828 | 0 | 1.75874e+09 | {'runtime': 4828} | 0.0486335 | 0.0103028 | {'1': 0.018424242424242423, '10': 0.07224242424242425, '20': 0.10521212121212122, '3': 0.03903030303030303, '5': 0.05212121212121213} | vidore/colqwen2-v0.1 | flatten |
|
||||
| vidore/colpali-v1.3_base | 10253 | 0 | 1.75874e+09 | {'runtime': 10253} | 0.0312334 | 0.93611 | {'1': 0.11272727272727272, '10': 0.551030303030303, '20': 0.6555151515151515, '3': 0.29987878787878786, '5': 0.4232727272727273} | vidore/colpali-v1.3 | base |
|
||||
| vidore/colqwen2-v0.1_flatten | 4745 | 0 | 1.75874e+09 | {'runtime': 4745} | 0.0472825 | 0.00990508 | {'1': 0.018424242424242423, '10': 0.07296969696969698, '20': 0.10496969696969696, '3': 0.03951515151515152, '5': 0.05236363636363636} | vidore/colqwen2-v0.1 | flatten |
|
||||
| vidore/colqwen2.5-v0.2_base | 17218 | 0 | 1.75875e+09 | {'runtime': 17218} | 0.0540356 | 0.694855 | {'1': 0.11903030303030304, '10': 0.7127272727272728, '20': 0.8366060606060606, '3': 0.336, '5': 0.5258181818181819} | vidore/colqwen2.5-v0.2 | base |
|
||||
| vidore/colqwen2.5-v0.2_rerank | 16859 | 0 | 1.75875e+09 | {'runtime': 16859} | 0.0518383 | 0.71693 | {'1': 0.0026666666666666666, '10': 0.025212121212121213, '20': 0.060848484848484846, '3': 0.006787878787878788, '5': 0.01187878787878788} | vidore/colqwen2.5-v0.2 | rerank |
|
||||
| vidore/colqwen2-v0.1_rerank | 9484 | 0 | 1.75875e+09 | {'runtime': 9484} | 0.0445599 | 0.691609 | {'1': 0.005333333333333333, '10': 0.030545454545454542, '20': 0.064, '3': 0.010666666666666666, '5': 0.017696969696969697} | vidore/colqwen2-v0.1 | rerank |
|
||||
| vidore/colSmol-256M_flatten | 6822 | 0 | 1.75875e+09 | {'runtime': 6822} | 0.0404544 | 0.00822329 | {'1': 0.01575757575757576, '10': 0.06836363636363636, '20': 0.10496969696969696, '3': 0.03442424242424243, '5': 0.04678787878787879} | vidore/colSmol-256M | flatten |
|
||||
| vidore/colSmol-500M_base | 12681 | 0 | 1.75875e+09 | {'runtime': 12681} | 0.0408026 | 0.850902 | {'1': 0.136, '10': 0.8029090909090909, '20': 0.8993939393939394, '3': 0.3806060606060606, '5': 0.5975757575757575} | vidore/colSmol-500M | base |
|
||||
| vidore/colSmol-500M_rerank | 13066 | 0 | 1.75876e+09 | {'runtime': 13066} | 0.0440974 | 0.927632 | {'1': 0.003636363636363637, '10': 0.028606060606060607, '20': 0.07054545454545455, '3': 0.00896969696969697, '5': 0.015030303030303033} | vidore/colSmol-500M | rerank |
|
||||
| vidore/colSmol-256M_rerank | 12646 | 0 | 1.75876e+09 | {'runtime': 12646} | 0.0391553 | 0.853279 | {'1': 0.003393939393939394, '10': 0.02909090909090909, '20': 0.07006060606060606, '3': 0.008727272727272728, '5': 0.015515151515151517} | vidore/colSmol-256M | rerank |
|
||||
| vidore/colqwen2.5-v0.2_flatten | 6348 | 0 | 1.75876e+09 | {'runtime': 6348} | 0.0509971 | 0.00772653 | {'1': 0.017696969696969697, '10': 0.06545454545454546, '20': 0.09284848484848485, '3': 0.03515151515151515, '5': 0.045575757575757575} | vidore/colqwen2.5-v0.2 | flatten |
|
||||
| vidore/colSmol-256M_base | 11554 | 0 | 1.75876e+09 | {'runtime': 11554} | 0.0366467 | 0.848463 | {'1': 0.1435151515151515, '10': 0.8426666666666667, '20': 0.9173333333333332, '3': 0.40824242424242424, '5': 0.6404848484848484} | vidore/colSmol-256M | base |
|
||||
| vidore/colSmol-500M_flatten | 6395 | 0 | 1.75876e+09 | {'runtime': 6395} | 0.0384664 | 0.00716238 | {'1': 0.018424242424242423, '10': 0.07345454545454545, '20': 0.11393939393939394, '3': 0.03903030303030303, '5': 0.05090909090909091} | vidore/colSmol-500M | flatten |
|
||||
| openai/clip-vit-base-patch32_base | 815 | 0 | 1.75876e+09 | {'runtime': 815} | 0.00533487 | 0.00794629 | {'1': 0.016, '10': 0.07636363636363637, '20': 0.11757575757575756, '3': 0.03296969696969697, '5': 0.04703030303030303} | openai/clip-vit-base-patch32 | base |
|
||||
| vidore/colqwen2-v0.1_max_pooling | 8758 | 0 | 1.7595e+09 | {'runtime': 8758} | 0.0985753 | 0.0106716 | {'1': 0.015515151515151517, '10': 0.07296969696969698, '20': 0.11248484848484848, '3': 0.032484848484848484, '5': 0.04703030303030303} | vidore/colqwen2-v0.1 | max_pooling |
|
||||
| vidore/colpali-v1.3_max_pooling | 4728 | 0 | 1.75949e+09 | {'runtime': 4728} | 0.0718573 | 0.0103053 | {'1': 0.011393939393939394, '10': 0.05672727272727273, '20': 0.08872727272727272, '3': 0.02666666666666667, '5': 0.03709090909090909} | vidore/colpali-v1.3 | max_pooling |
|
||||
| vidore/colqwen2-v1.0_max_pooling | 8760 | 0 | 1.7595e+09 | {'runtime': 8760} | 0.0981102 | 0.0106316 | {'1': 0.013575757575757576, '10': 0.06933333333333333, '20': 0.112, '3': 0.02812121212121212, '5': 0.041939393939393936} | vidore/colqwen2-v1.0 | max_pooling |
|
||||
| vidore/colqwen2-v0.1_max_pooling | 7696 | 0 | 1.7595e+09 | {'runtime': 7696} | 0.105203 | 0.011907 | {'1': 0.016242424242424242, '10': 0.07321212121212121, '20': 0.1132121212121212, '3': 0.03442424242424243, '5': 0.04945454545454545} | vidore/colqwen2-v0.1 | max_pooling |
|
||||
| vidore/colSmol-256M_max_pooling | 13982 | 0 | 1.75951e+09 | {'runtime': 13982} | 0.0999708 | 0.0121211 | {'1': 0.00993939393939394, '10': 0.05818181818181818, '20': 0.09187878787878788, '3': 0.025212121212121213, '5': 0.037575757575757575} | vidore/colSmol-256M | max_pooling |
|
||||
| vidore/colSmol-500M_max_pooling | 14191 | 0 | 1.75951e+09 | {'runtime': 14191} | 0.111858 | 0.0121703 | {'1': 0.012363636363636365, '10': 0.07539393939393939, '20': 0.13187878787878787, '3': 0.02787878787878788, '5': 0.0416969696969697} | vidore/colSmol-500M | max_pooling |
|
||||
| vidore/colqwen2-v0.1_hierarchical_pooling | 4507 | 0 | 1.7599e+09 | {'runtime': 4507} | 0.0320023 | 0.133653 | {'1': 0.1296969696969697, '10': 0.8504242424242424, '20': 0.9343030303030304, '3': 0.37527272727272726, '5': 0.6041212121212122} | vidore/colqwen2-v0.1 | hierarchical_pooling |
|
||||
| vidore/colpali-v1.3_hierarchical_pooling | 5816 | 0 | 1.7599e+09 | {'runtime': 5816} | 0.0217625 | 0.188727 | {'1': 0.10763636363636364, '10': 0.5372121212121213, '20': 0.6482424242424243, '3': 0.29333333333333333, '5': 0.4167272727272727} | vidore/colpali-v1.3 | hierarchical_pooling |
|
||||
| vidore/colqwen2.5-v0.2_hierarchical_pooling | 8430 | 0 | 1.75991e+09 | {'runtime': 8430} | 0.043073 | 0.141276 | {'1': 0.11103030303030303, '10': 0.6892121212121212, '20': 0.8203636363636364, '3': 0.3185454545454545, '5': 0.4989090909090909} | vidore/colqwen2.5-v0.2 | hierarchical_pooling |
|
||||
| vidore/colqwen2-v1.0_hierarchical_pooling | 5685 | 0 | 1.75991e+09 | {'runtime': 5685} | 0.0343824 | 0.144062 | {'1': 0.13745454545454547, '10': 0.822060606060606, '20': 0.9156363636363636, '3': 0.3856969696969697, '5': 0.6050909090909091} | vidore/colqwen2-v1.0 | hierarchical_pooling |
|
||||
```
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: "Multi-vector search with XTR"
|
||||
sidebarTitle: "Multi-vector XTR"
|
||||
description: "Represent a document with many vectors, and speed retrieval with ConteXtualized Token Retrieval."
|
||||
---
|
||||
|
||||
Covers [multi-vector search](/search/vector-search) enhanced by XTR
|
||||
(ConteXtualized Token Retriever): representing complex data with several vectors
|
||||
for more nuanced meaning, and using XTR to speed retrieval by prioritizing the
|
||||
most important tokens.
|
||||
|
||||
For a comparison of retrieval strategies on a token-level benchmark, see
|
||||
[Needle in a haystack](/tutorials/search/multivector-needle-in-a-haystack).
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multivector_xtr/main.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/multivector_xtr">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "NER-powered vector search"
|
||||
sidebarTitle: "NER Search"
|
||||
description: "Extract named entities and embed them to sharpen retrieval."
|
||||
---
|
||||
|
||||
Uses Named Entity Recognition to power vector search. Extracting entities --
|
||||
people, places, organizations -- and embedding them alongside the text improves
|
||||
accuracy when queries are about specific things rather than general topics.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/NER-powered-Semantic-Search/NER_powered_Semantic_Search_with_LanceDB.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/NER-powered-Semantic-Search">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Reddit concept summarizer"
|
||||
sidebarTitle: "Reddit Summarizer"
|
||||
description: "An end-to-end pipeline from Reddit text to embeddings to summarization and search."
|
||||
---
|
||||
|
||||
A complete pipeline: acquire text from Reddit, turn it into embeddings, store
|
||||
and manage those vectors in LanceDB, then build summarization and semantic
|
||||
search on top. Useful as a shape for any ingest-embed-serve workflow.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Reddit-summarization-and-search/subreddit_summarization_querying.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/Reddit-summarization-and-search">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: "Semantic result merging"
|
||||
sidebarTitle: "Vector Arithmetic"
|
||||
description: "Add and subtract embeddings to steer a query toward or away from a concept."
|
||||
---
|
||||
|
||||
Explores vector arithmetic: manipulating embeddings to capture relationships a
|
||||
single query vector cannot. Adding or subtracting the vector representation of a
|
||||
concept shifts the query, which makes searches more nuanced than a literal
|
||||
similarity lookup.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Vector-Arithmetic-with-LanceDB/main.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/Vector-Arithmetic-with-LanceDB">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: "Semantic search across videos"
|
||||
sidebarTitle: "Video Search"
|
||||
description: "Embed video with V-JEPA and search it by similarity."
|
||||
---
|
||||
|
||||
Builds a video search application on V-JEPA (Video Joint Embedding Predictive
|
||||
Architecture). Generates vector embeddings for video content and uses
|
||||
[vector search](/search/vector-search) to find visually similar clips -- a
|
||||
multimodal case where the searchable column is not text.
|
||||
|
||||
<Note>
|
||||
This example is a runnable notebook in the
|
||||
[vectordb-recipes](https://github.com/lancedb/vectordb-recipes) repository rather than a page here.
|
||||
</Note>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Open in Colab" icon="google" href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/v-jepa-video-search/intra-video.ipynb">
|
||||
Run it end to end in the browser.
|
||||
</Card>
|
||||
<Card title="View on GitHub" icon="github" href="https://github.com/lancedb/vectordb-recipes/tree/main/examples/v-jepa-video-search">
|
||||
Read the source.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
Reference in New Issue
Block a user