From 51966a84f582476566daf289589318e9cc2cd0a1 Mon Sep 17 00:00:00 2001 From: Ayush Chaurasia Date: Sat, 31 Aug 2024 04:09:14 +0530 Subject: [PATCH] docs: add multi-vector reranking, answerdotai and studies section (#1579) --- docs/mkdocs.yml | 7 +++++++ docs/src/reranking/index.md | 17 +++++++++++++++++ docs/src/studies/overview.md | 4 ++++ 3 files changed, 28 insertions(+) create mode 100644 docs/src/studies/overview.md diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 77dcb19d..0230caef 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -108,6 +108,7 @@ nav: - ColBERT Reranker: reranking/colbert.md - Jina Reranker: reranking/jina.md - OpenAI Reranker: reranking/openai.md + - AnswerDotAi Rerankers: reranking/answerdotai.md - Building Custom Rerankers: reranking/custom_reranker.md - Example: notebooks/lancedb_reranking.ipynb - Filtering: sql.md @@ -179,6 +180,8 @@ nav: - TransformersJS Embedding Search: examples/transformerjs_embedding_search_nodejs.md - 🦀 Rust: - Overview: examples/examples_rust.md + - Studies: + - ↗Improve retrievers with hybrid search and reranking: https://blog.lancedb.com/hybrid-search-and-reranking-report/ - 💭 FAQs: faq.md - ⚙️ API reference: - 🐍 Python: python/python.md @@ -219,6 +222,7 @@ nav: - ColBERT Reranker: reranking/colbert.md - Jina Reranker: reranking/jina.md - OpenAI Reranker: reranking/openai.md + - AnswerDotAi Rerankers: reranking/answerdotai.md - Building Custom Rerankers: reranking/custom_reranker.md - Example: notebooks/lancedb_reranking.ipynb - Filtering: sql.md @@ -286,6 +290,9 @@ nav: - TransformersJS Embedding Search: examples/transformerjs_embedding_search_nodejs.md - 🦀 Rust: - Overview: examples/examples_rust.md + - Studies: + - studies/overview.md + - ↗Improve retrievers with hybrid search and reranking: https://blog.lancedb.com/hybrid-search-and-reranking-report/ - API reference: - Overview: api_reference.md - Python: python/python.md diff --git a/docs/src/reranking/index.md b/docs/src/reranking/index.md index d2a15d6b..43e2c555 100644 --- a/docs/src/reranking/index.md +++ b/docs/src/reranking/index.md @@ -45,6 +45,23 @@ tbl.create_fts_index("text") result = tbl.query("hello", query_type="hybrid").rerank(reranker).to_list() ``` +### Multi-vector reranking +Most rerankers support reranking based on multiple vectors. To rerank based on multiple vectors, you can pass a list of vectors to the `rerank` method. Here's an example of how to rerank based on multiple vector columns using the `CrossEncoderReranker`: + +```python +from lancedb.rerankers import CrossEncoderReranker + +reranker = CrossEncoderReranker() + +query = "hello" + +res1 = table.search(query, vector_column_name="vector").limit(3) +res2 = table.search(query, vector_column_name="text_vector").limit(3) +res3 = table.search(query, vector_column_name="meta_vector").limit(3) + +reranked = reranker.rerank_multivector([res1, res2, res3], deduplicate=True) +``` + ## Available Rerankers LanceDB comes with some built-in rerankers. Here are some of the rerankers that are available in LanceDB: diff --git a/docs/src/studies/overview.md b/docs/src/studies/overview.md new file mode 100644 index 00000000..917f39c3 --- /dev/null +++ b/docs/src/studies/overview.md @@ -0,0 +1,4 @@ +This is a list of benchmarks and reports we've worked on at LanceDB. Some of these are continuously updated, while others are one-off reports. + +- [Improve retrievers with hybrid search and reranking](https://blog.lancedb.com/hybrid-search-and-reranking-report/) +