mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-15 00:02:26 +00:00
fix(python): rank each result list separately in RRF multivector rerank (#4325)
## What's wrong
`RRFReranker.rerank_multivector` concatenates every vector result list
and passes the concatenation to `rerank_hybrid` as a single list.
`rerank_hybrid` then scores each row by its position in that
concatenation, so a row's rank depends on how many rows came before it
in *earlier* lists.
With two searches of 3 rows each (`K=60`):
| row | rank in its own list | score today | RRF score |
|---|---|---|---|
| 1 (top of list 1) | 1 | 1/61 | 1/61 |
| 3 (third of list 1) | 3 | 1/63 | 1/63 |
| 4 (top of list 2) | 1 | **1/64** | 1/61 |
| 5 (second of list 2) | 2 | **1/65** | 1/62 |
So the best hit of the second search ranks below the worst hit of the
first, and the order of the lists passed in decides the fused ranking.
`MRRReranker.rerank_multivector` already scores each list on its own
("MRR semantics require treating each vector result as a separate
ranking system"). RRF is defined the same way.
## Fix
Score each result list with its own ranks (`1 / (rank + K)`, summed
across lists for rows found by more than one), then dedupe the
concatenation by `_rowid`, attach `_relevance_score`, and sort. The
`return_score` handling is unchanged.
## Tests
`test_rrf_multivector_ranks_each_result_list` in
`python/python/tests/test_rerankers.py`:
- On `main`, it fails with `assert 0.015625 == 0.01639344262295082` (the
top hit of list 2 got `1/64` instead of `1/61`).
- With the fix, it passes, and so do the other tests in
`test_rerankers.py` (16 passed, 12 skipped for missing API keys or
models).
- `ruff format --check` and `ruff check` are clean.
The fix was found and written by an AI agent (Breken) and checked
against the tests above.
Co-authored-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
This commit is contained in:
@@ -395,6 +395,38 @@ def test_mrr_multivector_rewards_consensus():
|
||||
assert result["_rowid"].to_pylist()[0] == 2
|
||||
|
||||
|
||||
def test_rrf_multivector_ranks_each_result_list():
|
||||
# RRF scores a document by its rank *within each* result list. The top hit
|
||||
# of the second list must score the same as the top hit of the first list,
|
||||
# not as if it were ranked after every row of the first list.
|
||||
reranker = RRFReranker(K=60)
|
||||
|
||||
def ranking(row_ids):
|
||||
return pa.table({"_rowid": pa.array(row_ids, type=pa.int64())})
|
||||
|
||||
rs1 = ranking([1, 2, 3])
|
||||
rs2 = ranking([4, 5, 6])
|
||||
|
||||
result = reranker.rerank_multivector([rs1, rs2])
|
||||
scores = dict(
|
||||
zip(result["_rowid"].to_pylist(), result["_relevance_score"].to_pylist())
|
||||
)
|
||||
|
||||
assert scores[1] == pytest.approx(1 / 61)
|
||||
assert scores[4] == pytest.approx(1 / 61)
|
||||
assert scores[5] == pytest.approx(1 / 62)
|
||||
# the second-ranked hit of the second list beats the third hit of the first
|
||||
assert scores[5] > scores[3]
|
||||
|
||||
# A document found by both lists sums its reciprocal ranks from each list.
|
||||
result = reranker.rerank_multivector([ranking([1, 2]), ranking([2, 3])])
|
||||
scores = dict(
|
||||
zip(result["_rowid"].to_pylist(), result["_relevance_score"].to_pylist())
|
||||
)
|
||||
assert scores[2] == pytest.approx(1 / 62 + 1 / 61)
|
||||
assert result["_rowid"].to_pylist() == [2, 1, 3]
|
||||
|
||||
|
||||
def test_rrf_reranker_distance():
|
||||
data = pa.table(
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user