Passing a record batch created by an ESM `apache-arrow` import to
`RRFReranker.rerankHybrid` throws an IPC footer parse error. The SDK's
CommonJS Arrow writer receives data types from another module instance.
Normalize single-record-batch input with the existing table sanitizer
before writing IPC. Extend the Arrow 15–21 round-trip test with
caller-created batches and nulls, and add a native ESM reranker
regression covering 64-bit row IDs, scores, and an empty FTS side.
Validation:
- Reproduced with published `0.42.0-beta.3`; CommonJS and explicitly
normalized ESM controls pass.
- Same native module and inputs: all 8 targeted cases fail with the
parent serializer and pass with this change.
- Local native build, TypeScript build, lint, and formatting pass.
- Current-head CI passes on Linux Node 22/24/26, macOS, and Arrow 19/20.
The macOS run passes 1,643 tests; Linux Node 24 also passes docs and
example checks.
Fixes#3269.
## What I observed
Using a reranker in a hybrid query could keep the Node.js process alive
even after `table.close()` and `db.close()`.
## Root cause
The reranker callback bridge used a `ThreadsafeFunction` in referenced
mode, which can keep the event loop alive longer than intended.
## Minimal fix
- In `nodejs/src/rerankers.rs`, create the reranker callback TSFN in
weak mode (`.weak::<true>()`).
- Add a regression test in `nodejs/__test__/rerankers.test.ts` that
spawns a child process, runs a rerank query, and asserts the process
exits naturally.
## Validation
- Built Node bindings successfully.
- Ran targeted tests: `rerankers.test.ts` passes (including new
regression test).
- Pre-commit checks for changed files were run and clean.
Support hybrid search in both rust and node SDKs.
- Adds a new rerankers package to rust LanceDB, with the implementation
of the default RRF reranker
- Adds a new hybrid package to lancedb, with some helper methods related
to hybrid search such as normalizing scores and converting score column
to rank columns
- Adds capability to LanceDB VectorQuery to perform hybrid search if it
has both a nearest vector and full text search parameters.
- Adds wrappers for reranker implementations to nodejs SDK.
Additional rerankers will be added in followup PRs
https://github.com/lancedb/lancedb/issues/1921
---
Notes about how the rust rerankers are wrapped for calling from JS:
I wanted to keep the core reranker logic, and the invocation of the
reranker by the query code, in Rust. This aligns with the philosophy of
the new node SDK where it's just a thin wrapper around Rust. However, I
also wanted to have support for users who want to add custom rerankers
written in Javascript.
When we add a reranker to the query from Javascript, it adds a special
Rust reranker that has a callback to the Javascript code (which could
then turn around and call an underlying Rust reranker implementation if
desired). This adds a bit of complexity, but overall I think it moves us
in the right direction of having the majority of the query logic in the
underlying Rust SDK while keeping the option open to support custom
Javascript Rerankers.