## Summary
`tokenizer` is a documented option, but initialization always passed the
model name to `AutoTokenizer.from_pretrained`. Preserve the resolved
tokenizer option, fall back to the model name when it is omitted, and
identify the selected tokenizer in loading errors.
Add five initialization regressions covering default model/tokenizer
selection, an explicit tokenizer, registry variables, and tokenizer
loading errors. They run in native Node child processes so the real ESM
import works without changing Jest/CI flags. Model loaders are mocked
and local/remote model access is disabled; these tests do not download
models or test inference.
## Validation
- Before: three regressions fail and two default-selection controls
pass. After: all five pass.
- New tests plus the existing registry and embedding-registry suites: 48
passed.
- TypeScript compilation, changed-file Biome checks, and `pnpm run docs`
pass; generated tracked docs are unchanged.
- Checked with Node 24.14.1 / pnpm 11.1.1 on Linux, both with default
Jest flags and the Linux CI ESM flag. The native binding and
declarations came from the official `0.40.0-beta.13` packages, with
registry integrity verified. Rebuilding Rust from source timed out
during Cargo metadata resolution, before compilation; no Rust code is
changed.
OAuth tests set `LANCEDB_OAUTH_BROWSER` inside Jest's sandbox, which
does not update the process environment read by Rust. As a result, the
native login flow launches a real browser; the [failing macOS main
job](https://github.com/lancedb/lancedb/actions/runs/35067074667/job/104699667840)
ends by terminating an orphaned Safari process.
Set the no-op browser helper while loading Jest configuration, before
creating sandboxes and workers, so both local and CI tests inherit it.
Check the inherited process environment before OAuth login to catch
regressions without opening a browser. Windows uses a no-op command
fixture. Test deadlines and worker counts are unchanged.
A negative control restoring the sandbox-only assignment fails in the
new pre-login check. The full macOS test suite passes with the standard
test launcher; the Windows helper has not been executed locally. The
[first hosted macOS
run](https://github.com/lancedb/lancedb/actions/runs/35071525843/job/104713928139)
passes all 843 tests (5 skipped), with no Safari process in the job log.
Further normal runs are needed to establish sustained stability.
## Summary
- lazily initialize built-in OpenAI and Hugging Face providers when
consumers call the public embedding registry API
- choose automatic vector versus FTS search from embedding metadata on a
fresh pinned table revision for every execution
- expose automatic string searches as an `AutoQuery` with only
operations common to both native query families
- keep the registry shared and built-in registration safe across
duplicated module graphs
## Root cause
Nitro treats dependency modules as side-effect-free and removes the bare
OpenAI provider import from its generated route. Registration therefore
never runs, so `getRegistry().get("openai")` remains undefined even when
the registry itself is shared globally. Bundlers may also duplicate the
provider and registry module graphs.
The public embedding entry point now initializes built-in providers only
when `getRegistry()` is explicitly called, keeping initialization on a
live path that Nitro retains. Each terminal automatic-search execution
pins the exact table revision visible at dispatch, reads embedding
metadata and computes an embedding from that snapshot, replays the
builder operations, and constructs and executes the selected native
query against the same snapshot. Pinned native snapshots execute locally
when namespace pushdown cannot carry their revision, while remote
snapshots are seeded directly from one version-and-schema response. The
public `AutoQuery` builder exposes only the operations shared by FTS and
vector search, so runtime class narrowing cannot expose invalid
vector-only methods. Repeated built-in registration replaces stale
constructors from duplicated module graphs while public `register()`
retains its duplicate-alias error.
## Validation
- `cargo fmt --all`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test --runInBand` (783 passed, 5 skipped)
- serial examples suite with a local OpenAI mock (11 passed), including
`sentence-transformers.test.ts`
- packaged Nitro 2.13.4 server route using the reported imports returned
`{"registered":true}`
- fresh-process FTS fixture initialized both public built-ins and
confirmed automatic string search still returned the indexed row
- schema-consistency regressions cover read-consistency refresh,
checkout, checkoutLatest, restore, runtime class narrowing, concurrent
overwrite during embedding computation, and reused automatic-search
builders
- focused regressions confirm pinned native snapshots bypass unversioned
namespace pushdown and remote snapshots use one describe request
Fixes#2429
<!-- lance-gatekeeper-fix:v1 agent=2adf0f21b8bfb634606ed8897a849e30
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
adenzhou1350
Xuanwo
lancedb-gatefixer[bot]andXuanwo