adenzhou1350 9c161869e1 fix(node): respect transformers tokenizer option (#4457) ...

## Summary

`tokenizer` is a documented option, but initialization always passed the
model name to `AutoTokenizer.from_pretrained`. Preserve the resolved
tokenizer option, fall back to the model name when it is omitted, and
identify the selected tokenizer in loading errors.

Add five initialization regressions covering default model/tokenizer
selection, an explicit tokenizer, registry variables, and tokenizer
loading errors. They run in native Node child processes so the real ESM
import works without changing Jest/CI flags. Model loaders are mocked
and local/remote model access is disabled; these tests do not download
models or test inference.

## Validation

- Before: three regressions fail and two default-selection controls
pass. After: all five pass.
- New tests plus the existing registry and embedding-registry suites: 48
passed.
- TypeScript compilation, changed-file Biome checks, and `pnpm run docs`
pass; generated tracked docs are unchanged.
- Checked with Node 24.14.1 / pnpm 11.1.1 on Linux, both with default
Jest flags and the Linux CI ESM flag. The native binding and
declarations came from the official `0.40.0-beta.13` packages, with
registry integrity verified. Rebuilding Rust from source timed out
during Cargo metadata resolution, before compilation; no Rust code is
changed.

2026-10-23 02:45:19 +08:00

..

auto_fts_search.cjs

fix(node): preserve embedding registration in server bundles (#3806)

2026-08-26 04:17:32 +08:00

oauth_browser.cmd

fix(node): prevent OAuth tests from launching browsers (#4196)

2026-09-16 16:32:34 +08:00

transformers_tokenizer.cjs

fix(node): respect transformers tokenizer option (#4457)

2026-10-23 02:45:19 +08:00