mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-23 20:22:37 +00:00
adenzhou1350
9c161869e1
## Summary `tokenizer` is a documented option, but initialization always passed the model name to `AutoTokenizer.from_pretrained`. Preserve the resolved tokenizer option, fall back to the model name when it is omitted, and identify the selected tokenizer in loading errors. Add five initialization regressions covering default model/tokenizer selection, an explicit tokenizer, registry variables, and tokenizer loading errors. They run in native Node child processes so the real ESM import works without changing Jest/CI flags. Model loaders are mocked and local/remote model access is disabled; these tests do not download models or test inference. ## Validation - Before: three regressions fail and two default-selection controls pass. After: all five pass. - New tests plus the existing registry and embedding-registry suites: 48 passed. - TypeScript compilation, changed-file Biome checks, and `pnpm run docs` pass; generated tracked docs are unchanged. - Checked with Node 24.14.1 / pnpm 11.1.1 on Linux, both with default Jest flags and the Linux CI ESM flag. The native binding and declarations came from the official `0.40.0-beta.13` packages, with registry integrity verified. Rebuilding Rust from source timed out during Cargo metadata resolution, before compilation; no Rust code is changed.
27 lines
836 B
TypeScript
27 lines
836 B
TypeScript
// SPDX-License-Identifier: Apache-2.0
|
|
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
|
|
|
|
import { execFileSync } from "node:child_process";
|
|
import { resolve } from "node:path";
|
|
|
|
describe("Transformers embedding tokenizer selection", () => {
|
|
// Run init's native ESM import outside Jest's CommonJS sandbox, like a user.
|
|
// The fixture mocks model loading and forbids local/remote model access.
|
|
test.each([
|
|
["default model", 0],
|
|
["default tokenizer", 1],
|
|
["custom tokenizer", 2],
|
|
["tokenizer registry variable", 3],
|
|
["tokenizer loading error", 4],
|
|
])("handles %s", (_name, testCase) => {
|
|
execFileSync(
|
|
process.execPath,
|
|
[
|
|
resolve(__dirname, "fixtures", "transformers_tokenizer.cjs"),
|
|
String(testCase),
|
|
],
|
|
{ stdio: "pipe", timeout: 15_000 },
|
|
);
|
|
});
|
|
});
|