lancedb

mirror of https://github.com/lancedb/lancedb.git synced 2026-10-23 20:22:37 +00:00

Files

T

Aviral Saxena d826b6115a fix(rust): explain client-side model files when remote tokenize fails (#4519 )

Addresses option 2 of #4440.

tokenize() builds the FTS analyzer on the client. For jieba/* and
lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and
when they’re missing you only got lance’s raw Invalid directory path:
... error, even though server-side search works.

Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps
the original cause and adds that tokenize() runs on the client, which
directory the files need to be in, and that for remote tables they must
match the server’s. I also updated the tokenize docs (Rust, Python, TS,
generated JS docs) to say a mismatch can’t be detected, so tokens may
differ from the server’s index.

Behavior when the files exist is unchanged. Detecting a mismatch needs a
server-side tokenize endpoint (option 1), so I left it out. The hint
isn’t limited to remote tables, since Table::as_native is slated for
removal and the wording fits local tables too.

Tests: Rust test with mocked jieba/default index_details and an empty
model dir, one checking other tokenizer errors don’t get the hint, and
Python sync/async tests.

2026-10-23 03:15:01 +08:00

embedding

fix(node): respect transformers tokenizer option (#4457 )

2026-10-23 02:45:19 +08:00

rerankers

ci(node): check docs in CI (#2084 )

2025-01-30 16:06:06 -08:00

arrow_type.ts

fix(node): validate inferred types across records (#3786 )

2026-08-24 16:41:05 -07:00

arrow.ts

fix!: normalize blob layouts and support external byte ranges (#4589 )