feat(python): support model-backed native FTS tokenizers (#3289)

mirror of https://github.com/lancedb/lancedb.git synced 2026-09-04 12:38:38 +00:00

This wires Lance's existing `jieba/*` and `lindera/*` native FTS
tokenizers through the Python SDK instead of leaving them behind
disabled features and narrow public typing. It also documents the
`LANCE_LANGUAGE_MODEL_HOME` model layout and adds Python coverage for
successful CJK indexing plus missing-model error guidance.

Closes #2168.

This commit is contained in:

Xuanwo

2026-05-08 23:53:14 +08:00

committed by

GitHub

parent 694aa48e19

commit 9b21c136c6

12 changed files with 725 additions and 51 deletions

									
										python/python/tests/models/lindera/ipadic/config.yml
									
		+4
		
												View File
												
				@@ -0,0 +1,4 @@

				segmenter:

				  mode: "normal"

				  dictionary:

				    path: "./python/tests/models/lindera/ipadic/main"