Bump version: 0.35.0-beta.1 → 0.35.0-beta.2

Updates the Rust workspace Lance dependencies and Java lance-core from
v9.0.0-beta.19 to v9.0.0-beta.23.

No compatibility fixes were required; strict workspace Clippy and Rust
formatting pass. Lance tag:
https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.23

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>

.agents/skills/lancedb-branch-ops/SKILL.md

@@ -1,6 +1,14 @@
 ---
 name: lancedb-branch-ops
-description: Branch management for LanceDB tables via the REST API. Use this skill whenever someone wants to create, delete, list, or switch branches on a LanceDB table — or needs to make sure a write (metadata update, index build, etc.) lands on a specific branch instead of main. Invoke it even without the word "branch" if context makes clear they want an experimental copy of a table, want to isolate changes, or want to confirm a mutation didn't touch main. Covers: branches/list, branches/create, branches/delete, and passing "branch" in describe/update_field_metadata/create_index to target a non-main version.
+description: >-
+  Manage LanceDB table branches through the REST API: list, create, and delete
+  branches; target schema reads, field-metadata updates, and index creation to a
+  named branch; and verify that branch changes remain isolated from main. Use
+  when a task involves branch lifecycle, an experimental or isolated table
+  version, directing an operation to a non-main branch, or confirming that a
+  mutation did not affect main. This skill also explains that LanceDB has no
+  checkout operation; each request selects its target branch in the request
+  body.
 ---
 
 ## Goal

.agents/skills/lancedb/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: lancedb
+description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, and apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics.
+---
+
+# Building LanceDB Pipelines
+
+Use this skill to produce LanceDB pipelines that are portable between local and remote tables (for LanceDB Enterprise/Cloud) and idiomatic for the selected SDK.
+
+## LanceDB Table Modes
+
+LanceDB has two common execution modes:
+
+- **Local table**: embedded, open source, in-process LanceDB. The client opens data from a local path or object storage URI and executes queries in the application process.
+- **Remote table**: LanceDB Enterprise/Cloud table opened through a `db://...` URI. The data may be very large, commonly backed by object storage, and queried through a remote service.
+
+Do NOT assume local-only table helpers exist on remote tables. If the user asks for LanceDB Enterprise, Cloud, `db://...`, production remote access, or a remote table, focus on the remote table path: use `search()` / `query()`, keep reads bounded with `select()` and `limit()`, and avoid table-level full materialization APIs.
+
+## Workflow
+
+1. Identify the SDK: Python, TypeScript, or both.
+2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path.
+3. Read the matching language branch before writing or changing code:
+   - Python patterns: `references/python/patterns.md`
+   - Python API quick reference: `references/python/api_reference.md`
+   - Python performance guidance: `references/python/performance.md`
+   - TypeScript patterns: `references/typescript/patterns.md`
+   - TypeScript API quick reference: `references/typescript/api_reference.md`
+   - TypeScript performance guidance: `references/typescript/performance.md`
+4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets.
+5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
+6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.
+7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
+8. After a successful embedded OSS ingestion, call `table.optimize()`. Do not call it for Enterprise/Cloud; remote maintenance is automatic.
+9. For remote Enterprise/Cloud writes, never drop-then-reuse or `mode="overwrite"` the same table name — see "Enterprise: never drop-then-reuse the same table name" below. This is the main local-vs-remote write pitfall.
+10. If reviewing an existing file or repo, run `scripts/check_materialization.py` on the relevant paths and inspect each finding before editing.
+11. Cross-check unfamiliar or non-trivial API claims against the source tree instead of relying on memory.
+
+## Core Portability Rule
+
+Do not write code that assumes a local table API will exist on a remote table. Remote tables can be very large, so whole-table materialization helpers are intentionally unavailable or unsafe.
+
+This does **not** mean result conversion is forbidden. Bounded query/search result collection is normal:
+
+- Python: `table.search(...).select([...]).limit(10).to_pandas()`
+- TypeScript: `await table.search(...).select([...]).limit(10).toArray()`
+
+The unsafe pattern is table-level or unbounded collection, plus local-only dataset escape hatches in remote code:
+
+- Python: `table.to_pandas()`, `table.to_arrow()`, `table.to_polars()`; `table.to_lance()` is local/OSS-only dataset access, not materialization
+- TypeScript: `await table.toArrow()`, `await table.query().toArray()` without `limit()`
+
+## Enterprise: never drop-then-reuse the same table name
+
+LanceDB Enterprise/Cloud splits a **control plane** (DDL: create/drop/rename) from a **data plane** (query nodes that serve reads). Query nodes cache the resolved dataset for a table name for up to `table_cache_ttl` — **default 300 seconds (5 minutes)**. After you drop or overwrite a table, the control plane updates immediately but the data plane keeps serving the *old* dataset until that cache entry expires. During the window the two planes disagree.
+
+The failure this causes: you `drop_table("t")` then immediately `create_table("t", ...)` (or `create_table("t", ..., mode="overwrite")`). The DDL returns success, but every query against `t` returns **`500 Internal Server Error`** (the query node resolves the stale/deleted dataset), and a fresh `describe` may still show the *old* schema/version. It looks like your write silently failed; it didn't — the name is cached.
+
+**`mode="overwrite"` has the same problem** — it is a drop+create of the same name under the hood.
+
+Rules for portable Enterprise ingestion:
+
+1. **Never reuse a table name you just dropped/overwrote within the cache TTL.** Do not use `mode="overwrite"` to replace an existing Enterprise table in place.
+2. To (re)load data, **write to a fresh table name** (e.g. `<table>_v2`, or a run-stamped suffix). A brand-new name has no cached data-plane entry, so writes and reads work immediately.
+3. Before creating, `list_tables()` and **fail loudly if the name already exists** rather than overwriting — prompt for a new name.
+4. To land on a specific final name that is currently occupied by an old table: drop the old table, **wait out the TTL (~5 min), then `rename_table(fresh_name, final_name)`**. Renaming onto a name whose old dataset is still cached hits the same race, so the wait is mandatory. `rename_table` is a supported control-plane op.
+5. When you hand a table name back to a human, tell them which step still needs the propagation wait (usually: "the old `t` was dropped; run the rename in ~5 minutes").
+
+This is Enterprise/Cloud-specific. Local/OSS tables have no separate data plane, so `mode="overwrite"` and immediate same-name reuse are fine there.
+
+## Script
+
+Run the scanner when reviewing or modifying an existing codebase:
+
+```bash
+python skills/lancedb/scripts/check_materialization.py path/to/file_or_dir
+```
+
+The script reports likely unsafe full-table materialization in Python and TypeScript. Treat results as review prompts, not automatic proof of a bug.

.agents/skills/lancedb/references/python/api_reference.md

@@ -0,0 +1,105 @@
+# Python API Reference
+
+Quick method reference for Python LanceDB code. Cross-check source for non-trivial claims.
+
+## Connect
+
+```python
+import lancedb
+
+db = lancedb.connect("./camelot-db")         # local/OSS
+db = lancedb.connect("db://my-db", api_key=api_key, region=region)  # remote
+```
+
+**Place the local database directory next to the script/entrypoint that opens it** (i.e. resolve the path relative to the script, `Path(__file__).parent / "camelot-db"`), not buried under a shared `data/` folder. The Lance dataset is the database, not a data file — keeping it beside its code makes ownership obvious and paths stable regardless of the working directory the script is launched from.
+
+**Do not name the directory `lancedb`** (e.g. `./lancedb`, `./data/lancedb`). It collides with the imported `lancedb` package name, which is confusing to read and easy to shadow in scripts. Give it a name derived from the repo or dataset with a clear prefix/suffix — for example `./<dataset>-db`, `./<repo>_lancedb`, or `./vectordb`.
+
+Async:
+
+```python
+db = await lancedb.connect_async("./camelot-db")
+```
+
+## Table Reads
+
+| Task | Preferred API |
+| --- | --- |
+| Vector search | `table.search(query_vector).limit(k)` |
+| Full scan with filters/projection (sync) | `table.search().where(...).select(...).limit(...)` |
+| Full scan with filters/projection (async) | `table.query().where(...).select(...).limit(...)` |
+| Filter | `.where("col > 10")` |
+| Projection | `.select(["id", "text"])` |
+| Bound result count | `.limit(20)` |
+| Collect bounded result as Python objects (default, no extra deps) | `.to_list()` on query/search result |
+| Collect bounded result as Arrow (default, `pyarrow` always available) | `.to_arrow()` on query/search result |
+| Collect bounded result as pandas (only if project uses pandas) | `.to_pandas()` on query/search result |
+| Collect bounded result as Polars (only if project uses polars) | `.to_polars()` on query/search result |
+
+## Sync vs Async Scan API
+
+The plain-scan entry point differs between the sync and async clients. **Verified against `lancedb` 0.34.0** — re-check if the pinned version changes:
+
+- **Sync** (`lancedb.connect(...)`): the table has **no `.query()` method**. Use `.search()` with no argument for a plain scan; it returns a query builder that supports `.where()`, `.select()`, `.limit()`, and the `.to_list()` / `.to_arrow()` / `.to_pandas()` / `.to_polars()` collectors.
+  ```python
+  rows = table.search().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
+  ```
+- **Async** (`lancedb.connect_async(...)`): the table has **both** `.query()` and `.search()`. Use `.query()` for a plain scan.
+  ```python
+  rows = await async_table.query().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
+  ```
+
+Do not call `table.query()` on a sync table — it raises `AttributeError`.
+
+## Local vs Remote Table Methods
+
+| API | Local table | Remote table | Agent guidance |
+| --- | --- | --- | --- |
+| `table.search(...)` | Yes | Yes | Preferred read path (sync + async) |
+| `table.query()` | Async only | Async only | Sync scan path is `table.search()`; `.query()` is the async scan builder |
+| `table.to_pandas()` | Yes | No / unsafe for portability | Avoid in portable code |
+| `table.to_arrow()` | Yes | No / unsafe for portability | Avoid in portable code |
+| `table.to_polars()` | Yes | No / unsafe for portability | Avoid in portable code |
+| `table.to_lance()` | Yes | No | Local/OSS escape hatch only |
+
+## Indexes
+
+Use `create_index(...)` for vector indexes and modern index configs. Use scalar indexes for filtered or merge keys.
+
+Common calls:
+
+```python
+table.create_index("vector")
+table.create_scalar_index("status")
+table.create_fts_index("text")
+```
+
+Check source docs before specifying advanced index config names or parameters.
+
+## Filtering And Recall Knobs
+
+```python
+table.search(query_vector).where("status = 'ready'")  # pre-filter by default
+table.search(query_vector).where("status = 'ready'", prefilter=False)
+table.search(query_vector).limit(10).refine_factor(20)
+table.search(query_vector).limit(10).nprobes(50)
+```
+
+Use post-filtering only when fewer than `limit` results are acceptable.
+
+## Diagnostics
+
+```python
+print(table.search(query_vector).where("year > 2000").limit(10).analyze_plan())
+print(table.index_stats("vector_idx"))
+```
+
+Use these before changing indexes or search tuning.
+
+## Maintenance
+
+```python
+table.optimize()
+```
+
+Call this after every successful local/OSS ingestion. It handles compaction, cleanup of old versions according to retention, and index optimization. Do not add this for LanceDB Enterprise/Cloud remote tables; Enterprise handles compaction and cleanup automatically from cluster configuration.

.agents/skills/lancedb/references/python/patterns.md

@@ -0,0 +1,173 @@
+# Python Patterns
+
+Use these patterns when writing Python code with `lancedb`.
+
+## Before Writing Code
+
+Choose the output type from what the project actually depends on. **Do not assume `pandas` or `polars` is installed** — they are heavy dependencies that many LanceDB projects do not use. `pyarrow`, by contrast, ships as a LanceDB dependency and is always available, so it is a safe default to lean on.
+
+Default output (after applying `select()` and `limit()`):
+
+- **Python objects**: `.to_list()` — a list of dicts, no extra dependencies. Prefer this for scripts, examples, and agent-generated code unless there is a reason to do otherwise.
+- **PyArrow**: `.to_arrow()` — a `pyarrow.Table`, when the surrounding code is Arrow-native or you need columnar/zero-copy handoff.
+
+Only reach for a DataFrame when the project *already* declares that dependency:
+
+- Pandas projects (pandas in `pyproject.toml`/requirements): `.to_pandas()`.
+- Polars projects (polars declared): `.to_polars()`.
+
+If unsure, check the dependency manifest or the imports in surrounding files. When in doubt, use `.to_list()` or `.to_arrow()`.
+
+## Schema Design and Validation
+
+Favor `LanceModel` and Pydantic validation for Python schemas. They keep field
+types readable, validate source records before a write, and map directly to a
+LanceDB schema. Use `Vector(dimension)` for fixed-size vectors:
+
+```python
+from lancedb.pydantic import LanceModel, Vector
+
+class Document(LanceModel):
+    id: int
+    text: str
+    vector: Vector(384, nullable=False)
+
+rows = [Document.model_validate(row) for row in source_rows]
+table = db.create_table("documents", schema=Document)
+table.add(rows)
+```
+
+Use PyArrow schemas instead when the pipeline is already Arrow-native, needs
+record-batch streaming, or has runtime schema requirements that would make a
+Pydantic model harder to understand. Declare Pydantic as a direct project
+dependency when application code imports it, even if LanceDB also depends on it.
+
+## Recommended Patterns
+
+### Bounded search or query
+
+Use this for application reads, examples, notebooks, and agent-generated scripts:
+
+```python
+results = (
+    table.search(query_vector)
+    .where("status = 'ready'")
+    .select(["id", "text"])
+    .limit(20)
+    .to_list()  # or .to_arrow(); .to_pandas()/.to_polars() only if the project uses them
+)
+```
+
+Why: `search()` works across local and remote tables and on both the sync and async clients. `select()` avoids fetching unused columns. `limit()` prevents accidental full-table reads. `.to_list()` and `.to_arrow()` avoid assuming pandas/polars is installed (see "Before Writing Code").
+
+For a **plain scan** (no query vector), the entry point differs by client:
+
+```python
+# Sync client: no .query() method — use .search() with no argument.
+rows = table.search().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
+
+# Async client: use .query().
+rows = await async_table.query().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
+```
+
+`table.query()` on a sync table raises `AttributeError` (verified on `lancedb` 0.34.0). See the "Sync vs Async Scan API" section in `api_reference.md`.
+
+### Bounded query result conversion
+
+It is fine to collect bounded query/search results:
+
+```python
+arrow_table = table.search().select(["id"]).limit(100).to_arrow()  # sync plain scan
+rows = table.search(query_vector).limit(10).to_list()
+df = table.search(query_vector).limit(10).to_pandas()  # only if pandas is a project dep
+```
+
+### Local-only Lance dataset API
+
+`table.to_lance()` does not itself materialize the full dataset. It returns the underlying `lance.LanceDataset`, making the table accessible through the PyLance dataset API. Use it when the task is explicitly local/OSS and needs Lance dataset methods not exposed by LanceDB:
+
+```python
+# Local/OSS only: RemoteTable does not expose table.to_lance().
+ds = table.to_lance()
+for batch in ds.to_batches(columns=["id", "text"], batch_size=10_000):
+    process(batch)
+```
+
+### Async Python
+
+Keep the same shape and bound the result before collecting:
+
+```python
+results = await (
+    async_table.query()
+    .where("status = 'ready'")
+    .select(["id", "text"])
+    .limit(20)
+    .to_list()  # or .to_arrow()
+)
+```
+
+## Anti-Patterns
+
+**Avoid the following anti-patterns in your code.**
+
+### Table-level full materialization
+
+Avoid whole-table collectors in portable or large-table code:
+
+```python
+df = table.to_pandas()
+arrow_table = table.to_arrow()
+polars_df = table.to_polars()
+```
+
+Why: local tables expose these whole-table collectors, but remote tables intentionally do not — a remote production table can be far larger than a local development table, so it is easy to accidentally pull the entire table into memory.
+
+`table.to_lance()` is different: it is not a full materialization call, but it is still local/OSS-only and should not appear in code meant to run against remote Enterprise tables.
+
+### Unbounded result collection
+
+Avoid query/search collection without a meaningful limit:
+
+```python
+rows = table.search().to_list()               # unbounded plain scan
+rows = table.search(query_vector).to_list()   # unbounded vector search
+```
+
+Prefer `select(...).limit(...)` before collecting; for large reads, stream in batches instead.
+
+### Per-row writes
+
+Avoid loops that write one row per call:
+
+```python
+for row in rows:
+    table.add([row])  # one commit + fragment per row
+```
+
+Each `add()` creates a new version and fragment. Pass the whole batch in a single call, or chunk very large inputs:
+
+```python
+table.add(rows)  # single commit
+# for very large inputs, add batches of several thousand rows
+```
+
+After the final successful write to an embedded OSS table, call
+`table.optimize()`. Skip this for Enterprise/Cloud tables because their
+maintenance is automatic.
+
+### Drop-then-reuse the same table name (Enterprise/Cloud)
+
+Avoid dropping or overwriting a remote table and then reusing that name right away:
+
+```python
+db.drop_table("my_table")
+table = db.create_table("my_table", data=rows)          # reads 500 for ~5 min
+table = db.create_table("my_table", data=rows, mode="overwrite")  # same problem
+```
+
+Why: Enterprise/Cloud splits DDL (control plane) from query serving (data plane). The data plane caches the dataset behind a table name for up to `table_cache_ttl` (default 300s / 5 min), so after a drop/overwrite the DDL succeeds but queries against the reused name return `500 Internal Server Error` until the cache expires — and a fresh `describe` may still show the old schema. Instead, write to a **fresh name**, use `list_tables()` and fail if it already exists, then `rename_table(fresh, final)` onto the final name only after the old table's drop has propagated (~5 min). See the "Enterprise: never drop-then-reuse the same table name" section in `SKILL.md`. Local/OSS tables have no separate data plane — overwrite freely there.
+
+### Guessing performance fixes
+
+Avoid changing `nprobes`, `refine_factor`, or index types before checking the query plan and index stats. Diagnose first, then tune one knob at a time.

.agents/skills/lancedb/references/python/performance.md

@@ -0,0 +1,131 @@
+# Python Performance Guidance
+
+Use this when writing Python code that ingests data, queries large tables, builds indexes, or investigates latency.
+
+## Ingestion
+
+### Recommended: validate schemas and records with Pydantic
+
+Favor `LanceModel` for readable Python schema definitions and validate source
+records before writing. Use PyArrow directly for Arrow-native or streaming
+pipelines where it is the clearer representation.
+
+```python
+from lancedb.pydantic import LanceModel, Vector
+
+class Document(LanceModel):
+    id: int
+    text: str
+    vector: Vector(384, nullable=False)
+
+rows = [Document.model_validate(row) for row in source_rows]
+table = db.create_table("documents", schema=Document)
+table.add(rows)
+```
+
+### Recommended: bulk ingestion for materialized data
+
+```python
+table.add(arrow_table)
+table.add(df)
+table.add(pa.dataset("data/", format="parquet"))
+```
+
+For very large initial loads, create the table empty first, then call `add(...)`. Passing data directly to `create_table(name, data)` can skip the auto-parallel write path.
+
+### Recommended: iterator ingestion for generated or streamed data
+
+```python
+def batches():
+    for raw in source:
+        vectors = model.encode(raw["text"])
+        yield pa.RecordBatch.from_pydict({**raw, "vector": vectors})
+
+table.add(batches())
+```
+
+Use chunks of several thousand rows or more when practical. Tiny batches and per-row writes create many small fragments.
+
+### Anti-pattern: per-row `add()`
+
+```python
+for row in rows:
+    table.add([row])
+```
+
+Each call creates a version and fragment. This slows ingestion and later queries.
+
+## Indexing
+
+- Build a vector index once brute-force vector search becomes too slow. As a rule of thumb, local brute force is fine below roughly 100K vectors; beyond that, build an index.
+- Use `IVF_PQ` as the general-purpose default. Enterprise builds this automatically.
+- Use scalar indexes for filtered columns and merge/upsert keys.
+- Use `BTREE` for mostly distinct numeric/string/temporal columns, `BITMAP` for booleans and low-cardinality columns, and `LABEL_LIST` for list membership queries.
+- Keep full-text defaults unless phrase queries require position data.
+
+## Querying
+
+Always be explicit:
+
+```python
+table.search(query_vector).select(["id", "title"]).limit(20)
+```
+
+- `select()` reduces bytes read and transferred.
+- `limit()` prevents accidental full-table materialization.
+- Pre-filtering is the default and guarantees returned rows satisfy the predicate.
+- Use post-filtering only when fewer than `limit` results are acceptable.
+
+## Recall Tuning
+
+Tune one knob at a time:
+
+- Quantized indexes: raise `refine_factor` to rescore more candidates on full vectors.
+- HNSW-backed indexes: raise `ef`; start around `1.5 * k`, increase toward `10 * k` if recall is short.
+- IVF candidate breadth: `nprobes` is auto-tuned; override only when a selective pre-filter leaves too few neighbors.
+
+## Maintenance
+
+After every successful embedded OSS/local ingestion, call `table.optimize()`.
+Do not add this to LanceDB Enterprise/Cloud remote table code; remote compaction
+and cleanup are handled automatically based on the Enterprise cluster
+configuration.
+
+Why local maintenance is needed:
+
+- Frequent writes can create many small fragments. Queries then need to scan across more files, which can increase latency.
+- Updates, deletes, and appends create new table versions. Old versions are retained for time travel and rollback, which can grow disk usage.
+- Indexes may have newly added rows that are not yet fully optimized into the index structure.
+
+For local/OSS tables, run `optimize()` after the final successful ingestion
+write. Also run it after later batches of update/delete operations or on a
+regular maintenance schedule:
+
+```python
+table.optimize()
+```
+
+If the user wants more aggressive local disk cleanup, pass a shorter cleanup retention window:
+
+```python
+from datetime import timedelta
+
+table.optimize(cleanup_older_than=timedelta(days=1))
+```
+
+Do not use very short cleanup windows when the application depends on time travel, rollback, or old versions.
+
+## Diagnostics
+
+Before changing code or indexes, inspect:
+
+```python
+print(table.search(query_vector).where("year > 2000").limit(10).analyze_plan())
+print(table.index_stats("vector_idx"))
+```
+
+Look for high scan bytes, missing indexes, fragmented data, and unindexed rows.
+
+## Python Multiprocessing
+
+When using multiprocessing, use `spawn` rather than `fork`. LanceDB is multi-threaded internally, and `fork` plus a multi-threaded process is unsafe.

.agents/skills/lancedb/references/typescript/api_reference.md

@@ -0,0 +1,78 @@
+# TypeScript API Reference
+
+Quick method reference for TypeScript LanceDB code. Cross-check source for non-trivial claims.
+
+## Connect
+
+```typescript
+import * as lancedb from "@lancedb/lancedb";
+
+const db = await lancedb.connect("./camelot-db");
+```
+
+**Place the local database directory next to the script/entrypoint that opens it** (resolve the path relative to the module, e.g. via `import.meta.dirname` / `__dirname`), not buried under a shared `data/` folder. The Lance dataset is the database, not a data file — keeping it beside its code makes ownership obvious and paths stable regardless of the working directory the script is launched from.
+
+**Do not name the directory `lancedb`** (e.g. `./lancedb`, `./data/lancedb`). It collides with the imported `lancedb` package/namespace, which is confusing to read. Give it a name derived from the repo or dataset with a clear prefix/suffix — for example `./<dataset>-db`, `./<repo>_lancedb`, or `./vectordb`.
+
+Remote connections use `db://...` plus Enterprise/Cloud credentials and deployment settings. Check current source/docs for exact connection options.
+
+## Table Reads
+
+| Task | Preferred API |
+| --- | --- |
+| Vector search | `table.search(queryVector).limit(k)` |
+| Full scan with filters/projection | `table.query().where(...).select(...).limit(...)` |
+| Filter | `.where("col > 10")` |
+| Projection | `.select(["id", "text"])` |
+| Bound result count | `.limit(20)` |
+| Collect bounded result as objects | `.toArray()` on query/search result |
+| Collect bounded result as Arrow | `.toArrow()` on query/search result |
+| Stream result batches | `for await (const batch of table.query()...)` |
+
+## Local vs Remote Safety
+
+| API | Agent guidance |
+| --- | --- |
+| `table.search(...)` | Preferred read path |
+| `table.query()` | Preferred scan/filter path |
+| `await table.toArrow()` | Avoid in portable or large-table code |
+| `await table.query().toArray()` with no `limit()` | Avoid; unbounded collection |
+| `await table.query().toArrow()` with no `limit()` | Avoid; unbounded collection |
+
+## Indexes
+
+```typescript
+await table.createIndex("vector");
+await table.createIndex("status");
+```
+
+Use vector indexes for large vector search workloads and scalar indexes for filtered columns or merge/upsert keys. Check source/docs before specifying advanced index options.
+
+## Filtering And Recall Knobs
+
+```typescript
+await table.search(queryVector).where("status = 'ready'").limit(10).toArray();
+await table.search(queryVector).limit(10).refineFactor(20).toArray();
+await table.search(queryVector).limit(10).nprobes(50).toArray();
+await table.search(queryVector).limit(10).ef(100).toArray();
+await table.search(queryVector).where("status = 'ready'").postfilter().limit(10).toArray();
+```
+
+Use `postfilter()` only when fewer than `limit` results are acceptable.
+
+## Diagnostics
+
+```typescript
+console.log(await table.search(queryVector).where("year > 2000").limit(10).analyzePlan());
+console.log(await table.indexStats("vector_idx"));
+```
+
+Use these before changing indexes or search tuning.
+
+## Maintenance
+
+```typescript
+await table.optimize();
+```
+
+Call this after every successful local/OSS ingestion. It handles compaction, cleanup of old versions according to retention, and index optimization. Do not add this for LanceDB Enterprise/Cloud remote tables; Enterprise handles compaction and cleanup automatically from cluster configuration.

.agents/skills/lancedb/references/typescript/patterns.md

@@ -0,0 +1,100 @@
+# TypeScript Patterns
+
+Use these patterns when writing TypeScript code with `@lancedb/lancedb`.
+
+## Recommended Patterns
+
+### Bounded query
+
+Use this for application reads, scripts, and examples:
+
+```typescript
+const rows = await table
+  .query()
+  .where("status = 'ready'")
+  .select(["id", "text"])
+  .limit(20)
+  .toArray();
+```
+
+### Bounded vector search
+
+```typescript
+const rows = await table
+  .search(queryVector)
+  .select(["id", "text"])
+  .limit(20)
+  .toArray();
+```
+
+### Batch streaming for larger reads
+
+When the task needs many rows, avoid collecting everything at once:
+
+```typescript
+for await (const batch of table
+  .query()
+  .where("status = 'ready'")
+  .select(["id", "text"])
+  .limit(10_000)) {
+  process(batch);
+}
+```
+
+## Anti-Patterns
+
+**Avoid the following anti-patterns in your code.**
+
+### Table-level full materialization
+
+Avoid whole-table collectors in portable or large-table code:
+
+```typescript
+const tableArrow = await table.toArrow();
+```
+
+Why: local tables expose these whole-table collectors, but remote tables intentionally do not — a remote production table can be far larger than a local development table, so it is easy to accidentally pull the entire table into memory.
+
+### Unbounded result collection
+
+Avoid query/search collection without a meaningful limit:
+
+```typescript
+const rows = await table.query().toArray();              // unbounded plain scan
+const rows = await table.search(queryVector).toArray();  // unbounded vector search
+```
+
+Prefer `select(...).limit(...)` before collecting; for large reads, stream in batches instead.
+
+### Per-row writes
+
+Avoid loops that write one row per call:
+
+```typescript
+for (const row of rows) {
+  await table.add([row]); // one commit + fragment per row
+}
+```
+
+Each `add()` creates a new version and fragment. Pass the whole batch in a single call, or chunk very large inputs:
+
+```typescript
+await table.add(rows); // single commit
+// for very large inputs, add in chunks of several thousand rows
+```
+
+### Drop-then-reuse the same table name (Enterprise/Cloud)
+
+Avoid dropping or overwriting a remote table and then reusing that name right away:
+
+```typescript
+await db.dropTable("my_table");
+const table = await db.createTable("my_table", rows);                    // reads 500 for ~5 min
+const table = await db.createTable("my_table", rows, { mode: "overwrite" }); // same problem
+```
+
+Why: Enterprise/Cloud splits DDL (control plane) from query serving (data plane). The data plane caches the dataset behind a table name for up to `table_cache_ttl` (default 300s / 5 min), so after a drop/overwrite the DDL succeeds but queries against the reused name return `500 Internal Server Error` until the cache expires — and a fresh `describe` may still show the old schema. Instead, write to a **fresh name**, use `tableNames()` and fail if it already exists, then `renameTable(fresh, final)` onto the final name only after the old table's drop has propagated (~5 min). See the "Enterprise: never drop-then-reuse the same table name" section in `SKILL.md`. Local/OSS tables have no separate data plane — overwrite freely there.
+
+### Guessing performance fixes
+
+Avoid changing `nprobes`, `refineFactor`, `ef`, or index settings before checking `analyzePlan()` and `indexStats(...)`. Diagnose first, then tune one knob at a time.

.agents/skills/lancedb/references/typescript/performance.md

@@ -0,0 +1,78 @@
+# TypeScript Performance Guidance
+
+Use this when writing TypeScript code that ingests data, queries large tables, builds indexes, or investigates latency.
+
+## Ingestion
+
+- Prefer bulk or batched writes.
+- Avoid per-row write loops; they create many small commits/fragments.
+- For generated data, accumulate reasonable batches before adding.
+- For file-backed data, prefer APIs that stream from Arrow/Parquet-style inputs when available.
+
+## Indexing
+
+- Build a vector index once brute-force vector search becomes too slow. As a rule of thumb, local brute force is fine below roughly 100K vectors; beyond that, build an index.
+- Use the general-purpose vector index defaults unless the task has explicit recall/latency requirements.
+- Build scalar indexes for filtered columns and merge/upsert keys.
+- Use full-text index phrase options only when phrase queries require them.
+
+## Querying
+
+Always be explicit:
+
+```typescript
+await table.search(queryVector).select(["id", "title"]).limit(20).toArray();
+```
+
+- `select()` reduces bytes read and transferred.
+- `limit()` prevents accidental full-table collection.
+- Pre-filtering is the default behavior. Use `postfilter()` only when fewer than `limit` results are acceptable.
+
+## Recall Tuning
+
+Tune one knob at a time:
+
+- Quantized indexes: raise `refineFactor(...)` to rescore more candidates on full vectors.
+- HNSW-backed indexes: raise `ef(...)`; start around `1.5 * k`, increase toward `10 * k` if recall is short.
+- IVF candidate breadth: `nprobes(...)` is usually auto-tuned; override only when a selective pre-filter leaves too few neighbors.
+
+## Maintenance
+
+After every successful embedded OSS/local ingestion, call `table.optimize()`.
+Do not add this to LanceDB Enterprise/Cloud remote table code; remote compaction
+and cleanup are handled automatically based on the Enterprise cluster
+configuration.
+
+Why local maintenance is needed:
+
+- Frequent writes can create many small fragments. Queries then need to scan across more files, which can increase latency.
+- Updates, deletes, and appends create new table versions. Old versions are retained for time travel and rollback, which can grow disk usage.
+- Indexes may have newly added rows that are not yet fully optimized into the index structure.
+
+For local/OSS tables, run `optimize()` after the final successful ingestion
+write. Also run it after later batches of update/delete operations or on a
+regular maintenance schedule:
+
+```typescript
+await table.optimize();
+```
+
+If the user wants more aggressive local disk cleanup, pass a shorter cleanup retention window:
+
+```typescript
+const olderThan = new Date(Date.now() - 24 * 60 * 60 * 1000);
+await table.optimize({ cleanupOlderThan: olderThan });
+```
+
+Do not use very short cleanup windows when the application depends on time travel, rollback, or old versions.
+
+## Diagnostics
+
+Before changing code or indexes, inspect:
+
+```typescript
+console.log(await table.search(queryVector).where("year > 2000").limit(10).analyzePlan());
+console.log(await table.indexStats("vector_idx"));
+```
+
+Look for high scan cost, missing indexes, fragmented data, and unindexed rows.

.agents/skills/lancedb/scripts/check_materialization.py

@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""Scan Python and TypeScript for likely unsafe LanceDB materialization."""
+
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+
+PY_FULL_TABLE = re.compile(r"\b\w+\.(to_pandas|to_arrow|to_polars)\s*\(")
+TS_TABLE_TO_ARROW = re.compile(r"\b\w+\.toArrow\s*\(")
+TS_QUERY_COLLECTOR = re.compile(r"\.query\s*\(\s*\)[\s\S]*?\.to(Array|Arrow)\s*\(")
+
+
+@dataclass
+class Finding:
+    path: Path
+    line: int
+    message: str
+    text: str
+
+
+def iter_files(paths: list[Path]) -> list[Path]:
+    files: list[Path] = []
+    for path in paths:
+        if path.is_dir():
+            files.extend(
+                p
+                for p in path.rglob("*")
+                if p.suffix in {".py", ".ts", ".tsx"} and "node_modules" not in p.parts
+            )
+        elif path.suffix in {".py", ".ts", ".tsx"}:
+            files.append(path)
+    return sorted(set(files))
+
+
+def line_number(text: str, offset: int) -> int:
+    return text.count("\n", 0, offset) + 1
+
+
+def scan_python(path: Path, text: str) -> list[Finding]:
+    findings: list[Finding] = []
+    for match in PY_FULL_TABLE.finditer(text):
+        line_start = text.rfind("\n", 0, match.start()) + 1
+        line_end = text.find("\n", match.start())
+        if line_end == -1:
+            line_end = len(text)
+        line = text[line_start:line_end].strip()
+        if ".search(" in line or ".query(" in line:
+            continue
+        findings.append(
+            Finding(
+                path,
+                line_number(text, match.start()),
+                f"Review Python `{match.group(1)}()` call; table-level materialization is not portable to remote tables.",
+                line,
+            )
+        )
+    return findings
+
+
+def statement_around(text: str, start: int, end: int) -> str:
+    before = max(text.rfind(";", 0, start), text.rfind("\n\n", 0, start))
+    after_candidates = [pos for pos in (text.find(";", end), text.find("\n\n", end)) if pos != -1]
+    after = min(after_candidates) if after_candidates else len(text)
+    return text[before + 1 : after].strip()
+
+
+def scan_typescript(path: Path, text: str) -> list[Finding]:
+    findings: list[Finding] = []
+    for match in TS_TABLE_TO_ARROW.finditer(text):
+        stmt = statement_around(text, match.start(), match.end())
+        if ".query(" in stmt or ".search(" in stmt:
+            continue
+        findings.append(
+            Finding(
+                path,
+                line_number(text, match.start()),
+                "Review TypeScript `table.toArrow()`-style call; table-level materialization is not portable for large/remote tables.",
+                stmt.splitlines()[0].strip(),
+            )
+        )
+
+    for match in TS_QUERY_COLLECTOR.finditer(text):
+        stmt = statement_around(text, match.start(), match.end())
+        if ".limit(" in stmt:
+            continue
+        findings.append(
+            Finding(
+                path,
+                line_number(text, match.start()),
+                "Review unbounded TypeScript query collection; add `limit()` or stream batches.",
+                stmt.splitlines()[0].strip(),
+            )
+        )
+    return findings
+
+
+def scan_file(path: Path) -> list[Finding]:
+    text = path.read_text(encoding="utf-8", errors="replace")
+    if path.suffix == ".py":
+        return scan_python(path, text)
+    if path.suffix in {".ts", ".tsx"}:
+        return scan_typescript(path, text)
+    return []
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("paths", nargs="+", type=Path)
+    parser.add_argument(
+        "--no-fail", action="store_true", help="Always exit 0 after reporting findings."
+    )
+    args = parser.parse_args()
+
+    findings: list[Finding] = []
+    for path in iter_files(args.paths):
+        findings.extend(scan_file(path))
+
+    for finding in findings:
+        print(f"{finding.path}:{finding.line}: {finding.message}")
+        print(f"  {finding.text}")
+
+    if findings:
+        print(
+            f"\n{len(findings)} finding(s). Review manually; bounded query result conversion may be OK."
+        )
+    return 0 if args.no_fail or not findings else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.31.0-beta.6"
+current_version = "0.32.0-beta.1"
 parse = """(?x)
     (?P<major>0|[1-9]\\d*)\\.
     (?P<minor>0|[1-9]\\d*)\\.

.github/workflows/build_linux_wheel/action.yml

@@ -34,15 +34,16 @@ runs:
         maturin-version: "1.12.4"
         command: build
         working-directory: python
-        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"
+        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc"
         target: x86_64-unknown-linux-gnu
         manylinux: ${{ inputs.manylinux }}
         args: ${{ inputs.args }}
         before-script-linux: |
           set -e
-          curl -L https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-$(uname -m).zip > /tmp/protoc.zip \
-            && unzip /tmp/protoc.zip -d /usr/local \
-            && rm /tmp/protoc.zip
+          curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-x86_64.zip -o /tmp/protoc.zip
+          unzip /tmp/protoc.zip -d /usr/local
+          rm /tmp/protoc.zip
+          /usr/local/bin/protoc --version
     - name: Build Arm Manylinux Wheel
       if: ${{ inputs.arm-build == 'true' }}
       uses: PyO3/maturin-action@v1
@@ -50,13 +51,14 @@ runs:
         maturin-version: "1.12.4"
         command: build
         working-directory: python
-        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"
+        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc"
         target: aarch64-unknown-linux-gnu
         manylinux: ${{ inputs.manylinux }}
         args: ${{ inputs.args }}
         before-script-linux: |
           set -e
-          yum install -y clang \
-            && curl -L https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-aarch_64.zip > /tmp/protoc.zip \
-            && unzip /tmp/protoc.zip -d /usr/local \
-            && rm /tmp/protoc.zip
+          yum install -y clang
+          curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-aarch_64.zip -o /tmp/protoc.zip
+          unzip /tmp/protoc.zip -d /usr/local
+          rm /tmp/protoc.zip
+          /usr/local/bin/protoc --version

.github/workflows/npm-publish.yml

@@ -103,7 +103,7 @@ jobs:
             features: fp16kernels
             pre_build: brew install protobuf
           - target: x86_64-pc-windows-msvc
-            host: windows-latest
+            host: windows-2025-8x-x64
             features: ","
             pre_build: |-
               choco install --no-progress protoc ninja nasm
@@ -111,12 +111,21 @@ jobs:
               # There is an issue where choco doesn't add nasm to the path
               export PATH="$PATH:/c/Program Files/NASM"
               nasm -v
+              # Fat LTO of the cdylib is single-threaded and the peak-memory
+              # step of the build, and had started hitting rustc-LLVM OOM on the
+              # Windows runners. ThinLTO parallelizes it across the runner's
+              # cores and keeps peak memory well under the limit.
+              export CARGO_PROFILE_RELEASE_LTO=thin
+              export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
           - target: aarch64-pc-windows-msvc
-            host: windows-latest
+            host: windows-2025-8x-x64
             features: ","
             pre_build: |-
                 choco install --no-progress protoc
                 rustup target add aarch64-pc-windows-msvc
+                # See ThinLTO note on the x86_64-pc-windows-msvc target above.
+                export CARGO_PROFILE_RELEASE_LTO=thin
+                export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
           - target: x86_64-unknown-linux-gnu
             host: ubuntu-latest
             features: fp16kernels

.github/workflows/rust.yml

@@ -125,10 +125,26 @@ jobs:
       - uses: rui314/setup-mold@v1
       - name: Make Swap
         run: |
-          sudo fallocate -l 16G /swapfile
-          sudo chmod 600 /swapfile
-          sudo mkswap /swapfile
-          sudo swapon /swapfile
+          swapfile=/swapfile
+          min_swap_bytes=$((15 * 1024 * 1024 * 1024))
+          active_swap_bytes="$(sudo swapon --show=NAME,SIZE --bytes --noheadings | awk '$1 == "/swapfile" { print $2 }')"
+          if [ -n "$active_swap_bytes" ]; then
+            if [ "$active_swap_bytes" -ge "$min_swap_bytes" ]; then
+              echo "/swapfile is already active with enough space; skipping swap creation"
+              exit 0
+            fi
+            echo "/swapfile is already active but smaller than 16G; using /mnt/lancedb-swapfile"
+            swapfile=/mnt/lancedb-swapfile
+          fi
+          if sudo swapon --show=NAME --noheadings | grep -Fxq "$swapfile"; then
+            echo "$swapfile is already active; skipping swap creation"
+            exit 0
+          fi
+          sudo rm -f "$swapfile"
+          sudo fallocate -l 16G "$swapfile"
+          sudo chmod 600 "$swapfile"
+          sudo mkswap "$swapfile"
+          sudo swapon "$swapfile"
       - name: Build
         run: cargo build --profile ci --all-features --tests --locked --examples
       - name: Run feature tests

Cargo.toml

@@ -13,20 +13,20 @@ categories = ["database-implementations"]
 rust-version = "1.91.0"
 
 [workspace.dependencies]
-lance = { "version" = "=9.0.0-beta.16", default-features = false, "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-core = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-datagen = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-file = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-io = { "version" = "=9.0.0-beta.16", default-features = false, "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-index = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-linalg = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace-impls = { "version" = "=9.0.0-beta.16", default-features = false, "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-table = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-testing = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-datafusion = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-encoding = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
-lance-arrow = { "version" = "=9.0.0-beta.16", "tag" = "v9.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
+lance = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-core = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-datagen = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-file = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-io = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-index = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-linalg = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace-impls = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-table = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-testing = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-datafusion = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-encoding = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance-arrow = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
 ahash = "0.8"
 # Note that this one does not include pyarrow
 arrow = { version = "58.0.0", optional = false }
@@ -54,6 +54,8 @@ half = { "version" = "2.7.1", default-features = false, features = [
 ] }
 futures = "0"
 log = "0.4"
+metrics = "0.24"
+metrics-util = "0.19"
 moka = { version = "0.12", features = ["future"] }
 object_store = "0.13.2"
 pin-project = "1.0.7"

deny.toml

@@ -51,18 +51,6 @@ ignore = [
     # https://rustsec.org/advisories/RUSTSEC-2024-0436
     { id = "RUSTSEC-2024-0436", reason = "transitive via datafusion; awaiting ecosystem migration" },
 
-    # encoding: unmaintained. Reached through lindera-dictionary, which is
-    # required by the native Lindera tokenizer path. Lindera has not migrated
-    # off this crate yet.
-    # https://rustsec.org/advisories/RUSTSEC-2021-0153
-    { id = "RUSTSEC-2021-0153", reason = "transitive via lindera-dictionary for native Lindera tokenizer" },
-
-    # fast-float: unsound and unmaintained. Reached only through polars-arrow
-    # from the optional Polars integration; replacement requires a Polars
-    # dependency upgrade.
-    # https://rustsec.org/advisories/RUSTSEC-2024-0379
-    { id = "RUSTSEC-2024-0379", reason = "transitive via polars-arrow; waiting on Polars migration" },
-
     # tantivy: segfault on malformed input due to missing bounds check.
     # Pulled in via lance for full-text search. We only feed tantivy
     # documents we construct ourselves, not attacker-controlled bytes.
@@ -80,18 +68,6 @@ ignore = [
     # https://rustsec.org/advisories/RUSTSEC-2025-0119
     { id = "RUSTSEC-2025-0119", reason = "transitive via hf-hub/indicatif; cosmetic formatting crate" },
 
-    # bincode: unmaintained. Reached through lindera and lindera-dictionary,
-    # which are required by the native Lindera tokenizer path. Lindera has not
-    # migrated to another serialization format yet.
-    # https://rustsec.org/advisories/RUSTSEC-2025-0141
-    { id = "RUSTSEC-2025-0141", reason = "transitive via lindera/lindera-dictionary for native Lindera tokenizer" },
-
-    # lru: soundness issue in IterMut. Reached only through aws-sdk-s3 in
-    # LanceDB's dev-dependency graph; LanceDB does not use that iterator
-    # directly. Clearing this requires the AWS SDK chain to update lru.
-    # https://rustsec.org/advisories/RUSTSEC-2026-0002
-    { id = "RUSTSEC-2026-0002", reason = "transitive via aws-sdk-s3 dev-dependency; waiting on AWS SDK lru upgrade" },
-
     # rustls-webpki 0.101.7 (old major line): name-constraint checks for
     # URI / wildcard names. Pulled in only via the legacy rustls 0.21 chain
     # from aws-smithy-http-client. The 0.103 line we actively use is patched.
@@ -108,12 +84,6 @@ ignore = [
     # https://rustsec.org/advisories/RUSTSEC-2026-0104
     { id = "RUSTSEC-2026-0104", reason = "only affects rustls-webpki 0.101 from legacy aws-smithy/rustls 0.21 chain" },
 
-    # rand 0.8.5: soundness issue only when ThreadRng reseeds inside a custom
-    # logger. Reached through several transitive chains. LanceDB does not use
-    # rand from a custom logger; upgrade once all pinned chains accept 0.8.6+.
-    # https://rustsec.org/advisories/RUSTSEC-2026-0097
-    { id = "RUSTSEC-2026-0097", reason = "transitive rand 0.8.5; LanceDB does not call ThreadRng from custom logging" },
-
     # pyo3 advisories in the Python bindings; tracked pending a patched pyo3 release.
     # https://rustsec.org/advisories/RUSTSEC-2026-0176
     # https://rustsec.org/advisories/RUSTSEC-2026-0177

docs/src/java/java.md

@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
 <dependency>
     <groupId>com.lancedb</groupId>
     <artifactId>lancedb-core</artifactId>
-    <version>0.31.0-beta.6</version>
+    <version>0.32.0-beta.1</version>
 </dependency>
 ```
 

docs/src/js/classes/Table.md

@@ -398,6 +398,26 @@ Drop an index from the table.
 
 ***
 
+### getLsmWriteSpec()
+
+```ts
+abstract getLsmWriteSpec(): Promise<undefined | LsmWriteSpec>
+```
+
+Read the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) currently installed on this table.
+
+Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
+spec has been set, or it was removed with [Table#unsetLsmWriteSpec](Table.md#unsetlsmwritespec)).
+The returned spec — including its `maintainedIndexes` and
+`writerConfigDefaults` — mirrors what was passed to
+[Table#setLsmWriteSpec](Table.md#setlsmwritespec).
+
+#### Returns
+
+`Promise`&lt;`undefined` \| [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)&gt;
+
+***
+
 ### indexStats()
 
 ```ts
@@ -914,6 +934,32 @@ Return the table as an arrow table
 
 ***
 
+### tokenize()
+
+```ts
+abstract tokenize(query, options): Promise<FtsToken[]>
+```
+
+Tokenize a full-text search query using the tokenizer configured on an FTS index.
+
+Specify exactly one of `column` or `indexName`.
+
+Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
+the client process from index metadata. For remote tables, this means the
+same tokenizer model files must also exist locally.
+
+#### Parameters
+
+* **query**: `string`
+
+* **options**: [`TokenizeTableOptions`](../type-aliases/TokenizeTableOptions.md)
+
+#### Returns
+
+`Promise`&lt;[`FtsToken`](../interfaces/FtsToken.md)[]&gt;
+
+***
+
 ### unsetLsmWriteSpec()
 
 ```ts

docs/src/js/functions/instrumentLanceDbMetrics.md

@@ -0,0 +1,42 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / instrumentLanceDbMetrics
+
+# Function: instrumentLanceDbMetrics()
+
+```ts
+function instrumentLanceDbMetrics(meterProvider?): boolean
+```
+
+Register LanceDB metrics as OpenTelemetry observable instruments.
+
+Installs a process-global metrics recorder and creates one observable
+instrument per LanceDB metric (currently object store request counts, bytes,
+latency, errors, and throttles) on the given (or global) `MeterProvider`. The
+configured `MetricReader` then collects them on its own schedule.
+
+Counters and gauges map directly to observable counters/gauges. Because
+OpenTelemetry has no asynchronous histogram instrument, each histogram is
+exported Prometheus-style as cumulative `le` bucket counts (`<name>_bucket`,
+with an `le` attribute) plus `<name>_count` and `<name>_sum`.
+
+Requires `@opentelemetry/api` (a dependency) and, to actually export, an
+OpenTelemetry SDK such as `@opentelemetry/sdk-metrics`.
+
+## Parameters
+
+* **meterProvider?**: `MeterProvider`
+    The provider to register instruments on. Defaults to the
+    global provider from `@opentelemetry/api`.
+
+## Returns
+
+`boolean`
+
+`true` if the recorder is installed and instruments are registered.
+  `false` if a different `metrics` recorder is already installed in this
+  process (only one global recorder is permitted), in which case a warning is
+  emitted and no instruments are created. Calling this more than once is safe;
+  instruments are created only on the first successful call.

docs/src/js/functions/tokenize.md

@@ -0,0 +1,26 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / tokenize
+
+# Function: tokenize()
+
+```ts
+function tokenize(query, options?): Promise<FtsToken[]>
+```
+
+Tokenize a full-text search query using an explicit tokenizer.
+
+This does not require a table or FTS index. The tokenizer options match
+[Index.fts](../classes/Index.md#fts).
+
+## Parameters
+
+* **query**: `string`
+
+* **options?**: `Partial`&lt;[`TokenizeOptions`](../interfaces/TokenizeOptions.md)&gt;
+
+## Returns
+
+`Promise`&lt;[`FtsToken`](../interfaces/FtsToken.md)[]&gt;

docs/src/js/globals.md

@@ -72,6 +72,7 @@
 - [FragmentStatistics](interfaces/FragmentStatistics.md)
 - [FragmentSummaryStats](interfaces/FragmentSummaryStats.md)
 - [FtsOptions](interfaces/FtsOptions.md)
+- [FtsToken](interfaces/FtsToken.md)
 - [FullTextQuery](interfaces/FullTextQuery.md)
 - [FullTextSearchOptions](interfaces/FullTextSearchOptions.md)
 - [HnswPqOptions](interfaces/HnswPqOptions.md)
@@ -107,6 +108,7 @@
 - [TimeoutConfig](interfaces/TimeoutConfig.md)
 - [TlsConfig](interfaces/TlsConfig.md)
 - [TokenResponse](interfaces/TokenResponse.md)
+- [TokenizeOptions](interfaces/TokenizeOptions.md)
 - [UpdateFieldMetadataResult](interfaces/UpdateFieldMetadataResult.md)
 - [UpdateOptions](interfaces/UpdateOptions.md)
 - [UpdateResult](interfaces/UpdateResult.md)
@@ -116,6 +118,7 @@
 
 ## Type Aliases
 
+- [BaseTokenizer](type-aliases/BaseTokenizer.md)
 - [Data](type-aliases/Data.md)
 - [DataLike](type-aliases/DataLike.md)
 - [FieldLike](type-aliases/FieldLike.md)
@@ -125,12 +128,15 @@
 - [RecordBatchLike](type-aliases/RecordBatchLike.md)
 - [SchemaLike](type-aliases/SchemaLike.md)
 - [TableLike](type-aliases/TableLike.md)
+- [TokenizeTableOptions](type-aliases/TokenizeTableOptions.md)
 
 ## Functions
 
 - [RecordBatchIterator](functions/RecordBatchIterator.md)
 - [connect](functions/connect.md)
 - [connectNamespace](functions/connectNamespace.md)
+- [instrumentLanceDbMetrics](functions/instrumentLanceDbMetrics.md)
 - [makeArrowTable](functions/makeArrowTable.md)
 - [packBits](functions/packBits.md)
 - [permutationBuilder](functions/permutationBuilder.md)
+- [tokenize](functions/tokenize.md)

docs/src/js/interfaces/FtsOptions.md

@@ -23,7 +23,7 @@ whether to remove punctuation
 ### baseTokenizer?
 
 ```ts
-optional baseTokenizer: "raw" | "simple" | "whitespace" | "ngram";
+optional baseTokenizer: BaseTokenizer;
 ```
 
 The tokenizer to use when building the index.
@@ -37,6 +37,10 @@ The following tokenizers are available:
 
 "raw" - Raw tokenizer. This tokenizer does not split the text into tokens and indexes the entire text as a single token.
 
+"icu" - ICU dictionary-based word segmentation.
+
+"icu/split" - ICU segmentation with simple-style delimiter splitting.
+
 ***
 
 ### language?

docs/src/js/interfaces/FtsToken.md

@@ -0,0 +1,29 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / FtsToken
+
+# Interface: FtsToken
+
+Token produced by the tokenizer configured on a full-text search index.
+
+## Properties
+
+### position
+
+```ts
+position: number;
+```
+
+Token position used by full-text query matching.
+
+***
+
+### text
+
+```ts
+text: string;
+```
+
+Token text after tokenizer filters have been applied.

docs/src/js/interfaces/TokenizeOptions.md

@@ -0,0 +1,109 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / TokenizeOptions
+
+# Interface: TokenizeOptions
+
+Options for tokenizing a full-text search query without a table index.
+
+## Properties
+
+### asciiFolding?
+
+```ts
+optional asciiFolding: boolean;
+```
+
+Whether to fold ASCII characters.
+
+***
+
+### baseTokenizer?
+
+```ts
+optional baseTokenizer: BaseTokenizer;
+```
+
+The tokenizer to use. The default is "simple".
+
+***
+
+### language?
+
+```ts
+optional language: string;
+```
+
+Language for stemming and stop words.
+
+***
+
+### lowercase?
+
+```ts
+optional lowercase: boolean;
+```
+
+Whether to lowercase tokens.
+
+***
+
+### maxTokenLength?
+
+```ts
+optional maxTokenLength: number;
+```
+
+Maximum token length; tokens longer than this are ignored.
+
+***
+
+### ngramMaxLength?
+
+```ts
+optional ngramMaxLength: number;
+```
+
+N-gram maximum length.
+
+***
+
+### ngramMinLength?
+
+```ts
+optional ngramMinLength: number;
+```
+
+N-gram minimum length.
+
+***
+
+### prefixOnly?
+
+```ts
+optional prefixOnly: boolean;
+```
+
+Whether to only emit token prefixes for the n-gram tokenizer.
+
+***
+
+### removeStopWords?
+
+```ts
+optional removeStopWords: boolean;
+```
+
+Whether to remove stop words.
+
+***
+
+### stem?
+
+```ts
+optional stem: boolean;
+```
+
+Whether to stem tokens.

docs/src/js/type-aliases/BaseTokenizer.md

@@ -0,0 +1,19 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / BaseTokenizer
+
+# Type Alias: BaseTokenizer
+
+```ts
+type BaseTokenizer:
+  | "simple"
+  | "whitespace"
+  | "raw"
+  | "ngram"
+  | "icu"
+  | "icu/split"
+  | `jieba/${string}`
+  | `lindera/${string}`;
+```

docs/src/js/type-aliases/TokenizeTableOptions.md

@@ -0,0 +1,11 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / TokenizeTableOptions
+
+# Type Alias: TokenizeTableOptions
+
+```ts
+type TokenizeTableOptions: object | object;
+```

java/lancedb-core/pom.xml

@@ -8,7 +8,7 @@
     <parent>
       <groupId>com.lancedb</groupId>
       <artifactId>lancedb-parent</artifactId>
-      <version>0.31.0-beta.6</version>
+      <version>0.32.0-beta.1</version>
       <relativePath>../pom.xml</relativePath>
     </parent>
 

java/pom.xml

@@ -6,7 +6,7 @@
 
     <groupId>com.lancedb</groupId>
     <artifactId>lancedb-parent</artifactId>
-    <version>0.31.0-beta.6</version>
+    <version>0.32.0-beta.1</version>
     <packaging>pom</packaging>
     <name>${project.artifactId}</name>
     <description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
     <properties>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         <arrow.version>15.0.0</arrow.version>
-        <lance-core.version>9.0.0-beta.16</lance-core.version>
+        <lance-core.version>9.0.0-beta.23</lance-core.version>
         <spotless.skip>false</spotless.skip>
         <spotless.version>2.30.0</spotless.version>
         <spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>

nodejs/Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "lancedb-nodejs"
 edition.workspace = true
-version = "0.31.0-beta.6"
+version = "0.32.0-beta.1"
 publish = false
 license.workspace = true
 description.workspace = true
@@ -44,6 +44,6 @@ aws-lc-rs = "=1.16.3"
 napi-build = "2.3.1"
 
 [features]
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/goosefs", "lancedb/metrics-otel"]
 fp16kernels = ["lancedb/fp16kernels"]
 remote = ["lancedb/remote"]

nodejs/__test__/otel.test.ts

@@ -0,0 +1,114 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+import {
+  MeterProvider,
+  type MetricData,
+  MetricReader,
+} from "@opentelemetry/sdk-metrics";
+import * as tmp from "tmp";
+import { connect, instrumentLanceDbMetrics } from "../lancedb";
+// snapshotLancedbMetrics is internal plumbing (not part of the public API), so
+// it is imported from the native module rather than the package entry point.
+import { snapshotLancedbMetrics } from "../lancedb/native";
+
+// The metrics recorder is process-global and installed once, so the whole
+// bridge is exercised in a single test to avoid cross-test global-state coupling.
+
+// A minimal pull-based reader whose `collect()` we drive directly, invoking the
+// observable-instrument callbacks. `@opentelemetry/sdk-metrics` ships no
+// in-memory reader, so we subclass the abstract base.
+class TestMetricReader extends MetricReader {
+  protected async onForceFlush(): Promise<void> {
+    // no-op: collection is driven directly via collect()
+  }
+  protected async onShutdown(): Promise<void> {
+    // no-op: nothing to release
+  }
+}
+
+async function metricsByName(
+  reader: TestMetricReader,
+): Promise<Map<string, MetricData>> {
+  const collected = await reader.collect();
+  const result = new Map<string, MetricData>();
+  for (const scope of collected.resourceMetrics.scopeMetrics) {
+    for (const metric of scope.metrics) {
+      result.set(metric.descriptor.name, metric);
+    }
+  }
+  return result;
+}
+
+describe("OpenTelemetry metrics bridge", () => {
+  let tmpDir: tmp.DirResult;
+  beforeEach(() => {
+    tmpDir = tmp.dirSync({ unsafeCleanup: true });
+  });
+  afterEach(() => tmpDir.removeCallback());
+
+  it("snapshot is safe to call regardless of install state", () => {
+    expect(Array.isArray(snapshotLancedbMetrics())).toBe(true);
+  });
+
+  it("exports object store metrics via observable instruments", async () => {
+    const reader = new TestMetricReader();
+    const provider = new MeterProvider({ readers: [reader] });
+    expect(instrumentLanceDbMetrics(provider)).toBe(true);
+
+    // Generate object store activity on the local filesystem (scheme "file").
+    const db = await connect(tmpDir.name);
+    const data = Array.from({ length: 256 }, (_, i) => ({ id: i }));
+    const table = await db.createTable("t", data);
+    expect(await table.countRows()).toBe(256);
+
+    const metrics = await metricsByName(reader);
+
+    const requests = metrics.get("lance_object_store_requests_total");
+    expect(requests).toBeDefined();
+    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
+    const requestPoints = (requests!.dataPoints as any[]) ?? [];
+    expect(requestPoints.length).toBeGreaterThan(0);
+    for (const p of requestPoints) {
+      // Labelled by `operation` and `base` (the store scheme by default).
+      expect(p.attributes).toHaveProperty("base");
+      expect(p.attributes).toHaveProperty("operation");
+    }
+    const totalRequests = requestPoints.reduce((acc, p) => acc + p.value, 0);
+    expect(totalRequests).toBeGreaterThan(0);
+
+    // Histograms are decomposed into bucket / count / sum observable counters.
+    const bucket = metrics.get(
+      "lance_object_store_request_duration_seconds_bucket",
+    );
+    expect(bucket).toBeDefined();
+    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
+    const bucketPoints = (bucket!.dataPoints as any[]) ?? [];
+    expect(bucketPoints.length).toBeGreaterThan(0);
+    expect(bucketPoints.every((p) => "le" in p.attributes)).toBe(true);
+    // The implicit +Inf bucket must be present.
+    expect(bucketPoints.some((p) => p.attributes.le === "+Inf")).toBe(true);
+
+    const count = metrics.get(
+      "lance_object_store_request_duration_seconds_count",
+    );
+    expect(count).toBeDefined();
+    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
+    const countPoints = (count!.dataPoints as any[]) ?? [];
+    expect(countPoints.reduce((acc, p) => acc + p.value, 0)).toBeGreaterThan(0);
+
+    const sum = metrics.get("lance_object_store_request_duration_seconds_sum");
+    expect(sum).toBeDefined();
+    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
+    const sumPoints = (sum!.dataPoints as any[]) ?? [];
+    expect(sumPoints.reduce((acc, p) => acc + p.value, 0)).toBeGreaterThan(0);
+
+    // Unit handling: only `_sum` keeps the histogram's unit (seconds); `_bucket`
+    // and `_count` observe cumulative counts and are unitless.
+    expect(sum!.descriptor.unit).toBe("s");
+    expect(bucket!.descriptor.unit).toBe("");
+    expect(count!.descriptor.unit).toBe("");
+
+    await provider.shutdown();
+  });
+});

nodejs/__test__/table.test.ts

@@ -16,6 +16,7 @@ import {
   PhraseQuery,
   Table,
   connect,
+  tokenize,
 } from "../lancedb";
 import {
   Table as ArrowTable,
@@ -2307,6 +2308,75 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
       expect(results2[0].text).toBe(data[1].text);
     });
 
+    test("tokenizes FTS queries by column or index name", async () => {
+      const db = await connect(tmpDir.name);
+      const data = [
+        {
+          text: "Running in cafés",
+          japanese: "Hello, こんにちは世界!",
+          vector: [0.1, 0.2, 0.3],
+        },
+      ];
+      const table = await db.createTable("test", data);
+      await table.createIndex("text", {
+        config: Index.fts({ baseTokenizer: "simple" }),
+      });
+      await table.createIndex("japanese", {
+        config: Index.fts({
+          baseTokenizer: "icu",
+          stem: false,
+          removeStopWords: false,
+        }),
+        name: "japanese_icu_idx",
+      });
+
+      await expect(table.tokenize("hello", {} as never)).rejects.toThrow(
+        "Specify exactly one",
+      );
+      await expect(
+        table.tokenize("hello", {
+          column: "text",
+          indexName: "text_idx",
+        } as never),
+      ).rejects.toThrow("Specify exactly one");
+
+      const simpleTokens = await table.tokenize("Running in cafés", {
+        column: "text",
+      });
+      expect(simpleTokens).toEqual([
+        { text: "run", position: 0 },
+        { text: "cafe", position: 2 },
+      ]);
+
+      const icuTokens = await table.tokenize("Hello, こんにちは世界!", {
+        indexName: "japanese_icu_idx",
+      });
+      expect(icuTokens).toEqual([
+        { text: "hello", position: 0 },
+        { text: "こんにちは", position: 1 },
+        { text: "世界", position: 2 },
+      ]);
+
+      const directSimpleTokens = await tokenize("Running in cafés", {
+        baseTokenizer: "simple",
+      });
+      expect(directSimpleTokens).toEqual([
+        { text: "run", position: 0 },
+        { text: "cafe", position: 2 },
+      ]);
+
+      const directIcuTokens = await tokenize("Hello, こんにちは世界!", {
+        baseTokenizer: "icu",
+        stem: false,
+        removeStopWords: false,
+      });
+      expect(directIcuTokens).toEqual([
+        { text: "hello", position: 0 },
+        { text: "こんにちは", position: 1 },
+        { text: "世界", position: 2 },
+      ]);
+    });
+
     test("full text search fast search", async () => {
       const db = await connect(tmpDir.name);
       const data = [{ text: "hello world", vector: [0.1, 0.2, 0.3], id: 1 }];
@@ -2992,6 +3062,56 @@ describe("setLsmWriteSpec / unsetLsmWriteSpec", () => {
       }),
     ).rejects.toThrow();
   });
+
+  it("reads back the installed spec via getLsmWriteSpec", async () => {
+    const conn = await connect(tmpDir.name);
+    const table = await makeTable(conn);
+    await table.setUnenforcedPrimaryKey("id");
+
+    // Nothing installed yet.
+    expect(await table.getLsmWriteSpec()).toBeUndefined();
+
+    // A real scalar index is needed to name it as a maintained index.
+    await table.add([{ id: 1 }, { id: 2 }, { id: 3 }]);
+    await table.createIndex("id");
+    const indexName = (await table.listIndices())[0].name;
+
+    // Bucket spec round-trips, including maintained indexes and writer config
+    // defaults. Lance writer-config keys are canonically snake_case.
+    // biome-ignore lint/style/useNamingConvention: Lance writer-config keys are snake_case
+    const writerConfigDefaults = { durable_write: "false" };
+    await table.setLsmWriteSpec({
+      specType: "bucket",
+      column: "id",
+      numBuckets: 4,
+      maintainedIndexes: [indexName],
+      writerConfigDefaults,
+    });
+    const spec = await table.getLsmWriteSpec();
+    expect(spec).toBeDefined();
+    expect(spec?.specType).toBe("bucket");
+    expect(spec?.column).toBe("id");
+    expect(spec?.numBuckets).toBe(4);
+    expect(spec?.maintainedIndexes).toEqual([indexName]);
+    expect(spec?.writerConfigDefaults).toEqual(writerConfigDefaults);
+
+    // After unset, undefined again.
+    await table.unsetLsmWriteSpec();
+    expect(await table.getLsmWriteSpec()).toBeUndefined();
+
+    // Identity round-trips (column recovered from the schema).
+    await table.setLsmWriteSpec({ specType: "identity", column: "id" });
+    const identity = await table.getLsmWriteSpec();
+    expect(identity?.specType).toBe("identity");
+    expect(identity?.column).toBe("id");
+    await table.unsetLsmWriteSpec();
+
+    // Unsharded round-trips (no routing column).
+    await table.setLsmWriteSpec({ specType: "unsharded" });
+    const unsharded = await table.getLsmWriteSpec();
+    expect(unsharded?.specType).toBe("unsharded");
+    expect(unsharded?.column).toBeFalsy();
+  });
 });
 
 describe("LSM merge insert", () => {

nodejs/lancedb/index.ts

@@ -13,13 +13,21 @@ import {
   Connection as LanceDbConnection,
   JsHeaderProvider as NativeJsHeaderProvider,
   Session,
+  tokenize as nativeTokenize,
 } from "./native.js";
 
 import { HeaderProvider } from "./header";
+import type { BaseTokenizer } from "./indices";
+import type { FtsToken } from "./table";
 
 // Re-export native header provider for use with connectWithHeaderProvider
 export { JsHeaderProvider as NativeJsHeaderProvider } from "./native.js";
 
+// OpenTelemetry metrics bridge. Only the high-level entry point is public; the
+// underlying recorder/catalog/snapshot functions remain internal plumbing that
+// `otel.ts` consumes from the native module.
+export { instrumentLanceDbMetrics } from "./otel";
+
 export {
   AddColumnsSql,
   ConnectionOptions,
@@ -109,6 +117,7 @@ export {
   HnswPqOptions,
   HnswSqOptions,
   FtsOptions,
+  BaseTokenizer,
 } from "./indices";
 
 export {
@@ -119,6 +128,8 @@ export {
   OptimizeOptions,
   Version,
   WriteProgress,
+  FtsToken,
+  TokenizeTableOptions,
   LsmWriteSpec,
   ColumnAlteration,
   FieldMetadataUpdate,
@@ -150,6 +161,68 @@ export {
 } from "./arrow";
 export { IntoSql, packBits } from "./util";
 
+/**
+ * Options for tokenizing a full-text search query without a table index.
+ */
+export interface TokenizeOptions {
+  /**
+   * The tokenizer to use. The default is "simple".
+   */
+  baseTokenizer?: BaseTokenizer;
+
+  /** Language for stemming and stop words. */
+  language?: string;
+
+  /** Maximum token length; tokens longer than this are ignored. */
+  maxTokenLength?: number;
+
+  /** Whether to lowercase tokens. */
+  lowercase?: boolean;
+
+  /** Whether to stem tokens. */
+  stem?: boolean;
+
+  /** Whether to remove stop words. */
+  removeStopWords?: boolean;
+
+  /** Whether to fold ASCII characters. */
+  asciiFolding?: boolean;
+
+  /** N-gram minimum length. */
+  ngramMinLength?: number;
+
+  /** N-gram maximum length. */
+  ngramMaxLength?: number;
+
+  /** Whether to only emit token prefixes for the n-gram tokenizer. */
+  prefixOnly?: boolean;
+}
+
+/**
+ * Tokenize a full-text search query using an explicit tokenizer.
+ *
+ * This does not require a table or FTS index. The tokenizer options match
+ * {@link Index.fts}.
+ */
+export async function tokenize(
+  query: string,
+  options?: Partial<TokenizeOptions>,
+): Promise<FtsToken[]> {
+  return await nativeTokenize(
+    query,
+    options?.baseTokenizer,
+    options?.language,
+    options?.maxTokenLength,
+    options?.lowercase,
+    options?.stem,
+    options?.removeStopWords,
+    options?.asciiFolding,
+    options?.ngramMinLength,
+    options?.ngramMaxLength,
+    options?.prefixOnly,
+  );
+}
+
 /**
  * Connect to a LanceDB instance at the given URI.
  *

nodejs/lancedb/indices.ts

@@ -486,6 +486,16 @@ export interface IvfFlatOptions {
   sampleRate?: number;
 }
 
+export type BaseTokenizer =
+  | "simple"
+  | "whitespace"
+  | "raw"
+  | "ngram"
+  | "icu"
+  | "icu/split"
+  | `jieba/${string}`
+  | `lindera/${string}`;
+
 /**
  * Options to create a full text search index
  */
@@ -509,8 +519,12 @@ export interface FtsOptions {
    * "whitespace" - Whitespace tokenizer. This tokenizer splits the text into tokens using whitespace as a delimiter.
    *
    * "raw" - Raw tokenizer. This tokenizer does not split the text into tokens and indexes the entire text as a single token.
+   *
+   * "icu" - ICU dictionary-based word segmentation.
+   *
+   * "icu/split" - ICU segmentation with simple-style delimiter splitting.
    */
-  baseTokenizer?: "simple" | "whitespace" | "raw" | "ngram";
+  baseTokenizer?: BaseTokenizer;
 
   /**
    * language for stemming and stop words

nodejs/lancedb/otel.ts

@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+import {
+  type Attributes,
+  type MeterProvider,
+  type ObservableResult,
+  metrics,
+} from "@opentelemetry/api";
+
+import {
+  lancedbMetricsCatalog,
+  registerLancedbMetricsRecorder,
+  snapshotLancedbMetrics,
+} from "./native";
+
+let instrumented = false;
+
+/**
+ * Register LanceDB metrics as OpenTelemetry observable instruments.
+ *
+ * Installs a process-global metrics recorder and creates one observable
+ * instrument per LanceDB metric (currently object store request counts, bytes,
+ * latency, errors, and throttles) on the given (or global) `MeterProvider`. The
+ * configured `MetricReader` then collects them on its own schedule.
+ *
+ * Counters and gauges map directly to observable counters/gauges. Because
+ * OpenTelemetry has no asynchronous histogram instrument, each histogram is
+ * exported Prometheus-style as cumulative `le` bucket counts (`<name>_bucket`,
+ * with an `le` attribute) plus `<name>_count` and `<name>_sum`.
+ *
+ * Requires `@opentelemetry/api` (a dependency) and, to actually export, an
+ * OpenTelemetry SDK such as `@opentelemetry/sdk-metrics`.
+ *
+ * @param meterProvider The provider to register instruments on. Defaults to the
+ *   global provider from `@opentelemetry/api`.
+ * @returns `true` if the recorder is installed and instruments are registered.
+ *   `false` if a different `metrics` recorder is already installed in this
+ *   process (only one global recorder is permitted), in which case a warning is
+ *   emitted and no instruments are created. Calling this more than once is safe;
+ *   instruments are created only on the first successful call.
+ */
+export function instrumentLanceDbMetrics(
+  meterProvider?: MeterProvider,
+): boolean {
+  if (!registerLancedbMetricsRecorder()) {
+    console.warn(
+      "Could not install the LanceDB metrics recorder: another `metrics` " +
+        "recorder is already installed in this process. LanceDB metrics will " +
+        "not be exported via OpenTelemetry.",
+    );
+    return false;
+  }
+
+  if (instrumented) {
+    return true;
+  }
+
+  const provider = meterProvider ?? metrics.getMeterProvider();
+  const meter = provider.getMeter("lancedb");
+
+  const scalarCallback = (metricName: string) => (result: ObservableResult) => {
+    for (const point of snapshotLancedbMetrics()) {
+      if (point.name === metricName && point.value != null) {
+        result.observe(point.value, point.attributes);
+      }
+    }
+  };
+
+  const bucketCallback = (metricName: string) => (result: ObservableResult) => {
+    for (const point of snapshotLancedbMetrics()) {
+      if (point.name !== metricName || point.buckets == null) {
+        continue;
+      }
+      for (const bucket of point.buckets) {
+        const attributes: Attributes = {
+          ...point.attributes,
+          le: bucket.le,
+        };
+        result.observe(bucket.cumulativeCount, attributes);
+      }
+    }
+  };
+
+  const fieldCallback =
+    (metricName: string, field: "count" | "sum") =>
+    (result: ObservableResult) => {
+      for (const point of snapshotLancedbMetrics()) {
+        if (point.name !== metricName) {
+          continue;
+        }
+        const value = point[field];
+        if (value != null) {
+          result.observe(value, point.attributes);
+        }
+      }
+    };
+
+  for (const desc of lancedbMetricsCatalog()) {
+    const unit = desc.unit ?? "";
+    if (desc.kind === "counter") {
+      const counter = meter.createObservableCounter(desc.name, {
+        unit,
+        description: desc.description,
+      });
+      counter.addCallback(scalarCallback(desc.name));
+    } else if (desc.kind === "gauge") {
+      const gauge = meter.createObservableGauge(desc.name, {
+        unit,
+        description: desc.description,
+      });
+      gauge.addCallback(scalarCallback(desc.name));
+    } else if (desc.kind === "histogram") {
+      // `_bucket` and `_count` observe cumulative sample counts, not the
+      // histogram's measured quantity, so they are unitless; only `_sum`
+      // carries the histogram's unit.
+      const bucket = meter.createObservableCounter(`${desc.name}_bucket`, {
+        description: `${desc.description} (cumulative buckets)`,
+      });
+      bucket.addCallback(bucketCallback(desc.name));
+
+      const count = meter.createObservableCounter(`${desc.name}_count`, {
+        description: `${desc.description} (count)`,
+      });
+      count.addCallback(fieldCallback(desc.name, "count"));
+
+      const sum = meter.createObservableCounter(`${desc.name}_sum`, {
+        unit,
+        description: `${desc.description} (sum)`,
+      });
+      sum.addCallback(fieldCallback(desc.name, "sum"));
+    }
+  }
+
+  instrumented = true;
+  return true;
+}

nodejs/lancedb/table.ts

@@ -158,6 +158,26 @@ export interface Version {
   metadata: Record<string, string>;
 }
 
+/** Token produced by the tokenizer configured on a full-text search index. */
+export interface FtsToken {
+  /** Token text after tokenizer filters have been applied. */
+  text: string;
+  /** Token position used by full-text query matching. */
+  position: number;
+}
+
+export type TokenizeTableOptions =
+  | {
+      /** FTS-indexed column whose tokenizer should be used. */
+      column: string;
+      indexName?: never;
+    }
+  | {
+      /** Name of the FTS index whose tokenizer should be used. */
+      indexName: string;
+      column?: never;
+    };
+
 /**
  * Specification selecting Lance's MemWAL LSM-style write path for
  * `mergeInsert`.
@@ -585,6 +605,17 @@ export abstract class Table {
    * @returns {Promise<void>}
    */
   abstract unsetLsmWriteSpec(): Promise<void>;
+  /**
+   * Read the {@link LsmWriteSpec} currently installed on this table.
+   *
+   * Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
+   * spec has been set, or it was removed with {@link Table#unsetLsmWriteSpec}).
+   * The returned spec — including its `maintainedIndexes` and
+   * `writerConfigDefaults` — mirrors what was passed to
+   * {@link Table#setLsmWriteSpec}.
+   * @returns {Promise<LsmWriteSpec | undefined>}
+   */
+  abstract getLsmWriteSpec(): Promise<LsmWriteSpec | undefined>;
   /**
    * Drain and close any cached MemWAL shard writers held for this table.
    *
@@ -705,6 +736,19 @@ export abstract class Table {
   abstract optimize(options?: Partial<OptimizeOptions>): Promise<OptimizeStats>;
   /** List all indices that have been created with {@link Table.createIndex} */
   abstract listIndices(): Promise<IndexConfig[]>;
+  /**
+   * Tokenize a full-text search query using the tokenizer configured on an FTS index.
+   *
+   * Specify exactly one of `column` or `indexName`.
+   *
+   * Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
+   * the client process from index metadata. For remote tables, this means the
+   * same tokenizer model files must also exist locally.
+   */
+  abstract tokenize(
+    query: string,
+    options: TokenizeTableOptions,
+  ): Promise<FtsToken[]>;
   /** Return the table as an arrow table */
   abstract toArrow(): Promise<ArrowTable>;
 
@@ -1091,6 +1135,15 @@ export class LocalTable extends Table {
     return await this.inner.unsetLsmWriteSpec();
   }
 
+  async getLsmWriteSpec(): Promise<LsmWriteSpec | undefined> {
+    // The native binding types `specType` as a plain `string`; narrow it back
+    // to the public union. The Rust `From` impl only ever emits one of the
+    // three valid values, so the cast is safe.
+    return ((await this.inner.getLsmWriteSpec()) ?? undefined) as
+      | LsmWriteSpec
+      | undefined;
+  }
+
   async closeLsmWriters(): Promise<void> {
     return await this.inner.closeLsmWriters();
   }
@@ -1153,6 +1206,17 @@ export class LocalTable extends Table {
     return await this.inner.listIndices();
   }
 
+  async tokenize(
+    query: string,
+    options: TokenizeTableOptions,
+  ): Promise<FtsToken[]> {
+    return await this.inner.tokenize(
+      query,
+      options?.column,
+      options?.indexName,
+    );
+  }
+
   async toArrow(): Promise<ArrowTable> {
     return await this.query().toArrow();
   }

nodejs/npm/darwin-arm64/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.31.0-beta.6",
+	"version": "0.32.0-beta.1",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",

nodejs/npm/linux-arm64-gnu/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.31.0-beta.6",
+	"version": "0.32.0-beta.1",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",

nodejs/npm/linux-arm64-musl/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-musl",
-	"version": "0.31.0-beta.6",
+	"version": "0.32.0-beta.1",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-musl.node",

nodejs/npm/linux-x64-gnu/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.31.0-beta.6",
+	"version": "0.32.0-beta.1",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",

nodejs/npm/linux-x64-musl/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-musl",
-	"version": "0.31.0-beta.6",
+	"version": "0.32.0-beta.1",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-musl.node",

nodejs/npm/win32-arm64-msvc/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lancedb/lancedb-win32-arm64-msvc",
-  "version": "0.31.0-beta.6",
+  "version": "0.32.0-beta.1",
   "os": [
     "win32"
   ],

nodejs/npm/win32-x64-msvc/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.31.0-beta.6",
+	"version": "0.32.0-beta.1",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",

nodejs/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "@lancedb/lancedb",
-  "version": "0.31.0-beta.6",
+  "version": "0.32.0-beta.1",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "@lancedb/lancedb",
-      "version": "0.31.0-beta.6",
+      "version": "0.32.0-beta.1",
       "cpu": [
         "x64",
         "arm64"
@@ -18,6 +18,7 @@
         "win32"
       ],
       "dependencies": {
+        "@opentelemetry/api": "^1.9.0",
         "reflect-metadata": "^0.2.2"
       },
       "devDependencies": {
@@ -27,6 +28,7 @@
         "@biomejs/biome": "^1.7.3",
         "@jest/globals": "^29.7.0",
         "@napi-rs/cli": "3.7.0",
+        "@opentelemetry/sdk-metrics": "^1.30.0",
         "@types/axios": "^0.14.0",
         "@types/jest": "^29.1.2",
         "@types/node": "22.7.4",
@@ -4148,6 +4150,75 @@
         "@octokit/openapi-types": "^27.0.0"
       }
     },
+    "node_modules/@opentelemetry/api": {
+      "version": "1.9.1",
+      "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.1.tgz",
+      "integrity": "sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==",
+      "license": "Apache-2.0",
+      "engines": {
+        "node": ">=8.0.0"
+      }
+    },
+    "node_modules/@opentelemetry/core": {
+      "version": "1.30.1",
+      "resolved": "https://registry.npmjs.org/@opentelemetry/core/-/core-1.30.1.tgz",
+      "integrity": "sha512-OOCM2C/QIURhJMuKaekP3TRBxBKxG/TWWA0TL2J6nXUtDnuCtccy49LUJF8xPFXMX+0LMcxFpCo8M9cGY1W6rQ==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "dependencies": {
+        "@opentelemetry/semantic-conventions": "1.28.0"
+      },
+      "engines": {
+        "node": ">=14"
+      },
+      "peerDependencies": {
+        "@opentelemetry/api": ">=1.0.0 <1.10.0"
+      }
+    },
+    "node_modules/@opentelemetry/resources": {
+      "version": "1.30.1",
+      "resolved": "https://registry.npmjs.org/@opentelemetry/resources/-/resources-1.30.1.tgz",
+      "integrity": "sha512-5UxZqiAgLYGFjS4s9qm5mBVo433u+dSPUFWVWXmLAD4wB65oMCoXaJP1KJa9DIYYMeHu3z4BZcStG3LC593cWA==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "dependencies": {
+        "@opentelemetry/core": "1.30.1",
+        "@opentelemetry/semantic-conventions": "1.28.0"
+      },
+      "engines": {
+        "node": ">=14"
+      },
+      "peerDependencies": {
+        "@opentelemetry/api": ">=1.0.0 <1.10.0"
+      }
+    },
+    "node_modules/@opentelemetry/sdk-metrics": {
+      "version": "1.30.1",
+      "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-metrics/-/sdk-metrics-1.30.1.tgz",
+      "integrity": "sha512-q9zcZ0Okl8jRgmy7eNW3Ku1XSgg3sDLa5evHZpCwjspw7E8Is4K/haRPDJrBcX3YSn/Y7gUvFnByNYEKQNbNog==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "dependencies": {
+        "@opentelemetry/core": "1.30.1",
+        "@opentelemetry/resources": "1.30.1"
+      },
+      "engines": {
+        "node": ">=14"
+      },
+      "peerDependencies": {
+        "@opentelemetry/api": ">=1.3.0 <1.10.0"
+      }
+    },
+    "node_modules/@opentelemetry/semantic-conventions": {
+      "version": "1.28.0",
+      "resolved": "https://registry.npmjs.org/@opentelemetry/semantic-conventions/-/semantic-conventions-1.28.0.tgz",
+      "integrity": "sha512-lp4qAiMTD4sNWW4DbKLBkfiMZ4jbAboJIGOQr5DvciMRI494OapieI9qiODpOt0XBr1LjIDy1xAGAnVs5supTA==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "engines": {
+        "node": ">=14"
+      }
+    },
     "node_modules/@protobufjs/aspromise": {
       "version": "1.1.2",
       "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",

nodejs/package.json

@@ -11,7 +11,7 @@
     "ann"
   ],
   "private": false,
-  "version": "0.31.0-beta.6",
+  "version": "0.32.0-beta.1",
   "main": "dist/index.js",
   "exports": {
     ".": "./dist/index.js",
@@ -44,6 +44,7 @@
     "@biomejs/biome": "^1.7.3",
     "@jest/globals": "^29.7.0",
     "@napi-rs/cli": "3.7.0",
+    "@opentelemetry/sdk-metrics": "^1.30.0",
     "@types/axios": "^0.14.0",
     "@types/jest": "^29.1.2",
     "@types/node": "22.7.4",
@@ -92,6 +93,7 @@
     "version": "napi version"
   },
   "dependencies": {
+    "@opentelemetry/api": "^1.9.0",
     "reflect-metadata": "^0.2.2"
   },
   "optionalDependencies": {

nodejs/pnpm-lock.yaml

@@ -8,6 +8,9 @@ importers:
 
   .:
     dependencies:
+      '@opentelemetry/api':
+        specifier: ^1.9.0
+        version: 1.9.1
       apache-arrow:
         specifier: '>=15.0.0 <=18.1.0'
         version: 18.1.0
@@ -33,6 +36,9 @@ importers:
       '@napi-rs/cli':
         specifier: 3.7.0
         version: 3.7.0(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)(@types/node@22.7.4)
+      '@opentelemetry/sdk-metrics':
+        specifier: ^1.30.0
+        version: 1.30.1(@opentelemetry/api@1.9.1)
       '@types/axios':
         specifier: ^0.14.0
         version: 0.14.4
@@ -1307,6 +1313,32 @@ packages:
   '@octokit/types@16.0.0':
     resolution: {integrity: sha512-sKq+9r1Mm4efXW1FCk7hFSeJo4QKreL/tTbR0rz/qx/r1Oa2VV83LTA/H/MuCOX7uCIJmQVRKBcbmWoySjAnSg==}
 
+  '@opentelemetry/api@1.9.1':
+    resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==}
+    engines: {node: '>=8.0.0'}
+
+  '@opentelemetry/core@1.30.1':
+    resolution: {integrity: sha512-OOCM2C/QIURhJMuKaekP3TRBxBKxG/TWWA0TL2J6nXUtDnuCtccy49LUJF8xPFXMX+0LMcxFpCo8M9cGY1W6rQ==}
+    engines: {node: '>=14'}
+    peerDependencies:
+      '@opentelemetry/api': '>=1.0.0 <1.10.0'
+
+  '@opentelemetry/resources@1.30.1':
+    resolution: {integrity: sha512-5UxZqiAgLYGFjS4s9qm5mBVo433u+dSPUFWVWXmLAD4wB65oMCoXaJP1KJa9DIYYMeHu3z4BZcStG3LC593cWA==}
+    engines: {node: '>=14'}
+    peerDependencies:
+      '@opentelemetry/api': '>=1.0.0 <1.10.0'
+
+  '@opentelemetry/sdk-metrics@1.30.1':
+    resolution: {integrity: sha512-q9zcZ0Okl8jRgmy7eNW3Ku1XSgg3sDLa5evHZpCwjspw7E8Is4K/haRPDJrBcX3YSn/Y7gUvFnByNYEKQNbNog==}
+    engines: {node: '>=14'}
+    peerDependencies:
+      '@opentelemetry/api': '>=1.3.0 <1.10.0'
+
+  '@opentelemetry/semantic-conventions@1.28.0':
+    resolution: {integrity: sha512-lp4qAiMTD4sNWW4DbKLBkfiMZ4jbAboJIGOQr5DvciMRI494OapieI9qiODpOt0XBr1LjIDy1xAGAnVs5supTA==}
+    engines: {node: '>=14'}
+
   '@protobufjs/aspromise@1.1.2':
     resolution: {integrity: sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==}
 
@@ -4925,6 +4957,27 @@ snapshots:
     dependencies:
       '@octokit/openapi-types': 27.0.0
 
+  '@opentelemetry/api@1.9.1': {}
+
+  '@opentelemetry/core@1.30.1(@opentelemetry/api@1.9.1)':
+    dependencies:
+      '@opentelemetry/api': 1.9.1
+      '@opentelemetry/semantic-conventions': 1.28.0
+
+  '@opentelemetry/resources@1.30.1(@opentelemetry/api@1.9.1)':
+    dependencies:
+      '@opentelemetry/api': 1.9.1
+      '@opentelemetry/core': 1.30.1(@opentelemetry/api@1.9.1)
+      '@opentelemetry/semantic-conventions': 1.28.0
+
+  '@opentelemetry/sdk-metrics@1.30.1(@opentelemetry/api@1.9.1)':
+    dependencies:
+      '@opentelemetry/api': 1.9.1
+      '@opentelemetry/core': 1.30.1(@opentelemetry/api@1.9.1)
+      '@opentelemetry/resources': 1.30.1(@opentelemetry/api@1.9.1)
+
+  '@opentelemetry/semantic-conventions@1.28.0': {}
+
   '@protobufjs/aspromise@1.1.2':
     optional: true
 

nodejs/src/index.rs

@@ -9,8 +9,11 @@ use lancedb::index::vector::{
     IvfFlatIndexBuilder, IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder, IvfPqIndexBuilder,
     IvfRqIndexBuilder,
 };
+use lancedb::tokenize as lancedb_tokenize;
 use napi_derive::napi;
 
+use crate::error::NapiErrorExt;
+use crate::table::FtsToken;
 use crate::util::parse_distance_type;
 
 #[napi]
@@ -30,6 +33,65 @@ impl Index {
     }
 }
 
+#[napi(catch_unwind)]
+#[allow(dead_code, clippy::too_many_arguments)]
+pub fn tokenize(
+    query: String,
+    base_tokenizer: Option<String>,
+    language: Option<String>,
+    max_token_length: Option<u32>,
+    lower_case: Option<bool>,
+    stem: Option<bool>,
+    remove_stop_words: Option<bool>,
+    ascii_folding: Option<bool>,
+    ngram_min_length: Option<u32>,
+    ngram_max_length: Option<u32>,
+    prefix_only: Option<bool>,
+) -> napi::Result<Vec<FtsToken>> {
+    let mut opts = FtsIndexBuilder::default();
+    if let Some(base_tokenizer) = base_tokenizer {
+        opts = opts.base_tokenizer(base_tokenizer);
+    }
+    if let Some(language) = language {
+        opts = opts.language(&language).map_err(|_| {
+            napi::Error::from_reason(format!(
+                "LanceDB does not support the requested language: '{}'",
+                language
+            ))
+        })?;
+    }
+    if let Some(max_token_length) = max_token_length {
+        opts = opts.max_token_length(Some(max_token_length as usize));
+    }
+    if let Some(lower_case) = lower_case {
+        opts = opts.lower_case(lower_case);
+    }
+    if let Some(stem) = stem {
+        opts = opts.stem(stem);
+    }
+    if let Some(remove_stop_words) = remove_stop_words {
+        opts = opts.remove_stop_words(remove_stop_words);
+    }
+    if let Some(ascii_folding) = ascii_folding {
+        opts = opts.ascii_folding(ascii_folding);
+    }
+    if let Some(ngram_min_length) = ngram_min_length {
+        opts = opts.ngram_min_length(ngram_min_length);
+    }
+    if let Some(ngram_max_length) = ngram_max_length {
+        opts = opts.ngram_max_length(ngram_max_length);
+    }
+    if let Some(prefix_only) = prefix_only {
+        opts = opts.ngram_prefix_only(prefix_only);
+    }
+
+    Ok(lancedb_tokenize(&query, &opts)
+        .default_error()?
+        .into_iter()
+        .map(FtsToken::from)
+        .collect())
+}
+
 #[napi]
 impl Index {
     #[napi(factory)]

nodejs/src/lib.rs

@@ -12,6 +12,7 @@ mod header;
 mod index;
 mod iterator;
 pub mod merge;
+pub mod otel;
 pub mod permutation;
 mod query;
 pub mod remote;

nodejs/src/otel.rs

@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+//! Node.js bindings over [`lancedb::metrics_otel`].
+//!
+//! The aggregation, catalog, and histogram bucketing all live in the LanceDB
+//! core crate; this module only converts the core snapshot types into napi
+//! objects and exposes the three entry points to JavaScript, where
+//! `lancedb/otel.ts` bridges them into the user's OpenTelemetry `MeterProvider`.
+
+use std::collections::HashMap;
+
+use lancedb::metrics_otel::{MetricPoint as CoreMetricPoint, MetricValue};
+use napi_derive::napi;
+
+/// One cumulative histogram bucket: all samples with value `<= le`.
+#[napi(object)]
+pub struct MetricBucket {
+    /// The inclusive upper bound of the bucket, or `"+Inf"` for the final bucket.
+    pub le: String,
+    /// Cumulative number of samples less than or equal to `le`.
+    pub cumulative_count: f64,
+}
+
+/// One aggregated metric data point. For counters and gauges only `value` is
+/// set; for histograms `buckets` (cumulative `le` counts), `count`, and `sum`
+/// are set.
+#[napi(object)]
+pub struct MetricPoint {
+    pub name: String,
+    pub kind: String,
+    pub attributes: HashMap<String, String>,
+    pub value: Option<f64>,
+    pub buckets: Option<Vec<MetricBucket>>,
+    pub count: Option<f64>,
+    pub sum: Option<f64>,
+}
+
+impl From<CoreMetricPoint> for MetricPoint {
+    fn from(point: CoreMetricPoint) -> Self {
+        let kind = point.kind.as_str().to_string();
+        let (value, buckets, count, sum) = match point.value {
+            MetricValue::Scalar(v) => (Some(v), None, None, None),
+            MetricValue::Histogram {
+                buckets,
+                count,
+                sum,
+            } => (
+                None,
+                Some(
+                    buckets
+                        .into_iter()
+                        // Counts stay well within the f64-exact integer range
+                        // (2^53), so this cast is lossless in practice and keeps
+                        // the values plain JS numbers for OpenTelemetry.
+                        .map(|(le, cumulative_count)| MetricBucket {
+                            le,
+                            cumulative_count: cumulative_count as f64,
+                        })
+                        .collect(),
+                ),
+                Some(count as f64),
+                Some(sum),
+            ),
+        };
+        Self {
+            name: point.name,
+            kind,
+            attributes: point.attributes,
+            value,
+            buckets,
+            count,
+            sum,
+        }
+    }
+}
+
+/// A described metric, used by the JavaScript layer to create instruments up front.
+#[napi(object)]
+pub struct MetricDescription {
+    pub name: String,
+    pub kind: String,
+    pub unit: Option<String>,
+    pub description: String,
+}
+
+/// Install the LanceDB metrics recorder as the process-global `metrics` recorder.
+///
+/// Returns `true` if the recorder is installed (now or previously). Returns
+/// `false` if a *different* recorder is already installed — `metrics` allows
+/// only one global recorder per process, so LanceDB cannot coexist with another.
+#[napi]
+pub fn register_lancedb_metrics_recorder() -> bool {
+    lancedb::metrics_otel::register_metrics_recorder()
+}
+
+/// The catalog of described LanceDB metrics. Empty until the recorder is installed.
+#[napi]
+pub fn lancedb_metrics_catalog() -> Vec<MetricDescription> {
+    lancedb::metrics_otel::metrics_catalog()
+        .into_iter()
+        .map(|desc| MetricDescription {
+            name: desc.name,
+            kind: desc.kind.as_str().to_string(),
+            unit: desc.unit,
+            description: desc.description,
+        })
+        .collect()
+}
+
+/// A point-in-time snapshot of every recorded metric. Empty until the recorder
+/// is installed.
+#[napi]
+pub fn snapshot_lancedb_metrics() -> Vec<MetricPoint> {
+    lancedb::metrics_otel::snapshot_metrics()
+        .into_iter()
+        .map(MetricPoint::from)
+        .collect()
+}

nodejs/src/table.rs

@@ -8,8 +8,8 @@ use chrono::{DateTime, Utc};
 use lancedb::ipc::{ipc_file_to_batches, ipc_file_to_schema};
 use lancedb::table::{
     AddDataMode, ColumnAlteration as LanceColumnAlteration, Duration,
-    FieldMetadataUpdate as LanceFieldMetadataUpdate, NewColumnTransform, OptimizeAction,
-    OptimizeOptions, Ref, Table as LanceDbTable,
+    FieldMetadataUpdate as LanceFieldMetadataUpdate, FtsToken as LanceDbFtsToken,
+    NewColumnTransform, OptimizeAction, OptimizeOptions, Ref, Table as LanceDbTable,
 };
 use napi::bindgen_prelude::*;
 use napi::threadsafe_function::{ThreadsafeFunction, ThreadsafeFunctionCallMode};
@@ -411,6 +411,16 @@ impl Table {
             .default_error()
     }
 
+    #[napi(catch_unwind)]
+    pub async fn get_lsm_write_spec(&self) -> napi::Result<Option<LsmWriteSpec>> {
+        let spec = self
+            .inner_ref()?
+            .get_lsm_write_spec()
+            .await
+            .default_error()?;
+        Ok(spec.map(LsmWriteSpec::from))
+    }
+
     #[napi(catch_unwind)]
     pub async fn close_lsm_writers(&self) -> napi::Result<()> {
         self.inner_ref()?.close_lsm_writers().await.default_error()
@@ -564,6 +574,27 @@ impl Table {
             .collect::<Vec<_>>())
     }
 
+    #[napi(catch_unwind)]
+    pub async fn tokenize(
+        &self,
+        query: String,
+        column: Option<String>,
+        index_name: Option<String>,
+    ) -> napi::Result<Vec<FtsToken>> {
+        let table = self.inner_ref()?;
+        let tokens = match (column.as_deref(), index_name.as_deref()) {
+            (Some(_), Some(_)) | (None, None) => {
+                return Err(napi::Error::from_reason(
+                    "Specify exactly one of 'column' or 'indexName'",
+                ));
+            }
+            (Some(column), None) => table.tokenize_with_column(&query, column).await,
+            (None, Some(index_name)) => table.tokenize(&query, index_name).await,
+        }
+        .default_error()?;
+        Ok(tokens.into_iter().map(FtsToken::from).collect())
+    }
+
     #[napi(catch_unwind)]
     pub async fn index_stats(&self, index_name: String) -> napi::Result<Option<IndexStatistics>> {
         let tbl = self.inner_ref()?;
@@ -671,6 +702,24 @@ impl From<lancedb::index::IndexConfig> for IndexConfig {
     }
 }
 
+#[napi(object)]
+/// A token produced by the tokenizer configured on a full-text search index.
+pub struct FtsToken {
+    /// The token text after the index tokenizer has applied its filters.
+    pub text: String,
+    /// The token position used by full-text query matching.
+    pub position: u32,
+}
+
+impl From<LanceDbFtsToken> for FtsToken {
+    fn from(token: LanceDbFtsToken) -> Self {
+        Self {
+            text: token.text,
+            position: token.position,
+        }
+    }
+}
+
 /// Specification selecting Lance's MemWAL LSM-style write path for
 /// `mergeInsert`.
 ///
@@ -728,6 +777,47 @@ impl TryFrom<LsmWriteSpec> for lancedb::table::LsmWriteSpec {
     }
 }
 
+impl From<lancedb::table::LsmWriteSpec> for LsmWriteSpec {
+    fn from(spec: lancedb::table::LsmWriteSpec) -> Self {
+        use lancedb::table::LsmWriteSpec as Native;
+        match spec {
+            Native::Bucket {
+                column,
+                num_buckets,
+                maintained_indexes,
+                writer_config_defaults,
+            } => Self {
+                spec_type: "bucket".to_string(),
+                column: Some(column),
+                num_buckets: Some(num_buckets),
+                maintained_indexes: Some(maintained_indexes),
+                writer_config_defaults: Some(writer_config_defaults),
+            },
+            Native::Identity {
+                column,
+                maintained_indexes,
+                writer_config_defaults,
+            } => Self {
+                spec_type: "identity".to_string(),
+                column: Some(column),
+                num_buckets: None,
+                maintained_indexes: Some(maintained_indexes),
+                writer_config_defaults: Some(writer_config_defaults),
+            },
+            Native::Unsharded {
+                maintained_indexes,
+                writer_config_defaults,
+            } => Self {
+                spec_type: "unsharded".to_string(),
+                column: None,
+                num_buckets: None,
+                maintained_indexes: Some(maintained_indexes),
+                writer_config_defaults: Some(writer_config_defaults),
+            },
+        }
+    }
+}
+
 /// Statistics about a compaction operation.
 #[napi(object)]
 #[derive(Clone, Debug)]

python/.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.34.0-beta.6"
+current_version = "0.35.0-beta.2"
 parse = """(?x)
     (?P<major>0|[1-9]\\d*)\\.
     (?P<minor>0|[1-9]\\d*)\\.

python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-python"
-version = "0.34.0-beta.6"
+version = "0.35.0-beta.2"
 publish = false
 edition.workspace = true
 description = "Python bindings for LanceDB"
@@ -47,6 +47,6 @@ pyo3-build-config = { version = "0.28", features = [
 ] }
 
 [features]
-default = ["remote",  "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/cos", "lancedb/goosefs", "lancedb/metrics-otel"]
 fp16kernels = ["lancedb/fp16kernels"]
 remote = ["lancedb/remote"]

python/pyproject.toml

@@ -47,6 +47,10 @@ repository = "https://github.com/lancedb/lancedb"
 pylance = [
     "pylance>=5.0.0b5",
 ]
+# A library only needs the OpenTelemetry API; the application supplies and
+# configures the SDK (the actual exporter/reader). See
+# https://opentelemetry.io/docs/languages/python/instrumentation/
+otel = ["opentelemetry-api"]
 tests = [
     "aiohttp>=3.9.0",
     "boto3>=1.28.57",
@@ -61,6 +65,7 @@ tests = [
     "pylance>=5.0.0b5",
     "requests>=2.31.0",
     "datafusion>=52,<53",
+    "opentelemetry-sdk>=1.30.0",
 ]
 dev = [
     "ruff>=0.3.0",

python/python/lancedb/__init__.py

@@ -6,19 +6,22 @@ import importlib.metadata
 import os
 from concurrent.futures import ThreadPoolExecutor
 from datetime import timedelta
-from typing import Dict, Optional, Union, Any, List
+from typing import Dict, Optional, Union, Any, List, Iterable
 
 __version__ = importlib.metadata.version("lancedb")
 
 from ._lancedb import connect as lancedb_connect
+from ._lancedb import FtsToken
+from ._lancedb import tokenize as _tokenize
 from .common import URI, sanitize_uri
 from urllib.parse import urlparse
 from .db import AsyncConnection, DBConnection, LanceDBConnection
 from .remote import ClientConfig
 from .remote.db import RemoteDBConnection
 from .expr import Expr, col, lit, func
-from .schema import vector
+from .schema import blob, vector, BlobType
 from .table import AsyncTable, Table
+from .types import BaseTokenizerType
 from ._lancedb import Session
 from .namespace import (
     connect_namespace,
@@ -149,8 +152,14 @@ def connect(
 
     For object storage, use a URI prefix:
 
-    >>> db = lancedb.connect("s3://my-bucket/lancedb",
-    ...                      storage_options={"aws_access_key_id": "***"})
+    >>> db = lancedb.connect(  # doctest: +SKIP
+    ...     "s3://my-bucket/lancedb",
+    ...     storage_options={
+    ...         "aws_access_key_id": "***",
+    ...         "aws_secret_access_key": "***",
+    ...         "aws_region": "us-east-1",
+    ...     },
+    ... )
 
     For tests and temporary data, use an in-memory database:
 
@@ -240,6 +249,40 @@ def connect(
     )
 
 
+def tokenize(
+    query: str,
+    *,
+    base_tokenizer: BaseTokenizerType = "simple",
+    language: str = "English",
+    max_token_length: Optional[int] = 40,
+    lower_case: bool = True,
+    stem: bool = True,
+    remove_stop_words: bool = True,
+    ascii_folding: bool = True,
+    ngram_min_length: int = 3,
+    ngram_max_length: int = 3,
+    prefix_only: bool = False,
+) -> Iterable[FtsToken]:
+    """Tokenize a full-text search query using an explicit tokenizer.
+
+    This does not require a table or FTS index. The tokenizer options match
+    :class:`lancedb.index.FTS`.
+    """
+    return _tokenize(
+        query,
+        base_tokenizer=base_tokenizer,
+        language=language,
+        max_token_length=max_token_length,
+        lower_case=lower_case,
+        stem=stem,
+        remove_stop_words=remove_stop_words,
+        ascii_folding=ascii_folding,
+        ngram_min_length=ngram_min_length,
+        ngram_max_length=ngram_max_length,
+        prefix_only=prefix_only,
+    )
+
+
 WORKER_PROPERTY_PREFIX = "_lancedb_worker_"
 
 
@@ -450,17 +493,21 @@ async def connect_async(
 __all__ = [
     "connect",
     "connect_async",
+    "tokenize",
     "connect_namespace",
     "connect_namespace_async",
     "AsyncConnection",
     "AsyncLanceNamespaceDBConnection",
     "AsyncTable",
+    "FtsToken",
     "col",
     "Expr",
     "func",
     "lit",
     "URI",
     "sanitize_uri",
+    "blob",
+    "BlobType",
     "vector",
     "DBConnection",
     "LanceDBConnection",

python/python/lancedb/_blob.py

@@ -0,0 +1,420 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+"""Blob fetch API and v2 projection helpers."""
+
+from __future__ import annotations
+
+import io
+from collections.abc import Awaitable, Callable, Iterable
+from typing import TYPE_CHECKING, Optional, Union
+
+import pyarrow as pa
+
+from .expr import Expr
+from .schema import blob_v2_column_paths
+from .types import BlobMode, QueryProjection, QueryProjectionSpec
+from .util import get_uri_scheme
+
+if TYPE_CHECKING:
+    from _typeshed import WriteableBuffer
+
+    from .remote.table import RemoteTable
+    from .table import AsyncTable, Table
+
+BLOB_MODE_TO_HANDLING = {
+    "lazy": "blobs_descriptions",
+    "bytes": "all_binary",
+    "descriptions": "blobs_descriptions",
+}
+
+ROW_ID_FIELD_NAME = "_lance_row_id"
+
+FetchBlobsSync = Callable[[str, pa.Table], pa.Array | pa.ChunkedArray]
+FetchBlobsAsync = Callable[[str, pa.Table], Awaitable[pa.Array | pa.ChunkedArray]]
+
+
+class BlobFile(io.RawIOBase):
+    """Seekable lazy handle from :meth:`~lancedb.table.Table.fetch_blob_files`.
+
+    Bytes load on ``read`` or ``read_range``, not when the handle is opened.
+    Use :meth:`aread` from async code.
+    """
+
+    def __init__(self, inner) -> None:
+        self._inner = inner
+
+    async def aread(self) -> bytes:
+        return await self._inner.read()
+
+    def close(self) -> None:
+        self._inner.close()
+
+    @property
+    def closed(self) -> bool:
+        return self._inner.is_closed()
+
+    def readable(self) -> bool:
+        return True
+
+    def seekable(self) -> bool:
+        return True
+
+    def seek(self, offset: int, whence: int = io.SEEK_SET) -> int:
+        if whence == io.SEEK_SET:
+            self._inner.seek(offset)
+        elif whence == io.SEEK_CUR:
+            self._inner.seek(self._inner.tell() + offset)
+        elif whence == io.SEEK_END:
+            self._inner.seek(self._inner.size() + offset)
+        else:
+            raise ValueError(f"invalid whence: {whence}")
+        return self._inner.tell()
+
+    def tell(self) -> int:
+        return self._inner.tell()
+
+    def size(self) -> int:
+        return self._inner.size()
+
+    def readall(self) -> bytes:
+        return self._inner.read_bytes()
+
+    def read(self, size: int = -1) -> bytes:
+        if size == -1:
+            return self._inner.read_bytes()
+        return super().read(size)
+
+    def read_range(self, offset: int, length: int) -> bytes:
+        return self._inner.read_range(offset, length)
+
+    def readinto(self, b: WriteableBuffer) -> int:
+        view = memoryview(b).cast("B")
+        chunk = self._inner.read_up_to(len(view))
+        view[: len(chunk)] = chunk
+        return len(chunk)
+
+    def __repr__(self) -> str:
+        return f"<BlobFile size={self.size()}>"
+
+
+def validate_blob_mode(blob_mode: BlobMode) -> None:
+    if blob_mode not in BLOB_MODE_TO_HANDLING:
+        modes = ", ".join(repr(mode) for mode in BLOB_MODE_TO_HANDLING)
+        raise ValueError(f"blob_mode must be one of {modes}, got {blob_mode!r}")
+
+
+def supports_blob_auto_row_id(table: Table | AsyncTable | RemoteTable) -> bool:
+    """Blob auto row-id applies to native tables, not LanceDB Cloud."""
+    from .remote.table import RemoteTable
+
+    if isinstance(table, RemoteTable):
+        return False
+
+    inner = getattr(table, "_inner", None)
+    if inner is not None:
+        uri = inner.database().uri
+        if isinstance(uri, str) and get_uri_scheme(uri) == "db":
+            return False
+
+    return True
+
+
+def projection_includes_blob_column(
+    projection: QueryProjection,
+    blob_columns: Iterable[str],
+) -> bool:
+    columns = set(blob_columns)
+    if not columns:
+        return False
+    if projection is None:
+        return True
+    for output, source in _iter_projection_pairs(projection):
+        if output in columns or source in columns:
+            return True
+    return False
+
+
+def blob_v2_projection_sources(
+    schema: pa.Schema,
+    projection: QueryProjection,
+) -> dict[str, str]:
+    blob_columns = blob_v2_column_paths(schema)
+    if not blob_columns:
+        return {}
+    columns = set(blob_columns)
+    if projection is None:
+        return {column: column for column in blob_columns}
+    return {
+        output: source
+        for output, source in _iter_projection_pairs(projection)
+        if source in columns
+    }
+
+
+def v2_projection_needs_row_id(
+    schema: pa.Schema,
+    projection: QueryProjection,
+    *,
+    with_row_id: bool,
+) -> bool:
+    if with_row_id:
+        return False
+    return projection_includes_blob_column(projection, blob_v2_column_paths(schema))
+
+
+def blob_auto_row_id_for_scan(
+    table: Table | AsyncTable | RemoteTable,
+    schema: pa.Schema,
+    projection: QueryProjection,
+    *,
+    with_row_id: bool | None,
+) -> bool:
+    if with_row_id is not None:
+        return False
+    if not supports_blob_auto_row_id(table):
+        return False
+    return v2_projection_needs_row_id(schema, projection, with_row_id=False)
+
+
+def finalize_blob_query_table(
+    tbl: pa.Table,
+    *,
+    user_requested_row_id: bool,
+    blob_auto_row_id: bool,
+    blob_paths: Iterable[str] = (),
+) -> pa.Table:
+    if user_requested_row_id or not blob_auto_row_id:
+        return tbl
+    return stash_auto_row_ids(tbl, blob_paths)
+
+
+async def replace_v2_blob_columns_with_bytes(
+    tbl: pa.Table,
+    blob_sources: dict[str, str],
+    fetch_blobs: FetchBlobsAsync,
+) -> pa.Table:
+    for output_name, source_name in blob_sources.items():
+        if output_name not in tbl.column_names:
+            continue
+        blobs = await fetch_blobs(source_name, tbl)
+        tbl = _set_blob_column(tbl, output_name, blobs)
+    return tbl
+
+
+def replace_v2_blob_columns_with_bytes_sync(
+    tbl: pa.Table,
+    blob_sources: dict[str, str],
+    fetch_blobs: FetchBlobsSync,
+) -> pa.Table:
+    for output_name, source_name in blob_sources.items():
+        if output_name not in tbl.column_names:
+            continue
+        blobs = fetch_blobs(source_name, tbl)
+        tbl = _set_blob_column(tbl, output_name, blobs)
+    return tbl
+
+
+def stash_auto_row_ids(tbl: pa.Table, blob_paths: Iterable[str]) -> pa.Table:
+    if "_rowid" not in tbl.column_names:
+        raise ValueError("query result has no '_rowid' column to hide")
+
+    present_paths = [p for p in blob_paths if p.split(".")[0] in tbl.column_names]
+    if not present_paths:
+        raise ValueError("query result has no blob v2 column to carry a row id")
+
+    row_ids = tbl["_rowid"]
+    if isinstance(row_ids, pa.ChunkedArray):
+        row_ids = row_ids.combine_chunks()
+    row_ids = row_ids.cast(pa.uint64())
+
+    for path in present_paths:
+        tbl = _embed_row_id_in_column(tbl, path, row_ids)
+    return tbl.drop_columns(["_rowid"])
+
+
+def read_row_ids_from_hits(hits: pa.Table, blob_column: str) -> list[int]:
+    if "_rowid" in hits.column_names:
+        return hits["_rowid"].to_pylist()
+
+    try:
+        leaf = _leaf_struct_column(hits, blob_column)
+        if ROW_ID_FIELD_NAME in leaf.type.names:
+            return leaf.field(ROW_ID_FIELD_NAME).to_pylist()
+    except KeyError:
+        pass
+
+    # blob_column is the source name; aliased projections use the output name in hits.
+    row_ids = _find_row_id_in_any_column(hits)
+    if row_ids is not None:
+        return row_ids
+
+    raise ValueError(
+        f"query result has no '_rowid' column and no '{ROW_ID_FIELD_NAME}' "
+        f"field on blob column '{blob_column}'. Pass fresh blob query "
+        "results, call .with_row_id(True), or pass a list of row ids."
+    )
+
+
+def _find_row_id_in_any_column(tbl: pa.Table) -> Optional[list[int]]:
+    for name in tbl.column_names:
+        column = tbl.column(name)
+        if isinstance(column, pa.ChunkedArray):
+            column = column.combine_chunks()
+        row_ids = _find_row_id_in_struct(column)
+        if row_ids is not None:
+            return row_ids
+    return None
+
+
+def _find_row_id_in_struct(array: pa.Array) -> Optional[list[int]]:
+    if not pa.types.is_struct(array.type):
+        return None
+    if ROW_ID_FIELD_NAME in array.type.names:
+        return array.field(ROW_ID_FIELD_NAME).to_pylist()
+    for i in range(array.type.num_fields):
+        row_ids = _find_row_id_in_struct(array.field(i))
+        if row_ids is not None:
+            return row_ids
+    return None
+
+
+def _iter_projection_pairs(
+    projection: QueryProjectionSpec,
+) -> Iterable[tuple[str, str]]:
+    if isinstance(projection, dict):
+        for name, expr in projection.items():
+            if isinstance(expr, str):
+                yield name, expr
+            elif isinstance(expr, Expr):
+                yield name, expr.to_sql()
+        return
+    for column in projection:
+        if isinstance(column, str):
+            yield column, column
+        elif isinstance(column, tuple) and len(column) == 2:
+            name, expr = column
+            if isinstance(expr, str):
+                yield name, expr
+            elif isinstance(expr, Expr):
+                yield name, expr.to_sql()
+
+
+def _set_blob_column(tbl: pa.Table, output_name: str, blobs: pa.Array) -> pa.Table:
+    index = tbl.schema.get_field_index(output_name)
+    return tbl.set_column(index, pa.field(output_name, blobs.type), [blobs])
+
+
+def _embed_row_id_in_column(tbl: pa.Table, path: str, row_ids: pa.Array) -> pa.Table:
+    def add_row_id(children: list, child_fields: list) -> None:
+        children.append(row_ids)
+        child_fields.append(pa.field(ROW_ID_FIELD_NAME, pa.uint64(), nullable=False))
+
+    return _transform_struct_column(tbl, path, add_row_id)
+
+
+def strip_auto_row_ids(tbl: pa.Table, blob_paths: Iterable[str]) -> pa.Table:
+    """Remove any `_lance_row_id` field embedded in blob descriptor structs.
+
+    For read-only descriptor views (`blob_mode="descriptions"`) that never
+    fetch bytes, so have no use for the row id.
+    """
+
+    def drop_row_id(children: list, child_fields: list) -> None:
+        for i, field in enumerate(child_fields):
+            if field.name == ROW_ID_FIELD_NAME:
+                del children[i], child_fields[i]
+                return
+
+    for path in blob_paths:
+        if path.split(".")[0] not in tbl.column_names:
+            continue
+        tbl = _transform_struct_column(tbl, path, drop_row_id)
+    return tbl
+
+
+def _transform_struct_column(
+    tbl: pa.Table, path: str, leaf_transform: Callable[[list, list], None]
+) -> pa.Table:
+    top_name, *rest = path.split(".")
+    top_index = tbl.schema.get_field_index(top_name)
+    top_field = tbl.schema.field(top_index)
+    top_array = tbl.column(top_name)
+    if isinstance(top_array, pa.ChunkedArray):
+        top_array = top_array.combine_chunks()
+
+    new_array, new_field = _rebuild_struct(top_array, top_field, rest, leaf_transform)
+    return tbl.set_column(top_index, new_field, new_array)
+
+
+def _rebuild_struct(
+    struct_array: pa.StructArray,
+    struct_field: pa.Field,
+    remaining_path: list[str],
+    leaf_transform: Callable[[list, list], None],
+) -> tuple[pa.StructArray, pa.Field]:
+    null_mask = struct_array.is_null()
+    if not remaining_path:
+        children = [struct_array.field(i) for i in range(struct_array.type.num_fields)]
+        child_fields = list(struct_array.type)
+        leaf_transform(children, child_fields)
+        new_array = pa.StructArray.from_arrays(
+            children, fields=child_fields, mask=null_mask
+        )
+    else:
+        child_name = remaining_path[0]
+        child_index = struct_array.type.get_field_index(child_name)
+        child_array = struct_array.field(child_index)
+        child_field = struct_array.type.field(child_index)
+        new_child_array, new_child_field = _rebuild_struct(
+            child_array, child_field, remaining_path[1:], leaf_transform
+        )
+
+        children = []
+        child_fields = []
+        for i in range(struct_array.type.num_fields):
+            field = struct_array.type.field(i)
+            if field.name == child_name:
+                children.append(new_child_array)
+                child_fields.append(new_child_field)
+            else:
+                children.append(struct_array.field(i))
+                child_fields.append(field)
+        new_array = pa.StructArray.from_arrays(
+            children, fields=child_fields, mask=null_mask
+        )
+
+    new_field = pa.field(
+        struct_field.name,
+        new_array.type,
+        nullable=struct_field.nullable,
+        metadata=struct_field.metadata,
+    )
+    return new_array, new_field
+
+
+def _leaf_struct_column(tbl: pa.Table, path: str) -> pa.StructArray:
+    parts = path.split(".")
+    column = tbl.column(parts[0])
+    if isinstance(column, pa.ChunkedArray):
+        column = column.combine_chunks()
+    for part in parts[1:]:
+        column = column.field(part)
+    return column
+
+
+def _normalize_blob_row_ids(
+    row_ids: Union[list[int], pa.Table], blob_column: str
+) -> list[int]:
+    if isinstance(row_ids, pa.Table):
+        return read_row_ids_from_hits(row_ids, blob_column)
+    if isinstance(row_ids, (pa.Array, pa.ChunkedArray)):
+        raise ValueError(
+            "pass a query table with _rowid, not a column array "
+            "(use fetch_blobs('image', hits), not fetch_blobs('image', hits['image']))"
+        )
+    return list(row_ids)
+
+
+def _wrap_blob_files(handles: Iterable[object]) -> list[Optional[BlobFile]]:
+    return [BlobFile(handle) if handle is not None else None for handle in handles]

python/python/lancedb/_lancedb.pyi

@@ -25,11 +25,45 @@ from lance_namespace import (
     ListTablesResponse,
 )
 from .remote import ClientConfig
+from .types import BaseTokenizerType
 
 IvfHnswPq: type[HnswPq] = HnswPq
 IvfHnswSq: type[HnswSq] = HnswSq
 IvfHnswFlat: type[HnswFlat] = HnswFlat
 
+class MetricPoint:
+    name: str
+    kind: str
+    attributes: Dict[str, str]
+    value: Optional[float]
+    buckets: Optional[List[Tuple[str, int]]]
+    count: Optional[int]
+    sum: Optional[float]
+
+class MetricDescription:
+    name: str
+    kind: str
+    unit: Optional[str]
+    description: str
+
+def register_lancedb_metrics_recorder() -> bool: ...
+def lancedb_metrics_catalog() -> List[MetricDescription]: ...
+def snapshot_lancedb_metrics() -> List[MetricPoint]: ...
+def tokenize(
+    query: str,
+    *,
+    base_tokenizer: BaseTokenizerType = "simple",
+    language: str = "English",
+    max_token_length: Optional[int] = 40,
+    lower_case: bool = True,
+    stem: bool = True,
+    remove_stop_words: bool = True,
+    ascii_folding: bool = True,
+    ngram_min_length: int = 3,
+    ngram_max_length: int = 3,
+    prefix_only: bool = False,
+) -> List["FtsToken"]: ...
+
 class PyExpr:
     """A type-safe DataFusion expression node (Rust-side handle)."""
 
@@ -162,6 +196,17 @@ class Connection(object):
         self,
     ) -> Dict[str, Any]: ...
 
+class BlobFile:
+    async def read(self) -> bytes: ...
+    def read_bytes(self) -> bytes: ...
+    def close(self) -> None: ...
+    def is_closed(self) -> bool: ...
+    def seek(self, position: int) -> None: ...
+    def tell(self) -> int: ...
+    def size(self) -> int: ...
+    def read_range(self, offset: int, length: int) -> bytes: ...
+    def read_up_to(self, length: int) -> bytes: ...
+
 class Table:
     def name(self) -> str: ...
     def __repr__(self) -> str: ...
@@ -208,6 +253,13 @@ class Table:
     async def prewarm_index(self, index_name: str) -> None: ...
     async def prewarm_data(self, columns: Optional[List[str]] = None) -> None: ...
     async def list_indices(self) -> list[IndexConfig]: ...
+    async def tokenize(
+        self,
+        query: str,
+        *,
+        column: Optional[str] = None,
+        index_name: Optional[str] = None,
+    ) -> list[FtsToken]: ...
     async def delete(self, filter: Union[str, PyExpr]) -> DeleteResult: ...
     async def add_columns(self, columns: list[tuple[str, str]]) -> AddColumnsResult: ...
     async def add_columns_with_schema(self, schema: pa.Schema) -> AddColumnsResult: ...
@@ -229,6 +281,7 @@ class Table:
     async def set_unenforced_primary_key(self, columns: List[str]) -> None: ...
     async def set_lsm_write_spec(self, spec: LsmWriteSpec) -> None: ...
     async def unset_lsm_write_spec(self) -> None: ...
+    async def get_lsm_write_spec(self) -> Optional[LsmWriteSpec]: ...
     async def close_lsm_writers(self) -> None: ...
     @property
     def tags(self) -> Tags: ...
@@ -238,6 +291,13 @@ class Table:
     def query(self) -> Query: ...
     def take_offsets(self, offsets: list[int]) -> TakeQuery: ...
     def take_row_ids(self, row_ids: list[int]) -> TakeQuery: ...
+    async def blob_columns(self) -> list[str]: ...
+    async def fetch_blobs(
+        self, column: str, row_ids: list[int]
+    ) -> pa.LargeBinaryArray: ...
+    async def fetch_blob_files(
+        self, column: str, row_ids: list[int]
+    ) -> list[Optional[BlobFile]]: ...
     def vector_search(self) -> VectorQuery: ...
 
 class Tags:
@@ -473,6 +533,10 @@ class MergeResult:
     num_attempts: int
     num_rows: int
 
+class FtsToken:
+    text: str
+    position: int
+
 class LsmWriteSpec:
     """Specification selecting Lance's MemWAL LSM-style write path for
     `merge_insert`."""

python/python/lancedb/embeddings/gemini_text.py

@@ -4,7 +4,7 @@
 
 import os
 from functools import cached_property
-from typing import List, Union
+from typing import List, Optional, Union
 
 import numpy as np
 
@@ -15,6 +15,8 @@ from .base import TextEmbeddingFunction
 from .registry import register
 from .utils import TEXT, api_key_not_found_help
 
+EMBEDDING_BATCH_SIZE = 100
+
 
 @register("gemini-text")
 class GeminiText(TextEmbeddingFunction):
@@ -81,6 +83,7 @@ class GeminiText(TextEmbeddingFunction):
     """
 
     name: str = "gemini-embedding-001"
+    dim: Optional[int] = None
     query_task_type: str = "retrieval_query"
     source_task_type: str = "retrieval_document"
 
@@ -93,6 +96,8 @@ class GeminiText(TextEmbeddingFunction):
         model_config["ignored_types"] = (cached_property,)
 
     def ndims(self):
+        if self.dim:
+            return self.dim
         # TODO: fix hardcoding
         return 768
 
@@ -133,22 +138,22 @@ class GeminiText(TextEmbeddingFunction):
                 contents.append({"parts": [{"text": text}]})
 
         # Build config
-        config_kwargs = {}
+        config_kwargs = {"output_dimensionality": self.ndims()}
         if task_type:
             config_kwargs["task_type"] = task_type.upper()  # API expects uppercase
 
-        # Call embed_content for each content
+        config = types.EmbedContentConfig(**config_kwargs) if config_kwargs else None
+
+        # Call embed_content in groups of at most EMBEDDING_BATCH_SIZE docs at a time
         embeddings = []
-        for content in contents:
-            config = (
-                types.EmbedContentConfig(**config_kwargs) if config_kwargs else None
-            )
+        for i in range(0, len(contents), EMBEDDING_BATCH_SIZE):
+            chunk = contents[i : i + EMBEDDING_BATCH_SIZE]
             response = self.client.models.embed_content(
                 model=self.name,
-                contents=content,
+                contents=chunk,
                 config=config,
             )
-            embeddings.append(response.embeddings[0].values)
+            embeddings.extend([np.array(e.values) for e in response.embeddings])
 
         return embeddings
 
@@ -160,5 +165,13 @@ class GeminiText(TextEmbeddingFunction):
             api_key_not_found_help("google")
 
         from google import genai as genai_module
+        from lancedb import __version__
 
-        return genai_module.Client(api_key=os.environ.get("GOOGLE_API_KEY"))
+        return genai_module.Client(
+            api_key=os.environ.get("GOOGLE_API_KEY"),
+            http_options={
+                "headers": {
+                    "x-goog-api-client": f"lancedb/{__version__}",
+                }
+            },
+        )

python/python/lancedb/embeddings/ollama.py

@@ -2,7 +2,7 @@
 # SPDX-FileCopyrightText: Copyright The LanceDB Authors
 
 from functools import cached_property
-from typing import TYPE_CHECKING, List, Optional, Sequence, Union
+from typing import TYPE_CHECKING, Any, List, Optional, Sequence, Union
 
 import numpy as np
 
@@ -56,6 +56,16 @@ class OllamaEmbeddings(TextEmbeddingFunction):
         embeddings = self._compute_embedding(texts)
         return list(embeddings)
 
+    def __getstate__(self) -> dict[str, Any]:
+        state = super().__getstate__()
+        state["__dict__"] = {
+            k: v for k, v in state["__dict__"].items() if k != "_ollama_client"
+        }
+        return state
+
+    def __setstate__(self, state: dict[str, Any]) -> None:
+        super().__setstate__(state)
+
     @cached_property
     def _ollama_client(self) -> "ollama.Client":
         ollama = attempt_import_or_raise("ollama")

python/python/lancedb/index.py

@@ -127,6 +127,8 @@ class FTS:
         - "whitespace": Split text by whitespace, but not punctuation.
         - "raw": No tokenization. The entire text is treated as a single token.
         - "ngram": N-gram tokenizer for substring-style matching.
+        - "icu": ICU dictionary-based word segmentation.
+        - "icu/split": ICU segmentation with simple-style delimiter splitting.
         - "jieba/*": Jieba tokenizer loaded from Lance's language model home.
         - "lindera/*": Lindera tokenizer loaded from Lance's language model home.
     language : str, default "English"

python/python/lancedb/otel.py

@@ -0,0 +1,170 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+"""Bridge LanceDB's internal metrics into OpenTelemetry.
+
+LanceDB (through Lance core) publishes metrics (currently object store request
+counts, bytes, latency, errors, and throttles) through the Rust ``metrics``
+facade. This module installs a process-global recorder that aggregates them and
+registers OpenTelemetry observable instruments that report the aggregated values
+into the user's ``MeterProvider``.
+
+The bridge is generic: every metric LanceDB describes is surfaced automatically,
+with no per-metric Python code. Histograms have no asynchronous OpenTelemetry
+instrument, so each is exported Prometheus-style as cumulative ``le`` buckets
+plus ``_count`` and ``_sum`` observable counters.
+"""
+
+from __future__ import annotations
+
+import warnings
+from typing import TYPE_CHECKING, Optional
+
+from ._lancedb import (
+    lancedb_metrics_catalog,
+    register_lancedb_metrics_recorder,
+    snapshot_lancedb_metrics,
+)
+
+if TYPE_CHECKING:
+    from opentelemetry.metrics import MeterProvider
+
+_INSTRUMENTED = False
+
+
+def instrument_lancedb_metrics(
+    meter_provider: Optional["MeterProvider"] = None,
+) -> bool:
+    """Register LanceDB metrics as OpenTelemetry observable instruments.
+
+    Installs a process-global metrics recorder and creates one observable
+    instrument per LanceDB metric on the given (or global) ``MeterProvider``. The
+    user's configured ``MetricReader`` then collects them on its own schedule.
+
+    Counters and gauges map directly to observable counters/gauges. Each
+    histogram is exported as cumulative ``le`` bucket counts (``<name>_bucket``,
+    with an ``le`` attribute) plus ``<name>_count`` and ``<name>_sum``.
+
+    Parameters
+    ----------
+    meter_provider : opentelemetry.metrics.MeterProvider, optional
+        The provider to register instruments on. Defaults to the global provider
+        from ``opentelemetry.metrics.get_meter_provider()``.
+
+    Returns
+    -------
+    bool
+        ``True`` if the recorder is installed and instruments are registered.
+        ``False`` if a different ``metrics`` recorder is already installed in
+        this process (``metrics`` permits only one global recorder), in which
+        case a warning is emitted and no instruments are created.
+
+    Notes
+    -----
+    Requires the OpenTelemetry API (``pip install lancedb[otel]``) and, to
+    actually export, an OpenTelemetry SDK (``pip install opentelemetry-sdk``)
+    configured by the application. Calling this more than once is safe;
+    instruments are created only on the first successful call.
+    """
+    global _INSTRUMENTED
+
+    try:
+        from opentelemetry.metrics import Observation, get_meter_provider
+    except ImportError as exc:
+        raise ImportError(
+            "instrument_lancedb_metrics requires the OpenTelemetry API/SDK. "
+            "Install it with `pip install lancedb[otel]` or "
+            "`pip install opentelemetry-sdk`."
+        ) from exc
+
+    if not register_lancedb_metrics_recorder():
+        warnings.warn(
+            "Could not install the LanceDB metrics recorder: another `metrics` "
+            "recorder is already installed in this process. LanceDB metrics will "
+            "not be exported via OpenTelemetry.",
+            stacklevel=2,
+        )
+        return False
+
+    if _INSTRUMENTED:
+        return True
+
+    provider = meter_provider or get_meter_provider()
+    meter = provider.get_meter("lancedb")
+
+    def scalar_callback(metric_name: str):
+        def callback(_options):
+            return [
+                Observation(point.value, point.attributes)
+                for point in snapshot_lancedb_metrics()
+                if point.name == metric_name and point.value is not None
+            ]
+
+        return callback
+
+    def bucket_callback(metric_name: str):
+        def callback(_options):
+            observations = []
+            for point in snapshot_lancedb_metrics():
+                if point.name != metric_name or point.buckets is None:
+                    continue
+                for le, cumulative in point.buckets:
+                    attributes = dict(point.attributes)
+                    attributes["le"] = le
+                    observations.append(Observation(cumulative, attributes))
+            return observations
+
+        return callback
+
+    def field_callback(metric_name: str, field: str):
+        def callback(_options):
+            observations = []
+            for point in snapshot_lancedb_metrics():
+                if point.name != metric_name:
+                    continue
+                value = getattr(point, field)
+                if value is not None:
+                    observations.append(Observation(value, point.attributes))
+            return observations
+
+        return callback
+
+    for desc in lancedb_metrics_catalog():
+        unit = desc.unit or ""
+        if desc.kind == "counter":
+            meter.create_observable_counter(
+                desc.name,
+                callbacks=[scalar_callback(desc.name)],
+                unit=unit,
+                description=desc.description,
+            )
+        elif desc.kind == "gauge":
+            meter.create_observable_gauge(
+                desc.name,
+                callbacks=[scalar_callback(desc.name)],
+                unit=unit,
+                description=desc.description,
+            )
+        elif desc.kind == "histogram":
+            # `_bucket` and `_count` observe cumulative sample counts, not the
+            # histogram's measured quantity, so they are unitless; only `_sum`
+            # carries the histogram's unit.
+            meter.create_observable_counter(
+                f"{desc.name}_bucket",
+                callbacks=[bucket_callback(desc.name)],
+                description=f"{desc.description} (cumulative buckets)",
+            )
+            meter.create_observable_counter(
+                f"{desc.name}_count",
+                callbacks=[field_callback(desc.name, "count")],
+                description=f"{desc.description} (count)",
+            )
+            meter.create_observable_counter(
+                f"{desc.name}_sum",
+                callbacks=[field_callback(desc.name, "sum")],
+                unit=unit,
+                description=f"{desc.description} (sum)",
+            )
+
+    _INSTRUMENTED = True
+    return True

python/python/lancedb/permutation.py

@@ -11,7 +11,7 @@ import pyarrow as pa
 from ._lancedb import async_permutation_builder, PermutationReader
 from .table import LanceTable, Table
 from .background_loop import LOOP
-from .util import batch_to_tensor, batch_to_tensor_rows
+from .util import batch_to_tensor, batch_to_tensor_dict, batch_to_tensor_rows
 from typing import Any, Callable, Iterator, Literal, Optional, TYPE_CHECKING, Union
 
 if TYPE_CHECKING:
@@ -946,6 +946,7 @@ class Permutation:
             "pandas",
             "arrow",
             "torch",
+            "torch_row",
             "torch_col",
             "polars",
         ],
@@ -961,15 +962,19 @@ class Permutation:
         - "python_col" - the batch will be a dict of lists (one entry per column)
         - "pandas" - the batch will be a pandas DataFrame
         - "arrow" - the batch will be a pyarrow RecordBatch
-        - "torch" - the batch will be a list of tensors, one per row
+        - "torch" - the batch will be a list of per-row dicts mapping column
+          name to a 0-D torch tensor. Works with the default
+          ``torch.utils.data.DataLoader`` collate, which stacks the per-row
+          dicts back into a dict of batched tensors.
+        - "torch_row" - the batch will be a list of tensors, one per row
         - "torch_col" - the batch will be a 2D torch tensor (first dim indexes columns)
         - "polars" - the batch will be a polars DataFrame
 
         Conversion may or may not involve a data copy.  Lance uses Arrow internally
         and so it is able to zero-copy to the arrow and polars formats.
 
-        Conversion to torch_col will be zero-copy but will only support a subset of data
-        types (numeric types).
+        Conversion to torch and torch_col will be zero-copy but will only support a
+        subset of data types (numeric types).
 
         Conversion to numpy and/or pandas will typically be zero-copy for numeric
         types.  Conversion of strings, lists, and structs will require creating python
@@ -990,6 +995,8 @@ class Permutation:
         elif format == "arrow":
             return self.with_transform(Transforms.arrow2arrow)
         elif format == "torch":
+            return self.with_transform(batch_to_tensor_dict)
+        elif format == "torch_row":
             return self.with_transform(batch_to_tensor_rows)
         elif format == "torch_col":
             return self.with_transform(batch_to_tensor)

python/python/lancedb/query.py

@@ -15,10 +15,12 @@ from typing import (
     List,
     Literal,
     Optional,
+    Protocol,
     Tuple,
     Type,
     TypeVar,
     Union,
+    runtime_checkable,
 )
 
 import deprecation
@@ -39,15 +41,21 @@ from .expr import Expr
 from .rerankers.base import Reranker
 from .rerankers.rrf import RRFReranker
 from .rerankers.util import check_reranker_result
+from .schema import is_blob_like_field, schema_has_blob_field
 from .util import flatten_columns
-
-BlobMode = Literal["lazy", "bytes", "descriptions"]
-
-_BLOB_MODE_TO_HANDLING = {
-    "lazy": "blobs_descriptions",
-    "bytes": "all_binary",
-    "descriptions": "blobs_descriptions",
-}
+from ._blob import (
+    BLOB_MODE_TO_HANDLING,
+    FetchBlobsAsync,
+    FetchBlobsSync,
+    blob_auto_row_id_for_scan,
+    blob_v2_projection_sources,
+    finalize_blob_query_table,
+    replace_v2_blob_columns_with_bytes,
+    replace_v2_blob_columns_with_bytes_sync,
+    supports_blob_auto_row_id,
+    validate_blob_mode,
+)
+from .types import BlobMode, QueryProjection
 
 if TYPE_CHECKING:
     import sys
@@ -73,25 +81,22 @@ if TYPE_CHECKING:
 T = TypeVar("T", bound="LanceModel")
 
 
-def _validate_blob_mode(blob_mode: BlobMode) -> None:
-    if blob_mode not in _BLOB_MODE_TO_HANDLING:
-        modes = ", ".join(repr(mode) for mode in _BLOB_MODE_TO_HANDLING)
-        raise ValueError(f"blob_mode must be one of {modes}, got {blob_mode!r}")
+@runtime_checkable
+class _LanceScanner(Protocol):
+    projected_schema: pa.Schema | None
+    schema: pa.Schema | None
 
+    def to_pandas(self, blob_mode: BlobMode | None = ..., **kwargs) -> pd.DataFrame: ...
 
-def _field_is_blob(field: pa.Field) -> bool:
-    metadata = field.metadata or {}
-    return metadata.get(b"lance-encoding:blob") == b"true" or (
-        metadata.get("lance-encoding:blob") == "true"
-    )
+    def to_pyarrow(self): ...
 
+    def to_table(self) -> pa.Table: ...
 
-def _schema_has_blob_field(schema: pa.Schema) -> bool:
-    return any(_field_is_blob(field) for field in schema)
+    def to_reader(self): ...
 
 
 def _blob_mode_requires_native_pandas(blob_mode: BlobMode, schema: pa.Schema) -> bool:
-    return blob_mode in _BLOB_MODE_TO_HANDLING and _schema_has_blob_field(schema)
+    return blob_mode in BLOB_MODE_TO_HANDLING and schema_has_blob_field(schema)
 
 
 def _unsupported_blob_pandas_error(reason: str) -> RuntimeError:
@@ -140,13 +145,7 @@ def _combine_where(
     return f"({existing_sql}) AND ({new_sql})"
 
 
-def _projection_to_scanner_kwargs(
-    columns: Optional[
-        Union[
-            List[str], List[Tuple[str, Union[str, Expr]]], Dict[str, Union[str, Expr]]
-        ]
-    ],
-) -> Dict[str, Any]:
+def _projection_to_scanner_kwargs(columns: QueryProjection) -> Dict[str, Any]:
     if columns is None:
         return {}
     if isinstance(columns, list):
@@ -171,7 +170,11 @@ def _projection_to_scanner_kwargs(
 
 
 def _scanner_kwargs_for_query(
-    query: Query, blob_mode: BlobMode, dataset: Optional[Any] = None
+    query: Query,
+    blob_mode: BlobMode,
+    dataset: Optional[Any] = None,
+    *,
+    with_row_id: Optional[bool] = None,
 ) -> Dict[str, Any]:
     fragments = _scanner_fragments_for_query(query, dataset)
     kwargs = {
@@ -179,10 +182,10 @@ def _scanner_kwargs_for_query(
         "filter": _filter_to_sql(query.filter),
         "limit": query.limit,
         "offset": query.offset,
-        "with_row_id": query.with_row_id,
+        "with_row_id": with_row_id if with_row_id is not None else query.with_row_id,
         "with_row_address": query.with_row_address,
         "fast_search": query.fast_search,
-        "blob_handling": _BLOB_MODE_TO_HANDLING[blob_mode],
+        "blob_handling": BLOB_MODE_TO_HANDLING[blob_mode],
         "fragments": fragments,
     }
     return {key: value for key, value in kwargs.items() if value is not None}
@@ -215,11 +218,11 @@ def _scanner_fragments_for_query(query: Query, dataset: Optional[Any]) -> Option
 def _ensure_lazy_blob_frame(
     df: "pd.DataFrame", schema: pa.Schema, blob_mode: BlobMode
 ) -> "pd.DataFrame":
-    if blob_mode != "lazy" or not _schema_has_blob_field(schema) or len(df) == 0:
+    if blob_mode != "lazy" or not schema_has_blob_field(schema) or len(df) == 0:
         return df
 
     for field in schema:
-        if not _field_is_blob(field) or field.name not in df.columns:
+        if not is_blob_like_field(field) or field.name not in df.columns:
             continue
         value = df[field.name].iloc[0]
         if value is not None and not hasattr(value, "readall"):
@@ -229,7 +232,7 @@ def _ensure_lazy_blob_frame(
     return df
 
 
-def _scanner_to_table(scanner: Any) -> pa.Table:
+def _scanner_to_table(scanner: _LanceScanner) -> pa.Table:
     if hasattr(scanner, "to_pyarrow"):
         reader = scanner.to_pyarrow()
         return reader.read_all()
@@ -239,7 +242,9 @@ def _scanner_to_table(scanner: Any) -> pa.Table:
     return reader.read_all()
 
 
-def _scanner_to_pandas(scanner: Any, blob_mode: BlobMode, **kwargs) -> "pd.DataFrame":
+def _scanner_to_pandas(
+    scanner: _LanceScanner, blob_mode: BlobMode, **kwargs
+) -> pd.DataFrame:
     schema = getattr(scanner, "projected_schema", None)
     if schema is None:
         schema = getattr(scanner, "schema", None)
@@ -260,13 +265,71 @@ def _scanner_to_pandas(scanner: Any, blob_mode: BlobMode, **kwargs) -> "pd.DataF
         return df
 
     tbl = _scanner_to_table(scanner)
-    if blob_mode == "lazy" and _schema_has_blob_field(tbl.schema):
+    if blob_mode == "lazy" and schema_has_blob_field(tbl.schema):
         raise _unsupported_blob_pandas_error(
             "the Lance scanner does not expose to_pandas"
         )
     return tbl.to_pandas(**kwargs)
 
 
+def _finish_plain_scan_pandas(
+    scanner: _LanceScanner,
+    *,
+    blob_mode: BlobMode,
+    blob_sources: dict[str, str],
+    fetch_blobs: FetchBlobsSync,
+    strip_auto_row_id: bool,
+    flatten: Optional[Union[int, bool]],
+    **kwargs,
+) -> pd.DataFrame:
+    if blob_sources:
+        tbl = _scanner_to_table(scanner)
+        tbl = replace_v2_blob_columns_with_bytes_sync(tbl, blob_sources, fetch_blobs)
+        if strip_auto_row_id and "_rowid" in tbl.column_names:
+            tbl = tbl.drop_columns(["_rowid"])
+        if flatten is not None:
+            tbl = flatten_columns(tbl, flatten)
+        return tbl.to_pandas(**kwargs)
+    if flatten is not None:
+        tbl = flatten_columns(_scanner_to_table(scanner), flatten)
+        if strip_auto_row_id and "_rowid" in tbl.column_names:
+            tbl = tbl.drop_columns(["_rowid"])
+        return tbl.to_pandas(**kwargs)
+    df = _scanner_to_pandas(scanner, blob_mode, **kwargs)
+    if strip_auto_row_id and "_rowid" in df.columns:
+        return df.drop(columns=["_rowid"])
+    return df
+
+
+async def _finish_plain_scan_pandas_async(
+    scanner: _LanceScanner,
+    *,
+    blob_mode: BlobMode,
+    blob_sources: dict[str, str],
+    fetch_blobs: FetchBlobsAsync,
+    strip_auto_row_id: bool,
+    flatten: Optional[Union[int, bool]],
+    **kwargs,
+) -> pd.DataFrame:
+    if blob_sources:
+        tbl = _scanner_to_table(scanner)
+        tbl = await replace_v2_blob_columns_with_bytes(tbl, blob_sources, fetch_blobs)
+        if strip_auto_row_id and "_rowid" in tbl.column_names:
+            tbl = tbl.drop_columns(["_rowid"])
+        if flatten is not None:
+            tbl = flatten_columns(tbl, flatten)
+        return tbl.to_pandas(**kwargs)
+    if flatten is not None:
+        tbl = flatten_columns(_scanner_to_table(scanner), flatten)
+        if strip_auto_row_id and "_rowid" in tbl.column_names:
+            tbl = tbl.drop_columns(["_rowid"])
+        return tbl.to_pandas(**kwargs)
+    df = _scanner_to_pandas(scanner, blob_mode, **kwargs)
+    if strip_auto_row_id and "_rowid" in df.columns:
+        return df.drop(columns=["_rowid"])
+    return df
+
+
 # Pydantic validation function for vector queries
 def ensure_vector_query(
     val: Any,
@@ -674,7 +737,7 @@ class Query(pydantic.BaseModel):
     distance_type: Optional[str] = None
 
     # which columns to return in the results (dict values may be str or Expr)
-    columns: Optional[Union[List[str], Dict[str, Union[str, Expr]]]] = None
+    columns: QueryProjection = None
 
     # minimum number of IVF partitions to search
     #
@@ -958,7 +1021,7 @@ class LanceQueryBuilder(ABC):
             Forwarded to pyarrow.Table.to_pandas after query execution and
             optional flattening.
         """
-        _validate_blob_mode(blob_mode)
+        validate_blob_mode(blob_mode)
         output_schema = getattr(self, "output_schema", None)
         if output_schema is not None:
             schema = output_schema()
@@ -1017,6 +1080,11 @@ class LanceQueryBuilder(ABC):
         Execute the query and return the results as a pyarrow
         [RecordBatchReader](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html)
 
+        For v2 blob projections, ``to_batches`` keeps the auto ``_rowid``
+        column visible so batch consumers can call ``fetch_blobs``. Use
+        ``to_arrow``, ``to_list``, or ``to_pandas`` if you want LanceDB to hide
+        auto row ids in the final collected result.
+
         Parameters
         ----------
         batch_size: int
@@ -1195,6 +1263,42 @@ class LanceQueryBuilder(ABC):
         self._with_row_id = with_row_id
         return self
 
+    def _user_requested_row_id(self) -> bool:
+        return self._with_row_id is True
+
+    def _blob_auto_row_id_enabled(self) -> bool:
+        if not supports_blob_auto_row_id(self._table):
+            return False
+        return blob_auto_row_id_for_scan(
+            self._table,
+            self._table.schema,
+            self._columns,
+            with_row_id=self._with_row_id,
+        )
+
+    def _scan_needs_row_id(self) -> bool:
+        return self._user_requested_row_id() or self._blob_auto_row_id_enabled()
+
+    def _query_for_scan(self) -> Query:
+        query = self.to_query_object()
+        if self._scan_needs_row_id():
+            query.with_row_id = True
+        return query
+
+    def _finalize_blob_query_table(self, tbl: pa.Table) -> pa.Table:
+        blob_auto_row_id = self._blob_auto_row_id_enabled()
+        blob_paths = (
+            blob_v2_projection_sources(self._table.schema, self._columns).keys()
+            if blob_auto_row_id
+            else ()
+        )
+        return finalize_blob_query_table(
+            tbl,
+            user_requested_row_id=self._user_requested_row_id(),
+            blob_auto_row_id=blob_auto_row_id,
+            blob_paths=blob_paths,
+        )
+
     def with_row_address(self, with_row_address: bool = True) -> Self:
         """Set whether to return row addresses.
 
@@ -1371,13 +1475,29 @@ class LanceQueryBuilder(ABC):
             return None
 
         dataset = self._table.to_lance()
-        scanner = dataset.scanner(
-            **_scanner_kwargs_for_query(query, blob_mode, dataset)
+        blob_auto_row_id = self._blob_auto_row_id_enabled()
+        blob_sources = (
+            blob_v2_projection_sources(self._table.schema, query.columns)
+            if blob_mode == "bytes"
+            else {}
+        )
+        scanner = dataset.scanner(
+            **_scanner_kwargs_for_query(
+                query,
+                "descriptions" if blob_sources else blob_mode,
+                dataset,
+                with_row_id=query.with_row_id or blob_auto_row_id or bool(blob_sources),
+            )
+        )
+        return _finish_plain_scan_pandas(
+            scanner,
+            blob_mode=blob_mode,
+            blob_sources=blob_sources,
+            fetch_blobs=self._table.fetch_blobs,
+            strip_auto_row_id=blob_auto_row_id,
+            flatten=flatten,
+            **kwargs,
         )
-        if flatten is not None:
-            tbl = flatten_columns(_scanner_to_table(scanner), flatten)
-            return tbl.to_pandas(**kwargs)
-        return _scanner_to_pandas(scanner, blob_mode, **kwargs)
 
     @abstractmethod
     def to_query_object(self) -> Query:
@@ -1625,7 +1745,9 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
             The maximum time to wait for the query to complete.
             If None, wait indefinitely.
         """
-        return self.to_batches(timeout=timeout).read_all()
+        return self._finalize_blob_query_table(
+            self.to_batches(timeout=timeout).read_all()
+        )
 
     def to_query_object(self) -> Query:
         """
@@ -1685,7 +1807,7 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
         vector = self._query if isinstance(self._query, list) else self._query.tolist()
         if isinstance(vector[0], np.ndarray):
             vector = [v.tolist() for v in vector]
-        query = self.to_query_object()
+        query = self._query_for_scan()
         result_set = self._table._execute_query(
             query, batch_size=batch_size, timeout=timeout
         )
@@ -1829,8 +1951,7 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
         Parameters
         ----------
         phrase_query: bool, default True
-            If True, then the query will be wrapped in quotes and
-            double quotes replaced by single quotes.
+            If True, then an unquoted string query will be wrapped in quotes.
 
         Returns
         -------
@@ -1840,6 +1961,21 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
         self._phrase_query = phrase_query
         return self
 
+    def _query_with_phrase_semantics(self) -> str | FullTextQuery:
+        query = self._query
+        if not self._phrase_query:
+            return query
+        if isinstance(query, str):
+            if not query.startswith('"') or not query.endswith('"'):
+                return f'"{query}"'
+            return query
+        if isinstance(query, PhraseQuery):
+            return query
+        raise TypeError(
+            "phrase_query() requires a string or PhraseQuery, "
+            f"got {type(query).__name__}"
+        )
+
     def fast_search(self) -> LanceFtsQueryBuilder:
         """
         Skip a flat search of unindexed data. This will improve
@@ -1864,7 +2000,7 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
             fragments=self._fragments,
             fragment_ids=self._fragment_ids,
             full_text_query=FullTextSearchQuery(
-                query=self._query, columns=self._fts_columns
+                query=self._query_with_phrase_semantics(), columns=self._fts_columns
             ),
             offset=self._offset,
             fast_search=self._fast_search,
@@ -1882,22 +2018,13 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
     def to_arrow(self, *, timeout: Optional[timedelta] = None) -> pa.Table:
         self._table._ensure_no_legacy_fts_index()
 
-        query = self._query
-        if self._phrase_query:
-            if isinstance(query, str):
-                if not query.startswith('"') or not query.endswith('"'):
-                    self._query = f'"{query}"'
-            elif isinstance(query, FullTextQuery) and not isinstance(
-                query, PhraseQuery
-            ):
-                raise TypeError("Please use PhraseQuery for phrase queries.")
-        query = self.to_query_object()
+        query = self._query_for_scan()
         results = self._table._execute_query(query, timeout=timeout)
         results = results.read_all()
         if self._reranker is not None:
             results = self._reranker.rerank_fts(self._query, results)
             check_reranker_result(results)
-        return results
+        return self._finalize_blob_query_table(results)
 
     def to_batches(
         self, /, batch_size: Optional[int] = None, timeout: Optional[timedelta] = None
@@ -1925,7 +2052,9 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
 
 class LanceEmptyQueryBuilder(LanceQueryBuilder):
     def to_arrow(self, *, timeout: Optional[timedelta] = None) -> pa.Table:
-        return self.to_batches(timeout=timeout).read_all()
+        return self._finalize_blob_query_table(
+            self.to_batches(timeout=timeout).read_all()
+        )
 
     def to_query_object(self) -> Query:
         return Query(
@@ -1947,7 +2076,7 @@ class LanceEmptyQueryBuilder(LanceQueryBuilder):
     def to_batches(
         self, /, batch_size: Optional[int] = None, timeout: Optional[timedelta] = None
     ) -> pa.RecordBatchReader:
-        query = self.to_query_object()
+        query = self._query_for_scan()
         return self._table._execute_query(query, batch_size=batch_size, timeout=timeout)
 
     def rerank(self, reranker: Reranker) -> LanceEmptyQueryBuilder:
@@ -2019,14 +2148,13 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
 
         return vector_query, text_query
 
-    def phrase_query(self, phrase_query: bool = None) -> LanceHybridQueryBuilder:
+    def phrase_query(self, phrase_query: bool = True) -> LanceHybridQueryBuilder:
         """Set whether to use phrase query.
 
         Parameters
         ----------
         phrase_query: bool, default True
-            If True, then the query will be wrapped in quotes and
-            double quotes replaced by single quotes.
+            If True, then an unquoted string query will be wrapped in quotes.
 
         Returns
         -------
@@ -2051,15 +2179,25 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
             fts_results = fts_future.result()
             vector_results = vector_future.result()
 
-        return self._combine_hybrid_results(
+        results = self._combine_hybrid_results(
             fts_results=fts_results,
             vector_results=vector_results,
             norm=self._norm,
             fts_query=self._fts_query._query,
             reranker=self._reranker,
             limit=self._limit,
-            with_row_ids=self._with_row_id,
+            with_row_ids=True,
         )
+        return self._finish_hybrid_results(results)
+
+    def _finish_hybrid_results(self, results: pa.Table) -> pa.Table:
+        if self._user_requested_row_id():
+            return results
+        if self._blob_auto_row_id_enabled():
+            return self._finalize_blob_query_table(results)
+        if "_rowid" in results.column_names:
+            return results.drop(["_rowid"])
+        return results
 
     @staticmethod
     def _combine_hybrid_results(
@@ -2500,7 +2638,7 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
             self._vector_query.ef(self._ef)
         if self._bypass_vector_index:
             self._vector_query.bypass_vector_index()
-        if self._lower_bound or self._upper_bound:
+        if self._lower_bound is not None or self._upper_bound is not None:
             self._vector_query.distance_range(
                 lower_bound=self._lower_bound, upper_bound=self._upper_bound
             )
@@ -2530,6 +2668,9 @@ class AsyncQueryBase(object):
         self._with_row_address = None
         self._fragments = None
         self._fragment_ids = None
+        self._with_row_id = None
+        self._blob_auto_row_id = False
+        self._blob_paths: tuple[str, ...] = ()
 
     def to_query_object(self) -> Query:
         """
@@ -2539,11 +2680,46 @@ class AsyncQueryBase(object):
         python and more easily serializable.
         """
         query = Query.from_inner(self._inner.to_query_request())
+        query.with_row_id = self._user_requested_row_id()
         query.with_row_address = self._with_row_address
         query.fragments = self._fragments
         query.fragment_ids = self._fragment_ids
         return query
 
+    def _user_requested_row_id(self) -> bool:
+        return self._with_row_id is True
+
+    def _blob_auto_row_id_enabled(self) -> bool:
+        return self._blob_auto_row_id
+
+    def _finalize_blob_query_table(self, tbl: pa.Table) -> pa.Table:
+        return finalize_blob_query_table(
+            tbl,
+            user_requested_row_id=self._user_requested_row_id(),
+            blob_auto_row_id=self._blob_auto_row_id_enabled(),
+            blob_paths=self._blob_paths,
+        )
+
+    async def _maybe_add_blob_row_id(self) -> None:
+        if self._table is None or not supports_blob_auto_row_id(self._table):
+            self._blob_auto_row_id = False
+            self._blob_paths = ()
+            return
+
+        req = self._inner.to_query_request()
+        schema = await self._table.schema()
+        self._blob_auto_row_id = blob_auto_row_id_for_scan(
+            self._table,
+            schema,
+            req.select,
+            with_row_id=self._with_row_id,
+        )
+        if not self._blob_auto_row_id:
+            self._blob_paths = ()
+            return
+        self._blob_paths = tuple(blob_v2_projection_sources(schema, req.select).keys())
+        self._inner.with_row_id()
+
     def select(self, columns: Union[List[str], dict[str, str]]) -> Self:
         """
         Return only the specified columns.
@@ -2596,6 +2772,7 @@ class AsyncQueryBase(object):
         """
         Include the _rowid column in the results.
         """
+        self._with_row_id = True
         self._inner.with_row_id()
         return self
 
@@ -2642,6 +2819,7 @@ class AsyncQueryBase(object):
             If not specified, no timeout is applied. If the query does not
             complete within the specified time, an error will be raised.
         """
+        await self._maybe_add_blob_row_id()
         return AsyncRecordBatchReader(
             await self._inner.execute(
                 max_batch_length=max_batch_length, timeout=timeout
@@ -2672,8 +2850,8 @@ class AsyncQueryBase(object):
             complete within the specified time, an error will be raised.
         """
         batch_iter = await self.to_batches(timeout=timeout)
-        return pa.Table.from_batches(
-            await batch_iter.read_all(), schema=batch_iter.schema
+        return self._finalize_blob_query_table(
+            pa.Table.from_batches(await batch_iter.read_all(), schema=batch_iter.schema)
         )
 
     async def to_list(self, timeout: Optional[timedelta] = None) -> List[dict]:
@@ -2740,7 +2918,7 @@ class AsyncQueryBase(object):
             Forwarded to pyarrow.Table.to_pandas after query execution and
             optional flattening.
         """
-        _validate_blob_mode(blob_mode)
+        validate_blob_mode(blob_mode)
         if hasattr(self._inner, "output_schema"):
             schema = await self.output_schema()
             if _blob_mode_requires_native_pandas(blob_mode, schema):
@@ -2781,14 +2959,36 @@ class AsyncQueryBase(object):
         if not _query_is_plain_scan(query):
             return None
 
+        schema = await self._table.schema()
+        blob_auto_row_id = blob_auto_row_id_for_scan(
+            self._table,
+            schema,
+            query.columns,
+            with_row_id=self._with_row_id,
+        )
+        blob_sources = (
+            blob_v2_projection_sources(schema, query.columns)
+            if blob_mode == "bytes"
+            else {}
+        )
         dataset = await self._table._to_lance()
         scanner = dataset.scanner(
-            **_scanner_kwargs_for_query(query, blob_mode, dataset)
+            **_scanner_kwargs_for_query(
+                query,
+                "descriptions" if blob_sources else blob_mode,
+                dataset,
+                with_row_id=query.with_row_id or blob_auto_row_id or bool(blob_sources),
+            )
+        )
+        return await _finish_plain_scan_pandas_async(
+            scanner,
+            blob_mode=blob_mode,
+            blob_sources=blob_sources,
+            fetch_blobs=self._table.fetch_blobs,
+            strip_auto_row_id=blob_auto_row_id,
+            flatten=flatten,
+            **kwargs,
         )
-        if flatten is not None:
-            tbl = flatten_columns(_scanner_to_table(scanner), flatten)
-            return tbl.to_pandas(**kwargs)
-        return _scanner_to_pandas(scanner, blob_mode, **kwargs)
 
     async def to_polars(
         self,
@@ -3573,9 +3773,24 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
         fts_query = AsyncFTSQuery(self._inner.to_fts_query(), self._table)
         vec_query = AsyncVectorQuery(self._inner.to_vector_query(), self._table)
 
-        # save the row ID choice that was made on the query builder and force it
-        # to actually fetch the row ids because we need this for reranking
-        with_row_ids = self._inner.get_with_row_id()
+        req = fts_query._inner.to_query_request()
+        blob_auto_row_id = False
+        blob_paths: tuple[str, ...] = ()
+        if self._table is not None and supports_blob_auto_row_id(self._table):
+            schema = await self._table.schema()
+            blob_auto_row_id = blob_auto_row_id_for_scan(
+                self._table,
+                schema,
+                req.select,
+                with_row_id=self._with_row_id,
+            )
+            if blob_auto_row_id:
+                blob_paths = tuple(
+                    blob_v2_projection_sources(schema, req.select).keys()
+                )
+        self._blob_auto_row_id = blob_auto_row_id
+        self._blob_paths = blob_paths
+
         fts_query.with_row_id()
         vec_query.with_row_id()
 
@@ -3591,8 +3806,14 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
             fts_query=fts_query.get_query(),
             reranker=self._reranker,
             limit=self._inner.get_limit(),
-            with_row_ids=with_row_ids,
+            with_row_ids=True,
         )
+        if (
+            not self._user_requested_row_id()
+            and not blob_auto_row_id
+            and "_rowid" in result.column_names
+        ):
+            result = result.drop(["_rowid"])
 
         return AsyncRecordBatchReader(result, max_batch_length=max_batch_length)
 

python/python/lancedb/remote/table.py

@@ -28,6 +28,7 @@ from lancedb._lancedb import (
     UpdateFieldMetadataResult,
     DeleteResult,
     DropColumnsResult,
+    FtsToken,
     IndexConfig,
     LsmWriteSpec,
     MergeResult,
@@ -244,6 +245,23 @@ class RemoteTable(Table):
         """List all the indices on the table"""
         return LOOP.run(self._table.list_indices())
 
+    def tokenize(
+        self,
+        query: str,
+        *,
+        column: Optional[str] = None,
+        index_name: Optional[str] = None,
+    ) -> Iterable[FtsToken]:
+        """Tokenize a query using the tokenizer configured on an FTS index.
+
+        Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
+        rebuilt in the client process from index metadata, so the same tokenizer
+        model files must exist locally.
+        """
+        return LOOP.run(
+            self._table.tokenize(query, column=column, index_name=index_name)
+        )
+
     def index_stats(self, index_uuid: str) -> Optional[IndexStatistics]:
         """List all the stats of a specified index"""
         return LOOP.run(self._table.index_stats(index_uuid))
@@ -912,6 +930,10 @@ class RemoteTable(Table):
         """Not supported on LanceDB Cloud."""
         return LOOP.run(self._table.unset_lsm_write_spec())
 
+    def get_lsm_write_spec(self) -> Optional["LsmWriteSpec"]:
+        """Read the installed LsmWriteSpec, or ``None``."""
+        return LOOP.run(self._table.get_lsm_write_spec())
+
     def close_lsm_writers(self) -> None:
         """No-op on LanceDB Cloud (no local shard writers)."""
         return LOOP.run(self._table.close_lsm_writers())
@@ -990,6 +1012,19 @@ class RemoteTable(Table):
             "migrate_v2_manifest_paths() is not supported on the LanceDB Cloud"
         )
 
+    def blob_columns(self) -> list[str]:
+        raise NotImplementedError(
+            "blob_columns() is not yet supported on the LanceDB Cloud"
+        )
+
+    def fetch_blobs(self, column: str, row_ids) -> pa.LargeBinaryArray:
+        raise NotImplementedError("fetch_blobs() is not supported on LanceDB Cloud")
+
+    def fetch_blob_files(self, column: str, row_ids):
+        raise NotImplementedError(
+            "fetch_blob_files() is not supported on LanceDB Cloud"
+        )
+
     def head(self, n=5) -> pa.Table:
         """
         Return the first `n` rows of the table.

python/python/lancedb/rerankers/__init__.py

@@ -12,6 +12,7 @@ from .rrf import RRFReranker
 from .mrr import MRRReranker
 from .answerdotai import AnswerdotaiRerankers
 from .voyageai import VoyageAIReranker
+from .watsonx import WatsonxReranker
 
 __all__ = [
     "Reranker",
@@ -25,4 +26,5 @@ __all__ = [
     "AnswerdotaiRerankers",
     "VoyageAIReranker",
     "MRRReranker",
+    "WatsonxReranker",
 ]

python/python/lancedb/rerankers/watsonx.py

@@ -0,0 +1,180 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+
+import os
+from functools import cached_property
+from typing import Dict, Optional
+
+import pyarrow as pa
+
+from ..util import attempt_import_or_raise
+from .base import Reranker
+
+DEFAULT_WATSONX_URL = "https://us-south.ml.cloud.ibm.com"
+
+
+class WatsonxReranker(Reranker):
+    """
+    Reranks the results using the IBM watsonx.ai Rerank API.
+
+    Uses the ``ibm_watsonx_ai`` SDK (``Rerank.generate``) under the hood.
+
+    API Docs:
+        https://cloud.ibm.com/docs/apis/watsonx-ai#text-rerank
+
+    Supported rerank models:
+        https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx#rerank
+
+    Parameters
+    ----------
+    model_name : str, default "cross-encoder/ms-marco-minilm-l-12-v2"
+        The ID of the rerank model to use.
+    column : str, default "text"
+        The name of the column to use as input to the reranker.
+    top_n : int, optional
+        Return only the top-n results. If ``None``, all results are returned.
+    return_score : str, default "relevance"
+        Options are ``"relevance"`` or ``"all"``.
+    api_key : str, optional
+        IBM Cloud API key.  Falls back to the ``WATSONX_API_KEY`` environment
+        variable when not provided.
+    project_id : str, optional
+        watsonx.ai project ID.  Falls back to the ``WATSONX_PROJECT_ID``
+        environment variable when not provided.  Mutually exclusive with
+        ``space_id`` — exactly one must be supplied.
+    space_id : str, optional
+        watsonx.ai deployment space ID.  Falls back to the ``WATSONX_SPACE_ID``
+        environment variable when not provided.  Mutually exclusive with
+        ``project_id`` — exactly one must be supplied.
+    url : str, optional
+        watsonx.ai service URL.  Defaults to
+        ``"https://us-south.ml.cloud.ibm.com"``.
+    truncate_input_tokens : int, optional
+        Truncate each input to this many tokens before scoring.  Passed
+        directly to the ``parameters`` dict of ``Rerank.generate``.
+    """
+
+    def __init__(
+        self,
+        model_name: str = "cross-encoder/ms-marco-minilm-l-12-v2",
+        column: str = "text",
+        top_n: Optional[int] = None,
+        return_score: str = "relevance",
+        api_key: Optional[str] = None,
+        project_id: Optional[str] = None,
+        space_id: Optional[str] = None,
+        url: Optional[str] = None,
+        truncate_input_tokens: Optional[int] = None,
+    ):
+        super().__init__(return_score)
+        self.model_name = model_name
+        self.column = column
+        self.top_n = top_n
+        self.api_key = api_key
+        self.project_id = project_id
+        self.space_id = space_id
+        self.url = url
+        self.truncate_input_tokens = truncate_input_tokens
+
+    def __str__(self) -> str:
+        return f"WatsonxReranker(model_name={self.model_name})"
+
+    @cached_property
+    def _client(self):
+        ibm_watsonx_ai = attempt_import_or_raise("ibm_watsonx_ai")
+        ibm_watsonx_ai_foundation_models = attempt_import_or_raise(
+            "ibm_watsonx_ai.foundation_models"
+        )
+
+        # --- credentials ---
+        api_key = self.api_key or os.environ.get("WATSONX_API_KEY")
+        if not api_key:
+            raise ValueError(
+                "WATSONX_API_KEY not set. Either set it in your environment or "
+                "pass it as `api_key` argument to WatsonxReranker."
+            )
+        credentials = ibm_watsonx_ai.Credentials(
+            api_key=api_key,
+            url=self.url or DEFAULT_WATSONX_URL,
+        )
+
+        # --- project_id / space_id (exactly one required) ---
+        project_id = self.project_id or os.environ.get("WATSONX_PROJECT_ID")
+        space_id = self.space_id or os.environ.get("WATSONX_SPACE_ID")
+
+        if project_id and space_id:
+            raise ValueError("Provide either `project_id` or `space_id`, not both.")
+        if not project_id and not space_id:
+            raise ValueError(
+                "Either WATSONX_PROJECT_ID or WATSONX_SPACE_ID must be set. "
+                "Pass one as an argument to WatsonxReranker or set the corresponding "
+                "environment variable."
+            )
+
+        kwargs: Dict = dict(model_id=self.model_name, credentials=credentials)
+        if project_id:
+            kwargs["project_id"] = project_id
+        else:
+            kwargs["space_id"] = space_id
+
+        return ibm_watsonx_ai_foundation_models.Rerank(**kwargs)
+
+    def _build_params(self) -> Dict:
+        """Build the ``parameters`` dict forwarded to ``Rerank.generate``."""
+        return_options: Dict = {"inputs": True}
+        if self.top_n is not None:
+            return_options["top_n"] = self.top_n
+        params: Dict = {"return_options": return_options}
+        if self.truncate_input_tokens is not None:
+            params["truncate_input_tokens"] = self.truncate_input_tokens
+        return params
+
+    def _rerank(self, result_set: pa.Table, query: str) -> pa.Table:
+        result_set = self._handle_empty_results(result_set)
+        if len(result_set) == 0:
+            return result_set
+
+        docs = result_set[self.column].to_pylist()
+        response = self._client.generate(
+            query=query,
+            inputs=docs,
+            params=self._build_params(),
+        )
+        results = response["results"]
+
+        indices, scores = zip(
+            *[(result["index"], result["score"]) for result in results]
+        )
+        result_set = result_set.take(list(indices))
+        result_set = result_set.append_column(
+            "_relevance_score", pa.array(scores, type=pa.float32())
+        )
+        return result_set
+
+    def rerank_hybrid(
+        self,
+        query: str,
+        vector_results: pa.Table,
+        fts_results: pa.Table,
+    ) -> pa.Table:
+        if self.score == "all":
+            combined_results = self._merge_and_keep_scores(vector_results, fts_results)
+        else:
+            combined_results = self.merge_results(vector_results, fts_results)
+        combined_results = self._rerank(combined_results, query)
+        if self.score == "relevance":
+            combined_results = self._keep_relevance_score(combined_results)
+        return combined_results
+
+    def rerank_vector(self, query: str, vector_results: pa.Table) -> pa.Table:
+        vector_results = self._rerank(vector_results, query)
+        if self.score == "relevance":
+            vector_results = vector_results.drop_columns(["_distance"])
+        return vector_results
+
+    def rerank_fts(self, query: str, fts_results: pa.Table) -> pa.Table:
+        fts_results = self._rerank(fts_results, query)
+        if self.score == "relevance":
+            fts_results = fts_results.drop_columns(["_score"])
+        return fts_results

python/python/lancedb/schema.py

@@ -2,10 +2,134 @@
 # SPDX-FileCopyrightText: Copyright The LanceDB Authors
 
 
-"""Schema related utilities."""
+"""Schema helpers for Lance blob columns."""
 
 import pyarrow as pa
 
+_BLOB_EXTENSION_NAME = "lance.blob.v2"
+_BLOB_V1_KEY = "lance-encoding:blob"
+_ARROW_EXT_NAME_KEY = "ARROW:extension:name"
+
+
+class BlobType(pa.ExtensionType):
+    """PyArrow extension type for a Lance blob v2 column.
+
+    Queries return descriptors; call :meth:`~lancedb.table.Table.fetch_blob_files`
+    for lazy reads or :meth:`~lancedb.table.Table.fetch_blobs` for eager bytes.
+    """
+
+    def __init__(self) -> None:
+        storage_type = pa.struct(
+            [
+                pa.field("data", pa.large_binary(), nullable=True),
+                pa.field("uri", pa.utf8(), nullable=True),
+                pa.field("position", pa.uint64(), nullable=True),
+                pa.field("size", pa.uint64(), nullable=True),
+            ]
+        )
+        super().__init__(storage_type, _BLOB_EXTENSION_NAME)
+
+    def __arrow_ext_serialize__(self) -> bytes:
+        return b""
+
+    @classmethod
+    def __arrow_ext_deserialize__(
+        cls, storage_type: pa.DataType, serialized: bytes
+    ) -> "BlobType":
+        return cls()
+
+    def __reduce__(self):
+        # Ensure pickle round-trips on older pyarrow (apache/arrow#35599).
+        return type(self).__arrow_ext_deserialize__, (
+            self.storage_type,
+            self.__arrow_ext_serialize__(),
+        )
+
+
+try:
+    pa.register_extension_type(BlobType())  # type: ignore[arg-type]
+except pa.ArrowKeyError:
+    pass
+
+
+def _metadata_value(metadata: dict, key: str):
+    return metadata.get(key.encode()) or metadata.get(key)
+
+
+def _metadata_marks_blob_v2(metadata: dict) -> bool:
+    if not metadata:
+        return False
+
+    extension_name = _metadata_value(metadata, _ARROW_EXT_NAME_KEY)
+    return extension_name in (_BLOB_EXTENSION_NAME, _BLOB_EXTENSION_NAME.encode())
+
+
+def _metadata_marks_legacy_blob(metadata: dict) -> bool:
+    if not metadata:
+        return False
+
+    return _metadata_value(metadata, _BLOB_V1_KEY) in ("true", b"true")
+
+
+def is_blob_v2_field(field: pa.Field) -> bool:
+    """Return True if `field` declares a blob v2 extension column."""
+    field_type = field.type
+    if (
+        isinstance(field_type, pa.ExtensionType)
+        and field_type.extension_name == _BLOB_EXTENSION_NAME
+    ):
+        return True
+    return _metadata_marks_blob_v2(field.metadata or {})
+
+
+def is_blob_like_field(field: pa.Field) -> bool:
+    """Blob detection for ``to_pandas(blob_mode=...)`` and scanner paths only.
+
+    Matches v2 extension fields on table schema, legacy ``lance-encoding:blob``
+    storage columns, and v2 query descriptor fields (the engine tags those with
+    the same metadata). Not used for fetch or auto ``_rowid``.
+    """
+    return is_blob_v2_field(field) or _metadata_marks_legacy_blob(field.metadata or {})
+
+
+def _collect_blob_paths(schema: pa.Schema, is_blob) -> list[str]:
+    paths: list[str] = []
+
+    def walk(fields, prefix: str) -> None:
+        for field in fields:
+            path = f"{prefix}.{field.name}" if prefix else field.name
+            if is_blob(field):
+                paths.append(path)
+            elif pa.types.is_struct(field.type):
+                walk(field.type, path)
+            elif (
+                pa.types.is_list(field.type)
+                or pa.types.is_large_list(field.type)
+                or pa.types.is_fixed_size_list(field.type)
+            ):
+                walk([field.type.value_field], path)
+
+    walk(schema, "")
+    return paths
+
+
+def blob_column_paths(schema: pa.Schema) -> list[str]:
+    """Dotted paths of blob-like columns (v2 extension or legacy metadata)."""
+    return _collect_blob_paths(schema, is_blob_like_field)
+
+
+def blob_v2_column_paths(schema: pa.Schema) -> list[str]:
+    return _collect_blob_paths(schema, is_blob_v2_field)
+
+
+def schema_has_blob_field(schema: pa.Schema) -> bool:
+    return bool(blob_column_paths(schema))
+
+
+def blob(name: str, nullable: bool = True) -> pa.Field:
+    """Create a Lance blob v2 column field."""
+    return pa.field(name, BlobType(), nullable=nullable)
+
 
 def vector(dimension: int, value_type: pa.DataType = pa.float32()) -> pa.DataType:
     """A help function to create a vector type.

python/python/lancedb/table.py

@@ -29,6 +29,14 @@ from urllib.parse import urlparse
 from lancedb.scannable import _register_optional_converters, to_scannable
 
 from . import __version__
+from ._blob import (
+    BlobFile,
+    _normalize_blob_row_ids,
+    _wrap_blob_files,
+    strip_auto_row_ids,
+    validate_blob_mode,
+)
+from .types import BlobMode
 from lancedb.arrow import peek_reader
 from lancedb.background_loop import LOOP, embedding_executor
 from .dependencies import (
@@ -88,10 +96,7 @@ from .util import (
     value_to_sql,
 )
 from .index import lang_mapping
-
-BlobMode = Literal["lazy", "bytes", "descriptions"]
-
-_VALID_BLOB_MODES = ("lazy", "bytes", "descriptions")
+from .schema import blob_v2_column_paths, schema_has_blob_field
 
 
 def _should_push_down_query_table(
@@ -100,23 +105,6 @@ def _should_push_down_query_table(
     return namespace_client is not None and "QueryTable" in pushdown_operations
 
 
-def _validate_blob_mode(blob_mode: BlobMode) -> None:
-    if blob_mode not in _VALID_BLOB_MODES:
-        modes = ", ".join(repr(mode) for mode in _VALID_BLOB_MODES)
-        raise ValueError(f"blob_mode must be one of {modes}, got {blob_mode!r}")
-
-
-def _field_is_blob(field: pa.Field) -> bool:
-    metadata = field.metadata or {}
-    return metadata.get(b"lance-encoding:blob") == b"true" or (
-        metadata.get("lance-encoding:blob") == "true"
-    )
-
-
-def _schema_has_blob_field(schema: pa.Schema) -> bool:
-    return any(_field_is_blob(field) for field in schema)
-
-
 _MODEL_BACKED_TOKENIZER_PREFIXES = ("jieba", "lindera")
 _MODEL_BACKED_TOKENIZER_ERRORS = (
     "unknown base tokenizer",
@@ -185,6 +173,7 @@ if TYPE_CHECKING:
         UpdateFieldMetadataResult,
         DeleteResult,
         DropColumnsResult,
+        FtsToken,
         LsmWriteSpec,
         MergeResult,
         UpdateResult,
@@ -1159,6 +1148,8 @@ class Table(ABC):
             - "whitespace": Split text by whitespace, but not punctuation.
             - "raw": No tokenization. The entire text is treated as a single token.
             - "ngram": N-Gram tokenizer.
+            - "icu": ICU dictionary-based word segmentation.
+            - "icu/split": ICU segmentation with simple-style delimiter splitting.
             - "jieba/*": Jieba tokenizer loaded from Lance's language model home.
             - "lindera/*": Lindera tokenizer loaded from Lance's language model home.
         language : str, default "English"
@@ -1523,6 +1514,31 @@ class Table(ABC):
             A query object that can be executed to get the rows.
         """
 
+    @abstractmethod
+    def blob_columns(self) -> list[str]:
+        """Names of the blob v2 columns declared on this table."""
+
+    @abstractmethod
+    def fetch_blobs(
+        self, column: str, row_ids: Union[list[int], pa.Table]
+    ) -> pa.LargeBinaryArray:
+        """Materialize full blob bytes for ``column`` at the given rows.
+
+        Convenience for small payloads. For large values use
+        :meth:`fetch_blob_files`.
+        """
+
+    @abstractmethod
+    def fetch_blob_files(
+        self, column: str, row_ids: Union[list[int], pa.Table]
+    ) -> "list[Optional[BlobFile]]":
+        """Open lazy, seekable :class:`~lancedb._blob.BlobFile` handles.
+
+        Prefer this over :meth:`fetch_blobs` for large payloads. ``row_ids`` is
+        a ``list[int]`` or query ``pyarrow.Table`` with ``_rowid`` (or stashed
+        row-id metadata). Null rows are ``None``. Local tables only.
+        """
+
     @abstractmethod
     def _execute_query(
         self,
@@ -1786,6 +1802,24 @@ class Table(ABC):
         [Table.create_index][lancedb.table.Table.create_index]
         """
 
+    @abstractmethod
+    def tokenize(
+        self,
+        query: str,
+        *,
+        column: Optional[str] = None,
+        index_name: Optional[str] = None,
+    ) -> Iterable[FtsToken]:
+        """
+        Tokenize a query using the tokenizer configured on an FTS index.
+
+        Specify exactly one of ``column`` or ``index_name``.
+
+        Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
+        rebuilt in the client process from index metadata. For remote tables,
+        this means the same tokenizer model files must also exist locally.
+        """
+
     @abstractmethod
     def index_stats(self, index_name: str) -> Optional[IndexStatistics]:
         """
@@ -2204,6 +2238,19 @@ class LanceTable(Table):
     def take_row_ids(self, row_ids: list[int]) -> LanceTakeQueryBuilder:
         return LanceTakeQueryBuilder(self._table.take_row_ids(row_ids))
 
+    def blob_columns(self) -> list[str]:
+        return LOOP.run(self._table.blob_columns())
+
+    def fetch_blobs(
+        self, column: str, row_ids: Union[list[int], pa.Table]
+    ) -> pa.LargeBinaryArray:
+        return LOOP.run(self._table.fetch_blobs(column, row_ids))
+
+    def fetch_blob_files(
+        self, column: str, row_ids: Union[list[int], pa.Table]
+    ) -> "list[Optional[BlobFile]]":
+        return LOOP.run(self._table.fetch_blob_files(column, row_ids))
+
     @property
     def tags(self) -> Tags:
         """Tag management for the table.
@@ -2399,9 +2446,14 @@ class LanceTable(Table):
         -------
         pd.DataFrame
         """
-        _validate_blob_mode(blob_mode)
-        if blob_mode == "descriptions" or not _schema_has_blob_field(self.schema):
-            return self.to_arrow().to_pandas(**kwargs)
+        validate_blob_mode(blob_mode)
+        if blob_mode == "descriptions" or not schema_has_blob_field(self.schema):
+            arrow_tbl = self.to_arrow()
+            if blob_mode == "descriptions":
+                arrow_tbl = strip_auto_row_ids(
+                    arrow_tbl, blob_v2_column_paths(self.schema)
+                )
+            return arrow_tbl.to_pandas(**kwargs)
 
         if (
             blob_mode == "lazy"
@@ -2410,6 +2462,9 @@ class LanceTable(Table):
         ):
             return self.to_arrow().to_pandas(**kwargs)
 
+        if blob_mode == "bytes" and blob_v2_column_paths(self.schema):
+            return self.search().to_pandas(blob_mode=blob_mode, **kwargs)
+
         return self.to_lance().to_pandas(blob_mode=blob_mode, **kwargs)
 
     def to_arrow(self) -> pa.Table:
@@ -3710,6 +3765,26 @@ class LanceTable(Table):
         """
         return LOOP.run(self._table.list_indices())
 
+    def tokenize(
+        self,
+        query: str,
+        *,
+        column: Optional[str] = None,
+        index_name: Optional[str] = None,
+    ) -> Iterable[FtsToken]:
+        """
+        Tokenize a query using the tokenizer configured on an FTS index.
+
+        Specify exactly one of ``column`` or ``index_name``.
+
+        Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
+        rebuilt in the client process from index metadata. For remote tables,
+        this means the same tokenizer model files must also exist locally.
+        """
+        return LOOP.run(
+            self._table.tokenize(query, column=column, index_name=index_name)
+        )
+
     def index_stats(self, index_name: str) -> Optional[IndexStatistics]:
         """
         Retrieve statistics about an index
@@ -3759,6 +3834,11 @@ class LanceTable(Table):
         [`AsyncTable.unset_lsm_write_spec`][lancedb.AsyncTable.unset_lsm_write_spec]."""
         return LOOP.run(self._table.unset_lsm_write_spec())
 
+    def get_lsm_write_spec(self) -> Optional["LsmWriteSpec"]:
+        """Read the installed LsmWriteSpec, or ``None``. See
+        [`AsyncTable.get_lsm_write_spec`][lancedb.AsyncTable.get_lsm_write_spec]."""
+        return LOOP.run(self._table.get_lsm_write_spec())
+
     def close_lsm_writers(self) -> None:
         """Close cached MemWAL shard writers. See
         [`AsyncTable.close_lsm_writers`][lancedb.AsyncTable.close_lsm_writers]."""
@@ -4074,17 +4154,58 @@ def _handle_bad_vector_column(
                 raise ValueError(
                     "`fill_value` must not be None if `on_bad_vectors` is 'fill'"
                 )
-            vec_arr = pc.if_else(
-                is_bad,
-                pa.scalar([fill_value] * dim, type=vec_arr.type),
-                vec_arr,
-            )
+            vec_arr = _fill_bad_vector_values(vec_arr, dim, fill_value)
         else:
             raise ValueError(f"Invalid value for on_bad_vectors: {on_bad_vectors}")
 
     return data.set_column(position, vector_column_name, vec_arr)
 
 
+def _fill_bad_vector_values(
+    arr: Union[pa.Array, pa.ChunkedArray],
+    dim: int,
+    fill_value: float,
+) -> pa.Array:
+    if not isinstance(arr, pa.ChunkedArray):
+        arr = pa.chunked_array([arr])
+    arr = arr.combine_chunks()
+
+    # A fixed-size slice truncates long vectors and pads short vectors with nulls.
+    # Slice an array marking the original child nulls in parallel so padding nulls
+    # can be distinguished from null values that were already present.
+    sliced = pc.list_slice(arr, 0, dim, return_fixed_size_list=True)
+    child_nulls = pc.is_null(arr.values)
+    parent_nulls = pc.is_null(arr)
+    if pa.types.is_list(arr.type):
+        original_child_nulls = pa.ListArray.from_arrays(
+            arr.offsets, child_nulls, mask=parent_nulls
+        )
+    elif pa.types.is_large_list(arr.type):
+        original_child_nulls = pa.LargeListArray.from_arrays(
+            arr.offsets, child_nulls, mask=parent_nulls
+        )
+    else:
+        original_child_nulls = pa.FixedSizeListArray.from_arrays(
+            child_nulls, arr.type.list_size, mask=parent_nulls
+        )
+    sliced_child_nulls = pc.list_slice(
+        original_child_nulls, 0, dim, return_fixed_size_list=True
+    )
+    needs_fill = pc.is_null(sliced_child_nulls.values)
+
+    values = sliced.values
+    if pa.types.is_floating(values.type):
+        values_for_nan_check = (
+            values.cast(pa.float32()) if pa.types.is_float16(values.type) else values
+        )
+        needs_fill = pc.or_kleene(needs_fill, pc.is_nan(values_for_nan_check))
+
+    fill_scalar = pa.scalar(fill_value).cast(values.type)
+    filled_values = pc.if_else(needs_fill, fill_scalar, values)
+    filled = pa.FixedSizeListArray.from_arrays(filled_values, dim)
+    return filled.cast(arr.type)
+
+
 def has_nan_values(arr: Union[pa.ListArray, pa.ChunkedArray]) -> pa.BooleanArray:
     if isinstance(arr, pa.ChunkedArray):
         values = pa.chunked_array([chunk.flatten() for chunk in arr.chunks])
@@ -4417,6 +4538,17 @@ class AsyncTable:
         """
         await self._inner.unset_lsm_write_spec()
 
+    async def get_lsm_write_spec(self) -> Optional["LsmWriteSpec"]:
+        """Read the LsmWriteSpec currently installed on this table.
+
+        Returns ``None`` when the MemWAL LSM write path is not enabled (no
+        spec has been set, or it was removed with `unset_lsm_write_spec`).
+        The returned spec — including its ``maintained_indexes`` and
+        ``writer_config_defaults`` — mirrors what was passed to
+        `set_lsm_write_spec`.
+        """
+        return await self._inner.get_lsm_write_spec()
+
     async def close_lsm_writers(self) -> None:
         """Drain and close any cached MemWAL shard writers for this table.
 
@@ -4523,14 +4655,18 @@ class AsyncTable:
         -------
         pd.DataFrame
         """
-        _validate_blob_mode(blob_mode)
-        if blob_mode == "descriptions" or not _schema_has_blob_field(
-            await self.schema()
-        ):
-            return (await self.to_arrow()).to_pandas(**kwargs)
+        validate_blob_mode(blob_mode)
+        schema = await self.schema()
+        if blob_mode == "descriptions" or not schema_has_blob_field(schema):
+            arrow_tbl = await self.to_arrow()
+            if blob_mode == "descriptions":
+                arrow_tbl = strip_auto_row_ids(arrow_tbl, blob_v2_column_paths(schema))
+            return arrow_tbl.to_pandas(**kwargs)
 
         if blob_mode == "lazy" and get_uri_scheme(await self.uri()) == "memory":
             return (await self.to_arrow()).to_pandas(**kwargs)
+        if blob_mode == "bytes" and blob_v2_column_paths(schema):
+            return await self.query().to_pandas(blob_mode=blob_mode, **kwargs)
         return (await self._to_lance()).to_pandas(blob_mode=blob_mode, **kwargs)
 
     async def to_arrow(self) -> pa.Table:
@@ -5631,6 +5767,24 @@ class AsyncTable:
         """
         return AsyncTakeQuery(self._inner.take_row_ids(row_ids), self)
 
+    async def blob_columns(self) -> list[str]:
+        return await self._inner.blob_columns()
+
+    async def fetch_blobs(
+        self, column: str, row_ids: Union[list[int], pa.Table]
+    ) -> pa.LargeBinaryArray:
+        return await self._inner.fetch_blobs(
+            column, _normalize_blob_row_ids(row_ids, column)
+        )
+
+    async def fetch_blob_files(
+        self, column: str, row_ids: Union[list[int], pa.Table]
+    ) -> "list[Optional[BlobFile]]":
+        handles = await self._inner.fetch_blob_files(
+            column, _normalize_blob_row_ids(row_ids, column)
+        )
+        return _wrap_blob_files(handles)
+
     @property
     def tags(self) -> AsyncTags:
         """Tag management for the dataset.
@@ -5732,6 +5886,24 @@ class AsyncTable:
         """
         return await self._inner.list_indices()
 
+    async def tokenize(
+        self,
+        query: str,
+        *,
+        column: Optional[str] = None,
+        index_name: Optional[str] = None,
+    ) -> Iterable[FtsToken]:
+        """
+        Tokenize a query using the tokenizer configured on an FTS index.
+
+        Specify exactly one of ``column`` or ``index_name``.
+
+        Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
+        rebuilt in the client process from index metadata. For remote tables,
+        this means the same tokenizer model files must also exist locally.
+        """
+        return await self._inner.tokenize(query, column=column, index_name=index_name)
+
     async def index_stats(self, index_name: str) -> Optional[IndexStatistics]:
         """
         Retrieve statistics about an index

python/python/lancedb/types.py

@@ -1,11 +1,24 @@
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileCopyrightText: Copyright The LanceDB Authors
 
-from typing import Literal
+from __future__ import annotations
+
+from typing import Dict, List, Literal, Optional, Tuple, Union
+
+from .expr import Expr
 
 # Query type literals
 QueryType = Literal["vector", "fts", "hybrid", "auto"]
 
+BlobMode = Literal["lazy", "bytes", "descriptions"]
+
+QueryProjectionSpec = Union[
+    List[str],
+    List[Tuple[str, Union[str, Expr]]],
+    Dict[str, Union[str, Expr]],
+]
+QueryProjection = Optional[QueryProjectionSpec]
+
 # Distance type literals
 DistanceType = Literal["l2", "cosine", "dot"]
 DistanceTypeWithHamming = Literal["l2", "cosine", "dot", "hamming"]
@@ -42,5 +55,7 @@ IndexType = Literal[
 ]
 
 # Tokenizer literals
-BuiltinTokenizerType = Literal["simple", "raw", "whitespace", "ngram"]
+BuiltinTokenizerType = Literal[
+    "simple", "raw", "whitespace", "ngram", "icu", "icu/split"
+]
 BaseTokenizerType = BuiltinTokenizerType | str

python/python/lancedb/util.py

@@ -177,7 +177,10 @@ def flatten_columns(tbl: pa.Table, flatten: Optional[Union[int, bool]] = None):
                 continue
             else:
                 break
-    elif isinstance(flatten, int):
+    # `bool` is a subclass of `int`, so guard against it explicitly: `flatten=False`
+    # (and `None`) must mean "do not flatten" rather than falling into the integer
+    # branch and raising on the `flatten <= 0` check.
+    elif isinstance(flatten, int) and not isinstance(flatten, bool):
         if flatten <= 0:
             raise ValueError(
                 "Please specify a positive integer for flatten or the boolean "
@@ -515,3 +518,38 @@ def batch_to_tensor_rows(batch: pa.RecordBatch):
     stacked = torch.tensor(numpy.column_stack(columns))
     rows = list(stacked.unbind(dim=0))
     return rows
+
+
+def batch_to_tensor_dict(batch: pa.RecordBatch):
+    """
+    Convert a PyArrow RecordBatch to a list of per-row dicts of PyTorch Tensors.
+
+    Each column is first converted to a 1-D tensor (zero-copy via DLPack), then
+    sliced per row. The result is a list whose length is ``batch.num_rows`` and
+    whose items are dicts keyed by column name. This shape composes directly
+    with PyTorch's default ``DataLoader`` collate, which stacks the per-row
+    dicts back into a dict of batched tensors.
+
+    Fails if torch is not installed.
+    Fails if a column's data type is not supported by PyTorch.
+
+    Parameters
+    ----------
+    batch : pa.RecordBatch
+        The record batch to convert.
+
+    Returns
+    -------
+    list[dict[str, torch.Tensor]]
+        One per-row dict per row in the batch. Each dict maps column name to a
+        0-D tensor view into the column.
+    """
+    torch = attempt_import_or_raise("torch", "torch")
+    tensors = {
+        name: torch.from_dlpack(col)
+        for name, col in zip(batch.schema.names, batch.columns)
+    }
+    return [
+        {name: tensor[i] for name, tensor in tensors.items()}
+        for i in range(batch.num_rows)
+    ]

python/python/tests/test_blob.py

@@ -0,0 +1,562 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+import io
+
+import pyarrow as pa
+import pyarrow.compute as pc
+import pytest
+
+import lancedb
+from lancedb._blob import read_row_ids_from_hits, stash_auto_row_ids
+from lancedb.index import FTS
+from lancedb.schema import blob_column_paths, blob_v2_column_paths
+
+
+def _blob_table(name, rows):
+    db = lancedb.connect("memory:///")
+    schema = pa.schema([pa.field("id", pa.int64()), lancedb.blob("image")])
+    table = db.create_table(name, schema=schema)
+    table.add(rows)
+    return table
+
+
+def _blob_array(name, values):
+    blob_type = lancedb.blob(name).type
+    storage_type = blob_type.storage_type
+    storage = pa.StructArray.from_arrays(
+        [
+            pa.array(values, type=pa.large_binary()),
+            pa.array([None] * len(values), type=pa.string()),
+            pa.array([None] * len(values), type=pa.uint64()),
+            pa.array([None] * len(values), type=pa.uint64()),
+        ],
+        fields=list(storage_type),
+    )
+    return pa.ExtensionArray.from_storage(blob_type, storage)
+
+
+def _row_ids_by_id(table):
+    hits = table.search().with_row_id(True).limit(1000).to_arrow()
+    assert "_rowid" in hits.column_names
+    return dict(zip(hits["id"].to_pylist(), hits["_rowid"].to_pylist()))
+
+
+def test_blob_factory_declares_v2_field():
+    field = lancedb.blob("image")
+    assert isinstance(field.type, pa.ExtensionType)
+    assert field.type.extension_name == "lance.blob.v2"
+
+
+def test_blob_v2_column_paths_include_list_children():
+    schema = pa.schema(
+        [
+            pa.field("id", pa.int64()),
+            pa.field("info", pa.struct([lancedb.blob("blob")])),
+            pa.field("images", pa.list_(lancedb.blob("image"))),
+            pa.field("large_images", pa.large_list(lancedb.blob("large_image"))),
+            pa.field(
+                "fixed_images",
+                pa.list_(lancedb.blob("fixed_image"), list_size=2),
+            ),
+        ]
+    )
+
+    assert blob_v2_column_paths(schema) == [
+        "info.blob",
+        "images.image",
+        "large_images.large_image",
+        "fixed_images.fixed_image",
+    ]
+
+
+def _legacy_v1_table(name):
+    db = lancedb.connect("memory:///")
+    schema = pa.schema(
+        [
+            pa.field("id", pa.int64()),
+            pa.field(
+                "legacy", pa.large_binary(), metadata={"lance-encoding:blob": "true"}
+            ),
+        ]
+    )
+    table = db.create_table(name, schema=schema)
+    table.add([{"id": 1, "legacy": b"old"}])
+    return table
+
+
+def test_blob_v2_column_paths_exclude_legacy_metadata():
+    schema = pa.schema(
+        [
+            pa.field("id", pa.int64()),
+            lancedb.blob("image"),
+            pa.field(
+                "legacy", pa.large_binary(), metadata={"lance-encoding:blob": "true"}
+            ),
+        ]
+    )
+    assert blob_v2_column_paths(schema) == ["image"]
+    assert blob_column_paths(schema) == ["image", "legacy"]
+
+
+def test_blob_v2_paths_match_blob_columns():
+    table = _blob_table("paths_match", [{"id": 1, "image": b"x"}])
+    assert blob_v2_column_paths(table.schema) == table.blob_columns()
+
+    db = lancedb.connect("memory:///")
+    info = pa.StructArray.from_arrays(
+        [
+            pa.array(["first"], type=pa.string()),
+            _blob_array("blob", [b"nested"]),
+        ],
+        names=["name", "blob"],
+    )
+    data = pa.Table.from_arrays(
+        [pa.array([1], type=pa.int64()), info],
+        names=["id", "info"],
+    )
+    nested = db.create_table("nested_paths", data=data)
+    assert blob_v2_column_paths(nested.schema) == nested.blob_columns()
+
+
+def test_auto_row_id_stash_round_trip():
+    table = _blob_table(
+        "stash_round_trip",
+        [{"id": 1, "image": b"alpha"}, {"id": 2, "image": b"beta"}],
+    )
+    hits = table.search().with_row_id(True).limit(10).to_arrow()
+    row_ids = hits["_rowid"].to_pylist()
+
+    stashed = stash_auto_row_ids(hits, ["image"])
+
+    assert "_rowid" not in stashed.column_names
+    assert stashed.schema.field("image").metadata == hits.schema.field("image").metadata
+    assert read_row_ids_from_hits(stashed, "image") == row_ids
+
+
+def test_blob_query_omits_auto_row_id():
+    table = _blob_table("rowid", [{"id": 1, "image": b"x"}])
+    hits = table.search().limit(10).to_arrow()
+    assert "_rowid" not in hits.column_names
+
+
+def test_blob_query_explicit_row_id_opt_in():
+    table = _blob_table("explicit_rowid", [{"id": 1, "image": b"x"}])
+    hits = table.search().with_row_id(True).limit(10).to_arrow()
+    assert "_rowid" in hits.column_names
+
+
+def test_table_to_pandas_descriptions_mode_omits_row_id():
+    table = _blob_table("descriptions_no_leak", [{"id": 1, "image": b"x"}])
+    df = table.to_pandas(blob_mode="descriptions")
+    descriptor = df["image"].iloc[0]
+    assert "_lance_row_id" not in descriptor
+    assert set(descriptor.keys()) == {"kind", "position", "size", "blob_id", "blob_uri"}
+
+
+@pytest.mark.asyncio
+async def test_async_table_to_pandas_descriptions_mode_omits_row_id():
+    db = await lancedb.connect_async("memory:///")
+    schema = pa.schema([pa.field("id", pa.int64()), lancedb.blob("image")])
+    table = await db.create_table("descriptions_no_leak_async", schema=schema)
+    await table.add([{"id": 1, "image": b"x"}])
+    df = await table.to_pandas(blob_mode="descriptions")
+    descriptor = df["image"].iloc[0]
+    assert "_lance_row_id" not in descriptor
+    assert set(descriptor.keys()) == {"kind", "position", "size", "blob_id", "blob_uri"}
+
+
+def test_fetch_blobs_round_trip():
+    table = _blob_table(
+        "round_trip",
+        [{"id": 1, "image": b"alpha"}, {"id": 2, "image": b"beta"}],
+    )
+    by_id = _row_ids_by_id(table)
+    blobs = table.fetch_blobs("image", [by_id[1], by_id[2]])
+    assert [blobs[0].as_py(), blobs[1].as_py()] == [b"alpha", b"beta"]
+
+
+def test_fetch_blobs_accepts_query_result():
+    table = _blob_table("from_result", [{"id": 1, "image": b"gamma"}])
+    hits = table.search().limit(10).to_arrow()
+    assert "_rowid" not in hits.column_names
+    blobs = table.fetch_blobs("image", hits)
+    assert {blobs[i].as_py() for i in range(len(blobs))} == {b"gamma"}
+
+
+def test_fetch_blobs_null_alignment():
+    table = _blob_table(
+        "nulls",
+        [{"id": 1, "image": b"present"}, {"id": 2, "image": None}],
+    )
+    by_id = _row_ids_by_id(table)
+    request = [by_id[1], by_id[2], by_id[1]]
+    blobs = table.fetch_blobs("image", request)
+    assert len(blobs) == len(request)
+    assert blobs[0].as_py() == b"present"
+    assert blobs[1].as_py() is None
+    assert blobs[2].as_py() == b"present"
+
+
+def test_fetch_blobs_nested_path():
+    db = lancedb.connect("memory:///")
+    info = pa.StructArray.from_arrays(
+        [
+            pa.array(["first", "second"], type=pa.string()),
+            _blob_array("blob", [b"nested-alpha", b"nested-beta"]),
+        ],
+        names=["name", "blob"],
+    )
+    data = pa.Table.from_arrays(
+        [pa.array([1, 2], type=pa.int64()), info],
+        names=["id", "info"],
+    )
+    table = db.create_table("nested", data=data)
+
+    by_id = _row_ids_by_id(table)
+    blobs = table.fetch_blobs("info.blob", [by_id[1], by_id[2]])
+    assert [blobs[0].as_py(), blobs[1].as_py()] == [b"nested-alpha", b"nested-beta"]
+
+
+def test_fetch_blob_files_lazy_read():
+    payload = b"lazy-read" * 100
+    table = _blob_table("lazy", [{"id": 1, "image": payload}])
+    by_id = _row_ids_by_id(table)
+    handles = table.fetch_blob_files("image", [by_id[1]])
+    assert len(handles) == 1
+    assert handles[0].read() == payload
+
+
+def test_fetch_blob_files_null_alignment():
+    table = _blob_table(
+        "lazy_nulls",
+        [{"id": 1, "image": b"here"}, {"id": 2, "image": None}],
+    )
+    by_id = _row_ids_by_id(table)
+    handles = table.fetch_blob_files("image", [by_id[2], by_id[1]])
+    assert len(handles) == 2
+    assert handles[0] is None
+    assert handles[1].read() == b"here"
+
+
+def test_fetch_blobs_rejects_non_blob_column():
+    table = _blob_table("reject", [{"id": 1, "image": b"x"}])
+    with pytest.raises(ValueError, match="not a blob column"):
+        table.fetch_blobs("id", [0])
+
+
+def test_legacy_v1_query_omits_auto_row_id():
+    table = _legacy_v1_table("legacy_v1")
+    hits = table.search().select(["legacy"]).limit(10).to_arrow()
+    assert "_rowid" not in hits.column_names
+
+
+def test_fetch_blobs_rejects_legacy_v1_column():
+    table = _legacy_v1_table("legacy_fetch")
+    with pytest.raises(ValueError, match="legacy blob column.*blob v2"):
+        table.fetch_blobs("legacy", [0])
+
+
+@pytest.mark.asyncio
+async def test_async_fetch_blob_files_lazy_read():
+    db = await lancedb.connect_async("memory:///")
+    schema = pa.schema([pa.field("id", pa.int64()), lancedb.blob("image")])
+    table = await db.create_table("async_lazy", schema=schema)
+    payload = b"async-lazy" * 100
+    await table.add([{"id": 1, "image": payload}])
+    hits = (
+        await table.query().select({"image_alias": "image"}).limit(10).to_arrow()
+    ).combine_chunks()
+    assert "_rowid" not in hits.column_names
+    handles = await table.fetch_blob_files("image", hits)
+    assert len(handles) == 1
+    assert await handles[0].aread() == payload
+
+
+def test_fetch_blobs_from_query_result_without_row_id_raises():
+    table = _blob_table("no_rowid", [{"id": 1, "image": b"x"}])
+    hits = table.search().select(["id"]).to_arrow()
+    assert "_rowid" not in hits.column_names
+    with pytest.raises(ValueError, match="_rowid"):
+        table.fetch_blobs("image", hits)
+
+
+_HYBRID_BLOB_SCHEMA = pa.schema(
+    [
+        pa.field("id", pa.int64()),
+        pa.field("text", pa.utf8()),
+        pa.field("vector", pa.list_(pa.float32(), list_size=2)),
+        lancedb.blob("image"),
+    ]
+)
+_HYBRID_BLOB_ROWS = [
+    {"id": 1, "text": "hello alpha", "vector": [1.0, 0.0], "image": b"alpha"},
+    {"id": 2, "text": "hello beta", "vector": [0.9, 0.1], "image": b"beta"},
+    {"id": 3, "text": "other", "vector": [0.0, 1.0], "image": b"other"},
+]
+
+
+def _hybrid_blob_table(db):
+    table = db.create_table("hybrid_blob_fetch", schema=_HYBRID_BLOB_SCHEMA)
+    table.add(_HYBRID_BLOB_ROWS)
+    table.create_index("text", config=FTS(with_position=False))
+    return table
+
+
+async def _hybrid_blob_table_async(db):
+    table = await db.create_table("hybrid_blob_fetch_async", schema=_HYBRID_BLOB_SCHEMA)
+    await table.add(_HYBRID_BLOB_ROWS)
+    await table.create_index("text", config=FTS(with_position=False))
+    return table
+
+
+def test_blob_v2_hybrid_fetch_blobs():
+    table = _hybrid_blob_table(lancedb.connect("memory:///"))
+    hits = (
+        table.search(query_type="hybrid")
+        .vector([1.0, 0.0])
+        .text("hello")
+        .select(["id", "image"])
+        .limit(2)
+        .to_arrow()
+    )
+
+    assert "_rowid" not in hits.column_names
+    assert "_lance_row_id" in hits.schema.field("image").type.names
+    blobs = table.fetch_blobs("image", hits)
+    assert {blobs[i].as_py() for i in range(len(blobs))} == {b"alpha", b"beta"}
+
+
+@pytest.mark.asyncio
+async def test_blob_v2_hybrid_fetch_blobs_async():
+    db = await lancedb.connect_async("memory:///hybrid_blob_fetch_async")
+    table = await _hybrid_blob_table_async(db)
+    hits = await (
+        table.query()
+        .nearest_to([1.0, 0.0])
+        .nearest_to_text("hello")
+        .select(["id", "image"])
+        .limit(2)
+        .to_arrow()
+    )
+
+    assert "_rowid" not in hits.column_names
+    assert "_lance_row_id" in hits.schema.field("image").type.names
+    blobs = await table.fetch_blobs("image", hits)
+    assert {blobs[i].as_py() for i in range(len(blobs))} == {b"alpha", b"beta"}
+
+
+def test_blob_file_seek_read_and_read_range():
+    payload = _identifiable_payload(1024)
+    table = _blob_table("seek_read", [{"id": 1, "image": payload}])
+    by_id = _row_ids_by_id(table)
+    handle = table.fetch_blob_files("image", [by_id[1]])[0]
+
+    assert handle.seek(100) == 100
+    assert handle.read(16) == payload[100:116]
+    handle.seek(100)
+    assert handle.read_range(500, 8) == payload[500:508]
+    assert handle.tell() == 100
+
+    with pytest.raises(ValueError, match="whence"):
+        handle.seek(0, 99)
+
+
+def test_fetch_blob_files_from_query_partial_read():
+    payload = _identifiable_payload(65536)
+    table = _blob_table("query_partial", [{"id": 1, "image": payload}])
+    hits = table.search().select(["id", "image"]).limit(1).to_arrow()
+    assert "_rowid" not in hits.column_names
+
+    handle = table.fetch_blob_files("image", hits)[0]
+    assert handle.size() == 65536
+    assert handle.read_range(0, 128) == payload[:128]
+    assert handle.tell() == 0
+    assert handle.seek(40000) == 40000
+    assert handle.read(16) == payload[40000:40016]
+
+
+def test_blob_file_buffered_reader():
+    payload = _identifiable_payload(4096)
+    table = _blob_table("buffered_reader", [{"id": 1, "image": payload}])
+    hits = table.search().select(["id", "image"]).limit(1).to_arrow()
+    handle = table.fetch_blob_files("image", hits)[0]
+    reader = io.BufferedReader(handle)
+    assert reader.read(8) == payload[:8]
+    assert reader.read(8) == payload[8:16]
+    assert reader.read() == payload[16:]
+
+
+def test_fetch_blob_files_cross_fragment_nulls_and_dups():
+    db = lancedb.connect("memory:///")
+    schema = pa.schema([pa.field("id", pa.int64()), lancedb.blob("image")])
+    table = db.create_table("cross_fragment", schema=schema)
+    table.add([{"id": 1, "image": b"alpha"}])
+    table.add([{"id": 2, "image": None}, {"id": 3, "image": b"beta"}])
+
+    by_id = _row_ids_by_id(table)
+    request = [by_id[3], by_id[2], by_id[1], by_id[3]]
+    handles = table.fetch_blob_files("image", request)
+    assert len(handles) == 4
+    assert handles[1] is None
+    assert handles[0].read() == b"beta"
+    assert handles[2].read() == b"alpha"
+    assert handles[3].seek(1) == 1
+    assert handles[3].read() == b"eta"
+
+
+def test_blob_file_pyav_decode_seek(tmp_path):
+    av = pytest.importorskip("av")
+    import fractions
+
+    clip = tmp_path / "clip.mp4"
+    with av.open(str(clip), mode="w") as container:
+        stream = container.add_stream("mpeg4", rate=5)
+        stream.width, stream.height, stream.pix_fmt = 32, 32, "yuv420p"
+        stream.time_base = fractions.Fraction(1, 5)
+        for pts in range(5):
+            frame = av.VideoFrame(32, 32, "yuv420p")
+            frame.pts = pts
+            container.mux(stream.encode(frame))
+        container.mux(stream.encode(None))
+
+    table = _blob_table("pyav", [{"id": 1, "image": clip.read_bytes()}])
+    hits = table.search().select(["image"]).limit(1).to_arrow()
+    handle = table.fetch_blob_files("image", hits)[0]
+
+    with av.open(handle) as container:
+        stream = container.streams.video[0]
+        container.seek(0)
+        assert next(container.decode(stream)) is not None
+
+
+def test_blob_v2_hybrid_fetch_blob_files_seek():
+    table = _hybrid_blob_table(lancedb.connect("memory:///"))
+    hits = (
+        table.search(query_type="hybrid")
+        .vector([1.0, 0.0])
+        .text("hello")
+        .select(["id", "image"])
+        .limit(2)
+        .to_arrow()
+    )
+    assert "_rowid" not in hits.column_names
+
+    handles = table.fetch_blob_files("image", hits)
+    assert len(handles) == 2
+    assert {handle.read_range(0, 2) for handle in handles} == {b"al", b"be"}
+    first = handles[0]
+    assert first.seek(1) == 1
+    assert first.read(2) in {b"lp", b"et"}
+
+
+def test_blob_file_header_sniff_from_search():
+    payload = b"%PDF-1.7\n" + bytes(4096)
+    table = _blob_table("header_sniff", [{"id": 1, "image": payload}])
+    hits = table.search().select(["id", "image"]).limit(1).to_arrow()
+    handle = table.fetch_blob_files("image", hits)[0]
+    assert handle.read_range(0, 4) == b"%PDF"
+    assert handle.tell() == 0
+
+
+def test_blob_file_multiple_handles_independent_cursors():
+    table = _blob_table(
+        "multi_handle",
+        [{"id": 1, "image": b"first-payload"}, {"id": 2, "image": b"second-payload"}],
+    )
+    by_id = _row_ids_by_id(table)
+    first, second = table.fetch_blob_files("image", [by_id[1], by_id[2]])
+    assert first.seek(6) == 6
+    assert second.tell() == 0
+    assert first.read(7) == b"payload"
+    assert second.read(6) == b"second"
+
+
+def test_fetch_blob_files_nested_path_seek():
+    db = lancedb.connect("memory:///")
+    info = pa.StructArray.from_arrays(
+        [
+            pa.array(["first", "second"], type=pa.string()),
+            _blob_array("blob", [b"nested-alpha", b"nested-beta"]),
+        ],
+        names=["name", "blob"],
+    )
+    data = pa.Table.from_arrays(
+        [pa.array([1, 2], type=pa.int64()), info],
+        names=["id", "info"],
+    )
+    table = db.create_table("nested_seek", data=data)
+    by_id = _row_ids_by_id(table)
+    handle = table.fetch_blob_files("info.blob", [by_id[2]])[0]
+    assert handle.seek(7) == 7
+    assert handle.read() == b"beta"
+
+
+def test_fetch_blobs_survives_sort_after_query():
+    table = _blob_table(
+        "sort_survives",
+        [{"id": i, "image": f"payload-{i}".encode()} for i in range(5)],
+    )
+    hits = table.search().select(["id", "image"]).to_arrow()
+    sort_idx = pc.sort_indices(hits["id"], sort_keys=[("id", "descending")])
+    sorted_hits = hits.take(sort_idx)
+
+    blobs = table.fetch_blobs("image", sorted_hits)
+    expected = [f"payload-{i}".encode() for i in sorted_hits["id"].to_pylist()]
+    assert [blobs[i].as_py() for i in range(len(blobs))] == expected
+
+
+def test_fetch_blobs_survives_filter_and_sort_after_query():
+    table = _blob_table(
+        "filter_sort_survives",
+        [{"id": i, "image": f"payload-{i}".encode()} for i in range(5)],
+    )
+    hits = table.search().select(["id", "image"]).to_arrow()
+    filtered = hits.filter(pc.field("id") >= 2)
+    sort_idx = pc.sort_indices(filtered["id"], sort_keys=[("id", "descending")])
+    filtered_sorted = filtered.take(sort_idx)
+
+    blobs = table.fetch_blobs("image", filtered_sorted)
+    expected = [f"payload-{i}".encode() for i in filtered_sorted["id"].to_pylist()]
+    assert [blobs[i].as_py() for i in range(len(blobs))] == expected
+
+
+def test_fetch_blob_files_survives_sort_after_query():
+    table = _blob_table(
+        "lazy_sort_survives",
+        [{"id": i, "image": f"payload-{i}".encode()} for i in range(5)],
+    )
+    hits = table.search().select(["id", "image"]).to_arrow()
+    sort_idx = pc.sort_indices(hits["id"], sort_keys=[("id", "descending")])
+    sorted_hits = hits.take(sort_idx)
+
+    handles = table.fetch_blob_files("image", sorted_hits)
+    expected = [f"payload-{i}".encode() for i in sorted_hits["id"].to_pylist()]
+    assert [handle.read() for handle in handles] == expected
+
+
+def test_fetch_blobs_nested_path_survives_sort_after_query():
+    db = lancedb.connect("memory:///")
+    values = [f"payload-{i}".encode() for i in range(4)]
+    info = pa.StructArray.from_arrays(
+        [pa.array(["row"] * 4, type=pa.string()), _blob_array("blob", values)],
+        names=["name", "blob"],
+    )
+    data = pa.Table.from_arrays(
+        [pa.array(range(4), type=pa.int64()), info],
+        names=["id", "info"],
+    )
+    table = db.create_table("nested_sort_survives", data=data)
+
+    hits = table.search().to_arrow()
+    sort_idx = pc.sort_indices(hits["id"], sort_keys=[("id", "descending")])
+    sorted_hits = hits.take(sort_idx)
+
+    blobs = table.fetch_blobs("info.blob", sorted_hits)
+    expected = [f"payload-{i}".encode() for i in sorted_hits["id"].to_pylist()]
+    assert [blobs[i].as_py() for i in range(len(blobs))] == expected
+
+
+def _identifiable_payload(size: int) -> bytes:
+    block = 256
+    return b"".join(bytes([i % 256]) * block for i in range(size // block))

python/python/tests/test_embeddings.py

@@ -2,6 +2,7 @@
 # SPDX-FileCopyrightText: Copyright The LanceDB Authors
 
 import os
+import pickle
 from typing import List, Optional, Union
 from unittest.mock import MagicMock, patch
 
@@ -591,6 +592,22 @@ def test_openai_no_retry_on_401(mock_sleep):
     assert mock_sleep.call_count == 0
 
 
+def test_ollama_embeddings_pickle():
+    """OllamaEmbeddings must pickle even after the cached client is created."""
+    registry = get_registry()
+    model = registry.get("ollama").create(name="nomic-embed-text")
+
+    # Simulate accessing the cached client, which stores it on the instance.
+    model.__dict__["_ollama_client"] = MagicMock()
+
+    pickled = pickle.dumps(model)
+    restored = pickle.loads(pickled)
+
+    assert restored.name == "nomic-embed-text"
+    assert restored.host == "http://localhost:11434"
+    assert "_ollama_client" not in restored.__dict__
+
+
 def test_url_retrieve_downloads_image():
     """
     Embedding functions like open-clip, siglip, and jinaai use url_retrieve()

python/python/tests/test_fts.py

@@ -786,6 +786,97 @@ def test_language(mem_db: DBConnection):
     assert len(results) == 0
 
 
+def test_tokenize_uses_simple_index_tokenizer(mem_db: DBConnection):
+    data = pa.table({"text": ["Running in cafés"], "other": ["Running in cafés"]})
+    table = mem_db.create_table("test_tokenize", data=data)
+    table.create_index("text", config=FTS(base_tokenizer="simple"))
+
+    tokens = table.tokenize("Running in cafés", column="text")
+
+    assert [(token.text, token.position) for token in tokens] == [
+        ("run", 0),
+        ("cafe", 2),
+    ]
+
+
+def test_tokenize_uses_icu_index_tokenizer_by_name(mem_db: DBConnection):
+    data = pa.table({"text": ["Hello, こんにちは世界!"]})
+    table = mem_db.create_table("test_tokenize_icu", data=data)
+    table.create_index(
+        "text",
+        config=FTS(
+            base_tokenizer="icu",
+            stem=False,
+            remove_stop_words=False,
+        ),
+        name="text_icu_idx",
+    )
+
+    tokens = table.tokenize("Hello, こんにちは世界!", index_name="text_icu_idx")
+
+    assert [(token.text, token.position) for token in tokens] == [
+        ("hello", 0),
+        ("こんにちは", 1),
+        ("世界", 2),
+    ]
+
+
+def test_tokenize_requires_one_selector(mem_db: DBConnection):
+    data = pa.table({"text": ["hello world"]})
+    table = mem_db.create_table("test_tokenize_selector", data=data)
+    table.create_index("text", config=FTS(), name="text_idx")
+
+    with pytest.raises(ValueError, match="Specify exactly one"):
+        table.tokenize("hello")
+
+    with pytest.raises(ValueError, match="Specify exactly one"):
+        table.tokenize("hello", column="text", index_name="text_idx")
+
+
+def test_tokenize_requires_fts_index(mem_db: DBConnection):
+    data = pa.table({"text": ["hello world"]})
+    table = mem_db.create_table("test_tokenize_no_index", data=data)
+
+    with pytest.raises(ValueError, match="does not have a full text search index"):
+        table.tokenize("hello", column="text")
+
+
+@pytest.mark.asyncio
+async def test_tokenize_async(async_table):
+    await async_table.create_index("text", config=FTS())
+
+    tokens = await async_table.tokenize("Running in cafés", column="text")
+
+    assert [(token.text, token.position) for token in tokens] == [
+        ("run", 0),
+        ("cafe", 2),
+    ]
+
+
+def test_tokenize_uses_explicit_simple_tokenizer():
+    tokens = ldb.tokenize("Running in cafés", base_tokenizer="simple")
+
+    assert [(token.text, token.position) for token in tokens] == [
+        ("run", 0),
+        ("cafe", 2),
+    ]
+
+
+def test_tokenize_uses_explicit_icu_tokenizer():
+    tokens = ldb.tokenize(
+        "Hello, こんにちは世界!",
+        base_tokenizer="icu",
+        stem=False,
+        remove_stop_words=False,
+    )
+
+    assert [(token.text, token.position) for token in tokens] == [
+        ("hello", 0),
+        ("こんにちは", 1),
+        ("世界", 2),
+    ]
+
+
 def test_fts_on_list(mem_db: DBConnection):
     data = pa.table(
         {
@@ -1084,6 +1175,84 @@ def test_fts_query_to_json():
     assert json_str == expected
 
 
+def test_fts_phrase_query_is_preserved_in_query_object():
+    query = LanceFtsQueryBuilder(mock.Mock(), "puppy runs").phrase_query()
+
+    query_object = query.to_query_object()
+
+    assert query_object.full_text_query.query == '"puppy runs"'
+
+
+def test_fts_phrase_query_execution_preserves_user_text():
+    table = mock.Mock()
+    table.schema = pa.schema([])
+    table._execute_query.return_value = pa.table({"text": ["result"]}).to_reader()
+
+    class CapturingReranker:
+        score = "relevance"
+
+        def __init__(self):
+            self.queries = []
+
+        def rerank_fts(self, query, results):
+            self.queries.append(query)
+            return results.append_column("_relevance_score", [[1.0]])
+
+    reranker = CapturingReranker()
+    query = (
+        LanceFtsQueryBuilder(table, "puppy runs")
+        .phrase_query()
+        .with_row_id(False)
+        .rerank(reranker)
+    )
+
+    query.to_arrow()
+
+    backend_query = table._execute_query.call_args.args[0]
+    assert (
+        backend_query.full_text_query.query,
+        reranker.queries,
+        query._query,
+    ) == ('"puppy runs"', ["puppy runs"], "puppy runs")
+
+
+def test_fts_phrase_query_false_preserves_string():
+    query = LanceFtsQueryBuilder(mock.Mock(), "puppy runs").phrase_query(False)
+
+    query_object = query.to_query_object()
+
+    assert query_object.full_text_query.query == "puppy runs"
+
+
+def test_fts_phrase_query_preserves_fully_quoted_string():
+    query = LanceFtsQueryBuilder(mock.Mock(), '"puppy runs"').phrase_query()
+
+    query_object = query.to_query_object()
+
+    assert query_object.full_text_query.query == '"puppy runs"'
+
+
+def test_fts_phrase_query_preserves_structured_phrase_query():
+    phrase_query = PhraseQuery("puppy runs", "text")
+    query = LanceFtsQueryBuilder(mock.Mock(), phrase_query).phrase_query()
+
+    query_object = query.to_query_object()
+
+    assert query_object.full_text_query.query == phrase_query
+
+
+def test_fts_phrase_query_rejects_other_structured_queries():
+    query = LanceFtsQueryBuilder(
+        mock.Mock(), MatchQuery("puppy", "text")
+    ).phrase_query()
+
+    with pytest.raises(
+        TypeError,
+        match=r"phrase_query\(\) requires a string or PhraseQuery, got MatchQuery",
+    ):
+        query.to_query_object()
+
+
 def test_fts_fast_search(table):
     table.create_fts_index("text")
 

python/python/tests/test_gemini.py

@@ -0,0 +1,183 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+"""Unit tests for GeminiText embedding function."""
+
+import sys
+from unittest.mock import MagicMock, patch
+
+# Mock google.genai modules before they are imported by gemini_text.py
+mock_google = MagicMock()
+mock_genai = MagicMock()
+mock_types = MagicMock()
+
+mock_google.genai = mock_genai
+mock_genai.types = mock_types
+
+sys.modules["google"] = mock_google
+sys.modules["google.genai"] = mock_genai
+sys.modules["google.genai.types"] = mock_types
+
+import pytest  # noqa: E402
+import numpy as np  # noqa: E402
+from lancedb.embeddings import get_registry  # noqa: E402
+from lancedb import __version__  # noqa: E402
+
+
+class TestGeminiText:
+    """Tests for GeminiText model registration, configuration, and execution."""
+
+    @pytest.fixture(autouse=True)
+    def setup_mocks(self):
+        """Set up standard mocks for google-genai Client and Config."""
+        # Reset mocks
+        mock_genai.reset_mock()
+        mock_types.reset_mock()
+
+        self.mock_client = MagicMock()
+        mock_genai.Client.return_value = self.mock_client
+
+        # Mock response for embed_content
+        self.mock_embedding_1 = MagicMock()
+        self.mock_embedding_1.values = [0.1] * 768
+        self.mock_embedding_2 = MagicMock()
+        self.mock_embedding_2.values = [0.2] * 768
+
+        self.mock_response = MagicMock()
+        self.mock_response.embeddings = [self.mock_embedding_1, self.mock_embedding_2]
+        self.mock_client.models.embed_content.return_value = self.mock_response
+
+    def test_gemini_registered(self):
+        """Test that gemini-text is registered in the embedding function registry."""
+        registry = get_registry()
+        assert registry.get("gemini-text") is not None
+
+    def test_client_init_headers(self):
+        """Test that Client is initialized with the partner-attribution header."""
+        with patch.dict("os.environ", {"GOOGLE_API_KEY": "test-key"}):
+            with patch("lancedb.embeddings.gemini_text.attempt_import_or_raise"):
+                registry = get_registry()
+                func = registry.get("gemini-text").create()
+
+                # Access the client property to trigger initialization
+                _ = func.client
+
+                mock_genai.Client.assert_called_once_with(
+                    api_key="test-key",
+                    http_options={
+                        "headers": {
+                            "x-goog-api-client": f"lancedb/{__version__}",
+                        }
+                    },
+                )
+
+    def test_generate_embeddings_batched(self):
+        """Test that multiple texts are sent in a single batched API request."""
+        with patch.dict("os.environ", {"GOOGLE_API_KEY": "test-key"}):
+            with patch("lancedb.embeddings.gemini_text.attempt_import_or_raise"):
+                registry = get_registry()
+                func = registry.get("gemini-text").create()
+
+                texts = ["hello", "world"]
+                embeddings = func.generate_embeddings(texts)
+
+                # Check embed_content was called exactly once
+                self.mock_client.models.embed_content.assert_called_once()
+
+                # Verify call arguments
+                call_kwargs = self.mock_client.models.embed_content.call_args.kwargs
+                assert call_kwargs["model"] == "gemini-embedding-001"
+                assert len(call_kwargs["contents"]) == 2
+                assert call_kwargs["contents"][0] == {"parts": [{"text": "hello"}]}
+                assert call_kwargs["contents"][1] == {"parts": [{"text": "world"}]}
+
+                # Verify returns are correct numpy arrays
+                assert len(embeddings) == 2
+                assert isinstance(embeddings[0], np.ndarray)
+                assert embeddings[0].shape == (768,)
+                assert np.allclose(embeddings[0], 0.1)
+                assert np.allclose(embeddings[1], 0.2)
+
+    def test_generate_embeddings_retrieval_document(self):
+        """Test that retrieval_document task type prepends the document title part."""
+        with patch.dict("os.environ", {"GOOGLE_API_KEY": "test-key"}):
+            with patch("lancedb.embeddings.gemini_text.attempt_import_or_raise"):
+                registry = get_registry()
+                func = registry.get("gemini-text").create(
+                    source_task_type="retrieval_document"
+                )
+
+                texts = ["doc text"]
+
+                # We need mock to return only 1 embedding since we only pass 1 text
+                mock_embedding = MagicMock()
+                mock_embedding.values = [0.3] * 768
+                self.mock_response.embeddings = [mock_embedding]
+
+                embeddings = func.generate_embeddings(
+                    texts, task_type="retrieval_document"
+                )
+
+                # Check call arguments for retrieval_document
+                call_kwargs = self.mock_client.models.embed_content.call_args.kwargs
+                assert call_kwargs["contents"][0] == {
+                    "parts": [{"text": "Embedding of a document"}, {"text": "doc text"}]
+                }
+                mock_types.EmbedContentConfig.assert_called_once_with(
+                    output_dimensionality=768, task_type="RETRIEVAL_DOCUMENT"
+                )
+
+                assert len(embeddings) == 1
+                assert np.allclose(embeddings[0], 0.3)
+
+    def test_custom_dimension(self):
+        """Test that custom dimension (dim) can be configured and passed to config."""
+        with patch.dict("os.environ", {"GOOGLE_API_KEY": "test-key"}):
+            with patch("lancedb.embeddings.gemini_text.attempt_import_or_raise"):
+                registry = get_registry()
+                func = registry.get("gemini-text").create(dim=3072)
+
+                assert func.ndims() == 3072
+
+                texts = ["hello"]
+                mock_embedding = MagicMock()
+                mock_embedding.values = [0.5] * 3072
+                self.mock_response.embeddings = [mock_embedding]
+
+                _ = func.generate_embeddings(texts)
+
+                mock_types.EmbedContentConfig.assert_called_once_with(
+                    output_dimensionality=3072
+                )
+
+    def test_generate_embeddings_chunked(self):
+        """Test that generate_embeddings chunks texts into groups of 100."""
+        with patch.dict("os.environ", {"GOOGLE_API_KEY": "test-key"}):
+            with patch("lancedb.embeddings.gemini_text.attempt_import_or_raise"):
+                registry = get_registry()
+                func = registry.get("gemini-text").create()
+
+                # Passing 250 texts should make 3 calls (100, 100, 50)
+                texts = [f"text_{i}" for i in range(250)]
+
+                # Mock client response to return correct number of embeddings per chunk
+                def mock_embed_side_effect(model, contents, config=None):
+                    mock_resp = MagicMock()
+                    mock_embeddings = []
+                    for _ in contents:
+                        emb = MagicMock()
+                        # Each embedding is length 768
+                        emb.values = [0.1] * 768
+                        mock_embeddings.append(emb)
+                    mock_resp.embeddings = mock_embeddings
+                    return mock_resp
+
+                self.mock_client.models.embed_content.side_effect = (
+                    mock_embed_side_effect
+                )
+
+                embeddings = func.generate_embeddings(texts)
+
+                # embed_content should be called 3 times
+                assert self.mock_client.models.embed_content.call_count == 3
+                assert len(embeddings) == 250

python/python/tests/test_hybrid_query.py

@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileCopyrightText: Copyright The LanceDB Authors
 
+from unittest import mock
+
 import lancedb
 
 from lancedb.query import LanceHybridQueryBuilder
@@ -139,6 +141,20 @@ def test_hybrid_query_distance_range(sync_table: Table):
             assert 0.2 <= dist.as_py() <= 0.5
 
 
+def test_hybrid_query_applies_zero_upper_distance_bound(sync_table: Table):
+    result = (
+        sync_table.search(query_type="hybrid")
+        .vector([0.0, 0.4])
+        .text("elephant")
+        .distance_range(upper_bound=0.0)
+        .rerank(RRFReranker(return_score="all"))
+        .limit(4)
+        .to_arrow()
+    )
+
+    assert len(result) == 0
+
+
 @pytest.mark.asyncio
 async def test_hybrid_query_distance_range_async(table: AsyncTable):
     reranker = RRFReranker(return_score="all")
@@ -177,6 +193,23 @@ async def test_analyze_plan(table: AsyncTable):
     assert "metrics=" in res
 
 
+def test_hybrid_phrase_query_is_preserved_in_analyze_plan():
+    table = mock.Mock()
+    analyzed_queries = []
+    table._analyze_plan.side_effect = lambda query: analyzed_queries.append(query) or ""
+
+    (
+        LanceHybridQueryBuilder(table)
+        .vector([0.1, 0.2])
+        .text("puppy runs")
+        .phrase_query()
+        .analyze_plan()
+    )
+
+    assert len(analyzed_queries) == 2
+    assert analyzed_queries[1].full_text_query.query == '"puppy runs"'
+
+
 @pytest.fixture
 def table_with_id(tmpdir_factory) -> Table:
     tmp_path = str(tmpdir_factory.mktemp("data"))

python/python/tests/test_lsm_write_spec.py

@@ -11,6 +11,7 @@ import lancedb
 import pyarrow as pa
 import pytest
 from lancedb._lancedb import LsmWriteSpec
+from lancedb.index import BTree
 
 SCHEMA = pa.schema(
     [
@@ -136,3 +137,76 @@ def test_lsm_write_spec_identity_and_writer_config_defaults():
     s = s.with_writer_config_defaults({"durable_write": "false"})
     assert s.writer_config_defaults == {"durable_write": "false"}
     assert "durable_write" in repr(s)
+
+
+def test_get_lsm_write_spec(tmp_path):
+    _db, table = _make_table(tmp_path)
+    table.set_unenforced_primary_key("id")
+
+    # None when nothing is installed.
+    assert table.get_lsm_write_spec() is None
+
+    # A real scalar index is needed to name it as a maintained index.
+    table.create_index("id", config=BTree())
+    idx_name = table.list_indices()[0].name
+
+    # Bucket spec round-trips, including maintained indexes and writer config
+    # defaults.
+    table.set_lsm_write_spec(
+        LsmWriteSpec.bucket("id", 4)
+        .with_maintained_indexes([idx_name])
+        .with_writer_config_defaults({"durable_write": "false"})
+    )
+    spec = table.get_lsm_write_spec()
+    assert spec is not None
+    assert spec.spec_type == "bucket"
+    assert spec.column == "id"
+    assert spec.num_buckets == 4
+    assert spec.maintained_indexes == [idx_name]
+    assert spec.writer_config_defaults == {"durable_write": "false"}
+
+    # After unset, None again.
+    table.unset_lsm_write_spec()
+    assert table.get_lsm_write_spec() is None
+
+    # Identity round-trips (column recovered from the schema).
+    table.set_lsm_write_spec(LsmWriteSpec.identity("id"))
+    spec = table.get_lsm_write_spec()
+    assert spec.spec_type == "identity"
+    assert spec.column == "id"
+    table.unset_lsm_write_spec()
+
+    # Unsharded round-trips (no routing column).
+    table.set_lsm_write_spec(LsmWriteSpec.unsharded())
+    spec = table.get_lsm_write_spec()
+    assert spec.spec_type == "unsharded"
+    assert spec.column is None
+
+
+@pytest.mark.asyncio
+async def test_async_get_lsm_write_spec(tmp_path):
+    db = await lancedb.connect_async(
+        tmp_path, read_consistency_interval=timedelta(seconds=0)
+    )
+    table = await db.create_table(
+        "t",
+        pa.RecordBatchReader.from_batches(SCHEMA, [_batch(["seed"], [0])]),
+    )
+
+    assert await table.get_lsm_write_spec() is None
+
+    # A real scalar index is needed to name it as a maintained index.
+    await table.create_index("id", config=BTree())
+    idx_name = (await table.list_indices())[0].name
+
+    await table.set_lsm_write_spec(
+        LsmWriteSpec.bucket("id", 8).with_maintained_indexes([idx_name])
+    )
+    spec = await table.get_lsm_write_spec()
+    assert spec is not None
+    assert spec.spec_type == "bucket"
+    assert spec.column == "id"
+    assert spec.num_buckets == 8
+    assert spec.maintained_indexes == [idx_name]
+    await table.unset_lsm_write_spec()
+    assert await table.get_lsm_write_spec() is None

python/python/tests/test_permutation.py

@@ -935,14 +935,41 @@ def test_transform_fn(mem_db):
     try:
         import torch
 
-        torch_result = list(
-            permutation.with_format("torch").iter(10, skip_last_batch=False)
+        # "torch" returns a list of per-row dicts. Default DataLoader collate
+        # stacks the per-row dicts back into a dict of batched tensors.
+        torch_perm = permutation.with_format("torch")
+        torch_batch = list(torch_perm.iter(10, skip_last_batch=False))[0]
+        assert isinstance(torch_batch, list)
+        assert len(torch_batch) == 10
+        assert isinstance(torch_batch[0], dict)
+        assert set(torch_batch[0].keys()) == {"id", "value"}
+        assert isinstance(torch_batch[0]["id"], torch.Tensor)
+        assert torch_batch[0]["id"].dtype == torch.int64
+
+        rows = torch_perm.__getitems__([0, 1, 2])
+        assert isinstance(rows, list)
+        assert len(rows) == 3
+        assert isinstance(rows[0], dict)
+        assert set(rows[0].keys()) == {"id", "value"}
+        assert isinstance(rows[0]["id"], torch.Tensor)
+
+        # "torch_row" returns a list of tensors, one per row.
+        torch_rows = list(
+            permutation.with_format("torch_row").iter(10, skip_last_batch=False)
         )[0]
-        assert isinstance(torch_result, list)
-        assert len(torch_result) == 10
-        assert isinstance(torch_result[0], torch.Tensor)
-        assert torch_result[0].shape == (2,)
-        assert torch_result[0].dtype == torch.int64
+        assert isinstance(torch_rows, list)
+        assert len(torch_rows) == 10
+        assert isinstance(torch_rows[0], torch.Tensor)
+        assert torch_rows[0].shape == (2,)
+        assert torch_rows[0].dtype == torch.int64
+
+        # "torch_col" stacks columns into a single 2D tensor.
+        torch_col = list(
+            permutation.with_format("torch_col").iter(10, skip_last_batch=False)
+        )[0]
+        assert isinstance(torch_col, torch.Tensor)
+        assert torch_col.shape == (2, 10)
+        assert torch_col.dtype == torch.int64
     except ImportError:
         # Skip check if torch is not installed
         pass

python/python/tests/test_query.py

@@ -11,6 +11,7 @@ import lancedb
 from lancedb.db import AsyncConnection
 from lancedb.embeddings.base import TextEmbeddingFunction
 from lancedb.embeddings.registry import get_registry, register
+from lancedb.expr import col
 from lancedb.index import FTS, IvfPq
 import lancedb.pydantic
 import numpy as np
@@ -63,11 +64,71 @@ def _blob_query_data():
     )
 
 
+def _create_blob_v2_query_table(db, name):
+    schema = pa.schema(
+        [
+            pa.field("id", pa.int64()),
+            pa.field("tag", pa.utf8()),
+            pa.field("vector", pa.list_(pa.float32(), list_size=2)),
+            lancedb.blob("blob"),
+        ]
+    )
+    table = db.create_table(name, schema=schema)
+    table.add(
+        [
+            {"id": 1, "tag": "drop", "vector": [1.0, 0.0], "blob": b"one"},
+            {"id": 2, "tag": "keep", "vector": [2.0, 0.0], "blob": b"two"},
+            {"id": 3, "tag": "keep", "vector": [3.0, 0.0], "blob": b"three"},
+            {"id": 4, "tag": "keep", "vector": [4.0, 0.0], "blob": b"four"},
+        ]
+    )
+    return table
+
+
+async def _create_blob_v2_query_table_async(db, name):
+    schema = pa.schema(
+        [
+            pa.field("id", pa.int64()),
+            pa.field("tag", pa.utf8()),
+            pa.field("vector", pa.list_(pa.float32(), list_size=2)),
+            lancedb.blob("blob"),
+        ]
+    )
+    table = await db.create_table(name, schema=schema)
+    await table.add(
+        [
+            {"id": 1, "tag": "drop", "vector": [1.0, 0.0], "blob": b"one"},
+            {"id": 2, "tag": "keep", "vector": [2.0, 0.0], "blob": b"two"},
+            {"id": 3, "tag": "keep", "vector": [3.0, 0.0], "blob": b"three"},
+            {"id": 4, "tag": "keep", "vector": [4.0, 0.0], "blob": b"four"},
+        ]
+    )
+    return table
+
+
 def _assert_lazy_blob(value, expected: bytes):
     assert hasattr(value, "readall")
     assert value.readall() == expected
 
 
+def _assert_blob_bytes_projection(df):
+    assert df["id_alias"].tolist() == [3, 4]
+    assert df["payload"].tolist() == [b"three", b"four"]
+    assert df["double_id"].tolist() == [6, 8]
+
+
+def _blob_query_table(db, name, blob_schema):
+    if blob_schema == "v1":
+        return db.create_table(name, _blob_query_data())
+    return _create_blob_v2_query_table(db, name)
+
+
+async def _blob_query_table_async(db, name, blob_schema):
+    if blob_schema == "v1":
+        return await db.create_table(name, _blob_query_data())
+    return await _create_blob_v2_query_table_async(db, name)
+
+
 @pytest.fixture(scope="module")
 def table(tmpdir_factory) -> lancedb.table.Table:
     tmp_path = str(tmpdir_factory.mktemp("data"))
@@ -235,10 +296,11 @@ def test_plain_scan_query_to_pandas_blob_modes(tmp_db, blob_mode):
         assert not hasattr(first, "readall")
 
 
-def test_plain_scan_query_to_pandas_blob_projection(tmp_db):
+@pytest.mark.parametrize("blob_schema", ["v1", "v2"])
+def test_plain_scan_query_to_pandas_blob_bytes_projection(tmp_db, blob_schema):
     pytest.importorskip("lance")
-    table = tmp_db.create_table(
-        "test_query_to_pandas_blob_projection", _blob_query_data()
+    table = _blob_query_table(
+        tmp_db, f"test_query_to_pandas_blob_{blob_schema}_bytes", blob_schema
     )
 
     df = (
@@ -250,9 +312,8 @@ def test_plain_scan_query_to_pandas_blob_projection(tmp_db):
         .to_pandas(blob_mode="bytes")
     )
 
-    assert df["id_alias"].tolist() == [3, 4]
-    assert df["payload"].tolist() == [b"three", b"four"]
-    assert df["double_id"].tolist() == [6, 8]
+    _assert_blob_bytes_projection(df)
+    assert "_rowid" not in df.columns
 
 
 @pytest.mark.parametrize("blob_mode", ["bytes", "descriptions"])
@@ -348,18 +409,6 @@ async def test_async_plain_scan_query_to_pandas_blob_projection(tmp_db_async):
     assert lazy_df["id"].tolist() == [1]
     _assert_lazy_blob(lazy_df["blob"].iloc[0], b"one")
 
-    bytes_df = await (
-        table.query()
-        .where("id >= 2")
-        .select({"id_alias": "id", "payload": "blob", "double_id": "id * 2"})
-        .limit(2)
-        .offset(1)
-        .to_pandas(blob_mode="bytes")
-    )
-    assert bytes_df["id_alias"].tolist() == [3, 4]
-    assert bytes_df["payload"].tolist() == [b"three", b"four"]
-    assert bytes_df["double_id"].tolist() == [6, 8]
-
     desc_df = await (
         table.query()
         .where("id = 1")
@@ -371,6 +420,31 @@ async def test_async_plain_scan_query_to_pandas_blob_projection(tmp_db_async):
     assert not hasattr(first, "readall")
 
 
+@pytest.mark.asyncio
+@pytest.mark.parametrize("blob_schema", ["v1", "v2"])
+async def test_async_plain_scan_query_to_pandas_blob_bytes_projection(
+    tmp_db_async, blob_schema
+):
+    pytest.importorskip("lance")
+    table = await _blob_query_table_async(
+        tmp_db_async,
+        f"test_async_query_to_pandas_blob_{blob_schema}_bytes",
+        blob_schema,
+    )
+
+    df = await (
+        table.query()
+        .where("id >= 2")
+        .select({"id_alias": "id", "payload": "blob", "double_id": "id * 2"})
+        .limit(2)
+        .offset(1)
+        .to_pandas(blob_mode="bytes")
+    )
+
+    _assert_blob_bytes_projection(df)
+    assert "_rowid" not in df.columns
+
+
 @pytest.mark.asyncio
 @pytest.mark.parametrize("blob_mode", ["bytes", "descriptions"])
 async def test_async_plain_scan_query_to_pandas_blob_mode_does_not_collect_arrow(
@@ -502,6 +576,18 @@ def test_with_row_id(table: lancedb.table.Table):
     assert rs["_rowid"].to_pylist() == [0, 1]
 
 
+def test_blob_v2_query_omits_auto_row_id(tmp_db):
+    table = _create_blob_v2_query_table(tmp_db, "test_blob_v2_omits_auto_rowid")
+
+    query_obj = table.search().select(["id", "blob"]).limit(2).to_query_object()
+    assert query_obj.with_row_id is None
+
+    rs = table.search().select(["id", "blob"]).limit(2).to_arrow()
+
+    assert "_rowid" not in rs.column_names
+    assert rs["id"].to_pylist() == [1, 2]
+
+
 def test_where_repeated_combines_with_and(table: lancedb.table.Table):
     # Calling where() more than once should AND the filters together instead of
     # silently replacing the previous one (regression test for #2649).
@@ -1946,3 +2032,39 @@ def test_fast_search(tmp_path):
     # 2. Fast Search -> Should NOT include "LanceScan" (Uses Index)
     plan = table.search(q).fast_search().explain_plan(True)
     assert "LanceScan" not in plan
+
+
+def test_blob_v2_with_row_id_bytes_pandas(tmp_db):
+    table = _create_blob_v2_query_table(tmp_db, "test_blob_v2_rowid_bytes_pandas")
+
+    df = (
+        table.search()
+        .with_row_id(True)
+        .select(["id", "blob"])
+        .to_pandas(blob_mode="bytes")
+    )
+
+    assert "_rowid" in df.columns
+    assert df["id"].tolist() == [1, 2, 3, 4]
+    assert df["blob"].tolist() == [b"one", b"two", b"three", b"four"]
+
+
+def test_blob_v2_expr_projection_stash(tmp_db):
+    table = _create_blob_v2_query_table(tmp_db, "test_blob_v2_expr_projection_stash")
+
+    hits = table.search().select({"blob_alias": col("blob")}).limit(2).to_arrow()
+
+    assert "_rowid" not in hits.column_names
+    assert "_lance_row_id" in hits.schema.field("blob_alias").type.names
+    blobs = table.fetch_blobs("blob", hits)
+    assert [blobs[i].as_py() for i in range(len(blobs))] == [b"one", b"two"]
+
+
+def test_blob_v2_to_batches_row_id(tmp_db):
+    table = _create_blob_v2_query_table(tmp_db, "test_blob_v2_to_batches_rowid")
+
+    hits = table.search().select(["id", "blob"]).limit(2).to_batches().read_all()
+
+    assert "_rowid" in hits.column_names
+    blobs = table.fetch_blobs("blob", hits)
+    assert [blobs[i].as_py() for i in range(len(blobs))] == [b"one", b"two"]

python/python/tests/test_rerankers.py

@@ -23,6 +23,7 @@ from lancedb.rerankers import (
     AnswerdotaiRerankers,
     VoyageAIReranker,
     MRRReranker,
+    WatsonxReranker,
 )
 from lancedb.table import LanceTable
 
@@ -727,3 +728,19 @@ def test_linear_combination_missing_fts_is_penalised():
         f"Document with FTS score (rowid 0, {scores[0]:.4f}) should beat "
         f"document with no FTS match (rowid 1, {scores[1]:.4f})"
     )
+
+
+@pytest.mark.skipif(
+    os.environ.get("WATSONX_API_KEY") is None
+    or (
+        os.environ.get("WATSONX_PROJECT_ID") is None
+        and os.environ.get("WATSONX_SPACE_ID") is None
+    ),
+    reason="WATSONX_API_KEY and one of WATSONX_PROJECT_ID / "
+    "WATSONX_SPACE_ID must be set",
+)
+def test_watsonx_reranker(tmp_path):
+    pytest.importorskip("ibm_watsonx_ai")
+    table, schema = get_test_table(tmp_path)
+    reranker = WatsonxReranker()
+    _run_test_reranker(reranker, table, "single player experience", None, schema)

python/python/tests/test_s3_bucket_dots.py

@@ -7,7 +7,8 @@ Tests for S3 bucket names containing dots.
 Related issue: https://github.com/lancedb/lancedb/issues/1898
 
 These tests validate the early error checking for S3 bucket names with dots.
-No actual S3 connection is made - validation happens before connection.
+When validation succeeds, eager namespace initialization may still fail later
+because these tests intentionally do not provide real S3 credentials.
 """
 
 import pytest
@@ -20,6 +21,13 @@ BUCKET_WITH_DOTS_AND_AWS_REGION = ("s3://my.bucket.name", {"aws_region": "us-eas
 BUCKET_WITHOUT_DOTS = "s3://my-bucket/path"
 
 
+def assert_not_rejected_for_bucket_dots(connect):
+    try:
+        connect()
+    except ValueError as err:
+        assert "contains dots" not in str(err)
+
+
 class TestS3BucketWithDotsSync:
     """Tests for connect()."""
 
@@ -27,19 +35,22 @@ class TestS3BucketWithDotsSync:
         with pytest.raises(ValueError, match="contains dots"):
             lancedb.connect(BUCKET_WITH_DOTS)
 
-    def test_bucket_with_dots_and_region_passes(self):
+    def test_bucket_with_dots_and_region_is_not_rejected(self):
         uri, opts = BUCKET_WITH_DOTS_AND_REGION
-        db = lancedb.connect(uri, storage_options=opts)
-        assert db is not None
+        assert_not_rejected_for_bucket_dots(
+            lambda: lancedb.connect(uri, storage_options=opts)
+        )
 
-    def test_bucket_with_dots_and_aws_region_passes(self):
+    def test_bucket_with_dots_and_aws_region_is_not_rejected(self):
         uri, opts = BUCKET_WITH_DOTS_AND_AWS_REGION
-        db = lancedb.connect(uri, storage_options=opts)
-        assert db is not None
+        assert_not_rejected_for_bucket_dots(
+            lambda: lancedb.connect(uri, storage_options=opts)
+        )
 
-    def test_bucket_without_dots_passes(self):
-        db = lancedb.connect(BUCKET_WITHOUT_DOTS)
-        assert db is not None
+    def test_bucket_without_dots_is_not_rejected(self):
+        assert_not_rejected_for_bucket_dots(
+            lambda: lancedb.connect(BUCKET_WITHOUT_DOTS)
+        )
 
 
 class TestS3BucketWithDotsAsync:
@@ -51,18 +62,24 @@ class TestS3BucketWithDotsAsync:
             await lancedb.connect_async(BUCKET_WITH_DOTS)
 
     @pytest.mark.asyncio
-    async def test_bucket_with_dots_and_region_passes(self):
+    async def test_bucket_with_dots_and_region_is_not_rejected(self):
         uri, opts = BUCKET_WITH_DOTS_AND_REGION
-        db = await lancedb.connect_async(uri, storage_options=opts)
-        assert db is not None
+        try:
+            await lancedb.connect_async(uri, storage_options=opts)
+        except ValueError as err:
+            assert "contains dots" not in str(err)
 
     @pytest.mark.asyncio
-    async def test_bucket_with_dots_and_aws_region_passes(self):
+    async def test_bucket_with_dots_and_aws_region_is_not_rejected(self):
         uri, opts = BUCKET_WITH_DOTS_AND_AWS_REGION
-        db = await lancedb.connect_async(uri, storage_options=opts)
-        assert db is not None
+        try:
+            await lancedb.connect_async(uri, storage_options=opts)
+        except ValueError as err:
+            assert "contains dots" not in str(err)
 
     @pytest.mark.asyncio
-    async def test_bucket_without_dots_passes(self):
-        db = await lancedb.connect_async(BUCKET_WITHOUT_DOTS)
-        assert db is not None
+    async def test_bucket_without_dots_is_not_rejected(self):
+        try:
+            await lancedb.connect_async(BUCKET_WITHOUT_DOTS)
+        except ValueError as err:
+            assert "contains dots" not in str(err)

python/python/tests/test_table.py

@@ -45,6 +45,32 @@ def _blob_test_data():
     )
 
 
+def _blob_v2_table(db: DBConnection, name: str):
+    schema = pa.schema([pa.field("id", pa.int64()), lancedb.blob("blob")])
+    table = db.create_table(name, schema=schema)
+    table.add([{"id": 1, "blob": b"hello"}, {"id": 2, "blob": b"world"}])
+    return table
+
+
+async def _blob_v2_table_async(db: AsyncConnection, name: str):
+    schema = pa.schema([pa.field("id", pa.int64()), lancedb.blob("blob")])
+    table = await db.create_table(name, schema=schema)
+    await table.add([{"id": 1, "blob": b"hello"}, {"id": 2, "blob": b"world"}])
+    return table
+
+
+def _blob_table(db: DBConnection, name: str, blob_schema: str):
+    if blob_schema == "v1":
+        return db.create_table(name, data=_blob_test_data())
+    return _blob_v2_table(db, name)
+
+
+async def _blob_table_async(db: AsyncConnection, name: str, blob_schema: str):
+    if blob_schema == "v1":
+        return await db.create_table(name, data=_blob_test_data())
+    return await _blob_v2_table_async(db, name)
+
+
 def _assert_lazy_blob(value, expected: bytes):
     assert hasattr(value, "readall")
     assert value.readall() == expected
@@ -107,6 +133,18 @@ def test_table_to_pandas_blob_modes(tmp_db: DBConnection, blob_mode):
         assert not hasattr(first, "readall")
 
 
+@pytest.mark.parametrize("blob_schema", ["v1", "v2"])
+def test_table_to_pandas_blob_bytes(tmp_db: DBConnection, blob_schema):
+    pytest.importorskip("lance")
+    table = _blob_table(tmp_db, f"test_to_pandas_blob_{blob_schema}_bytes", blob_schema)
+
+    df = table.to_pandas(blob_mode="bytes")
+
+    assert list(df.columns) == ["id", "blob"]
+    assert df["blob"].tolist() == [b"hello", b"world"]
+    assert "_rowid" not in df.columns
+
+
 def test_table_to_pandas_kwargs(tmp_db: DBConnection):
     pd = pytest.importorskip("pandas")
     data = pa.table({"id": pa.array([1, 2], pa.int64())})
@@ -118,15 +156,20 @@ def test_table_to_pandas_kwargs(tmp_db: DBConnection):
 
 
 @pytest.mark.asyncio
-async def test_async_table_to_pandas_blob_bytes(tmp_db_async: AsyncConnection):
+@pytest.mark.parametrize("blob_schema", ["v1", "v2"])
+async def test_async_table_to_pandas_blob_bytes(
+    tmp_db_async: AsyncConnection, blob_schema
+):
     pytest.importorskip("lance")
-    table = await tmp_db_async.create_table(
-        "test_async_to_pandas_blob_bytes", data=_blob_test_data()
+    table = await _blob_table_async(
+        tmp_db_async, f"test_async_to_pandas_blob_{blob_schema}_bytes", blob_schema
     )
 
     df = await table.to_pandas(blob_mode="bytes")
 
+    assert list(df.columns) == ["id", "blob"]
     assert df["blob"].tolist() == [b"hello", b"world"]
+    assert "_rowid" not in df.columns
 
 
 @pytest.mark.asyncio
@@ -1568,16 +1611,23 @@ def test_create_with_nans(mem_db: DBConnection):
         "fill_test",
         data=[
             {"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
+            {"vector": [2.1, 4.1], "item": "foo", "price": 9.0},
             {"vector": [np.nan], "item": "bar", "price": 20.0},
-            {"vector": [np.nan, np.nan], "item": "bar", "price": 20.0},
+            {"vector": [np.nan, 5.0], "item": "bar", "price": 21.0},
+            {"vector": [5], "item": "bar", "price": 22.0},
         ],
         on_bad_vectors="fill",
         fill_value=0.0,
     )
-    assert len(table) == 3
+    assert len(table) == 5
     arrow_tbl = table.search().where("item == 'bar'").to_arrow()
-    v = arrow_tbl["vector"].to_pylist()[0]
-    assert np.allclose(v, np.array([0.0, 0.0]))
+    filled_vectors = {
+        row["price"]: row["vector"]
+        for row in arrow_tbl.select(["price", "vector"]).to_pylist()
+    }
+    assert np.allclose(filled_vectors[20.0], np.array([0.0, 0.0]))
+    assert np.allclose(filled_vectors[21.0], np.array([0.0, 5.0]))
+    assert np.allclose(filled_vectors[22.0], np.array([5.0, 0.0]))
 
 
 def test_add_with_nans(mem_db: DBConnection):
@@ -1620,15 +1670,21 @@ def test_add_with_nans(mem_db: DBConnection):
         data=[
             {"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
             {"vector": [np.nan], "item": "bar", "price": 20.0},
-            {"vector": [np.nan, np.nan], "item": "bar", "price": 20.0},
+            {"vector": [np.nan, 5.0], "item": "bar", "price": 21.0},
+            {"vector": [5], "item": "bar", "price": 22.0},
         ],
         on_bad_vectors="fill",
         fill_value=0.0,
     )
-    assert len(table) == 3
+    assert len(table) == 4
     arrow_tbl = table.search().where("item == 'bar'").to_arrow()
-    v = arrow_tbl["vector"].to_pylist()[0]
-    assert np.allclose(v, np.array([0.0, 0.0]))
+    filled_vectors = {
+        row["price"]: row["vector"]
+        for row in arrow_tbl.select(["price", "vector"]).to_pylist()
+    }
+    assert np.allclose(filled_vectors[20.0], np.array([0.0, 0.0]))
+    assert np.allclose(filled_vectors[21.0], np.array([0.0, 5.0]))
+    assert np.allclose(filled_vectors[22.0], np.array([5.0, 0.0]))
 
 
 def test_add_with_empty_fixed_size_list_drops_bad_rows(mem_db: DBConnection):
@@ -1789,7 +1845,9 @@ def test_on_bad_vectors_fill_preserves_arrow_nested_vector_type(mem_db: DBConnec
         fill_value=0.0,
     )
 
-    assert table.to_arrow()["vector"].to_pylist() == [[1.0, 2.0], [0.0, 0.0]]
+    vector = table.to_arrow()["vector"]
+    assert vector.type == pa.list_(pa.float32())
+    assert vector.to_pylist() == [[1.0, 2.0], [0.0, 3.0]]
 
 
 @pytest.mark.parametrize(

python/python/tests/test_torch.py

@@ -148,15 +148,47 @@ def test_permutation_dataloader(mem_db):
     for batch in dataloader:
         assert batch["a"].size(0) == 10
 
-    permutation = permutation.with_format("torch")
-    dataloader = torch.utils.data.DataLoader(permutation, batch_size=10, shuffle=True)
+    # "torch" produces a list of per-row dicts per batch. The default
+    # DataLoader collate stacks the per-row dicts back into a batched dict.
+    torch_perm = permutation.with_format("torch")
+    batch = next(torch_perm.iter(10, skip_last_batch=False))
+    assert isinstance(batch, list)
+    assert len(batch) == 10
+    assert isinstance(batch[0], dict)
+    assert isinstance(batch[0]["a"], torch.Tensor)
+    rows = torch_perm.__getitems__([0, 1, 2])
+    assert isinstance(rows, list)
+    assert len(rows) == 3
+    assert isinstance(rows[0], dict)
+    assert isinstance(rows[0]["a"], torch.Tensor)
+    dataloader = torch.utils.data.DataLoader(torch_perm, batch_size=10, shuffle=True)
+    for batch in dataloader:
+        assert isinstance(batch, dict)
+        assert batch["a"].shape == (10,)
+    # Spawn-based workers exercise the pickle round-trip path: the new
+    # transform-as-list shape must survive pickling so workers produce the
+    # same per-row dicts the parent does.
+    spawn_loader = torch.utils.data.DataLoader(
+        torch_perm,
+        batch_size=10,
+        num_workers=2,
+        multiprocessing_context="spawn",
+    )
+    for batch in spawn_loader:
+        assert isinstance(batch, dict)
+        assert batch["a"].shape == (10,)
+
+    # "torch_row" returns a list of row tensors. Works with the default
+    # DataLoader collate (stacks rows into 2D).
+    row_perm = permutation.with_format("torch_row")
+    dataloader = torch.utils.data.DataLoader(row_perm, batch_size=10, shuffle=True)
     for batch in dataloader:
         assert batch.size(0) == 10
         assert batch.size(1) == 1
 
-    permutation = permutation.with_format("torch_col")
+    col_perm = permutation.with_format("torch_col")
     dataloader = torch.utils.data.DataLoader(
-        permutation, collate_fn=lambda x: x, batch_size=10, shuffle=True
+        col_perm, collate_fn=lambda x: x, batch_size=10, shuffle=True
     )
     for batch in dataloader:
         assert batch.size(0) == 1

python/python/tests/test_util.py

@@ -13,6 +13,7 @@ from lancedb.embeddings.registry import EmbeddingFunctionRegistry
 from lancedb.table import (
     _append_vector_columns,
     _cast_to_target_schema,
+    _fill_bad_vector_values,
     _handle_bad_vectors,
     _into_pyarrow_reader,
     _infer_target_schema,
@@ -25,10 +26,42 @@ import pandas as pd
 import polars as pl
 import pytest
 import lancedb
-from lancedb.util import get_uri_scheme, join_uri, value_to_sql
+from lancedb.util import flatten_columns, get_uri_scheme, join_uri, value_to_sql
 from utils import exception_output
 
 
+def _struct_table() -> pa.Table:
+    return pa.table(
+        {
+            "id": [1, 2],
+            "nested": pa.array([{"a": 1, "b": 2}, {"a": 3, "b": 4}]),
+        }
+    )
+
+
+def test_flatten_columns():
+    tbl = _struct_table()
+
+    # None / False mean "do not flatten": the struct column is preserved.
+    # `False` is a regression guard: because bool is a subclass of int it used
+    # to fall into the integer branch and raise ValueError (see issue).
+    for no_flatten in (None, False):
+        result = flatten_columns(tbl, no_flatten)
+        assert result.column_names == ["id", "nested"]
+
+    # True flattens all nested levels.
+    flattened = flatten_columns(tbl, True)
+    assert flattened.column_names == ["id", "nested.a", "nested.b"]
+
+    # A positive integer flattens up to that depth.
+    flattened = flatten_columns(tbl, 1)
+    assert flattened.column_names == ["id", "nested.a", "nested.b"]
+
+    # Non-positive integers are still rejected.
+    with pytest.raises(ValueError):
+        flatten_columns(tbl, 0)
+
+
 def test_normalize_uri():
     uris = [
         "relative/path",
@@ -255,7 +288,9 @@ def test_append_vector_columns():
 
 @pytest.mark.parametrize("on_bad_vectors", ["error", "drop", "fill", "null"])
 def test_handle_bad_vectors_jagged(on_bad_vectors):
-    vector = pa.array([[1.0, 2.0], [3.0], [4.0, 5.0]])
+    vector = pa.array(
+        [[1.0, 2.0], [3.0], [4.0, 5.0], [6.0, 7.0, 8.0], [None, 9.0], None]
+    )
     schema = pa.schema({"vector": pa.list_(pa.float64())})
     data = pa.table({"vector": vector}, schema=schema)
 
@@ -281,15 +316,54 @@ def test_handle_bad_vectors_jagged(on_bad_vectors):
         ).read_all()
 
     if on_bad_vectors == "drop":
-        expected = pa.array([[1.0, 2.0], [4.0, 5.0]])
+        expected = pa.array([[1.0, 2.0], [4.0, 5.0], [None, 9.0]])
     elif on_bad_vectors == "fill":
-        expected = pa.array([[1.0, 2.0], [42.0, 42.0], [4.0, 5.0]])
+        expected = pa.array(
+            [
+                [1.0, 2.0],
+                [3.0, 42.0],
+                [4.0, 5.0],
+                [6.0, 7.0],
+                [None, 9.0],
+                [42.0, 42.0],
+            ]
+        )
     elif on_bad_vectors == "null":
-        expected = pa.array([[1.0, 2.0], None, [4.0, 5.0]])
+        expected = pa.array([[1.0, 2.0], None, [4.0, 5.0], None, [None, 9.0], None])
 
     assert output["vector"].combine_chunks() == expected
 
 
+@pytest.mark.parametrize(
+    ("vector_type", "vectors", "expected"),
+    [
+        (
+            pa.list_(pa.float64()),
+            [[1.0, float("nan")], [2.0], None, [None, 3.0], [4.0, 5.0, 6.0]],
+            [[1.0, 42.0], [2.0, 42.0], [42.0, 42.0], [None, 3.0], [4.0, 5.0]],
+        ),
+        (
+            pa.large_list(pa.float64()),
+            [[1.0, float("nan")], [2.0], None, [None, 3.0], [4.0, 5.0, 6.0]],
+            [[1.0, 42.0], [2.0, 42.0], [42.0, 42.0], [None, 3.0], [4.0, 5.0]],
+        ),
+        (
+            pa.list_(pa.float64(), 2),
+            [[1.0, float("nan")], None, [None, 3.0]],
+            [[1.0, 42.0], [42.0, 42.0], [None, 3.0]],
+        ),
+    ],
+)
+def test_fill_bad_vector_values_arrow_types(vector_type, vectors, expected):
+    arr = pa.array([[0.0, 0.0], *vectors, [9.0, 9.0]], type=vector_type)
+    arr = arr.slice(1, len(vectors))
+
+    actual = _fill_bad_vector_values(arr, dim=2, fill_value=42.0)
+
+    assert actual.type == vector_type
+    assert actual.to_pylist() == expected
+
+
 @pytest.mark.parametrize("on_bad_vectors", ["error", "drop", "fill", "null"])
 def test_handle_bad_vectors_nan(on_bad_vectors):
     vector = pa.array([[1.0, float("nan")], [3.0, 4.0]])
@@ -319,7 +393,7 @@ def test_handle_bad_vectors_nan(on_bad_vectors):
     if on_bad_vectors == "drop":
         expected = pa.array([[3.0, 4.0]])
     elif on_bad_vectors == "fill":
-        expected = pa.array([[42.0, 42.0], [3.0, 4.0]])
+        expected = pa.array([[1.0, 42.0], [3.0, 4.0]])
     elif on_bad_vectors == "null":
         expected = pa.array([None, [3.0, 4.0]])
 

python/src/lib.rs

@@ -15,8 +15,8 @@ use pyo3::{
 use query::{FTSQuery, HybridQuery, Query, VectorQuery};
 use session::Session;
 use table::{
-    AddColumnsResult, AddResult, AlterColumnsResult, DeleteResult, DropColumnsResult, LsmWriteSpec,
-    MergeResult, Table, UpdateFieldMetadataResult, UpdateResult,
+    AddColumnsResult, AddResult, AlterColumnsResult, DeleteResult, DropColumnsResult, FtsToken,
+    LsmWriteSpec, MergeResult, PyBlobFile, Table, UpdateFieldMetadataResult, UpdateResult,
 };
 
 pub mod arrow;
@@ -27,6 +27,7 @@ pub mod header;
 pub mod index;
 pub mod namespace;
 pub mod oauth;
+pub mod otel;
 pub mod permutation;
 pub mod query;
 pub mod runtime;
@@ -43,6 +44,7 @@ pub fn _lancedb(_py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
     m.add_class::<Connection>()?;
     m.add_class::<Session>()?;
     m.add_class::<Table>()?;
+    m.add_class::<PyBlobFile>()?;
     m.add_class::<IndexConfig>()?;
     m.add_class::<Query>()?;
     m.add_class::<FTSQuery>()?;
@@ -58,12 +60,23 @@ pub fn _lancedb(_py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
     m.add_class::<DeleteResult>()?;
     m.add_class::<DropColumnsResult>()?;
     m.add_class::<UpdateResult>()?;
+    m.add_class::<FtsToken>()?;
     m.add_class::<PyAsyncPermutationBuilder>()?;
     m.add_class::<PyPermutationReader>()?;
     m.add_class::<PyExpr>()?;
+    // OpenTelemetry metrics bridge
+    m.add_class::<otel::PyMetricPoint>()?;
+    m.add_class::<otel::PyMetricDescription>()?;
+    m.add_function(wrap_pyfunction!(
+        otel::register_lancedb_metrics_recorder,
+        m
+    )?)?;
+    m.add_function(wrap_pyfunction!(otel::lancedb_metrics_catalog, m)?)?;
+    m.add_function(wrap_pyfunction!(otel::snapshot_lancedb_metrics, m)?)?;
     m.add_function(wrap_pyfunction!(connect, m)?)?;
     m.add_function(wrap_pyfunction!(connect_namespace, m)?)?;
     m.add_function(wrap_pyfunction!(connect_namespace_client, m)?)?;
+    m.add_function(wrap_pyfunction!(table::tokenize, m)?)?;
     m.add_function(wrap_pyfunction!(permutation::async_permutation_builder, m)?)?;
     m.add_function(wrap_pyfunction!(util::validate_table_name, m)?)?;
     m.add_function(wrap_pyfunction!(query::fts_query_to_json, m)?)?;

python/src/otel.rs

@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+//! Python-facing wrappers over [`lancedb::metrics_otel`].
+//!
+//! The aggregation, catalog, and histogram bucketing all live in the LanceDB
+//! core crate; this module only converts the core snapshot types into PyO3
+//! classes and exposes the three entry points to Python, where
+//! `lancedb/otel.py` bridges them into the user's OpenTelemetry `MeterProvider`.
+
+use std::collections::HashMap;
+
+use lancedb::metrics_otel::{MetricPoint, MetricValue};
+use pyo3::prelude::*;
+
+/// One metric data point exposed to Python. For counters and gauges only
+/// `value` is set; for histograms `buckets` (cumulative `le` counts), `count`,
+/// and `sum` are set.
+#[pyclass(name = "MetricPoint", get_all)]
+pub struct PyMetricPoint {
+    name: String,
+    kind: String,
+    attributes: HashMap<String, String>,
+    value: Option<f64>,
+    buckets: Option<Vec<(String, u64)>>,
+    count: Option<u64>,
+    sum: Option<f64>,
+}
+
+impl From<MetricPoint> for PyMetricPoint {
+    fn from(point: MetricPoint) -> Self {
+        let kind = point.kind.as_str().to_string();
+        let (value, buckets, count, sum) = match point.value {
+            MetricValue::Scalar(v) => (Some(v), None, None, None),
+            MetricValue::Histogram {
+                buckets,
+                count,
+                sum,
+            } => (None, Some(buckets), Some(count), Some(sum)),
+        };
+        Self {
+            name: point.name,
+            kind,
+            attributes: point.attributes,
+            value,
+            buckets,
+            count,
+            sum,
+        }
+    }
+}
+
+/// A described metric, used by the Python layer to create instruments up front.
+#[pyclass(name = "MetricDescription", get_all)]
+pub struct PyMetricDescription {
+    name: String,
+    kind: String,
+    unit: Option<String>,
+    description: String,
+}
+
+/// Install the LanceDB metrics recorder as the process-global `metrics` recorder.
+///
+/// Returns `True` if the recorder is installed (now or previously). Returns
+/// `False` if a *different* recorder is already installed — `metrics` allows
+/// only one global recorder per process, so LanceDB cannot coexist with another.
+#[pyfunction]
+pub fn register_lancedb_metrics_recorder() -> bool {
+    lancedb::metrics_otel::register_metrics_recorder()
+}
+
+/// The catalog of described LanceDB metrics. Empty until the recorder is installed.
+#[pyfunction]
+pub fn lancedb_metrics_catalog() -> Vec<PyMetricDescription> {
+    lancedb::metrics_otel::metrics_catalog()
+        .into_iter()
+        .map(|desc| PyMetricDescription {
+            name: desc.name,
+            kind: desc.kind.as_str().to_string(),
+            unit: desc.unit,
+            description: desc.description,
+        })
+        .collect()
+}
+
+/// A point-in-time snapshot of every recorded metric. Empty until the recorder
+/// is installed.
+///
+/// The read is lock-free but not O(1): it walks every registered series and
+/// allocates owned copies of their names and labels. The GIL is released across
+/// that work so a periodic collection doesn't stall other Python threads.
+#[pyfunction]
+pub fn snapshot_lancedb_metrics(py: Python<'_>) -> Vec<PyMetricPoint> {
+    let points = py.detach(lancedb::metrics_otel::snapshot_metrics);
+    points.into_iter().map(PyMetricPoint::from).collect()
+}

python/src/table.rs

@@ -2,7 +2,7 @@
 // SPDX-FileCopyrightText: Copyright The LanceDB Authors
 use std::{collections::HashMap, sync::Arc};
 
-use crate::runtime::future_into_py;
+use crate::runtime::{block_on, future_into_py};
 use crate::{
     connection::Connection,
     error::PythonErrorExt,
@@ -12,19 +12,23 @@ use crate::{
     table::scannable::PyScannable,
 };
 use arrow::{
+    array::{Array, LargeBinaryArray},
     datatypes::{DataType, Schema},
     ffi_stream::ArrowArrayStreamReader,
     pyarrow::{FromPyArrow, PyArrowType, ToPyArrow},
 };
+use lancedb::blob::BlobFile;
+use lancedb::index::scalar::FtsIndexBuilder;
 use lancedb::table::{
-    AddDataMode, ColumnAlteration, Duration, FieldMetadataUpdate, NewColumnTransform,
-    OptimizeAction, OptimizeOptions, Ref, Table as LanceDbTable,
+    AddDataMode, ColumnAlteration, Duration, FieldMetadataUpdate, FtsToken as LanceDbFtsToken,
+    NewColumnTransform, OptimizeAction, OptimizeOptions, Ref, Table as LanceDbTable,
 };
+use lancedb::tokenize as lancedb_tokenize;
 use pyo3::{
     Bound, FromPyObject, Py, PyAny, PyRef, PyResult, Python,
     exceptions::{PyRuntimeError, PyValueError},
-    pyclass, pymethods,
-    types::{IntoPyDict, PyAnyMethods, PyDict, PyDictMethods},
+    pyclass, pyfunction, pymethods,
+    types::{IntoPyDict, PyAnyMethods, PyBytes, PyDict, PyDictMethods},
 };
 
 mod scannable;
@@ -322,6 +326,12 @@ impl From<LsmWriteSpec> for lancedb::table::LsmWriteSpec {
     }
 }
 
+impl From<lancedb::table::LsmWriteSpec> for LsmWriteSpec {
+    fn from(inner: lancedb::table::LsmWriteSpec) -> Self {
+        Self { inner }
+    }
+}
+
 #[pyclass(get_all, from_py_object)]
 #[derive(Clone, Debug)]
 pub struct AddColumnsResult {
@@ -406,6 +416,150 @@ impl From<lancedb::table::DropColumnsResult> for DropColumnsResult {
     }
 }
 
+/// Lazy blob handle from ``Table.fetch_blob_files``.
+#[pyclass(name = "BlobFile")]
+pub struct PyBlobFile {
+    inner: Arc<BlobFile>,
+}
+
+#[pymethods]
+impl PyBlobFile {
+    fn read_bytes(self_: PyRef<'_, Self>) -> PyResult<Py<PyBytes>> {
+        let inner = self_.inner.clone();
+        let bytes = block_on(async move { inner.read().await })
+            .map_err(|e| PyRuntimeError::new_err(format!("blob read failed: {e}")))?;
+        Ok(PyBytes::new(self_.py(), bytes.as_ref()).unbind())
+    }
+
+    pub fn read(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner.clone();
+        future_into_py(self_.py(), async move {
+            let bytes = inner
+                .read()
+                .await
+                .map_err(|e| PyRuntimeError::new_err(format!("blob read failed: {e}")))?;
+            Python::attach(|py| Ok(PyBytes::new(py, bytes.as_ref()).unbind()))
+        })
+    }
+
+    fn close(self_: PyRef<'_, Self>) -> PyResult<()> {
+        let inner = self_.inner.clone();
+        block_on(async move { inner.close().await })
+            .map_err(|e| PyRuntimeError::new_err(format!("blob close failed: {e}")))
+    }
+
+    fn is_closed(self_: PyRef<'_, Self>) -> bool {
+        let inner = self_.inner.clone();
+        block_on(async move { inner.is_closed().await })
+    }
+
+    fn seek(self_: PyRef<'_, Self>, position: u64) -> PyResult<()> {
+        let inner = self_.inner.clone();
+        block_on(async move { inner.seek(position).await })
+            .map_err(|e| PyRuntimeError::new_err(format!("blob seek failed: {e}")))
+    }
+
+    fn tell(self_: PyRef<'_, Self>) -> PyResult<u64> {
+        let inner = self_.inner.clone();
+        block_on(async move { inner.tell().await })
+            .map_err(|e| PyRuntimeError::new_err(format!("blob tell failed: {e}")))
+    }
+
+    fn size(self_: PyRef<'_, Self>) -> u64 {
+        self_.inner.size()
+    }
+
+    /// Read a blob-local byte range without moving the cursor.
+    fn read_range(self_: PyRef<'_, Self>, offset: u64, length: usize) -> PyResult<Py<PyBytes>> {
+        let end = offset
+            .checked_add(length as u64)
+            .ok_or_else(|| PyValueError::new_err("offset + length overflowed"))?;
+        let inner = self_.inner.clone();
+        let bytes = block_on(async move { inner.read_range(offset..end).await })
+            .map_err(|e| PyRuntimeError::new_err(format!("blob read_range failed: {e}")))?;
+        Ok(PyBytes::new(self_.py(), bytes.as_ref()).unbind())
+    }
+
+    fn read_up_to(self_: PyRef<'_, Self>, length: usize) -> PyResult<Py<PyBytes>> {
+        let inner = self_.inner.clone();
+        let bytes = block_on(async move { inner.read_up_to(length).await })
+            .map_err(|e| PyRuntimeError::new_err(format!("blob read failed: {e}")))?;
+        Ok(PyBytes::new(self_.py(), bytes.as_ref()).unbind())
+    }
+}
+
+#[pyclass(get_all, from_py_object)]
+#[derive(Clone, Debug)]
+pub struct FtsToken {
+    pub text: String,
+    pub position: u32,
+}
+
+#[pymethods]
+impl FtsToken {
+    pub fn __repr__(&self) -> String {
+        format!("FtsToken(text={:?}, position={})", self.text, self.position)
+    }
+}
+
+impl From<LanceDbFtsToken> for FtsToken {
+    fn from(token: LanceDbFtsToken) -> Self {
+        Self {
+            text: token.text,
+            position: token.position,
+        }
+    }
+}
+
+#[pyfunction(signature = (
+    query,
+    *,
+    base_tokenizer = "simple".to_string(),
+    language = "English".to_string(),
+    max_token_length = Some(40),
+    lower_case = true,
+    stem = true,
+    remove_stop_words = true,
+    ascii_folding = true,
+    ngram_min_length = 3,
+    ngram_max_length = 3,
+    prefix_only = false
+))]
+#[allow(clippy::too_many_arguments)]
+pub fn tokenize(
+    query: String,
+    base_tokenizer: String,
+    language: String,
+    max_token_length: Option<u32>,
+    lower_case: bool,
+    stem: bool,
+    remove_stop_words: bool,
+    ascii_folding: bool,
+    ngram_min_length: u32,
+    ngram_max_length: u32,
+    prefix_only: bool,
+) -> PyResult<Vec<FtsToken>> {
+    let params = FtsIndexBuilder::default()
+        .base_tokenizer(base_tokenizer)
+        .language(&language)
+        .map_err(|_| {
+            PyValueError::new_err(format!(
+                "LanceDB does not support the requested language: '{}'",
+                language
+            ))
+        })?
+        .max_token_length(max_token_length.map(|value| value as usize))
+        .lower_case(lower_case)
+        .stem(stem)
+        .remove_stop_words(remove_stop_words)
+        .ascii_folding(ascii_folding)
+        .ngram_min_length(ngram_min_length)
+        .ngram_max_length(ngram_max_length)
+        .ngram_prefix_only(prefix_only);
+    let tokens = lancedb_tokenize(&query, &params).infer_error()?;
+    Ok(tokens.into_iter().map(FtsToken::from).collect())
+}
+
 #[pyclass]
 pub struct Table {
     // We keep a copy of the name to use if the inner table is dropped
@@ -704,6 +858,29 @@ impl Table {
         })
     }
 
+    #[pyo3(signature = (query, *, column=None, index_name=None))]
+    pub fn tokenize(
+        self_: PyRef<'_, Self>,
+        query: String,
+        column: Option<String>,
+        index_name: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        future_into_py(self_.py(), async move {
+            let tokens = match (column.as_deref(), index_name.as_deref()) {
+                (Some(_), Some(_)) | (None, None) => {
+                    return Err(PyValueError::new_err(
+                        "Specify exactly one of 'column' or 'index_name'",
+                    ));
+                }
+                (Some(column), None) => inner.tokenize_with_column(&query, column).await,
+                (None, Some(index_name)) => inner.tokenize(&query, index_name).await,
+            }
+            .infer_error()?;
+            Ok(tokens.into_iter().map(FtsToken::from).collect::<Vec<_>>())
+        })
+    }
+
     pub fn index_stats(self_: PyRef<'_, Self>, index_name: String) -> PyResult<Bound<'_, PyAny>> {
         let inner = self_.inner_ref()?.clone();
         future_into_py(self_.py(), async move {
@@ -895,6 +1072,55 @@ impl Table {
         ))
     }
 
+    /// Names of the blob v2 columns declared on this table, in declaration order.
+    pub fn blob_columns(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        future_into_py(self_.py(), async move {
+            inner.blob_columns().await.infer_error()
+        })
+    }
+
+    /// Read blob bytes for `row_ids` from blob v2 column `column`.
+    #[pyo3(signature = (column, row_ids))]
+    pub fn fetch_blobs(
+        self_: PyRef<'_, Self>,
+        column: String,
+        row_ids: Vec<u64>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        future_into_py(self_.py(), async move {
+            let blobs: LargeBinaryArray = inner
+                .fetch_blobs(column.as_str(), &row_ids)
+                .await
+                .infer_error()?;
+            Python::attach(|py| blobs.to_data().to_pyarrow(py).map(|obj| obj.unbind()))
+        })
+    }
+
+    /// Open lazy blob handles for `row_ids` from blob v2 column `column`.
+    #[pyo3(signature = (column, row_ids))]
+    pub fn fetch_blob_files(
+        self_: PyRef<'_, Self>,
+        column: String,
+        row_ids: Vec<u64>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        future_into_py(self_.py(), async move {
+            let handles = inner
+                .fetch_blob_files(column.as_str(), &row_ids)
+                .await
+                .infer_error()?;
+            Ok(handles
+                .into_iter()
+                .map(|handle| {
+                    handle.map(|file| PyBlobFile {
+                        inner: Arc::new(file),
+                    })
+                })
+                .collect::<Vec<_>>())
+        })
+    }
+
     /// Optimize the on-disk data by compacting and pruning old data, for better performance.
     #[pyo3(signature = (cleanup_since_ms=None, delete_unverified=None))]
     pub fn optimize(
@@ -1029,6 +1255,14 @@ impl Table {
         })
     }
 
+    pub fn get_lsm_write_spec(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        future_into_py(self_.py(), async move {
+            let spec = inner.get_lsm_write_spec().await.infer_error()?;
+            Ok(spec.map(LsmWriteSpec::from))
+        })
+    }
+
     pub fn close_lsm_writers(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
         let inner = self_.inner_ref()?.clone();
         future_into_py(self_.py(), async move {

python/tests/test_otel.py

@@ -0,0 +1,101 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+import lancedb
+import pyarrow as pa
+import pytest
+
+# The metrics recorder is process-global and installed once, so the whole
+# bridge is exercised in a single test to avoid cross-test global-state coupling.
+
+
+def _metrics_by_name(reader):
+    data = reader.get_metrics_data()
+    result = {}
+    for resource_metrics in data.resource_metrics:
+        for scope_metrics in resource_metrics.scope_metrics:
+            for metric in scope_metrics.metrics:
+                result[metric.name] = metric
+    return result
+
+
+def test_instrument_lancedb_metrics_exports_object_store_metrics(tmp_path):
+    pytest.importorskip("opentelemetry.sdk.metrics")
+    from lancedb.otel import instrument_lancedb_metrics
+    from opentelemetry.sdk.metrics import MeterProvider
+    from opentelemetry.sdk.metrics.export import InMemoryMetricReader
+
+    reader = InMemoryMetricReader()
+    provider = MeterProvider(metric_readers=[reader])
+    assert instrument_lancedb_metrics(provider)
+
+    # The catalog is populated once the recorder is installed.
+    from lancedb._lancedb import lancedb_metrics_catalog
+
+    catalog = {desc.name: desc for desc in lancedb_metrics_catalog()}
+    # Every metric kind emitted by the object store must be described so it is
+    # surfaced by the bridge (counter, histogram, and gauge).
+    assert catalog["lance_object_store_requests_total"].kind == "counter"
+    assert catalog["lance_object_store_request_duration_seconds"].kind == "histogram"
+    assert catalog["lance_object_store_in_flight_requests"].kind == "gauge"
+    assert catalog["lance_object_store_retryable_responses_total"].kind == "counter"
+
+    # Generate object store activity on the local filesystem (scheme "file").
+    db = lancedb.connect(str(tmp_path))
+    table = db.create_table("t", pa.table({"id": pa.array(range(256))}))
+    assert table.count_rows() == 256
+    assert table.to_arrow().num_rows == 256
+
+    metrics = _metrics_by_name(reader)
+
+    requests = metrics["lance_object_store_requests_total"]
+    points = list(requests.data.data_points)
+    assert points, "expected at least one request data point"
+    # Object store metrics are labelled by `operation` and `base` (the store
+    # scheme, e.g. "file", by default).
+    assert all("base" in p.attributes and "operation" in p.attributes for p in points)
+    assert sum(p.value for p in points) > 0
+
+    # Histograms are decomposed into bucket / count / sum observable counters.
+    bucket = metrics["lance_object_store_request_duration_seconds_bucket"]
+    bucket_points = list(bucket.data.data_points)
+    assert bucket_points
+    assert all("le" in p.attributes for p in bucket_points)
+    # The implicit +Inf bucket must be present and is the cumulative maximum.
+    assert any(p.attributes["le"] == "+Inf" for p in bucket_points)
+
+    count = metrics["lance_object_store_request_duration_seconds_count"]
+    assert sum(p.value for p in count.data.data_points) > 0
+
+    # The `_sum` instrument must also be wired and report positive latency.
+    duration_sum = metrics["lance_object_store_request_duration_seconds_sum"]
+    assert sum(p.value for p in duration_sum.data.data_points) > 0
+
+    # Unit handling: only `_sum` keeps the histogram's unit (seconds); `_bucket`
+    # and `_count` observe cumulative counts and are unitless.
+    assert duration_sum.unit == "s"
+    assert bucket.unit == ""
+    assert count.unit == ""
+
+
+def test_snapshot_empty_before_install_is_safe():
+    # snapshot is callable regardless of installation state and never raises.
+    from lancedb._lancedb import snapshot_lancedb_metrics
+
+    assert isinstance(snapshot_lancedb_metrics(), list)
+
+
+def test_instrument_warns_when_recorder_unavailable(monkeypatch):
+    # A foreign `metrics` recorder already installed -> register returns False;
+    # instrument_lancedb_metrics must warn and return False without instrumenting.
+    pytest.importorskip("opentelemetry.sdk.metrics")
+    import lancedb.otel as otel
+    from opentelemetry.sdk.metrics import MeterProvider
+    from opentelemetry.sdk.metrics.export import InMemoryMetricReader
+
+    monkeypatch.setattr(otel, "register_lancedb_metrics_recorder", lambda: False)
+
+    reader = InMemoryMetricReader()
+    provider = MeterProvider(metric_readers=[reader])
+    with pytest.warns(UserWarning, match="recorder"):
+        assert otel.instrument_lancedb_metrics(provider) is False

python/uv.lock

@@ -780,7 +780,7 @@ name = "cuda-bindings"
 version = "13.3.1"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "cuda-pathfinder" },
+    { name = "cuda-pathfinder", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
 ]
 wheels = [
     { url = "https://files.pythonhosted.org/packages/a9/21/8464d133752951c154feafb3b65c297e7d80f301183d220bec4c830f1441/cuda_bindings-13.3.1-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:120fcc53d57903df529c3486962c56528cba5b7d6c57c99537320ed9922c8b86", size = 6073403, upload-time = "2026-05-29T23:11:36.22Z" },
@@ -815,37 +815,37 @@ wheels = [
 
 [package.optional-dependencies]
 cublas = [
-    { name = "nvidia-cublas", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-cublas", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 cudart = [
-    { name = "nvidia-cuda-runtime", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-cuda-runtime", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 cufft = [
-    { name = "nvidia-cufft", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-cufft", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 cufile = [
     { name = "nvidia-cufile", marker = "sys_platform == 'linux'" },
 ]
 cupti = [
-    { name = "nvidia-cuda-cupti", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-cuda-cupti", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 curand = [
-    { name = "nvidia-curand", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-curand", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 cusolver = [
-    { name = "nvidia-cusolver", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-cusolver", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 cusparse = [
-    { name = "nvidia-cusparse", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-cusparse", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 nvjitlink = [
-    { name = "nvidia-nvjitlink", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 nvrtc = [
-    { name = "nvidia-cuda-nvrtc", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-cuda-nvrtc", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 nvtx = [
-    { name = "nvidia-nvtx", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
+    { name = "nvidia-nvtx", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
 ]
 
 [[package]]
@@ -1909,6 +1909,9 @@ embeddings = [
     { name = "sentencepiece" },
     { name = "torch" },
 ]
+otel = [
+    { name = "opentelemetry-api" },
+]
 pylance = [
     { name = "pylance" },
 ]
@@ -1923,6 +1926,7 @@ tests = [
     { name = "boto3" },
     { name = "datafusion" },
     { name = "duckdb" },
+    { name = "opentelemetry-sdk" },
     { name = "pandas", version = "2.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
     { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.14'" },
     { name = "pandas", version = "3.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.14'" },
@@ -1963,6 +1967,8 @@ requires-dist = [
     { name = "open-clip-torch", marker = "extra == 'clip'" },
     { name = "open-clip-torch", marker = "extra == 'embeddings'", specifier = ">=2.20.0" },
     { name = "openai", marker = "extra == 'embeddings'", specifier = ">=1.6.1" },
+    { name = "opentelemetry-api", marker = "extra == 'otel'" },
+    { name = "opentelemetry-sdk", marker = "extra == 'tests'", specifier = ">=1.30.0" },
     { name = "overrides", marker = "python_full_version < '3.12'", specifier = ">=0.7" },
     { name = "packaging", specifier = ">=23.0" },
     { name = "pandas", marker = "extra == 'tests'", specifier = ">=1.4" },
@@ -1994,7 +2000,7 @@ requires-dist = [
     { name = "transformers", marker = "extra == 'siglip'", specifier = ">=4.41.0" },
     { name = "typing-extensions", marker = "python_full_version < '3.11' and extra == 'dev'", specifier = ">=4.0.0" },
 ]
-provides-extras = ["azure", "clip", "dev", "docs", "embeddings", "pylance", "siglip", "tests"]
+provides-extras = ["azure", "clip", "dev", "docs", "embeddings", "otel", "pylance", "siglip", "tests"]
 
 [[package]]
 name = "lomond"
@@ -2775,7 +2781,7 @@ name = "nvidia-cudnn-cu13"
 version = "9.19.0.56"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-cublas" },
+    { name = "nvidia-cublas", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
 ]
 wheels = [
     { url = "https://files.pythonhosted.org/packages/f1/84/26025437c1e6b61a707442184fa0c03d083b661adf3a3eecfd6d21677740/nvidia_cudnn_cu13-9.19.0.56-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:6ed29ffaee1176c612daf442e4dd6cfeb6a0caa43ddcbeb59da94953030b1be4", size = 433781201, upload-time = "2026-02-03T20:40:53.805Z" },
@@ -2787,7 +2793,7 @@ name = "nvidia-cufft"
 version = "12.0.0.61"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-nvjitlink" },
+    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
 ]
 wheels = [
     { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
@@ -2817,9 +2823,9 @@ name = "nvidia-cusolver"
 version = "12.0.4.66"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-cublas" },
-    { name = "nvidia-cusparse" },
-    { name = "nvidia-nvjitlink" },
+    { name = "nvidia-cublas", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
+    { name = "nvidia-cusparse", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
+    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
 ]
 wheels = [
     { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
@@ -2831,7 +2837,7 @@ name = "nvidia-cusparse"
 version = "12.6.3.3"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-nvjitlink" },
+    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
 ]
 wheels = [
     { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
@@ -2934,6 +2940,45 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/f6/46/180e14be801a75bc13f234cb1b594b232adeb9c84e60a9ab1832e8333591/openai-2.40.0-py3-none-any.whl", hash = "sha256:2b205637ff214477f9ce9ab035e9f494db0e3fa8f1e599008953735fbf6ff1ff", size = 1350935, upload-time = "2026-06-01T21:48:21.462Z" },
 ]
 
+[[package]]
+name = "opentelemetry-api"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ae/cc/e4c9584181f86494df0f6bdec1a4f3280c50db44704dc2a407e994fc87bb/opentelemetry_api-1.43.0.tar.gz", hash = "sha256:107d0d03857ea8fc7c5fcbbbd83f800c281f0d560553d61c1d675fccfd1761c1", size = 73476, upload-time = "2026-06-24T15:19:55.323Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/17/83/6dba32b85f31868400440dc7ad2ca1eab94cbbf3a7b0459ed39f8311a9e2/opentelemetry_api-1.43.0-py3-none-any.whl", hash = "sha256:20acf45e9b21851926835292e4045d290acade1edd2ff3de86d2f069687ba1fd", size = 61912, upload-time = "2026-06-24T15:19:35.434Z" },
+]
+
+[[package]]
+name = "opentelemetry-sdk"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "opentelemetry-api" },
+    { name = "opentelemetry-semantic-conventions" },
+    { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3e/eb/5041074274ac0956b03637cc039d434569112468e875eddfcc9a0674ce06/opentelemetry_sdk-1.43.0.tar.gz", hash = "sha256:d8187c81c162df9913e4003dd6485f7390d9a24fc17026ec7387b8b8218b08e9", size = 254744, upload-time = "2026-06-24T15:20:08.467Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/49/e3/b17be23af124201c9f52eececd4cc8ddfed1597d37b4ee771895d325805c/opentelemetry_sdk-1.43.0-py3-none-any.whl", hash = "sha256:d1323a547c1ce69d6a069a17a44b7da82bb8b332051ecb074041f87642c86823", size = 178852, upload-time = "2026-06-24T15:19:52.169Z" },
+]
+
+[[package]]
+name = "opentelemetry-semantic-conventions"
+version = "0.64b0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "opentelemetry-api" },
+    { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5a/30/5f26df29509eccd86b99b481ac9ffa39da49ba9577cc69071c552ae30447/opentelemetry_semantic_conventions-0.64b0.tar.gz", hash = "sha256:72f76fb2d1582d9d033dd1fcd84532e961e6ff3d90d24ba6fabc72975a83864c", size = 148340, upload-time = "2026-06-24T15:20:09.267Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/f2/ca/23ba87a221b574a7c5a99d48849d80bfe8b047624681357e2b002e566187/opentelemetry_semantic_conventions-0.64b0-py3-none-any.whl", hash = "sha256:ea77e85e354b8f604ddbe5f3d9135216f982fa4d77e5859ac30f6d8a50505aa6", size = 203713, upload-time = "2026-06-24T15:19:53.339Z" },
+]
+
 [[package]]
 name = "overrides"
 version = "7.7.0"

rust/lancedb/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lancedb"
-version = "0.31.0-beta.6"
+version = "0.32.0-beta.1"
 edition.workspace = true
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license.workspace = true
@@ -44,11 +44,12 @@ lance-io = { workspace = true }
 lance-index = { workspace = true, features = ["tokenizer-jieba", "tokenizer-lindera"] }
 lance-table = { workspace = true }
 lance-linalg = { workspace = true }
-lance-testing = { workspace = true }
 lance-encoding = { workspace = true }
 lance-arrow = { workspace = true }
 lance-namespace = { workspace = true }
 lance-namespace-impls = { workspace = true }
+metrics = { workspace = true, optional = true }
+metrics-util = { workspace = true, optional = true }
 moka = { workspace = true }
 pin-project = { workspace = true }
 tokio = { version = "1.23", features = ["rt-multi-thread", "sync"] }
@@ -93,6 +94,7 @@ semver = { workspace = true }
 
 [dev-dependencies]
 anyhow = "1"
+lance-testing = { workspace = true }
 tempfile = "3.5.0"
 random_word = { version = "0.4.3", features = ["en"] }
 tokio = { version = "1.23", features = ["io-util", "macros", "net", "rt-multi-thread", "sync"] }
@@ -128,6 +130,12 @@ azure = [
     "lance-namespace-impls/dir-azure",
     "lance-namespace-impls/credential-vendor-azure",
 ]
+cos = ["lance/tencent", "lance-io/tencent"]
+goosefs = [
+    "lance/goosefs",
+    "lance-io/goosefs",
+    "lance-namespace-impls/dir-goosefs",
+]
 huggingface = [
     "lance/huggingface",
     "lance-io/huggingface",
@@ -141,6 +149,15 @@ remote = [
     "lance-namespace-impls/rest",
     "lance-namespace-impls/rest-adapter",
 ]
+# Publish LanceDB's internal metrics (currently object store request counts,
+# bytes, latency, errors, and throttles) through the `metrics` crate facade,
+# and re-export the `metrics` crate as `lancedb::metrics`. Install any
+# `metrics`-compatible recorder to collect them.
+metrics = ["dep:metrics", "lance/metrics", "lance-io/metrics"]
+# Additional adapter on top of `metrics` that installs a process-global recorder
+# and exposes a pull-based snapshot/catalog API (see `lancedb::metrics_otel`)
+# for bridging metrics into OpenTelemetry or other pull-based exporters.
+metrics-otel = ["metrics", "dep:metrics-util"]
 fp16kernels = ["lance-linalg/fp16kernels"]
 s3-test = []
 bedrock = ["dep:aws-sdk-bedrockruntime"]

rust/lancedb/src/database/listing.rs

@@ -342,6 +342,19 @@ impl ListingDatabase {
         ))
     }
 
+    fn storage_base_uri(uri: &str) -> String {
+        let Ok(mut url) = url::Url::parse(uri) else {
+            return uri.to_string();
+        };
+        url.set_query(None);
+        let Some((storage_scheme, _commit_scheme)) = url.scheme().split_once('+') else {
+            return url.to_string();
+        };
+        let storage_scheme = storage_scheme.to_string();
+        let _ = url.set_scheme(&storage_scheme);
+        url.to_string()
+    }
+
     async fn prepare_namespace_root(
         uri: &str,
         storage_options: &HashMap<String, String>,
@@ -520,6 +533,8 @@ impl ListingDatabase {
                 // will add a trailing '?' to the url
                 url.set_query(None);
 
+                let storage_base_uri = Self::storage_base_uri(url.as_str());
+
                 let table_base_uri = if let Some(store) = engine {
                     static WARN_ONCE: std::sync::Once = std::sync::Once::new();
                     WARN_ONCE.call_once(|| {
@@ -532,8 +547,6 @@ impl ListingDatabase {
                     url.to_string()
                 };
 
-                let plain_uri = url.to_string();
-
                 let session = request
                     .session
                     .clone()
@@ -550,13 +563,13 @@ impl ListingDatabase {
                 };
                 let (object_store, base_path) = ObjectStore::from_uri_and_params(
                     session.store_registry(),
-                    &plain_uri,
+                    &storage_base_uri,
                     &os_params,
                 )
                 .await?;
                 if object_store.is_local() {
-                    Self::try_create_dir(&plain_uri).context(CreateDirSnafu {
-                        path: plain_uri.clone(),
+                    Self::try_create_dir(&storage_base_uri).context(CreateDirSnafu {
+                        path: storage_base_uri.clone(),
                     })?;
                 }
 
@@ -570,7 +583,7 @@ impl ListingDatabase {
                 };
 
                 let namespace_database = Self::connect_namespace_database(
-                    &table_base_uri,
+                    &storage_base_uri,
                     options.storage_options.clone(),
                     request.namespace_client_properties.clone(),
                     request.read_consistency_interval,
@@ -1309,6 +1322,60 @@ mod tests {
         (tempdir, db)
     }
 
+    #[tokio::test]
+    async fn test_listing_database_root_ops_do_not_create_manifest() {
+        let tempdir = tempdir().unwrap();
+        let uri = tempdir.path().to_str().unwrap();
+
+        let request = ConnectRequest {
+            uri: uri.to_string(),
+            #[cfg(feature = "remote")]
+            client_config: Default::default(),
+            options: Default::default(),
+            namespace_client_properties: Default::default(),
+            manifest_enabled: false,
+            read_consistency_interval: None,
+            session: None,
+        };
+
+        let db = ListingDatabase::connect_with_options(&request)
+            .await
+            .unwrap();
+
+        assert!(!tempdir.path().join("__manifest").exists());
+
+        let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int32, false)]));
+        db.create_table(CreateTableRequest {
+            name: "root_table".to_string(),
+            namespace_path: vec![],
+            data: Box::new(RecordBatch::new_empty(schema)) as Box<dyn Scannable>,
+            mode: CreateTableMode::Create,
+            write_options: Default::default(),
+            location: None,
+            namespace_client: None,
+        })
+        .await
+        .unwrap();
+
+        db.open_table(OpenTableRequest {
+            name: "root_table".to_string(),
+            namespace_path: vec![],
+            index_cache_size: None,
+            lance_read_params: None,
+            location: None,
+            namespace_client: None,
+            managed_versioning: None,
+        })
+        .await
+        .unwrap();
+
+        #[allow(deprecated)]
+        let table_names = db.table_names(TableNamesRequest::default()).await.unwrap();
+
+        assert_eq!(table_names, vec!["root_table".to_string()]);
+        assert!(!tempdir.path().join("__manifest").exists());
+    }
+
     #[tokio::test]
     async fn test_clone_table_basic() {
         let (_tempdir, db) = setup_database().await;
@@ -2283,6 +2350,24 @@ mod tests {
         assert_eq!(captured.as_deref(), Some("foo=bar"));
     }
 
+    #[test]
+    fn test_storage_base_uri_strips_commit_engine_scheme() {
+        assert_eq!(
+            ListingDatabase::storage_base_uri("s3+ddb://bucket/prefix?ddbTableName=commit_table"),
+            "s3://bucket/prefix"
+        );
+
+        assert_eq!(
+            ListingDatabase::storage_base_uri("s3://bucket/prefix?foo=bar"),
+            "s3://bucket/prefix"
+        );
+
+        assert_eq!(
+            ListingDatabase::storage_base_uri("/tmp/lancedb"),
+            "/tmp/lancedb"
+        );
+    }
+
     /// Regression: connecting via a URL-style URI (which goes through
     /// `url::Url::parse` and the `query_pairs_mut()` path) must not
     /// append a trailing `?` to per-table URIs when the input URI has

rust/lancedb/src/embeddings.rs

@@ -198,28 +198,36 @@ fn compute_embedding_arrays(
     batch: &RecordBatch,
     embeddings: &[(EmbeddingDefinition, Arc<dyn EmbeddingFunction>)],
 ) -> Result<Vec<Arc<dyn Array>>> {
-    if embeddings.len() == 1 {
-        let (fld, func) = &embeddings[0];
-        let src_column =
-            batch
-                .column_by_name(&fld.source_column)
-                .ok_or_else(|| Error::InvalidInput {
-                    message: format!("Source column '{}' not found", fld.source_column),
-                })?;
+    let input_columns = embeddings
+        .iter()
+        .map(|(fld, func)| {
+            let src_column =
+                batch
+                    .column_by_name(&fld.source_column)
+                    .ok_or_else(|| Error::InvalidInput {
+                        message: format!("Source column '{}' not found", fld.source_column),
+                    })?;
+            Ok((src_column.clone(), func))
+        })
+        .collect::<Result<Vec<_>>>()?;
+
+    if batch.num_rows() == 0 {
+        return input_columns
+            .iter()
+            .map(|(_, func)| Ok(arrow_array::new_empty_array(func.dest_type()?.as_ref())))
+            .collect();
+    }
+
+    if input_columns.len() == 1 {
+        let (src_column, func) = &input_columns[0];
         return Ok(vec![func.compute_source_embeddings(src_column.clone())?]);
     }
 
     // Parallel path: multiple embeddings
     std::thread::scope(|s| {
-        let handles: Vec<_> = embeddings
+        let handles: Vec<_> = input_columns
             .iter()
-            .map(|(fld, func)| {
-                let src_column = batch.column_by_name(&fld.source_column).ok_or_else(|| {
-                    Error::InvalidInput {
-                        message: format!("Source column '{}' not found", fld.source_column),
-                    }
-                })?;
-
+            .map(|(src_column, func)| {
                 let handle = s.spawn(move || func.compute_source_embeddings(src_column.clone()));
 
                 Ok(handle)
@@ -392,3 +400,104 @@ impl<R: RecordBatchReader> RecordBatchReader for WithEmbeddings<R> {
             .into_rich_schema()
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::sync::{
+        Arc,
+        atomic::{AtomicUsize, Ordering},
+    };
+
+    use arrow_array::{Array, ArrayRef, FixedSizeListArray, RecordBatch, StringArray};
+    use arrow_schema::DataType;
+
+    use super::*;
+
+    #[derive(Debug)]
+    struct FailingEmbedding {
+        calls: AtomicUsize,
+    }
+
+    impl EmbeddingFunction for FailingEmbedding {
+        fn name(&self) -> &str {
+            "failing"
+        }
+
+        fn source_type(&self) -> Result<Cow<'_, DataType>> {
+            Ok(Cow::Owned(DataType::Utf8))
+        }
+
+        fn dest_type(&self) -> Result<Cow<'_, DataType>> {
+            Ok(Cow::Owned(DataType::new_fixed_size_list(
+                DataType::Float32,
+                3,
+                false,
+            )))
+        }
+
+        fn compute_source_embeddings(&self, _source: Arc<dyn Array>) -> Result<Arc<dyn Array>> {
+            self.calls.fetch_add(1, Ordering::SeqCst);
+            Err(Error::Runtime {
+                message: "embedding function must not receive an empty batch".to_string(),
+            })
+        }
+
+        fn compute_query_embeddings(&self, _input: Arc<dyn Array>) -> Result<Arc<dyn Array>> {
+            unreachable!("query embeddings are not exercised by this test")
+        }
+    }
+
+    #[test]
+    fn empty_batch_skips_embedding_functions() {
+        let embedding_function = Arc::new(FailingEmbedding {
+            calls: AtomicUsize::new(0),
+        });
+        let source: ArrayRef = Arc::new(StringArray::from(Vec::<&str>::new()));
+        let batch = RecordBatch::try_from_iter([("text", source)]).unwrap();
+        let embeddings = vec![(
+            EmbeddingDefinition::new("text", "failing", Some("text_embedding")),
+            embedding_function.clone() as Arc<dyn EmbeddingFunction>,
+        )];
+
+        let result = compute_embeddings_for_batch(batch, &embeddings).unwrap();
+
+        assert_eq!(embedding_function.calls.load(Ordering::SeqCst), 0);
+        assert_eq!(result.num_rows(), 0);
+
+        let embedding = result.column_by_name("text_embedding").unwrap();
+        assert_eq!(
+            embedding.data_type(),
+            &DataType::new_fixed_size_list(DataType::Float32, 3, false)
+        );
+        assert_eq!(embedding.null_count(), 0);
+
+        let embedding = embedding
+            .as_any()
+            .downcast_ref::<FixedSizeListArray>()
+            .unwrap();
+        assert_eq!(embedding.len(), 0);
+        assert_eq!(embedding.value_length(), 3);
+        assert_eq!(embedding.values().len(), 0);
+    }
+
+    #[test]
+    fn empty_batch_still_validates_source_column() {
+        let embedding_function = Arc::new(FailingEmbedding {
+            calls: AtomicUsize::new(0),
+        });
+        let source: ArrayRef = Arc::new(StringArray::from(Vec::<&str>::new()));
+        let batch = RecordBatch::try_from_iter([("text", source)]).unwrap();
+        let embeddings = vec![(
+            EmbeddingDefinition::new("missing_column", "failing", Some("text_embedding")),
+            embedding_function.clone() as Arc<dyn EmbeddingFunction>,
+        )];
+
+        let result = compute_embeddings_for_batch(batch, &embeddings);
+        assert!(result.is_err());
+        assert!(
+            matches!(result.unwrap_err(), Error::InvalidInput { .. }),
+            "expected InvalidInput error when source column is missing"
+        );
+        assert_eq!(embedding_function.calls.load(Ordering::SeqCst), 0);
+    }
+}

rust/lancedb/src/index.rs

@@ -317,6 +317,8 @@ pub enum IndexType {
     // FTS
     #[serde(alias = "INVERTED", alias = "Inverted")]
     FTS,
+    /// Catch-all for index types not recognized by this version of LanceDB.
+    Unknown,
 }
 
 impl std::fmt::Display for IndexType {
@@ -334,6 +336,7 @@ impl std::fmt::Display for IndexType {
             Self::LabelList => write!(f, "LABEL_LIST"),
             Self::Fm => write!(f, "FM"),
             Self::FTS => write!(f, "FTS"),
+            Self::Unknown => write!(f, "UNKNOWN"),
         }
     }
 }
@@ -355,9 +358,7 @@ impl std::str::FromStr for IndexType {
             "IVF_HNSW_PQ" => Ok(Self::IvfHnswPq),
             "IVF_HNSW_SQ" => Ok(Self::IvfHnswSq),
             "IVF_HNSW_FLAT" => Ok(Self::IvfHnswFlat),
-            _ => Err(Error::InvalidInput {
-                message: format!("the input value {} is not a valid IndexType", value),
-            }),
+            _ => Ok(Self::Unknown),
         }
     }
 }
@@ -425,20 +426,15 @@ pub struct IndexConfig {
 #[derive(Debug, Deserialize)]
 pub(crate) struct IndexMetadata {
     pub metric_type: Option<DistanceType>,
-    // Sometimes the index type is provided at this level.
-    pub index_type: Option<IndexType>,
 }
 
-// This struct is used to deserialize the JSON data returned from the Lance API
-// Dataset::index_statistics().
+// Deserializes the JSON returned by Dataset::index_statistics().
 #[skip_serializing_none]
 #[derive(Debug, Deserialize)]
 pub(crate) struct IndexStatisticsImpl {
     pub num_indexed_rows: usize,
     pub num_unindexed_rows: usize,
     pub indices: Vec<IndexMetadata>,
-    // Sometimes, the index type is provided at this level.
-    pub index_type: Option<IndexType>,
     pub num_indices: Option<u32>,
 }
 

rust/lancedb/src/lib.rs

@@ -33,6 +33,11 @@
 //! - `remote` - Enable remote client to connect to LanceDB cloud.
 //! - `huggingface` - Enable HuggingFace Hub integration for loading datasets from the Hub.
 //! - `fp16kernels` - Enable FP16 kernels for faster vector search on CPU.
+//! - `metrics` - Publish LanceDB's internal metrics through the
+//!   [`metrics`](https://docs.rs/metrics) crate facade and re-export that crate.
+//!   Install any `metrics`-compatible recorder to collect them.
+//! - `metrics-otel` - Add a pull-based adapter (the `metrics_otel` module) over
+//!   the `metrics` facade for bridging metrics into OpenTelemetry or similar.
 //!
 //! ### Quick Start
 //!
@@ -174,6 +179,8 @@ pub mod expr;
 pub mod index;
 pub mod io;
 pub mod ipc;
+#[cfg(feature = "metrics-otel")]
+pub mod metrics_otel;
 #[cfg(feature = "polars")]
 mod polars_arrow_convertors;
 pub mod query;
@@ -194,7 +201,21 @@ pub use connection::{ConnectNamespaceBuilder, Connection};
 pub use error::{Error, Result};
 use lance_index::vector::ApproxMode as LanceApproxMode;
 use lance_linalg::distance::DistanceType as LanceDistanceType;
-pub use table::Table;
+/// Re-export of the [`metrics`](https://docs.rs/metrics) crate facade. Enable
+/// the `metrics` feature to publish LanceDB's internal metrics; install any
+/// `metrics`-compatible recorder to collect them. See also [`metrics_otel`] for
+/// a built-in pull-based adapter.
+#[cfg(feature = "metrics")]
+pub use metrics;
+pub use table::{FtsToken, Table};
+
+/// Tokenize a full-text search query using an explicit FTS tokenizer configuration.
+///
+/// This does not require a table or FTS index. The tokenizer options are the
+/// same [`index::scalar::FtsIndexBuilder`] values used when creating an FTS index.
+pub fn tokenize(query: &str, params: &index::scalar::FtsIndexBuilder) -> Result<Vec<FtsToken>> {
+    table::tokenize(query, params)
+}
 
 #[derive(Debug, Copy, Clone, PartialEq, Serialize, Deserialize, Default)]
 #[non_exhaustive]

rust/lancedb/src/metrics_otel.rs

@@ -0,0 +1,594 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+//! A pull-based adapter over the [`metrics`] crate facade.
+//!
+//! LanceDB (through Lance core) publishes metrics — currently object store
+//! request counts, bytes, latency, errors, and throttles — through the global
+//! [`metrics`] facade without choosing a backend. This module installs a
+//! process-global [`metrics::Recorder`] that aggregates those metrics into
+//! lock-free cumulative storage and exposes that state as a snapshot, so callers
+//! can feed it into a pull-based exporter such as OpenTelemetry.
+//!
+//! The language bindings build their OpenTelemetry integrations on top of the
+//! three public entry points here: [`register_metrics_recorder`],
+//! [`metrics_catalog`], and [`snapshot_metrics`].
+//!
+//! The recorder is *generic*: it records any metric emitted through the facade,
+//! keyed by name and labels. Object store metrics are the first producer, but
+//! nothing here is specific to them. New metrics flow through automatically;
+//! they only need to be described (via the `metrics` `describe_*!` macros) so
+//! callers can discover their name, kind, and unit up front.
+//!
+//! ## Why pull, not push
+//!
+//! OpenTelemetry collects on its own schedule and invokes observable-instrument
+//! callbacks at collection time. Cumulative counters map directly onto OTel's
+//! `ObservableCounter` semantics. So the adapter aggregates in Rust and lets the
+//! collection thread pull a [`snapshot`](snapshot_metrics) on demand.
+//!
+//! ## Histograms
+//!
+//! OpenTelemetry has no asynchronous histogram instrument, so histograms cannot
+//! be pulled as-is. Instead each histogram is aggregated into fixed buckets
+//! (Prometheus style) and exposed as cumulative `le` bucket counts plus a count
+//! and sum, which the caller can surface as observable counters.
+
+use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, LazyLock, Mutex, OnceLock, RwLock};
+
+use metrics::{Counter, Gauge, Histogram, Key, KeyName, Metadata, Recorder, SharedString, Unit};
+use metrics_util::registry::{Registry, Storage};
+
+/// Bucket boundaries used when a histogram has no registered bounds. Covers a
+/// broad latency range so unknown histograms still produce useful buckets.
+const DEFAULT_BOUNDS: &[f64] = &[
+    0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0,
+];
+
+/// The kind of a metric, mirroring the three `metrics` instrument types.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum MetricKind {
+    Counter,
+    Gauge,
+    Histogram,
+}
+
+impl MetricKind {
+    /// The lowercase name of this kind (`"counter"`, `"gauge"`, `"histogram"`).
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Counter => "counter",
+            Self::Gauge => "gauge",
+            Self::Histogram => "histogram",
+        }
+    }
+}
+
+/// A described metric, used to create one exporter instrument per metric up front.
+#[derive(Debug, Clone)]
+pub struct MetricDescription {
+    /// The metric name (e.g. `lance_object_store_requests_total`).
+    pub name: String,
+    /// Whether the metric is a counter, gauge, or histogram.
+    pub kind: MetricKind,
+    /// The canonical unit label, if the producer described one.
+    pub unit: Option<String>,
+    /// Human-readable help text describing the metric.
+    pub description: String,
+}
+
+/// The aggregated value of a metric at snapshot time.
+#[derive(Debug, Clone)]
+pub enum MetricValue {
+    /// A counter or gauge value.
+    Scalar(f64),
+    /// A histogram, decomposed into cumulative `le` buckets plus count and sum.
+    Histogram {
+        /// Cumulative `(le, count)` buckets, ending in the implicit `+Inf` bucket.
+        buckets: Vec<(String, u64)>,
+        /// Total number of recorded samples.
+        count: u64,
+        /// Sum of all recorded sample values.
+        sum: f64,
+    },
+}
+
+/// One aggregated metric data point exposed to a caller.
+#[derive(Debug, Clone)]
+pub struct MetricPoint {
+    /// The metric name.
+    pub name: String,
+    /// Whether the point is a counter, gauge, or histogram.
+    pub kind: MetricKind,
+    /// The label set for this point (e.g. `operation`, `base`).
+    pub attributes: HashMap<String, String>,
+    /// The aggregated value.
+    pub value: MetricValue,
+}
+
+/// Catalog of described metrics, keyed by metric name.
+static CATALOG: LazyLock<Mutex<HashMap<String, CatalogEntry>>> =
+    LazyLock::new(|| Mutex::new(HashMap::new()));
+
+struct CatalogEntry {
+    kind: MetricKind,
+    unit: Option<String>,
+    description: String,
+}
+
+/// Per-metric histogram bucket boundaries, keyed by metric name. Producers
+/// register their recommended bounds before any metric is recorded.
+static HISTOGRAM_BOUNDS: LazyLock<RwLock<HashMap<String, Arc<[f64]>>>> =
+    LazyLock::new(|| RwLock::new(HashMap::new()));
+
+/// The installed recorder's registry, available once installation succeeds.
+static REGISTRY: OnceLock<Arc<Registry<Key, LanceStorage>>> = OnceLock::new();
+
+fn bounds_for(name: &str) -> Arc<[f64]> {
+    HISTOGRAM_BOUNDS
+        .read()
+        .unwrap()
+        .get(name)
+        .cloned()
+        .unwrap_or_else(|| Arc::from(DEFAULT_BOUNDS))
+}
+
+/// A histogram that buckets samples at record time into fixed boundaries,
+/// keeping a cumulative count and sum. Bucketing eagerly keeps memory bounded
+/// (unlike retaining raw samples) and produces Prometheus-style `le` buckets.
+struct BucketedHistogram {
+    /// Sorted, finite upper bounds. A sample `v` falls in the first bucket whose
+    /// bound is `>= v`; samples above all bounds fall in the implicit `+Inf`
+    /// bucket stored as the final entry of `counts`.
+    bounds: Arc<[f64]>,
+    /// Per-bucket (non-cumulative) counts; length is `bounds.len() + 1`.
+    counts: Box<[AtomicU64]>,
+    count: AtomicU64,
+    /// Running sum of recorded values, stored as `f64` bits (there is no atomic
+    /// f64, so the bit pattern is held in a `u64`; see [`Self::add_to_sum`]).
+    sum_bits: AtomicU64,
+}
+
+// All atomics here use `Ordering::Relaxed`: each metric counter is independent,
+// so no happens-before relationship is needed between them, and a snapshot
+// reader tolerates slightly stale values. This matches `metrics_util`'s
+// `AtomicStorage`.
+
+impl BucketedHistogram {
+    fn new(bounds: Arc<[f64]>) -> Self {
+        let counts = (0..bounds.len() + 1)
+            .map(|_| AtomicU64::new(0))
+            .collect::<Vec<_>>()
+            .into_boxed_slice();
+        Self {
+            bounds,
+            counts,
+            count: AtomicU64::new(0),
+            sum_bits: AtomicU64::new(0),
+        }
+    }
+
+    fn add_to_sum(&self, value: f64) {
+        // No atomic offers an f64 add, so read the current bit pattern, add in
+        // float space, and CAS it back, retrying if another thread won the race.
+        let mut current = self.sum_bits.load(Ordering::Relaxed);
+        loop {
+            let updated = (f64::from_bits(current) + value).to_bits();
+            match self.sum_bits.compare_exchange_weak(
+                current,
+                updated,
+                Ordering::Relaxed,
+                Ordering::Relaxed,
+            ) {
+                Ok(_) => break,
+                Err(actual) => current = actual,
+            }
+        }
+    }
+
+    /// Cumulative `le` buckets, total count, and sum at this instant.
+    fn snapshot(&self) -> MetricValue {
+        let mut cumulative = 0u64;
+        let mut buckets = Vec::with_capacity(self.bounds.len() + 1);
+        for (i, bound) in self.bounds.iter().enumerate() {
+            cumulative += self.counts[i].load(Ordering::Relaxed);
+            buckets.push((format!("{}", bound), cumulative));
+        }
+        cumulative += self.counts[self.bounds.len()].load(Ordering::Relaxed);
+        buckets.push(("+Inf".to_string(), cumulative));
+        MetricValue::Histogram {
+            buckets,
+            count: self.count.load(Ordering::Relaxed),
+            sum: f64::from_bits(self.sum_bits.load(Ordering::Relaxed)),
+        }
+    }
+}
+
+impl metrics::HistogramFn for BucketedHistogram {
+    fn record(&self, value: f64) {
+        let idx = self.bounds.partition_point(|&bound| bound < value);
+        self.counts[idx].fetch_add(1, Ordering::Relaxed);
+        self.count.fetch_add(1, Ordering::Relaxed);
+        self.add_to_sum(value);
+    }
+}
+
+/// Storage backing the registry. Counters and gauges are plain atomics (as in
+/// `metrics_util`'s `AtomicStorage`); histograms use [`BucketedHistogram`].
+struct LanceStorage;
+
+impl Storage<Key> for LanceStorage {
+    type Counter = Arc<AtomicU64>;
+    type Gauge = Arc<AtomicU64>;
+    type Histogram = Arc<BucketedHistogram>;
+
+    fn counter(&self, _key: &Key) -> Self::Counter {
+        Arc::new(AtomicU64::new(0))
+    }
+
+    fn gauge(&self, _key: &Key) -> Self::Gauge {
+        // The `metrics` facade writes the f64 bit pattern into this `u64` (the
+        // snapshot decodes it with `f64::from_bits`), matching `AtomicStorage`.
+        // `0` decodes to `0.0`, the correct initial value.
+        Arc::new(AtomicU64::new(0))
+    }
+
+    fn histogram(&self, key: &Key) -> Self::Histogram {
+        Arc::new(BucketedHistogram::new(bounds_for(key.name())))
+    }
+}
+
+struct LanceRecorder {
+    registry: Arc<Registry<Key, LanceStorage>>,
+}
+
+impl LanceRecorder {
+    fn describe(
+        &self,
+        key: KeyName,
+        kind: MetricKind,
+        unit: Option<Unit>,
+        description: SharedString,
+    ) {
+        CATALOG.lock().unwrap().insert(
+            key.as_str().to_string(),
+            CatalogEntry {
+                kind,
+                unit: unit.map(|u| u.as_canonical_label().to_string()),
+                description: description.into_owned(),
+            },
+        );
+    }
+}
+
+impl Recorder for LanceRecorder {
+    fn describe_counter(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
+        self.describe(key, MetricKind::Counter, unit, description);
+    }
+
+    fn describe_gauge(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
+        self.describe(key, MetricKind::Gauge, unit, description);
+    }
+
+    fn describe_histogram(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
+        self.describe(key, MetricKind::Histogram, unit, description);
+    }
+
+    fn register_counter(&self, key: &Key, _metadata: &Metadata<'_>) -> Counter {
+        self.registry
+            .get_or_create_counter(key, |c| Counter::from_arc(c.clone()))
+    }
+
+    fn register_gauge(&self, key: &Key, _metadata: &Metadata<'_>) -> Gauge {
+        self.registry
+            .get_or_create_gauge(key, |g| Gauge::from_arc(g.clone()))
+    }
+
+    fn register_histogram(&self, key: &Key, _metadata: &Metadata<'_>) -> Histogram {
+        self.registry
+            .get_or_create_histogram(key, |h| Histogram::from_arc(h.clone()))
+    }
+}
+
+/// Register the recommended histogram bounds for every metric-emitting
+/// subsystem. New subsystems add their `histogram_bounds()` here.
+fn register_bounds() {
+    let mut bounds = HISTOGRAM_BOUNDS.write().unwrap();
+    for (name, values) in lance_io::object_store::metrics::histogram_bounds() {
+        bounds.insert((*name).to_string(), Arc::from(*values));
+    }
+}
+
+/// Describe every metric-emitting subsystem so the catalog is populated. Must
+/// run after the recorder is installed. New subsystems add their
+/// `describe_metrics()` here.
+fn describe_all() {
+    lance_io::object_store::metrics::describe_metrics();
+}
+
+fn labels(key: &Key) -> HashMap<String, String> {
+    key.labels()
+        .map(|label| (label.key().to_string(), label.value().to_string()))
+        .collect()
+}
+
+fn collect_points(registry: &Registry<Key, LanceStorage>) -> Vec<MetricPoint> {
+    let mut points = Vec::new();
+    for (key, handle) in registry.get_counter_handles() {
+        points.push(MetricPoint {
+            name: key.name().to_string(),
+            kind: MetricKind::Counter,
+            attributes: labels(&key),
+            // OpenTelemetry observations are float; counts stay well within the
+            // f64-exact integer range (2^53), so this cast is lossless in practice.
+            value: MetricValue::Scalar(handle.load(Ordering::Relaxed) as f64),
+        });
+    }
+    for (key, handle) in registry.get_gauge_handles() {
+        points.push(MetricPoint {
+            name: key.name().to_string(),
+            kind: MetricKind::Gauge,
+            attributes: labels(&key),
+            value: MetricValue::Scalar(f64::from_bits(handle.load(Ordering::Relaxed))),
+        });
+    }
+    for (key, handle) in registry.get_histogram_handles() {
+        points.push(MetricPoint {
+            name: key.name().to_string(),
+            kind: MetricKind::Histogram,
+            attributes: labels(&key),
+            value: handle.snapshot(),
+        });
+    }
+    points
+}
+
+/// Install the LanceDB metrics recorder as the process-global `metrics` recorder.
+///
+/// Returns `true` if the recorder is installed (now or previously). Returns
+/// `false` if a *different* recorder is already installed — `metrics` allows
+/// only one global recorder per process, so LanceDB cannot coexist with another.
+pub fn register_metrics_recorder() -> bool {
+    if REGISTRY.get().is_some() {
+        return true;
+    }
+    let registry = Arc::new(Registry::new(LanceStorage));
+    let recorder = LanceRecorder {
+        registry: registry.clone(),
+    };
+    // Register bounds *before* installing the recorder. Bounds don't depend on
+    // the recorder, and once it is installed a concurrent histogram emission
+    // could otherwise create a handle with the fallback bounds and keep them for
+    // the process lifetime.
+    register_bounds();
+    match metrics::set_global_recorder(recorder) {
+        Ok(()) => {
+            let _ = REGISTRY.set(registry);
+            // Describe metrics only after install so the `describe_*!` macros
+            // route through this recorder and populate the catalog.
+            describe_all();
+            true
+        }
+        Err(_) => false,
+    }
+}
+
+/// The catalog of described LanceDB metrics. Empty until the recorder is installed.
+pub fn metrics_catalog() -> Vec<MetricDescription> {
+    CATALOG
+        .lock()
+        .unwrap()
+        .iter()
+        .map(|(name, entry)| MetricDescription {
+            name: name.clone(),
+            kind: entry.kind,
+            unit: entry.unit.clone(),
+            description: entry.description.clone(),
+        })
+        .collect()
+}
+
+/// A point-in-time snapshot of every recorded metric. Empty until the recorder
+/// is installed. The read is lock-free.
+pub fn snapshot_metrics() -> Vec<MetricPoint> {
+    let Some(registry) = REGISTRY.get() else {
+        return Vec::new();
+    };
+    collect_points(registry)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use metrics::HistogramFn;
+
+    fn bucket_count(buckets: &[(String, u64)], le: &str) -> u64 {
+        buckets
+            .iter()
+            .find(|(b, _)| b == le)
+            .map(|(_, c)| *c)
+            .unwrap_or_else(|| panic!("no bucket with le={le}"))
+    }
+
+    #[test]
+    fn bucketed_histogram_records_cumulative_buckets() {
+        let hist = BucketedHistogram::new(Arc::from([0.1f64, 1.0, 10.0].as_slice()));
+        hist.record(0.05); // le=0.1
+        hist.record(0.5); // le=1
+        hist.record(0.5); // le=1
+        hist.record(50.0); // +Inf
+
+        let MetricValue::Histogram {
+            buckets,
+            count,
+            sum,
+        } = hist.snapshot()
+        else {
+            panic!("expected histogram");
+        };
+
+        // Buckets are cumulative (Prometheus `le` semantics).
+        assert_eq!(bucket_count(&buckets, "0.1"), 1);
+        assert_eq!(bucket_count(&buckets, "1"), 3);
+        assert_eq!(bucket_count(&buckets, "10"), 3);
+        assert_eq!(bucket_count(&buckets, "+Inf"), 4);
+        assert_eq!(count, 4);
+        assert!((sum - 51.05).abs() < 1e-9);
+    }
+
+    #[test]
+    fn bucketed_histogram_boundary_is_inclusive() {
+        let hist = BucketedHistogram::new(Arc::from([1.0f64].as_slice()));
+        hist.record(1.0); // exactly the bound -> le=1, not +Inf
+        let MetricValue::Histogram { buckets, .. } = hist.snapshot() else {
+            panic!("expected histogram");
+        };
+        assert_eq!(bucket_count(&buckets, "1"), 1);
+        assert_eq!(bucket_count(&buckets, "+Inf"), 1);
+    }
+
+    #[test]
+    fn bucketed_histogram_boundary_is_inclusive_mid_range() {
+        // A value equal to a middle bound lands in that bucket, not the next.
+        let hist = BucketedHistogram::new(Arc::from([0.1f64, 1.0, 10.0].as_slice()));
+        hist.record(1.0);
+        let MetricValue::Histogram { buckets, .. } = hist.snapshot() else {
+            panic!("expected histogram");
+        };
+        assert_eq!(bucket_count(&buckets, "0.1"), 0);
+        assert_eq!(bucket_count(&buckets, "1"), 1);
+        assert_eq!(bucket_count(&buckets, "10"), 1); // cumulative, so still 1
+        assert_eq!(bucket_count(&buckets, "+Inf"), 1);
+    }
+
+    #[test]
+    fn recorder_aggregates_counters_with_labels() {
+        let registry = Arc::new(Registry::new(LanceStorage));
+        let recorder = LanceRecorder {
+            registry: registry.clone(),
+        };
+        metrics::with_local_recorder(&recorder, || {
+            metrics::counter!("test_requests_total", "operation" => "get", "scheme" => "s3")
+                .increment(2);
+            metrics::counter!("test_requests_total", "operation" => "get", "scheme" => "s3")
+                .increment(3);
+            // A distinct label set must produce a separate point, not merge.
+            metrics::counter!("test_requests_total", "operation" => "put", "scheme" => "gs")
+                .increment(7);
+        });
+
+        let scalar = |attrs: &[(&str, &str)]| {
+            let points = collect_points(&registry);
+            let point = points
+                .into_iter()
+                .find(|p| {
+                    p.name == "test_requests_total"
+                        && attrs
+                            .iter()
+                            .all(|(k, v)| p.attributes.get(*k).map(String::as_str) == Some(*v))
+                })
+                .expect("counter recorded for label set");
+            assert_eq!(point.kind, MetricKind::Counter);
+            match point.value {
+                MetricValue::Scalar(v) => v,
+                _ => panic!("expected scalar"),
+            }
+        };
+
+        // Same labels aggregate; distinct labels stay separate.
+        assert!((scalar(&[("operation", "get"), ("scheme", "s3")]) - 5.0).abs() < 1e-9);
+        assert!((scalar(&[("operation", "put"), ("scheme", "gs")]) - 7.0).abs() < 1e-9);
+    }
+
+    #[test]
+    fn recorder_records_gauges() {
+        let registry = Arc::new(Registry::new(LanceStorage));
+        let recorder = LanceRecorder {
+            registry: registry.clone(),
+        };
+        // Gauges store the f64 bit pattern in a u64; the snapshot must decode it.
+        metrics::with_local_recorder(&recorder, || {
+            metrics::gauge!("test_gauge", "scheme" => "s3").set(3.5);
+        });
+
+        let points = collect_points(&registry);
+        let point = points
+            .iter()
+            .find(|p| p.name == "test_gauge")
+            .expect("gauge recorded");
+        assert_eq!(point.kind, MetricKind::Gauge);
+        assert!(matches!(point.value, MetricValue::Scalar(v) if (v - 3.5).abs() < 1e-9));
+    }
+
+    #[test]
+    fn recorder_falls_back_to_default_bounds() {
+        // A histogram with no registered bounds uses DEFAULT_BOUNDS.
+        let name = "test_unregistered_histogram";
+        assert!(!HISTOGRAM_BOUNDS.read().unwrap().contains_key(name));
+
+        let registry = Arc::new(Registry::new(LanceStorage));
+        let recorder = LanceRecorder {
+            registry: registry.clone(),
+        };
+        metrics::with_local_recorder(&recorder, || {
+            metrics::histogram!(name).record(0.02);
+        });
+
+        let points = collect_points(&registry);
+        let point = points.iter().find(|p| p.name == name).expect("recorded");
+        let MetricValue::Histogram { buckets, count, .. } = &point.value else {
+            panic!("expected histogram");
+        };
+        assert_eq!(*count, 1);
+        // DEFAULT_BOUNDS yields one bucket per bound plus the implicit `+Inf`.
+        assert_eq!(buckets.len(), DEFAULT_BOUNDS.len() + 1);
+        // 0.02 falls in the le=0.025 bucket (the third DEFAULT_BOUNDS entry).
+        assert_eq!(bucket_count(buckets, "0.025"), 1);
+        assert_eq!(bucket_count(buckets, "0.01"), 0);
+        assert_eq!(bucket_count(buckets, "+Inf"), 1);
+    }
+
+    #[test]
+    fn recorder_uses_registered_histogram_bounds() {
+        let name = "test_recorder_bounds_seconds";
+        HISTOGRAM_BOUNDS
+            .write()
+            .unwrap()
+            .insert(name.to_string(), Arc::from([0.1f64, 1.0].as_slice()));
+
+        let registry = Arc::new(Registry::new(LanceStorage));
+        let recorder = LanceRecorder {
+            registry: registry.clone(),
+        };
+        metrics::with_local_recorder(&recorder, || {
+            metrics::histogram!(name).record(0.05);
+            metrics::histogram!(name).record(5.0);
+        });
+
+        let points = collect_points(&registry);
+        let point = points.iter().find(|p| p.name == name).expect("recorded");
+        let MetricValue::Histogram { buckets, count, .. } = &point.value else {
+            panic!("expected histogram");
+        };
+        assert_eq!(*count, 2);
+        assert_eq!(bucket_count(buckets, "0.1"), 1);
+        assert_eq!(bucket_count(buckets, "+Inf"), 2);
+    }
+
+    #[test]
+    fn describe_populates_catalog() {
+        let name = "test_describe_catalog_total";
+        let registry = Arc::new(Registry::new(LanceStorage));
+        let recorder = LanceRecorder { registry };
+        metrics::with_local_recorder(&recorder, || {
+            metrics::describe_counter!(name, Unit::Count, "a test counter");
+        });
+
+        let catalog = CATALOG.lock().unwrap();
+        let entry = catalog.get(name).expect("described");
+        assert_eq!(entry.kind, MetricKind::Counter);
+        assert_eq!(entry.description, "a test counter");
+    }
+}

rust/lancedb/src/remote/table.rs

@@ -20,6 +20,7 @@ use crate::table::AddColumnsResult;
 use crate::table::AddResult;
 use crate::table::DeleteResult;
 use crate::table::DropColumnsResult;
+use crate::table::LsmWriteSpec;
 use crate::table::MergeResult;
 use crate::table::Tags;
 use crate::table::UpdateResult;
@@ -2269,8 +2270,7 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
         })
     }
 
-    async fn set_lsm_write_spec(&self, spec: crate::table::LsmWriteSpec) -> Result<()> {
-        use crate::table::LsmWriteSpec;
+    async fn set_lsm_write_spec(&self, spec: LsmWriteSpec) -> Result<()> {
         self.check_mutable().await?;
 
         // Map the spec onto the server's request DTO. `sharding` is internally
@@ -2322,6 +2322,69 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
         Ok(())
     }
 
+    async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
+        // Read counterpart to set/unset, resolved server-side against HEAD. The
+        // server reads the spec from the `__lance_mem_wal` system index (shard
+        // column mapped from its Lance field id against the current schema) and
+        // re-encodes it into the same sophon-owned shape the set endpoint
+        // accepts — no lance/lancedb types cross the wire. `lsm_write_spec` is
+        // null when the LSM write path is not enabled for the table.
+        let request = self.post_read(&format!(
+            "/v1/table/{}/get_lsm_write_spec/",
+            self.identifier
+        ));
+        let (request_id, response) = self.send(request, true).await?;
+        let response = self.check_table_response(&request_id, response).await?;
+        let body = response.text().await.err_to_http(request_id.clone())?;
+
+        // Mirror of sophon's `Sharding` (internally tagged on `mode`) and
+        // `LsmWriteSpecBody` / `GetLsmWriteSpecResponse`.
+        #[derive(Deserialize)]
+        #[serde(tag = "mode", rename_all = "snake_case")]
+        enum Sharding {
+            Unsharded,
+            Bucket { column: String, num_buckets: u32 },
+            Identity { column: String },
+        }
+        #[derive(Deserialize)]
+        struct LsmWriteSpecBody {
+            sharding: Sharding,
+            #[serde(default)]
+            maintained_indexes: Vec<String>,
+            #[serde(default)]
+            writer_config_defaults: std::collections::HashMap<String, String>,
+        }
+        #[derive(Deserialize)]
+        struct GetLsmWriteSpecResponse {
+            lsm_write_spec: Option<LsmWriteSpecBody>,
+        }
+
+        let parsed: GetLsmWriteSpecResponse =
+            serde_json::from_str(&body).map_err(|e| Error::Http {
+                source: format!("Failed to parse get_lsm_write_spec response: {}", e).into(),
+                request_id,
+                status_code: None,
+            })?;
+
+        let Some(body) = parsed.lsm_write_spec else {
+            // The LSM write path is not enabled for this table.
+            return Ok(None);
+        };
+
+        let spec = match body.sharding {
+            Sharding::Bucket {
+                column,
+                num_buckets,
+            } => LsmWriteSpec::bucket(column, num_buckets),
+            Sharding::Identity { column } => LsmWriteSpec::identity(column),
+            Sharding::Unsharded => LsmWriteSpec::unsharded(),
+        }
+        .with_maintained_indexes(body.maintained_indexes)
+        .with_writer_config_defaults(body.writer_config_defaults);
+
+        Ok(Some(spec))
+    }
+
     async fn tags(&self) -> Result<Box<dyn Tags + '_>> {
         Ok(Box::new(RemoteTags { inner: self }))
     }
@@ -2754,8 +2817,7 @@ mod tests {
     use super::*;
 
     use crate::remote::client::{ClientConfig, RetryConfig};
-    use crate::table::AddDataMode;
-    use crate::table::FieldMetadataUpdate;
+    use crate::table::{AddDataMode, FieldMetadataUpdate, FtsToken};
 
     use arrow::{array::AsArray, compute::concat_batches, datatypes::Int32Type};
     use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, record_batch};
@@ -4825,6 +4887,141 @@ mod tests {
         assert_eq!(text_idx.created_at, None);
     }
 
+    #[tokio::test]
+    async fn test_tokenize_uses_remote_index_details() {
+        let schema = Schema::new(vec![Field::new("text", DataType::Utf8, false)]);
+        let index_details = serde_json::json!({
+            "base_tokenizer": "icu",
+            "language": "English",
+            "with_position": false,
+            "max_token_length": 40,
+            "lower_case": true,
+            "stem": false,
+            "remove_stop_words": false,
+            "ascii_folding": true,
+        })
+        .to_string();
+        let table = Table::new_with_handler("my_table", move |request| {
+            assert_eq!(request.method(), "POST");
+            match request.url().path() {
+                "/v1/table/my_table/describe/" => http::Response::builder()
+                    .status(200)
+                    .body(describe_response(&schema))
+                    .unwrap(),
+                "/v1/table/my_table/index/list/" => {
+                    let body = serde_json::json!({
+                        "indexes": [
+                            {
+                                "index_name": "text_idx",
+                                "columns": ["text"],
+                                "index_type": "FTS",
+                                "index_details": index_details,
+                            },
+                        ]
+                    });
+                    http::Response::builder()
+                        .status(200)
+                        .body(serde_json::to_string(&body).unwrap())
+                        .unwrap()
+                }
+                path => panic!("Unexpected path: {}", path),
+            }
+        });
+
+        let tokens = table
+            .tokenize("Hello, こんにちは世界!", "text_idx")
+            .await
+            .unwrap();
+
+        assert_eq!(
+            tokens,
+            vec![
+                FtsToken {
+                    text: "hello".to_string(),
+                    position: 0,
+                },
+                FtsToken {
+                    text: "こんにちは".to_string(),
+                    position: 1,
+                },
+                FtsToken {
+                    text: "世界".to_string(),
+                    position: 2,
+                },
+            ]
+        );
+    }
+
+    #[tokio::test]
+    async fn test_tokenize_requires_existing_index_name() {
+        let schema = Schema::new(vec![Field::new("text", DataType::Utf8, false)]);
+        let table = Table::new_with_handler("my_table", move |request| -> http::Response<String> {
+            assert_eq!(request.method(), "POST");
+            match request.url().path() {
+                "/v1/table/my_table/describe/" => http::Response::builder()
+                    .status(200)
+                    .body(describe_response(&schema))
+                    .unwrap(),
+                "/v1/table/my_table/index/list/" => {
+                    let body = serde_json::json!({ "indexes": [] });
+                    http::Response::builder()
+                        .status(200)
+                        .body(serde_json::to_string(&body).unwrap())
+                        .unwrap()
+                }
+                path => panic!("Unexpected path: {}", path),
+            }
+        });
+
+        let err = table.tokenize("hello", "text_idx").await.unwrap_err();
+        assert!(matches!(
+            err,
+            Error::InvalidInput { message }
+                if message.contains("No index named 'text_idx'")
+        ));
+    }
+
+    #[tokio::test]
+    async fn test_tokenize_with_column_remote_requires_index_details() {
+        let schema = Schema::new(vec![Field::new("text", DataType::Utf8, false)]);
+        let table = Table::new_with_handler("my_table", move |request| {
+            assert_eq!(request.method(), "POST");
+            match request.url().path() {
+                "/v1/table/my_table/describe/" => http::Response::builder()
+                    .status(200)
+                    .body(describe_response(&schema))
+                    .unwrap(),
+                "/v1/table/my_table/index/list/" => {
+                    let body = serde_json::json!({
+                        "indexes": [
+                            {
+                                "index_name": "text_idx",
+                                "columns": ["text"],
+                                "index_type": "FTS",
+                            },
+                        ]
+                    });
+                    http::Response::builder()
+                        .status(200)
+                        .body(serde_json::to_string(&body).unwrap())
+                        .unwrap()
+                }
+                path => panic!("Unexpected path: {}", path),
+            }
+        });
+
+        let err = table
+            .tokenize_with_column("hello", "text")
+            .await
+            .unwrap_err();
+
+        assert!(matches!(
+            err,
+            Error::InvalidInput { message }
+                if message.contains("does not include tokenizer details")
+        ));
+    }
+
     #[test]
     fn test_deserialize_created_at() {
         #[derive(Deserialize)]
@@ -5361,6 +5558,74 @@ mod tests {
         table.unset_lsm_write_spec().await.unwrap();
     }
 
+    #[tokio::test]
+    async fn test_get_lsm_write_spec() {
+        let table = Table::new_with_handler("my_table", |request| {
+            assert_eq!(request.method(), "POST");
+            assert_eq!(
+                request.url().path(),
+                "/v1/table/my_table/get_lsm_write_spec/"
+            );
+
+            // The server resolves the spec and re-encodes it into the same
+            // sophon-owned shape the set endpoint accepts (`Sharding` internally
+            // tagged on `mode`, wrapped in `lsm_write_spec`).
+            let response = serde_json::json!({
+                "lsm_write_spec": {
+                    "sharding": { "mode": "bucket", "column": "id", "num_buckets": 4 },
+                    "maintained_indexes": ["id_idx"],
+                    "writer_config_defaults": { "durable_write": "false" },
+                }
+            });
+            http::Response::builder()
+                .status(200)
+                .body(response.to_string())
+                .unwrap()
+        });
+
+        let spec = table
+            .get_lsm_write_spec()
+            .await
+            .unwrap()
+            .expect("a spec should be reported");
+        match spec {
+            crate::table::LsmWriteSpec::Bucket {
+                column,
+                num_buckets,
+                maintained_indexes,
+                writer_config_defaults,
+            } => {
+                assert_eq!(column, "id");
+                assert_eq!(num_buckets, 4);
+                assert_eq!(maintained_indexes, vec!["id_idx".to_string()]);
+                assert_eq!(
+                    writer_config_defaults
+                        .get("durable_write")
+                        .map(String::as_str),
+                    Some("false")
+                );
+            }
+            other => panic!("expected a bucket spec, got {:?}", other),
+        }
+    }
+
+    #[tokio::test]
+    async fn test_get_lsm_write_spec_absent() {
+        let table = Table::new_with_handler("my_table", |request| {
+            assert_eq!(
+                request.url().path(),
+                "/v1/table/my_table/get_lsm_write_spec/"
+            );
+            // Null spec → the LSM write path is not enabled.
+            let response = serde_json::json!({ "lsm_write_spec": null });
+            http::Response::builder()
+                .status(200)
+                .body(response.to_string())
+                .unwrap()
+        });
+        assert!(table.get_lsm_write_spec().await.unwrap().is_none());
+    }
+
     #[tokio::test]
     async fn test_wait_for_index() {
         let table = _make_table_with_indices(0);

rust/lancedb/src/table.rs

@@ -23,10 +23,13 @@ use lance::dataset::{InsertBuilder, WriteParams};
 use lance::index::DatasetIndexExt;
 use lance::io::{ObjectStoreParams, WrappingObjectStore};
 use lance_datafusion::utils::StreamingWriteSource;
+use lance_index::IndexCriteria;
 use lance_io::object_store::{LanceNamespaceStorageOptionsProvider, StorageOptionsAccessor};
 pub use query::AnyQuery;
 
 use lance::io::commit::namespace_manifest::LanceNamespaceExternalManifestStore;
+use lance_index::scalar::InvertedIndexParams;
+use lance_index::scalar::inverted::query::collect_query_tokens;
 use lance_namespace::LanceNamespace;
 use lance_namespace::error::NamespaceError;
 use lance_namespace::models::DescribeTableRequest;
@@ -42,6 +45,7 @@ use std::sync::Arc;
 
 use crate::connection::NamespaceClientPushdownOperation;
 
+use crate::DistanceType;
 use crate::data::scannable::{PeekedScannable, Scannable, estimate_write_partitions};
 use crate::database::Database;
 use crate::database::read_freshness::TableFreshness;
@@ -49,10 +53,10 @@ use crate::embeddings::{EmbeddingDefinition, EmbeddingRegistry, MemoryRegistry};
 use crate::error::{Error, Result};
 use crate::index::IndexStatistics;
 use crate::index::{Index, IndexBuilder};
-use crate::index::{IndexConfig, IndexStatisticsImpl};
+use crate::index::{IndexConfig, IndexStatisticsImpl, IndexType};
 use crate::query::{IntoQueryVector, Query, QueryExecutionOptions, TakeQuery, VectorQuery};
 use crate::table::datafusion::insert::InsertExec;
-use crate::utils::{PatchReadParam, PatchWriteParam};
+use crate::utils::{PatchReadParam, PatchWriteParam, resolve_arrow_field_path};
 
 use self::dataset::DatasetConsistencyWrapper;
 use self::merge::MergeInsertBuilder;
@@ -471,6 +475,33 @@ impl LsmWriteSpec {
     }
 }
 
+/// A token produced by the tokenizer configured on a full-text search index.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct FtsToken {
+    /// The token text after the index tokenizer has applied its filters.
+    pub text: String,
+    /// The token position used by full-text query matching.
+    pub position: u32,
+}
+
+/// Tokenize a full-text search query using an explicit FTS tokenizer configuration.
+///
+/// This does not require a table or FTS index. Use
+/// [`crate::index::scalar::FtsIndexBuilder`] to supply the same tokenizer
+/// options used when creating an FTS index.
+pub fn tokenize(query: &str, params: &InvertedIndexParams) -> Result<Vec<FtsToken>> {
+    let mut tokenizer = params.build().map_err(|err| Error::InvalidInput {
+        message: format!("Failed to build tokenizer: {}", err),
+    })?;
+    let tokens = collect_query_tokens(query, &mut tokenizer);
+    Ok((0..tokens.len())
+        .map(|idx| FtsToken {
+            text: tokens.get_token(idx).to_string(),
+            position: tokens.position(idx),
+        })
+        .collect())
+}
+
 /// A trait for anything "table-like".  This is used for both native tables (which target
 /// Lance datasets) and remote tables (which target LanceDB cloud)
 ///
@@ -581,6 +612,16 @@ pub trait BaseTable: std::fmt::Display + std::fmt::Debug + Send + Sync {
             message: "unset_lsm_write_spec is not supported on this table type".into(),
         })
     }
+    /// Read the [`LsmWriteSpec`] currently installed on this table, returning
+    /// `None` when the MemWAL LSM write path is not enabled.
+    ///
+    /// The default implementation returns `NotSupported`. Implementations that
+    /// support the MemWAL LSM write path must override this.
+    async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
+        Err(Error::NotSupported {
+            message: "get_lsm_write_spec is not supported on this table type".into(),
+        })
+    }
     /// Drain and close any cached MemWAL shard writers for this table.
     ///
     /// The default implementation is a no-op; table types that maintain
@@ -1538,6 +1579,29 @@ impl Table {
         self.inner.unset_lsm_write_spec().await
     }
 
+    /// Read the [`LsmWriteSpec`] currently installed on this table.
+    ///
+    /// Returns `Ok(None)` when the MemWAL LSM write path is not enabled (no
+    /// spec has been set, or it was removed with [`Table::unset_lsm_write_spec`]).
+    /// The returned spec — including its [`LsmWriteSpec::maintained_indexes`] and
+    /// [`LsmWriteSpec::writer_config_defaults`] — mirrors what was passed to
+    /// [`Table::set_lsm_write_spec`].
+    ///
+    /// # Example
+    ///
+    /// ```
+    /// # use lancedb::table::Table;
+    /// # async fn example(table: &Table) -> Result<(), Box<dyn std::error::Error>> {
+    /// if let Some(spec) = table.get_lsm_write_spec().await? {
+    ///     println!("LSM write path enabled: {:?}", spec);
+    /// }
+    /// # Ok(())
+    /// # }
+    /// ```
+    pub async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
+        self.inner.get_lsm_write_spec().await
+    }
+
     /// Drain and close any cached MemWAL shard writers held for this table.
     ///
     /// When an [`LsmWriteSpec`] is installed, `merge_insert` opens MemWAL shard
@@ -1626,6 +1690,111 @@ impl Table {
         self.inner.list_indices().await
     }
 
+    /// Tokenize a full-text search query using the tokenizer configured on an FTS index.
+    ///
+    /// Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
+    /// the client process from index metadata. For remote tables, this means the
+    /// same tokenizer model files must also exist locally.
+    pub async fn tokenize(&self, query: &str, index_name: &str) -> Result<Vec<FtsToken>> {
+        let indices = self.inner.list_indices().await?;
+        let matches = indices
+            .iter()
+            .filter(|idx| idx.name == index_name)
+            .collect::<Vec<_>>();
+        let index = match matches.as_slice() {
+            [index] => *index,
+            [] => {
+                return Err(Error::InvalidInput {
+                    message: format!("No index named '{}'", index_name),
+                });
+            }
+            _ => {
+                return Err(Error::InvalidInput {
+                    message: format!("Index name '{}' is ambiguous", index_name),
+                });
+            }
+        };
+        if index.index_type != IndexType::FTS {
+            return Err(Error::InvalidInput {
+                message: format!("Index '{}' is not a full text search index", index_name),
+            });
+        }
+        self.tokenize_with_index(query, index, index_name)
+    }
+
+    /// Tokenize a full-text search query using the tokenizer configured on the
+    /// FTS index for a column.
+    ///
+    /// The column must have exactly one FTS index. Model-backed tokenizers such
+    /// as `jieba/*` and `lindera/*` are rebuilt in the client process from
+    /// index metadata. For remote tables, this means the same tokenizer model
+    /// files must also exist locally.
+    pub async fn tokenize_with_column(&self, query: &str, column: &str) -> Result<Vec<FtsToken>> {
+        let schema = self.inner.schema().await?;
+        let (column, _) = resolve_arrow_field_path(schema.as_ref(), column)?;
+        let indices = self.inner.list_indices().await?;
+        let matches = indices
+            .iter()
+            .filter(|idx| {
+                idx.index_type == IndexType::FTS
+                    && idx.columns.len() == 1
+                    && idx.columns[0] == column
+            })
+            .collect::<Vec<_>>();
+        let index = match matches.as_slice() {
+            [index] => *index,
+            [] => {
+                return Err(Error::InvalidInput {
+                    message: format!("Column '{}' does not have a full text search index", column),
+                });
+            }
+            _ => {
+                return Err(Error::InvalidInput {
+                    message: format!(
+                        "Column '{}' has multiple full text search indexes; tokenization by column is ambiguous",
+                        column
+                    ),
+                });
+            }
+        };
+        self.tokenize(query, &index.name).await
+    }
+
+    fn tokenize_with_index(
+        &self,
+        query: &str,
+        index: &IndexConfig,
+        index_name: &str,
+    ) -> Result<Vec<FtsToken>> {
+        let selector_description = format!("index name '{}'", index_name);
+        let details = index
+            .index_details
+            .as_deref()
+            .ok_or_else(|| Error::InvalidInput {
+                message: format!(
+                    "Full text search index '{}' for {} does not include tokenizer details",
+                    index.name, selector_description
+                ),
+            })?;
+        let params = serde_json::from_str::<InvertedIndexParams>(details).map_err(|err| {
+            Error::InvalidInput {
+                message: format!(
+                    "Failed to parse tokenizer details for full text search index '{}' for {}: {}",
+                    index.name, selector_description, err
+                ),
+            }
+        })?;
+        tokenize(query, &params).map_err(|err| match err {
+            Error::InvalidInput { message } => Error::InvalidInput {
+                message: format!(
+                    "{} for full text search index '{}' for {}",
+                    message, index.name, selector_description
+                ),
+            },
+            err => err,
+        })
+    }
+
     /// Get the table URI (storage location)
     ///
     /// Returns the full storage location of the table (e.g., S3/GCS path).
@@ -2850,6 +3019,10 @@ impl BaseTable for NativeTable {
         merge::lsm::unset_lsm_write_spec(self).await
     }
 
+    async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
+        merge::lsm::get_lsm_write_spec(self).await
+    }
+
     async fn close_lsm_writers(&self) -> Result<()> {
         merge::lsm::close_lsm_writers(self).await
     }
@@ -2930,17 +3103,21 @@ impl BaseTable for NativeTable {
             .await?
             .into_iter()
             .filter_map(|idx_desc| {
-                let index_type: crate::index::IndexType = match idx_desc.index_type().parse() {
-                    Ok(index_type) => index_type,
-                    Err(e) => {
-                        log::warn!(
-                            "Failed to parse index type for index {}: {}",
-                            idx_desc.name(),
-                            e
-                        );
-                        return None;
-                    }
-                };
+                let index_type: crate::index::IndexType = idx_desc
+                    .index_type()
+                    .parse()
+                    .unwrap_or(crate::index::IndexType::Unknown);
+                if index_type == crate::index::IndexType::Unknown {
+                    // Internal or future index types that this version doesn't recognize
+                    // (e.g. Lance's internal FragReuseIndex) are silently excluded from
+                    // the user-visible index listing.
+                    log::debug!(
+                        "Skipping unrecognized index '{}' (type '{}') in list_indices",
+                        idx_desc.name(),
+                        idx_desc.index_type(),
+                    );
+                    return None;
+                }
 
                 let field_ids = idx_desc.field_ids();
                 let mut columns = Vec::with_capacity(field_ids.len());
@@ -3007,40 +3184,83 @@ impl BaseTable for NativeTable {
     }
 
     async fn index_stats(&self, index_name: &str) -> Result<Option<IndexStatistics>> {
-        let stats = match self
-            .dataset
-            .get()
-            .await?
-            .index_statistics(index_name.as_ref())
-            .await
-        {
-            Ok(stats) => stats,
-            Err(lance_core::Error::IndexNotFound { .. }) => return Ok(None),
-            Err(e) => return Err(Error::from(e)),
+        // describe_indices() reads only manifest-level metadata (no index file I/O).
+        // VectorIndexDetails in the manifest carries distance_type for indices written
+        // by recent Lance versions. For older datasets that didn't write those details
+        // we fall back to index_statistics() for vector index types.
+        let dataset = self.dataset.get().await?;
+
+        let mut descriptions = dataset
+            .describe_indices(Some(IndexCriteria::default().with_name(index_name)))
+            .await?;
+        let Some(description) = descriptions.pop() else {
+            return Ok(None);
         };
 
-        let mut stats: IndexStatisticsImpl =
-            serde_json::from_str(&stats).map_err(|e| Error::InvalidInput {
-                message: format!("error deserializing index statistics: {}", e),
-            })?;
+        let index_type: crate::index::IndexType = description
+            .index_type()
+            .parse()
+            .unwrap_or(crate::index::IndexType::Unknown);
 
-        let first_index = stats.indices.pop().ok_or_else(|| Error::InvalidInput {
-            message: "index statistics is empty".to_string(),
-        })?;
-        // Index type should be present at one of the levels.
-        let index_type =
-            stats
-                .index_type
-                .or(first_index.index_type)
-                .ok_or_else(|| Error::InvalidInput {
-                    message: "index statistics was missing index type".to_string(),
-                })?;
-        Ok(Some(IndexStatistics {
-            num_indexed_rows: stats.num_indexed_rows,
-            num_unindexed_rows: stats.num_unindexed_rows,
+        let is_vector = matches!(
             index_type,
-            distance_type: first_index.metric_type,
-            num_indices: stats.num_indices,
+            crate::index::IndexType::IvfFlat
+                | crate::index::IndexType::IvfSq
+                | crate::index::IndexType::IvfPq
+                | crate::index::IndexType::IvfRq
+                | crate::index::IndexType::IvfHnswPq
+                | crate::index::IndexType::IvfHnswSq
+                | crate::index::IndexType::IvfHnswFlat
+        );
+
+        // details() serializes VectorIndexDetails to JSON with an uppercase "metric_type"
+        // field (e.g. "L2", "COSINE"). Parse it with a case-insensitive match.
+        let distance_type = description.details().ok().and_then(|json| {
+            #[derive(serde::Deserialize)]
+            struct Details {
+                metric_type: Option<String>,
+            }
+            serde_json::from_str::<Details>(&json)
+                .ok()
+                .and_then(|d| d.metric_type)
+                .and_then(|m| match m.to_uppercase().as_str() {
+                    "L2" => Some(DistanceType::L2),
+                    "COSINE" => Some(DistanceType::Cosine),
+                    "DOT" => Some(DistanceType::Dot),
+                    "HAMMING" => Some(DistanceType::Hamming),
+                    _ => None,
+                })
+        });
+
+        // Older Lance datasets didn't write VectorIndexDetails, so distance_type won't
+        // be in the manifest. Fall back to index_statistics() only in that case.
+        if is_vector && distance_type.is_none() {
+            let stats = dataset.index_statistics(index_name).await?;
+            let mut stats: IndexStatisticsImpl =
+                serde_json::from_str(&stats).map_err(|e| Error::InvalidInput {
+                    message: format!("error deserializing index statistics: {}", e),
+                })?;
+            let first_index = stats.indices.pop().ok_or_else(|| Error::InvalidInput {
+                message: "index statistics is empty".to_string(),
+            })?;
+            return Ok(Some(IndexStatistics {
+                num_indexed_rows: stats.num_indexed_rows,
+                num_unindexed_rows: stats.num_unindexed_rows,
+                index_type,
+                distance_type: first_index.metric_type,
+                num_indices: stats.num_indices,
+            }));
+        }
+
+        let num_indexed_rows = description.rows_indexed() as usize;
+        let total_rows = dataset.count_rows(None).await?;
+        let num_unindexed_rows = total_rows.saturating_sub(num_indexed_rows);
+        Ok(Some(IndexStatistics {
+            num_indexed_rows,
+            num_unindexed_rows,
+            index_type,
+            distance_type,
+            num_indices: Some(description.metadata().len() as u32),
         }))
     }
 
@@ -3197,6 +3417,27 @@ mod tests {
     use crate::query::{ExecutableQuery, QueryBase};
     use crate::test_utils::connection::new_test_connection;
 
+    #[test]
+    fn test_tokenize_uses_explicit_simple_tokenizer() {
+        let params =
+            crate::index::scalar::FtsIndexBuilder::default().base_tokenizer("simple".to_string());
+        let tokens = crate::tokenize("Running in cafés", &params).unwrap();
+
+        assert_eq!(
+            tokens,
+            vec![
+                FtsToken {
+                    text: "run".to_string(),
+                    position: 0,
+                },
+                FtsToken {
+                    text: "cafe".to_string(),
+                    position: 2,
+                },
+            ]
+        );
+    }
+
     #[tokio::test]
     async fn test_open() {
         let tmp_dir = tempdir().unwrap();
@@ -4411,6 +4652,67 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_get_lsm_write_spec() {
+        let tmp_dir = tempdir().unwrap();
+        let uri = tmp_dir.path().to_str().unwrap();
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", DataType::Int64, false),
+            Field::new("region", DataType::Utf8, true),
+        ]));
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(arrow_array::Int64Array::from(vec![1, 2, 3])),
+                Arc::new(StringArray::from(vec!["a", "b", "c"])),
+            ],
+        )
+        .unwrap();
+        let reader: Box<dyn arrow_array::RecordBatchReader + Send> =
+            Box::new(RecordBatchIterator::new(vec![Ok(batch)], schema.clone()));
+        let conn = ConnectBuilder::new(uri)
+            .read_consistency_interval(Duration::from_secs(0))
+            .execute()
+            .await
+            .unwrap();
+        let table = conn.create_table("t", reader).execute().await.unwrap();
+
+        // No spec installed yet.
+        assert_eq!(table.get_lsm_write_spec().await.unwrap(), None);
+
+        // A real scalar index is needed to name it as a maintained index.
+        table
+            .create_index(&["id"], Index::Auto)
+            .execute()
+            .await
+            .unwrap();
+        let idx_name = table.list_indices().await.unwrap()[0].name.clone();
+
+        // Bucket spec round-trips exactly, including the routing column (recovered
+        // from its field id), maintained indexes, and writer config defaults.
+        let spec = LsmWriteSpec::bucket("id", 4)
+            .with_maintained_indexes([idx_name])
+            .with_writer_config_defaults([("durable_write", "false")]);
+        table.set_lsm_write_spec(spec.clone()).await.unwrap();
+        assert_eq!(table.get_lsm_write_spec().await.unwrap(), Some(spec));
+
+        // After unset, no spec is reported.
+        table.unset_lsm_write_spec().await.unwrap();
+        assert_eq!(table.get_lsm_write_spec().await.unwrap(), None);
+
+        // Identity sharding round-trips (column recovered from the schema).
+        let spec = LsmWriteSpec::identity("region");
+        table.set_lsm_write_spec(spec.clone()).await.unwrap();
+        assert_eq!(table.get_lsm_write_spec().await.unwrap(), Some(spec));
+        table.unset_lsm_write_spec().await.unwrap();
+
+        // Unsharded round-trips (no routing column).
+        let spec = LsmWriteSpec::unsharded();
+        table.set_lsm_write_spec(spec.clone()).await.unwrap();
+        assert_eq!(table.get_lsm_write_spec().await.unwrap(), Some(spec));
+    }
+
     #[tokio::test]
     pub async fn test_stats() {
         let tmp_dir = tempdir().unwrap();

rust/lancedb/src/table/create_index.rs

@@ -589,6 +589,7 @@ mod tests {
         let stats = table.index_stats(index_name).await.unwrap().unwrap();
         assert_eq!(stats.num_indexed_rows, 512);
         assert_eq!(stats.num_unindexed_rows, 0);
+        assert_eq!(stats.distance_type, Some(crate::DistanceType::L2));
     }
 
     #[tokio::test]
@@ -646,6 +647,7 @@ mod tests {
         let stats = table.index_stats(index_name).await.unwrap().unwrap();
         assert_eq!(stats.num_indexed_rows, 512);
         assert_eq!(stats.num_unindexed_rows, 0);
+        assert_eq!(stats.distance_type, Some(crate::DistanceType::L2));
     }
 
     #[tokio::test]
@@ -690,6 +692,10 @@ mod tests {
         assert_eq!(index.index_type, crate::index::IndexType::IvfHnswFlat);
         assert_eq!(index.columns, vec!["embeddings".to_string()]);
         assert_eq!(table.count_rows(None).await.unwrap(), 512);
+        let stats = table.index_stats(&index.name).await.unwrap().unwrap();
+        assert_eq!(stats.num_indexed_rows, 512);
+        assert_eq!(stats.num_unindexed_rows, 0);
+        assert_eq!(stats.distance_type, Some(crate::DistanceType::L2));
     }
 
     #[tokio::test]
@@ -747,6 +753,15 @@ mod tests {
         let stats = table.index_stats(index_name).await.unwrap().unwrap();
         assert_eq!(stats.num_indexed_rows, 1);
         assert_eq!(stats.num_unindexed_rows, 0);
+        assert_eq!(stats.index_type, crate::index::IndexType::BTree);
+        assert_eq!(stats.distance_type, None);
+
+        // Rows added after the index was built appear as unindexed.
+        let new_batch = record_batch!(("i", Int32, [2])).unwrap();
+        table.add(new_batch).execute().await.unwrap();
+        let stats = table.index_stats(index_name).await.unwrap().unwrap();
+        assert_eq!(stats.num_indexed_rows, 1);
+        assert_eq!(stats.num_unindexed_rows, 1);
     }
 
     #[tokio::test]
@@ -795,6 +810,12 @@ mod tests {
             .map(|b| b.num_rows())
             .sum::<usize>();
         assert_eq!(count, 1);
+
+        let stats = table.index_stats("text_idx").await.unwrap().unwrap();
+        assert_eq!(stats.num_indexed_rows, 1);
+        assert_eq!(stats.num_unindexed_rows, 0);
+        assert_eq!(stats.index_type, crate::index::IndexType::Fm);
+        assert_eq!(stats.distance_type, None);
     }
 
     #[tokio::test]
@@ -1188,6 +1209,12 @@ mod tests {
         let index = configs_iter.next().unwrap();
         assert_eq!(index.index_type, crate::index::IndexType::Bitmap);
         assert_eq!(index.columns, vec!["large_data".to_string()]);
+
+        let stats = table.index_stats("category_idx").await.unwrap().unwrap();
+        assert_eq!(stats.num_indexed_rows, 100);
+        assert_eq!(stats.num_unindexed_rows, 0);
+        assert_eq!(stats.index_type, crate::index::IndexType::Bitmap);
+        assert_eq!(stats.distance_type, None);
     }
 
     #[tokio::test]
@@ -1256,6 +1283,12 @@ mod tests {
         let index = index_configs.into_iter().next().unwrap();
         assert_eq!(index.index_type, crate::index::IndexType::LabelList);
         assert_eq!(index.columns, vec!["tags".to_string()]);
+
+        let stats = table.index_stats("tags_idx").await.unwrap().unwrap();
+        assert_eq!(stats.num_indexed_rows, 40);
+        assert_eq!(stats.num_unindexed_rows, 0);
+        assert_eq!(stats.index_type, crate::index::IndexType::LabelList);
+        assert_eq!(stats.distance_type, None);
     }
 
     #[tokio::test]

rust/lancedb/src/table/merge/lsm.rs

@@ -32,7 +32,7 @@ use lance::dataset::mem_wal::{
 };
 use lance::index::DatasetIndexExt;
 use lance_core::datatypes::Schema as LanceSchema;
-use lance_index::mem_wal::{MemWalIndexDetails, ShardingSpec};
+use lance_index::mem_wal::{MemWalIndexDetails, ShardingField, ShardingSpec};
 use tokio::sync::RwLock;
 use uuid::Uuid;
 
@@ -148,6 +148,97 @@ pub(crate) async fn unset_lsm_write_spec(table: &NativeTable) -> Result<()> {
     Ok(())
 }
 
+// =============================================================================
+// get_lsm_write_spec
+// =============================================================================
+
+/// Read the [`LsmWriteSpec`] currently installed on the table, if any.
+///
+/// Reconstructs the spec from the MemWAL index's stored details — the same
+/// metadata [`set_lsm_write_spec`] writes — read directly via
+/// [`mem_wal_index_details`](DatasetMemWalExt::mem_wal_index_details) (the raw
+/// manifest record, not the `describe_indices` enrichment path, so it is
+/// unaffected by system-index filtering there). Returns `Ok(None)` when no
+/// MemWAL index is installed.
+#[allow(clippy::redundant_pub_crate)]
+pub(crate) async fn get_lsm_write_spec(table: &NativeTable) -> Result<Option<LsmWriteSpec>> {
+    let dataset = table.dataset.get().await?;
+    let Some(details) = dataset.mem_wal_index_details().await? else {
+        return Ok(None);
+    };
+    let spec = lsm_write_spec_from_details(&details, dataset.schema())?;
+    Ok(Some(spec))
+}
+
+/// Reconstruct the public [`LsmWriteSpec`] from stored MemWAL index details.
+///
+/// The sharding transform and its parameters recover the mode; the routing
+/// column is resolved from the sharding field's source id back to its schema
+/// name (only bucket / identity carry a column). `maintained_indexes` and
+/// `writer_config_defaults` are copied verbatim, so the result round-trips
+/// what `set_lsm_write_spec` installed.
+fn lsm_write_spec_from_details(
+    details: &MemWalIndexDetails,
+    schema: &LanceSchema,
+) -> Result<LsmWriteSpec> {
+    let spec = details
+        .sharding_specs
+        .first()
+        .ok_or_else(|| Error::Runtime {
+            message: "get_lsm_write_spec: MemWAL index has no sharding spec".to_string(),
+        })?;
+    let field = spec.fields.first().ok_or_else(|| Error::Runtime {
+        message: "get_lsm_write_spec: MemWAL index has an empty sharding spec".to_string(),
+    })?;
+
+    let base = match field.transform.as_deref() {
+        Some(BUCKET_TRANSFORM) => {
+            let num_buckets = field
+                .parameters
+                .get(NUM_BUCKETS_PARAM)
+                .and_then(|s| s.parse::<u32>().ok())
+                .filter(|n| *n > 0)
+                .ok_or_else(|| Error::Runtime {
+                    message: "get_lsm_write_spec: MemWAL bucket spec has a missing or invalid num_buckets parameter".to_string(),
+                })?;
+            LsmWriteSpec::bucket(sharding_column(field, schema)?, num_buckets)
+        }
+        Some(IDENTITY_TRANSFORM) => LsmWriteSpec::identity(sharding_column(field, schema)?),
+        Some(UNSHARDED_TRANSFORM) => LsmWriteSpec::unsharded(),
+        other => {
+            return Err(Error::Runtime {
+                message: format!(
+                    "get_lsm_write_spec: MemWAL index has an unsupported sharding transform {:?}",
+                    other
+                ),
+            });
+        }
+    };
+
+    Ok(base
+        .with_maintained_indexes(details.maintained_indexes.clone())
+        .with_writer_config_defaults(details.writer_config_defaults.clone()))
+}
+
+/// Resolve the single routing column name from a sharding field's source id.
+///
+/// `set_lsm_write_spec` records the shard column by its Lance field id, so the
+/// name is recovered by looking that id up in the current schema.
+fn sharding_column(field: &ShardingField, schema: &LanceSchema) -> Result<String> {
+    let source_id = *field.source_ids.first().ok_or_else(|| Error::Runtime {
+        message: "get_lsm_write_spec: sharding field has no source column".to_string(),
+    })?;
+    schema
+        .field_by_id(source_id)
+        .map(|f| f.name.clone())
+        .ok_or_else(|| Error::Runtime {
+            message: format!(
+                "get_lsm_write_spec: sharding source field id {} not found in schema",
+                source_id
+            ),
+        })
+}
+
 // =============================================================================
 // close_lsm_writers
 // =============================================================================