test: update python expectations for lance 9.1

test: relax hash split distribution assertions
chore: update lance dependency to v9.1.0-beta.2
2026-08-27 08:28:28 +00:00 · 2026-07-16 13:46:54 -07:00 · 2026-07-16 10:45:35 -07:00 · 2026-07-16 10:45:05 -07:00 · 2026-07-16 10:31:30 -07:00 · 2026-07-15 21:21:40 -07:00
56 changed files with 1083 additions and 723 deletions
@@ -1,145 +0,0 @@
---
-name: lancedb-branch-ops
-description: >-
-  Manage LanceDB table branches through the REST API: list, create, and delete
-  branches; target schema reads, field-metadata updates, and index creation to a
-  named branch; and verify that branch changes remain isolated from main. Use
-  when a task involves branch lifecycle, an experimental or isolated table
-  version, directing an operation to a non-main branch, or confirming that a
-  mutation did not affect main. This skill also explains that LanceDB has no
-  checkout operation; each request selects its target branch in the request
-  body.
---
-
-## Goal
-
-Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main.
-
-## Step 0: Establish the connection
-
-Use the `lancedb-connect` skill to resolve the base URL and auth headers (`x-api-key`, `x-lancedb-database`). Skip this only if the connection is already known from the current conversation.
-
-All examples below use `{base_url}` — substitute the resolved endpoint and include the auth headers on every request.
-
-## The branch model (important)
-
-LanceDB branches are named snapshots that diverge from the table's current state at creation time. There is **no checkout command** — you never switch the whole table to a branch. Instead, you **pass `"branch": "<name>"` in the request body** of any operation to target that branch. Omitting the key (or sending an empty body) always targets main.
-
-`branches/list` returns only non-main branches. Main always exists and is not listed.
-
-## List branches
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/list
-Content-Type: application/json
-
-{}
-```
-
-Response:
-```json
-{
-  "branches": {
-    "experiment-reindex": {"parentVersion": 1, "createAt": 1782506085, "manifestSize": 1029}
-  }
-}
-```
-
-If `branches` is `{}`, the table has no branches besides main.
-
-## Create a branch
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/create
-Content-Type: application/json
-
-{"name": "experiment-reindex"}
-```
-
-HTTP 200 with `{}` body = success. The branch is created off the table's current state on main.
-
-Verify by calling `branches/list` and confirming the new name appears.
-
-## Delete a branch
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/delete
-Content-Type: application/json
-
-{"name": "stale-2024"}
-```
-
-HTTP 200 with `{}` body = success. Only the branch pointer is removed — main and all row data remain intact.
-
-Verify by calling `branches/list` (name gone) and `describe` with no branch param (main still responds).
-
-## Operate on a specific branch
-
-Pass `"branch": "<name>"` in the body of any operation to scope it to that branch:
-
-**Read schema on a branch:**
-```http
-POST {base_url}/v1/table/{table_id}/describe
-Content-Type: application/json
-
-{"branch": "wip-branch"}
-```
-
-**Write metadata to a branch (not main):**
-```http
-POST {base_url}/v1/table/{table_id}/update_field_metadata
-Content-Type: application/json
-
-{
-  "branch": "wip-branch",
-  "updates": [
-    {
-      "path": "category",
-      "metadata": {"lancedb:description": "Product category label."},
-      "replace": false
-    }
-  ]
-}
-```
-
-**Build an index on a branch:**
-```http
-POST {base_url}/v1/table/{table_id}/create_index
-Content-Type: application/json
-
-{
-  "branch": "wip-branch",
-  "column": "category",
-  "index_type": "BTREE"
-}
-```
-
-## Verifying isolation
-
-After writing to a branch, always confirm the change did NOT land on main:
-
-```bash
-# Should show the new metadata
-curl -s -X POST {base_url}/v1/table/{table_id}/describe \
-  -H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
-  -H "content-type: application/json" \
-  -d '{"branch": "wip-branch"}'
-
-# Should NOT show the new metadata
-curl -s -X POST {base_url}/v1/table/{table_id}/describe \
-  -H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
-  -H "content-type: application/json" \
-  -d '{}'
-```
-
-## Quick reference
-
-| Goal | Endpoint | Body |
-|------|----------|------|
-| List all branches | `branches/list` | `{}` |
-| Create a branch | `branches/create` | `{"name": "..."}` |
-| Delete a branch | `branches/delete` | `{"name": "..."}` |
-| Read schema on branch | `describe` | `{"branch": "..."}` |
-| Write metadata on branch | `update_field_metadata` | `{"branch": "...", "updates": [...]}` |
-| Build index on branch | `create_index` | `{"branch": "...", "column": ..., "index_type": ...}` |
-| Target main (default) | any endpoint | omit `"branch"` key |
@@ -1,178 +0,0 @@
---
-name: lancedb-column-metadata
-description: Column metadata authoring for LanceDB tables via the REST API. This skill is required for tasks like writing field descriptions, setting tags on columns (field_type, model, project_id, version), classifying columns as embeddings vs labels vs eval metrics, or grouping versioned columns into logical families — because it has the API integration needed to read the schema and persist metadata back. Invoke whenever someone wants to document, annotate, tag, or classify what their table columns ARE. Trigger even without an explicit "LanceDB" mention, as long as the context is column-level documentation or tagging for an ML or vector database table.
-metadata:
-  short-description: Write column descriptions, tags, and logical groupings to a LanceDB table
---
-
-## Overview
-
-This skill authors column-level metadata for a LanceDB table. It connects to a LanceDB deployment over its REST API, inspects the table schema, generates appropriate metadata, and writes it back.
-
-## Step 0: Establish the connection
-
-Use the `lancedb-connect` skill (invoke it via the Skill tool) to resolve the base URL and auth headers (`x-api-key`, `x-lancedb-database`) for whichever deployment the user is working against — enterprise/self-hosted or a local dev server. Skip it only if the connection details are already established in the conversation.
-
-All examples below use `{base_url}` — substitute the resolved endpoint and include the resolved headers on every request.
-
-## Metadata keys
-
-All metadata uses namespaced keys:
-
-| Key | Purpose | Example value |
-|-----|---------|---------------|
-| `lancedb:description` | Human-readable explanation of what the column contains | `"CLIP ViT-L/14 image embedding, L2-normalized (768-dim)"` |
-| `lancedb:tag:<name>` | Flexible key-value tag; the suffix names the tag category | `lancedb:tag:field_type: "embedding"`, `lancedb:tag:model: "clip"`, `lancedb:tag:project_id: "foo"` |
-| `lancedb:logical-column` | Logical group/family this column belongs to | `"clip_features"` |
-
-Tags are open-ended — use whatever key suffix and value make sense given the user's intent. The tag suffix should describe *what is being classified* (e.g., `field_type`, `model`, `project_id`) and the value describes *how*.
-
-## Step 1: Resolve the table identifier
-
-You need:
- **Table name** (required) — e.g., `my_table` or `my_namespace.my_table`
- **Database name** — ask if not provided and not inferable from context; it goes in the `x-lancedb-database` header, never in the URL path
-
-The table identifier in the URL path is typically `table_name` for a top-level table, or `namespace$table_name` if the table lives in a namespace. The API accepts a `delimiter` query parameter to parse compound identifiers (default `$`).
-
-## Step 2: Describe the table
-
-```http
-POST {base_url}/v1/table/{table_id}/describe
-Content-Type: application/json
-
-{}
-```
-
-The response contains `schema.fields` — an array of field objects:
-
-```json
-{
-  "schema": {
-    "fields": [
-      {
-        "name": "clip_embedding_v3",
-        "type": { "type": "FixedSizeList", "fields": [...], "listSize": 768 },
-        "nullable": true,
-        "metadata": { "lancedb:description": "..." }
-      }
-    ]
-  }
-}
-```
-
-Each field has:
- `name` — field name
- `type` — Arrow data type (check `type.type` for the type string)
- `nullable` — boolean
- `metadata` — existing key-value metadata (read this before writing to avoid redundant updates)
-
-For struct/nested fields, recurse into `type.fields` and represent them as dot-notation paths (e.g., `parent.child`).
-
-If the user hasn't specified which columns to update, work with all columns.
-
-## Step 3: Generate metadata
-
-Decide what to generate based on the user's request.
-
-### Writing descriptions (`lancedb:description`)
-
-Base descriptions on:
- The column name and Arrow type (e.g., `FixedSizeList` of floats → likely an embedding)
- User-supplied context (upstream pipeline, sample values, domain knowledge)
- Name patterns: `_embedding`/`_vec`/`_embed` → vector; `_label`/`_class` → label; `_score`/`_eval`/`_metric` → evaluation metric
-
-Be specific and concise. Good: `"Sentence-BERT embedding of the query text (768-dim)."` Not: `"An embedding column."`
-
-### Tagging columns (`lancedb:tag:<name>`)
-
-Choose tag key names that match what the user asked to annotate. Common patterns:
-
- Semantic field type → `lancedb:tag:field_type: "embedding"` / `"text"` / `"image"` / `"label"` / `"eval"` / `"id"` / `"metadata"`
- Model or source → `lancedb:tag:model: "clip"` / `"bert"` / `"vit"`
- Project affiliation → `lancedb:tag:project_id: "<name>"`
- Version → `lancedb:tag:version: "v3"` (and `lancedb:tag:latest: "true"` for the newest)
-
-Use Arrow type as a hint: `FixedSizeList` + float → embedding; `Utf8`/`LargeUtf8` → text; `Binary` → image or blob.
-
-Multiple tags on the same column are fine — each is a separate key.
-
-### Grouping into logical columns (`lancedb:logical-column`)
-
-Look for naming patterns across columns:
- `clip_v1`, `clip_v2`, `clip_v3` → logical column `"clip"`, latest is `v3`
- `text_embed_20240101`, `text_embed_20240601` → logical column `"text_embed"`, latest is the most recent date suffix
-
-Write `lancedb:logical-column` on all members of a group. Mark the newest with `lancedb:tag:latest: "true"` (in addition to its version tag).
-
-## Step 4: Write the metadata
-
-```http
-POST {base_url}/v1/table/{table_id}/update_field_metadata
-Content-Type: application/json
-
-{
-  "updates": [
-    {
-      "path": "clip_v3",
-      "metadata": {
-        "lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
-        "lancedb:tag:field_type": "embedding",
-        "lancedb:tag:model": "clip",
-        "lancedb:tag:version": "v3",
-        "lancedb:tag:latest": "true",
-        "lancedb:logical-column": "clip"
-      },
-      "replace": false
-    },
-    {
-      "path": "clip_v2",
-      "metadata": {
-        "lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
-        "lancedb:tag:field_type": "embedding",
-        "lancedb:tag:model": "clip",
-        "lancedb:tag:version": "v2",
-        "lancedb:logical-column": "clip"
-      },
-      "replace": false
-    }
-  ]
-}
-```
-
-Rules:
- **Use `"replace": false`** (merge) by default — this preserves existing metadata the user didn't ask to change
- Use `"replace": true` only if the user explicitly asks to overwrite all existing metadata on a column
- Set a value to `null` to delete a specific key
- Batch all updates in a single request when possible
-
-The response includes `version` (new table version) and `fields` (the updated metadata per field).
-
-## Step 5: Confirm
-
-Report back:
- Which columns were updated and what was written
- The new table version number
- Any columns skipped (e.g., already had up-to-date metadata)
-
---
-
-## Quick examples
-
-**"Write descriptions for all columns in the `product_embeddings` table"**
-1. POST `/v1/table/product_embeddings/describe` → get all fields
-2. Generate a `lancedb:description` for each column based on name + type
-3. POST `update_field_metadata` with descriptions
-4. Report
-
-**"Tag the columns in `model_outputs` with their field type and model"**
-1. Describe `model_outputs`
-2. For each field, classify by name + Arrow type → set `lancedb:tag:field_type` and `lancedb:tag:model` where applicable
-3. POST `update_field_metadata`
-4. Report
-
-**"Group the feature columns in `training_features` into logical families and mark the latest version"**
-1. Describe the table
-2. Find version patterns → assign `lancedb:logical-column` and `lancedb:tag:version`; mark newest with `lancedb:tag:latest: "true"`
-3. POST `update_field_metadata`
-4. Show the grouping
@@ -1,42 +0,0 @@
---
-name: lancedb-connect
-description: Resolve how to connect to a LanceDB deployment over the REST API — figure out the base URL, API key, and database header. Use this before making any REST requests to a LanceDB table, whenever the endpoint or auth setup is not already known. Also useful on its own when someone asks how to connect, authenticate, or curl their LanceDB instance.
-metadata:
-  short-description: Resolve the base URL and auth headers for a LanceDB deployment
---
-
-## Goal
-
-Produce two things every REST request needs:
-
-1. **Base URL** — the endpoint
-2. **Headers** — `x-api-key`, and usually `x-lancedb-database`
-
-## Resolution steps
-
-1. If the user already gave a URL and API key (or said which environment they're working against), use that.
-2. Otherwise, look for credentials already available in the environment:
-   - Env vars like `LANCEDB_URI` / `LANCEDB_HOST` / `LANCEDB_API_KEY`
-   - A LanceDB endpoint already running or port-forwarded locally (the REST default port is 2333, i.e. `http://localhost:2333`)
-3. If you didn't find both pieces, ask the user directly: **"What's your LanceDB endpoint's URL, and what's your API key?"** Also ask which database to use if it isn't obvious. Don't guess or probe further — the user knows their deployment.
-
-## Validating the connection
-
-Make a cheap authenticated request and check the status:
-
-```bash
-curl -s -w "\n%{http_code}" "{base_url}/v1/table/?limit=1" \
-  -H "x-api-key: <key>" \
-  -H "x-lancedb-database: <database>"
-```
-
- `200` — connection, key, and database header all good
- `401` — API key missing or wrong
- `400` mentioning a database header — this deployment expects `x-lancedb-database`
-
-## Non-REST equivalents
-
-If the caller would rather use the SDK or CLI than raw REST, the same credentials work:
-
- Python SDK: `lancedb.connect("db://<database>", api_key="<key>", host_override="<base_url>")`
- `lancedb` CLI: a `[profiles.<name>]` entry in `~/.lancedb/config.toml` with `http_server_url`, `api_key`, `database`
@@ -27,7 +27,9 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
   - TypeScript patterns: `references/typescript/patterns.md`
   - TypeScript API quick reference: `references/typescript/api_reference.md`
   - TypeScript performance guidance: `references/typescript/performance.md`
-4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets.
+   - Column metadata authoring (both SDKs): `references/column_metadata.md`
+   - Branch operations (both SDKs): `references/branch_ops.md`
+4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main.
 5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
 6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.
 7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
@@ -0,0 +1,117 @@
+# Branch Operations
+
+Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main. Use for branch lifecycle tasks, experimental/isolated table versions, targeting an operation at a non-main branch, or confirming a mutation did not affect main.
+
+Works on local/OSS and remote Enterprise/Cloud tables.
+
+## The branch model (important)
+
+Branches are isolated, writable lines of history forked from another branch (or a specific version). Writes on a branch never affect `main`.
+
+There is **no global "switch branch" state** — you never repoint the whole table at a branch. Instead, **operations are scoped by which table handle you use**:
+
+- The handle you got from `open_table(name)` / `openTable(name)` targets `main`.
+- `branches.create(...)` and `branches.checkout(...)` return a **new table handle scoped to that branch**. Every read/write on that handle (add, update, `update_field_metadata`, `create_index`, search, …) lands on the branch.
+- The original main handle is unaffected — keep it around to verify isolation.
+
+`branches.list()` returns only non-main branches. Main always exists and is not listed.
+
+## Python
+
+`table.branches` is a property returning the branch manager; `table.current_branch()` tells you what a handle is scoped to (`None` = main).
+
+```python
+table = db.open_table("products")            # scoped to main
+
+# list — dict of name -> metadata (parent_branch, parent_version, ...); {} = only main
+table.branches.list()
+
+# create: forks from main by default and returns a handle scoped to the new branch
+exp = table.branches.create("experiment-reindex")
+exp = table.branches.create("exp2", from_ref="main", from_version=None)  # optional fork point
+
+# checkout an existing branch -> branch-scoped handle
+wip = table.branches.checkout("wip-branch")
+# with version= it pins to that version (read-only detached view); omit to track latest, writable
+
+# operate on the branch simply by using its handle
+wip.update_field_metadata(
+    {"path": "category", "metadata": {"lancedb:description": "Product category label."}}
+)
+wip.create_scalar_index("category")
+
+# delete: removes only the branch pointer; main and row data remain intact
+table.branches.delete("stale-2024")
+
+# alternatively, open a branch handle directly from the connection
+wip = db.open_table("products", branch="wip-branch")
+
+exp.current_branch()    # "experiment-reindex"
+table.current_branch()  # None (main)
+```
+
+Async: same shape — `table.branches` returns `AsyncBranches`; `await table.branches.create(...)` etc.
+
+## TypeScript
+
+`table.branches()` is an **async method** returning the `Branches` manager; `table.currentBranch()` returns the scoped branch or `null` for main.
+
+```typescript
+const table = await db.openTable("products"); // scoped to main
+const branches = await table.branches();
+
+// list — Record<string, BranchContents>; {} = only main
+await branches.list();
+
+// create: forks from main by default, returns a Table scoped to the new branch
+const exp = await branches.create("experiment-reindex");
+const exp2 = await branches.create("exp2", "main" /* fromRef */, undefined /* fromVersion */);
+
+// checkout an existing branch -> branch-scoped Table
+const wip = await branches.checkout("wip-branch");
+// with a version arg it pins (read-only detached view); omit to track latest, writable
+
+// operate on the branch simply by using its handle
+await wip.updateFieldMetadata([
+  { path: "category", metadata: { "lancedb:description": "Product category label." } },
+]);
+await wip.createIndex("category");
+
+// delete: removes only the branch pointer; main and row data remain intact
+await branches.delete("stale-2024");
+
+// alternatively, open a branch handle directly from the connection
+const wip2 = await db.openTable("products", { branch: "wip-branch" });
+
+exp.currentBranch();   // "experiment-reindex"
+table.currentBranch(); // null (main)
+```
+
+## Verifying isolation
+
+After writing to a branch, confirm the change did NOT land on main by reading through both handles:
+
+```python
+wip = table.branches.checkout("wip-branch")
+wip.update_field_metadata({"path": "category", "metadata": {"lancedb:description": "..."}})
+
+assert b"lancedb:description" in (wip.schema.field("category").metadata or {})
+assert b"lancedb:description" not in (table.schema.field("category").metadata or {})  # main untouched
+```
+
+Two handles on the same branch see each other's writes (e.g. `table.branches.create("exp")` and `db.open_table(name, branch="exp")`); main stays isolated.
+
+## Quick reference
+
+| Goal | Python | TypeScript |
+|------|--------|------------|
+| List branches (non-main) | `table.branches.list()` | `await (await table.branches()).list()` |
+| Create branch (off main) | `table.branches.create(name)` → branch handle | `await branches.create(name)` → branch `Table` |
+| Create from a fork point | `table.branches.create(name, from_ref=..., from_version=...)` | `await branches.create(name, fromRef, fromVersion)` |
+| Get a branch handle | `table.branches.checkout(name)` or `db.open_table(t, branch=name)` | `await branches.checkout(name)` or `await db.openTable(t, { branch: name })` |
+| Pin to a branch version (read-only) | `table.branches.checkout(name, version=v)` | `await branches.checkout(name, v)` |
+| Delete branch | `table.branches.delete(name)` | `await branches.delete(name)` |
+| Which branch is this handle on? | `table.current_branch()` (`None` = main) | `table.currentBranch()` (`null` = main) |
+| Target main | use the original (non-branch) handle | use the original (non-branch) handle |
+
+Branch names must be non-empty; empty names raise a validation error.
@@ -0,0 +1,183 @@
+# Column Metadata Authoring
+
+Write column-level descriptions, tags, and logical groupings onto a LanceDB table's schema. Use this when the user wants to document, annotate, tag, or classify what their table columns ARE (embeddings vs labels vs eval metrics, model provenance, version families, etc.).
+
+Works on local/OSS and remote Enterprise/Cloud tables alike — read the schema through the table handle, write through `update_field_metadata` (Python) / `updateFieldMetadata` (TypeScript).
+
+## Metadata key conventions
+
+All metadata uses namespaced keys:
+
+| Key | Purpose | Example value |
+|-----|---------|---------------|
+| `lancedb:description` | Human-readable explanation of what the column contains | `"CLIP ViT-L/14 image embedding, L2-normalized (768-dim)"` |
+| `lancedb:tag:<name>` | Flexible key-value tag; the suffix names the tag category | `lancedb:tag:field_type: "embedding"`, `lancedb:tag:model: "clip"`, `lancedb:tag:project_id: "foo"` |
+| `lancedb:logical-column` | Logical group/family this column belongs to | `"clip_features"` |
+
+Tags are open-ended — use whatever key suffix and value make sense given the user's intent. The tag suffix should describe *what is being classified* (e.g., `field_type`, `model`, `project_id`) and the value describes *how*. Multiple tags on the same column are fine — each is a separate key. All values are strings.
+
+## Step 1: Read the schema and existing metadata
+
+Read existing metadata before writing, to avoid redundant updates.
+
+Python — `table.schema` (sync property; async: `await table.schema()`) returns a `pyarrow.Schema`. **Arrow field metadata is bytes-keyed in Python**:
+
+```python
+schema = table.schema
+for field in schema:
+    meta = field.metadata or {}  # dict[bytes, bytes], e.g. {b"lancedb:description": b"..."}
+    print(field.name, field.type, field.nullable, meta)
+```
+
+TypeScript — `await table.schema()` returns an Arrow `Schema`; field metadata is a `Map<string, string>`:
+
+```typescript
+const schema = await table.schema();
+for (const field of schema.fields) {
+  console.log(field.name, field.type, field.nullable, field.metadata); // Map
+  // field.metadata.get("lancedb:description")
+}
+```
+
+For struct/nested fields, recurse into the field's children and address them as dot-paths (e.g., `parent.child`).
+
+If the user hasn't specified which columns to update, work with all columns.
+
+## Step 2: Generate metadata
+
+Decide what to generate based on the user's request.
+
+### Descriptions (`lancedb:description`)
+
+Base descriptions on:
+- The column name and Arrow type (e.g., `FixedSizeList` of floats → likely an embedding)
+- User-supplied context (upstream pipeline, sample values, domain knowledge)
+- Name patterns: `_embedding`/`_vec`/`_embed` → vector; `_label`/`_class` → label; `_score`/`_eval`/`_metric` → evaluation metric
+
+Be specific and concise. Good: `"Sentence-BERT embedding of the query text (768-dim)."` Not: `"An embedding column."`
+
+### Tags (`lancedb:tag:<name>`)
+
+Choose tag key names that match what the user asked to annotate. Common patterns:
+
+- Semantic field type → `lancedb:tag:field_type: "embedding"` / `"text"` / `"image"` / `"label"` / `"eval"` / `"id"` / `"metadata"`
+- Model or source → `lancedb:tag:model: "clip"` / `"bert"` / `"vit"`
+- Project affiliation → `lancedb:tag:project_id: "<name>"`
+- Version → `lancedb:tag:version: "v3"` (and `lancedb:tag:latest: "true"` for the newest)
+
+Use Arrow type as a hint: `FixedSizeList` + float → embedding; `Utf8`/`LargeUtf8` → text; `Binary` → image or blob.
+
+### Logical groupings (`lancedb:logical-column`)
+
+Look for naming patterns across columns:
+- `clip_v1`, `clip_v2`, `clip_v3` → logical column `"clip"`, latest is `v3`
+- `text_embed_20240101`, `text_embed_20240601` → logical column `"text_embed"`, latest is the most recent date suffix
+
+Write `lancedb:logical-column` on all members of a group. Mark the newest with `lancedb:tag:latest: "true"` (in addition to its version tag).
+
+## Step 3: Write the metadata
+
+Each update names a field by dot-path and carries a metadata map. Semantics (identical in both SDKs):
+
+- **Merge by default** (`replace` omitted/false) — preserves existing metadata the user didn't ask to change
+- `replace: true` swaps the field's entire metadata map — only if the user explicitly asks to overwrite
+- A value of `None`/`null` deletes that specific key
+- Batch all field updates into a single call when possible
+- Returns the new table version
+
+Python (sync and async take one dict per field, as varargs):
+
+```python
+res = table.update_field_metadata(
+    {
+        "path": "clip_v3",
+        "metadata": {
+            "lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
+            "lancedb:tag:field_type": "embedding",
+            "lancedb:tag:model": "clip",
+            "lancedb:tag:version": "v3",
+            "lancedb:tag:latest": "true",
+            "lancedb:logical-column": "clip",
+        },
+    },
+    {
+        "path": "clip_v2",
+        "metadata": {
+            "lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
+            "lancedb:tag:field_type": "embedding",
+            "lancedb:tag:model": "clip",
+            "lancedb:tag:version": "v2",
+            "lancedb:logical-column": "clip",
+        },
+    },
+)
+print(res.version)  # new table version
+
+# merge semantics: add a key, delete one via None, keep the rest
+table.update_field_metadata(
+    {"path": "clip_v2", "metadata": {"lancedb:tag:archived": "true", "lancedb:tag:latest": None}}
+)
+```
+
+(`replace_field_metadata` is deprecated — use `update_field_metadata`.)
+
+TypeScript (takes an array of `FieldMetadataUpdate`):
+
+```typescript
+const res = await table.updateFieldMetadata([
+  {
+    path: "clip_v3",
+    metadata: {
+      "lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
+      "lancedb:tag:field_type": "embedding",
+      "lancedb:tag:model": "clip",
+      "lancedb:tag:version": "v3",
+      "lancedb:tag:latest": "true",
+      "lancedb:logical-column": "clip",
+    },
+  },
+  {
+    path: "clip_v2",
+    metadata: {
+      "lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
+      "lancedb:tag:field_type": "embedding",
+      "lancedb:tag:model": "clip",
+      "lancedb:tag:version": "v2",
+      "lancedb:logical-column": "clip",
+    },
+  },
+]);
+console.log(res.version); // new table version
+
+// merge semantics: add a key, delete one via null, keep the rest
+await table.updateFieldMetadata([
+  { path: "clip_v2", metadata: { "lancedb:tag:archived": "true", "lancedb:tag:latest": null } },
+]);
+```
+
+## Step 4: Confirm
+
+Report back:
+- Which columns were updated and what was written
+- The new table version number (from the result)
+- Any columns skipped (e.g., already had up-to-date metadata)
+
+## Quick examples
+
+**"Write descriptions for all columns in the `product_embeddings` table"**
+1. Read `table.schema` → all fields + existing metadata
+2. Generate a `lancedb:description` for each column based on name + type
+3. One `update_field_metadata` call with all descriptions
+4. Report
+
+**"Tag the columns in `model_outputs` with their field type and model"**
+1. Read the schema
+2. For each field, classify by name + Arrow type → set `lancedb:tag:field_type` and `lancedb:tag:model` where applicable
+3. Write in one batched call
+4. Report
+
+**"Group the feature columns in `training_features` into logical families and mark the latest version"**
+1. Read the schema
+2. Find version patterns → assign `lancedb:logical-column` and `lancedb:tag:version`; mark newest with `lancedb:tag:latest: "true"`
+3. Write in one batched call
+4. Show the grouping
@@ -96,6 +96,32 @@ print(table.index_stats("vector_idx"))

 Use these before changing indexes or search tuning.

+## Column (Field) Metadata
+
+```python
+schema = table.schema                       # sync property; async: await table.schema()
+meta = schema.field("category").metadata    # dict[bytes, bytes] — Arrow metadata is bytes-keyed
+res = table.update_field_metadata(          # varargs: one dict per field; works local + remote
+    {"path": "category", "metadata": {"lancedb:description": "...", "lancedb:tag:field_type": "label"}}
+)
+res.version  # new table version
+```
+
+Merges by default; a `None` value deletes that key; `"replace": True` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). `replace_field_metadata` is deprecated. See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
+
+## Branches
+
+```python
+table.branches.list()                          # non-main branches; {} = only main
+exp = table.branches.create("exp")             # fork off main -> handle scoped to the branch
+wip = table.branches.checkout("wip")           # existing branch -> scoped handle (version= pins read-only)
+wip = db.open_table("t", branch="wip")         # or open scoped directly
+table.branches.delete("stale")                 # removes only the branch pointer
+table.current_branch()                         # None = main
+```
+
+There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
+
 ## Maintenance

 ```python
@@ -69,6 +69,33 @@ console.log(await table.indexStats("vector_idx"));

 Use these before changing indexes or search tuning.

+## Column (Field) Metadata
+
+```typescript
+const schema = await table.schema();
+const meta = schema.fields.find((f) => f.name === "category")?.metadata; // Map<string, string>
+const res = await table.updateFieldMetadata([
+  { path: "category", metadata: { "lancedb:description": "...", "lancedb:tag:field_type": "label" } },
+]);
+res.version; // new table version
+```
+
+Merges by default; a `null` value deletes that key; `replace: true` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
+
+## Branches
+
+```typescript
+const branches = await table.branches();       // async manager
+await branches.list();                         // non-main branches; {} = only main
+const exp = await branches.create("exp");      // fork off main -> Table scoped to the branch
+const wip = await branches.checkout("wip");    // existing branch -> scoped Table (version arg pins read-only)
+const wip2 = await db.openTable("t", { branch: "wip" }); // or open scoped directly
+await branches.delete("stale");                // removes only the branch pointer
+table.currentBranch();                         // null = main
+```
+
+There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
+
 ## Maintenance

 ```typescript
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.32.0-beta.1"
+current_version = "0.32.0-beta.2"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
@@ -1750,7 +1750,7 @@ version = "3.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
 dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -2288,14 +2288,13 @@ dependencies = [

 [[package]]
 name = "datafusion"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "93db0e623840612f7f2cd757f7e8a8922064192363732c88692e0870016e141b"
+checksum = "997a31e15872606a49478e670c58302094c97cb96abb0a7d60720f8e92170040"
 dependencies = [
 "arrow",
 "arrow-schema",
 "async-trait",
- "bytes",
 "chrono",
 "datafusion-catalog",
 "datafusion-catalog-listing",
@@ -2322,13 +2321,12 @@ dependencies = [
 "datafusion-session",
 "datafusion-sql",
 "futures",
+ "indexmap 2.14.0",
 "itertools 0.14.0",
 "log",
 "object_store",
 "parking_lot",
- "rand 0.9.5",
- "regex",
- "sqlparser 0.61.0",
+ "sqlparser 0.62.0",
 "tempfile",
 "tokio",
 "url",
@@ -2337,9 +2335,9 @@ dependencies = [

 [[package]]
 name = "datafusion-catalog"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "37cefde60b26a7f4ff61e9d2ff2833322f91df2b568d7238afe67bde5bdffb66"
+checksum = "f7dd61161508f8f5fa1107774ea687bd753c22d83a32eebf963549f89de14139"
 dependencies = [
 "arrow",
 "async-trait",
@@ -2362,9 +2360,9 @@ dependencies = [

 [[package]]
 name = "datafusion-catalog-listing"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "17e112307715d6a7a331111a4c2330ff54bc237183511c319e3708a4cff431fb"
+checksum = "897c70f871277f9ce99aa38347be0d679bbe3e617156c4d2a8378cec8a2a0891"
 dependencies = [
 "arrow",
 "async-trait",
@@ -2385,32 +2383,33 @@ dependencies = [

 [[package]]
 name = "datafusion-common"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d72a11ca44a95e1081870d3abb80c717496e8a7acb467a1d3e932bb636af5cc2"
+checksum = "121c9ded5d87d9172319e006f2afdb9928d72dbacd6a90a458d8acb1e3b43a65"
 dependencies = [
- "ahash",
 "arrow",
 "arrow-ipc",
+ "arrow-schema",
 "chrono",
+ "foldhash 0.2.0",
 "half",
- "hashbrown 0.16.1",
+ "hashbrown 0.17.1",
 "indexmap 2.14.0",
 "itertools 0.14.0",
 "libc",
 "log",
 "object_store",
- "paste",
- "sqlparser 0.61.0",
+ "sqlparser 0.62.0",
 "tokio",
+ "uuid",
 "web-time",
 ]

 [[package]]
 name = "datafusion-common-runtime"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "89f4afaed29670ec4fd6053643adc749fe3f4bc9d1ce1b8c5679b22c67d12def"
+checksum = "981b9dae74f78ee3d9f714fb49b01919eab975461b56149510c3ba9ea11287d1"
 dependencies = [
 "futures",
 "log",
@@ -2419,9 +2418,9 @@ dependencies = [

 [[package]]
 name = "datafusion-datasource"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e9fb386e1691355355a96419978a0022b7947b44d4a24a6ea99f00b6b485cbb6"
+checksum = "ffd7d295b2ec7c00d8a56562f41ed41062cf0af75549ed891c12a0a09eddfefe"
 dependencies = [
 "arrow",
 "async-trait",
@@ -2441,6 +2440,7 @@ dependencies = [
 "itertools 0.14.0",
 "log",
 "object_store",
+ "parking_lot",
 "rand 0.9.5",
 "tokio",
 "url",
@@ -2448,9 +2448,9 @@ dependencies = [

 [[package]]
 name = "datafusion-datasource-arrow"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ffa6c52cfed0734c5f93754d1c0175f558175248bf686c944fb05c373e5fc096"
+checksum = "552b0b3f342f7ec41b3fbd70f6339dc82a30cfd0349e7f280e7852528085349f"
 dependencies = [
 "arrow",
 "arrow-ipc",
@@ -2472,9 +2472,9 @@ dependencies = [

 [[package]]
 name = "datafusion-datasource-csv"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "503f29e0582c1fc189578d665ff57d9300da1f80c282777d7eb67bb79fb8cdca"
+checksum = "68850aa426b897e879c8b87e512ea8124f1d0a2869a4e51808ddaaddf1bc0ada"
 dependencies = [
 "arrow",
 "async-trait",
@@ -2495,9 +2495,9 @@ dependencies = [

 [[package]]
 name = "datafusion-datasource-json"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e33804749abc8d0c8cb7473228483cb8070e524c6f6086ee1b85a64debe2b3d2"
+checksum = "402f93242ae08ef99139ee2c528a49d087efe88d5c7b2c3ff5480855a40ce54f"
 dependencies = [
 "arrow",
 "async-trait",
@@ -2512,27 +2512,25 @@ dependencies = [
 "datafusion-session",
 "futures",
 "object_store",
- "serde_json",
 "tokio",
 "tokio-stream",
 ]

 [[package]]
 name = "datafusion-doc"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8de6ac0df1662b9148ad3c987978b32cbec7c772f199b1d53520c8fa764a87ee"
+checksum = "cb9e7e5d11130c48c8bd4e80c79a9772dd28ce6dc330baca9246205d245b9e2e"

 [[package]]
 name = "datafusion-execution"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c03c7fbdaefcca4ef6ffe425a5fc2325763bfb426599bb0bf4536466efabe709"
+checksum = "37a8643ab852eb68864e1b72ae789e8066282dce48eea6347ffb0aee33d1ccc0"
 dependencies = [
 "arrow",
 "arrow-buffer",
 "async-trait",
- "chrono",
 "dashmap",
 "datafusion-common",
 "datafusion-expr",
@@ -2548,11 +2546,12 @@ dependencies = [

 [[package]]
 name = "datafusion-expr"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "574b9b6977fedbd2a611cbff12e5caf90f31640ad9dc5870f152836d94bad0dd"
+checksum = "6932f4d71eed9c8d9341476a2b845aadfabde5495d08dbcd8fc23881f49fa7a0"
 dependencies = [
 "arrow",
+ "arrow-schema",
 "async-trait",
 "chrono",
 "datafusion-common",
@@ -2563,29 +2562,27 @@ dependencies = [
 "datafusion-physical-expr-common",
 "indexmap 2.14.0",
 "itertools 0.14.0",
- "paste",
 "serde_json",
- "sqlparser 0.61.0",
+ "sqlparser 0.62.0",
 ]

 [[package]]
 name = "datafusion-expr-common"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7d7c3adf3db8bf61e92eb90cb659c8e8b734593a8f7c8e12a843c7ddba24b87e"
+checksum = "0225491839a31b1f7d2cb8092c2d50792e2fe1c1724e4e6d08e011f5feaf4ed2"
 dependencies = [
 "arrow",
 "datafusion-common",
 "indexmap 2.14.0",
 "itertools 0.14.0",
- "paste",
 ]

 [[package]]
 name = "datafusion-functions"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f28aa4e10384e782774b10e72aca4d93ef7b31aa653095d9d4536b0a3dbc51b6"
+checksum = "14872c47bfc3d21e53ec82f57074e6987a15941c1e2f43cde4ac6ae2746634e3"
 dependencies = [
 "arrow",
 "arrow-buffer",
@@ -2600,26 +2597,25 @@ dependencies = [
 "datafusion-expr",
 "datafusion-expr-common",
 "datafusion-macros",
+ "datafusion-physical-expr-common",
 "hex",
 "itertools 0.14.0",
 "log",
- "md-5 0.10.6",
+ "md-5 0.11.0",
 "memchr",
 "num-traits",
 "rand 0.9.5",
 "regex",
- "sha2 0.10.9",
- "unicode-segmentation",
+ "sha2 0.11.0",
 "uuid",
 ]

 [[package]]
 name = "datafusion-functions-aggregate"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "00aa6217e56098ba84e0a338176fe52f0a84cca398021512c6c8c5eff806d0ad"
+checksum = "75a2ca14e1b609be21e657e2d3130b2f446456b08393b377bb721a33952d2e09"
 dependencies = [
- "ahash",
 "arrow",
 "datafusion-common",
 "datafusion-doc",
@@ -2629,19 +2625,18 @@ dependencies = [
 "datafusion-macros",
 "datafusion-physical-expr",
 "datafusion-physical-expr-common",
+ "foldhash 0.2.0",
 "half",
 "log",
 "num-traits",
- "paste",
 ]

 [[package]]
 name = "datafusion-functions-aggregate-common"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b511250349407db7c43832ab2de63f5557b19a20dfd236b39ca2c04468b50d47"
+checksum = "1ece74ba09092d2ef9c9b54a38445450aea292a1f8b04faf531936b723a24b3c"
 dependencies = [
- "ahash",
 "arrow",
 "datafusion-common",
 "datafusion-expr-common",
@@ -2650,9 +2645,9 @@ dependencies = [

 [[package]]
 name = "datafusion-functions-nested"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ef13a858e20d50f0a9bb5e96e7ac82b4e7597f247515bccca4fdd2992df0212a"
+checksum = "3f3e3f9ee8ca59bf70518802107de6f1b88a9509efdc629fadc5de9d6b2d5ef5"
 dependencies = [
 "arrow",
 "arrow-ord",
@@ -2666,34 +2661,34 @@ dependencies = [
 "datafusion-functions-aggregate-common",
 "datafusion-macros",
 "datafusion-physical-expr-common",
- "hashbrown 0.16.1",
+ "hashbrown 0.17.1",
 "itertools 0.14.0",
 "itoa",
 "log",
- "paste",
+ "memchr",
 ]

 [[package]]
 name = "datafusion-functions-table"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "72b40d3f5bbb3905f9ccb1ce9485a9595c77b69758a7c24d3ba79e334ff51e7e"
+checksum = "89161dffc22cf2b50f9f4b1bee83b5221d3b4ed7c2e37fd7aa2b22a5297b3a26"
 dependencies = [
 "arrow",
 "async-trait",
 "datafusion-catalog",
 "datafusion-common",
 "datafusion-expr",
+ "datafusion-physical-expr",
 "datafusion-physical-plan",
 "parking_lot",
- "paste",
 ]

 [[package]]
 name = "datafusion-functions-window"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d4e88ec9d57c9b685d02f58bfee7be62d72610430ddcedb82a08e5d9925dbfb6"
+checksum = "d7339345b226b3874037708bf5023ba1c2de705128f8457a095aae5ae9cb9c78"
 dependencies = [
 "arrow",
 "datafusion-common",
@@ -2704,14 +2699,13 @@ dependencies = [
 "datafusion-physical-expr",
 "datafusion-physical-expr-common",
 "log",
- "paste",
 ]

 [[package]]
 name = "datafusion-functions-window-common"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8307bb93519b1a91913723a1130cfafeee3f72200d870d88e91a6fc5470ede5c"
+checksum = "fa84836dc2392df6f43d6a29d37fb56a8ebdc8b3f4e10ae8dc15861fd20278fb"
 dependencies = [
 "datafusion-common",
 "datafusion-physical-expr-common",
@@ -2719,9 +2713,9 @@ dependencies = [

 [[package]]
 name = "datafusion-macros"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2e367e6a71051d0ebdd29b2f85d12059b38b1d1f172c6906e80016da662226bd"
+checksum = "587164e03ad68732aa9e7bfe5686e3f25970d4c64fd4bd80790749840892dae5"
 dependencies = [
 "datafusion-doc",
 "quote",
@@ -2730,9 +2724,9 @@ dependencies = [

 [[package]]
 name = "datafusion-optimizer"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e929015451a67f77d9d8b727b2bf3a40c4445fdef6cdc53281d7d97c76888ace"
+checksum = "77f20e8cf9e8654d92f4c16b24c487353ee5bf153ffc12d5772cd399ab8cd281"
 dependencies = [
 "arrow",
 "chrono",
@@ -2749,11 +2743,10 @@ dependencies = [

 [[package]]
 name = "datafusion-physical-expr"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4b1e68aba7a4b350401cfdf25a3d6f989ad898a7410164afe9ca52080244cb59"
+checksum = "f015a4a82f6f7ff7e1d8d4bf3870a936752fa38b17705dfcc14adef95aa8922c"
 dependencies = [
- "ahash",
 "arrow",
 "datafusion-common",
 "datafusion-expr",
@@ -2761,20 +2754,19 @@ dependencies = [
 "datafusion-functions-aggregate-common",
 "datafusion-physical-expr-common",
 "half",
- "hashbrown 0.16.1",
+ "hashbrown 0.17.1",
 "indexmap 2.14.0",
 "itertools 0.14.0",
 "parking_lot",
- "paste",
 "petgraph",
 "tokio",
 ]

 [[package]]
 name = "datafusion-physical-expr-adapter"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ea22315f33cf2e0adc104e8ec42e285f6ed93998d565c65e82fec6a9ee9f9db4"
+checksum = "51e6ffff8acdfe54e0ea15ccf38115c4a9184433b0439f42907637928d00a235"
 dependencies = [
 "arrow",
 "datafusion-common",
@@ -2787,26 +2779,26 @@ dependencies = [

 [[package]]
 name = "datafusion-physical-expr-common"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b04b45ea8ad3ac2d78f2ea2a76053e06591c9629c7a603eda16c10649ecf4362"
+checksum = "7967a3e171c6a4bf09474b3f7a14f1a3db13ed1714ba12156f33fcce2bba54e8"
 dependencies = [
- "ahash",
 "arrow",
 "chrono",
 "datafusion-common",
 "datafusion-expr-common",
- "hashbrown 0.16.1",
+ "hashbrown 0.17.1",
 "indexmap 2.14.0",
 "itertools 0.14.0",
 "parking_lot",
+ "pin-project",
 ]

 [[package]]
 name = "datafusion-physical-optimizer"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7cb13397809a425918f608dfe8653f332015a3e330004ab191b4404187238b95"
+checksum = "59ff803e2a96054cb6d83f35f9e60fd4f42eac515e1932bd1b2dbc91d5fcbf36"
 dependencies = [
 "arrow",
 "datafusion-common",
@@ -2822,12 +2814,13 @@ dependencies = [

 [[package]]
 name = "datafusion-physical-plan"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5edc023675791af9d5fb4cc4c24abf5f7bd3bd4dcf9e5bd90ea1eff6976dcc79"
+checksum = "776ee54d47d15bdb126452f9ca17b03761e3b004682914beaedd3f86eb507fbc"
 dependencies = [
- "ahash",
 "arrow",
+ "arrow-data",
+ "arrow-ipc",
 "arrow-ord",
 "arrow-schema",
 "async-trait",
@@ -2842,7 +2835,7 @@ dependencies = [
 "datafusion-physical-expr-common",
 "futures",
 "half",
- "hashbrown 0.16.1",
+ "hashbrown 0.17.1",
 "indexmap 2.14.0",
 "itertools 0.14.0",
 "log",
@@ -2854,9 +2847,9 @@ dependencies = [

 [[package]]
 name = "datafusion-pruning"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ac8c76860e355616555081cab5968cec1af7a80701ff374510860bcd567e365a"
+checksum = "d5fb9e5774660aa69c3ba93c610f175f75b65cb8c3776edb3626de8f3a4f4ee3"
 dependencies = [
 "arrow",
 "datafusion-common",
@@ -2865,15 +2858,14 @@ dependencies = [
 "datafusion-physical-expr",
 "datafusion-physical-expr-common",
 "datafusion-physical-plan",
- "itertools 0.14.0",
 "log",
 ]

 [[package]]
 name = "datafusion-session"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5412111aa48e2424ba926112e192f7a6b7e4ccb450145d25ce5ede9f19dc491e"
+checksum = "15ce715fa2a61f4623cc234bcc14a3ef6a91f189128d5b14b468a6a17cdfc417"
 dependencies = [
 "async-trait",
 "datafusion-common",
@@ -2885,9 +2877,9 @@ dependencies = [

 [[package]]
 name = "datafusion-sql"
-version = "53.1.0"
+version = "54.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fa0d133ddf8b9b3b872acac900157f783e7b879fe9a6bccf389abebbfac45ec1"
+checksum = "6094ad36a3ed6d7ac87b20b479b2d0b118250f66cf997603829fdc65b44a7099"
 dependencies = [
 "arrow",
 "bigdecimal",
@@ -2898,7 +2890,7 @@ dependencies = [
 "indexmap 2.14.0",
 "log",
 "regex",
- "sqlparser 0.61.0",
+ "sqlparser 0.62.0",
 ]

 [[package]]
@@ -3014,7 +3006,7 @@ dependencies = [
 "libc",
 "option-ext",
 "redox_users",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -3237,7 +3229,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
 dependencies = [
 "libc",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -3429,8 +3421,8 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"

 [[package]]
 name = "fsst"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "rand 0.9.5",
@@ -3917,6 +3909,11 @@ name = "hashbrown"
 version = "0.17.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
+dependencies = [
+ "allocator-api2",
+ "equivalent",
+ "foldhash 0.2.0",
+]

 [[package]]
 name = "heapify"
@@ -4221,7 +4218,7 @@ dependencies = [
 "libc",
 "percent-encoding",
 "pin-project-lite",
- "socket2 0.6.3",
+ "socket2 0.5.10",
 "system-configuration",
 "tokio",
 "tower-service",
@@ -4520,7 +4517,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
 dependencies = [
 "hermit-abi",
 "libc",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -4614,7 +4611,7 @@ dependencies = [
 "portable-atomic-util",
 "serde_core",
 "wasm-bindgen",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -4780,8 +4777,8 @@ checksum = "e037a2e1d8d5fdbd49b16a4ea09d5d6401c1f29eca5ff29d03d3824dba16256a"

 [[package]]
 name = "lance"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arc-swap",
 "arrow",
@@ -4855,8 +4852,8 @@ dependencies = [

 [[package]]
 name = "lance-arrow"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4878,7 +4875,7 @@ dependencies = [
 [[package]]
 name = "lance-arrow-scalar"
 version = "58.0.0"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4892,7 +4889,7 @@ dependencies = [
 [[package]]
 name = "lance-arrow-stats"
 version = "58.0.0"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "arrow-schema",
@@ -4901,8 +4898,8 @@ dependencies = [

 [[package]]
 name = "lance-bitpacking"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrayref",
 "crunchy",
@@ -4912,8 +4909,8 @@ dependencies = [

 [[package]]
 name = "lance-core"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4951,8 +4948,8 @@ dependencies = [

 [[package]]
 name = "lance-datafusion"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -4982,8 +4979,8 @@ dependencies = [

 [[package]]
 name = "lance-datagen"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -5000,8 +4997,8 @@ dependencies = [

 [[package]]
 name = "lance-derive"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -5010,8 +5007,8 @@ dependencies = [

 [[package]]
 name = "lance-encoding"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-arith",
 "arrow-array",
@@ -5046,8 +5043,8 @@ dependencies = [

 [[package]]
 name = "lance-file"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-arith",
 "arrow-array",
@@ -5077,8 +5074,8 @@ dependencies = [

 [[package]]
 name = "lance-index"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arc-swap",
 "arrow",
@@ -5113,6 +5110,7 @@ dependencies = [
 "lance-datagen",
 "lance-encoding",
 "lance-file",
+ "lance-index-core",
 "lance-io",
 "lance-linalg",
 "lance-select",
@@ -5141,10 +5139,33 @@ dependencies = [
 "uuid",
 ]

+[[package]]
+name = "lance-index-core"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
+dependencies = [
+ "arrow-array",
+ "arrow-schema",
+ "arrow-select",
+ "async-trait",
+ "bytes",
+ "datafusion",
+ "datafusion-common",
+ "datafusion-expr",
+ "futures",
+ "lance-core",
+ "lance-io",
+ "lance-select",
+ "prost-types",
+ "roaring",
+ "serde",
+ "serde_json",
+]
+
 [[package]]
 name = "lance-io"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow",
 "arrow-arith",
@@ -5162,6 +5183,7 @@ dependencies = [
 "bytes",
 "chrono",
 "futures",
+ "goosefs-sdk",
 "http 1.4.2",
 "io-uring",
 "lance-arrow",
@@ -5186,8 +5208,8 @@ dependencies = [

 [[package]]
 name = "lance-linalg"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5203,8 +5225,8 @@ dependencies = [

 [[package]]
 name = "lance-namespace"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow",
 "async-trait",
@@ -5216,8 +5238,8 @@ dependencies = [

 [[package]]
 name = "lance-namespace-impls"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow",
 "arrow-ipc",
@@ -5271,8 +5293,8 @@ dependencies = [

 [[package]]
 name = "lance-select"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5287,8 +5309,8 @@ dependencies = [

 [[package]]
 name = "lance-table"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -5327,8 +5349,8 @@ dependencies = [

 [[package]]
 name = "lance-testing"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "arrow-array",
 "arrow-schema",
@@ -5341,8 +5363,8 @@ dependencies = [

 [[package]]
 name = "lance-tokenizer"
-version = "9.0.0-beta.23"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.23#0acc51eb8f013985395bf3ac7f0ef4f8a23a377d"
+version = "9.1.0-beta.2"
+source = "git+https://github.com/lance-format/lance.git?tag=v9.1.0-beta.2#9139486ed371dd6caeb9c708b39fba7f5649a9b0"
 dependencies = [
 "icu_segmenter",
 "jieba-rs",
@@ -5355,7 +5377,7 @@ dependencies = [

 [[package]]
 name = "lancedb"
-version = "0.32.0-beta.1"
+version = "0.32.0-beta.2"
 dependencies = [
 "ahash",
 "anyhow",
@@ -5391,6 +5413,7 @@ dependencies = [
 "datafusion-physical-plan",
 "datafusion-sql",
 "futures",
+ "goosefs-sdk",
 "half",
 "hf-hub",
 "http 1.4.2",
@@ -5443,7 +5466,7 @@ dependencies = [

 [[package]]
 name = "lancedb-nodejs"
-version = "0.32.0-beta.1"
+version = "0.32.0-beta.2"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5468,7 +5491,7 @@ dependencies = [

 [[package]]
 name = "lancedb-python"
-version = "0.35.0-beta.1"
+version = "0.35.0-beta.2"
 dependencies = [
 "arrow",
 "async-trait",
@@ -6184,7 +6207,7 @@ version = "0.50.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
 dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -7563,8 +7586,8 @@ version = "0.14.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "343d3bd7056eda839b03204e68deff7d1b13aba7af2b2fd16890697274262ee7"
 dependencies = [
- "heck 0.5.0",
- "itertools 0.14.0",
+ "heck 0.4.1",
+ "itertools 0.11.0",
 "log",
 "multimap",
 "petgraph",
@@ -7583,7 +7606,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b"
 dependencies = [
 "anyhow",
- "itertools 0.14.0",
+ "itertools 0.11.0",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -7792,7 +7815,7 @@ dependencies = [
 "quinn-udp",
 "rustc-hash",
 "rustls 0.23.40",
- "socket2 0.6.3",
+ "socket2 0.5.10",
 "thiserror 2.0.18",
 "tokio",
 "tracing",
@@ -7830,9 +7853,9 @@ dependencies = [
 "cfg_aliases",
 "libc",
 "once_cell",
- "socket2 0.6.3",
+ "socket2 0.5.10",
 "tracing",
- "windows-sys 0.60.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -8606,7 +8629,7 @@ dependencies = [
 "errno",
 "libc",
 "linux-raw-sys",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -8677,7 +8700,7 @@ dependencies = [
 "security-framework",
 "security-framework-sys",
 "webpki-root-certs",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -9245,7 +9268,7 @@ version = "0.8.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451"
 dependencies = [
- "heck 0.5.0",
+ "heck 0.4.1",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -9257,7 +9280,7 @@ version = "0.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "54254b8531cafa275c5e096f62d48c81435d1015405a91198ddb11e967301d40"
 dependencies = [
- "heck 0.5.0",
+ "heck 0.4.1",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -9352,9 +9375,9 @@ dependencies = [

 [[package]]
 name = "sqlparser"
-version = "0.61.0"
+version = "0.62.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dbf5ea8d4d7c808e1af1cbabebca9a2abe603bcefc22294c5b95018d53200cb7"
+checksum = "13c6d1b651dc4edf07eead2a0c6c78016ce971bc2c10da5266861b13f25e7cec"
 dependencies = [
 "log",
 "sqlparser_derive",
@@ -9690,7 +9713,7 @@ dependencies = [
 "getrandom 0.4.2",
 "once_cell",
 "rustix",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -10667,7 +10690,7 @@ version = "0.1.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
 dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -13,20 +13,20 @@ categories = ["database-implementations"]
 rust-version = "1.91.0"

 [workspace.dependencies]
-lance = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-core = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-datagen = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-file = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-io = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-index = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-linalg = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace-impls = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-table = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-testing = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-datafusion = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-encoding = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-arrow = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance = { "version" = "=9.1.0-beta.2", default-features = false, "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-core = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-datagen = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-file = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-io = { "version" = "=9.1.0-beta.2", default-features = false, "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-index = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-linalg = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace-impls = { "version" = "=9.1.0-beta.2", default-features = false, "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-table = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-testing = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-datafusion = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-encoding = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance-arrow = { "version" = "=9.1.0-beta.2", "tag" = "v9.1.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
 ahash = "0.8"
 # Note that this one does not include pyarrow
 arrow = { version = "58.0.0", optional = false }
@@ -39,15 +39,15 @@ arrow-schema = "58.0.0"
 arrow-select = "58.0.0"
 arrow-cast = "58.0.0"
 async-trait = "0"
-datafusion = { version = "53.0.0", default-features = false }
-datafusion-catalog = "53.0.0"
-datafusion-common = { version = "53.0.0", default-features = false }
-datafusion-execution = "53.0.0"
-datafusion-expr = "53.0.0"
-datafusion-functions = "53.0.0"
-datafusion-physical-plan = "53.0.0"
-datafusion-physical-expr = "53.0.0"
-datafusion-sql = "53.0.0"
+datafusion = { version = "54.0.0", default-features = false }
+datafusion-catalog = "54.0.0"
+datafusion-common = { version = "54.0.0", default-features = false }
+datafusion-execution = "54.0.0"
+datafusion-expr = "54.0.0"
+datafusion-functions = "54.0.0"
+datafusion-physical-plan = "54.0.0"
+datafusion-physical-expr = "54.0.0"
+datafusion-sql = "54.0.0"
 env_logger = "0.11"
 half = { "version" = "2.7.1", default-features = false, features = [
    "num-traits",
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
 <dependency>
    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-core</artifactId>
-    <version>0.32.0-beta.1</version>
+    <version>0.32.0-beta.2</version>
 </dependency>
 ```

@@ -33,7 +33,7 @@ protected inner: Query | Promise<Query>;
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -41,6 +41,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -38,7 +38,7 @@ protected inner: NativeQueryType | Promise<NativeQueryType>;
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -46,6 +46,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -29,7 +29,7 @@ protected inner: TakeQuery | Promise<TakeQuery>;
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -37,6 +37,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -51,7 +51,7 @@ addQueryVector(vector): VectorQuery
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -59,6 +59,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -118,6 +118,7 @@

 ## Type Aliases

+- [AnalyzePlanDistributedMetrics](type-aliases/AnalyzePlanDistributedMetrics.md)
 - [BaseTokenizer](type-aliases/BaseTokenizer.md)
 - [Data](type-aliases/Data.md)
 - [DataLike](type-aliases/DataLike.md)
@@ -0,0 +1,11 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / AnalyzePlanDistributedMetrics
+
+# Type Alias: AnalyzePlanDistributedMetrics
+
+```ts
+type AnalyzePlanDistributedMetrics: "aggregate" | "per_worker" | "full";
+```
@@ -8,7 +8,7 @@
    <parent>
      <groupId>com.lancedb</groupId>
      <artifactId>lancedb-parent</artifactId>
-      <version>0.32.0-beta.1</version>
+      <version>0.32.0-beta.2</version>
      <relativePath>../pom.xml</relativePath>
    </parent>

@@ -6,7 +6,7 @@

    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-parent</artifactId>
-    <version>0.32.0-beta.1</version>
+    <version>0.32.0-beta.2</version>
    <packaging>pom</packaging>
    <name>${project.artifactId}</name>
    <description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <arrow.version>15.0.0</arrow.version>
-        <lance-core.version>9.0.0-beta.23</lance-core.version>
+        <lance-core.version>9.1.0-beta.2</lance-core.version>
        <spotless.skip>false</spotless.skip>
        <spotless.version>2.30.0</spotless.version>
        <spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
@@ -1,7 +1,7 @@
 [package]
 name = "lancedb-nodejs"
 edition.workspace = true
-version = "0.32.0-beta.1"
+version = "0.32.0-beta.2"
 publish = false
 license.workspace = true
 description.workspace = true
@@ -2775,8 +2775,13 @@ describe("when calling analyzePlan", () => {
      .fill(1)
      .map(() => Math.random());
    const plan = await table.query().nearestTo(queryVec).analyzePlan();
-    console.log("Query Plan:\n", plan); // <--- Print the plan
    expect(plan).toMatch("AnalyzeExec");
+
+    const fullPlan = await table
+      .query()
+      .nearestTo(queryVec)
+      .analyzePlan("full");
+    expect(fullPlan).toMatch("AnalyzeExec");
  });
 });

@@ -93,6 +93,7 @@ export {
  QueryBase,
  VectorQuery,
  TakeQuery,
+  AnalyzePlanDistributedMetrics,
  QueryExecutionOptions,
  ColumnOrdering,
  FullTextSearchOptions,
@@ -79,6 +79,8 @@ export interface QueryExecutionOptions {
  timeoutMs?: number;
 }

+export type AnalyzePlanDistributedMetrics = "aggregate" | "per_worker" | "full";
+
 export interface ColumnOrdering {
  columnName: string;
  ascending?: boolean;
@@ -311,13 +313,20 @@ export class QueryBase<
   *        KNNVectorDistance: metric=l2, metrics=[output_rows=1, elapsed_compute=114.333µs, output_batches=1]
   *         LanceScan: uri=/path/to/data, projection=[vector], row_id=true, row_addr=false, ordered=false, metrics=[output_rows=1, elapsed_compute=103.626µs, bytes_read=549, iops=2, requests=2]
   *
+   * @param distributedMetrics - How distributed worker metrics are displayed for remote query plans.
+   * Defaults to `"aggregate"`.
   * @returns A query execution plan with runtime metrics for each step.
   */
-  async analyzePlan(): Promise<string> {
+  async analyzePlan(
+    distributedMetrics?: AnalyzePlanDistributedMetrics,
+  ): Promise<string> {
+    const distributedMetricsMode = distributedMetrics ?? "aggregate";
    if (this.inner instanceof Promise) {
-      return this.inner.then((inner) => inner.analyzePlan());
+      return this.inner.then((inner) =>
+        inner.analyzePlan(distributedMetricsMode),
+      );
    } else {
-      return this.inner.analyzePlan();
+      return this.inner.analyzePlan(distributedMetricsMode);
    }
  }

@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-musl",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-musl.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-musl",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-musl.node",
@@ -1,6 +1,6 @@
 {
  "name": "@lancedb/lancedb-win32-arm64-msvc",
-  "version": "0.32.0-beta.1",
+  "version": "0.32.0-beta.2",
  "os": [
    "win32"
  ],
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",
@@ -1,12 +1,12 @@
 {
  "name": "@lancedb/lancedb",
-  "version": "0.32.0-beta.1",
+  "version": "0.32.0-beta.2",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@lancedb/lancedb",
-      "version": "0.32.0-beta.1",
+      "version": "0.32.0-beta.2",
      "cpu": [
        "x64",
        "arm64"
@@ -11,7 +11,7 @@
    "ann"
  ],
  "private": false,
-  "version": "0.32.0-beta.1",
+  "version": "0.32.0-beta.2",
  "main": "dist/index.js",
  "exports": {
    ".": "./dist/index.js",
@@ -19,6 +19,7 @@ use lancedb::index::scalar::{
    BooleanQuery, BoostQuery, FtsQuery, FullTextSearchQuery, MatchQuery, MultiMatchQuery, Occur,
    Operator, PhraseQuery,
 };
+use lancedb::query::AnalyzePlanDistributedMetrics;
 use lancedb::query::ExecutableQuery;
 use lancedb::query::Query as LanceDbQuery;
 use lancedb::query::QueryBase;
@@ -47,6 +48,28 @@ impl From<ColumnOrdering> for LanceDbColumnOrdering {
    }
 }

+fn analyze_plan_options(
+    distributed_metrics: Option<String>,
+) -> napi::Result<QueryExecutionOptions> {
+    let analyze_plan_distributed_metrics =
+        match distributed_metrics.as_deref().unwrap_or("aggregate") {
+            "aggregate" => AnalyzePlanDistributedMetrics::Aggregate,
+            "per_worker" => AnalyzePlanDistributedMetrics::PerWorker,
+            "full" => AnalyzePlanDistributedMetrics::Full,
+            mode => {
+                return Err(napi::Error::from_reason(format!(
+                    "Invalid distributedMetrics value '{}'. Expected one of: \
+                 'aggregate', 'per_worker', 'full'",
+                    mode
+                )));
+            }
+        };
+
+    let mut options = QueryExecutionOptions::default();
+    options.analyze_plan_distributed_metrics = analyze_plan_distributed_metrics;
+    Ok(options)
+}
+
 fn bytes_to_arrow_array(data: Uint8Array, dtype: String) -> napi::Result<Arc<dyn Array>> {
    let buf = arrow_buffer::Buffer::from(data.to_vec());
    let num_bytes = buf.len();
@@ -200,13 +223,17 @@ impl Query {
    }

    #[napi(catch_unwind)]
-    pub async fn analyze_plan(&self) -> napi::Result<String> {
-        self.inner.analyze_plan().await.map_err(|e| {
-            napi::Error::from_reason(format!(
-                "Failed to execute analyze plan: {}",
-                convert_error(&e)
-            ))
-        })
+    pub async fn analyze_plan(&self, distributed_metrics: Option<String>) -> napi::Result<String> {
+        let options = analyze_plan_options(distributed_metrics)?;
+        self.inner
+            .analyze_plan_with_options(options)
+            .await
+            .map_err(|e| {
+                napi::Error::from_reason(format!(
+                    "Failed to execute analyze plan: {}",
+                    convert_error(&e)
+                ))
+            })
    }
 }

@@ -412,13 +439,17 @@ impl VectorQuery {
    }

    #[napi(catch_unwind)]
-    pub async fn analyze_plan(&self) -> napi::Result<String> {
-        self.inner.analyze_plan().await.map_err(|e| {
-            napi::Error::from_reason(format!(
-                "Failed to execute analyze plan: {}",
-                convert_error(&e)
-            ))
-        })
+    pub async fn analyze_plan(&self, distributed_metrics: Option<String>) -> napi::Result<String> {
+        let options = analyze_plan_options(distributed_metrics)?;
+        self.inner
+            .analyze_plan_with_options(options)
+            .await
+            .map_err(|e| {
+                napi::Error::from_reason(format!(
+                    "Failed to execute analyze plan: {}",
+                    convert_error(&e)
+                ))
+            })
    }
 }

@@ -491,13 +522,17 @@ impl TakeQuery {
    }

    #[napi(catch_unwind)]
-    pub async fn analyze_plan(&self) -> napi::Result<String> {
-        self.inner.analyze_plan().await.map_err(|e| {
-            napi::Error::from_reason(format!(
-                "Failed to execute analyze plan: {}",
-                convert_error(&e)
-            ))
-        })
+    pub async fn analyze_plan(&self, distributed_metrics: Option<String>) -> napi::Result<String> {
+        let options = analyze_plan_options(distributed_metrics)?;
+        self.inner
+            .analyze_plan_with_options(options)
+            .await
+            .map_err(|e| {
+                napi::Error::from_reason(format!(
+                    "Failed to execute analyze plan: {}",
+                    convert_error(&e)
+                ))
+            })
    }
 }

@@ -61,10 +61,11 @@ tests = [
    "duckdb>=0.9.0",
    "pytz>=2023.3",
    "polars>=0.19, <=1.3.0",
+    "pyarrow<25",
    "pyarrow-stubs>=16.0",
-    "pylance>=5.0.0b5",
+    "pylance==9.0.0rc1",
    "requests>=2.31.0",
-    "datafusion>=52,<53",
+    "datafusion>=54,<55",
    "opentelemetry-sdk>=1.30.0",
 ]
 dev = [
@@ -30,6 +30,7 @@ from .types import BaseTokenizerType
 IvfHnswPq: type[HnswPq] = HnswPq
 IvfHnswSq: type[HnswSq] = HnswSq
 IvfHnswFlat: type[HnswFlat] = HnswFlat
+AnalyzePlanDistributedMetrics = Literal["aggregate", "per_worker", "full"]

 class MetricPoint:
    name: str
@@ -393,7 +394,9 @@ class Query:
        self, max_batch_length: Optional[int], timeout: Optional[timedelta]
    ) -> RecordBatchStream: ...
    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
-    async def analyze_plan(self) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class TakeQuery:
@@ -401,6 +404,10 @@ class TakeQuery:
    def with_row_id(self): ...
    async def output_schema(self) -> pa.Schema: ...
    async def execute(self) -> RecordBatchStream: ...
+    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class FTSQuery:
@@ -421,6 +428,10 @@ class FTSQuery:
    async def execute(
        self, max_batch_length: Optional[int], timeout: Optional[timedelta]
    ) -> RecordBatchStream: ...
+    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class VectorQuery:
@@ -443,6 +454,10 @@ class VectorQuery:
    def bypass_vector_index(self): ...
    def nearest_to_text(self, query: dict) -> HybridQuery: ...
    def order_by(self, ordering: Optional[List[ColumnOrdering]]): ...
+    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class HybridQuery:
@@ -79,6 +79,7 @@ if TYPE_CHECKING:
        from typing_extensions import Self

 T = TypeVar("T", bound="LanceModel")
+AnalyzePlanDistributedMetrics = Literal["aggregate", "per_worker", "full"]


@runtime_checkable
@@ -1372,7 +1373,9 @@ class LanceQueryBuilder(ABC):
        self._order_by = ordering
        return self

-    def analyze_plan(self) -> str:
+    def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """
        Run the query and return its execution plan with runtime metrics.

@@ -1410,12 +1413,22 @@ class LanceQueryBuilder(ABC):
                      fragments_scanned=..., ranges_scanned=1, rows_scanned=1,
                      bytes_read=..., iops=..., requests=..., task_wait_time=...]

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+            "aggregate" preserves the legacy summary, "per_worker" shows each
+            worker separately, and "full" includes both.
+
        Returns
        -------
        plan : str
            The physical query execution plan with runtime metrics.
        """
-        return self._table._analyze_plan(self.to_query_object())
+        return self._table._analyze_plan(
+            self.to_query_object(), distributed_metrics=distributed_metrics
+        )

    def vector(self, vector: Union[np.ndarray, list]) -> Self:
        """Set the vector to search for.
@@ -2581,9 +2594,17 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
        indented_fts = "\n".join("  " + line for line in fts_plan.splitlines())
        return f"{reranker_label}\n  {indented_vector}\n  {indented_fts}"

-    def analyze_plan(self):
+    def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """Execute the query and display with runtime metrics.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
@@ -2591,9 +2612,19 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
        self._create_query_builders()

        results = ["Vector Search Plan:"]
-        results.append(self._table._analyze_plan(self._vector_query.to_query_object()))
+        results.append(
+            self._table._analyze_plan(
+                self._vector_query.to_query_object(),
+                distributed_metrics=distributed_metrics,
+            )
+        )
        results.append("FTS Search Plan:")
-        results.append(self._table._analyze_plan(self._fts_query.to_query_object()))
+        results.append(
+            self._table._analyze_plan(
+                self._fts_query.to_query_object(),
+                distributed_metrics=distributed_metrics,
+            )
+        )
        return "\n".join(results)

    def _create_query_builders(self):
@@ -3080,14 +3111,22 @@ class AsyncQueryBase(object):
        """  # noqa: E501
        return await self._inner.explain_plan(verbose)

-    async def analyze_plan(self):
+    async def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """Execute the query and display with runtime metrics.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
        """
-        return await self._inner.analyze_plan()
+        return await self._inner.analyze_plan(distributed_metrics)


 class AsyncStandardQuery(AsyncQueryBase):
@@ -3836,18 +3875,18 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
        >>> asyncio.run(doctest_example()) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
        RRFReranker(K=60)
            ProjectionExec: expr=[vector@0 as vector, text@3 as text, _distance@2 as _distance]
-              Take: columns="vector, _rowid, _distance, (text)"
-                CoalesceBatchesExec: target_batch_size=1024
-                  GlobalLimitExec: skip=0, fetch=10
-                    FilterExec: _distance@2 IS NOT NULL
-                      SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
-                        KNNVectorDistance: metric=l2
-                          LanceRead: uri=..., projection=[vector], ...
+            Take: columns="vector, _rowid, _distance, (text)"
+              CoalesceBatchesExec: target_batch_size=1024
+                GlobalLimitExec: skip=0, fetch=10
+                  FilterExec: _distance@2 IS NOT NULL
+                    SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
+                      KNNVectorDistance: metric=l2
+                        LanceRead: uri=..., projection=[vector], ...
            ProjectionExec: expr=[vector@2 as vector, text@3 as text, _score@1 as _score]
-              Take: columns="_rowid, _score, (vector), (text)"
-                CoalesceBatchesExec: target_batch_size=1024
-                  GlobalLimitExec: skip=0, fetch=10
-                    MatchQuery: column=text, query=hello
+            Take: columns="_rowid, _score, (vector), (text)"
+              CoalesceBatchesExec: target_batch_size=1024
+                GlobalLimitExec: skip=0, fetch=10
+                  MatchQuery: column=text, query=[hello]

        Parameters
        ----------
@@ -3866,7 +3905,9 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
        indented_fts = "\n".join("  " + line for line in fts_plan.splitlines())
        return f"{self._reranker}\n  {indented_vector}\n  {indented_fts}"

-    async def analyze_plan(self):
+    async def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """
        Execute the query and return the physical execution plan with runtime metrics.

@@ -3875,14 +3916,24 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
        elapsed time, I/O stats, and more. It’s useful for debugging and
        performance analysis.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
        """
        results = ["Vector Search Query:"]
-        results.append(await self._inner.to_vector_query().analyze_plan())
+        results.append(
+            await self._inner.to_vector_query().analyze_plan(distributed_metrics)
+        )
        results.append("FTS Search Query:")
-        results.append(await self._inner.to_fts_query().analyze_plan())
+        results.append(
+            await self._inner.to_fts_query().analyze_plan(distributed_metrics)
+        )

        return "\n".join(results)

@@ -4166,14 +4217,22 @@ class BaseQueryBuilder(object):
        """  # noqa: E501
        return LOOP.run(self._inner.explain_plan(verbose))

-    def analyze_plan(self):
+    def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """Execute the query and display with runtime metrics.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
        """
-        return LOOP.run(self._inner.analyze_plan())
+        return LOOP.run(self._inner.analyze_plan(distributed_metrics))


 class LanceTakeQueryBuilder(BaseQueryBuilder):
@@ -56,7 +56,12 @@ from lancedb.merge import LanceMergeInsertBuilder
 from lancedb.embeddings import EmbeddingFunctionRegistry
 from lancedb.table import _normalize_progress

-from ..query import LanceVectorQueryBuilder, LanceQueryBuilder, LanceTakeQueryBuilder
+from ..query import (
+    AnalyzePlanDistributedMetrics,
+    LanceQueryBuilder,
+    LanceTakeQueryBuilder,
+    LanceVectorQueryBuilder,
+)
 from ..table import AsyncTable, BlobMode, Branches, IndexStatistics, Query, Table, Tags
 from ..types import BaseTokenizerType

@@ -718,8 +723,15 @@ class RemoteTable(Table):
    def _explain_plan(self, query: Query, verbose: Optional[bool] = False) -> str:
        return LOOP.run(self._table._explain_plan(query, verbose))

-    def _analyze_plan(self, query: Query) -> str:
-        return LOOP.run(self._table._analyze_plan(query))
+    def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str:
+        return LOOP.run(
+            self._table._analyze_plan(query, distributed_metrics=distributed_metrics)
+        )

    def _output_schema(self, query: Query) -> pa.Schema:
        return LOOP.run(self._table._output_schema(query))
@@ -73,6 +73,7 @@ from .expr import Expr
 from .merge import LanceMergeInsertBuilder
 from .pydantic import LanceModel, model_to_dict
 from .query import (
+    AnalyzePlanDistributedMetrics,
    AsyncFTSQuery,
    AsyncHybridQuery,
    AsyncQuery,
@@ -1552,7 +1553,12 @@ class Table(ABC):
    def _explain_plan(self, query: Query, verbose: Optional[bool] = False) -> str: ...

    @abstractmethod
-    def _analyze_plan(self, query: Query) -> str: ...
+    def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str: ...

    @abstractmethod
    def _output_schema(self, query: Query) -> pa.Schema: ...
@@ -3630,8 +3636,15 @@ class LanceTable(Table):
    def _explain_plan(self, query: Query, verbose: Optional[bool] = False) -> str:
        return LOOP.run(self._table._explain_plan(query, verbose))

-    def _analyze_plan(self, query: Query) -> str:
-        return LOOP.run(self._table._analyze_plan(query))
+    def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str:
+        return LOOP.run(
+            self._table._analyze_plan(query, distributed_metrics=distributed_metrics)
+        )

    def _output_schema(self, query: Query) -> pa.Schema:
        return LOOP.run(self._table._output_schema(query))
@@ -5390,10 +5403,15 @@ class AsyncTable:
        async_query = self._sync_query_to_async(query)
        return await async_query.explain_plan(verbose)

-    async def _analyze_plan(self, query: Query) -> str:
+    async def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str:
        # This method is used by the sync table
        async_query = self._sync_query_to_async(query)
-        return await async_query.analyze_plan()
+        return await async_query.analyze_plan(distributed_metrics)

    async def _output_schema(self, query: Query) -> pa.Schema:
        async_query = self._sync_query_to_async(query)
@@ -196,18 +196,26 @@ async def test_analyze_plan(table: AsyncTable):
 def test_hybrid_phrase_query_is_preserved_in_analyze_plan():
    table = mock.Mock()
    analyzed_queries = []
-    table._analyze_plan.side_effect = lambda query: analyzed_queries.append(query) or ""
+    distributed_metric_modes = []
+
+    def capture_query(query, *, distributed_metrics="aggregate"):
+        analyzed_queries.append(query)
+        distributed_metric_modes.append(distributed_metrics)
+        return ""
+
+    table._analyze_plan.side_effect = capture_query

    (
        LanceHybridQueryBuilder(table)
        .vector([0.1, 0.2])
        .text("puppy runs")
        .phrase_query()
-        .analyze_plan()
+        .analyze_plan(distributed_metrics="full")
    )

    assert len(analyzed_queries) == 2
    assert analyzed_queries[1].full_text_query.query == '"puppy runs"'
+    assert distributed_metric_modes == ["full", "full"]


@pytest.fixture
@@ -128,21 +128,33 @@ def test_split_hash(mem_db):

 def test_split_hash_with_discard(mem_db):
    """Test hash-based splitting with discard weight."""
+    total_rows = 1000
    tbl = mem_db.create_table(
        "test_table",
-        pa.table({"id": range(100), "category": ["A", "B"] * 50, "value": range(100)}),
+        pa.table(
+            {
+                "id": range(total_rows),
+                "category": [f"category-{i}" for i in range(total_rows)],
+                "value": range(total_rows),
+            }
+        ),
    )

    permutation_tbl = (
+        # Hash a high-cardinality column: "category" has only two distinct
+        # values, so whether anything is discarded would hinge on where those
+        # two hashes land rather than on the discard weight.
        permutation_builder(tbl)
-        .split_hash(["category"], [1, 1], discard_weight=2)  # Should discard ~50%
+        .split_hash(["id"], [1, 1], discard_weight=2)  # Should discard ~50%
        .execute()
    )

-    # Should have fewer than 100 rows due to discard
+    # Should have fewer rows due to discard, but should not be empty.
    row_count = permutation_tbl.count_rows()
-    assert row_count < 100
-    assert row_count > 0  # But not empty
+    assert 0 < row_count < total_rows
+
+    data = permutation_tbl.search(None).to_arrow().to_pydict()
+    assert set(data["split_id"]) == {0, 1}


 def test_split_sequential(mem_db):
@@ -1273,7 +1273,7 @@ async def test_explain_plan_fts(table_async: AsyncTable):
    query = await table_async.search("dog", query_type="fts", fts_columns="text")
    plan = await query.explain_plan()
    # Should show FTS details (issue #2465 is now fixed)
-    assert "MatchQuery: column=text, query=dog" in plan
+    assert "MatchQuery: column=text, query=[dog]" in plan
    assert "GlobalLimitExec" in plan  # Default limit

    # Test FTS query with limit
@@ -1281,7 +1281,7 @@ async def test_explain_plan_fts(table_async: AsyncTable):
        "dog", query_type="fts", fts_columns="text"
    )
    plan_with_limit = await query_with_limit.limit(1).explain_plan()
-    assert "MatchQuery: column=text, query=dog" in plan_with_limit
+    assert "MatchQuery: column=text, query=[dog]" in plan_with_limit
    assert "GlobalLimitExec: skip=0, fetch=1" in plan_with_limit

    # Test FTS query with offset and limit
@@ -1289,7 +1289,7 @@ async def test_explain_plan_fts(table_async: AsyncTable):
        "dog", query_type="fts", fts_columns="text"
    )
    plan_with_offset = await query_with_offset.offset(1).limit(1).explain_plan()
-    assert "MatchQuery: column=text, query=dog" in plan_with_offset
+    assert "MatchQuery: column=text, query=[dog]" in plan_with_offset
    assert "GlobalLimitExec: skip=1, fetch=1" in plan_with_offset


@@ -1333,7 +1333,7 @@ async def test_explain_plan_with_filters(table_async: AsyncTable):
        "dog", query_type="fts", fts_columns="text"
    )
    plan_fts_filter = await query_fts_filter.where("id = 1").explain_plan()
-    assert "MatchQuery: column=text, query=dog" in plan_fts_filter
+    assert "MatchQuery: column=text, query=[dog]" in plan_fts_filter
    assert "LanceRead" in plan_fts_filter
    assert "full_filter=id = Int64(1)" in plan_fts_filter  # Should show filter details

@@ -19,6 +19,7 @@ use lancedb::index::scalar::{
    BooleanQuery, BoostQuery, FtsQuery, FullTextSearchQuery, MatchQuery, MultiMatchQuery, Occur,
    Operator, PhraseQuery,
 };
+use lancedb::query::AnalyzePlanDistributedMetrics;
 use lancedb::query::QueryBase;
 use lancedb::query::QueryExecutionOptions;
 use lancedb::query::QueryFilter;
@@ -42,6 +43,25 @@ use pyo3::{Borrowed, FromPyObject, exceptions::PyRuntimeError};
 use pyo3::{PyErr, pyclass};
 use pyo3::{exceptions::PyValueError, intern};

+fn analyze_plan_options(distributed_metrics: Option<&str>) -> PyResult<QueryExecutionOptions> {
+    let analyze_plan_distributed_metrics = match distributed_metrics.unwrap_or("aggregate") {
+        "aggregate" => AnalyzePlanDistributedMetrics::Aggregate,
+        "per_worker" => AnalyzePlanDistributedMetrics::PerWorker,
+        "full" => AnalyzePlanDistributedMetrics::Full,
+        mode => {
+            return Err(PyValueError::new_err(format!(
+                "Invalid distributed_metrics value '{}'. Expected one of: \
+                 'aggregate', 'per_worker', 'full'",
+                mode
+            )));
+        }
+    };
+
+    let mut options = QueryExecutionOptions::default();
+    options.analyze_plan_distributed_metrics = analyze_plan_distributed_metrics;
+    Ok(options)
+}
+
 impl<'a, 'py> FromPyObject<'a, 'py> for PyLanceDB<FtsQuery> {
    type Error = PyErr;

@@ -571,11 +591,16 @@ impl Query {
        })
    }

-    pub fn analyze_plan(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+    #[pyo3(signature = (distributed_metrics=None))]
+    pub fn analyze_plan(
+        self_: PyRef<'_, Self>,
+        distributed_metrics: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
        let inner = self_.inner.clone();
+        let options = analyze_plan_options(distributed_metrics.as_deref())?;
        future_into_py(self_.py(), async move {
            inner
-                .analyze_plan()
+                .analyze_plan_with_options(options)
                .await
                .map_err(|e| PyRuntimeError::new_err(e.to_string()))
        })
@@ -650,11 +675,16 @@ impl TakeQuery {
        })
    }

-    pub fn analyze_plan(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+    #[pyo3(signature = (distributed_metrics=None))]
+    pub fn analyze_plan(
+        self_: PyRef<'_, Self>,
+        distributed_metrics: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
        let inner = self_.inner.clone();
+        let options = analyze_plan_options(distributed_metrics.as_deref())?;
        future_into_py(self_.py(), async move {
            inner
-                .analyze_plan()
+                .analyze_plan_with_options(options)
                .await
                .map_err(|e| PyRuntimeError::new_err(e.to_string()))
        })
@@ -777,14 +807,19 @@ impl FTSQuery {
        })
    }

-    pub fn analyze_plan(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+    #[pyo3(signature = (distributed_metrics=None))]
+    pub fn analyze_plan(
+        self_: PyRef<'_, Self>,
+        distributed_metrics: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
        let inner = self_
            .inner
            .clone()
            .full_text_search(self_.fts_query.clone());
+        let options = analyze_plan_options(distributed_metrics.as_deref())?;
        future_into_py(self_.py(), async move {
            inner
-                .analyze_plan()
+                .analyze_plan_with_options(options)
                .await
                .map_err(|e| PyRuntimeError::new_err(e.to_string()))
        })
@@ -958,11 +993,16 @@ impl VectorQuery {
        })
    }

-    pub fn analyze_plan(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+    #[pyo3(signature = (distributed_metrics=None))]
+    pub fn analyze_plan(
+        self_: PyRef<'_, Self>,
+        distributed_metrics: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
        let inner = self_.inner.clone();
+        let options = analyze_plan_options(distributed_metrics.as_deref())?;
        future_into_py(self_.py(), async move {
            inner
-                .analyze_plan()
+                .analyze_plan_with_options(options)
                .await
                .map_err(|e| PyRuntimeError::new_err(e.to_string()))
        })
@@ -657,6 +657,15 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/c7/0d/67e5b4109ea4a837e80daa87c2c696711955e40449a97e8926672534def2/click-8.4.1-py3-none-any.whl", hash = "sha256:482be17c6991b8c19c5429a1e995d9b0efdbb63172824c41f99965dc0ade8ec2", size = 116639, upload-time = "2026-05-22T04:08:35.26Z" },
 ]

+[[package]]
+name = "cloudpickle"
+version = "3.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/27/fb/576f067976d320f5f0114a8d9fa1215425441bb35627b1993e5afd8111e5/cloudpickle-3.1.2.tar.gz", hash = "sha256:7fda9eb655c9c230dab534f1983763de5835249750e85fbcef43aaa30a9a2414", size = 22330, upload-time = "2025-11-03T09:25:26.604Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl", hash = "sha256:9acb47f6afd73f60dc1df93bb801b472f05ff42fa6c84167d25cb206be1fbf4a", size = 22228, upload-time = "2025-11-03T09:25:25.534Z" },
+]
+
 [[package]]
 name = "cohere"
 version = "7.0.3"
@@ -850,19 +859,25 @@ nvtx = [

 [[package]]
 name = "datafusion"
-version = "52.3.0"
+version = "54.0.0"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
+    { name = "cloudpickle" },
    { name = "pyarrow" },
    { name = "typing-extensions", marker = "python_full_version < '3.13'" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/db/d4/a5ad7b665a80008901892fde61dc667318db0652a955d706ddca3a224b5a/datafusion-52.3.0.tar.gz", hash = "sha256:2e8b02ad142b1a0d673f035d96a0944a640ac78275003d7e453cee4afe4a20a4", size = 205026, upload-time = "2026-03-16T10:54:07.739Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/60/90/886f7e9cf827f07ebd60bd293e54e0a028a50dd49bbaef0ee42aae1981ea/datafusion-54.0.0.tar.gz", hash = "sha256:cfe7e8dfc026efc05824f49b53ad6a72caf5c2d6820759b6212a09e245a427ed", size = 276448, upload-time = "2026-06-29T11:19:34.816Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/55/63/1bb0737988cefa77274b459d64fa4b57ba4cf755639a39733e9581b5d599/datafusion-52.3.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:a73f02406b2985b9145dd97f8221a929c9ef3289a8ba64c6b52043e240938528", size = 31503230, upload-time = "2026-03-16T10:53:50.312Z" },
-    { url = "https://files.pythonhosted.org/packages/d6/e3/ea3b79239953c3044d19d8e9581015da025b6640796db03799e435b17910/datafusion-52.3.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:118a1f0add6a3f91fcbc90c71819fe08750e2981637d5e7b346e099e94a20b8b", size = 28159497, upload-time = "2026-03-16T10:53:54.032Z" },
-    { url = "https://files.pythonhosted.org/packages/24/c8/7d325feb4b7509ae03857fd7e164e95ec72e8c9f3dfd3178ec7f80d53977/datafusion-52.3.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:253ce7aee5fe84bd6ee290c20608114114bdb5115852617f97d3855d36ad9341", size = 30769154, upload-time = "2026-03-16T10:53:57.835Z" },
-    { url = "https://files.pythonhosted.org/packages/37/ee/478689c69b3cb1ccabb2d52feac0c181f6cdf20b51a81df35344b1dab9a6/datafusion-52.3.0-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2af3469d2f06959bec88579ab107a72f965de18b32e607069bbdd0b859ed8dbb", size = 33060335, upload-time = "2026-03-16T10:54:01.715Z" },
-    { url = "https://files.pythonhosted.org/packages/1c/48/01906ab5c1a70373c6874ac5192d03646fa7b94d9ff06e3f676cb6b0f43f/datafusion-52.3.0-cp310-abi3-win_amd64.whl", hash = "sha256:9fb35738cf4dbff672dbcfffc7332813024cb0ad2ab8cda1fb90b9054277ab0c", size = 33765807, upload-time = "2026-03-16T10:54:05.728Z" },
+    { url = "https://files.pythonhosted.org/packages/46/58/4c5b981e3d9ade32a906c15a4941eef50c9b862781cdc14bf4dff48d026a/datafusion-54.0.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:946f55e48b8d523d7b4ac106bdf588b4493c2c66f81877d6952aafeaf7c3ec73", size = 39810553, upload-time = "2026-06-29T11:19:02.1Z" },
+    { url = "https://files.pythonhosted.org/packages/66/e5/5e4dbd42ce9a2affb3be90d9ab17cebde1a6f28b0d9fb4b83d612d5c8e42/datafusion-54.0.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:2a3bf43185c7e43e25242e5fb17b6a11b86bf976434c0bc493fdedbd9a080969", size = 37145255, upload-time = "2026-06-29T11:19:05.491Z" },
+    { url = "https://files.pythonhosted.org/packages/c6/5e/dbb9e6e3e5006d34f295d7ac73f1302c8f2df140666402a06e6c55028edb/datafusion-54.0.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:9432bf162381e9282cbc74915b8b773895de18be836f7e3f6d0de4d981f24630", size = 38853856, upload-time = "2026-06-29T11:19:08.732Z" },
+    { url = "https://files.pythonhosted.org/packages/a8/81/e69008e3479f4d0134875bc4ae39503bedcd55ca2597e71392c963c651b4/datafusion-54.0.0-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:3bcd4d213fa74710e75e6e182cc468c2bdbc5ffc74a08c8155d414fbbfa1b3f6", size = 41050149, upload-time = "2026-06-29T11:19:12.108Z" },
+    { url = "https://files.pythonhosted.org/packages/61/d4/8ba6e3fe3291c9ccc94b5ca3ec3c1fbcbfbe5ece5ffb965e4550844e2c56/datafusion-54.0.0-cp310-abi3-win_amd64.whl", hash = "sha256:b934e097e1bdca7d5768a81ac1bc4a1812cb459269f8b1a5d892a5d930f18376", size = 43444869, upload-time = "2026-06-29T11:19:15.963Z" },
+    { url = "https://files.pythonhosted.org/packages/9d/41/5608323226f21a0fa180823c531dbc0ed270e9b694f299b7647505cb6a06/datafusion-54.0.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:c4e79048da82ad89b768bd0be7df39254cd2a0afe2b719d1f129e8a7229af683", size = 39796248, upload-time = "2026-06-29T11:19:19.208Z" },
+    { url = "https://files.pythonhosted.org/packages/18/81/392ee323104ab14ca689384723b69e137064a828233c165574f97a74c0e9/datafusion-54.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:fe57038003b18e28b90752c1e32b44af74ec4f552a1904aee725e1129a00c447", size = 37153577, upload-time = "2026-06-29T11:19:22.397Z" },
+    { url = "https://files.pythonhosted.org/packages/40/c4/ebd5ef5349ecbea7f5f9da76c213581c13e7bbe1b5735c9925b279eeb4eb/datafusion-54.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:574f642832a106456cfc4f32aa82484c504fc32f4be2b510202bcb579de8e6d1", size = 38849839, upload-time = "2026-06-29T11:19:25.783Z" },
+    { url = "https://files.pythonhosted.org/packages/5a/b9/2383d30d317bb913cab97dbf2e6e1d5f37f594860d5c5bc176e025cf7d4a/datafusion-54.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:796fd5683927443c5bc61999d00b9007ef9b5ce107725ea8d241df718860985d", size = 41074623, upload-time = "2026-06-29T11:19:29.119Z" },
+    { url = "https://files.pythonhosted.org/packages/35/5c/553fd1107dede0a56727fda7216a7198d41394f2d19697f4fb104cc695ea/datafusion-54.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:64973c63874ec31670dd97b32b18af7b07fad679cb20d58ed154038e3a5c204e", size = 43438801, upload-time = "2026-06-29T11:19:32.799Z" },
 ]

 [[package]]
@@ -1828,19 +1843,19 @@ wheels = [

 [[package]]
 name = "lance-namespace"
-version = "0.7.7"
+version = "0.8.6"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "lance-namespace-urllib3-client" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/06/5c/9822af615fc1bd3ee1073994696c739aecde377be32435ec3303aed1bc5d/lance_namespace-0.7.7.tar.gz", hash = "sha256:d00b525f2e26993a6c61668e798bca6c808605ab8a79f29f86a1a1af92d91ae2", size = 10754, upload-time = "2026-05-20T17:32:59.45Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/af/12/f7ab93b29be3edbf5fc3610714bf2d06088e7f4524bfb38dfd6852458b08/lance_namespace-0.8.6.tar.gz", hash = "sha256:18232e721c8188145f4ec9389cc2dfbeeabf54a619d94885ea1b3375bee9f4af", size = 11529, upload-time = "2026-06-12T17:36:41.651Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/11/43/186acc1156da20c351db196e2b6241b2453b16dc1b4cc8e0a626667ca471/lance_namespace-0.7.7-py3-none-any.whl", hash = "sha256:477a7ca6b5e1f673a2c9ba52f42d6e8e3ff7c27a601392a21eb90fba98d0309b", size = 12581, upload-time = "2026-05-20T17:32:57.389Z" },
+    { url = "https://files.pythonhosted.org/packages/a0/1b/5b1668ee2dc8910965f390640359112a31157092fcf8e000b89c79b58708/lance_namespace-0.8.6-py3-none-any.whl", hash = "sha256:571eae34f9aad70e5b05020416c2860889b9ec82993ccd0eb015e7b39c3ea309", size = 13383, upload-time = "2026-06-12T17:36:43.456Z" },
 ]

 [[package]]
 name = "lance-namespace-urllib3-client"
-version = "0.7.7"
+version = "0.8.6"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "pydantic" },
@@ -1848,9 +1863,9 @@ dependencies = [
    { name = "typing-extensions" },
    { name = "urllib3" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/07/95/38ab81ccc1e09beeecd8ddfc61b8bc73831dc5053db1e3f9021f64a4896b/lance_namespace_urllib3_client-0.7.7.tar.gz", hash = "sha256:4d8c066628c17c6a10cf643b51a7f7ae1bfb8a614d9cc54a5af38a4ba2b4b102", size = 202930, upload-time = "2026-05-20T17:32:58.308Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c7/80/fb224b4a89c1c1638cde949cb6cce6c3aca7759effbfea46a3d9c3960b21/lance_namespace_urllib3_client-0.8.6.tar.gz", hash = "sha256:b6fb1d306e74a7576e5309919020be744527de484a63dbf5eed10f8b368548df", size = 228772, upload-time = "2026-06-12T17:36:42.609Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/35/96/5483e48e40433b1d078183c15a92c99e59a156041b0260e7f18ee34e7c08/lance_namespace_urllib3_client-0.7.7-py3-none-any.whl", hash = "sha256:9221c3e00fd89f0c811953d94b32d2ea527765280460a174f5872dc8a74c0ed6", size = 334767, upload-time = "2026-05-20T17:32:55.883Z" },
+    { url = "https://files.pythonhosted.org/packages/c5/90/1e27de15cd1b16785a1c7312beb0a59e75c8344a815f600f58173a565bd1/lance_namespace_urllib3_client-0.8.6-py3-none-any.whl", hash = "sha256:9d78249c3fb15aa3d15d668f78f04a275af3d08d800a7027492f37996ac4968b", size = 369950, upload-time = "2026-06-12T17:36:40.438Z" },
 ]

 [[package]]
@@ -1931,6 +1946,7 @@ tests = [
    { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.14'" },
    { name = "pandas", version = "3.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.14'" },
    { name = "polars" },
+    { name = "pyarrow" },
    { name = "pyarrow-stubs" },
    { name = "pylance" },
    { name = "pytest" },
@@ -1950,7 +1966,7 @@ requires-dist = [
    { name = "botocore", marker = "extra == 'embeddings'", specifier = ">=1.31.57" },
    { name = "cohere", marker = "extra == 'embeddings'", specifier = ">=4.0" },
    { name = "colpali-engine", marker = "extra == 'embeddings'", specifier = ">=0.3.10" },
-    { name = "datafusion", marker = "extra == 'tests'", specifier = ">=52,<53" },
+    { name = "datafusion", marker = "extra == 'tests'", specifier = ">=54,<55" },
    { name = "deprecation", specifier = ">=2.1.0" },
    { name = "duckdb", marker = "extra == 'tests'", specifier = ">=0.9.0" },
    { name = "google-genai", marker = "extra == 'embeddings'", specifier = ">=1.0.0" },
@@ -1978,10 +1994,11 @@ requires-dist = [
    { name = "polars", marker = "extra == 'tests'", specifier = ">=0.19,<=1.3.0" },
    { name = "pre-commit", marker = "extra == 'dev'", specifier = ">=3.5.0" },
    { name = "pyarrow", specifier = ">=16" },
+    { name = "pyarrow", marker = "extra == 'tests'", specifier = "<25" },
    { name = "pyarrow-stubs", marker = "extra == 'tests'", specifier = ">=16.0" },
    { name = "pydantic", specifier = ">=1.10" },
    { name = "pylance", marker = "extra == 'pylance'", specifier = ">=5.0.0b5" },
-    { name = "pylance", marker = "extra == 'tests'", specifier = ">=5.0.0b5" },
+    { name = "pylance", marker = "extra == 'tests'", specifier = "==9.0.0rc1" },
    { name = "pyright", marker = "extra == 'dev'", specifier = ">=1.1.350" },
    { name = "pytest", marker = "extra == 'tests'", specifier = ">=7.0" },
    { name = "pytest-asyncio", marker = "extra == 'tests'", specifier = ">=0.21" },
@@ -3837,8 +3854,8 @@ crypto = [

 [[package]]
 name = "pylance"
-version = "7.0.0"
-source = { registry = "https://pypi.org/simple" }
+version = "9.0.0rc1"
+source = { registry = "https://pypi.fury.io/lance-format" }
 dependencies = [
    { name = "lance-namespace" },
    { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
@@ -3846,12 +3863,12 @@ dependencies = [
    { name = "pyarrow" },
 ]
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/ac/ad/2f64921bf346e7075aef24a72595db44821724a3d89a9a92dd24e79632aa/pylance-7.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:98422021975be76e72b1572f41b8c9abb3bee5bdc9bfa5e9ce731110a65ed4d1", size = 62134146, upload-time = "2026-05-27T21:59:37.459Z" },
-    { url = "https://files.pythonhosted.org/packages/73/1c/c5a01bee0160b55d9a98895cbd33091d038f0a0995b121ab72e629008d02/pylance-7.0.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4bec86ee5b6fbd8bfc493e653f0a1fba0303cfe5492b9b46fc25ab908edc7183", size = 65373684, upload-time = "2026-05-27T22:04:01.584Z" },
-    { url = "https://files.pythonhosted.org/packages/eb/da/1fe8b8f7dbfe734d76af76acc994fc360a0d0c79a4874ef69f5a72a58fe3/pylance-7.0.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:881491432c53184e52f8d1db8d5f872f39a03f36fb104bec77b33d379519d8b5", size = 69458555, upload-time = "2026-05-27T22:16:50.567Z" },
-    { url = "https://files.pythonhosted.org/packages/76/f0/dd505cf3fd0226ab9d94759acd713125af1d3bfacfd80bbd52e3b9f89509/pylance-7.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:18453999e7fff4f76b16d6b7882c9df0628bd142ff95e2461bd7dd5ee3fe0af3", size = 65394430, upload-time = "2026-05-27T22:05:30.923Z" },
-    { url = "https://files.pythonhosted.org/packages/17/ba/2357b81034f28eb00790e258ed140289a6a887a7468ca9df6349fd186b27/pylance-7.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:04a58051d408c60fe76d41a220dcaf8fea8fb6d1aa0ca78a709b60bc3cc8d19a", size = 69473470, upload-time = "2026-05-27T22:17:18.935Z" },
-    { url = "https://files.pythonhosted.org/packages/1f/ec/5c00b6303a67d787f9475141832cbdc513d674ac3dcaeef8a7b169905e65/pylance-7.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:467d4864af047eaab4e1370e2f1e88e2c6f507c079874421116cb41d78bc3629", size = 74792863, upload-time = "2026-05-27T22:19:23.875Z" },
+    { url = "https://pypi.fury.io/lance-format/-/ver_vEHBE/pylance-9.0.0rc1-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:f0b6b02a1808bb3072ee7fe4e36614cae6f86302513e73ec7f55b2234a963b24" },
+    { url = "https://pypi.fury.io/lance-format/-/ver_1Jipm4/pylance-9.0.0rc1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:30f0ebf0d88034301819eb964f9236ce555aaa58e7ab89c5975a3e2250bbb405" },
+    { url = "https://pypi.fury.io/lance-format/-/ver_IvKxo/pylance-9.0.0rc1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:44609ea2615ea6e684b85478d1694af2026458f61cf7895ecc75e238bfd17aa8" },
+    { url = "https://pypi.fury.io/lance-format/-/ver_2hidj1/pylance-9.0.0rc1-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:182167a8dba9eeabffbffd53bd5b8548613d4d459b7cd7b34a840dd00cbb806f" },
+    { url = "https://pypi.fury.io/lance-format/-/ver_1dFx3r/pylance-9.0.0rc1-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:8a63b11e814b7eab758bcaf0d6f97eb05ea86203d9fb0af718c462c24c7d6c9c" },
+    { url = "https://pypi.fury.io/lance-format/-/ver_2a8dSh/pylance-9.0.0rc1-cp310-abi3-win_amd64.whl", hash = "sha256:2ff8b953ae2b0550490c1a7efd210aa91bc223d200ffac28849056cfd7436d97" },
 ]

 [[package]]
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb"
-version = "0.32.0-beta.1"
+version = "0.32.0-beta.2"
 edition.workspace = true
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license.workspace = true
@@ -50,6 +50,8 @@ lance-namespace = { workspace = true }
 lance-namespace-impls = { workspace = true }
 metrics = { workspace = true, optional = true }
 metrics-util = { workspace = true, optional = true }
+# Pin the transitive GooseFS SDK until the 0.1.6 compile break is fixed upstream.
+goosefs-sdk = { version = "=0.1.5", optional = true }
 moka = { workspace = true }
 pin-project = { workspace = true }
 tokio = { version = "1.23", features = ["rt-multi-thread", "sync"] }
@@ -132,6 +134,7 @@ azure = [
 ]
 cos = ["lance/tencent", "lance-io/tencent"]
 goosefs = [
+    "dep:goosefs-sdk",
    "lance/goosefs",
    "lance-io/goosefs",
    "lance-namespace-impls/dir-goosefs",
@@ -8,13 +8,13 @@ use arrow_array::{RecordBatch, UInt64Array};
 use futures::{StreamExt, TryStreamExt};
 use lance::io::ObjectStore;
 use lance_core::{cache::LanceCache, utils::futures::FinallyStreamExt};
-use lance_encoding::decoder::DecoderPlugins;
+use lance_encoding::decoder::{DecoderPlugins, FilterExpression};
 use lance_file::{
    reader::{FileReader, FileReaderOptions},
    writer::{FileWriter, FileWriterOptions},
 };
-use lance_index::scalar::IndexReader;
 use lance_io::{
+    ReadBatchParams,
    scheduler::{ScanScheduler, SchedulerConfig},
    utils::CachedFileSize,
 };
@@ -216,6 +216,7 @@ impl Shuffler {
        let scan_scheduler = ScanScheduler::new(Arc::new(object_store), scheduler_config);
        let job_id = self.id.clone();
        let rng = Arc::new(Mutex::new(rng));
+        let read_schema = arrow_schema.clone();

        // Second pass, read each file as a single batch and shuffle
        let stream = futures::stream::iter(0..num_files)
@@ -224,6 +225,7 @@ impl Shuffler {
                let rng = rng.clone();
                let tmp_dir = tmp_dir.clone();
                let job_id = job_id.clone();
+                let read_schema = read_schema.clone();
                async move {
                    let path = tmp_dir.join(format!("shuffle_{}_{file_index}.lance", job_id));
                    let path = object_store::path::Path::from_absolute_path(path).unwrap();
@@ -239,7 +241,19 @@ impl Shuffler {
                    )
                    .await?;
                    // Need to read the entire file in a single batch for in-memory shuffling
-                    let batch = reader.read_record_batch(0, reader.num_rows()).await?;
+                    let batches = reader
+                        .read_stream(
+                            ReadBatchParams::RangeFull,
+                            reader.num_rows() as u32,
+                            1,
+                            FilterExpression::no_filter(),
+                        )
+                        .await?
+                        .try_collect::<Vec<_>>()
+                        .await?;
+                    // An empty file yields no batches; fall back to an empty batch
+                    // with the expected schema so shuffling handles it gracefully.
+                    let batch = concat_batches(&read_schema, &batches)?;
                    let mut rng = rng.lock().unwrap_or_else(|e| e.into_inner());
                    Self::shuffle_batch(&batch, &mut rng, clump_size)
                }
@@ -8,7 +8,7 @@ use std::sync::{

 use arrow_array::{Array, BooleanArray, RecordBatch, UInt64Array};
 use arrow_schema::{DataType, Field, Schema};
-use datafusion_common::hash_utils::create_hashes;
+use datafusion_common::hash_utils::{RandomState, create_hashes};
 use futures::{StreamExt, TryStreamExt};
 use lance_arrow::SchemaExt;

@@ -234,7 +234,7 @@ impl Splitter {
            .cloned()
            .collect::<Vec<_>>();
        let mut hashes = vec![0; batch.num_rows()];
-        let random_state = ahash::RandomState::with_seeds(0, 0, 0, 0);
+        let random_state = RandomState::with_seed(0);
        create_hashes(&arrays, &random_state, &mut hashes).unwrap();
        // As an example, let's assume the weights are 1, 2.  Our total weight is 3.
        //
@@ -761,8 +761,8 @@ mod tests {
        verify_splitter(splitter, test_data(), 50, &[11, 8, 9], false).await;
    }

-    #[tokio::test]
-    async fn test_hash_split() {
+    async fn collect_hash_split() -> RecordBatch {
+        let total_rows = 50;
        let data = lance_datagen::gen_batch()
            .with_seed(Seed::from(42))
            .col(
@@ -783,7 +783,7 @@ mod tests {
        );

        let split_batches = splitter
-            .apply(data, 10)
+            .apply(data, total_rows)
            .await
            .unwrap()
            .try_collect::<Vec<_>>()
@@ -791,20 +791,35 @@ mod tests {
            .unwrap();

        let schema = split_batches[0].schema();
-        let split_batch = concat_batches(&schema, &split_batches).unwrap();
+        concat_batches(&schema, &split_batches).unwrap()
+    }

-        // These assertions are all based on fixed seed in data generation but they match
-        // up roughly to what we expect (25% discarded, 25% in split 0, 50% in split 1)
+    #[tokio::test]
+    async fn test_hash_split() {
+        let total_rows = 50;
+        let split_batch = collect_hash_split().await;
+        let split_batch_again = collect_hash_split().await;

-        // 14 rows (28%) are discarded because discard_weight is 1
-        assert_eq!(split_batch.num_rows(), 36);
+        assert_eq!(split_batch.num_rows(), split_batch_again.num_rows());
+        assert_eq!(split_batch.num_columns(), split_batch_again.num_columns());
+        for (left, right) in split_batch
+            .columns()
+            .iter()
+            .zip(split_batch_again.columns().iter())
+        {
+            assert_eq!(left, right);
+        }
+
+        assert!(split_batch.num_rows() > 0);
+        assert!(split_batch.num_rows() < total_rows);
        assert_eq!(split_batch.num_columns(), 2);

        let split_ids = split_batch.column(1).as_primitive::<UInt64Type>().values();
        let num_in_split_0 = split_ids.iter().filter(|v| **v == 0).count();
        let num_in_split_1 = split_ids.iter().filter(|v| **v == 1).count();

-        assert_eq!(num_in_split_0, 11); // 22%
-        assert_eq!(num_in_split_1, 25); // 50%
+        assert_eq!(num_in_split_0 + num_in_split_1, split_batch.num_rows());
+        assert!(num_in_split_0 > 0);
+        assert!(num_in_split_1 > num_in_split_0);
    }
 }
@@ -614,6 +614,12 @@ pub struct QueryExecutionOptions {
    pub max_batch_length: u32,
    /// Max duration to wait for the query to execute before timing out.
    pub timeout: Option<Duration>,
+    /// How distributed worker metrics should be displayed by
+    /// [`ExecutableQuery::analyze_plan`].
+    ///
+    /// This only affects remote distributed query plans. Local query execution
+    /// ignores this option.
+    pub analyze_plan_distributed_metrics: AnalyzePlanDistributedMetrics,
 }

 impl Default for QueryExecutionOptions {
@@ -621,6 +627,7 @@ impl Default for QueryExecutionOptions {
        Self {
            max_batch_length: 1024,
            timeout: None,
+            analyze_plan_distributed_metrics: AnalyzePlanDistributedMetrics::Aggregate,
        }
    }
 }
@@ -633,6 +640,29 @@ impl QueryExecutionOptions {
    }
 }

+/// How distributed worker metrics are displayed in analyzed query plans.
+#[non_exhaustive]
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub enum AnalyzePlanDistributedMetrics {
+    /// Preserve the legacy output: aggregate worker metrics into one synthetic tree.
+    #[default]
+    Aggregate,
+    /// Render one raw worker-side tree per distributed worker.
+    PerWorker,
+    /// Render the aggregate tree followed by the raw per-worker trees.
+    Full,
+}
+
+impl AnalyzePlanDistributedMetrics {
+    pub(crate) fn as_query_param(self) -> &'static str {
+        match self {
+            Self::Aggregate => "aggregate",
+            Self::PerWorker => "per_worker",
+            Self::Full => "full",
+        }
+    }
+}
+
 /// A trait for a query object that can be executed to get results
 ///
 /// There are various kinds of queries but they all return results
@@ -36,7 +36,7 @@ use crate::{DistanceType, Error};
 use crate::{
    error::Result,
    index::{IndexBuilder, IndexConfig},
-    query::QueryExecutionOptions,
+    query::{AnalyzePlanDistributedMetrics, QueryExecutionOptions},
    table::{
        AddDataBuilder, BaseTable, OptimizeAction, OptimizeStats, TableDefinition, UpdateBuilder,
        merge::MergeInsertBuilder,
@@ -1250,8 +1250,7 @@ impl<S: HttpSend + 'static> RemoteTable<S> {

            match result {
                Ok(_) => {
-                    let add_result = insert
-                        .as_any()
+                    let add_result = (insert.as_ref() as &dyn std::any::Any)
                        .downcast_ref::<RemoteInsertExec<S>>()
                        .and_then(|i| i.add_result())
                        .unwrap_or(AddResult { version: 0 });
@@ -1993,9 +1992,16 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
    async fn analyze_plan(
        &self,
        query: &AnyQuery,
-        _options: QueryExecutionOptions,
+        options: QueryExecutionOptions,
    ) -> Result<String> {
-        let request = self.post_read(&format!("/v1/table/{}/analyze_plan/", self.identifier));
+        let mut request = self.post_read(&format!("/v1/table/{}/analyze_plan/", self.identifier));
+
+        if options.analyze_plan_distributed_metrics != AnalyzePlanDistributedMetrics::Aggregate {
+            request = request.query(&[(
+                "distributed_metrics",
+                options.analyze_plan_distributed_metrics.as_query_param(),
+            )]);
+        }

        let query_bodies = self.prepare_query_bodies(query).await?;
        let requests: Vec<reqwest::RequestBuilder> = query_bodies
@@ -2840,7 +2846,10 @@ mod tests {
    use crate::{
        DistanceType, Error, Table,
        index::{Index, IndexStatistics, IndexType, vector::IvfPqIndexBuilder},
-        query::{ColumnOrdering, ExecutableQuery, QueryBase},
+        query::{
+            AnalyzePlanDistributedMetrics, ColumnOrdering, ExecutableQuery, QueryBase,
+            QueryExecutionOptions,
+        },
        remote::ARROW_FILE_CONTENT_TYPE,
    };

@@ -4048,6 +4057,42 @@ mod tests {
            .unwrap();
    }

+    #[tokio::test]
+    async fn test_analyze_plan_distributed_metrics_query_param() {
+        let table = Table::new_with_handler("my_table", |request| {
+            assert_eq!(request.method(), "POST");
+            assert_eq!(request.url().path(), "/v1/table/my_table/analyze_plan/");
+            assert_eq!(
+                request
+                    .url()
+                    .query_pairs()
+                    .find(|(k, _)| k == "distributed_metrics"),
+                Some(("distributed_metrics".into(), "per_worker".into()))
+            );
+
+            let body = request.body().unwrap().as_bytes().unwrap();
+            let body: serde_json::Value = serde_json::from_slice(body).unwrap();
+            assert_eq!(body["k"], serde_json::json!(1));
+
+            http::Response::builder()
+                .status(200)
+                .body(r#""analyzed plan""#)
+                .unwrap()
+        });
+
+        let result = table
+            .query()
+            .limit(1)
+            .analyze_plan_with_options(QueryExecutionOptions {
+                analyze_plan_distributed_metrics: AnalyzePlanDistributedMetrics::PerWorker,
+                ..Default::default()
+            })
+            .await
+            .unwrap();
+
+        assert_eq!(result, "analyzed plan");
+    }
+
    #[tokio::test]
    async fn test_query_structured_fts() {
        let table =
@@ -3,7 +3,6 @@

 //! DataFusion ExecutionPlan for inserting data into remote LanceDB tables.

-use std::any::Any;
 use std::sync::{Arc, Mutex};

 use arrow_array::{ArrayRef, RecordBatch, UInt64Array};
@@ -237,10 +236,6 @@ impl<S: HttpSend + 'static> ExecutionPlan for RemoteInsertExec<S> {
        Self::static_name()
    }

-    fn as_any(&self) -> &dyn Any {
-        self
-    }
-
    fn properties(&self) -> &Arc<PlanProperties> {
        &self.properties
    }
@@ -89,10 +89,6 @@ impl ExecutionPlan for MetadataEraserExec {
        "MetadataEraserExec"
    }

-    fn as_any(&self) -> &dyn std::any::Any {
-        self
-    }
-
    fn properties(&self) -> &Arc<PlanProperties> {
        &self.properties
    }
@@ -138,7 +134,7 @@ impl ExecutionPlan for MetadataEraserExec {
        )
    }

-    fn partition_statistics(&self, partition: Option<usize>) -> DataFusionResult<Statistics> {
+    fn partition_statistics(&self, partition: Option<usize>) -> DataFusionResult<Arc<Statistics>> {
        self.input.partition_statistics(partition)
    }

@@ -188,10 +184,6 @@ impl BaseTableAdapter {

 #[async_trait]
 impl TableProvider for BaseTableAdapter {
-    fn as_any(&self) -> &dyn std::any::Any {
-        self
-    }
-
    fn schema(&self) -> Arc<ArrowSchema> {
        self.schema.clone()
    }
@@ -3,7 +3,6 @@

 //! DataFusion ExecutionPlan for inserting data into LanceDB tables.

-use std::any::Any;
 use std::sync::atomic::{AtomicU64, Ordering};
 use std::sync::{Arc, LazyLock, Mutex};

@@ -146,10 +145,6 @@ impl ExecutionPlan for InsertExec {
        Self::static_name()
    }

-    fn as_any(&self) -> &dyn Any {
-        self
-    }
-
    fn properties(&self) -> &Arc<PlanProperties> {
        &self.properties
    }
@@ -3,7 +3,6 @@

 //! A DataFusion projection that rejects vectors containing NaN values.

-use std::any::Any;
 use std::sync::{Arc, LazyLock};

 use arrow_array::{Array, FixedSizeListArray};
@@ -86,10 +85,6 @@ impl RejectNanUdf {
 }

 impl ScalarUDFImpl for RejectNanUdf {
-    fn as_any(&self) -> &dyn Any {
-        self
-    }
-
    fn name(&self) -> &str {
        "reject_nan"
    }
@@ -66,10 +66,6 @@ impl ExecutionPlan for ScannableExec {
        "ScannableExec"
    }

-    fn as_any(&self) -> &dyn std::any::Any {
-        self
-    }
-
    fn properties(&self) -> &Arc<PlanProperties> {
        &self.properties
    }
@@ -121,14 +117,14 @@ impl ExecutionPlan for ScannableExec {
        )))
    }

-    fn partition_statistics(&self, _partition: Option<usize>) -> DFResult<Statistics> {
-        Ok(Statistics {
+    fn partition_statistics(&self, _partition: Option<usize>) -> DFResult<Arc<Statistics>> {
+        Ok(Arc::new(Statistics {
            num_rows: self
                .num_rows
                .map(Precision::Exact)
                .unwrap_or(Precision::Absent),
            total_byte_size: Precision::Absent,
            column_statistics: vec![],
-        })
+        }))
    }
 }
@@ -137,8 +137,7 @@ mod tests {
            };

            // Downcast to BaseTableAdapter and apply FTS query
-            let base_adapter = table_provider
-                .as_any()
+            let base_adapter = (table_provider.as_ref() as &dyn std::any::Any)
                .downcast_ref::<BaseTableAdapter>()
                .ok_or_else(|| {
                    DataFusionError::Internal(
@@ -982,10 +982,10 @@ mod tests {
        use crate::table::query::create_plan;

        fn find_ann_approx_mode(plan: &dyn ExecutionPlan) -> Option<ApproxMode> {
-            if let Some(ann) = plan.as_any().downcast_ref::<ANNIvfSubIndexExec>() {
+            if let Some(ann) = (plan as &dyn std::any::Any).downcast_ref::<ANNIvfSubIndexExec>() {
                return Some(ann.query().approx_mode);
            }
-            if let Some(ann) = plan.as_any().downcast_ref::<ANNIvfPartitionExec>() {
+            if let Some(ann) = (plan as &dyn std::any::Any).downcast_ref::<ANNIvfPartitionExec>() {
                return Some(ann.query.approx_mode);
            }
            plan.children()
Author	SHA1	Message	Date
Jack Ye	6416840c33	test: update python expectations for lance 9.1	2026-07-16 13:46:54 -07:00
Jack Ye	b030361d86	test: relax hash split distribution assertions	2026-07-16 10:45:35 -07:00
lancedb automation	325cab394b	chore: update lance dependency to v9.1.0-beta.2	2026-07-16 10:45:05 -07:00
LanceDB Robot	bc8674ab22	chore!: update lance dependency to v9.0.0-rc.1 (#3673 ) BREAKING CHANGE: splits generated by the permutation data loader will not be the same, due to a change in hash function. Updates the Lance dependencies and Java lance-core to [v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1). Includes the required DataFusion 54 and Lance file-reader compatibility updates. --------- Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:31:30 -07:00
Jack Ye	37032151d3	feat: support distributed analyze plan metrics in clients (#3675 ) Adds client-side support for analyze_plan distributed metrics modes across Rust, Python, and TypeScript clients. Defaults to aggregate for backward compatibility and sends the remote distributed_metrics parameter only when a non-default mode is requested.	2026-07-15 21:21:40 -07:00
Dan Tasse	00c4a7b843	chore: consolidate skills into one (#3672 ) Consolidating skills so we have only one `lancedb` skill, making it easier to install and work with, vs. installing and using different skills for "lancedb-column-metadata", "lancedb-branch-ops", etc. Also deleted lancedb-connect, because the new monoskill uses the python/TS APIs so it doesn't need extra handholding to connect to the REST API. ## does it work? Test 1: do some column metadata operations with 1. no skills, 2. our previous baseline lancedb skill, 3. the baseline lancedb skill with the lancedb-column-metadata skill folded in: ``` ┌───────────────────────────┬──────────────────────┬────────────────────┬───────────────────────┐ │ eval │ no-skill │ lancedb (original) │ lancedb2-incl-columns │ ├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤ │ 2-add-all-metadata-types │ 2.5/5 · 116s · $0.44 │ 0/5 · 154s · $0.60 │ 5/5 · 58s · $0.26 │ ├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤ │ 3-delete-one-metadata-key │ 4/4 · 67s · $0.23 │ 4/4 · 100s · $0.45 │ 4/4 · 45s · $0.20 │ ├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤ │ TOTAL (per rep avg) │ 6.5/9 · 183s · $0.66 │ 4/9 · 254s · $1.05 │ 9/9 · 103s · $0.46 │ └───────────────────────────┴──────────────────────┴────────────────────┴───────────────────────┘ ``` without column-metadata-specific content, it failed because it wrote keys like `description` instead of `lancedb:description`. That's pretty undiscoverable without the skill. Test 2: do some simple branch operations with 1. no skills, 2. our previous baseline lancedb skill, 3. the combined skill (in this PR): ``` ┌───────────────────────────┬────────────────────┬────────────────────┬────────────────────────────────┐ │ eval │ no-skill │ lancedb (original) │ lancedb3-incl-columns-branches │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ 5-create-branch │ 2/2 · 64s · $0.30 │ 2/2 · 57s · $0.34 │ 2/2 · 38s · $0.22 │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ 6-delete-branch │ 2/2 · 38s · $0.21 │ 2/2 · 44s · $0.27 │ 2/2 · 40s · $0.22 │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ 7-switch-branch-and-write │ 1/2 · 96s · $0.49 │ 1/2 · 106s · $0.58 │ 2/2 · 66s · $0.40 │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ TOTAL (per rep avg) │ 5/6 · 199s · $1.00 │ 5/6 · 207s · $1.20 │ 6/6 · 143s · $0.83 │ └───────────────────────────┴────────────────────┴────────────────────┴────────────────────────────────┘ ``` Test 3: run everything, with the lancedb (original) skill, lancedb(original) + all the separate skills, and lancedb3-incl-columns-branches ``` ┌────────────────────────────────┬──────────────────────┬────────────────────────────────┬─────────────────────────────────────────────────────────────────┐ │ eval │ lancedb (original) │ lancedb3-incl-columns-branches │ all-separate (lancedb + connect + column-metadata + branch-ops) │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 1-pick-column-for-image-search │ 2/2 · 141s · $0.67 │ 2/2 · 128s · $0.43 │ 1/2 · 257s · $0.81 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 2-add-all-metadata-types │ 5/5 · 93s · $0.52 │ 5/5 · 54s · $0.30 │ 5/5 · 40s · $0.24 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 3-delete-one-metadata-key │ 4/4 · 84s · $0.44 │ 4/4 · 36s · $0.24 │ 4/4 · 28s · $0.20 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 4-build-index │ 3/3 · 275s · $0.52 │ 3/3 · 84s · $0.39 │ 3/3 · 68s · $0.50 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 5-create-branch │ 2/2 · 39s · $0.31 │ 2/2 · 41s · $0.20 │ 2/2 · 15s · $0.17 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 6-delete-branch │ 2/2 · 59s · $0.25 │ 2/2 · 69s · $0.29 │ 2/2 · 26s · $0.17 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 7-switch-branch-and-write │ 1/2 · 84s · $0.46 │ 2/2 · 62s · $0.40 │ 2/2 · 27s · $0.23 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ TOTAL │ 19/20 · 774s · $3.17 │ 20/20 · 474s · $2.24 │ 19/20 · 462s · $2.32 │ └────────────────────────────────┴──────────────────────┴────────────────────────────────┴─────────────────────────────────────────────────────────────────┘ ``` ("lancedb3-incl-columns-branches" is the combined skill in this PR, all-separate is using the four separate skills.) For overall performance, it helps to have the specialized skills for metadata and branching; doesn't really matter whether they're separate skills or all together. Also doesn't matter much whether it's REST or Python. So let's merge these skills to make it easier for users.	2026-07-15 16:54:23 -04:00
LanceDB Robot	1773fb2239	chore: update lance dependency to v9.0.0-beta.24 (#3667 ) Updates the Rust workspace Lance dependencies and Java lance-core dependency to v9.0.0-beta.24. Lance tag: https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.24	2026-07-15 12:42:29 -05:00
Lance Release	8a4eaaa8b9	Bump version: 0.32.0-beta.1 → 0.32.0-beta.2	2026-07-14 23:28:32 +00:00