Bump version: 0.35.0-beta.2 → 0.35.0-beta.3

feat(fts): add block size configuration (#3691 )
## What changed - add `block_size` to Python FTS configuration and the deprecated local/remote helpers - add `blockSize` to the TypeScript FTS options and propagate it through the NAPI binding - serialize the value as `block_size` for remote index creation - document the existing Rust builder API and generate the TypeScript API reference - add local, remote, metadata, search, and invalid-value regression coverage ## Why Lance supports configuring the number of documents per compressed FTS posting block, but LanceDB's Python and TypeScript APIs did not expose the setting. This made the experimental FTS V3 layout unavailable through those clients and allowed the value to be dropped before index creation. ## How it works The default remains `128`. Supported values are `128` and `256`; selecting `256` uses the experimental FTS V3 format. Invalid values are rejected by the Lance builder and surfaced as Python or JavaScript errors. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo +1.94.0 clippy --quiet --features remote --tests --examples -- -D warnings` - targeted Rust local and remote index tests - Rust doctests: 34 passed - Python Ruff checks, doctest, and targeted local/remote tests: 5 passed - TypeScript build, Biome lint, generated docs, and targeted Jest tests: 9 passed - `git diff --check` ## Limitations The Java client remains unchanged because its external remote REST model does not currently expose `block_size`. Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
2026-08-26 16:08:43 +00:00 · 2026-07-24 22:03:30 +00:00 · 2026-07-24 15:02:38 -07:00 · 2026-07-24 15:02:23 -07:00 · 2026-07-24 15:01:30 -07:00 · 2026-07-22 14:11:57 -07:00
128 changed files with 5561 additions and 967 deletions
@@ -0,0 +1,20 @@
+{
+  "name": "lancedb",
+  "interface": {
+    "displayName": "LanceDB"
+  },
+  "plugins": [
+    {
+      "name": "lancedb",
+      "source": {
+        "source": "local",
+        "path": "./plugins/lancedb"
+      },
+      "policy": {
+        "installation": "AVAILABLE",
+        "authentication": "ON_INSTALL"
+      },
+      "category": "Developer Tools"
+    }
+  ]
+}
@@ -5,3 +5,7 @@ This directory contains repo-scoped code agent skills for the LanceDB project.
 Each skill is a folder that contains a required `SKILL.md` and optional bundled resources.

 Codex discovers skills from `.agents/skills` in the current working directory and parent directories.
+
+The `lancedb` skill lives in the `plugins/lancedb` plugin (see `plugins/lancedb/skills/lancedb`)
+so it can be installed via the plugin marketplaces (`.claude-plugin/marketplace.json` and
+`.agents/plugins/marketplace.json`); the `lancedb` entry here is a symlink into that plugin.
@@ -0,0 +1 @@
+../../plugins/lancedb/skills/lancedb
@@ -1,145 +0,0 @@
---
-name: lancedb-branch-ops
-description: >-
-  Manage LanceDB table branches through the REST API: list, create, and delete
-  branches; target schema reads, field-metadata updates, and index creation to a
-  named branch; and verify that branch changes remain isolated from main. Use
-  when a task involves branch lifecycle, an experimental or isolated table
-  version, directing an operation to a non-main branch, or confirming that a
-  mutation did not affect main. This skill also explains that LanceDB has no
-  checkout operation; each request selects its target branch in the request
-  body.
---
-
-## Goal
-
-Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main.
-
-## Step 0: Establish the connection
-
-Use the `lancedb-connect` skill to resolve the base URL and auth headers (`x-api-key`, `x-lancedb-database`). Skip this only if the connection is already known from the current conversation.
-
-All examples below use `{base_url}` — substitute the resolved endpoint and include the auth headers on every request.
-
-## The branch model (important)
-
-LanceDB branches are named snapshots that diverge from the table's current state at creation time. There is **no checkout command** — you never switch the whole table to a branch. Instead, you **pass `"branch": "<name>"` in the request body** of any operation to target that branch. Omitting the key (or sending an empty body) always targets main.
-
-`branches/list` returns only non-main branches. Main always exists and is not listed.
-
-## List branches
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/list
-Content-Type: application/json
-
-{}
-```
-
-Response:
-```json
-{
-  "branches": {
-    "experiment-reindex": {"parentVersion": 1, "createAt": 1782506085, "manifestSize": 1029}
-  }
-}
-```
-
-If `branches` is `{}`, the table has no branches besides main.
-
-## Create a branch
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/create
-Content-Type: application/json
-
-{"name": "experiment-reindex"}
-```
-
-HTTP 200 with `{}` body = success. The branch is created off the table's current state on main.
-
-Verify by calling `branches/list` and confirming the new name appears.
-
-## Delete a branch
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/delete
-Content-Type: application/json
-
-{"name": "stale-2024"}
-```
-
-HTTP 200 with `{}` body = success. Only the branch pointer is removed — main and all row data remain intact.
-
-Verify by calling `branches/list` (name gone) and `describe` with no branch param (main still responds).
-
-## Operate on a specific branch
-
-Pass `"branch": "<name>"` in the body of any operation to scope it to that branch:
-
-**Read schema on a branch:**
-```http
-POST {base_url}/v1/table/{table_id}/describe
-Content-Type: application/json
-
-{"branch": "wip-branch"}
-```
-
-**Write metadata to a branch (not main):**
-```http
-POST {base_url}/v1/table/{table_id}/update_field_metadata
-Content-Type: application/json
-
-{
-  "branch": "wip-branch",
-  "updates": [
-    {
-      "path": "category",
-      "metadata": {"lancedb:description": "Product category label."},
-      "replace": false
-    }
-  ]
-}
-```
-
-**Build an index on a branch:**
-```http
-POST {base_url}/v1/table/{table_id}/create_index
-Content-Type: application/json
-
-{
-  "branch": "wip-branch",
-  "column": "category",
-  "index_type": "BTREE"
-}
-```
-
-## Verifying isolation
-
-After writing to a branch, always confirm the change did NOT land on main:
-
-```bash
-# Should show the new metadata
-curl -s -X POST {base_url}/v1/table/{table_id}/describe \
-  -H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
-  -H "content-type: application/json" \
-  -d '{"branch": "wip-branch"}'
-
-# Should NOT show the new metadata
-curl -s -X POST {base_url}/v1/table/{table_id}/describe \
-  -H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
-  -H "content-type: application/json" \
-  -d '{}'
-```
-
-## Quick reference
-
-| Goal | Endpoint | Body |
-|------|----------|------|
-| List all branches | `branches/list` | `{}` |
-| Create a branch | `branches/create` | `{"name": "..."}` |
-| Delete a branch | `branches/delete` | `{"name": "..."}` |
-| Read schema on branch | `describe` | `{"branch": "..."}` |
-| Write metadata on branch | `update_field_metadata` | `{"branch": "...", "updates": [...]}` |
-| Build index on branch | `create_index` | `{"branch": "...", "column": ..., "index_type": ...}` |
-| Target main (default) | any endpoint | omit `"branch"` key |
@@ -1,178 +0,0 @@
---
-name: lancedb-column-metadata
-description: Column metadata authoring for LanceDB tables via the REST API. This skill is required for tasks like writing field descriptions, setting tags on columns (field_type, model, project_id, version), classifying columns as embeddings vs labels vs eval metrics, or grouping versioned columns into logical families — because it has the API integration needed to read the schema and persist metadata back. Invoke whenever someone wants to document, annotate, tag, or classify what their table columns ARE. Trigger even without an explicit "LanceDB" mention, as long as the context is column-level documentation or tagging for an ML or vector database table.
-metadata:
-  short-description: Write column descriptions, tags, and logical groupings to a LanceDB table
---
-
-## Overview
-
-This skill authors column-level metadata for a LanceDB table. It connects to a LanceDB deployment over its REST API, inspects the table schema, generates appropriate metadata, and writes it back.
-
-## Step 0: Establish the connection
-
-Use the `lancedb-connect` skill (invoke it via the Skill tool) to resolve the base URL and auth headers (`x-api-key`, `x-lancedb-database`) for whichever deployment the user is working against — enterprise/self-hosted or a local dev server. Skip it only if the connection details are already established in the conversation.
-
-All examples below use `{base_url}` — substitute the resolved endpoint and include the resolved headers on every request.
-
-## Metadata keys
-
-All metadata uses namespaced keys:
-
-| Key | Purpose | Example value |
-|-----|---------|---------------|
-| `lancedb:description` | Human-readable explanation of what the column contains | `"CLIP ViT-L/14 image embedding, L2-normalized (768-dim)"` |
-| `lancedb:tag:<name>` | Flexible key-value tag; the suffix names the tag category | `lancedb:tag:field_type: "embedding"`, `lancedb:tag:model: "clip"`, `lancedb:tag:project_id: "foo"` |
-| `lancedb:logical-column` | Logical group/family this column belongs to | `"clip_features"` |
-
-Tags are open-ended — use whatever key suffix and value make sense given the user's intent. The tag suffix should describe *what is being classified* (e.g., `field_type`, `model`, `project_id`) and the value describes *how*.
-
-## Step 1: Resolve the table identifier
-
-You need:
- **Table name** (required) — e.g., `my_table` or `my_namespace.my_table`
- **Database name** — ask if not provided and not inferable from context; it goes in the `x-lancedb-database` header, never in the URL path
-
-The table identifier in the URL path is typically `table_name` for a top-level table, or `namespace$table_name` if the table lives in a namespace. The API accepts a `delimiter` query parameter to parse compound identifiers (default `$`).
-
-## Step 2: Describe the table
-
-```http
-POST {base_url}/v1/table/{table_id}/describe
-Content-Type: application/json
-
-{}
-```
-
-The response contains `schema.fields` — an array of field objects:
-
-```json
-{
-  "schema": {
-    "fields": [
-      {
-        "name": "clip_embedding_v3",
-        "type": { "type": "FixedSizeList", "fields": [...], "listSize": 768 },
-        "nullable": true,
-        "metadata": { "lancedb:description": "..." }
-      }
-    ]
-  }
-}
-```
-
-Each field has:
- `name` — field name
- `type` — Arrow data type (check `type.type` for the type string)
- `nullable` — boolean
- `metadata` — existing key-value metadata (read this before writing to avoid redundant updates)
-
-For struct/nested fields, recurse into `type.fields` and represent them as dot-notation paths (e.g., `parent.child`).
-
-If the user hasn't specified which columns to update, work with all columns.
-
-## Step 3: Generate metadata
-
-Decide what to generate based on the user's request.
-
-### Writing descriptions (`lancedb:description`)
-
-Base descriptions on:
- The column name and Arrow type (e.g., `FixedSizeList` of floats → likely an embedding)
- User-supplied context (upstream pipeline, sample values, domain knowledge)
- Name patterns: `_embedding`/`_vec`/`_embed` → vector; `_label`/`_class` → label; `_score`/`_eval`/`_metric` → evaluation metric
-
-Be specific and concise. Good: `"Sentence-BERT embedding of the query text (768-dim)."` Not: `"An embedding column."`
-
-### Tagging columns (`lancedb:tag:<name>`)
-
-Choose tag key names that match what the user asked to annotate. Common patterns:
-
- Semantic field type → `lancedb:tag:field_type: "embedding"` / `"text"` / `"image"` / `"label"` / `"eval"` / `"id"` / `"metadata"`
- Model or source → `lancedb:tag:model: "clip"` / `"bert"` / `"vit"`
- Project affiliation → `lancedb:tag:project_id: "<name>"`
- Version → `lancedb:tag:version: "v3"` (and `lancedb:tag:latest: "true"` for the newest)
-
-Use Arrow type as a hint: `FixedSizeList` + float → embedding; `Utf8`/`LargeUtf8` → text; `Binary` → image or blob.
-
-Multiple tags on the same column are fine — each is a separate key.
-
-### Grouping into logical columns (`lancedb:logical-column`)
-
-Look for naming patterns across columns:
- `clip_v1`, `clip_v2`, `clip_v3` → logical column `"clip"`, latest is `v3`
- `text_embed_20240101`, `text_embed_20240601` → logical column `"text_embed"`, latest is the most recent date suffix
-
-Write `lancedb:logical-column` on all members of a group. Mark the newest with `lancedb:tag:latest: "true"` (in addition to its version tag).
-
-## Step 4: Write the metadata
-
-```http
-POST {base_url}/v1/table/{table_id}/update_field_metadata
-Content-Type: application/json
-
-{
-  "updates": [
-    {
-      "path": "clip_v3",
-      "metadata": {
-        "lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
-        "lancedb:tag:field_type": "embedding",
-        "lancedb:tag:model": "clip",
-        "lancedb:tag:version": "v3",
-        "lancedb:tag:latest": "true",
-        "lancedb:logical-column": "clip"
-      },
-      "replace": false
-    },
-    {
-      "path": "clip_v2",
-      "metadata": {
-        "lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
-        "lancedb:tag:field_type": "embedding",
-        "lancedb:tag:model": "clip",
-        "lancedb:tag:version": "v2",
-        "lancedb:logical-column": "clip"
-      },
-      "replace": false
-    }
-  ]
-}
-```
-
-Rules:
- **Use `"replace": false`** (merge) by default — this preserves existing metadata the user didn't ask to change
- Use `"replace": true` only if the user explicitly asks to overwrite all existing metadata on a column
- Set a value to `null` to delete a specific key
- Batch all updates in a single request when possible
-
-The response includes `version` (new table version) and `fields` (the updated metadata per field).
-
-## Step 5: Confirm
-
-Report back:
- Which columns were updated and what was written
- The new table version number
- Any columns skipped (e.g., already had up-to-date metadata)
-
---
-
-## Quick examples
-
-**"Write descriptions for all columns in the `product_embeddings` table"**
-1. POST `/v1/table/product_embeddings/describe` → get all fields
-2. Generate a `lancedb:description` for each column based on name + type
-3. POST `update_field_metadata` with descriptions
-4. Report
-
-**"Tag the columns in `model_outputs` with their field type and model"**
-1. Describe `model_outputs`
-2. For each field, classify by name + Arrow type → set `lancedb:tag:field_type` and `lancedb:tag:model` where applicable
-3. POST `update_field_metadata`
-4. Report
-
-**"Group the feature columns in `training_features` into logical families and mark the latest version"**
-1. Describe the table
-2. Find version patterns → assign `lancedb:logical-column` and `lancedb:tag:version`; mark newest with `lancedb:tag:latest: "true"`
-3. POST `update_field_metadata`
-4. Show the grouping
@@ -1,42 +0,0 @@
---
-name: lancedb-connect
-description: Resolve how to connect to a LanceDB deployment over the REST API — figure out the base URL, API key, and database header. Use this before making any REST requests to a LanceDB table, whenever the endpoint or auth setup is not already known. Also useful on its own when someone asks how to connect, authenticate, or curl their LanceDB instance.
-metadata:
-  short-description: Resolve the base URL and auth headers for a LanceDB deployment
---
-
-## Goal
-
-Produce two things every REST request needs:
-
-1. **Base URL** — the endpoint
-2. **Headers** — `x-api-key`, and usually `x-lancedb-database`
-
-## Resolution steps
-
-1. If the user already gave a URL and API key (or said which environment they're working against), use that.
-2. Otherwise, look for credentials already available in the environment:
-   - Env vars like `LANCEDB_URI` / `LANCEDB_HOST` / `LANCEDB_API_KEY`
-   - A LanceDB endpoint already running or port-forwarded locally (the REST default port is 2333, i.e. `http://localhost:2333`)
-3. If you didn't find both pieces, ask the user directly: **"What's your LanceDB endpoint's URL, and what's your API key?"** Also ask which database to use if it isn't obvious. Don't guess or probe further — the user knows their deployment.
-
-## Validating the connection
-
-Make a cheap authenticated request and check the status:
-
-```bash
-curl -s -w "\n%{http_code}" "{base_url}/v1/table/?limit=1" \
-  -H "x-api-key: <key>" \
-  -H "x-lancedb-database: <database>"
-```
-
- `200` — connection, key, and database header all good
- `401` — API key missing or wrong
- `400` mentioning a database header — this deployment expects `x-lancedb-database`
-
-## Non-REST equivalents
-
-If the caller would rather use the SDK or CLI than raw REST, the same credentials work:
-
- Python SDK: `lancedb.connect("db://<database>", api_key="<key>", host_override="<base_url>")`
- `lancedb` CLI: a `[profiles.<name>]` entry in `~/.lancedb/config.toml` with `http_server_url`, `api_key`, `database`
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.32.0-beta.1"
+current_version = "0.32.0-beta.2"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
@@ -0,0 +1,19 @@
+{
+  "name": "lancedb",
+  "owner": {
+    "name": "LanceDB"
+  },
+  "description": "LanceDB plugins for Claude Code.",
+  "plugins": [
+    {
+      "name": "lancedb",
+      "source": "./plugins/lancedb",
+      "description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables.",
+      "version": "0.1.0",
+      "author": {
+        "name": "LanceDB"
+      },
+      "category": "development"
+    }
+  ]
+}
@@ -18,6 +18,14 @@ inputs:
    description: "The manylinux version to build for"
    required: false
    default: "2_17"
+  package-name:
+    description: "Override [project] name in python/pyproject.toml (e.g. 'lancedb-compat'). Default keeps 'lancedb'."
+    required: false
+    default: "lancedb"
+  rustflags:
+    description: "RUSTFLAGS for the build container, as a single whitespace-free token (e.g. '-Ctarget-cpu=x86-64-v2'). Empty leaves RUSTFLAGS unset, keeping the defaults from .cargo/config.toml."
+    required: false
+    default: ""
 runs:
  using: "composite"
  steps:
@@ -27,6 +35,18 @@ runs:
        ARM_BUILD: ${{ inputs.arm-build }}
      run: |
        echo "ARM BUILD: $ARM_BUILD"
+    - name: Patch package name for variant build
+      if: ${{ inputs.package-name != 'lancedb' }}
+      shell: bash
+      env:
+        PACKAGE_NAME: ${{ inputs.package-name }}
+      run: |
+        # Swap the [project] name so this build produces e.g. lancedb-compat
+        # wheels. The package still installs files under the lancedb/
+        # namespace -- import lancedb still works after pip install.
+        sed -i.bak 's/^name = "lancedb"$/name = "'"$PACKAGE_NAME"'"/' python/pyproject.toml
+        rm -f python/pyproject.toml.bak
+        grep '^name = ' python/pyproject.toml
    - name: Build x86_64 Manylinux wheel
      if: ${{ inputs.arm-build == 'false' }}
      uses: PyO3/maturin-action@v1
@@ -34,7 +54,7 @@ runs:
        maturin-version: "1.12.4"
        command: build
        working-directory: python
-        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc"
+        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc ${{ inputs.rustflags != '' && format('-e RUSTFLAGS={0}', inputs.rustflags) || '' }}"
        target: x86_64-unknown-linux-gnu
        manylinux: ${{ inputs.manylinux }}
        args: ${{ inputs.args }}
@@ -51,7 +71,7 @@ runs:
        maturin-version: "1.12.4"
        command: build
        working-directory: python
-        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc"
+        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc ${{ inputs.rustflags != '' && format('-e RUSTFLAGS={0}', inputs.rustflags) || '' }}"
        target: aarch64-unknown-linux-gnu
        manylinux: ${{ inputs.manylinux }}
        args: ${{ inputs.args }}
@@ -87,7 +87,7 @@ jobs:
          bash ci/update_lockfiles.sh --amend
      - name: Push new version tag
        if: ${{ !inputs.dry_run }}
-        uses: ad-m/github-push-action@master
+        uses: ad-m/github-push-action@881a6320fdb16eb5318c5054f31c218aec2b324c # v1.3.0
        with:
          # Need to use PAT here too to trigger next workflow. See comment above.
          github_token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
@@ -22,7 +22,7 @@ permissions:

 jobs:
  linux:
-    name: Python ${{ matrix.config.platform }} manylinux${{ matrix.config.manylinux }}
+    name: Python ${{ matrix.config.package_name }} ${{ matrix.config.platform }} manylinux${{ matrix.config.manylinux }}
    timeout-minutes: 60
    strategy:
      matrix:
@@ -31,11 +31,28 @@ jobs:
            manylinux: "2_28"
            extra_args: "--features fp16kernels"
            runner: ubuntu-22.04
+            package_name: "lancedb"
+            rustflags: ""
          # For successful fat LTO builds, we need a large runner to avoid OOM errors.
          - platform: aarch64
            manylinux: "2_28"
            extra_args: "--features fp16kernels"
            runner: ubuntu-2404-8x-arm64
+            package_name: "lancedb"
+            rustflags: ""
+          # `lancedb-compat`: pre-Haswell-friendly variant for x86_64 hosts
+          # without AVX2 (Sandy Bridge / Ivy Bridge / Westmere on Intel,
+          # Bulldozer / Piledriver / Steamroller on AMD). Compiled at the
+          # `x86-64-v2` baseline; runtime SIMD dispatch in lance-linalg
+          # picks the appropriate tier (scalar / AVX / AVX+FMA / AVX2+FMA
+          # / AVX-512) at load time. Same import as `lancedb` -- conflicts
+          # at install time, so users pick one.
+          - platform: x86_64
+            manylinux: "2_28"
+            extra_args: ""
+            runner: ubuntu-22.04
+            package_name: "lancedb-compat"
+            rustflags: "-Ctarget-cpu=x86-64-v2"
    runs-on: ${{ matrix.config.runner }}
    steps:
      - uses: actions/checkout@v6
@@ -52,11 +69,13 @@ jobs:
          args: "--release --strip ${{ matrix.config.extra_args }}"
          arm-build: ${{ matrix.config.platform == 'aarch64' }}
          manylinux: ${{ matrix.config.manylinux }}
+          package-name: ${{ matrix.config.package_name }}
+          rustflags: ${{ matrix.config.rustflags }}
      - uses: actions/upload-artifact@v7
        if: startsWith(github.ref, 'refs/tags/python-v')
        with:
-          name: wheels-linux-${{ matrix.config.platform }}-${{ matrix.config.manylinux }}
-          path: target/wheels/lancedb-*.whl
+          name: wheels-linux-${{ matrix.config.package_name }}-${{ matrix.config.platform }}-${{ matrix.config.manylinux }}
+          path: target/wheels/*.whl
          if-no-files-found: error
  mac:
    timeout-minutes: 90
@@ -145,7 +164,7 @@ jobs:
          FURY_TOKEN: ${{ secrets.FURY_TOKEN }}
        run: |
          shopt -s nullglob
-          WHEELS=(target/wheels/lancedb-*.whl)
+          WHEELS=(target/wheels/*.whl)
          if [[ ${#WHEELS[@]} -eq 0 ]]; then
            echo "No wheels found in target/wheels/" >&2
            exit 1
@@ -98,7 +98,7 @@ jobs:
          cargo build --profile ci --benches --all-features --tests

  linux:
-    timeout-minutes: 30
+    timeout-minutes: 60
    # To build all features, we need more disk space than is available
    # on the free OSS github runner. This is mostly due to the the
    # sentence-transformers feature.
@@ -158,7 +158,7 @@ jobs:
        run: CARGO_ARGS="--profile ci" make -C ./lancedb remote-tests

  macos:
-    timeout-minutes: 30
+    timeout-minutes: 60
    strategy:
      matrix:
        mac-runner: ["macos-14", "macos-15"]
@@ -13,20 +13,20 @@ categories = ["database-implementations"]
 rust-version = "1.91.0"

 [workspace.dependencies]
-lance = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-core = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-datagen = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-file = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-io = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-index = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-linalg = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace-impls = { "version" = "=9.0.0-beta.23", default-features = false, "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-table = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-testing = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-datafusion = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-encoding = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
-lance-arrow = { "version" = "=9.0.0-beta.23", "tag" = "v9.0.0-beta.23", "git" = "https://github.com/lance-format/lance.git" }
+lance = { "version" = "=10.0.0-beta.3", default-features = false, "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-core = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-datagen = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-file = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-io = { "version" = "=10.0.0-beta.3", default-features = false, "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-index = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-linalg = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace-impls = { "version" = "=10.0.0-beta.3", default-features = false, "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-table = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-testing = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-datafusion = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-encoding = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
+lance-arrow = { "version" = "=10.0.0-beta.3", "tag" = "v10.0.0-beta.3", "git" = "https://github.com/lance-format/lance.git" }
 ahash = "0.8"
 # Note that this one does not include pyarrow
 arrow = { version = "58.0.0", optional = false }
@@ -39,15 +39,15 @@ arrow-schema = "58.0.0"
 arrow-select = "58.0.0"
 arrow-cast = "58.0.0"
 async-trait = "0"
-datafusion = { version = "53.0.0", default-features = false }
-datafusion-catalog = "53.0.0"
-datafusion-common = { version = "53.0.0", default-features = false }
-datafusion-execution = "53.0.0"
-datafusion-expr = "53.0.0"
-datafusion-functions = "53.0.0"
-datafusion-physical-plan = "53.0.0"
-datafusion-physical-expr = "53.0.0"
-datafusion-sql = "53.0.0"
+datafusion = { version = "54.0.0", default-features = false }
+datafusion-catalog = "54.0.0"
+datafusion-common = { version = "54.0.0", default-features = false }
+datafusion-execution = "54.0.0"
+datafusion-expr = "54.0.0"
+datafusion-functions = "54.0.0"
+datafusion-physical-plan = "54.0.0"
+datafusion-physical-expr = "54.0.0"
+datafusion-sql = "54.0.0"
 env_logger = "0.11"
 half = { "version" = "2.7.1", default-features = false, features = [
    "num-traits",
@@ -64,7 +64,6 @@ snafu = "0.8"
 url = "2"
 num-traits = "0.2"
 regex = "1.10"
-lazy_static = "1"
 semver = "1.0.25"
 chrono = "0.4"

@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
 <dependency>
    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-core</artifactId>
-    <version>0.32.0-beta.1</version>
+    <version>0.32.0-beta.2</version>
 </dependency>
 ```

@@ -249,6 +249,57 @@ try (BufferAllocator allocator = new RootAllocator();
 }
 ```

+### Creating an Empty Table
+
+To create an empty table, send an Arrow IPC stream that contains the table schema and no record batches.
+The schema in the IPC stream becomes the table schema, and rows can be inserted later.
+
+```java
+import org.lance.namespace.model.CreateTableRequest;
+import org.lance.namespace.model.CreateTableResponse;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.channels.Channels;
+import java.util.Arrays;
+
+Schema schema = new Schema(Arrays.asList(
+    new Field("id", FieldType.nullable(new ArrowType.Int(32, true)), null),
+    new Field("name", FieldType.nullable(new ArrowType.Utf8()), null),
+    new Field("embedding",
+        FieldType.nullable(new ArrowType.FixedSizeList(128)),
+        Arrays.asList(new Field("item",
+            FieldType.nullable(new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE)),
+            null)))
+));
+
+byte[] emptyTableData;
+try (BufferAllocator allocator = new RootAllocator();
+     VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
+    root.setRowCount(0);
+
+    ByteArrayOutputStream out = new ByteArrayOutputStream();
+    try (ArrowStreamWriter writer = new ArrowStreamWriter(root, null, Channels.newChannel(out))) {
+        writer.start();
+        writer.end();
+    }
+    emptyTableData = out.toByteArray();
+}
+
+CreateTableRequest request = new CreateTableRequest();
+request.setId(Arrays.asList("my_namespace", "empty_table"));
+
+CreateTableResponse response = namespaceClient.createTable(request, emptyTableData);
+```
+
 ### Insert

 ```java
@@ -431,9 +482,88 @@ query.setVector(vector);
 byte[] result = namespaceClient.queryTable(query);
 ```

-### Reading Query Results
+## Indexing

-Query results are returned in Apache Arrow IPC file format. Here's how to read them:
+The Java SDK exposes the REST namespace index operations through the same `LanceNamespace` client.
+Index creation runs asynchronously, so use `listTableIndices` or `describeTableIndexStats` to check progress.
+
+### Creating a Vector Index
+
+```java
+import org.lance.namespace.model.CreateTableIndexRequest;
+import org.lance.namespace.model.CreateTableIndexResponse;
+
+CreateTableIndexRequest request = new CreateTableIndexRequest();
+request.setId(Arrays.asList("my_namespace", "my_table"));
+request.setColumn("embedding");
+request.setIndexType("IVF_PQ");
+request.setDistanceType("cosine");
+request.setName("embedding_idx");
+
+CreateTableIndexResponse response = namespaceClient.createTableIndex(request);
+System.out.println("Index transaction: " + response.getTransactionId());
+```
+
+### Creating a Scalar Index
+
+```java
+import org.lance.namespace.model.CreateTableIndexRequest;
+import org.lance.namespace.model.CreateTableScalarIndexResponse;
+
+CreateTableIndexRequest request = new CreateTableIndexRequest();
+request.setId(Arrays.asList("my_namespace", "my_table"));
+request.setColumn("category");
+request.setIndexType("BTREE");
+request.setName("category_idx");
+
+CreateTableScalarIndexResponse response = namespaceClient.createTableScalarIndex(request);
+System.out.println("Index transaction: " + response.getTransactionId());
+```
+
+### Creating a Full Text Search Index
+
+```java
+import org.lance.namespace.model.CreateTableIndexRequest;
+import org.lance.namespace.model.CreateTableScalarIndexResponse;
+
+CreateTableIndexRequest request = new CreateTableIndexRequest();
+request.setId(Arrays.asList("my_namespace", "my_table"));
+request.setColumn("text_column");
+request.setIndexType("FTS");
+request.setName("text_idx");
+request.setBaseTokenizer("simple");
+request.setLowerCase(true);
+request.setWithPosition(true);
+
+CreateTableScalarIndexResponse response = namespaceClient.createTableScalarIndex(request);
+System.out.println("Index transaction: " + response.getTransactionId());
+```
+
+### Listing Indexes
+
+```java
+import org.lance.namespace.model.IndexContent;
+import org.lance.namespace.model.ListTableIndicesRequest;
+import org.lance.namespace.model.ListTableIndicesResponse;
+
+ListTableIndicesRequest request = new ListTableIndicesRequest();
+request.setId(Arrays.asList("my_namespace", "my_table"));
+
+ListTableIndicesResponse response = namespaceClient.listTableIndices(request);
+for (IndexContent index : response.getIndexes()) {
+    System.out.println(index.getIndexName() + ": " + index.getStatus());
+}
+```
+
+!!! note
+    The current Java namespace API exposes index type, index name, distance type, and full text search tokenizer options.
+    IVF training parameters such as `num_partitions` are not exposed by `CreateTableIndexRequest` yet.
+    To make those configurable from Java, the namespace API must add those fields first.
+
+## Reading Query Results
+
+Query results are returned as bytes in Apache Arrow IPC file format. Put the byte-channel
+adapter behind a small helper so query code can work with `ArrowFileReader` directly:

 ```java
 import org.apache.arrow.vector.ipc.ArrowFileReader;
@@ -441,45 +571,50 @@ import org.apache.arrow.vector.VectorSchemaRoot;
 import org.apache.arrow.memory.BufferAllocator;
 import org.apache.arrow.memory.RootAllocator;

+import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.nio.channels.SeekableByteChannel;

-// Helper class to read Arrow data from byte array
-class ByteArraySeekableByteChannel implements SeekableByteChannel {
-    private final byte[] data;
-    private long position = 0;
-    private boolean isOpen = true;
-
-    public ByteArraySeekableByteChannel(byte[] data) {
-        this.data = data;
+final class ArrowIpc {
+    static ArrowFileReader openFileReader(byte[] data, BufferAllocator allocator) throws IOException {
+        return new ArrowFileReader(new ByteArraySeekableByteChannel(data), allocator);
    }

-    @Override
-    public int read(ByteBuffer dst) {
-        int remaining = dst.remaining();
-        int available = (int) (data.length - position);
-        if (available <= 0) return -1;
-        int toRead = Math.min(remaining, available);
-        dst.put(data, (int) position, toRead);
-        position += toRead;
-        return toRead;
-    }
+    private static final class ByteArraySeekableByteChannel implements SeekableByteChannel {
+        private final byte[] data;
+        private long position = 0;
+        private boolean isOpen = true;

-    @Override public long position() { return position; }
-    @Override public SeekableByteChannel position(long newPosition) { position = newPosition; return this; }
-    @Override public long size() { return data.length; }
-    @Override public boolean isOpen() { return isOpen; }
-    @Override public void close() { isOpen = false; }
-    @Override public int write(ByteBuffer src) { throw new UnsupportedOperationException(); }
-    @Override public SeekableByteChannel truncate(long size) { throw new UnsupportedOperationException(); }
+        private ByteArraySeekableByteChannel(byte[] data) {
+            this.data = data;
+        }
+
+        @Override
+        public int read(ByteBuffer dst) {
+            int remaining = dst.remaining();
+            int available = (int) (data.length - position);
+            if (available <= 0) return -1;
+            int toRead = Math.min(remaining, available);
+            dst.put(data, (int) position, toRead);
+            position += toRead;
+            return toRead;
+        }
+
+        @Override public long position() { return position; }
+        @Override public SeekableByteChannel position(long newPosition) { position = newPosition; return this; }
+        @Override public long size() { return data.length; }
+        @Override public boolean isOpen() { return isOpen; }
+        @Override public void close() { isOpen = false; }
+        @Override public int write(ByteBuffer src) { throw new UnsupportedOperationException(); }
+        @Override public SeekableByteChannel truncate(long size) { throw new UnsupportedOperationException(); }
+    }
 }

 // Read query results
 byte[] queryResult = namespaceClient.queryTable(query);

 try (BufferAllocator allocator = new RootAllocator();
-     ArrowFileReader reader = new ArrowFileReader(
-         new ByteArraySeekableByteChannel(queryResult), allocator)) {
+     ArrowFileReader reader = ArrowIpc.openFileReader(queryResult, allocator)) {

    for (int i = 0; i < reader.getRecordBlocks().size(); i++) {
        reader.loadRecordBatch(reader.getRecordBlocks().get(i));
@@ -83,6 +83,24 @@ Delete a branch.

 ***

+### diff()
+
+```ts
+diff(fromBranch): Promise<BranchDiff>
+```
+
+Compare a branch against main without modifying either branch.
+
+#### Parameters
+
+* **fromBranch**: `string`
+
+#### Returns
+
+`Promise`&lt;[`BranchDiff`](../interfaces/BranchDiff.md)&gt;
+
+***
+
 ### list()

 ```ts
@@ -94,3 +112,28 @@ List all branches, mapping name to branch metadata.
 #### Returns

 `Promise`&lt;`Record`&lt;`string`, [`BranchContents`](BranchContents.md)&gt;&gt;
+
+***
+
+### merge()
+
+```ts
+merge(fromBranch, dryRun): Promise<MergeBranchResult>
+```
+
+Merge a branch into main.
+
+Set `dryRun` to `true` to preview the merge. A rejected merge resolves
+with `status: "rejected"` instead of throwing.
+
+#### Parameters
+
+* **fromBranch**: `string`
+    Branch to merge from.
+
+* **dryRun**: `boolean` = `false`
+    When true, only preview the merge. Defaults to false.
+
+#### Returns
+
+`Promise`&lt;[`MergeBranchResult`](../interfaces/MergeBranchResult.md)&gt;
@@ -33,7 +33,7 @@ protected inner: Query | Promise<Query>;
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -41,6 +41,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -38,7 +38,7 @@ protected inner: NativeQueryType | Promise<NativeQueryType>;
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -46,6 +46,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -29,7 +29,7 @@ protected inner: TakeQuery | Promise<TakeQuery>;
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -37,6 +37,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -51,7 +51,7 @@ addQueryVector(vector): VectorQuery
 ### analyzePlan()

 ```ts
-analyzePlan(): Promise<string>
+analyzePlan(distributedMetrics?): Promise<string>
 ```

 Executes the query and returns the physical query plan annotated with runtime metrics.
@@ -59,6 +59,12 @@ Executes the query and returns the physical query plan annotated with runtime me
 This is useful for debugging and performance analysis, as it shows how the query was executed
 and includes metrics such as elapsed time, rows processed, and I/O statistics.

+#### Parameters
+
+* **distributedMetrics?**: [`AnalyzePlanDistributedMetrics`](../type-aliases/AnalyzePlanDistributedMetrics.md)
+    How distributed worker metrics are displayed for remote query plans.
+    Defaults to `"aggregate"`.
+
 #### Returns

 `Promise`&lt;`string`&gt;
@@ -52,6 +52,11 @@
 - [AddDataOptions](interfaces/AddDataOptions.md)
 - [AddResult](interfaces/AddResult.md)
 - [AlterColumnsResult](interfaces/AlterColumnsResult.md)
+- [BranchColumnChange](interfaces/BranchColumnChange.md)
+- [BranchColumnSummary](interfaces/BranchColumnSummary.md)
+- [BranchDiff](interfaces/BranchDiff.md)
+- [BranchIndexSummary](interfaces/BranchIndexSummary.md)
+- [BranchRowCountSummary](interfaces/BranchRowCountSummary.md)
 - [ClientConfig](interfaces/ClientConfig.md)
 - [ColumnAlteration](interfaces/ColumnAlteration.md)
 - [ColumnOrdering](interfaces/ColumnOrdering.md)
@@ -86,6 +91,9 @@
 - [ListNamespacesOptions](interfaces/ListNamespacesOptions.md)
 - [ListNamespacesResponse](interfaces/ListNamespacesResponse.md)
 - [LsmWriteSpec](interfaces/LsmWriteSpec.md)
+- [MergeBlocker](interfaces/MergeBlocker.md)
+- [MergeBranchResult](interfaces/MergeBranchResult.md)
+- [MergePreview](interfaces/MergePreview.md)
 - [MergeResult](interfaces/MergeResult.md)
 - [NativeOAuthConfig](interfaces/NativeOAuthConfig.md)
 - [OAuthConfig](interfaces/OAuthConfig.md)
@@ -118,6 +126,7 @@

 ## Type Aliases

+- [AnalyzePlanDistributedMetrics](type-aliases/AnalyzePlanDistributedMetrics.md)
 - [BaseTokenizer](type-aliases/BaseTokenizer.md)
 - [Data](type-aliases/Data.md)
 - [DataLike](type-aliases/DataLike.md)
@@ -0,0 +1,33 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / BranchColumnChange
+
+# Interface: BranchColumnChange
+
+A column whose definition differs between main and the branch.
+
+## Properties
+
+### branch
+
+```ts
+branch: BranchColumnSummary;
+```
+
+***
+
+### main
+
+```ts
+main: BranchColumnSummary;
+```
+
+***
+
+### name
+
+```ts
+name: string;
+```
@@ -0,0 +1,33 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / BranchColumnSummary
+
+# Interface: BranchColumnSummary
+
+Summary of a column in a branch diff.
+
+## Properties
+
+### dataType
+
+```ts
+dataType: string;
+```
+
+***
+
+### name
+
+```ts
+name: string;
+```
+
+***
+
+### nullable
+
+```ts
+nullable: boolean;
+```
@@ -0,0 +1,129 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / BranchDiff
+
+# Interface: BranchDiff
+
+Read-only comparison of a branch against main.
+
+## Properties
+
+### addedColumns
+
+```ts
+addedColumns: BranchColumnSummary[];
+```
+
+***
+
+### addedIndexes
+
+```ts
+addedIndexes: BranchIndexSummary[];
+```
+
+***
+
+### baseMoved
+
+```ts
+baseMoved: boolean;
+```
+
+***
+
+### branchVersion
+
+```ts
+branchVersion: number;
+```
+
+***
+
+### changedColumns
+
+```ts
+changedColumns: BranchColumnChange[];
+```
+
+***
+
+### fromBranch
+
+```ts
+fromBranch: string;
+```
+
+***
+
+### mainVersion
+
+```ts
+mainVersion: number;
+```
+
+***
+
+### mergeBlockers
+
+```ts
+mergeBlockers: MergeBlocker[];
+```
+
+***
+
+### mergeable
+
+```ts
+mergeable: boolean;
+```
+
+***
+
+### parentVersion
+
+```ts
+parentVersion: number;
+```
+
+***
+
+### removedColumns
+
+```ts
+removedColumns: BranchColumnSummary[];
+```
+
+***
+
+### removedIndexes
+
+```ts
+removedIndexes: BranchIndexSummary[];
+```
+
+***
+
+### rowCountBranch
+
+```ts
+rowCountBranch: number;
+```
+
+***
+
+### rowCountMain
+
+```ts
+rowCountMain: number;
+```
+
+***
+
+### rowSummary
+
+```ts
+rowSummary: BranchRowCountSummary;
+```
@@ -0,0 +1,41 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / BranchIndexSummary
+
+# Interface: BranchIndexSummary
+
+Summary of an index in a branch diff.
+
+## Properties
+
+### columns
+
+```ts
+columns: string[];
+```
+
+***
+
+### indexName
+
+```ts
+indexName: string;
+```
+
+***
+
+### indexType?
+
+```ts
+optional indexType: string;
+```
+
+***
+
+### status
+
+```ts
+status: string;
+```
@@ -0,0 +1,57 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / BranchRowCountSummary
+
+# Interface: BranchRowCountSummary
+
+Row-level comparison between main and the branch.
+
+## Properties
+
+### deltaAvailable
+
+```ts
+deltaAvailable: boolean;
+```
+
+***
+
+### inputsChanged
+
+```ts
+inputsChanged: number;
+```
+
+***
+
+### newOnBase
+
+```ts
+newOnBase: number;
+```
+
+***
+
+### newOnBranch
+
+```ts
+newOnBranch: number;
+```
+
+***
+
+### staleRecompute
+
+```ts
+staleRecompute: number;
+```
+
+***
+
+### unchanged
+
+```ts
+unchanged: number;
+```
@@ -43,6 +43,19 @@ The following tokenizers are available:

 ***

+### blockSize?
+
+```ts
+optional blockSize: 128 | 256;
+```
+
+Number of documents per compressed posting block.
+
+The default is 128. Supported values are 128 and 256. A value of 256 uses
+the experimental FTS V3 format and may introduce breaking changes.
+
+***
+
 ### language?

 ```ts
@@ -0,0 +1,25 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / MergeBlocker
+
+# Interface: MergeBlocker
+
+A reason why a branch cannot currently be merged.
+
+## Properties
+
+### code
+
+```ts
+code: string;
+```
+
+***
+
+### message
+
+```ts
+message: string;
+```
@@ -0,0 +1,46 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / MergeBranchResult
+
+# Interface: MergeBranchResult
+
+Result of previewing or attempting a branch merge.
+
+## Properties
+
+### diff
+
+```ts
+diff: BranchDiff;
+```
+
+***
+
+### mainVersionAfter?
+
+```ts
+optional mainVersionAfter: number;
+```
+
+***
+
+### preview
+
+```ts
+preview: MergePreview;
+```
+
+***
+
+### status
+
+```ts
+status:
+  | "unknown"
+  | "rejected"
+  | "ready"
+  | "notImplemented"
+  | "merged";
+```
@@ -0,0 +1,17 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / MergePreview
+
+# Interface: MergePreview
+
+Changes that would be, or were, promoted by a branch merge.
+
+## Properties
+
+### promotedColumns
+
+```ts
+promotedColumns: string[];
+```
@@ -0,0 +1,11 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / AnalyzePlanDistributedMetrics
+
+# Type Alias: AnalyzePlanDistributedMetrics
+
+```ts
+type AnalyzePlanDistributedMetrics: "aggregate" | "per_worker" | "full";
+```
@@ -8,7 +8,7 @@
    <parent>
      <groupId>com.lancedb</groupId>
      <artifactId>lancedb-parent</artifactId>
-      <version>0.32.0-beta.1</version>
+      <version>0.32.0-beta.2</version>
      <relativePath>../pom.xml</relativePath>
    </parent>

@@ -6,7 +6,7 @@

    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-parent</artifactId>
-    <version>0.32.0-beta.1</version>
+    <version>0.32.0-beta.2</version>
    <packaging>pom</packaging>
    <name>${project.artifactId}</name>
    <description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <arrow.version>15.0.0</arrow.version>
-        <lance-core.version>9.0.0-beta.23</lance-core.version>
+        <lance-core.version>10.0.0-beta.3</lance-core.version>
        <spotless.skip>false</spotless.skip>
        <spotless.version>2.30.0</spotless.version>
        <spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
@@ -1,7 +1,7 @@
 [package]
 name = "lancedb-nodejs"
 edition.workspace = true
-version = "0.32.0-beta.1"
+version = "0.32.0-beta.2"
 publish = false
 license.workspace = true
 description.workspace = true
@@ -52,6 +52,7 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
      Float64,
      Struct,
      List,
+      Map_,
      Int16,
      Int32,
      Int64,
@@ -69,6 +70,30 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
    type Schema = ApacheArrow["Schema"];
    type Table = ApacheArrow["Table"];

+    function expectValidMapField(
+      // biome-ignore lint/suspicious/noExplicitAny: Arrow Field types vary across supported versions
+      field: any,
+    ): void {
+      expect(DataType.isMap(field.type)).toBe(true);
+      expect(field.type.keysSorted).toBe(true);
+      expect(field.type.children).toHaveLength(1);
+
+      const entries = field.type.children[0];
+      expect(entries.name).toBe("entries");
+      expect(entries.nullable).toBe(false);
+      expect(DataType.isStruct(entries.type)).toBe(true);
+      expect(entries.type.children).toHaveLength(2);
+
+      const [key, value] = entries.type.children;
+      expect([key.name, value.name]).toEqual(["key", "value"]);
+      expect(key.nullable).toBe(false);
+      expect(DataType.isUtf8(key.type)).toBe(true);
+      expect(value.nullable).toBe(true);
+      expect(DataType.isInt(value.type)).toBe(true);
+      expect(value.type.bitWidth).toBe(32);
+      expect(value.type.isSigned).toBe(true);
+    }
+
    // Helper method to verify various ways to create a table
    async function checkTableCreation(
      tableCreationMethod: (
@@ -938,6 +963,34 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
          false,
        );
      });
+
+      it("will make an empty table with a Map field", async function () {
+        const schema = new Schema([
+          new Field(
+            "attributes",
+            new Map_(
+              new Field(
+                "entries",
+                new Struct([
+                  new Field("key", new Utf8(), false),
+                  new Field("value", new Int32(), true),
+                ]),
+                false,
+              ),
+              true,
+            ),
+          ),
+        ]);
+
+        const table = makeEmptyTable(schema);
+
+        expectValidMapField(table.schema.fields[0]);
+
+        const buffer = await fromTableToBuffer(table);
+        const roundTripped = tableFromIPC(buffer);
+
+        expectValidMapField(roundTripped.schema.fields[0]);
+      });
    });

    describe("when using two versions of arrow", function () {
@@ -15,6 +15,7 @@ import {
  OAuthHeaderProvider,
  StaticHeaderProvider,
 } from "../lancedb/header";
+import { Index } from "../lancedb/indices";

 // Test-only header providers
 class CustomProvider extends HeaderProvider {
@@ -225,6 +226,161 @@ describe("remote connection", () => {
    );
  });

+  it("sends the FTS posting block size to remote tables", async () => {
+    let createIndexBody: Record<string, unknown> | undefined;
+
+    await withMockDatabase(
+      (req, res) => {
+        const path = req.url ?? "";
+        if (path.endsWith("/describe/")) {
+          res.writeHead(200, { "Content-Type": "application/json" }).end(
+            JSON.stringify({
+              name: "t",
+              version: 1,
+              schema: {
+                fields: [
+                  { name: "text", type: { type: "string" }, nullable: false },
+                ],
+              },
+            }),
+          );
+          return;
+        }
+
+        if (path.endsWith("/create_index/")) {
+          let raw = "";
+          req.on("data", (chunk) => {
+            raw += chunk;
+          });
+          req.on("end", () => {
+            createIndexBody = JSON.parse(raw);
+            res.writeHead(200).end();
+          });
+          return;
+        }
+
+        res.writeHead(404).end();
+      },
+      async (db) => {
+        const table = await db.openTable("t");
+        await table.createIndex("text", {
+          config: Index.fts({ blockSize: 256 }),
+        });
+      },
+    );
+
+    expect(createIndexBody?.["column"]).toBe("text");
+    expect(createIndexBody?.["index_type"]).toBe("FTS");
+    expect(createIndexBody?.["block_size"]).toBe(256);
+  });
+
+  it("diffs and merges remote branches", async () => {
+    const sampleDiff = {
+      fromBranch: "exp",
+      parentVersion: 1,
+      mainVersion: 2,
+      branchVersion: 3,
+      baseMoved: false,
+      rowCountMain: 3,
+      rowCountBranch: 3,
+      rowSummary: {
+        unchanged: 3,
+        newOnBase: 0,
+        newOnBranch: 0,
+        staleRecompute: 0,
+        inputsChanged: 0,
+        deltaAvailable: false,
+      },
+      addedColumns: [{ name: "tag", dataType: "utf8", nullable: true }],
+      removedColumns: [],
+      changedColumns: [],
+      addedIndexes: [],
+      removedIndexes: [],
+      mergeable: true,
+      mergeBlockers: [],
+    };
+    const mergeBodies: Record<string, unknown>[] = [];
+
+    await withMockDatabase(
+      (req, res) => {
+        const path = req.url ?? "";
+        if (path.endsWith("/describe/")) {
+          res.writeHead(200, { "Content-Type": "application/json" }).end(
+            JSON.stringify({
+              name: "t",
+              version: 2,
+              schema: { fields: [] },
+            }),
+          );
+          return;
+        }
+
+        let raw = "";
+        req.on("data", (chunk) => {
+          raw += chunk;
+        });
+        req.on("end", () => {
+          const body = raw ? JSON.parse(raw) : {};
+          if (path.endsWith("/branches/diff/")) {
+            // biome-ignore lint/style/useNamingConvention: snake_case mandated by the server wire format
+            expect(body).toEqual({ from_branch: "exp" });
+            res
+              .writeHead(200, { "Content-Type": "application/json" })
+              .end(JSON.stringify(sampleDiff));
+            return;
+          }
+          if (path.endsWith("/branches/merge/")) {
+            mergeBodies.push(body);
+            const dryRun = body["dry_run"] === true;
+            const response = {
+              status: dryRun ? "ready" : "rejected",
+              diff: dryRun
+                ? sampleDiff
+                : {
+                    ...sampleDiff,
+                    mergeable: false,
+                    mergeBlockers: [
+                      { code: "baseMoved", message: "main has advanced" },
+                    ],
+                  },
+              preview: { promotedColumns: dryRun ? ["tag"] : [] },
+            };
+            res
+              .writeHead(dryRun ? 200 : 409, {
+                "Content-Type": "application/json",
+              })
+              .end(JSON.stringify(response));
+            return;
+          }
+          res.writeHead(404).end();
+        });
+      },
+      async (db) => {
+        const table = await db.openTable("t");
+        const branches = await table.branches();
+
+        await expect(branches.diff("exp")).resolves.toEqual(sampleDiff);
+
+        const rejected = await branches.merge("exp");
+        expect(rejected.status).toBe("rejected");
+        expect(rejected.diff.mergeBlockers).toEqual([
+          { code: "baseMoved", message: "main has advanced" },
+        ]);
+
+        const preview = await branches.merge("exp", true);
+        expect(preview.status).toBe("ready");
+        expect(preview.preview.promotedColumns).toEqual(["tag"]);
+      },
+    );
+
+    expect(mergeBodies).toEqual([
+      // biome-ignore lint/style/useNamingConvention: snake_case mandated by the server wire format
+      { from_branch: "exp", dry_run: false },
+      // biome-ignore lint/style/useNamingConvention: snake_case mandated by the server wire format
+      { from_branch: "exp", dry_run: true },
+    ]);
+  });
+
  describe("TlsConfig", () => {
    it("should create TlsConfig with all fields", () => {
      const tlsConfig: TlsConfig = {
@@ -2,7 +2,7 @@
 // SPDX-FileCopyrightText: Copyright The LanceDB Authors

 import * as arrow from "../lancedb/arrow";
-import { sanitizeField, sanitizeType } from "../lancedb/sanitize";
+import { sanitizeField, sanitizeMap, sanitizeType } from "../lancedb/sanitize";

 describe("sanitize", function () {
  describe("sanitizeType function", function () {
@@ -181,4 +181,15 @@ describe("sanitize", function () {
      );
    });
  });
+
+  describe("sanitizeMap function", function () {
+    it.each([
+      ["no children", []],
+      ["two children", [{}, {}]],
+    ])("should reject a Map type with %s", function (_, children) {
+      expect(() => sanitizeMap({ children, keysSorted: false })).toThrow(
+        "Expected a Map type to have exactly one child",
+      );
+    });
+  });
 });
@@ -2527,6 +2527,35 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
      expect(results3.length).toBe(1);
    });

+    test("full text search with custom posting block size", async () => {
+      const db = await connect(tmpDir.name);
+      const data = [
+        { text: "hello world", vector: [0.1, 0.2, 0.3] },
+        { text: "goodbye world", vector: [0.4, 0.5, 0.6] },
+      ];
+      const table = await db.createTable("test", data);
+      await table.createIndex("text", {
+        config: Index.fts({ blockSize: 256 }),
+      });
+
+      const index = (await table.listIndices()).find(
+        (index) => index.indexType === "FTS",
+      );
+      expect(index?.indexVersion).toBe(3);
+      expect(
+        (index?.indexDetails as Record<string, unknown>)["block_size"],
+      ).toBe(256);
+
+      const results = await table.search("hello").toArray();
+      expect(results[0].text).toBe(data[0].text);
+    });
+
+    test("rejects invalid full text posting block size", () => {
+      expect(() => Index.fts({ blockSize: 129 as 128 | 256 })).toThrow(
+        "128 or 256",
+      );
+    });
+
    test("full text search without lowercase", async () => {
      const db = await connect(tmpDir.name);
      const data = [
@@ -2775,8 +2804,13 @@ describe("when calling analyzePlan", () => {
      .fill(1)
      .map(() => Math.random());
    const plan = await table.query().nearestTo(queryVec).analyzePlan();
-    console.log("Query Plan:\n", plan); // <--- Print the plan
    expect(plan).toMatch("AnalyzeExec");
+
+    const fullPlan = await table
+      .query()
+      .nearestTo(queryVec)
+      .analyzePlan("full");
+    expect(fullPlan).toMatch("AnalyzeExec");
  });
 });

@@ -93,6 +93,7 @@ export {
  QueryBase,
  VectorQuery,
  TakeQuery,
+  AnalyzePlanDistributedMetrics,
  QueryExecutionOptions,
  ColumnOrdering,
  FullTextSearchOptions,
@@ -123,6 +124,14 @@ export {
 export {
  Table,
  Branches,
+  BranchColumnSummary,
+  BranchColumnChange,
+  BranchIndexSummary,
+  BranchRowCountSummary,
+  MergeBlocker,
+  BranchDiff,
+  MergePreview,
+  MergeBranchResult,
  AddDataOptions,
  UpdateOptions,
  OptimizeOptions,
@@ -572,6 +572,14 @@ export interface FtsOptions {
   * whether to only index the prefix of the token for ngram tokenizer
   */
  prefixOnly?: boolean;
+
+  /**
+   * Number of documents per compressed posting block.
+   *
+   * The default is 128. Supported values are 128 and 256. A value of 256 uses
+   * the experimental FTS V3 format and may introduce breaking changes.
+   */
+  blockSize?: 128 | 256;
 }

 export class Index {
@@ -751,6 +759,7 @@ export class Index {
        options?.ngramMinLength,
        options?.ngramMaxLength,
        options?.prefixOnly,
+        options?.blockSize,
      ),
    );
  }
@@ -79,6 +79,8 @@ export interface QueryExecutionOptions {
  timeoutMs?: number;
 }

+export type AnalyzePlanDistributedMetrics = "aggregate" | "per_worker" | "full";
+
 export interface ColumnOrdering {
  columnName: string;
  ascending?: boolean;
@@ -311,13 +313,20 @@ export class QueryBase<
   *        KNNVectorDistance: metric=l2, metrics=[output_rows=1, elapsed_compute=114.333µs, output_batches=1]
   *         LanceScan: uri=/path/to/data, projection=[vector], row_id=true, row_addr=false, ordered=false, metrics=[output_rows=1, elapsed_compute=103.626µs, bytes_read=549, iops=2, requests=2]
   *
+   * @param distributedMetrics - How distributed worker metrics are displayed for remote query plans.
+   * Defaults to `"aggregate"`.
   * @returns A query execution plan with runtime metrics for each step.
   */
-  async analyzePlan(): Promise<string> {
+  async analyzePlan(
+    distributedMetrics?: AnalyzePlanDistributedMetrics,
+  ): Promise<string> {
+    const distributedMetricsMode = distributedMetrics ?? "aggregate";
    if (this.inner instanceof Promise) {
-      return this.inner.then((inner) => inner.analyzePlan());
+      return this.inner.then((inner) =>
+        inner.analyzePlan(distributedMetricsMode),
+      );
    } else {
-      return this.inner.analyzePlan();
+      return this.inner.analyzePlan(distributedMetricsMode);
    }
  }

@@ -288,12 +288,11 @@ export function sanitizeMap(typeLike: object) {
  if (!("keysSorted" in typeLike) || typeof typeLike.keysSorted !== "boolean") {
    throw Error("Expected a Map type to have a `keysSorted` property");
  }
+  if (typeLike.children.length !== 1) {
+    throw Error("Expected a Map type to have exactly one child");
+  }

-  return new Map_(
-    // biome-ignore lint/suspicious/noExplicitAny: skip
-    typeLike.children.map((field) => sanitizeField(field)) as any,
-    typeLike.keysSorted,
-  );
+  return new Map_(sanitizeField(typeLike.children[0]), typeLike.keysSorted);
 }

 export function sanitizeDuration(typeLike: object) {
@@ -1329,6 +1329,76 @@ export interface FieldMetadataUpdate {
  replace?: boolean;
 }

+/** Summary of a column in a branch diff. */
+export interface BranchColumnSummary {
+  name: string;
+  dataType: string;
+  nullable: boolean;
+}
+
+/** A column whose definition differs between main and the branch. */
+export interface BranchColumnChange {
+  name: string;
+  main: BranchColumnSummary;
+  branch: BranchColumnSummary;
+}
+
+/** Summary of an index in a branch diff. */
+export interface BranchIndexSummary {
+  indexName: string;
+  columns: string[];
+  indexType?: string;
+  status: string;
+}
+
+/** Row-level comparison between main and the branch. */
+export interface BranchRowCountSummary {
+  unchanged: number;
+  newOnBase: number;
+  newOnBranch: number;
+  staleRecompute: number;
+  inputsChanged: number;
+  deltaAvailable: boolean;
+}
+
+/** A reason why a branch cannot currently be merged. */
+export interface MergeBlocker {
+  code: string;
+  message: string;
+}
+
+/** Read-only comparison of a branch against main. */
+export interface BranchDiff {
+  fromBranch: string;
+  parentVersion: number;
+  mainVersion: number;
+  branchVersion: number;
+  baseMoved: boolean;
+  rowCountMain: number;
+  rowCountBranch: number;
+  rowSummary: BranchRowCountSummary;
+  addedColumns: BranchColumnSummary[];
+  removedColumns: BranchColumnSummary[];
+  changedColumns: BranchColumnChange[];
+  addedIndexes: BranchIndexSummary[];
+  removedIndexes: BranchIndexSummary[];
+  mergeable: boolean;
+  mergeBlockers: MergeBlocker[];
+}
+
+/** Changes that would be, or were, promoted by a branch merge. */
+export interface MergePreview {
+  promotedColumns: string[];
+}
+
+/** Result of previewing or attempting a branch merge. */
+export interface MergeBranchResult {
+  status: "ready" | "rejected" | "notImplemented" | "merged" | "unknown";
+  diff: BranchDiff;
+  preview: MergePreview;
+  mainVersionAfter?: number;
+}
+
 /**
 * Branch manager for a {@link Table}.
 *
@@ -1381,4 +1451,28 @@ export class Branches {
  async delete(name: string): Promise<void> {
    return await this.#inner.delete(name);
  }
+
+  /** Compare a branch against main without modifying either branch. */
+  async diff(fromBranch: string): Promise<BranchDiff> {
+    return (await this.#inner.diff(fromBranch)) as unknown as BranchDiff;
+  }
+
+  /**
+   * Merge a branch into main.
+   *
+   * Set `dryRun` to `true` to preview the merge. A rejected merge resolves
+   * with `status: "rejected"` instead of throwing.
+   *
+   * @param fromBranch Branch to merge from.
+   * @param dryRun When true, only preview the merge. Defaults to false.
+   */
+  async merge(
+    fromBranch: string,
+    dryRun: boolean = false,
+  ): Promise<MergeBranchResult> {
+    return (await this.#inner.merge(
+      fromBranch,
+      dryRun,
+    )) as unknown as MergeBranchResult;
+  }
 }
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-musl",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-musl.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-musl",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-musl.node",
@@ -1,6 +1,6 @@
 {
  "name": "@lancedb/lancedb-win32-arm64-msvc",
-  "version": "0.32.0-beta.1",
+  "version": "0.32.0-beta.2",
  "os": [
    "win32"
  ],
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.32.0-beta.1",
+	"version": "0.32.0-beta.2",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",
@@ -1,12 +1,12 @@
 {
  "name": "@lancedb/lancedb",
-  "version": "0.32.0-beta.1",
+  "version": "0.32.0-beta.2",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@lancedb/lancedb",
-      "version": "0.32.0-beta.1",
+      "version": "0.32.0-beta.2",
      "cpu": [
        "x64",
        "arm64"
@@ -11,7 +11,7 @@
    "ann"
  ],
  "private": false,
-  "version": "0.32.0-beta.1",
+  "version": "0.32.0-beta.2",
  "main": "dist/index.js",
  "exports": {
    ".": "./dist/index.js",
@@ -226,7 +226,8 @@ impl Index {
        ngram_min_length: Option<u32>,
        ngram_max_length: Option<u32>,
        prefix_only: Option<bool>,
-    ) -> Self {
+        block_size: Option<u32>,
+    ) -> napi::Result<Self> {
        let mut opts = FtsIndexBuilder::default();
        if let Some(with_position) = with_position {
            opts = opts.with_position(with_position);
@@ -261,10 +262,15 @@ impl Index {
        if let Some(prefix_only) = prefix_only {
            opts = opts.ngram_prefix_only(prefix_only);
        }
-
-        Self {
-            inner: Mutex::new(Some(LanceDbIndex::FTS(opts))),
+        if let Some(block_size) = block_size {
+            opts = opts
+                .block_size(block_size as usize)
+                .map_err(|err| napi::Error::from_reason(err.to_string()))?;
        }
+
+        Ok(Self {
+            inner: Mutex::new(Some(LanceDbIndex::FTS(opts))),
+        })
    }

    #[napi(factory)]
@@ -19,6 +19,7 @@ use lancedb::index::scalar::{
    BooleanQuery, BoostQuery, FtsQuery, FullTextSearchQuery, MatchQuery, MultiMatchQuery, Occur,
    Operator, PhraseQuery,
 };
+use lancedb::query::AnalyzePlanDistributedMetrics;
 use lancedb::query::ExecutableQuery;
 use lancedb::query::Query as LanceDbQuery;
 use lancedb::query::QueryBase;
@@ -47,6 +48,28 @@ impl From<ColumnOrdering> for LanceDbColumnOrdering {
    }
 }

+fn analyze_plan_options(
+    distributed_metrics: Option<String>,
+) -> napi::Result<QueryExecutionOptions> {
+    let analyze_plan_distributed_metrics =
+        match distributed_metrics.as_deref().unwrap_or("aggregate") {
+            "aggregate" => AnalyzePlanDistributedMetrics::Aggregate,
+            "per_worker" => AnalyzePlanDistributedMetrics::PerWorker,
+            "full" => AnalyzePlanDistributedMetrics::Full,
+            mode => {
+                return Err(napi::Error::from_reason(format!(
+                    "Invalid distributedMetrics value '{}'. Expected one of: \
+                 'aggregate', 'per_worker', 'full'",
+                    mode
+                )));
+            }
+        };
+
+    let mut options = QueryExecutionOptions::default();
+    options.analyze_plan_distributed_metrics = analyze_plan_distributed_metrics;
+    Ok(options)
+}
+
 fn bytes_to_arrow_array(data: Uint8Array, dtype: String) -> napi::Result<Arc<dyn Array>> {
    let buf = arrow_buffer::Buffer::from(data.to_vec());
    let num_bytes = buf.len();
@@ -200,13 +223,17 @@ impl Query {
    }

    #[napi(catch_unwind)]
-    pub async fn analyze_plan(&self) -> napi::Result<String> {
-        self.inner.analyze_plan().await.map_err(|e| {
-            napi::Error::from_reason(format!(
-                "Failed to execute analyze plan: {}",
-                convert_error(&e)
-            ))
-        })
+    pub async fn analyze_plan(&self, distributed_metrics: Option<String>) -> napi::Result<String> {
+        let options = analyze_plan_options(distributed_metrics)?;
+        self.inner
+            .analyze_plan_with_options(options)
+            .await
+            .map_err(|e| {
+                napi::Error::from_reason(format!(
+                    "Failed to execute analyze plan: {}",
+                    convert_error(&e)
+                ))
+            })
    }
 }

@@ -412,13 +439,17 @@ impl VectorQuery {
    }

    #[napi(catch_unwind)]
-    pub async fn analyze_plan(&self) -> napi::Result<String> {
-        self.inner.analyze_plan().await.map_err(|e| {
-            napi::Error::from_reason(format!(
-                "Failed to execute analyze plan: {}",
-                convert_error(&e)
-            ))
-        })
+    pub async fn analyze_plan(&self, distributed_metrics: Option<String>) -> napi::Result<String> {
+        let options = analyze_plan_options(distributed_metrics)?;
+        self.inner
+            .analyze_plan_with_options(options)
+            .await
+            .map_err(|e| {
+                napi::Error::from_reason(format!(
+                    "Failed to execute analyze plan: {}",
+                    convert_error(&e)
+                ))
+            })
    }
 }

@@ -491,13 +522,17 @@ impl TakeQuery {
    }

    #[napi(catch_unwind)]
-    pub async fn analyze_plan(&self) -> napi::Result<String> {
-        self.inner.analyze_plan().await.map_err(|e| {
-            napi::Error::from_reason(format!(
-                "Failed to execute analyze plan: {}",
-                convert_error(&e)
-            ))
-        })
+    pub async fn analyze_plan(&self, distributed_metrics: Option<String>) -> napi::Result<String> {
+        let options = analyze_plan_options(distributed_metrics)?;
+        self.inner
+            .analyze_plan_with_options(options)
+            .await
+            .map_err(|e| {
+                napi::Error::from_reason(format!(
+                    "Failed to execute analyze plan: {}",
+                    convert_error(&e)
+                ))
+            })
    }
 }

@@ -232,6 +232,10 @@ impl From<ClientConfig> for lancedb::remote::ClientConfig {
            tls_config: config.tls_config.map(Into::into),
            header_provider: None, // the header provider is set separately later
            user_id: config.user_id,
+            // Resolved from LANCE_CLIENT_MAX_BYTES_PER_REQUEST or the default.
+            max_bytes_per_request: None,
+            // Resolved from LANCE_CLIENT_MAX_REQUEST_DURATION or the read timeout.
+            max_request_duration: None,
        }
    }
 }
@@ -1355,4 +1355,28 @@ impl Branches {
    pub async fn delete(&self, name: String) -> napi::Result<()> {
        self.inner.delete_branch(&name).await.default_error()
    }
+
+    #[napi(ts_return_type = "Promise<Record<string, unknown>>")]
+    pub async fn diff(&self, from_branch: String) -> napi::Result<serde_json::Value> {
+        let diff = self.inner.diff_branch(&from_branch).await.default_error()?;
+        serde_json::to_value(diff).map_err(|err| {
+            napi::Error::from_reason(format!("failed to serialize branch diff: {err}"))
+        })
+    }
+
+    #[napi(ts_return_type = "Promise<Record<string, unknown>>")]
+    pub async fn merge(
+        &self,
+        from_branch: String,
+        dry_run: Option<bool>,
+    ) -> napi::Result<serde_json::Value> {
+        let result = self
+            .inner
+            .merge_branch(&from_branch, dry_run.unwrap_or(false))
+            .await
+            .default_error()?;
+        serde_json::to_value(result).map_err(|err| {
+            napi::Error::from_reason(format!("failed to serialize branch merge result: {err}"))
+        })
+    }
 }
@@ -0,0 +1,21 @@
+{
+  "name": "lancedb",
+  "description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables, with idiomatic query/search patterns and performance defaults for ingestion, indexing, filtering, and diagnostics.",
+  "version": "0.1.0",
+  "author": {
+    "name": "LanceDB"
+  },
+  "homepage": "https://www.lancedb.com",
+  "keywords": [
+    "lancedb",
+    "vector-search",
+    "full-text-search",
+    "hybrid-search",
+    "python",
+    "typescript",
+    "pipelines",
+    "ingestion",
+    "indexing",
+    "performance"
+  ]
+}
@@ -0,0 +1,33 @@
+{
+  "name": "lancedb",
+  "version": "0.1.0",
+  "description": "Codex plugin for building LanceDB pipelines in Python and TypeScript.",
+  "author": {
+    "name": "LanceDB"
+  },
+  "keywords": [
+    "lancedb",
+    "vector-search",
+    "full-text-search",
+    "hybrid-search",
+    "python",
+    "typescript",
+    "pipelines"
+  ],
+  "skills": "./skills/",
+  "interface": {
+    "displayName": "LanceDB",
+    "shortDescription": "Build LanceDB pipelines in Python and TypeScript.",
+    "longDescription": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables, with idiomatic query/search patterns and performance defaults for ingestion, indexing, filtering, and diagnostics.",
+    "developerName": "LanceDB",
+    "websiteURL": "https://www.lancedb.com",
+    "category": "Developer Tools",
+    "capabilities": [
+      "Developer Tools"
+    ],
+    "defaultPrompt": "Create a LanceDB table, embed sample text, and run a vector search.",
+    "composerIcon": "./assets/logo.png",
+    "logo": "./assets/logo.png",
+    "logoDark": "./assets/logo-dark.png"
+  }
+}
@@ -1,6 +1,6 @@
 ---
 name: lancedb
-description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, and apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics.
+description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics, and resolve connections to the remote server for Enterprise-only operations such as jobs.
 ---

 # Building LanceDB Pipelines
@@ -19,7 +19,7 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
 ## Workflow

 1. Identify the SDK: Python, TypeScript, or both.
-2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path.
+2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path. If the task involves jobs in any way (listing, inspecting, creating, or canceling jobs), it is always the remote path and requires a remote server connection — see "Connecting to the LanceDB remote server" below before doing anything else.
 3. Read the matching language branch before writing or changing code:
   - Python patterns: `references/python/patterns.md`
   - Python API quick reference: `references/python/api_reference.md`
@@ -27,7 +27,11 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
   - TypeScript patterns: `references/typescript/patterns.md`
   - TypeScript API quick reference: `references/typescript/api_reference.md`
   - TypeScript performance guidance: `references/typescript/performance.md`
-4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets.
+   - Column metadata authoring (both SDKs): `references/column_metadata.md`
+   - Branch operations (both SDKs): `references/branch_ops.md`
+   - Remote server connection resolution (jobs, raw REST): `references/remote_connect.md`
+   - Job operations REST API (list/describe/cancel/query_events): `references/remote_jobs.md`
+4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read `remote_connect.md` when the task involves jobs or direct REST access to an Enterprise deployment, and `remote_jobs.md` for the job REST methods themselves (list, describe, cancel, query_events).
 5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
 6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.
 7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
@@ -68,6 +72,10 @@ Rules for portable Enterprise ingestion:

 This is Enterprise/Cloud-specific. Local/OSS tables have no separate data plane, so `mode="overwrite"` and immediate same-name reuse are fine there.

+## Connecting to the LanceDB remote server
+
+LanceDB Enterprise/Cloud deployments are served by a server implementing the lance-namespace OpenAPI spec (<https://github.com/lance-format/lance-namespace/blob/main/docs/src/spec.yaml>). Every remote (`db://...`) connection talks to such a server, and some operations exist only there. In particular, **all operations around jobs (listing, inspecting, creating, or canceling jobs) run server-side** — there is no local/OSS equivalent. Before any job work, or any direct REST call to an Enterprise deployment, read `references/remote_connect.md` to resolve the base URL, credentials, and database header and to validate the connection. Then use the four job REST methods documented in `references/remote_jobs.md` (list, describe, cancel, query_events).
+
 ## Script

 Run the scanner when reviewing or modifying an existing codebase:
@@ -0,0 +1,6 @@
+interface:
+  display_name: "LanceDB"
+  short_description: "Build LanceDB pipelines in Python and TypeScript"
+  default_prompt: "Use $lancedb to create a table, embed sample text, and run a vector search."
+  icon_small: "./assets/icon.png"
+  icon_large: "./assets/icon.png"
@@ -0,0 +1,182 @@
+# Branch Operations
+
+Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main. Use for branch lifecycle tasks, experimental/isolated table versions, targeting an operation at a non-main branch, or confirming a mutation did not affect main.
+
+Works on local/OSS and remote Enterprise/Cloud tables, except merging a branch into main, which is Enterprise-only.
+
+## The branch model (important)
+
+Branches are isolated, writable lines of history forked from another branch (or a specific version). Writes on a branch never affect `main`.
+
+There is **no global "switch branch" state** — you never repoint the whole table at a branch. Instead, **operations are scoped by which table handle you use**:
+
+- The handle you got from `open_table(name)` / `openTable(name)` targets `main`.
+- `branches.create(...)` and `branches.checkout(...)` return a **new table handle scoped to that branch**. Every read/write on that handle (add, update, `update_field_metadata`, `create_index`, search, …) lands on the branch.
+- The original main handle is unaffected — keep it around to verify isolation.
+
+`branches.list()` returns only non-main branches. Main always exists and is not listed.
+
+## Python
+
+`table.branches` is a property returning the branch manager; `table.current_branch()` tells you what a handle is scoped to (`None` = main).
+
+```python
+table = db.open_table("products")            # scoped to main
+
+# list — dict of name -> metadata (parent_branch, parent_version, ...); {} = only main
+table.branches.list()
+
+# create: forks from main by default and returns a handle scoped to the new branch
+exp = table.branches.create("experiment-reindex")
+exp = table.branches.create("exp2", from_ref="main", from_version=None)  # optional fork point
+
+# checkout an existing branch -> branch-scoped handle
+wip = table.branches.checkout("wip-branch")
+# with version= it pins to that version (read-only detached view); omit to track latest, writable
+
+# operate on the branch simply by using its handle
+wip.update_field_metadata(
+    {"path": "category", "metadata": {"lancedb:description": "Product category label."}}
+)
+wip.create_scalar_index("category")
+
+# delete: removes only the branch pointer; main and row data remain intact
+table.branches.delete("stale-2024")
+
+# alternatively, open a branch handle directly from the connection
+wip = db.open_table("products", branch="wip-branch")
+
+exp.current_branch()    # "experiment-reindex"
+table.current_branch()  # None (main)
+```
+
+Async: same shape — `table.branches` returns `AsyncBranches`; `await table.branches.create(...)` etc.
+
+## TypeScript
+
+`table.branches()` is an **async method** returning the `Branches` manager; `table.currentBranch()` returns the scoped branch or `null` for main.
+
+```typescript
+const table = await db.openTable("products"); // scoped to main
+const branches = await table.branches();
+
+// list — Record<string, BranchContents>; {} = only main
+await branches.list();
+
+// create: forks from main by default, returns a Table scoped to the new branch
+const exp = await branches.create("experiment-reindex");
+const exp2 = await branches.create("exp2", "main" /* fromRef */, undefined /* fromVersion */);
+
+// checkout an existing branch -> branch-scoped Table
+const wip = await branches.checkout("wip-branch");
+// with a version arg it pins (read-only detached view); omit to track latest, writable
+
+// operate on the branch simply by using its handle
+await wip.updateFieldMetadata([
+  { path: "category", metadata: { "lancedb:description": "Product category label." } },
+]);
+await wip.createIndex("category");
+
+// delete: removes only the branch pointer; main and row data remain intact
+await branches.delete("stale-2024");
+
+// alternatively, open a branch handle directly from the connection
+const wip2 = await db.openTable("products", { branch: "wip-branch" });
+
+exp.currentBranch();   // "experiment-reindex"
+table.currentBranch(); // null (main)
+```
+
+## Verifying isolation
+
+After writing to a branch, confirm the change did NOT land on main by reading through both handles:
+
+```python
+wip = table.branches.checkout("wip-branch")
+wip.update_field_metadata({"path": "category", "metadata": {"lancedb:description": "..."}})
+
+assert b"lancedb:description" in (wip.schema.field("category").metadata or {})
+assert b"lancedb:description" not in (table.schema.field("category").metadata or {})  # main untouched
+```
+
+Two handles on the same branch see each other's writes (e.g. `table.branches.create("exp")` and `db.open_table(name, branch="exp")`); main stays isolated.
+
+## Merging a branch into main (Enterprise only)
+
+Merge is available through the SDKs (`table.branches.merge(...)`) on **Enterprise tables only** — it is not supported on Cloud or local/OSS tables, which raise `NotSupported`.
+
+`merge` takes the branch to merge **from** and a `dry_run` flag. Both the SDK method and the underlying REST endpoint **actually merge by default** (`dry_run=False`); pass `dry_run=True` to only preview. A rejected merge is **not an exception** — it returns a result with `status="rejected"` rather than raising, so inspect the return value. Use `branches.diff(from_branch)` to inspect a branch's pending diff without attempting a merge.
+
+```python
+exp = "experiment-reindex"
+
+# preview only — returns status="ready" if it would merge cleanly
+preview = table.branches.merge(exp, dry_run=True)
+
+# actually merge (default)
+result = table.branches.merge(exp)
+if result["status"] == "merged":
+    print("landed at", result["mainVersionAfter"])
+elif result["status"] == "rejected":
+    print(result["diff"]["mergeBlockers"])   # why it was refused
+
+# inspect a branch's pending diff without merging
+diff = table.branches.diff(exp)
+```
+
+Async: `await table.branches.merge(exp)`, `await table.branches.diff(exp)`.
+
+```typescript
+const branches = await table.branches();
+const exp = "experiment-reindex";
+
+// preview only (second arg is dryRun)
+const preview = await branches.merge(exp, true);
+
+// actually merge (default)
+const result = await branches.merge(exp);
+if (result.status === "merged") {
+  console.log("landed at", result.mainVersionAfter);
+} else if (result.status === "rejected") {
+  console.log(result.diff.mergeBlockers);
+}
+
+const diff = await branches.diff(exp);
+```
+
+The result is the wire JSON, containing `status` (`ready` on a passing dry run, `merged` on success, `rejected` when refused — also `notImplemented`/`unknown`), the branch `diff` (including `mergeBlockers` explaining any rejection), a `preview` of the columns that would be promoted, and — after a real merge — `mainVersionAfter`.
+
+### Merge preconditions
+
+Merge only **promotes newly added columns** onto main; it does not replay arbitrary commits. Practically, a branch is mergeable only if it has **exactly one commit since it was created, and that commit added a column**. The merge is rejected (`status: "rejected"`, with `mergeBlockers` set) if:
+
+- the branch was forked from another branch rather than directly from main
+- main has advanced since the branch was forked
+- the branch's rows changed since the fork (row counts must match main exactly)
+- the branch removed columns or changed a column's type/nullability
+- the branch added no columns (index-only changes are not merged)
+
+### Adding a column in a single commit
+
+Because the branch must contain just one column-adding commit, add the column with its values in one operation rather than add-then-backfill:
+
+1. **SQL transformation** — `add_columns` with a SQL expression computed from existing columns, so the column lands populated in one commit.
+2. **Precompute the values** — compute the column's values externally, then add the fully-populated column in a single operation (e.g. via `merge_insert`/`add_columns` with the data ready).
+3. **Lance-format-level data evolution (pylance)** — use Lance's data evolution with backfill, documented at <https://lance.org/guide/data_evolution/#with-data-backfill>.
+
+## Quick reference
+
+| Goal | Python | TypeScript |
+|------|--------|------------|
+| List branches (non-main) | `table.branches.list()` | `await (await table.branches()).list()` |
+| Create branch (off main) | `table.branches.create(name)` → branch handle | `await branches.create(name)` → branch `Table` |
+| Create from a fork point | `table.branches.create(name, from_ref=..., from_version=...)` | `await branches.create(name, fromRef, fromVersion)` |
+| Get a branch handle | `table.branches.checkout(name)` or `db.open_table(t, branch=name)` | `await branches.checkout(name)` or `await db.openTable(t, { branch: name })` |
+| Pin to a branch version (read-only) | `table.branches.checkout(name, version=v)` | `await branches.checkout(name, v)` |
+| Delete branch | `table.branches.delete(name)` | `await branches.delete(name)` |
+| Which branch is this handle on? | `table.current_branch()` (`None` = main) | `table.currentBranch()` (`null` = main) |
+| Target main | use the original (non-branch) handle | use the original (non-branch) handle |
+| Merge branch into main (Enterprise only) | `table.branches.merge(from_branch, dry_run=False)` | `await branches.merge(fromBranch, dryRun)` |
+| Preview a branch's pending diff (Enterprise only) | `table.branches.diff(from_branch)` | `await branches.diff(fromBranch)` |
+
+Branch names must be non-empty; empty names raise a validation error.
@@ -0,0 +1,183 @@
+# Column Metadata Authoring
+
+Write column-level descriptions, tags, and logical groupings onto a LanceDB table's schema. Use this when the user wants to document, annotate, tag, or classify what their table columns ARE (embeddings vs labels vs eval metrics, model provenance, version families, etc.).
+
+Works on local/OSS and remote Enterprise/Cloud tables alike — read the schema through the table handle, write through `update_field_metadata` (Python) / `updateFieldMetadata` (TypeScript).
+
+## Metadata key conventions
+
+All metadata uses namespaced keys:
+
+| Key | Purpose | Example value |
+|-----|---------|---------------|
+| `lancedb:description` | Human-readable explanation of what the column contains | `"CLIP ViT-L/14 image embedding, L2-normalized (768-dim)"` |
+| `lancedb:tag:<name>` | Flexible key-value tag; the suffix names the tag category | `lancedb:tag:field_type: "embedding"`, `lancedb:tag:model: "clip"`, `lancedb:tag:project_id: "foo"` |
+| `lancedb:logical-column` | Logical group/family this column belongs to | `"clip_features"` |
+
+Tags are open-ended — use whatever key suffix and value make sense given the user's intent. The tag suffix should describe *what is being classified* (e.g., `field_type`, `model`, `project_id`) and the value describes *how*. Multiple tags on the same column are fine — each is a separate key. All values are strings.
+
+## Step 1: Read the schema and existing metadata
+
+Read existing metadata before writing, to avoid redundant updates.
+
+Python — `table.schema` (sync property; async: `await table.schema()`) returns a `pyarrow.Schema`. **Arrow field metadata is bytes-keyed in Python**:
+
+```python
+schema = table.schema
+for field in schema:
+    meta = field.metadata or {}  # dict[bytes, bytes], e.g. {b"lancedb:description": b"..."}
+    print(field.name, field.type, field.nullable, meta)
+```
+
+TypeScript — `await table.schema()` returns an Arrow `Schema`; field metadata is a `Map<string, string>`:
+
+```typescript
+const schema = await table.schema();
+for (const field of schema.fields) {
+  console.log(field.name, field.type, field.nullable, field.metadata); // Map
+  // field.metadata.get("lancedb:description")
+}
+```
+
+For struct/nested fields, recurse into the field's children and address them as dot-paths (e.g., `parent.child`).
+
+If the user hasn't specified which columns to update, work with all columns.
+
+## Step 2: Generate metadata
+
+Decide what to generate based on the user's request.
+
+### Descriptions (`lancedb:description`)
+
+Base descriptions on:
+- The column name and Arrow type (e.g., `FixedSizeList` of floats → likely an embedding)
+- User-supplied context (upstream pipeline, sample values, domain knowledge)
+- Name patterns: `_embedding`/`_vec`/`_embed` → vector; `_label`/`_class` → label; `_score`/`_eval`/`_metric` → evaluation metric
+
+Be specific and concise. Good: `"Sentence-BERT embedding of the query text (768-dim)."` Not: `"An embedding column."`
+
+### Tags (`lancedb:tag:<name>`)
+
+Choose tag key names that match what the user asked to annotate. Common patterns:
+
+- Semantic field type → `lancedb:tag:field_type: "embedding"` / `"text"` / `"image"` / `"label"` / `"eval"` / `"id"` / `"metadata"`
+- Model or source → `lancedb:tag:model: "clip"` / `"bert"` / `"vit"`
+- Project affiliation → `lancedb:tag:project_id: "<name>"`
+- Version → `lancedb:tag:version: "v3"` (and `lancedb:tag:latest: "true"` for the newest)
+
+Use Arrow type as a hint: `FixedSizeList` + float → embedding; `Utf8`/`LargeUtf8` → text; `Binary` → image or blob.
+
+### Logical groupings (`lancedb:logical-column`)
+
+Look for naming patterns across columns:
+- `clip_v1`, `clip_v2`, `clip_v3` → logical column `"clip"`, latest is `v3`
+- `text_embed_20240101`, `text_embed_20240601` → logical column `"text_embed"`, latest is the most recent date suffix
+
+Write `lancedb:logical-column` on all members of a group. Mark the newest with `lancedb:tag:latest: "true"` (in addition to its version tag).
+
+## Step 3: Write the metadata
+
+Each update names a field by dot-path and carries a metadata map. Semantics (identical in both SDKs):
+
+- **Merge by default** (`replace` omitted/false) — preserves existing metadata the user didn't ask to change
+- `replace: true` swaps the field's entire metadata map — only if the user explicitly asks to overwrite
+- A value of `None`/`null` deletes that specific key
+- Batch all field updates into a single call when possible
+- Returns the new table version
+
+Python (sync and async take one dict per field, as varargs):
+
+```python
+res = table.update_field_metadata(
+    {
+        "path": "clip_v3",
+        "metadata": {
+            "lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
+            "lancedb:tag:field_type": "embedding",
+            "lancedb:tag:model": "clip",
+            "lancedb:tag:version": "v3",
+            "lancedb:tag:latest": "true",
+            "lancedb:logical-column": "clip",
+        },
+    },
+    {
+        "path": "clip_v2",
+        "metadata": {
+            "lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
+            "lancedb:tag:field_type": "embedding",
+            "lancedb:tag:model": "clip",
+            "lancedb:tag:version": "v2",
+            "lancedb:logical-column": "clip",
+        },
+    },
+)
+print(res.version)  # new table version
+
+# merge semantics: add a key, delete one via None, keep the rest
+table.update_field_metadata(
+    {"path": "clip_v2", "metadata": {"lancedb:tag:archived": "true", "lancedb:tag:latest": None}}
+)
+```
+
+(`replace_field_metadata` is deprecated — use `update_field_metadata`.)
+
+TypeScript (takes an array of `FieldMetadataUpdate`):
+
+```typescript
+const res = await table.updateFieldMetadata([
+  {
+    path: "clip_v3",
+    metadata: {
+      "lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
+      "lancedb:tag:field_type": "embedding",
+      "lancedb:tag:model": "clip",
+      "lancedb:tag:version": "v3",
+      "lancedb:tag:latest": "true",
+      "lancedb:logical-column": "clip",
+    },
+  },
+  {
+    path: "clip_v2",
+    metadata: {
+      "lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
+      "lancedb:tag:field_type": "embedding",
+      "lancedb:tag:model": "clip",
+      "lancedb:tag:version": "v2",
+      "lancedb:logical-column": "clip",
+    },
+  },
+]);
+console.log(res.version); // new table version
+
+// merge semantics: add a key, delete one via null, keep the rest
+await table.updateFieldMetadata([
+  { path: "clip_v2", metadata: { "lancedb:tag:archived": "true", "lancedb:tag:latest": null } },
+]);
+```
+
+## Step 4: Confirm
+
+Report back:
+- Which columns were updated and what was written
+- The new table version number (from the result)
+- Any columns skipped (e.g., already had up-to-date metadata)
+
+## Quick examples
+
+**"Write descriptions for all columns in the `product_embeddings` table"**
+1. Read `table.schema` → all fields + existing metadata
+2. Generate a `lancedb:description` for each column based on name + type
+3. One `update_field_metadata` call with all descriptions
+4. Report
+
+**"Tag the columns in `model_outputs` with their field type and model"**
+1. Read the schema
+2. For each field, classify by name + Arrow type → set `lancedb:tag:field_type` and `lancedb:tag:model` where applicable
+3. Write in one batched call
+4. Report
+
+**"Group the feature columns in `training_features` into logical families and mark the latest version"**
+1. Read the schema
+2. Find version patterns → assign `lancedb:logical-column` and `lancedb:tag:version`; mark newest with `lancedb:tag:latest: "true"`
+3. Write in one batched call
+4. Show the grouping
@@ -4,12 +4,19 @@ Quick method reference for Python LanceDB code. Cross-check source for non-trivi

 ## Connect

+If you're connecting to a remote database, use this:
 ```python
 import lancedb

-db = lancedb.connect("./camelot-db")         # local/OSS
-db = lancedb.connect("db://my-db", api_key=api_key, region=region)  # remote
+db = lancedb.connect("db://my-db", api_key=api_key, host_override=host_override) # remote
 ```
+(values may be found in LANCEDB_API_KEY and LANCEDB_HOST_OVERRIDE, either in env vars or a .env file)
+
+If you're connecting to a local table using OSS LanceDB, use this:
+```python
+db = lancedb.connect("./camelot-db") # local/OSS
+```
+If you're not sure which, or if you can't find the api_key or host_override params, ask the user.

 **Place the local database directory next to the script/entrypoint that opens it** (i.e. resolve the path relative to the script, `Path(__file__).parent / "camelot-db"`), not buried under a shared `data/` folder. The Lance dataset is the database, not a data file — keeping it beside its code makes ownership obvious and paths stable regardless of the working directory the script is launched from.

@@ -96,6 +103,32 @@ print(table.index_stats("vector_idx"))

 Use these before changing indexes or search tuning.

+## Column (Field) Metadata
+
+```python
+schema = table.schema                       # sync property; async: await table.schema()
+meta = schema.field("category").metadata    # dict[bytes, bytes] — Arrow metadata is bytes-keyed
+res = table.update_field_metadata(          # varargs: one dict per field; works local + remote
+    {"path": "category", "metadata": {"lancedb:description": "...", "lancedb:tag:field_type": "label"}}
+)
+res.version  # new table version
+```
+
+Merges by default; a `None` value deletes that key; `"replace": True` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). `replace_field_metadata` is deprecated. See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
+
+## Branches
+
+```python
+table.branches.list()                          # non-main branches; {} = only main
+exp = table.branches.create("exp")             # fork off main -> handle scoped to the branch
+wip = table.branches.checkout("wip")           # existing branch -> scoped handle (version= pins read-only)
+wip = db.open_table("t", branch="wip")         # or open scoped directly
+table.branches.delete("stale")                 # removes only the branch pointer
+table.current_branch()                         # None = main
+```
+
+There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
+
 ## Maintenance

 ```python
@@ -0,0 +1,45 @@
+# Connecting to a LanceDB remote server
+
+LanceDB Enterprise/Cloud deployments are served by a server implementing the
+lance-namespace OpenAPI spec
+(<https://github.com/lance-format/lance-namespace/blob/main/docs/src/spec.yaml>).
+Every remote (`db://...`) connection talks to such a server, and some operations
+exist only there. In particular, all operations around jobs (listing, inspecting,
+creating, or canceling jobs) run server-side — there is no local/OSS equivalent, so
+resolve a server connection before attempting any job work. The job REST methods
+themselves are documented in `references/remote_jobs.md`.
+
+Every request needs two things:
+
+1. **Base URL** — the server endpoint
+2. **Credentials** — an API key (`x-api-key` header over REST), and usually a database name (`x-lancedb-database` header)
+
+## Resolution steps
+
+1. If the user already gave a URL and API key (or said which environment they're working against), use that.
+2. Otherwise, look for credentials already available in the environment:
+   - Env vars like `LANCEDB_URI` / `LANCEDB_HOST` / `LANCEDB_API_KEY`
+   - A server endpoint already running or port-forwarded locally (the REST default port is 2333, i.e. `http://localhost:2333`)
+3. If you didn't find both pieces, ask the user directly: **"What's your LanceDB endpoint's URL, and what's your API key?"** Also ask which database to use if it isn't obvious. Don't guess or probe further — the user knows their deployment.
+
+## Validating the connection
+
+Make a cheap authenticated request and check the status before starting real work:
+
+```bash
+curl -s -w "\n%{http_code}" "{base_url}/v1/table/?limit=1" \
+  -H "x-api-key: <key>" \
+  -H "x-lancedb-database: <database>"
+```
+
+- `200` — connection, key, and database header all good
+- `401` — API key missing or wrong
+- `400` mentioning a database header — this deployment expects `x-lancedb-database`
+
+## Non-REST equivalents
+
+The same credentials work through the SDKs and CLI:
+
+- Python SDK: `lancedb.connect("db://<database>", api_key="<key>", host_override="<base_url>")`
+- TypeScript SDK: `await lancedb.connect("db://<database>", { apiKey: "<key>", hostOverride: "<base_url>" })`
+- `lancedb` CLI: a `[profiles.<name>]` entry in `~/.lancedb/config.toml` with `http_server_url`, `api_key`, `database`
@@ -0,0 +1,151 @@
+# Job operations over the LanceDB remote server REST API
+
+Jobs are server-side background operations on LanceDB Enterprise/Cloud — index builds,
+column backfills, materialized view refreshes, and similar async work. Endpoints that
+trigger async work (e.g. the column backfill or materialized view refresh endpoints)
+return a `job_id`; these four methods are how you track and manage those jobs.
+
+Resolve the connection first — see `references/remote_connect.md`. All four methods
+are **POST** requests under `{base_url}/v1/jobs/` with JSON bodies, and take the usual
+`x-api-key` / `x-lancedb-database` headers. If every job call returns `501`, job APIs
+are disabled on that deployment (the server has no job registry configured) — report
+that rather than retrying.
+
+## 1. List jobs — `POST /v1/jobs/list`
+
+The body is optional; an empty body lists everything. All fields are filters:
+
+```json
+{
+  "limit": 100,
+  "table_name": "my_table",
+  "job_type": "...",
+  "job_subtype": "...",
+  "state": "...",
+  "page_token": "..."
+}
+```
+
+```bash
+curl -s -X POST "{base_url}/v1/jobs/list" \
+  -H "x-api-key: <key>" -H "x-lancedb-database: <database>" \
+  -H "content-type: application/json" \
+  -d '{"table_name": "my_table"}'
+```
+
+Response:
+
+```json
+{
+  "jobs": [
+    {
+      "job_id": "...",
+      "table": "my_table",
+      "job_type": "...",
+      "job_subtype": "...",
+      "state": "done",
+      "created_at_millis": 1720000000000
+    }
+  ],
+  "page_token": "..."
+}
+```
+
+A `page_token` in the response means there are more results — pass it back in the next
+request to continue. Note list rows use a lowercase `state` string, while describe uses
+an uppercase `job_state`.
+
+## 2. Describe a job — `POST /v1/jobs/describe`
+
+Body: `{"job_id": "<id>"}`. Returns full detail for one job:
+
+```json
+{
+  "job_id": "...",
+  "job_type": "...",
+  "job_subtype": "...",
+  "job_state": "IN_PROGRESS",
+  "creation_ms": 1720000000000,
+  "spec": {},
+  "status": {}
+}
+```
+
+`job_state` is one of `IN_PROGRESS`, `CANCELLED`, `FAILED`, `DONE`. `spec` and `status`
+are job-type-specific JSON objects (the job's input specification and its current
+progress/status). Returns `404` for an unknown job id.
+
+## 3. Cancel a job — `POST /v1/jobs/cancel`
+
+Body: `{"job_id": "<id>"}`; response echoes `{"job_id": "<id>"}`. Cancellation is a
+service-level operation requiring the same administrative authorization as the
+`/admin` routes — a database-scoped API key that can list and describe jobs may still
+get a permission error here. Other errors: `404` unknown job, `409` state conflict
+(e.g. already in a terminal state), `429` too much write contention (safe to retry).
+
+## 4. Query job event history — `POST /v1/jobs/query_events`
+
+Returns the event history (state transitions, progress updates) for one or more jobs.
+Body: `{"job_id": "<id>"}` for one job, or `{"job_ids": ["<id>", ...]}` for a batch.
+Optional fields: `limit` (max event rows), `limit_per_job` (per job in a batch query),
+and `filter` — a SQL-like expression over the columns `state`, `updated_by`,
+`owner_component`, and `claim_entity`. (`full_text_search` is reserved and currently
+rejected as not implemented.)
+
+The response is **not JSON** — it is an Arrow IPC stream
+(`content-type: application/vnd.apache.arrow.stream`). Decode it, e.g. in Python:
+
+```python
+import pyarrow.ipc
+import requests
+
+resp = requests.post(
+    f"{base_url}/v1/jobs/query_events",
+    headers={"x-api-key": key, "x-lancedb-database": database},
+    json={"job_id": job_id},
+)
+resp.raise_for_status()
+events = pyarrow.ipc.open_stream(resp.content).read_all()
+```
+
+## Feature engineering (Geneva) jobs
+
+Feature engineering jobs — UDF column backfills and materialized view refreshes run
+through Geneva — are tracked **separately** from the `/v1/jobs` registry above. Their
+records live in a `geneva_jobs` table inside the database itself (in the `__system`
+namespace), and you access them through a Python `geneva` connection rather than the
+REST endpoints above:
+
+```python
+import geneva
+from geneva.jobs import JobStateManager
+
+# Same credentials as lancedb.connect / the REST API
+conn = geneva.connect("db://<database>", api_key="<key>", host_override="<base_url>")
+jsm = JobStateManager(conn)
+
+# List jobs. NOTE: status defaults to "RUNNING"; pass status=None for all jobs.
+# Statuses: PENDING | RUNNING | DONE | FAILED | CANCELLED
+jobs = jsm.list_jobs(table_name="my_table", status=None)
+
+# Fetch one job by id (returns a list of JobRecord)
+records = jsm.get("<job_id>")
+```
+
+Each `JobRecord` has `table_name`, `column_name`, `job_id`, `job_type`, `status`,
+`launched_at`, `completed_at`, `config`, `launched_by`, `manifest_id`, `cluster_name`,
+`metrics` (progress counters), `events` (human-readable history), and `updated_at`.
+For filters `list_jobs` doesn't support (e.g. time ranges), query the underlying table
+directly: `jsm.get_table(True).search().where("launched_at >= TIMESTAMP '...'")` —
+pass `True` to check out the latest version, since other processes update job state.
+
+Stale-status caveat: nothing reaps dead Geneva jobs, so a job can sit in
+`RUNNING`/`PENDING` forever if its worker died. Treat a job as effectively `FAILED`
+when it has been running longer than ~36 hours, or its `updated_at` is more than ~2
+hours old (this matches the heuristic the Geneva console UI applies on read).
+
+## Workflow tips
+
+- To wait for async work (a backfill, an index build), poll `describe` until
+  `job_state` leaves `IN_PROGRESS`; on `FAILED`, pull `status` and `query_events` for
+  the failure detail.
@@ -69,6 +69,33 @@ console.log(await table.indexStats("vector_idx"));

 Use these before changing indexes or search tuning.

+## Column (Field) Metadata
+
+```typescript
+const schema = await table.schema();
+const meta = schema.fields.find((f) => f.name === "category")?.metadata; // Map<string, string>
+const res = await table.updateFieldMetadata([
+  { path: "category", metadata: { "lancedb:description": "...", "lancedb:tag:field_type": "label" } },
+]);
+res.version; // new table version
+```
+
+Merges by default; a `null` value deletes that key; `replace: true` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
+
+## Branches
+
+```typescript
+const branches = await table.branches();       // async manager
+await branches.list();                         // non-main branches; {} = only main
+const exp = await branches.create("exp");      // fork off main -> Table scoped to the branch
+const wip = await branches.checkout("wip");    // existing branch -> scoped Table (version arg pins read-only)
+const wip2 = await db.openTable("t", { branch: "wip" }); // or open scoped directly
+await branches.delete("stale");                // removes only the branch pointer
+table.currentBranch();                         // null = main
+```
+
+There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
+
 ## Maintenance

 ```typescript
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.35.0-beta.2"
+current_version = "0.35.0-beta.3"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-python"
-version = "0.35.0-beta.2"
+version = "0.35.0-beta.3"
 publish = false
 edition.workspace = true
 description = "Python bindings for LanceDB"
@@ -8,6 +8,27 @@ A Python library for [LanceDB](https://github.com/lancedb/lancedb).
 pip install lancedb
 ```

+### Pre-Haswell x86_64 hosts: `lancedb-compat`
+
+The default `lancedb` wheel targets `x86-64-haswell` (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with `Illegal instruction` at `import lancedb`.
+
+For those hosts, install the `lancedb-compat` package instead:
+
+```bash
+pip install lancedb-compat
+```
+
+Same Python API (`import lancedb` works as usual). The compat wheel is compiled at the `x86-64-v2` baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use `lance.simd_info()` from Python to verify which tier was selected.
+
+`lancedb` and `lancedb-compat` install to the same `lancedb/` namespace and conflict at install time. Pick one. To switch, `pip uninstall lancedb` first, then `pip install lancedb-compat` (or vice-versa).
+
+If you need a custom baseline (or `lancedb-compat` isn't yet published for your platform), build from source with the override:
+
+```bash
+RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
+pip install ./target/wheels/lancedb-*.whl
+```
+
 ### Preview Releases

 Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
@@ -61,10 +61,11 @@ tests = [
    "duckdb>=0.9.0",
    "pytz>=2023.3",
    "polars>=0.19, <=1.3.0",
+    "pyarrow<25",
    "pyarrow-stubs>=16.0",
-    "pylance>=5.0.0b5",
+    "pylance==9.0.0rc1",
    "requests>=2.31.0",
-    "datafusion>=52,<53",
+    "datafusion>=54,<55",
    "opentelemetry-sdk>=1.30.0",
 ]
 dev = [
@@ -30,6 +30,7 @@ from .types import BaseTokenizerType
 IvfHnswPq: type[HnswPq] = HnswPq
 IvfHnswSq: type[HnswSq] = HnswSq
 IvfHnswFlat: type[HnswFlat] = HnswFlat
+AnalyzePlanDistributedMetrics = Literal["aggregate", "per_worker", "full"]

 class MetricPoint:
    name: str
@@ -218,6 +219,7 @@ class Table:
        data: pa.RecordBatchReader,
        mode: Literal["append", "overwrite"],
        progress: Optional[Any] = None,
+        write_parallelism: Optional[int] = None,
    ) -> AddResult: ...
    async def update(
        self, updates: Dict[str, str], where: Optional[str]
@@ -317,6 +319,10 @@ class Branches:
    ) -> Table: ...
    async def checkout(self, name: str, version: Optional[int] = None) -> Table: ...
    async def delete(self, name: str) -> None: ...
+    async def diff(self, from_branch: str) -> Dict[str, Any]: ...
+    async def merge(
+        self, from_branch: str, dry_run: bool = False
+    ) -> Dict[str, Any]: ...

 class IndexConfig:
    name: str
@@ -393,7 +399,9 @@ class Query:
        self, max_batch_length: Optional[int], timeout: Optional[timedelta]
    ) -> RecordBatchStream: ...
    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
-    async def analyze_plan(self) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class TakeQuery:
@@ -401,6 +409,10 @@ class TakeQuery:
    def with_row_id(self): ...
    async def output_schema(self) -> pa.Schema: ...
    async def execute(self) -> RecordBatchStream: ...
+    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class FTSQuery:
@@ -421,6 +433,10 @@ class FTSQuery:
    async def execute(
        self, max_batch_length: Optional[int], timeout: Optional[timedelta]
    ) -> RecordBatchStream: ...
+    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class VectorQuery:
@@ -443,6 +459,10 @@ class VectorQuery:
    def bypass_vector_index(self): ...
    def nearest_to_text(self, query: dict) -> HybridQuery: ...
    def order_by(self, ordering: Optional[List[ColumnOrdering]]): ...
+    async def explain_plan(self, verbose: Optional[bool]) -> str: ...
+    async def analyze_plan(
+        self, distributed_metrics: Optional[AnalyzePlanDistributedMetrics] = None
+    ) -> str: ...
    def to_query_request(self) -> PyQueryRequest: ...

 class HybridQuery:
@@ -41,6 +41,7 @@ from lance_namespace import (
    ListTablesResponse,
    connect as namespace_connect,
 )
+from lance_namespace.errors import NamespaceNotEmptyError, TableNotFoundError

 from . import __version__
 from ._lancedb import connect as lancedb_connect  # type: ignore
@@ -746,10 +747,12 @@ class LanceDBConnection(DBConnection):
        """
        if namespace_path is None:
            namespace_path = []
-        return self._namespace_conn().list_namespaces(
-            namespace_path=namespace_path,
-            page_token=page_token,
-            limit=limit,
+        return LOOP.run(
+            self._conn.list_namespaces(
+                namespace_path=namespace_path,
+                page_token=page_token,
+                limit=limit,
+            )
        )

    @override
@@ -759,10 +762,12 @@ class LanceDBConnection(DBConnection):
        mode: Optional[str] = None,
        properties: Optional[Dict[str, str]] = None,
    ) -> CreateNamespaceResponse:
-        return self._namespace_conn().create_namespace(
-            namespace_path=namespace_path,
-            mode=mode,
-            properties=properties,
+        return LOOP.run(
+            self._conn.create_namespace(
+                namespace_path=namespace_path,
+                mode=mode,
+                properties=properties,
+            )
        )

    @override
@@ -772,19 +777,24 @@ class LanceDBConnection(DBConnection):
        mode: Optional[str] = None,
        behavior: Optional[str] = None,
    ) -> DropNamespaceResponse:
-        return self._namespace_conn().drop_namespace(
-            namespace_path=namespace_path,
-            mode=mode,
-            behavior=behavior,
-        )
+        try:
+            return LOOP.run(
+                self._conn.drop_namespace(
+                    namespace_path=namespace_path,
+                    mode=mode,
+                    behavior=behavior,
+                )
+            )
+        except RuntimeError as e:
+            if "Namespace not empty" in str(e):
+                raise NamespaceNotEmptyError(str(e)) from e
+            raise

    @override
    def describe_namespace(
        self, namespace_path: List[str]
    ) -> DescribeNamespaceResponse:
-        return self._namespace_conn().describe_namespace(
-            namespace_path=namespace_path,
-        )
+        return LOOP.run(self._conn.describe_namespace(namespace_path=namespace_path))

    @override
    def list_tables(
@@ -813,12 +823,6 @@ class LanceDBConnection(DBConnection):
        """
        if namespace_path is None:
            namespace_path = []
-        if namespace_path:
-            return self._namespace_conn().list_tables(
-                namespace_path=namespace_path,
-                page_token=page_token,
-                limit=limit,
-            )
        return LOOP.run(
            self._conn.list_tables(
                namespace_path=namespace_path, page_token=page_token, limit=limit
@@ -916,22 +920,6 @@ class LanceDBConnection(DBConnection):
            raise ValueError("mode must be either 'create' or 'overwrite'")
        validate_table_name(name)

-        if namespace_path:
-            return self._namespace_conn().create_table(
-                name,
-                data=data,
-                schema=schema,
-                mode=mode,
-                exist_ok=exist_ok,
-                on_bad_vectors=on_bad_vectors,
-                fill_value=fill_value,
-                embedding_functions=embedding_functions,
-                namespace_path=namespace_path,
-                storage_options=storage_options,
-                data_storage_version=data_storage_version,
-                enable_v2_manifest_paths=enable_v2_manifest_paths,
-            )
-
        tbl = LanceTable.create(
            self,
            name,
@@ -944,22 +932,11 @@ class LanceDBConnection(DBConnection):
            embedding_functions=embedding_functions,
            namespace_path=namespace_path,
            storage_options=storage_options,
+            data_storage_version=data_storage_version,
+            enable_v2_manifest_paths=enable_v2_manifest_paths,
        )
        return tbl

-    def _namespace_conn(self) -> DBConnection:
-        """Return a LanceNamespaceDBConnection backed by this connection's
-        directory namespace.  Used to delegate child-namespace operations."""
-        from lancedb.namespace import LanceNamespaceDBConnection
-
-        return LanceNamespaceDBConnection(
-            self.namespace_client(),
-            read_consistency_interval=self.read_consistency_interval,
-            storage_options=self.storage_options,
-            namespace_client_impl=None,
-            namespace_client_properties=None,
-        )
-
    @override
    def open_table(
        self,
@@ -1006,14 +983,7 @@ class LanceDBConnection(DBConnection):
                stacklevel=2,
            )

-        if namespace_path:
-            tbl = self._namespace_conn().open_table(
-                name,
-                namespace_path=namespace_path,
-                storage_options=storage_options,
-                index_cache_size=index_cache_size,
-            )
-        else:
+        try:
            tbl = LanceTable.open(
                self,
                name,
@@ -1021,6 +991,15 @@ class LanceDBConnection(DBConnection):
                storage_options=storage_options,
                index_cache_size=index_cache_size,
            )
+        except (RuntimeError, ValueError) as e:
+            if namespace_path and (
+                "Table not found" in str(e) or "was not found" in str(e)
+            ):
+                table_id = namespace_path + [name]
+                raise TableNotFoundError(
+                    f"Table not found: {'$'.join(table_id)}"
+                ) from e
+            raise

        if branch is not None:
            tbl = tbl.branches.checkout(branch, version)
@@ -1104,9 +1083,6 @@ class LanceDBConnection(DBConnection):
        """
        if namespace_path is None:
            namespace_path = []
-        if namespace_path:
-            self._namespace_conn().drop_table(name, namespace_path=namespace_path)
-            return
        LOOP.run(
            self._conn.drop_table(
                name, namespace_path=namespace_path, ignore_missing=ignore_missing
@@ -14,29 +14,76 @@ import numpy as np

 DEFAULT_WATSONX_URL = "https://us-south.ml.cloud.ibm.com"

-MODELS_DIMS = {
+# Models currently available on the watsonx.ai SaaS platform.
+# These are the IDs advertised to new users via model_names() and shown in
+# validation error messages.  Regional availability and withdrawal dates are
+# documented at:
+# https://www.ibm.com/docs/en/watsonx/saas?topic=models-supported-encoder
+CURRENT_MODELS: dict[str, int] = {
+    "ibm/granite-embedding-278m-multilingual": 768,
+    "ibm/slate-125m-english-rtrvr-v2": 768,
+    "ibm/slate-30m-english-rtrvr-v2": 384,
+    "intfloat/multilingual-e5-large": 1024,
+}
+
+# Full dimension map including legacy model IDs from earlier releases.
+# Kept so that existing tables whose stored metadata uses these names can still
+# resolve dimensions on load without raising an error.  These IDs are NOT
+# advertised to new users.
+MODELS_DIMS: dict[str, int] = {
+    **CURRENT_MODELS,
+    # Deprecated — withdrawal announced but still functional until the dates above.
+    "sentence-transformers/all-minilm-l6-v2": 384,
+    # Pre-v2 legacy names retained for metadata compatibility only.
    "ibm/slate-125m-english-rtrvr": 768,
    "ibm/slate-30m-english-rtrvr": 384,
    "sentence-transformers/all-minilm-l12-v2": 384,
-    "intfloat/multilingual-e5-large": 1024,
 }


@register("watsonx")
 class WatsonxEmbeddings(TextEmbeddingFunction):
    """
+    An embedding function that uses the IBM watsonx.ai Embeddings API.
+
    API Docs:
-    ---------
-    https://cloud.ibm.com/apidocs/watsonx-ai#text-embeddings
+        https://cloud.ibm.com/apidocs/watsonx-ai#text-embeddings

    Supported embedding models:
-    ---------------------------
-    https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx
+        https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx
+
+    Parameters
+    ----------
+    name : str, default "ibm/slate-125m-english-rtrvr"
+        The ID of the embedding model to use.  For new tables,
+        ``"ibm/granite-embedding-278m-multilingual"`` is recommended.
+    api_key : str, optional
+        IBM Cloud API key.  Falls back to the ``WATSONX_API_KEY`` environment
+        variable when not provided.
+    project_id : str, optional
+        watsonx.ai project ID.  Explicit value takes precedence over the
+        ``WATSONX_PROJECT_ID`` environment variable.  Mutually exclusive with
+        ``space_id`` — exactly one must be supplied.
+    space_id : str, optional
+        watsonx.ai deployment space ID.  Explicit value takes precedence over
+        the ``WATSONX_SPACE_ID`` environment variable.  Mutually exclusive with
+        ``project_id`` — exactly one must be supplied.
+    url : str, optional
+        watsonx.ai service URL.  Defaults to
+        ``"https://us-south.ml.cloud.ibm.com"``.
+    params : dict, optional
+        Extra parameters forwarded verbatim to ``Embeddings`` (e.g.
+        ``{"truncate_input_tokens": 512}``).
    """

+    # Intentionally kept at the original pre-PR default so that existing tables
+    # whose stored metadata contains model:{} reload with the same model they
+    # were created with.  New users should pass name= explicitly, e.g.
+    # name="ibm/granite-embedding-278m-multilingual".
    name: str = "ibm/slate-125m-english-rtrvr"
    api_key: Optional[str] = None
    project_id: Optional[str] = None
+    space_id: Optional[str] = None
    url: Optional[str] = None
    params: Optional[Dict] = None

@@ -46,12 +93,13 @@ class WatsonxEmbeddings(TextEmbeddingFunction):

    @staticmethod
    def model_names():
-        return [
-            "ibm/slate-125m-english-rtrvr",
-            "ibm/slate-30m-english-rtrvr",
-            "sentence-transformers/all-minilm-l12-v2",
-            "intfloat/multilingual-e5-large",
-        ]
+        """Return the IDs of models currently available for new tables.
+
+        Legacy / deprecated IDs are intentionally excluded.  They remain
+        resolvable for dimension lookups on existing tables via ``MODELS_DIMS``,
+        but should not be used when creating new tables.
+        """
+        return list(CURRENT_MODELS.keys())

    def ndims(self):
        return self._ndims
@@ -59,7 +107,10 @@ class WatsonxEmbeddings(TextEmbeddingFunction):
    @cached_property
    def _ndims(self):
        if self.name not in MODELS_DIMS:
-            raise ValueError(f"Unknown model name {self.name}")
+            raise ValueError(
+                f"Unknown model '{self.name}'. "
+                f"Available models: {list(CURRENT_MODELS.keys())}"
+            )
        return MODELS_DIMS[self.name]

    def generate_embeddings(
@@ -81,27 +132,45 @@ class WatsonxEmbeddings(TextEmbeddingFunction):
            "ibm_watsonx_ai.foundation_models"
        )

-        kwargs = {"model_id": self.name}
+        # --- credentials ---
+        # Explicit field takes priority; env var is the fallback.
+        api_key = self.api_key or os.environ.get("WATSONX_API_KEY")
+        if not api_key:
+            raise ValueError(
+                "WATSONX_API_KEY not set. Either set it in your environment or "
+                "pass it as `api_key` argument to WatsonxEmbeddings."
+            )
+        credentials = ibm_watsonx_ai.Credentials(
+            api_key=api_key,
+            url=self.url or DEFAULT_WATSONX_URL,
+        )
+
+        # --- project_id / space_id (exactly one required) ---
+        # Explicit field always wins; env var is consulted only when the
+        # corresponding field was not set, so passing project_id= never
+        # conflicts with a stray WATSONX_SPACE_ID env var and vice-versa.
+        space_id, project_id = self.space_id, self.project_id
+
+        if project_id is None and space_id is None:
+            # Neither was passed explicitly — fall back to env vars.
+            project_id = os.environ.get("WATSONX_PROJECT_ID")
+            space_id = os.environ.get("WATSONX_SPACE_ID")
+
+        if project_id and space_id:
+            raise ValueError("Provide either `project_id` or `space_id`, not both.")
+        if not project_id and not space_id:
+            raise ValueError(
+                "Either WATSONX_PROJECT_ID or WATSONX_SPACE_ID must be set. "
+                "Pass one as an argument to WatsonxEmbeddings or set the "
+                "corresponding environment variable."
+            )
+
+        client_kwargs: Dict = dict(model_id=self.name, credentials=credentials)
        if self.params:
-            kwargs["params"] = self.params
-        if self.project_id:
-            kwargs["project_id"] = self.project_id
-        elif "WATSONX_PROJECT_ID" in os.environ:
-            kwargs["project_id"] = os.environ["WATSONX_PROJECT_ID"]
+            client_kwargs["params"] = self.params
+        if project_id:
+            client_kwargs["project_id"] = project_id
        else:
-            raise ValueError("WATSONX_PROJECT_ID must be set or passed")
+            client_kwargs["space_id"] = space_id

-        creds_kwargs = {}
-        if self.api_key:
-            creds_kwargs["api_key"] = self.api_key
-        elif "WATSONX_API_KEY" in os.environ:
-            creds_kwargs["api_key"] = os.environ["WATSONX_API_KEY"]
-        else:
-            raise ValueError("WATSONX_API_KEY must be set or passed")
-        if self.url:
-            creds_kwargs["url"] = self.url
-        else:
-            creds_kwargs["url"] = DEFAULT_WATSONX_URL
-        kwargs["credentials"] = ibm_watsonx_ai.Credentials(**creds_kwargs)
-
-        return ibm_watsonx_ai_foundation_models.Embeddings(**kwargs)
+        return ibm_watsonx_ai_foundation_models.Embeddings(**client_kwargs)
@@ -115,6 +115,12 @@ class FTS:

    For example, it works with `title`, `description`, `content`, etc.

+    Examples
+    --------
+    Create an index configuration that uses 256-document posting blocks:
+
+    >>> config = FTS(block_size=256)
+
    Attributes
    ----------
    with_position : bool, default False
@@ -148,6 +154,10 @@ class FTS:
    ascii_folding : bool, default True
        Whether to fold ASCII characters. This converts accented characters to
        their ASCII equivalent. For example, "café" would be converted to "cafe".
+    block_size : int, default 128
+        The number of documents per compressed posting block. Supported values
+        are 128 and 256. A value of 256 uses the experimental FTS V3 format
+        and may introduce breaking changes.

    Notes
    -----
@@ -168,6 +178,7 @@ class FTS:
    ngram_min_length: int = 3
    ngram_max_length: int = 3
    prefix_only: bool = False
+    block_size: int = 128


@dataclass
@@ -885,7 +885,7 @@ class Permutation:
        This method refines the current selection, potentially removing columns.  It
        will not add back columns that were previously removed.

-        If any of the columns do not exist then an error will be raised
+        If any of the columns do not exist then an error will be raised.

        This does not introduce a post-processing step.  It simply reduces the amount
        of data we read.
@@ -898,9 +898,14 @@ class Permutation:
        for name in columns:
            value = self.selection.get(name, None)
            if value is None:
-                raise ValueError(
-                    f"Cannot select column `{name}` because it does not exist"
-                )
+                if name == "_rowid":
+                    # _rowid is a system column not in the default schema
+                    # but can be explicitly selected
+                    value = "_rowid"
+                else:
+                    raise ValueError(
+                        f"Cannot select column `{name}` because it does not exist"
+                    )
            new_selection[name] = value
        return self._with_selection(new_selection)

@@ -79,6 +79,7 @@ if TYPE_CHECKING:
        from typing_extensions import Self

 T = TypeVar("T", bound="LanceModel")
+AnalyzePlanDistributedMetrics = Literal["aggregate", "per_worker", "full"]


@runtime_checkable
@@ -1372,7 +1373,9 @@ class LanceQueryBuilder(ABC):
        self._order_by = ordering
        return self

-    def analyze_plan(self) -> str:
+    def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """
        Run the query and return its execution plan with runtime metrics.

@@ -1410,12 +1413,22 @@ class LanceQueryBuilder(ABC):
                      fragments_scanned=..., ranges_scanned=1, rows_scanned=1,
                      bytes_read=..., iops=..., requests=..., task_wait_time=...]

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+            "aggregate" preserves the legacy summary, "per_worker" shows each
+            worker separately, and "full" includes both.
+
        Returns
        -------
        plan : str
            The physical query execution plan with runtime metrics.
        """
-        return self._table._analyze_plan(self.to_query_object())
+        return self._table._analyze_plan(
+            self.to_query_object(), distributed_metrics=distributed_metrics
+        )

    def vector(self, vector: Union[np.ndarray, list]) -> Self:
        """Set the vector to search for.
@@ -2581,9 +2594,17 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
        indented_fts = "\n".join("  " + line for line in fts_plan.splitlines())
        return f"{reranker_label}\n  {indented_vector}\n  {indented_fts}"

-    def analyze_plan(self):
+    def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """Execute the query and display with runtime metrics.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
@@ -2591,9 +2612,19 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
        self._create_query_builders()

        results = ["Vector Search Plan:"]
-        results.append(self._table._analyze_plan(self._vector_query.to_query_object()))
+        results.append(
+            self._table._analyze_plan(
+                self._vector_query.to_query_object(),
+                distributed_metrics=distributed_metrics,
+            )
+        )
        results.append("FTS Search Plan:")
-        results.append(self._table._analyze_plan(self._fts_query.to_query_object()))
+        results.append(
+            self._table._analyze_plan(
+                self._fts_query.to_query_object(),
+                distributed_metrics=distributed_metrics,
+            )
+        )
        return "\n".join(results)

    def _create_query_builders(self):
@@ -3080,14 +3111,22 @@ class AsyncQueryBase(object):
        """  # noqa: E501
        return await self._inner.explain_plan(verbose)

-    async def analyze_plan(self):
+    async def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """Execute the query and display with runtime metrics.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
        """
-        return await self._inner.analyze_plan()
+        return await self._inner.analyze_plan(distributed_metrics)


 class AsyncStandardQuery(AsyncQueryBase):
@@ -3836,18 +3875,16 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
        >>> asyncio.run(doctest_example()) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
        RRFReranker(K=60)
            ProjectionExec: expr=[vector@0 as vector, text@3 as text, _distance@2 as _distance]
-              Take: columns="vector, _rowid, _distance, (text)"
-                CoalesceBatchesExec: target_batch_size=1024
-                  GlobalLimitExec: skip=0, fetch=10
-                    FilterExec: _distance@2 IS NOT NULL
-                      SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
-                        KNNVectorDistance: metric=l2
-                          LanceRead: uri=..., projection=[vector], ...
+              LanceRead: uri=..., projection=[text], source=stream(_rowid)
+                GlobalLimitExec: skip=0, fetch=10
+                  FilterExec: _distance@2 IS NOT NULL
+                    SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
+                      KNNVectorDistance: metric=l2
+                        LanceRead: uri=..., projection=[vector], ...
            ProjectionExec: expr=[vector@2 as vector, text@3 as text, _score@1 as _score]
-              Take: columns="_rowid, _score, (vector), (text)"
-                CoalesceBatchesExec: target_batch_size=1024
-                  GlobalLimitExec: skip=0, fetch=10
-                    MatchQuery: column=text, query=hello
+              LanceRead: uri=..., projection=[vector, text], source=stream(_rowid)
+                GlobalLimitExec: skip=0, fetch=10
+                  MatchQuery: column=text, query=[hello]

        Parameters
        ----------
@@ -3866,7 +3903,9 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
        indented_fts = "\n".join("  " + line for line in fts_plan.splitlines())
        return f"{self._reranker}\n  {indented_vector}\n  {indented_fts}"

-    async def analyze_plan(self):
+    async def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """
        Execute the query and return the physical execution plan with runtime metrics.

@@ -3875,14 +3914,24 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
        elapsed time, I/O stats, and more. It’s useful for debugging and
        performance analysis.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
        """
        results = ["Vector Search Query:"]
-        results.append(await self._inner.to_vector_query().analyze_plan())
+        results.append(
+            await self._inner.to_vector_query().analyze_plan(distributed_metrics)
+        )
        results.append("FTS Search Query:")
-        results.append(await self._inner.to_fts_query().analyze_plan())
+        results.append(
+            await self._inner.to_fts_query().analyze_plan(distributed_metrics)
+        )

        return "\n".join(results)

@@ -4166,14 +4215,22 @@ class BaseQueryBuilder(object):
        """  # noqa: E501
        return LOOP.run(self._inner.explain_plan(verbose))

-    def analyze_plan(self):
+    def analyze_plan(
+        self, distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate"
+    ) -> str:
        """Execute the query and display with runtime metrics.

+        Parameters
+        ----------
+        distributed_metrics : Literal["aggregate", "per_worker", "full"]
+            Defaults to "aggregate".
+            How distributed worker metrics are displayed for remote query plans.
+
        Returns
        -------
        plan : str
        """
-        return LOOP.run(self._inner.analyze_plan())
+        return LOOP.run(self._inner.analyze_plan(distributed_metrics))


 class LanceTakeQueryBuilder(BaseQueryBuilder):
@@ -56,7 +56,12 @@ from lancedb.merge import LanceMergeInsertBuilder
 from lancedb.embeddings import EmbeddingFunctionRegistry
 from lancedb.table import _normalize_progress

-from ..query import LanceVectorQueryBuilder, LanceQueryBuilder, LanceTakeQueryBuilder
+from ..query import (
+    AnalyzePlanDistributedMetrics,
+    LanceQueryBuilder,
+    LanceTakeQueryBuilder,
+    LanceVectorQueryBuilder,
+)
 from ..table import AsyncTable, BlobMode, Branches, IndexStatistics, Query, Table, Tags
 from ..types import BaseTokenizerType

@@ -339,6 +344,7 @@ class RemoteTable(Table):
        ngram_min_length: int = 3,
        ngram_max_length: int = 3,
        prefix_only: bool = False,
+        block_size: int = 128,
        name: Optional[str] = None,
    ):
        """Create a full-text search index on a column.
@@ -359,6 +365,7 @@ class RemoteTable(Table):
            ngram_min_length=ngram_min_length,
            ngram_max_length=ngram_max_length,
            prefix_only=prefix_only,
+            block_size=block_size,
        )
        LOOP.run(
            self._table.create_index(
@@ -569,6 +576,7 @@ class RemoteTable(Table):
        on_bad_vectors: str = "error",
        fill_value: float = 0.0,
        progress: Optional[Union[bool, Callable, Any]] = None,
+        write_parallelism: Optional[int] = None,
    ) -> AddResult:
        """Add more data to the [Table](Table). It has the same API signature as
        the OSS version.
@@ -594,6 +602,12 @@ class RemoteTable(Table):
        progress: bool, callable, or tqdm-like, optional
            A callback or tqdm-compatible progress bar. See
            :meth:`Table.add` for details.
+        write_parallelism: int, optional
+            Number of partitions to write in parallel. Higher values increase
+            throughput but also peak memory use, since each partition buffers
+            data in flight. Defaults to an estimate based on the data size,
+            capped at the number of CPU cores. Lower this if bulk ingestion is
+            using too much memory.

        Returns
        -------
@@ -609,6 +623,7 @@ class RemoteTable(Table):
                    on_bad_vectors=on_bad_vectors,
                    fill_value=fill_value,
                    progress=progress,
+                    write_parallelism=write_parallelism,
                )
            )
        finally:
@@ -718,8 +733,15 @@ class RemoteTable(Table):
    def _explain_plan(self, query: Query, verbose: Optional[bool] = False) -> str:
        return LOOP.run(self._table._explain_plan(query, verbose))

-    def _analyze_plan(self, query: Query) -> str:
-        return LOOP.run(self._table._analyze_plan(query))
+    def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str:
+        return LOOP.run(
+            self._table._analyze_plan(query, distributed_metrics=distributed_metrics)
+        )

    def _output_schema(self, query: Query) -> pa.Schema:
        return LOOP.run(self._table._output_schema(query))
@@ -23,7 +23,7 @@ class AnswerdotaiRerankers(Reranker):
    column : str, default "text"
        The name of the column to use as input to the cross encoder model.
    return_score : str, default "relevance"
-        options are "relevance" or "all". Only "relevance" is supported for now.
+        options are "relevance" or "all".
    **kwargs
        Additional keyword arguments to pass to the model. For example, 'device'.
        See AnswerDotAI/rerankers for more information.
@@ -77,12 +77,13 @@ class AnswerdotaiRerankers(Reranker):
        vector_results: pa.Table,
        fts_results: pa.Table,
    ):
-        combined_results = self.merge_results(vector_results, fts_results)
+        if self.score == "all":
+            combined_results = self._merge_and_keep_scores(vector_results, fts_results)
+        else:
+            combined_results = self.merge_results(vector_results, fts_results)
        combined_results = self._rerank(combined_results, query)
        if self.score == "relevance":
            combined_results = self._keep_relevance_score(combined_results)
-        elif self.score == "all":
-            combined_results = self._merge_and_keep_scores(vector_results, fts_results)
        combined_results = combined_results.sort_by(
            [("_relevance_score", "descending")]
        )
@@ -16,7 +16,7 @@ class ColbertReranker(AnswerdotaiRerankers):
    column : str, default "text"
        The name of the column to use as input to the cross encoder model.
    return_score : str, default "relevance"
-        options are "relevance" or "all". Only "relevance" is supported for now.
+        options are "relevance" or "all".
    **kwargs
        Additional keyword arguments to pass to the model, for example, 'device'.
        See AnswerDotAI/rerankers for more information.
@@ -40,12 +40,12 @@ class WatsonxReranker(Reranker):
        IBM Cloud API key.  Falls back to the ``WATSONX_API_KEY`` environment
        variable when not provided.
    project_id : str, optional
-        watsonx.ai project ID.  Falls back to the ``WATSONX_PROJECT_ID``
-        environment variable when not provided.  Mutually exclusive with
+        watsonx.ai project ID.  Explicit value takes precedence over the
+        ``WATSONX_PROJECT_ID`` environment variable.  Mutually exclusive with
        ``space_id`` — exactly one must be supplied.
    space_id : str, optional
-        watsonx.ai deployment space ID.  Falls back to the ``WATSONX_SPACE_ID``
-        environment variable when not provided.  Mutually exclusive with
+        watsonx.ai deployment space ID.  Explicit value takes precedence over
+        the ``WATSONX_SPACE_ID`` environment variable.  Mutually exclusive with
        ``project_id`` — exactly one must be supplied.
    url : str, optional
        watsonx.ai service URL.  Defaults to
@@ -100,8 +100,16 @@ class WatsonxReranker(Reranker):
        )

        # --- project_id / space_id (exactly one required) ---
-        project_id = self.project_id or os.environ.get("WATSONX_PROJECT_ID")
-        space_id = self.space_id or os.environ.get("WATSONX_SPACE_ID")
+        # Explicit field always wins; env vars are consulted only when neither
+        # was passed explicitly, so a stray WATSONX_SPACE_ID never overrides an
+        # explicit project_id and vice-versa.
+        project_id = self.project_id
+        space_id = self.space_id
+
+        if project_id is None and space_id is None:
+            # Neither was passed explicitly — fall back to env vars.
+            project_id = os.environ.get("WATSONX_PROJECT_ID")
+            space_id = os.environ.get("WATSONX_SPACE_ID")

        if project_id and space_id:
            raise ValueError("Provide either `project_id` or `space_id`, not both.")
@@ -7,10 +7,158 @@ import sys
 from typing import Callable, Iterator, Optional
 from lancedb.arrow import to_arrow
 import pyarrow as pa
+import pyarrow.compute as pc
 import pyarrow.dataset as ds

 from .pydantic import LanceModel

+# pyarrow's default scanner settings are tuned for narrow rows. For wide rows
+# (e.g. embedding columns) they buffer a huge read-ahead window in host memory
+# and can OOM the client during bulk ingestion. We size the scanner so the
+# estimated in-flight memory stays within a budget, while leaving narrow
+# datasets on pyarrow's defaults (no throughput regression).
+_SCAN_MEMORY_BUDGET_BYTES = 1024 * 1024 * 1024  # ~1 GiB in-flight target
+_TARGET_BATCH_BYTES = 16 * 1024 * 1024  # ~16 MiB per batch
+_MIN_BATCH_ROWS = 512
+# pyarrow defaults (see arrow/dataset ScanOptions); we never exceed these.
+_PA_DEFAULT_BATCH_ROWS = 131_072
+_PA_DEFAULT_BATCH_READAHEAD = 16
+_PA_DEFAULT_FRAGMENT_READAHEAD = 4
+# Read-ahead used for wide rows. pyarrow reads a whole parquet row group at a
+# time and keeps `batch_readahead` of them resident, so read-ahead depth (not
+# batch size) dominates peak memory for wide data; keep both small but leave a
+# little prefetch for throughput. Tuned empirically against embedding datasets.
+_WIDE_BATCH_READAHEAD = 2
+_WIDE_FRAGMENT_READAHEAD = 1
+# Estimate for variable-width columns (string/binary/list) whose true width is
+# unknown from the schema alone. Only needs to be large enough to flag "wide".
+_VARIABLE_WIDTH_ESTIMATE = 128
+# Rows peeked from a rescannable source to refine the list-length guess for
+# variable-length list columns (e.g. embeddings stored as `list<float32>`
+# instead of `list<float32, N>`), whose per-row width the schema can't tell us.
+_SAMPLE_ROWS = 10
+
+
+def _observed_list_length(sample: pa.ChunkedArray) -> Optional[int]:
+    """Average element count per row observed in a list/large_list sample."""
+    if len(sample) == 0:
+        return None
+    mean = pc.mean(pc.list_value_length(sample)).as_py()
+    return None if mean is None else max(1, round(mean))
+
+
+def _estimate_field_width(
+    dtype: pa.DataType, sample: Optional[pa.ChunkedArray] = None
+) -> int:
+    if pa.types.is_fixed_size_list(dtype):
+        return dtype.list_size * _estimate_field_width(dtype.value_type)
+    if pa.types.is_struct(dtype):
+        return sum(
+            _estimate_field_width(
+                dtype.field(i).type,
+                pc.struct_field(sample, [i]) if sample is not None else None,
+            )
+            for i in range(dtype.num_fields)
+        )
+    if pa.types.is_dictionary(dtype):
+        return _estimate_field_width(dtype.value_type)
+    if pa.types.is_fixed_size_binary(dtype):
+        return dtype.byte_width
+    if pa.types.is_boolean(dtype):
+        return 1
+    if (pa.types.is_list(dtype) or pa.types.is_large_list(dtype)) and (
+        sample is not None
+    ):
+        observed_length = _observed_list_length(sample)
+        if observed_length is not None:
+            return observed_length * _estimate_field_width(dtype.value_type)
+    # Fixed-width scalars (ints, floats, temporal, decimal) expose bit_width;
+    # variable-width types (string, binary, list, map, ...) raise ValueError.
+    try:
+        return max(1, dtype.bit_width // 8)
+    except (ValueError, AttributeError):
+        return _VARIABLE_WIDTH_ESTIMATE
+
+
+def _estimate_bytes_per_row(
+    schema: pa.Schema, sample: Optional[pa.Table] = None
+) -> int:
+    return max(
+        1,
+        sum(
+            _estimate_field_width(
+                field.type, sample.column(field.name) if sample is not None else None
+            )
+            for field in schema
+        ),
+    )
+
+
+def _sample_head(head: Callable[..., pa.Table]) -> Optional[pa.Table]:
+    """Best-effort peek at a few rows to refine the bytes-per-row estimate.
+
+    Uses a tight batch size and no read-ahead so the peek itself can't trigger
+    the wide-row memory blowup this module exists to avoid. Returns None (fall
+    back to the schema-only estimate) if sampling isn't possible for any
+    reason, e.g. an empty dataset.
+    """
+    try:
+        sample = head(
+            _SAMPLE_ROWS,
+            batch_size=_SAMPLE_ROWS,
+            batch_readahead=1,
+            fragment_readahead=1,
+        )
+    except Exception:
+        return None
+    return sample if sample.num_rows > 0 else None
+
+
+def _bounded_scanner_kwargs(
+    schema: pa.Schema, sample: Optional[pa.Table] = None
+) -> dict:
+    """Scanner kwargs that cap in-flight memory for wide rows.
+
+    Narrow datasets keep pyarrow's defaults unchanged (no throughput
+    regression). For wide rows (e.g. embedding columns) pyarrow's default
+    read-ahead buffers many large batches/row-groups at once, which can OOM the
+    client during bulk ingestion, so we shrink the batch size and read-ahead to
+    keep the estimated in-flight memory near the budget.
+
+    Read-ahead (not just batch size) has to drop: pyarrow reads a whole parquet
+    row group at a time and keeps `batch_readahead`/`fragment_readahead` of them
+    resident, so a small batch size alone still pins large row-group buffers.
+
+    `sample`, if given, is a small (see `_SAMPLE_ROWS`) table of rows from the
+    source used to refine the estimate for variable-length list columns (e.g.
+    embeddings stored without a fixed size), whose width the schema alone
+    can't tell us.
+    """
+    bytes_per_row = _estimate_bytes_per_row(schema, sample)
+
+    # If pyarrow's defaults already stay within budget, leave them alone so
+    # narrow datasets keep their throughput. A "unit" of in-flight memory is one
+    # default-sized batch, held `batch_readahead + fragment_readahead` deep.
+    default_in_flight = (
+        _PA_DEFAULT_BATCH_ROWS
+        * bytes_per_row
+        * (_PA_DEFAULT_BATCH_READAHEAD + _PA_DEFAULT_FRAGMENT_READAHEAD)
+    )
+    if default_in_flight <= _SCAN_MEMORY_BUDGET_BYTES:
+        return {}
+
+    # Wide rows: cap batch bytes and pull read-ahead down so only a couple of
+    # large row-group buffers are resident at once.
+    batch_size = min(
+        _PA_DEFAULT_BATCH_ROWS,
+        max(_MIN_BATCH_ROWS, _TARGET_BATCH_BYTES // bytes_per_row),
+    )
+    return {
+        "batch_size": batch_size,
+        "batch_readahead": _WIDE_BATCH_READAHEAD,
+        "fragment_readahead": _WIDE_FRAGMENT_READAHEAD,
+    }
+

@dataclass
 class Scannable:
@@ -56,10 +204,12 @@ def _from_table(data: pa.Table) -> Scannable:

@to_scannable.register(ds.Dataset)
 def _from_dataset(data: ds.Dataset) -> Scannable:
+    sample = _sample_head(data.head)
+    scanner_kwargs = _bounded_scanner_kwargs(data.schema, sample)
    return Scannable(
        schema=data.schema,
        num_rows=data.count_rows(),
-        reader=lambda: data.scanner().to_reader(),
+        reader=lambda: data.scanner(**scanner_kwargs).to_reader(),
    )


@@ -206,10 +356,12 @@ def _register_optional_converters():

        @to_scannable.register(lance.LanceDataset)
        def _from_lance(data: lance.LanceDataset) -> Scannable:
+            sample = _sample_head(data.head)
+            scanner_kwargs = _bounded_scanner_kwargs(data.schema, sample)
            return Scannable(
                schema=data.schema,
                num_rows=data.count_rows(),
-                reader=lambda: data.scanner().to_reader(),
+                reader=lambda: data.scanner(**scanner_kwargs).to_reader(),
            )


@@ -73,6 +73,7 @@ from .expr import Expr
 from .merge import LanceMergeInsertBuilder
 from .pydantic import LanceModel, model_to_dict
 from .query import (
+    AnalyzePlanDistributedMetrics,
    AsyncFTSQuery,
    AsyncHybridQuery,
    AsyncQuery,
@@ -1105,6 +1106,7 @@ class Table(ABC):
        ngram_min_length: int = 3,
        ngram_max_length: int = 3,
        prefix_only: bool = False,
+        block_size: int = 128,
        wait_timeout: Optional[timedelta] = None,
        name: Optional[str] = None,
    ):
@@ -1176,6 +1178,10 @@ class Table(ABC):
            The maximum length of an n-gram.
        prefix_only: bool, default False
            Whether to only index the prefix of the token for ngram tokenizer.
+        block_size: int, default 128
+            The number of documents per compressed posting block. Must be 128
+            or 256. A value of 256 uses the experimental FTS V3 format and
+            may introduce breaking changes.
        wait_timeout: timedelta, optional
            The timeout to wait if indexing is asynchronous.
        name: str, optional
@@ -1198,6 +1204,7 @@ class Table(ABC):
        on_bad_vectors: OnBadVectorsType = "error",
        fill_value: float = 0.0,
        progress: Optional[Union[bool, Callable, Any]] = None,
+        write_parallelism: Optional[int] = None,
    ) -> AddResult:
        """Add more data to the [Table](Table).

@@ -1243,6 +1250,13 @@ class Table(ABC):
                with tqdm() as pbar:
                    table.add(data, progress=pbar)

+        write_parallelism: int, optional
+            Number of partitions to write in parallel. Higher values increase
+            throughput but also peak memory use, since each partition buffers
+            data in flight. Defaults to an estimate based on the data size,
+            capped at the number of CPU cores. Lower this if bulk ingestion is
+            using too much memory.
+
        Returns
        -------
        AddResult
@@ -1552,7 +1566,12 @@ class Table(ABC):
    def _explain_plan(self, query: Query, verbose: Optional[bool] = False) -> str: ...

    @abstractmethod
-    def _analyze_plan(self, query: Query) -> str: ...
+    def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str: ...

    @abstractmethod
    def _output_schema(self, query: Query) -> pa.Schema: ...
@@ -2189,7 +2208,7 @@ class LanceTable(Table):
        namespace_client = self._namespace_client
        if namespace_client is None:
            conn_uri = getattr(self._conn, "uri", "")
-            if get_uri_scheme(conn_uri) == "namespace":
+            if get_uri_scheme(conn_uri) == "namespace" or self._namespace_path:
                namespace_client = self._conn.namespace_client()
                self._namespace_client = namespace_client

@@ -3012,6 +3031,7 @@ class LanceTable(Table):
        ngram_min_length: int = 3,
        ngram_max_length: int = 3,
        prefix_only: bool = False,
+        block_size: int = 128,
        name: Optional[str] = None,
    ):
        """Create a full-text search index on a column.
@@ -3061,9 +3081,7 @@ class LanceTable(Table):
        else:
            tokenizer_configs = self.infer_tokenizer_configs(tokenizer_name)

-        config = FTS(
-            **tokenizer_configs,
-        )
+        config = FTS(block_size=block_size, **tokenizer_configs)

        try:
            LOOP.run(
@@ -3152,6 +3170,7 @@ class LanceTable(Table):
        on_bad_vectors: OnBadVectorsType = "error",
        fill_value: float = 0.0,
        progress: Optional[Union[bool, Callable, Any]] = None,
+        write_parallelism: Optional[int] = None,
    ) -> AddResult:
        """Add data to the table.
        If vector columns are missing and the table
@@ -3173,6 +3192,12 @@ class LanceTable(Table):
        progress: bool, callable, or tqdm-like, optional
            A callback or tqdm-compatible progress bar. See
            :meth:`Table.add` for details.
+        write_parallelism: int, optional
+            Number of partitions to write in parallel. Higher values increase
+            throughput but also peak memory use, since each partition buffers
+            data in flight. Defaults to an estimate based on the data size,
+            capped at the number of CPU cores. Lower this if bulk ingestion is
+            using too much memory.

        Returns
        -------
@@ -3188,6 +3213,7 @@ class LanceTable(Table):
                    on_bad_vectors=on_bad_vectors,
                    fill_value=fill_value,
                    progress=progress,
+                    write_parallelism=write_parallelism,
                )
            )
        finally:
@@ -3630,8 +3656,15 @@ class LanceTable(Table):
    def _explain_plan(self, query: Query, verbose: Optional[bool] = False) -> str:
        return LOOP.run(self._table._explain_plan(query, verbose))

-    def _analyze_plan(self, query: Query) -> str:
-        return LOOP.run(self._table._analyze_plan(query))
+    def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str:
+        return LOOP.run(
+            self._table._analyze_plan(query, distributed_metrics=distributed_metrics)
+        )

    def _output_schema(self, query: Query) -> pa.Schema:
        return LOOP.run(self._table._output_schema(query))
@@ -4923,6 +4956,7 @@ class AsyncTable:
        on_bad_vectors: Optional[OnBadVectorsType] = None,
        fill_value: Optional[float] = None,
        progress: Optional[Union[bool, Callable, Any]] = None,
+        write_parallelism: Optional[int] = None,
    ) -> AddResult:
        """Add more data to the [Table](Table).

@@ -4947,6 +4981,12 @@ class AsyncTable:
        progress: callable or tqdm-like, optional
            A callback or tqdm-compatible progress bar. See
            :meth:`Table.add` for details.
+        write_parallelism: int, optional
+            Number of partitions to write in parallel. Higher values increase
+            throughput but also peak memory use, since each partition buffers
+            data in flight. Defaults to an estimate based on the data size,
+            capped at the number of CPU cores. Lower this if bulk ingestion is
+            using too much memory.

        """
        schema = await self.schema()
@@ -4978,7 +5018,12 @@ class AsyncTable:
        data = to_scannable(data)
        progress, owns = _normalize_progress(progress)
        try:
-            return await self._inner.add(data, mode or "append", progress=progress)
+            return await self._inner.add(
+                data,
+                mode or "append",
+                progress=progress,
+                write_parallelism=write_parallelism,
+            )
        except RuntimeError as e:
            if "Cast error" in str(e):
                raise ValueError(e)
@@ -5390,10 +5435,15 @@ class AsyncTable:
        async_query = self._sync_query_to_async(query)
        return await async_query.explain_plan(verbose)

-    async def _analyze_plan(self, query: Query) -> str:
+    async def _analyze_plan(
+        self,
+        query: Query,
+        *,
+        distributed_metrics: AnalyzePlanDistributedMetrics = "aggregate",
+    ) -> str:
        # This method is used by the sync table
        async_query = self._sync_query_to_async(query)
-        return await async_query.analyze_plan()
+        return await async_query.analyze_plan(distributed_metrics)

    async def _output_schema(self, query: Query) -> pa.Schema:
        async_query = self._sync_query_to_async(query)
@@ -6221,6 +6271,24 @@ class Branches:
        """Delete a branch."""
        LOOP.run(self._table.branches.delete(name))

+    def diff(self, from_branch: str) -> Dict[str, Any]:
+        """Diff a branch against main."""
+        return LOOP.run(self._table.branches.diff(from_branch))
+
+    def merge(self, from_branch: str, dry_run: bool = False) -> Dict[str, Any]:
+        """Merge a branch into main, or dry-run.
+
+        Parameters
+        ----------
+        from_branch: str
+            Branch to merge from.
+        dry_run: bool, default False
+            When True, only preview. When False, attempt the merge.
+
+        A rejected merge returns ``status="rejected"`` instead of raising.
+        """
+        return LOOP.run(self._table.branches.merge(from_branch, dry_run))
+
    def _wrap(
        self, async_table: "AsyncTable", version: Optional[int] = None
    ) -> "Table":
@@ -6350,3 +6418,14 @@ class AsyncBranches:
    async def delete(self, name: str) -> None:
        """Delete a branch."""
        await self._table.branches.delete(name)
+
+    async def diff(self, from_branch: str) -> Dict[str, Any]:
+        """Diff a branch against main."""
+        return await self._table.branches.diff(from_branch)
+
+    async def merge(self, from_branch: str, dry_run: bool = False) -> Dict[str, Any]:
+        """Merge a branch into main, or dry-run.
+
+        A rejected merge returns ``status="rejected"`` instead of raising.
+        """
+        return await self._table.branches.merge(from_branch, dry_run)
@@ -307,13 +307,19 @@ def infer_vector_column_name(
        # FTS queries do not require a vector column
        return None

-    if query is not None or query_type == "hybrid":
-        try:
-            vector_column_name = inf_vector_column_query(
-                schema, dim=_query_vector_dim(query)
-            )
-        except Exception as e:
-            raise e
+    if query is None and query_type != "hybrid":
+        # No vector search was requested (e.g. a plain scan), so there's
+        # nothing to infer.
+        return None
+
+    vector_column_name = inf_vector_column_query(schema, dim=_query_vector_dim(query))
+
+    if vector_column_name is None:
+        raise ValueError(
+            "No vector column found in the schema. Please specify the "
+            "vector column name explicitly via the `vector_column_name` "
+            "parameter."
+        )

    return vector_column_name

@@ -13,6 +13,7 @@ import numpy as np
 import pandas as pd
 import pyarrow as pa
 import pytest
+from lance_namespace.errors import NamespaceNotEmptyError, TableNotFoundError
 from lancedb.pydantic import LanceModel, Vector


@@ -955,6 +956,47 @@ def test_local_namespace_operations(tmp_path):
    assert db.list_namespaces().namespaces == []


+def test_local_sync_namespace_uses_rust_without_python_client(tmp_path, monkeypatch):
+    """Sync local namespace operations should avoid the Python namespace client."""
+    db = lancedb.connect(tmp_path)
+
+    def fail_namespace_client():
+        raise AssertionError("Python namespace client should not be constructed")
+
+    monkeypatch.setattr(db, "namespace_client", fail_namespace_client)
+
+    db.create_namespace(["child"])
+    assert "child" in db.list_namespaces().namespaces
+
+    schema = pa.schema([pa.field("id", pa.int64())])
+    table = db.create_table("tbl", schema=schema, namespace_path=["child"])
+    assert table.namespace == ["child"]
+    assert "tbl" in db.table_names(namespace_path=["child"])
+    assert db.list_tables(namespace_path=["child"]).tables == ["tbl"]
+
+    opened = db.open_table("tbl", namespace_path=["child"])
+    assert opened.namespace == ["child"]
+
+    db.drop_table("tbl", namespace_path=["child"])
+    assert db.list_tables(namespace_path=["child"]).tables == []
+    db.drop_namespace(["child"])
+    assert db.list_namespaces().namespaces == []
+
+
+def test_local_sync_namespace_preserves_public_errors(tmp_path):
+    db = lancedb.connect(tmp_path)
+    db.create_namespace(["child"])
+    db.create_table(
+        "tbl", schema=pa.schema([pa.field("id", pa.int64())]), namespace_path=["child"]
+    )
+
+    with pytest.raises(TableNotFoundError, match="child\\$missing"):
+        db.open_table("missing", namespace_path=["child"])
+
+    with pytest.raises(NamespaceNotEmptyError):
+        db.drop_namespace(["child"])
+
+
 def test_create_namespace_invalid_mode_raises(tmp_path):
    """Unrecognized create namespace modes raise a clear error."""
    db = lancedb.connect(tmp_path)
@@ -226,6 +226,23 @@ def test_create_inverted_index(table, with_position):
    assert any(i.name == "custom_fts_index" for i in fts_indices)


+@pytest.mark.parametrize("block_size", [128, 256])
+def test_create_inverted_index_block_size(table, block_size):
+    table.create_index("text", config=FTS(block_size=block_size))
+
+    index = next(index for index in table.list_indices() if index.index_type == "FTS")
+    assert index.index_details["block_size"] == block_size
+    assert index.index_version == (2 if block_size == 128 else 3)
+
+    results = table.search("puppy").limit(5).to_list()
+    assert len(results) == 5
+
+
+def test_create_inverted_index_rejects_invalid_block_size(table):
+    with pytest.raises(ValueError, match="128 or 256"):
+        table.create_index("text", config=FTS(block_size=129))
+
+
 def test_search_fts(table):
    table.create_fts_index("text")
    results = table.search("puppy").select(["id", "text"]).limit(5).to_list()
@@ -196,18 +196,26 @@ async def test_analyze_plan(table: AsyncTable):
 def test_hybrid_phrase_query_is_preserved_in_analyze_plan():
    table = mock.Mock()
    analyzed_queries = []
-    table._analyze_plan.side_effect = lambda query: analyzed_queries.append(query) or ""
+    distributed_metric_modes = []
+
+    def capture_query(query, *, distributed_metrics="aggregate"):
+        analyzed_queries.append(query)
+        distributed_metric_modes.append(distributed_metrics)
+        return ""
+
+    table._analyze_plan.side_effect = capture_query

    (
        LanceHybridQueryBuilder(table)
        .vector([0.1, 0.2])
        .text("puppy runs")
        .phrase_query()
-        .analyze_plan()
+        .analyze_plan(distributed_metrics="full")
    )

    assert len(analyzed_queries) == 2
    assert analyzed_queries[1].full_text_query.query == '"puppy runs"'
+    assert distributed_metric_modes == ["full", "full"]


@pytest.fixture
@@ -134,8 +134,11 @@ def test_split_hash_with_discard(mem_db):
    )

    permutation_tbl = (
+        # Hash a high-cardinality column: "category" has only two distinct
+        # values, so whether anything is discarded would hinge on where those
+        # two hashes land rather than on the discard weight.
        permutation_builder(tbl)
-        .split_hash(["category"], [1, 1], discard_weight=2)  # Should discard ~50%
+        .split_hash(["id"], [1, 1], discard_weight=2)  # Should discard ~50%
        .execute()
    )

@@ -1133,3 +1136,61 @@ def test_take_offsets_empty_permutation(some_permutation: Permutation):
    result = some_permutation.take_offsets([])

    assert result == []
+
+
+def test_select_rowid(some_permutation: Permutation):
+    """Test that _rowid can be selected alongside regular columns."""
+    perm_with_rowid = some_permutation.select_columns(["_rowid", "id"])
+    assert "_rowid" in perm_with_rowid.column_names
+    batches = list(perm_with_rowid.iter(100, skip_last_batch=False))
+    for batch in batches:
+        assert "_rowid" in batch[0]
+
+
+def test_select_rowid_only(some_permutation: Permutation):
+    """Test that _rowid can be selected as the sole column."""
+    perm_rowid_only = some_permutation.select_columns(["_rowid"])
+    assert perm_rowid_only.column_names == ["_rowid"]
+    batches = list(perm_rowid_only.iter(100, skip_last_batch=False))
+    assert len(batches) > 0
+    for batch in batches:
+        assert list(batch[0].keys()) == ["_rowid"]
+
+
+def test_select_rowid_not_in_default(some_permutation: Permutation):
+    """Test that _rowid is NOT in the default column_names or schema."""
+    assert "_rowid" not in some_permutation.column_names
+    assert "_rowid" not in some_permutation.schema.names
+
+
+def test_select_rowid_identity_permutation(mem_db):
+    """Test that _rowid works with an identity permutation."""
+    tbl = mem_db.create_table(
+        "test_rowid_identity", pa.table({"id": range(10), "value": range(10)})
+    )
+    perm = Permutation.identity(tbl)
+    perm_with_rowid = perm.select_columns(["_rowid", "id"])
+    batches = list(perm_with_rowid.iter(10, skip_last_batch=False))
+    assert len(batches) == 1
+    assert "_rowid" in batches[0][0]
+
+
+def test_rename_rowid(some_permutation: Permutation):
+    """Test that _rowid can be selected and then renamed."""
+    perm_with_rowid = some_permutation.select_columns(["_rowid", "id"])
+    renamed = perm_with_rowid.rename_column("_rowid", "my_row_id")
+    assert "my_row_id" in renamed.column_names
+    assert "_rowid" not in renamed.column_names
+    batches = list(renamed.iter(100, skip_last_batch=False))
+    for batch in batches:
+        assert "my_row_id" in batch[0]
+        assert "_rowid" not in batch[0]
+
+
+def test_remove_rowid_after_select(some_permutation: Permutation):
+    """Test that _rowid can be selected and then removed."""
+    perm_with_rowid = some_permutation.select_columns(["_rowid", "id"])
+    assert "_rowid" in perm_with_rowid.column_names
+    perm_without_rowid = perm_with_rowid.remove_columns(["_rowid"])
+    assert "_rowid" not in perm_without_rowid.column_names
+    assert perm_without_rowid.column_names == ["id"]
@@ -1273,7 +1273,7 @@ async def test_explain_plan_fts(table_async: AsyncTable):
    query = await table_async.search("dog", query_type="fts", fts_columns="text")
    plan = await query.explain_plan()
    # Should show FTS details (issue #2465 is now fixed)
-    assert "MatchQuery: column=text, query=dog" in plan
+    assert "MatchQuery: column=text, query=[dog]" in plan
    assert "GlobalLimitExec" in plan  # Default limit

    # Test FTS query with limit
@@ -1281,7 +1281,7 @@ async def test_explain_plan_fts(table_async: AsyncTable):
        "dog", query_type="fts", fts_columns="text"
    )
    plan_with_limit = await query_with_limit.limit(1).explain_plan()
-    assert "MatchQuery: column=text, query=dog" in plan_with_limit
+    assert "MatchQuery: column=text, query=[dog]" in plan_with_limit
    assert "GlobalLimitExec: skip=0, fetch=1" in plan_with_limit

    # Test FTS query with offset and limit
@@ -1289,7 +1289,7 @@ async def test_explain_plan_fts(table_async: AsyncTable):
        "dog", query_type="fts", fts_columns="text"
    )
    plan_with_offset = await query_with_offset.offset(1).limit(1).explain_plan()
-    assert "MatchQuery: column=text, query=dog" in plan_with_offset
+    assert "MatchQuery: column=text, query=[dog]" in plan_with_offset
    assert "GlobalLimitExec: skip=1, fetch=1" in plan_with_offset


@@ -1333,7 +1333,7 @@ async def test_explain_plan_with_filters(table_async: AsyncTable):
        "dog", query_type="fts", fts_columns="text"
    )
    plan_fts_filter = await query_fts_filter.where("id = 1").explain_plan()
-    assert "MatchQuery: column=text, query=dog" in plan_fts_filter
+    assert "MatchQuery: column=text, query=[dog]" in plan_fts_filter
    assert "LanceRead" in plan_fts_filter
    assert "full_filter=id = Int64(1)" in plan_fts_filter  # Should show filter details

@@ -236,6 +236,65 @@ def test_remote_table_branches_sync():
        table.branches.delete("exp")


+def test_remote_table_branch_merge_defaults_to_execute():
+    merge_bodies = []
+    diff = {
+        "fromBranch": "exp",
+        "parentVersion": 1,
+        "mainVersion": 2,
+        "branchVersion": 3,
+        "baseMoved": False,
+        "rowCountMain": 3,
+        "rowCountBranch": 3,
+        "rowSummary": {
+            "unchanged": 3,
+            "newOnBase": 0,
+            "newOnBranch": 0,
+            "staleRecompute": 0,
+            "inputsChanged": 0,
+            "deltaAvailable": False,
+        },
+        "addedColumns": [],
+        "removedColumns": [],
+        "changedColumns": [],
+        "addedIndexes": [],
+        "removedIndexes": [],
+        "mergeable": True,
+        "mergeBlockers": [],
+    }
+
+    def handler(request):
+        if request.path.endswith("/describe/"):
+            status = 200
+            body = {"version": 2, "schema": {"fields": []}}
+        else:
+            content_len = int(request.headers.get("Content-Length"))
+            request_body = json.loads(request.rfile.read(content_len))
+            merge_bodies.append(request_body)
+            dry_run = request_body["dry_run"]
+            status = 200 if dry_run else 409
+            body = {
+                "status": "ready" if dry_run else "rejected",
+                "diff": diff,
+                "preview": {"promotedColumns": []},
+            }
+
+        request.send_response(status)
+        request.send_header("Content-Type", "application/json")
+        request.end_headers()
+        request.wfile.write(json.dumps(body).encode())
+
+    with mock_lancedb_connection(handler) as db:
+        branches = db.open_table("test").branches
+        assert branches.merge("exp")["status"] == "rejected"
+        assert branches.merge("exp", dry_run=True)["status"] == "ready"
+
+    assert merge_bodies == [
+        {"from_branch": "exp", "dry_run": False},
+        {"from_branch": "exp", "dry_run": True},
+    ]
+
+
@pytest.mark.asyncio
 async def test_async_remote_open_table_branch_and_version():
    async with mock_lancedb_connection_async(_branch_open_handler) as db:
@@ -709,7 +768,10 @@ def test_table_create_indices():
        # Test create_fts_index with custom name (legacy method)
        with pytest.warns(DeprecationWarning, match="create_fts_index"):
            table.create_fts_index(
-                "text", wait_timeout=timedelta(seconds=2), name="custom_fts_idx"
+                "text",
+                wait_timeout=timedelta(seconds=2),
+                block_size=256,
+                name="custom_fts_idx",
            )

        # Test create_index with custom name (legacy form: vector_column_name kwarg)
@@ -732,6 +794,7 @@ def test_table_create_indices():
        fts_req = received_requests[1]
        assert "name" in fts_req
        assert fts_req["name"] == "custom_fts_idx"
+        assert fts_req["block_size"] == 256

        # Check vector index request has custom name
        vector_req = received_requests[2]
@@ -817,7 +880,7 @@ def test_remote_create_index_new_api():
            _warnings.simplefilter("error", DeprecationWarning)
            table.create_index("vector", config=IvfPq(distance_type="l2"))
            table.create_index("category", config=BTree())
-            table.create_index("text", config=FTS())
+            table.create_index("text", config=FTS(block_size=256))
            # IvfRq via new API
            table.create_index("vector", config=IvfRq(distance_type="l2"))

@@ -837,6 +900,7 @@ def test_remote_create_index_new_api():
            "vector",
            "vector",
        ]
+        assert received_requests[2]["block_size"] == 256


 def test_table_wait_for_index_timeout():
@@ -644,6 +644,21 @@ def test_cross_encoder_reranker_return_all(tmp_path):
    assert "_distance" in result.column_names


+def test_answerdotai_reranker_return_all(tmp_path):
+    pytest.importorskip("rerankers")
+    reranker = AnswerdotaiRerankers(return_score="all")
+    table, schema = get_test_table(tmp_path)
+    query = "single player experience"
+    result = (
+        table.search(query, query_type="hybrid", vector_column_name="vector")
+        .rerank(reranker=reranker)
+        .to_arrow()
+    )
+    assert "_relevance_score" in result.column_names
+    assert "_score" in result.column_names
+    assert "_distance" in result.column_names
+
+
 # ---------------------------------------------------------------------------
 # Regression tests for LinearCombinationReranker scoring bugs (issue #3154)
 # ---------------------------------------------------------------------------
@@ -0,0 +1,183 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.dataset as ds
+import pyarrow.parquet as pq
+
+from lancedb.scannable import (
+    _PA_DEFAULT_BATCH_ROWS,
+    _SAMPLE_ROWS,
+    _VARIABLE_WIDTH_ESTIMATE,
+    _WIDE_BATCH_READAHEAD,
+    _WIDE_FRAGMENT_READAHEAD,
+    _bounded_scanner_kwargs,
+    _estimate_bytes_per_row,
+    _sample_head,
+    to_scannable,
+)
+
+
+def test_estimate_bytes_per_row():
+    # fixed-width scalars
+    assert _estimate_bytes_per_row(pa.schema([("a", pa.int64())])) == 8
+    assert (
+        _estimate_bytes_per_row(pa.schema([("a", pa.int32()), ("b", pa.float64())]))
+        == 12
+    )
+    assert _estimate_bytes_per_row(pa.schema([("a", pa.bool_())])) == 1
+    # fixed-size list (embedding) dominates
+    assert (
+        _estimate_bytes_per_row(pa.schema([("v", pa.list_(pa.float32(), 768))]))
+        == 768 * 4
+    )
+    # struct sums its children
+    struct = pa.struct([("x", pa.int32()), ("y", pa.int32())])
+    assert _estimate_bytes_per_row(pa.schema([("s", struct)])) == 8
+    # variable-width columns get a flat estimate, not zero
+    assert _estimate_bytes_per_row(pa.schema([("s", pa.string())])) > 0
+
+
+def test_estimate_bytes_per_row_uses_sample_for_variable_length_lists():
+    # A vector column without a fixed size (e.g. `list<float32>` instead of
+    # `list<float32, 768>`) has no width the schema alone can tell us.
+    schema = pa.schema([("v", pa.list_(pa.float32()))])
+    assert _estimate_bytes_per_row(schema) == _VARIABLE_WIDTH_ESTIMATE
+
+    sample = pa.table({"v": pa.array([[0.0] * 768], type=pa.list_(pa.float32()))})
+    assert _estimate_bytes_per_row(schema, sample) == 768 * 4
+
+
+def test_estimate_bytes_per_row_sample_ignores_missing_or_null_lists():
+    schema = pa.schema([("v", pa.list_(pa.float32()))])
+    # an all-null sample column can't tell us anything either
+    sample = pa.table({"v": pa.array([None], type=pa.list_(pa.float32()))})
+    assert _estimate_bytes_per_row(schema, sample) == _VARIABLE_WIDTH_ESTIMATE
+
+
+def test_bounded_scanner_kwargs_narrow_uses_defaults():
+    # Narrow rows stay on pyarrow defaults (empty kwargs) so throughput is
+    # unchanged.
+    for schema in [
+        pa.schema([("a", pa.int64()), ("b", pa.int32()), ("c", pa.string())]),
+        pa.schema([("a", pa.int64()), ("t", pa.string()), ("u", pa.string())]),
+        # a 100-dim float32 vector is still under the per-row budget
+        pa.schema([("id", pa.int64()), ("v", pa.list_(pa.float32(), 100))]),
+    ]:
+        assert _bounded_scanner_kwargs(schema) == {}, schema
+
+
+def test_bounded_scanner_kwargs_wide_is_bounded():
+    schema = pa.schema(
+        [
+            ("uid", pa.string()),
+            ("img", pa.list_(pa.float32(), 768)),
+            ("txt", pa.list_(pa.float32(), 768)),
+        ]
+    )
+    kwargs = _bounded_scanner_kwargs(schema)
+    assert kwargs, "wide schema should be throttled"
+    assert kwargs["batch_readahead"] == _WIDE_BATCH_READAHEAD
+    assert kwargs["fragment_readahead"] == _WIDE_FRAGMENT_READAHEAD
+    # batch is capped well below the pyarrow default for wide rows
+    assert kwargs["batch_size"] < _PA_DEFAULT_BATCH_ROWS
+
+
+def test_bounded_scanner_kwargs_variable_length_list_needs_sample():
+    # Without a sample, a variable-length (not fixed-size) vector column looks
+    # narrow because its true width is unknown from the schema alone.
+    schema = pa.schema([("uid", pa.string()), ("vec", pa.list_(pa.float32()))])
+    assert _bounded_scanner_kwargs(schema) == {}
+
+    sample = pa.table(
+        {
+            "uid": pa.array(["a"]),
+            "vec": pa.array([[0.0] * 768], type=pa.list_(pa.float32())),
+        }
+    )
+    kwargs = _bounded_scanner_kwargs(schema, sample)
+    assert kwargs, "sample should reveal the wide vector column"
+    assert kwargs["batch_size"] < _PA_DEFAULT_BATCH_ROWS
+
+
+def _write_wide_dataset(
+    path, *, files=2, rows_per_file=20_000, dim=768, fixed_size=True
+):
+    rng = np.random.default_rng(0)
+    for i in range(files):
+        emb = rng.standard_normal((rows_per_file, dim), dtype=np.float32)
+        vec_type = pa.list_(pa.float32(), dim) if fixed_size else pa.list_(pa.float32())
+        vec_array = (
+            pa.FixedSizeListArray.from_arrays(pa.array(emb.reshape(-1)), dim)
+            if fixed_size
+            else pa.array(emb.tolist(), type=vec_type)
+        )
+        table = pa.table(
+            {
+                "uid": pa.array([f"{i}_{j}" for j in range(rows_per_file)]),
+                "vec": vec_array,
+            }
+        )
+        pq.write_table(table, f"{path}/part-{i}.parquet")
+
+
+def test_dataset_reader_respects_bounded_batch_size(tmp_path):
+    # The Dataset path should stream small batches for wide rows, not pyarrow's
+    # 131072-row default, and still return every row.
+    _write_wide_dataset(str(tmp_path))
+    dataset = ds.dataset(str(tmp_path), format="parquet")
+
+    expected = _bounded_scanner_kwargs(dataset.schema)["batch_size"]
+
+    scannable = to_scannable(dataset)
+    assert scannable.rescannable
+    assert scannable.num_rows == 40_000
+
+    total = 0
+    for batch in scannable.reader():
+        assert batch.num_rows <= expected
+        total += batch.num_rows
+    assert total == 40_000
+
+    # factory can be called again (rescannable)
+    assert sum(b.num_rows for b in scannable.reader()) == 40_000
+
+
+def test_dataset_reader_samples_variable_length_list_width(tmp_path):
+    # A vector column stored without a fixed size (e.g. produced by tools that
+    # don't tag list columns with their length) is invisible to the
+    # schema-only estimate, so `to_scannable` must peek a sample of rows to
+    # detect that it's wide and bound the scanner accordingly.
+    _write_wide_dataset(str(tmp_path), fixed_size=False)
+    dataset = ds.dataset(str(tmp_path), format="parquet")
+
+    schema_only_kwargs = _bounded_scanner_kwargs(dataset.schema)
+    assert schema_only_kwargs == {}, "schema alone can't see the list width"
+
+    scannable = to_scannable(dataset)
+    assert scannable.rescannable
+    assert scannable.num_rows == 40_000
+
+    total = 0
+    for batch in scannable.reader():
+        assert batch.num_rows < _PA_DEFAULT_BATCH_ROWS
+        total += batch.num_rows
+    assert total == 40_000
+
+
+def test_sample_head_is_bounded_rows(tmp_path):
+    # The peek itself must not read the whole dataset.
+    _write_wide_dataset(str(tmp_path), files=1, rows_per_file=1000, fixed_size=False)
+    dataset = ds.dataset(str(tmp_path), format="parquet")
+
+    sample = _sample_head(dataset.head)
+    assert sample.num_rows == _SAMPLE_ROWS
+
+
+def test_sample_head_returns_none_for_empty_dataset(tmp_path):
+    table = pa.table({"v": pa.array([], type=pa.list_(pa.float32()))})
+    pq.write_table(table, f"{tmp_path}/empty.parquet")
+    dataset = ds.dataset(str(tmp_path), format="parquet")
+
+    assert _sample_head(dataset.head) is None
@@ -434,6 +434,29 @@ def test_add(mem_db: DBConnection):
    _add(table, schema)


+def test_add_write_parallelism(mem_db: DBConnection):
+    schema = pa.schema([pa.field("id", pa.int64())])
+    table = mem_db.create_table("test", schema=schema)
+
+    data = pa.table({"id": list(range(1000))}, schema=schema)
+    table.add(data, write_parallelism=4)
+    assert len(table) == 1000
+
+    # invalid parallelism is rejected
+    with pytest.raises(ValueError, match="write_parallelism"):
+        table.add(data, write_parallelism=0)
+
+
+@pytest.mark.asyncio
+async def test_add_write_parallelism_async(mem_db_async: AsyncConnection):
+    schema = pa.schema([pa.field("id", pa.int64())])
+    table = await mem_db_async.create_table("test", schema=schema)
+
+    data = pa.table({"id": list(range(1000))}, schema=schema)
+    await table.add(data, write_parallelism=4)
+    assert await table.count_rows() == 1000
+
+
 def test_add_struct(mem_db: DBConnection):
    # https://github.com/lancedb/lancedb/issues/2114
    schema = pa.schema(
--- a/Show More
+++ b/Show More