chore: point lance dependency at jack branch

fix(udf): JobHandle.wait() terminates on failed jobs
wait() (sync + async) only stopped on finished/stale/committed, so a job the server already reported as state=failed was polled until the (default 3600s) timeout, then raised a misleading TimeoutError instead of the real cause. A doomed backfill -- e.g. a multi-column REFRESH COLUMN of a scalar UDF -- hung the client even though get_job surfaced the failure within ~3s. Add a terminal failed branch that raises JobFailedError carrying the server error, exported from the package. Verified end-to-end against the cluster: raises in 3.6s instead of hanging. Unit-tested with a mock conn (sync+async, failure + success + committed paths). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-27 16:38:31 +00:00 · 2026-06-29 22:46:07 -07:00 · 2026-06-29 22:35:35 -07:00 · 2026-06-29 22:35:35 -07:00 · 2026-06-29 22:35:35 -07:00 · 2026-06-29 22:35:34 -07:00
116 changed files with 4311 additions and 7258 deletions
@@ -1,137 +0,0 @@
---
-name: lancedb-branch-ops
-description: Branch management for LanceDB tables via the REST API. Use this skill whenever someone wants to create, delete, list, or switch branches on a LanceDB table — or needs to make sure a write (metadata update, index build, etc.) lands on a specific branch instead of main. Invoke it even without the word "branch" if context makes clear they want an experimental copy of a table, want to isolate changes, or want to confirm a mutation didn't touch main. Covers: branches/list, branches/create, branches/delete, and passing "branch" in describe/update_field_metadata/create_index to target a non-main version.
---
-
-## Goal
-
-Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main.
-
-## Step 0: Establish the connection
-
-Use the `lancedb-connect` skill to resolve the base URL and auth headers (`x-api-key`, `x-lancedb-database`). Skip this only if the connection is already known from the current conversation.
-
-All examples below use `{base_url}` — substitute the resolved endpoint and include the auth headers on every request.
-
-## The branch model (important)
-
-LanceDB branches are named snapshots that diverge from the table's current state at creation time. There is **no checkout command** — you never switch the whole table to a branch. Instead, you **pass `"branch": "<name>"` in the request body** of any operation to target that branch. Omitting the key (or sending an empty body) always targets main.
-
-`branches/list` returns only non-main branches. Main always exists and is not listed.
-
-## List branches
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/list
-Content-Type: application/json
-
-{}
-```
-
-Response:
-```json
-{
-  "branches": {
-    "experiment-reindex": {"parentVersion": 1, "createAt": 1782506085, "manifestSize": 1029}
-  }
-}
-```
-
-If `branches` is `{}`, the table has no branches besides main.
-
-## Create a branch
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/create
-Content-Type: application/json
-
-{"name": "experiment-reindex"}
-```
-
-HTTP 200 with `{}` body = success. The branch is created off the table's current state on main.
-
-Verify by calling `branches/list` and confirming the new name appears.
-
-## Delete a branch
-
-```http
-POST {base_url}/v1/table/{table_id}/branches/delete
-Content-Type: application/json
-
-{"name": "stale-2024"}
-```
-
-HTTP 200 with `{}` body = success. Only the branch pointer is removed — main and all row data remain intact.
-
-Verify by calling `branches/list` (name gone) and `describe` with no branch param (main still responds).
-
-## Operate on a specific branch
-
-Pass `"branch": "<name>"` in the body of any operation to scope it to that branch:
-
-**Read schema on a branch:**
-```http
-POST {base_url}/v1/table/{table_id}/describe
-Content-Type: application/json
-
-{"branch": "wip-branch"}
-```
-
-**Write metadata to a branch (not main):**
-```http
-POST {base_url}/v1/table/{table_id}/update_field_metadata
-Content-Type: application/json
-
-{
-  "branch": "wip-branch",
-  "updates": [
-    {
-      "path": "category",
-      "metadata": {"lancedb:description": "Product category label."},
-      "replace": false
-    }
-  ]
-}
-```
-
-**Build an index on a branch:**
-```http
-POST {base_url}/v1/table/{table_id}/create_index
-Content-Type: application/json
-
-{
-  "branch": "wip-branch",
-  "column": "category",
-  "index_type": "BTREE"
-}
-```
-
-## Verifying isolation
-
-After writing to a branch, always confirm the change did NOT land on main:
-
-```bash
-# Should show the new metadata
-curl -s -X POST {base_url}/v1/table/{table_id}/describe \
-  -H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
-  -H "content-type: application/json" \
-  -d '{"branch": "wip-branch"}'
-
-# Should NOT show the new metadata
-curl -s -X POST {base_url}/v1/table/{table_id}/describe \
-  -H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
-  -H "content-type: application/json" \
-  -d '{}'
-```
-
-## Quick reference
-
-| Goal | Endpoint | Body |
-|------|----------|------|
-| List all branches | `branches/list` | `{}` |
-| Create a branch | `branches/create` | `{"name": "..."}` |
-| Delete a branch | `branches/delete` | `{"name": "..."}` |
-| Read schema on branch | `describe` | `{"branch": "..."}` |
-| Write metadata on branch | `update_field_metadata` | `{"branch": "...", "updates": [...]}` |
-| Build index on branch | `create_index` | `{"branch": "...", "column": ..., "index_type": ...}` |
-| Target main (default) | any endpoint | omit `"branch"` key |
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.32.0-beta.0"
+current_version = "0.31.0-beta.4"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
@@ -1,21 +0,0 @@
-# CODEOWNERS
-#
-# These owners will be the default owners for everything in the repo.
-# They will be requested for review when someone opens a pull request.
-#
-# See https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
-
-# Default owners for everything
-* @jackye1995 @wjones127
-
-# Release and publish workflows — changes here can affect supply chain security
-/.github/workflows/ @jackye1995 @wjones127 @Xuanwo
-
-# Remote client and auth — sensitive networking and auth code
-/rust/lancedb/src/remote/ @jackye1995 @wjones127
-
-# Python FFI boundary
-/python/src/ @jackye1995 @wjones127 @AyushExel
-
-# NodeJS FFI boundary
-/nodejs/src/ @jackye1995 @wjones127
@@ -34,16 +34,15 @@ runs:
        maturin-version: "1.12.4"
        command: build
        working-directory: python
-        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc"
+        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"
        target: x86_64-unknown-linux-gnu
        manylinux: ${{ inputs.manylinux }}
        args: ${{ inputs.args }}
        before-script-linux: |
          set -e
-          curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-x86_64.zip -o /tmp/protoc.zip
-          unzip /tmp/protoc.zip -d /usr/local
-          rm /tmp/protoc.zip
-          /usr/local/bin/protoc --version
+          curl -L https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-$(uname -m).zip > /tmp/protoc.zip \
+            && unzip /tmp/protoc.zip -d /usr/local \
+            && rm /tmp/protoc.zip
    - name: Build Arm Manylinux Wheel
      if: ${{ inputs.arm-build == 'true' }}
      uses: PyO3/maturin-action@v1
@@ -51,14 +50,13 @@ runs:
        maturin-version: "1.12.4"
        command: build
        working-directory: python
-        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc"
+        docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"
        target: aarch64-unknown-linux-gnu
        manylinux: ${{ inputs.manylinux }}
        args: ${{ inputs.args }}
        before-script-linux: |
          set -e
-          yum install -y clang
-          curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-aarch_64.zip -o /tmp/protoc.zip
-          unzip /tmp/protoc.zip -d /usr/local
-          rm /tmp/protoc.zip
-          /usr/local/bin/protoc --version
+          yum install -y clang \
+            && curl -L https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-aarch_64.zip > /tmp/protoc.zip \
+            && unzip /tmp/protoc.zip -d /usr/local \
+            && rm /tmp/protoc.zip
@@ -25,7 +25,7 @@ jobs:
    # Only runs on tags that matches the make-release action
    if: startsWith(github.ref, 'refs/tags/v')
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - uses: Swatinem/rust-cache@v2
        with:
          workspaces: rust
@@ -47,7 +47,7 @@ jobs:
      contents: read
      issues: write
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - uses: ./.github/actions/create-failure-issue
        with:
          job-results: ${{ toJSON(needs) }}
@@ -36,14 +36,14 @@ jobs:
          echo "guidelines = ${{ inputs.guidelines }}"

      - name: Checkout Repo
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
        with:
          ref: ${{ inputs.branch }}
          fetch-depth: 0
          persist-credentials: true

      - name: Set up Node.js
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v4
        with:
          # pnpm 11 (used by the nodejs install step below) requires
          # Node >= 22.13; use 24 since 22 hits EOL in October.
@@ -82,7 +82,7 @@ jobs:
          cache: maven

      - name: Setup pnpm
-        uses: pnpm/action-setup@v6
+        uses: pnpm/action-setup@v4
        with:
          version: 11.1.1
      - name: Install Node.js dependencies for TypeScript bindings
@@ -30,13 +30,13 @@ jobs:
          echo "tag = ${{ inputs.tag || 'latest' }}"

      - name: Checkout Repo LanceDB
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
        with:
          fetch-depth: 0
          persist-credentials: true

      - name: Set up Node.js
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v4
        with:
          node-version: 20

@@ -27,7 +27,7 @@ jobs:
    name: Verify PR title / description conforms to semantic-release
    runs-on: ubuntu-latest
    steps:
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v4
        with:
          node-version: "18"
      # These rules are disabled because Github will always ensure there
@@ -35,7 +35,7 @@ jobs:
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
      - name: Install dependencies needed for ubuntu
        run: |
          sudo apt install -y protobuf-compiler libssl-dev
@@ -53,7 +53,7 @@ jobs:
          python -m pip install --extra-index-url https://pypi.fury.io/lance-format/ --extra-index-url https://pypi.fury.io/lancedb/ -e .
          python -m pip install --extra-index-url https://pypi.fury.io/lance-format/ --extra-index-url https://pypi.fury.io/lancedb/ -r ../docs/requirements.txt
      - name: Set up node
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
@@ -32,7 +32,7 @@ jobs:
        working-directory: ./java
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
      - name: Set up Java 8
        uses: actions/setup-java@v4
        with:
@@ -73,7 +73,7 @@ jobs:
      contents: read
      issues: write
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - uses: ./.github/actions/create-failure-issue
        with:
          job-results: ${{ toJSON(needs) }}
@@ -36,7 +36,7 @@ jobs:
        working-directory: ./java
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
      - name: Set up Java 17
        uses: actions/setup-java@v4
        with:
@@ -19,7 +19,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
      - name: Install license-header-checker
        working-directory: /tmp
        run: |
@@ -49,7 +49,7 @@ jobs:
    steps:
      - name: Output Inputs
        run: echo "${{ toJSON(github.event.inputs) }}"
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -38,14 +38,14 @@ jobs:
      CC: gcc-12
      CXX: g++-12
    steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
      with:
        fetch-depth: 0
        lfs: true
-    - uses: pnpm/action-setup@v6
+    - uses: pnpm/action-setup@v4
      with:
        version: 11.1.1
-    - uses: actions/setup-node@v6
+    - uses: actions/setup-node@v4
      with:
        # pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
        # in October. The library itself still supports Node >= 18
@@ -86,14 +86,14 @@ jobs:
        shell: bash
        working-directory: nodejs
    steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
      with:
        fetch-depth: 0
        lfs: true
-    - uses: pnpm/action-setup@v6
+    - uses: pnpm/action-setup@v4
      with:
        version: 11.1.1
-    - uses: actions/setup-node@v6
+    - uses: actions/setup-node@v4
      name: Setup Node.js 24 for build
      with:
        # pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
@@ -130,7 +130,7 @@ jobs:
          echo "Run 'pnpm run docs', fix any warnings, and commit the changes."
          exit 1
        fi
-    - uses: actions/setup-node@v6
+    - uses: actions/setup-node@v4
      name: Setup Node.js ${{ matrix.node-version }} for test
      with:
        node-version: ${{ matrix.node-version }}
@@ -166,14 +166,14 @@ jobs:
        shell: bash
        working-directory: nodejs
    steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
      with:
        fetch-depth: 0
        lfs: true
-    - uses: pnpm/action-setup@v6
+    - uses: pnpm/action-setup@v4
      with:
        version: 11.1.1
-    - uses: actions/setup-node@v6
+    - uses: actions/setup-node@v4
      with:
        # pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
        # in October.
@@ -32,7 +32,7 @@ jobs:
    permissions:
      contents: write
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -103,7 +103,7 @@ jobs:
            features: fp16kernels
            pre_build: brew install protobuf
          - target: x86_64-pc-windows-msvc
-            host: windows-2025-8x-x64
+            host: windows-latest
            features: ","
            pre_build: |-
              choco install --no-progress protoc ninja nasm
@@ -111,21 +111,12 @@ jobs:
              # There is an issue where choco doesn't add nasm to the path
              export PATH="$PATH:/c/Program Files/NASM"
              nasm -v
-              # Fat LTO of the cdylib is single-threaded and the peak-memory
-              # step of the build, and had started hitting rustc-LLVM OOM on the
-              # Windows runners. ThinLTO parallelizes it across the runner's
-              # cores and keeps peak memory well under the limit.
-              export CARGO_PROFILE_RELEASE_LTO=thin
-              export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
          - target: aarch64-pc-windows-msvc
-            host: windows-2025-8x-x64
+            host: windows-latest
            features: ","
            pre_build: |-
                choco install --no-progress protoc
                rustup target add aarch64-pc-windows-msvc
-                # See ThinLTO note on the x86_64-pc-windows-msvc target above.
-                export CARGO_PROFILE_RELEASE_LTO=thin
-                export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
          - target: x86_64-unknown-linux-gnu
            host: ubuntu-latest
            features: fp16kernels
@@ -179,13 +170,13 @@ jobs:
      run:
        working-directory: nodejs
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - name: Setup pnpm
-        uses: pnpm/action-setup@v6
+        uses: pnpm/action-setup@v4
        with:
          version: 11.1.1
      - name: Setup node
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v4
        with:
          # pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
          # in October.
@@ -199,7 +190,7 @@ jobs:
          toolchain: stable
          targets: ${{ matrix.settings.target }}
      - name: Cache cargo
-        uses: actions/cache@v5
+        uses: actions/cache@v4
        with:
          path: |
            ~/.cargo/registry/index/
@@ -253,7 +244,7 @@ jobs:
        if: ${{ !matrix.settings.docker }}
        shell: bash
      - name: Upload artifact
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@v4
        with:
          name: lancedb-${{ matrix.settings.target }}
          path: nodejs/dist/*.node
@@ -265,7 +256,7 @@ jobs:
        run: pnpm tsc
      - name: Upload Generic Artifacts
        if: ${{ matrix.settings.target == 'aarch64-apple-darwin' }}
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@v4
        with:
          name: nodejs-dist
          path: |
@@ -296,13 +287,13 @@ jobs:
        shell: bash
        working-directory: nodejs
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - name: Setup pnpm
-        uses: pnpm/action-setup@v6
+        uses: pnpm/action-setup@v4
        with:
          version: 11.1.1
      - name: Setup Node.js 24 for install
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v4
        with:
          # pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
          # in October.
@@ -312,18 +303,18 @@ jobs:
      - name: Install dependencies
        run: pnpm install --frozen-lockfile
      - name: Setup Node.js ${{ matrix.node }} for test
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
      - name: Download artifacts
-        uses: actions/download-artifact@v8
+        uses: actions/download-artifact@v4
        with:
          name: lancedb-${{ matrix.settings.target }}
          path: nodejs/dist/
          # For testing purposes:
          # run-id: 13982782871
          # github-token: ${{ secrets.GITHUB_TOKEN }} # token with actions:read permissions on target repo
-      - uses: actions/download-artifact@v8
+      - uses: actions/download-artifact@v4
        with:
          name: nodejs-dist
          path: nodejs/dist
@@ -348,13 +339,13 @@ jobs:
    needs:
      - test-lancedb
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - name: Setup pnpm
-        uses: pnpm/action-setup@v6
+        uses: pnpm/action-setup@v4
        with:
          version: 11.1.1
      - name: Setup node
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: pnpm
@@ -362,14 +353,14 @@ jobs:
          registry-url: "https://registry.npmjs.org"
      - name: Install dependencies
        run: pnpm install --frozen-lockfile
-      - uses: actions/download-artifact@v8
+      - uses: actions/download-artifact@v4
        with:
          name: nodejs-dist
          path: nodejs/dist
          # For testing purposes:
          # run-id: 13982782871
          # github-token: ${{ secrets.GITHUB_TOKEN }} # token with actions:read permissions on target repo
-      - uses: actions/download-artifact@v8
+      - uses: actions/download-artifact@v4
        name: Download arch-specific binaries
        with:
          pattern: lancedb-*
@@ -407,7 +398,7 @@ jobs:
      contents: read
      issues: write
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - uses: ./.github/actions/create-failure-issue
        with:
          job-results: ${{ toJSON(needs) }}
@@ -41,7 +41,7 @@ jobs:
        shell: bash
        working-directory: python
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -66,7 +66,7 @@ jobs:
        shell: bash
        working-directory: python
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -95,7 +95,7 @@ jobs:
        shell: bash
        working-directory: python
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -126,7 +126,7 @@ jobs:
        shell: bash
        working-directory: python
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -160,7 +160,7 @@ jobs:
        shell: bash
        working-directory: python
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -189,7 +189,7 @@ jobs:
        shell: bash
        working-directory: python
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -212,7 +212,7 @@ jobs:
        shell: bash
        working-directory: python
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -40,7 +40,7 @@ jobs:
      CC: clang-18
      CXX: clang++-18
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -65,7 +65,7 @@ jobs:
    timeout-minutes: 10
    runs-on: ubuntu-24.04
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - uses: EmbarkStudios/cargo-deny-action@v2
        with:
          command: check advisories bans licenses sources
@@ -78,7 +78,7 @@ jobs:
      CC: clang
      CXX: clang++
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      # Building without a lock file often requires the latest Rust version since downstream
      # dependencies may have updated their minimum Rust version.
      - uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -113,7 +113,7 @@ jobs:
      CXX: clang++-18
      GH_TOKEN: ${{ secrets.SOPHON_READ_TOKEN }}
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -152,7 +152,7 @@ jobs:
        shell: bash
        working-directory: rust
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          lfs: true
@@ -181,7 +181,7 @@ jobs:
      run:
        working-directory: rust/lancedb
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
      - name: Set target
        run: rustup target add ${{ matrix.target }}
      - uses: Swatinem/rust-cache@v2
@@ -210,7 +210,7 @@ jobs:
      CC: clang-18
      CXX: clang++-18
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v4
        with:
          submodules: true
      - name: Install dependencies
@@ -11,7 +11,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
        with:
          ref: main
          persist-credentials: false
@@ -11,7 +11,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v4
        with:
          ref: main
          persist-credentials: false
@@ -1750,7 +1750,7 @@ version = "3.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
 dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -2039,9 +2039,9 @@ dependencies = [

 [[package]]
 name = "crossbeam-epoch"
-version = "0.9.20"
+version = "0.9.18"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f"
+checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e"
 dependencies = [
 "crossbeam-utils",
 ]
@@ -3014,7 +3014,7 @@ dependencies = [
 "libc",
 "option-ext",
 "redox_users",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -3151,12 +3151,6 @@ dependencies = [
 "encoding_rs",
 ]

-[[package]]
-name = "endian-type"
-version = "0.1.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c34f04666d835ff5d62e058c3995147c06f42fe86ff053337632bca83e42702d"
-
 [[package]]
 name = "enum-as-inner"
 version = "0.6.1"
@@ -3237,7 +3231,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
 dependencies = [
 "libc",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -3429,8 +3423,8 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"

 [[package]]
 name = "fsst"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "rand 0.9.4",
@@ -3785,30 +3779,6 @@ dependencies = [
 "wasm-bindgen",
 ]

-[[package]]
-name = "goosefs-sdk"
-version = "0.1.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9ae079b88ffe7772d12cfc5c40a5a324babb357893d95b5e3a22ae857f236c5f"
-dependencies = [
- "async-trait",
- "bytes",
- "dashmap",
- "hostname",
- "prost",
- "prost-types",
- "rand 0.9.4",
- "reqwest 0.12.28",
- "serde",
- "thiserror 2.0.18",
- "tokio",
- "tokio-stream",
- "tonic",
- "tonic-prost",
- "tracing",
- "uuid",
-]
-
 [[package]]
 name = "group"
 version = "0.12.1"
@@ -4020,17 +3990,6 @@ dependencies = [
 "windows-sys 0.61.2",
 ]

-[[package]]
-name = "hostname"
-version = "0.4.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "617aaa3557aef3810a6369d0a99fac8a080891b68bd9f9812a1eeda0c0730cbd"
-dependencies = [
- "cfg-if 1.0.4",
- "libc",
- "windows-link",
-]
-
 [[package]]
 name = "http"
 version = "0.2.12"
@@ -4191,19 +4150,6 @@ dependencies = [
 "webpki-roots 1.0.7",
 ]

-[[package]]
-name = "hyper-timeout"
-version = "0.5.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0"
-dependencies = [
- "hyper 1.9.0",
- "hyper-util",
- "pin-project-lite",
- "tokio",
- "tower-service",
-]
-
 [[package]]
 name = "hyper-util"
 version = "0.1.20"
@@ -4520,7 +4466,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
 dependencies = [
 "hermit-abi",
 "libc",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -4614,7 +4560,7 @@ dependencies = [
 "portable-atomic-util",
 "serde_core",
 "wasm-bindgen",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -4727,7 +4673,7 @@ dependencies = [
 "jiff",
 "nom 8.0.0",
 "num-traits",
- "ordered-float 5.3.0",
+ "ordered-float",
 "rand 0.9.4",
 "serde",
 "serde_json",
@@ -4780,8 +4726,8 @@ checksum = "e037a2e1d8d5fdbd49b16a4ea09d5d6401c1f29eca5ff29d03d3824dba16256a"

 [[package]]
 name = "lance"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arc-swap",
 "arrow",
@@ -4855,8 +4801,8 @@ dependencies = [

 [[package]]
 name = "lance-arrow"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4865,7 +4811,6 @@ dependencies = [
 "arrow-ord",
 "arrow-schema",
 "arrow-select",
- "bytemuck",
 "bytes",
 "futures",
 "getrandom 0.2.17",
@@ -4878,7 +4823,7 @@ dependencies = [
 [[package]]
 name = "lance-arrow-scalar"
 version = "58.0.0"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4892,7 +4837,7 @@ dependencies = [
 [[package]]
 name = "lance-arrow-stats"
 version = "58.0.0"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "arrow-schema",
@@ -4901,8 +4846,8 @@ dependencies = [

 [[package]]
 name = "lance-bitpacking"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrayref",
 "crunchy",
@@ -4912,8 +4857,8 @@ dependencies = [

 [[package]]
 name = "lance-core"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4951,8 +4896,8 @@ dependencies = [

 [[package]]
 name = "lance-datafusion"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -4982,8 +4927,8 @@ dependencies = [

 [[package]]
 name = "lance-datagen"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -5000,8 +4945,8 @@ dependencies = [

 [[package]]
 name = "lance-derive"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -5010,8 +4955,8 @@ dependencies = [

 [[package]]
 name = "lance-encoding"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-arith",
 "arrow-array",
@@ -5046,8 +4991,8 @@ dependencies = [

 [[package]]
 name = "lance-file"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-arith",
 "arrow-array",
@@ -5077,8 +5022,8 @@ dependencies = [

 [[package]]
 name = "lance-index"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arc-swap",
 "arrow",
@@ -5143,8 +5088,8 @@ dependencies = [

 [[package]]
 name = "lance-io"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow",
 "arrow-arith",
@@ -5168,7 +5113,6 @@ dependencies = [
 "lance-core",
 "lance-namespace",
 "log",
- "metrics",
 "moka",
 "object_store",
 "object_store_opendal",
@@ -5186,8 +5130,8 @@ dependencies = [

 [[package]]
 name = "lance-linalg"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5203,8 +5147,8 @@ dependencies = [

 [[package]]
 name = "lance-namespace"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow",
 "async-trait",
@@ -5216,8 +5160,8 @@ dependencies = [

 [[package]]
 name = "lance-namespace-impls"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow",
 "arrow-ipc",
@@ -5240,7 +5184,7 @@ dependencies = [
 "lance-table",
 "log",
 "object_store",
- "quick-xml 0.40.1",
+ "quick-xml 0.38.4",
 "rand 0.9.4",
 "reqwest 0.12.28",
 "roaring",
@@ -5271,8 +5215,8 @@ dependencies = [

 [[package]]
 name = "lance-select"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5287,8 +5231,8 @@ dependencies = [

 [[package]]
 name = "lance-table"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -5327,22 +5271,22 @@ dependencies = [

 [[package]]
 name = "lance-testing"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "arrow-array",
 "arrow-schema",
 "criterion",
 "lance-arrow",
 "num-traits",
- "pprof 0.15.0",
+ "pprof",
 "rand 0.9.4",
 ]

 [[package]]
 name = "lance-tokenizer"
-version = "9.0.0-beta.19"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.19#8f0e6d3a7c53438275b134c0ac1afbc80600616e"
+version = "9.0.0-beta.10"
+source = "git+https://github.com/jackye1995/lance.git?branch=jack%2Fsophon-pr-6325#1c5b5061c60934b4c18dbe86c5e91b4961105989"
 dependencies = [
 "icu_segmenter",
 "jieba-rs",
@@ -5355,13 +5299,12 @@ dependencies = [

 [[package]]
 name = "lancedb"
-version = "0.32.0-beta.0"
+version = "0.31.0-beta.4"
 dependencies = [
 "ahash",
 "anyhow",
 "arrow",
 "arrow-array",
- "arrow-buffer",
 "arrow-cast",
 "arrow-data",
 "arrow-ipc",
@@ -5411,15 +5354,12 @@ dependencies = [
 "lance-testing",
 "lazy_static",
 "log",
- "metrics",
- "metrics-util",
 "moka",
 "num-traits",
 "object_store",
 "pin-project",
 "polars",
 "polars-arrow",
- "pprof 0.14.1",
 "rand 0.9.4",
 "random_word",
 "regex",
@@ -5443,7 +5383,7 @@ dependencies = [

 [[package]]
 name = "lancedb-nodejs"
-version = "0.32.0-beta.0"
+version = "0.31.0-beta.4"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5468,7 +5408,7 @@ dependencies = [

 [[package]]
 name = "lancedb-python"
-version = "0.35.0-beta.0"
+version = "0.34.0-beta.4"
 dependencies = [
 "arrow",
 "async-trait",
@@ -5870,36 +5810,6 @@ dependencies = [
 "stable_deref_trait",
 ]

-[[package]]
-name = "metrics"
-version = "0.24.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "89550ee9f79e88fef3119de263694973a8adb26c21d75322164fb8c493039fe2"
-dependencies = [
- "portable-atomic",
- "rapidhash",
-]
-
-[[package]]
-name = "metrics-util"
-version = "0.19.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b8496cc523d1f94c1385dd8f0f0c2c480b2b8aeccb5b7e4485ad6365523ae376"
-dependencies = [
- "aho-corasick",
- "crossbeam-epoch",
- "crossbeam-utils",
- "hashbrown 0.15.5",
- "indexmap 2.14.0",
- "metrics",
- "ordered-float 4.6.0",
- "quanta",
- "radix_trie",
- "rand 0.9.4",
- "rand_xoshiro",
- "sketches-ddsketch",
-]
-
 [[package]]
 name = "mime"
 version = "0.3.17"
@@ -6041,9 +5951,9 @@ dependencies = [

 [[package]]
 name = "napi"
-version = "3.10.3"
+version = "3.9.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0c71997d6f7ad4a756966e452426848ac27d3b37a295302d63afbbcce0270f93"
+checksum = "b41bda2ac390efb5e8d22025d925ccc3f3807d8c1bea6d19b36127247c4b8f83"
 dependencies = [
 "bitflags 2.11.1",
 "chrono",
@@ -6066,9 +5976,9 @@ checksum = "c9c366d2c8c60b86fa632df75f745509b52f9128f91a6bad4c796e44abb505e1"

 [[package]]
 name = "napi-derive"
-version = "3.5.9"
+version = "3.5.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d4ba572deef53e2c386759a8c2014175a62679d74ff83adc205c8bc0e0285727"
+checksum = "61d66f70256ad5aef58659966064471d0ad90e2897bc36a5a5e0389c85aabc1e"
 dependencies = [
 "convert_case",
 "ctor 1.0.5",
@@ -6080,9 +5990,9 @@ dependencies = [

 [[package]]
 name = "napi-derive-backend"
-version = "5.1.1"
+version = "5.0.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ddd961eb2aa8965e3f29722d754f3a86907eb1984e2fbcbe3fe87b9a02d6bfba"
+checksum = "81b4b08f15eed7a2a20c3f4c6314013fc3ac890a3afa9892b594485299ebdb2d"
 dependencies = [
 "convert_case",
 "proc-macro2",
@@ -6115,15 +6025,6 @@ dependencies = [
 "rawpointer",
 ]

-[[package]]
-name = "nibble_vec"
-version = "0.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "77a5d83df9f36fe23f0c3648c6bbb8b0298bb5f1939c8f2704431371f4b84d43"
-dependencies = [
- "smallvec",
-]
-
 [[package]]
 name = "nix"
 version = "0.26.4"
@@ -6184,7 +6085,7 @@ version = "0.50.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
 dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -6439,9 +6340,7 @@ dependencies = [
 "opendal-layer-timeout",
 "opendal-service-azblob",
 "opendal-service-azdls",
- "opendal-service-cos",
 "opendal-service-gcs",
- "opendal-service-goosefs",
 "opendal-service-hf",
 "opendal-service-oss",
 "opendal-service-s3",
@@ -6569,23 +6468,6 @@ dependencies = [
 "opendal-core",
 ]

-[[package]]
-name = "opendal-service-cos"
-version = "0.57.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aa8cafe9729213375c7331019b0cb756ad3e1aff7f45cd32c45eae91ebde8901"
-dependencies = [
- "bytes",
- "http 1.4.2",
- "log",
- "opendal-core",
- "quick-xml 0.39.4",
- "reqsign-core",
- "reqsign-file-read-tokio",
- "reqsign-tencent-cos",
- "serde",
-]
-
 [[package]]
 name = "opendal-service-gcs"
 version = "0.57.0"
@@ -6607,20 +6489,6 @@ dependencies = [
 "tokio",
 ]

-[[package]]
-name = "opendal-service-goosefs"
-version = "0.57.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "69e43048bde419947ba826fbdc2f134d6c03f44ebf48bd33a03b72f9fc45fcb4"
-dependencies = [
- "bytes",
- "goosefs-sdk",
- "log",
- "opendal-core",
- "serde",
- "tokio",
-]
-
 [[package]]
 name = "opendal-service-hf"
 version = "0.57.0"
@@ -6688,15 +6556,6 @@ version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"

-[[package]]
-name = "ordered-float"
-version = "4.6.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7bb71e1b3fa6ca1c61f383464aaf2bb0e2f8e772a1f01d486832464de363b951"
-dependencies = [
- "num-traits",
-]
-
 [[package]]
 name = "ordered-float"
 version = "5.3.0"
@@ -7466,28 +7325,6 @@ version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"

-[[package]]
-name = "pprof"
-version = "0.14.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "afad4d4df7b31280028245f152d5a575083e2abb822d05736f5e47653e77689f"
-dependencies = [
- "aligned-vec",
- "backtrace",
- "cfg-if 1.0.4",
- "findshlibs",
- "inferno",
- "libc",
- "log",
- "nix",
- "once_cell",
- "smallvec",
- "spin 0.10.0",
- "symbolic-demangle",
- "tempfile",
- "thiserror 1.0.69",
-]
-
 [[package]]
 name = "pprof"
 version = "0.15.0"
@@ -7563,8 +7400,8 @@ version = "0.14.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "343d3bd7056eda839b03204e68deff7d1b13aba7af2b2fd16890697274262ee7"
 dependencies = [
- "heck 0.5.0",
- "itertools 0.14.0",
+ "heck 0.4.1",
+ "itertools 0.11.0",
 "log",
 "multimap",
 "petgraph",
@@ -7583,7 +7420,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b"
 dependencies = [
 "anyhow",
- "itertools 0.14.0",
+ "itertools 0.11.0",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -7736,21 +7573,6 @@ dependencies = [
 "syn 2.0.117",
 ]

-[[package]]
-name = "quanta"
-version = "0.12.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f3ab5a9d756f0d97bdc89019bd2e4ea098cf9cde50ee7564dde6b81ccc8f06c7"
-dependencies = [
- "crossbeam-utils",
- "libc",
- "once_cell",
- "raw-cpuid",
- "wasi 0.11.1+wasi-snapshot-preview1",
- "web-sys",
- "winapi",
-]
-
 [[package]]
 name = "quick-xml"
 version = "0.26.0"
@@ -7760,6 +7582,15 @@ dependencies = [
 "memchr",
 ]

+[[package]]
+name = "quick-xml"
+version = "0.38.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b66c2058c55a409d601666cffe35f04333cf1013010882cec174a7467cd4e21c"
+dependencies = [
+ "memchr",
+]
+
 [[package]]
 name = "quick-xml"
 version = "0.39.4"
@@ -7770,15 +7601,6 @@ dependencies = [
 "serde",
 ]

-[[package]]
-name = "quick-xml"
-version = "0.40.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2474bd2e5029e7ccb6abb2ba48cf2383a333851dedf495901544281590c7da7f"
-dependencies = [
- "memchr",
-]
-
 [[package]]
 name = "quinn"
 version = "0.11.9"
@@ -7832,7 +7654,7 @@ dependencies = [
 "once_cell",
 "socket2 0.6.3",
 "tracing",
- "windows-sys 0.60.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -7862,16 +7684,6 @@ version = "0.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09"

-[[package]]
-name = "radix_trie"
-version = "0.2.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c069c179fcdc6a2fe24d8d18305cf085fdbd4f922c041943e203685d6a1c58fd"
-dependencies = [
- "endian-type",
- "nibble_vec",
-]
-
 [[package]]
 name = "rancor"
 version = "0.1.1"
@@ -8006,15 +7818,6 @@ version = "1.7.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "973443cf09a9c8656b574a866ab68dfa19f0867d0340648c7d2f6a71b8a8ea68"

-[[package]]
-name = "rapidhash"
-version = "4.5.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5da7e78a036ce858e8d55b7e7dc8ba3a88b78350fd2155d3591bbd966b58589e"
-dependencies = [
- "rustversion",
-]
-
 [[package]]
 name = "raw-cpuid"
 version = "11.6.0"
@@ -8299,21 +8102,6 @@ dependencies = [
 "tokio",
 ]

-[[package]]
-name = "reqsign-tencent-cos"
-version = "3.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e128f19525861dbded59e1e7c17653a8ed63d573ca04aed708d552dbef5bb32a"
-dependencies = [
- "anyhow",
- "http 1.4.2",
- "log",
- "percent-encoding",
- "reqsign-core",
- "serde",
- "serde_json",
-]
-
 [[package]]
 name = "reqwest"
 version = "0.12.28"
@@ -8606,7 +8394,7 @@ dependencies = [
 "errno",
 "libc",
 "linux-raw-sys",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -8677,7 +8465,7 @@ dependencies = [
 "security-framework",
 "security-framework-sys",
 "webpki-root-certs",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -9192,12 +8980,6 @@ version = "1.0.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"

-[[package]]
-name = "sketches-ddsketch"
-version = "0.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0c6f73aeb92d671e0cc4dca167e59b2deb6387c375391bc99ee743f326994a2b"
-
 [[package]]
 name = "slab"
 version = "0.4.12"
@@ -9245,7 +9027,7 @@ version = "0.8.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451"
 dependencies = [
- "heck 0.5.0",
+ "heck 0.4.1",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -9257,7 +9039,7 @@ version = "0.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "54254b8531cafa275c5e096f62d48c81435d1015405a91198ddb11e967301d40"
 dependencies = [
- "heck 0.5.0",
+ "heck 0.4.1",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -9690,7 +9472,7 @@ dependencies = [
 "getrandom 0.4.2",
 "once_cell",
 "rustix",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -10005,45 +9787,6 @@ dependencies = [
 "winnow",
 ]

-[[package]]
-name = "tonic"
-version = "0.14.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ac2a5518c70fa84342385732db33fb3f44bc4cc748936eb5833d2df34d6445ef"
-dependencies = [
- "async-trait",
- "base64 0.22.1",
- "bytes",
- "h2 0.4.14",
- "http 1.4.2",
- "http-body 1.0.1",
- "http-body-util",
- "hyper 1.9.0",
- "hyper-timeout",
- "hyper-util",
- "percent-encoding",
- "pin-project",
- "socket2 0.6.3",
- "sync_wrapper",
- "tokio",
- "tokio-stream",
- "tower",
- "tower-layer",
- "tower-service",
- "tracing",
-]
-
-[[package]]
-name = "tonic-prost"
-version = "0.14.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "50849f68853be452acf590cde0b146665b8d507b3b8af17261df47e02c209ea0"
-dependencies = [
- "bytes",
- "prost",
- "tonic",
-]
-
 [[package]]
 name = "tower"
 version = "0.5.3"
@@ -10052,12 +9795,9 @@ checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4"
 dependencies = [
 "futures-core",
 "futures-util",
- "indexmap 2.14.0",
 "pin-project-lite",
- "slab",
 "sync_wrapper",
 "tokio",
- "tokio-util",
 "tower-layer",
 "tower-service",
 "tracing",
@@ -10667,7 +10407,7 @@ version = "0.1.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
 dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
 ]

 [[package]]
@@ -13,25 +13,24 @@ categories = ["database-implementations"]
 rust-version = "1.91.0"

 [workspace.dependencies]
-lance = { "version" = "=9.0.0-beta.19", default-features = false, "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-core = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-datagen = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-file = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-io = { "version" = "=9.0.0-beta.19", default-features = false, "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-index = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-linalg = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace-impls = { "version" = "=9.0.0-beta.19", default-features = false, "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-table = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-testing = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-datafusion = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-encoding = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
-lance-arrow = { "version" = "=9.0.0-beta.19", "tag" = "v9.0.0-beta.19", "git" = "https://github.com/lance-format/lance.git" }
+lance = { "version" = "=9.0.0-beta.10", default-features = false, "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-core = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-datagen = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-file = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-io = { "version" = "=9.0.0-beta.10", default-features = false, "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-index = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-linalg = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-namespace = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-namespace-impls = { "version" = "=9.0.0-beta.10", default-features = false, "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-table = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-testing = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-datafusion = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-encoding = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
+lance-arrow = { "version" = "=9.0.0-beta.10", "branch" = "jack/sophon-pr-6325", "git" = "https://github.com/jackye1995/lance.git" }
 ahash = "0.8"
 # Note that this one does not include pyarrow
 arrow = { version = "58.0.0", optional = false }
 arrow-array = "58.0.0"
-arrow-buffer = "58.0.0"
 arrow-data = "58.0.0"
 arrow-ipc = "58.0.0"
 arrow-ord = "58.0.0"
@@ -54,8 +53,6 @@ half = { "version" = "2.7.1", default-features = false, features = [
 ] }
 futures = "0"
 log = "0.4"
-metrics = "0.24"
-metrics-util = "0.19"
 moka = { version = "0.12", features = ["future"] }
 object_store = "0.13.2"
 pin-project = "1.0.7"
@@ -51,6 +51,18 @@ ignore = [
    # https://rustsec.org/advisories/RUSTSEC-2024-0436
    { id = "RUSTSEC-2024-0436", reason = "transitive via datafusion; awaiting ecosystem migration" },

+    # encoding: unmaintained. Reached through lindera-dictionary, which is
+    # required by the native Lindera tokenizer path. Lindera has not migrated
+    # off this crate yet.
+    # https://rustsec.org/advisories/RUSTSEC-2021-0153
+    { id = "RUSTSEC-2021-0153", reason = "transitive via lindera-dictionary for native Lindera tokenizer" },
+
+    # fast-float: unsound and unmaintained. Reached only through polars-arrow
+    # from the optional Polars integration; replacement requires a Polars
+    # dependency upgrade.
+    # https://rustsec.org/advisories/RUSTSEC-2024-0379
+    { id = "RUSTSEC-2024-0379", reason = "transitive via polars-arrow; waiting on Polars migration" },
+
    # tantivy: segfault on malformed input due to missing bounds check.
    # Pulled in via lance for full-text search. We only feed tantivy
    # documents we construct ourselves, not attacker-controlled bytes.
@@ -68,6 +80,18 @@ ignore = [
    # https://rustsec.org/advisories/RUSTSEC-2025-0119
    { id = "RUSTSEC-2025-0119", reason = "transitive via hf-hub/indicatif; cosmetic formatting crate" },

+    # bincode: unmaintained. Reached through lindera and lindera-dictionary,
+    # which are required by the native Lindera tokenizer path. Lindera has not
+    # migrated to another serialization format yet.
+    # https://rustsec.org/advisories/RUSTSEC-2025-0141
+    { id = "RUSTSEC-2025-0141", reason = "transitive via lindera/lindera-dictionary for native Lindera tokenizer" },
+
+    # lru: soundness issue in IterMut. Reached only through aws-sdk-s3 in
+    # LanceDB's dev-dependency graph; LanceDB does not use that iterator
+    # directly. Clearing this requires the AWS SDK chain to update lru.
+    # https://rustsec.org/advisories/RUSTSEC-2026-0002
+    { id = "RUSTSEC-2026-0002", reason = "transitive via aws-sdk-s3 dev-dependency; waiting on AWS SDK lru upgrade" },
+
    # rustls-webpki 0.101.7 (old major line): name-constraint checks for
    # URI / wildcard names. Pulled in only via the legacy rustls 0.21 chain
    # from aws-smithy-http-client. The 0.103 line we actively use is patched.
@@ -84,23 +108,17 @@ ignore = [
    # https://rustsec.org/advisories/RUSTSEC-2026-0104
    { id = "RUSTSEC-2026-0104", reason = "only affects rustls-webpki 0.101 from legacy aws-smithy/rustls 0.21 chain" },

+    # rand 0.8.5: soundness issue only when ThreadRng reseeds inside a custom
+    # logger. Reached through several transitive chains. LanceDB does not use
+    # rand from a custom logger; upgrade once all pinned chains accept 0.8.6+.
+    # https://rustsec.org/advisories/RUSTSEC-2026-0097
+    { id = "RUSTSEC-2026-0097", reason = "transitive rand 0.8.5; LanceDB does not call ThreadRng from custom logging" },
+
    # pyo3 advisories in the Python bindings; tracked pending a patched pyo3 release.
    # https://rustsec.org/advisories/RUSTSEC-2026-0176
    # https://rustsec.org/advisories/RUSTSEC-2026-0177
    { id = "RUSTSEC-2026-0176", reason = "pyo3 in Python bindings; awaiting patched pyo3 release" },
    { id = "RUSTSEC-2026-0177", reason = "pyo3 in Python bindings; awaiting patched pyo3 release" },
-
-    # quick-xml < 0.41.0: quadratic runtime on duplicate attribute names (DoS).
-    # quick-xml < 0.41.0: unbounded namespace-declaration allocation in NsReader (DoS).
-    # Pulled in transitively by inferno (dev-only flame-graph dep), lance-namespace-impls
-    # (git dep from lance), and opendal/reqsign (cloud storage XML parsing). The XML
-    # parsed by opendal/reqsign comes from trusted cloud-storage endpoints (S3, GCS,
-    # Azure), not attacker-controlled input. Clearing requires upstream crates to migrate
-    # to quick-xml >= 0.41.0.
-    # https://rustsec.org/advisories/RUSTSEC-2026-0194
-    # https://rustsec.org/advisories/RUSTSEC-2026-0195
-    { id = "RUSTSEC-2026-0194", reason = "transitive via inferno/lance/opendal; XML from trusted cloud endpoints, not attacker-controlled" },
-    { id = "RUSTSEC-2026-0195", reason = "transitive via inferno/lance/opendal; XML from trusted cloud endpoints, not attacker-controlled" },
 ]

 # ---------------------------------------------------------------------------
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
 <dependency>
    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-core</artifactId>
-    <version>0.32.0-beta.0</version>
+    <version>0.31.0-beta.4</version>
 </dependency>
 ```

@@ -518,9 +518,6 @@ x > 5 OR y = 'test'

 Filtering performance can often be improved by creating a scalar index
 on the filter column(s).
-
-Calling this multiple times combines the filters with a logical AND rather
-than replacing the previous filter.
 ```

 #### Inherited from
@@ -398,26 +398,6 @@ Drop an index from the table.

 ***

-### getLsmWriteSpec()
-
-```ts
-abstract getLsmWriteSpec(): Promise<undefined | LsmWriteSpec>
-```
-
-Read the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) currently installed on this table.
-
-Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
-spec has been set, or it was removed with [Table#unsetLsmWriteSpec](Table.md#unsetlsmwritespec)).
-The returned spec — including its `maintainedIndexes` and
-`writerConfigDefaults` — mirrors what was passed to
-[Table#setLsmWriteSpec](Table.md#setlsmwritespec).
-
-#### Returns
-
-`Promise`&lt;`undefined` \| [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)&gt;
-
-***
-
 ### indexStats()

 ```ts
@@ -767,9 +767,6 @@ x > 5 OR y = 'test'

 Filtering performance can often be improved by creating a scalar index
 on the filter column(s).
-
-Calling this multiple times combines the filters with a logical AND rather
-than replacing the previous filter.
 ```

 #### Inherited from
@@ -1,42 +0,0 @@
-[**@lancedb/lancedb**](../README.md) • **Docs**
-
-***
-
-[@lancedb/lancedb](../globals.md) / instrumentLanceDbMetrics
-
-# Function: instrumentLanceDbMetrics()
-
-```ts
-function instrumentLanceDbMetrics(meterProvider?): boolean
-```
-
-Register LanceDB metrics as OpenTelemetry observable instruments.
-
-Installs a process-global metrics recorder and creates one observable
-instrument per LanceDB metric (currently object store request counts, bytes,
-latency, errors, and throttles) on the given (or global) `MeterProvider`. The
-configured `MetricReader` then collects them on its own schedule.
-
-Counters and gauges map directly to observable counters/gauges. Because
-OpenTelemetry has no asynchronous histogram instrument, each histogram is
-exported Prometheus-style as cumulative `le` bucket counts (`<name>_bucket`,
-with an `le` attribute) plus `<name>_count` and `<name>_sum`.
-
-Requires `@opentelemetry/api` (a dependency) and, to actually export, an
-OpenTelemetry SDK such as `@opentelemetry/sdk-metrics`.
-
-## Parameters
-
-* **meterProvider?**: `MeterProvider`
-    The provider to register instruments on. Defaults to the
-    global provider from `@opentelemetry/api`.
-
-## Returns
-
-`boolean`
-
-`true` if the recorder is installed and instruments are registered.
-  `false` if a different `metrics` recorder is already installed in this
-  process (only one global recorder is permitted), in which case a warning is
-  emitted and no instruments are created. Calling this more than once is safe;
-  instruments are created only on the first successful call.
@@ -131,7 +131,6 @@
 - [RecordBatchIterator](functions/RecordBatchIterator.md)
 - [connect](functions/connect.md)
 - [connectNamespace](functions/connectNamespace.md)
- [instrumentLanceDbMetrics](functions/instrumentLanceDbMetrics.md)
 - [makeArrowTable](functions/makeArrowTable.md)
 - [packBits](functions/packBits.md)
 - [permutationBuilder](functions/permutationBuilder.md)
@@ -8,14 +8,6 @@

 ## Properties

-### clumpSize?
-
-```ts
-optional clumpSize: number;
-```
-
-***
-
 ### counts?

 ```ts
@@ -8,7 +8,7 @@
    <parent>
      <groupId>com.lancedb</groupId>
      <artifactId>lancedb-parent</artifactId>
-      <version>0.32.0-beta.0</version>
+      <version>0.31.0-beta.4</version>
      <relativePath>../pom.xml</relativePath>
    </parent>

@@ -6,7 +6,7 @@

    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-parent</artifactId>
-    <version>0.32.0-beta.0</version>
+    <version>0.31.0-beta.4</version>
    <packaging>pom</packaging>
    <name>${project.artifactId}</name>
    <description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <arrow.version>15.0.0</arrow.version>
-        <lance-core.version>9.0.0-beta.19</lance-core.version>
+        <lance-core.version>9.0.0-beta.10</lance-core.version>
        <spotless.skip>false</spotless.skip>
        <spotless.version>2.30.0</spotless.version>
        <spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
@@ -1,7 +1,7 @@
 [package]
 name = "lancedb-nodejs"
 edition.workspace = true
-version = "0.32.0-beta.0"
+version = "0.31.0-beta.4"
 publish = false
 license.workspace = true
 description.workspace = true
@@ -44,6 +44,6 @@ aws-lc-rs = "=1.16.3"
 napi-build = "2.3.1"

 [features]
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/goosefs", "lancedb/metrics-otel"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
 fp16kernels = ["lancedb/fp16kernels"]
 remote = ["lancedb/remote"]
@@ -1,114 +0,0 @@
-// SPDX-License-Identifier: Apache-2.0
-// SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-import {
-  MeterProvider,
-  type MetricData,
-  MetricReader,
-} from "@opentelemetry/sdk-metrics";
-import * as tmp from "tmp";
-import { connect, instrumentLanceDbMetrics } from "../lancedb";
-// snapshotLancedbMetrics is internal plumbing (not part of the public API), so
-// it is imported from the native module rather than the package entry point.
-import { snapshotLancedbMetrics } from "../lancedb/native";
-
-// The metrics recorder is process-global and installed once, so the whole
-// bridge is exercised in a single test to avoid cross-test global-state coupling.
-
-// A minimal pull-based reader whose `collect()` we drive directly, invoking the
-// observable-instrument callbacks. `@opentelemetry/sdk-metrics` ships no
-// in-memory reader, so we subclass the abstract base.
-class TestMetricReader extends MetricReader {
-  protected async onForceFlush(): Promise<void> {
-    // no-op: collection is driven directly via collect()
-  }
-  protected async onShutdown(): Promise<void> {
-    // no-op: nothing to release
-  }
-}
-
-async function metricsByName(
-  reader: TestMetricReader,
-): Promise<Map<string, MetricData>> {
-  const collected = await reader.collect();
-  const result = new Map<string, MetricData>();
-  for (const scope of collected.resourceMetrics.scopeMetrics) {
-    for (const metric of scope.metrics) {
-      result.set(metric.descriptor.name, metric);
-    }
-  }
-  return result;
-}
-
-describe("OpenTelemetry metrics bridge", () => {
-  let tmpDir: tmp.DirResult;
-  beforeEach(() => {
-    tmpDir = tmp.dirSync({ unsafeCleanup: true });
-  });
-  afterEach(() => tmpDir.removeCallback());
-
-  it("snapshot is safe to call regardless of install state", () => {
-    expect(Array.isArray(snapshotLancedbMetrics())).toBe(true);
-  });
-
-  it("exports object store metrics via observable instruments", async () => {
-    const reader = new TestMetricReader();
-    const provider = new MeterProvider({ readers: [reader] });
-    expect(instrumentLanceDbMetrics(provider)).toBe(true);
-
-    // Generate object store activity on the local filesystem (scheme "file").
-    const db = await connect(tmpDir.name);
-    const data = Array.from({ length: 256 }, (_, i) => ({ id: i }));
-    const table = await db.createTable("t", data);
-    expect(await table.countRows()).toBe(256);
-
-    const metrics = await metricsByName(reader);
-
-    const requests = metrics.get("lance_object_store_requests_total");
-    expect(requests).toBeDefined();
-    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
-    const requestPoints = (requests!.dataPoints as any[]) ?? [];
-    expect(requestPoints.length).toBeGreaterThan(0);
-    for (const p of requestPoints) {
-      // Labelled by `operation` and `base` (the store scheme by default).
-      expect(p.attributes).toHaveProperty("base");
-      expect(p.attributes).toHaveProperty("operation");
-    }
-    const totalRequests = requestPoints.reduce((acc, p) => acc + p.value, 0);
-    expect(totalRequests).toBeGreaterThan(0);
-
-    // Histograms are decomposed into bucket / count / sum observable counters.
-    const bucket = metrics.get(
-      "lance_object_store_request_duration_seconds_bucket",
-    );
-    expect(bucket).toBeDefined();
-    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
-    const bucketPoints = (bucket!.dataPoints as any[]) ?? [];
-    expect(bucketPoints.length).toBeGreaterThan(0);
-    expect(bucketPoints.every((p) => "le" in p.attributes)).toBe(true);
-    // The implicit +Inf bucket must be present.
-    expect(bucketPoints.some((p) => p.attributes.le === "+Inf")).toBe(true);
-
-    const count = metrics.get(
-      "lance_object_store_request_duration_seconds_count",
-    );
-    expect(count).toBeDefined();
-    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
-    const countPoints = (count!.dataPoints as any[]) ?? [];
-    expect(countPoints.reduce((acc, p) => acc + p.value, 0)).toBeGreaterThan(0);
-
-    const sum = metrics.get("lance_object_store_request_duration_seconds_sum");
-    expect(sum).toBeDefined();
-    // biome-ignore lint/suspicious/noExplicitAny: SDK point shape
-    const sumPoints = (sum!.dataPoints as any[]) ?? [];
-    expect(sumPoints.reduce((acc, p) => acc + p.value, 0)).toBeGreaterThan(0);
-
-    // Unit handling: only `_sum` keeps the histogram's unit (seconds); `_bucket`
-    // and `_count` observe cumulative counts and are unitless.
-    expect(sum!.descriptor.unit).toBe("s");
-    expect(bucket!.descriptor.unit).toBe("");
-    expect(count!.descriptor.unit).toBe("");
-
-    await provider.shutdown();
-  });
-});
@@ -215,20 +215,6 @@ describe("Query orderBy", () => {
    expect(results[2].score).toBeCloseTo(4.1, 0.001);
  });

-  it("should combine repeated where clauses with AND", async () => {
-    const results = await table
-      .query()
-      .where("score > 1.0")
-      .where("score < 3.0")
-      .orderBy({ columnName: "score" })
-      .toArray();
-    // Only rows matching both predicates should be returned, rather than the
-    // second where() silently replacing the first.
-    expect(results.length).toBe(2);
-    expect(results[0].score).toBeCloseTo(1.2, 0.001);
-    expect(results[1].score).toBeCloseTo(2.8, 0.001);
-  });
-
  it("should support method chaining with limit", async () => {
    const results = await table
      .query()
@@ -2992,56 +2992,6 @@ describe("setLsmWriteSpec / unsetLsmWriteSpec", () => {
      }),
    ).rejects.toThrow();
  });
-
-  it("reads back the installed spec via getLsmWriteSpec", async () => {
-    const conn = await connect(tmpDir.name);
-    const table = await makeTable(conn);
-    await table.setUnenforcedPrimaryKey("id");
-
-    // Nothing installed yet.
-    expect(await table.getLsmWriteSpec()).toBeUndefined();
-
-    // A real scalar index is needed to name it as a maintained index.
-    await table.add([{ id: 1 }, { id: 2 }, { id: 3 }]);
-    await table.createIndex("id");
-    const indexName = (await table.listIndices())[0].name;
-
-    // Bucket spec round-trips, including maintained indexes and writer config
-    // defaults. Lance writer-config keys are canonically snake_case.
-    // biome-ignore lint/style/useNamingConvention: Lance writer-config keys are snake_case
-    const writerConfigDefaults = { durable_write: "false" };
-    await table.setLsmWriteSpec({
-      specType: "bucket",
-      column: "id",
-      numBuckets: 4,
-      maintainedIndexes: [indexName],
-      writerConfigDefaults,
-    });
-    const spec = await table.getLsmWriteSpec();
-    expect(spec).toBeDefined();
-    expect(spec?.specType).toBe("bucket");
-    expect(spec?.column).toBe("id");
-    expect(spec?.numBuckets).toBe(4);
-    expect(spec?.maintainedIndexes).toEqual([indexName]);
-    expect(spec?.writerConfigDefaults).toEqual(writerConfigDefaults);
-
-    // After unset, undefined again.
-    await table.unsetLsmWriteSpec();
-    expect(await table.getLsmWriteSpec()).toBeUndefined();
-
-    // Identity round-trips (column recovered from the schema).
-    await table.setLsmWriteSpec({ specType: "identity", column: "id" });
-    const identity = await table.getLsmWriteSpec();
-    expect(identity?.specType).toBe("identity");
-    expect(identity?.column).toBe("id");
-    await table.unsetLsmWriteSpec();
-
-    // Unsharded round-trips (no routing column).
-    await table.setLsmWriteSpec({ specType: "unsharded" });
-    const unsharded = await table.getLsmWriteSpec();
-    expect(unsharded?.specType).toBe("unsharded");
-    expect(unsharded?.column).toBeFalsy();
-  });
 });

 describe("LSM merge insert", () => {
@@ -20,11 +20,6 @@ import { HeaderProvider } from "./header";
 // Re-export native header provider for use with connectWithHeaderProvider
 export { JsHeaderProvider as NativeJsHeaderProvider } from "./native.js";

-// OpenTelemetry metrics bridge. Only the high-level entry point is public; the
-// underlying recorder/catalog/snapshot functions remain internal plumbing that
-// `otel.ts` consumes from the native module.
-export { instrumentLanceDbMetrics } from "./otel";
-
 export {
  AddColumnsSql,
  ConnectionOptions,
@@ -1,137 +0,0 @@
-// SPDX-License-Identifier: Apache-2.0
-// SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-import {
-  type Attributes,
-  type MeterProvider,
-  type ObservableResult,
-  metrics,
-} from "@opentelemetry/api";
-
-import {
-  lancedbMetricsCatalog,
-  registerLancedbMetricsRecorder,
-  snapshotLancedbMetrics,
-} from "./native";
-
-let instrumented = false;
-
-/**
- * Register LanceDB metrics as OpenTelemetry observable instruments.
- *
- * Installs a process-global metrics recorder and creates one observable
- * instrument per LanceDB metric (currently object store request counts, bytes,
- * latency, errors, and throttles) on the given (or global) `MeterProvider`. The
- * configured `MetricReader` then collects them on its own schedule.
- *
- * Counters and gauges map directly to observable counters/gauges. Because
- * OpenTelemetry has no asynchronous histogram instrument, each histogram is
- * exported Prometheus-style as cumulative `le` bucket counts (`<name>_bucket`,
- * with an `le` attribute) plus `<name>_count` and `<name>_sum`.
- *
- * Requires `@opentelemetry/api` (a dependency) and, to actually export, an
- * OpenTelemetry SDK such as `@opentelemetry/sdk-metrics`.
- *
- * @param meterProvider The provider to register instruments on. Defaults to the
- *   global provider from `@opentelemetry/api`.
- * @returns `true` if the recorder is installed and instruments are registered.
- *   `false` if a different `metrics` recorder is already installed in this
- *   process (only one global recorder is permitted), in which case a warning is
- *   emitted and no instruments are created. Calling this more than once is safe;
- *   instruments are created only on the first successful call.
- */
-export function instrumentLanceDbMetrics(
-  meterProvider?: MeterProvider,
-): boolean {
-  if (!registerLancedbMetricsRecorder()) {
-    console.warn(
-      "Could not install the LanceDB metrics recorder: another `metrics` " +
-        "recorder is already installed in this process. LanceDB metrics will " +
-        "not be exported via OpenTelemetry.",
-    );
-    return false;
-  }
-
-  if (instrumented) {
-    return true;
-  }
-
-  const provider = meterProvider ?? metrics.getMeterProvider();
-  const meter = provider.getMeter("lancedb");
-
-  const scalarCallback = (metricName: string) => (result: ObservableResult) => {
-    for (const point of snapshotLancedbMetrics()) {
-      if (point.name === metricName && point.value != null) {
-        result.observe(point.value, point.attributes);
-      }
-    }
-  };
-
-  const bucketCallback = (metricName: string) => (result: ObservableResult) => {
-    for (const point of snapshotLancedbMetrics()) {
-      if (point.name !== metricName || point.buckets == null) {
-        continue;
-      }
-      for (const bucket of point.buckets) {
-        const attributes: Attributes = {
-          ...point.attributes,
-          le: bucket.le,
-        };
-        result.observe(bucket.cumulativeCount, attributes);
-      }
-    }
-  };
-
-  const fieldCallback =
-    (metricName: string, field: "count" | "sum") =>
-    (result: ObservableResult) => {
-      for (const point of snapshotLancedbMetrics()) {
-        if (point.name !== metricName) {
-          continue;
-        }
-        const value = point[field];
-        if (value != null) {
-          result.observe(value, point.attributes);
-        }
-      }
-    };
-
-  for (const desc of lancedbMetricsCatalog()) {
-    const unit = desc.unit ?? "";
-    if (desc.kind === "counter") {
-      const counter = meter.createObservableCounter(desc.name, {
-        unit,
-        description: desc.description,
-      });
-      counter.addCallback(scalarCallback(desc.name));
-    } else if (desc.kind === "gauge") {
-      const gauge = meter.createObservableGauge(desc.name, {
-        unit,
-        description: desc.description,
-      });
-      gauge.addCallback(scalarCallback(desc.name));
-    } else if (desc.kind === "histogram") {
-      // `_bucket` and `_count` observe cumulative sample counts, not the
-      // histogram's measured quantity, so they are unitless; only `_sum`
-      // carries the histogram's unit.
-      const bucket = meter.createObservableCounter(`${desc.name}_bucket`, {
-        description: `${desc.description} (cumulative buckets)`,
-      });
-      bucket.addCallback(bucketCallback(desc.name));
-
-      const count = meter.createObservableCounter(`${desc.name}_count`, {
-        description: `${desc.description} (count)`,
-      });
-      count.addCallback(fieldCallback(desc.name, "count"));
-
-      const sum = meter.createObservableCounter(`${desc.name}_sum`, {
-        unit,
-        description: `${desc.description} (sum)`,
-      });
-      sum.addCallback(fieldCallback(desc.name, "sum"));
-    }
-  }
-
-  instrumented = true;
-  return true;
-}
@@ -362,9 +362,6 @@ export class StandardQueryBase<
   *
   * Filtering performance can often be improved by creating a scalar index
   * on the filter column(s).
-   *
-   * Calling this multiple times combines the filters with a logical AND rather
-   * than replacing the previous filter.
   */
  where(predicate: string): this {
    this.doCall((inner: NativeQueryType) => inner.onlyIf(predicate));
@@ -585,17 +585,6 @@ export abstract class Table {
   * @returns {Promise<void>}
   */
  abstract unsetLsmWriteSpec(): Promise<void>;
-  /**
-   * Read the {@link LsmWriteSpec} currently installed on this table.
-   *
-   * Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
-   * spec has been set, or it was removed with {@link Table#unsetLsmWriteSpec}).
-   * The returned spec — including its `maintainedIndexes` and
-   * `writerConfigDefaults` — mirrors what was passed to
-   * {@link Table#setLsmWriteSpec}.
-   * @returns {Promise<LsmWriteSpec | undefined>}
-   */
-  abstract getLsmWriteSpec(): Promise<LsmWriteSpec | undefined>;
  /**
   * Drain and close any cached MemWAL shard writers held for this table.
   *
@@ -1102,15 +1091,6 @@ export class LocalTable extends Table {
    return await this.inner.unsetLsmWriteSpec();
  }

-  async getLsmWriteSpec(): Promise<LsmWriteSpec | undefined> {
-    // The native binding types `specType` as a plain `string`; narrow it back
-    // to the public union. The Rust `From` impl only ever emits one of the
-    // three valid values, so the cast is safe.
-    return ((await this.inner.getLsmWriteSpec()) ?? undefined) as
-      | LsmWriteSpec
-      | undefined;
-  }
-
  async closeLsmWriters(): Promise<void> {
    return await this.inner.closeLsmWriters();
  }
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.32.0-beta.0",
+	"version": "0.31.0-beta.4",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.32.0-beta.0",
+	"version": "0.31.0-beta.4",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-musl",
-	"version": "0.32.0-beta.0",
+	"version": "0.31.0-beta.4",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-musl.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.32.0-beta.0",
+	"version": "0.31.0-beta.4",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-musl",
-	"version": "0.32.0-beta.0",
+	"version": "0.31.0-beta.4",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-musl.node",
@@ -1,6 +1,6 @@
 {
  "name": "@lancedb/lancedb-win32-arm64-msvc",
-  "version": "0.32.0-beta.0",
+  "version": "0.31.0-beta.4",
  "os": [
    "win32"
  ],
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.32.0-beta.0",
+	"version": "0.31.0-beta.4",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",
@@ -1,12 +1,12 @@
 {
  "name": "@lancedb/lancedb",
-  "version": "0.32.0-beta.0",
+  "version": "0.31.0-beta.4",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@lancedb/lancedb",
-      "version": "0.32.0-beta.0",
+      "version": "0.31.0-beta.4",
      "cpu": [
        "x64",
        "arm64"
@@ -18,7 +18,6 @@
        "win32"
      ],
      "dependencies": {
-        "@opentelemetry/api": "^1.9.0",
        "reflect-metadata": "^0.2.2"
      },
      "devDependencies": {
@@ -28,7 +27,6 @@
        "@biomejs/biome": "^1.7.3",
        "@jest/globals": "^29.7.0",
        "@napi-rs/cli": "3.7.0",
-        "@opentelemetry/sdk-metrics": "^1.30.0",
        "@types/axios": "^0.14.0",
        "@types/jest": "^29.1.2",
        "@types/node": "22.7.4",
@@ -4150,75 +4148,6 @@
        "@octokit/openapi-types": "^27.0.0"
      }
    },
-    "node_modules/@opentelemetry/api": {
-      "version": "1.9.1",
-      "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.1.tgz",
-      "integrity": "sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==",
-      "license": "Apache-2.0",
-      "engines": {
-        "node": ">=8.0.0"
-      }
-    },
-    "node_modules/@opentelemetry/core": {
-      "version": "1.30.1",
-      "resolved": "https://registry.npmjs.org/@opentelemetry/core/-/core-1.30.1.tgz",
-      "integrity": "sha512-OOCM2C/QIURhJMuKaekP3TRBxBKxG/TWWA0TL2J6nXUtDnuCtccy49LUJF8xPFXMX+0LMcxFpCo8M9cGY1W6rQ==",
-      "dev": true,
-      "license": "Apache-2.0",
-      "dependencies": {
-        "@opentelemetry/semantic-conventions": "1.28.0"
-      },
-      "engines": {
-        "node": ">=14"
-      },
-      "peerDependencies": {
-        "@opentelemetry/api": ">=1.0.0 <1.10.0"
-      }
-    },
-    "node_modules/@opentelemetry/resources": {
-      "version": "1.30.1",
-      "resolved": "https://registry.npmjs.org/@opentelemetry/resources/-/resources-1.30.1.tgz",
-      "integrity": "sha512-5UxZqiAgLYGFjS4s9qm5mBVo433u+dSPUFWVWXmLAD4wB65oMCoXaJP1KJa9DIYYMeHu3z4BZcStG3LC593cWA==",
-      "dev": true,
-      "license": "Apache-2.0",
-      "dependencies": {
-        "@opentelemetry/core": "1.30.1",
-        "@opentelemetry/semantic-conventions": "1.28.0"
-      },
-      "engines": {
-        "node": ">=14"
-      },
-      "peerDependencies": {
-        "@opentelemetry/api": ">=1.0.0 <1.10.0"
-      }
-    },
-    "node_modules/@opentelemetry/sdk-metrics": {
-      "version": "1.30.1",
-      "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-metrics/-/sdk-metrics-1.30.1.tgz",
-      "integrity": "sha512-q9zcZ0Okl8jRgmy7eNW3Ku1XSgg3sDLa5evHZpCwjspw7E8Is4K/haRPDJrBcX3YSn/Y7gUvFnByNYEKQNbNog==",
-      "dev": true,
-      "license": "Apache-2.0",
-      "dependencies": {
-        "@opentelemetry/core": "1.30.1",
-        "@opentelemetry/resources": "1.30.1"
-      },
-      "engines": {
-        "node": ">=14"
-      },
-      "peerDependencies": {
-        "@opentelemetry/api": ">=1.3.0 <1.10.0"
-      }
-    },
-    "node_modules/@opentelemetry/semantic-conventions": {
-      "version": "1.28.0",
-      "resolved": "https://registry.npmjs.org/@opentelemetry/semantic-conventions/-/semantic-conventions-1.28.0.tgz",
-      "integrity": "sha512-lp4qAiMTD4sNWW4DbKLBkfiMZ4jbAboJIGOQr5DvciMRI494OapieI9qiODpOt0XBr1LjIDy1xAGAnVs5supTA==",
-      "dev": true,
-      "license": "Apache-2.0",
-      "engines": {
-        "node": ">=14"
-      }
-    },
    "node_modules/@protobufjs/aspromise": {
      "version": "1.1.2",
      "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
@@ -11,7 +11,7 @@
    "ann"
  ],
  "private": false,
-  "version": "0.32.0-beta.0",
+  "version": "0.31.0-beta.4",
  "main": "dist/index.js",
  "exports": {
    ".": "./dist/index.js",
@@ -44,7 +44,6 @@
    "@biomejs/biome": "^1.7.3",
    "@jest/globals": "^29.7.0",
    "@napi-rs/cli": "3.7.0",
-    "@opentelemetry/sdk-metrics": "^1.30.0",
    "@types/axios": "^0.14.0",
    "@types/jest": "^29.1.2",
    "@types/node": "22.7.4",
@@ -93,7 +92,6 @@
    "version": "napi version"
  },
  "dependencies": {
-    "@opentelemetry/api": "^1.9.0",
    "reflect-metadata": "^0.2.2"
  },
  "optionalDependencies": {
@@ -8,9 +8,6 @@ importers:

  .:
    dependencies:
-      '@opentelemetry/api':
-        specifier: ^1.9.0
-        version: 1.9.1
      apache-arrow:
        specifier: '>=15.0.0 <=18.1.0'
        version: 18.1.0
@@ -36,9 +33,6 @@ importers:
      '@napi-rs/cli':
        specifier: 3.7.0
        version: 3.7.0(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)(@types/node@22.7.4)
-      '@opentelemetry/sdk-metrics':
-        specifier: ^1.30.0
-        version: 1.30.1(@opentelemetry/api@1.9.1)
      '@types/axios':
        specifier: ^0.14.0
        version: 0.14.4
@@ -1313,32 +1307,6 @@ packages:
  '@octokit/types@16.0.0':
    resolution: {integrity: sha512-sKq+9r1Mm4efXW1FCk7hFSeJo4QKreL/tTbR0rz/qx/r1Oa2VV83LTA/H/MuCOX7uCIJmQVRKBcbmWoySjAnSg==}

-  '@opentelemetry/api@1.9.1':
-    resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==}
-    engines: {node: '>=8.0.0'}
-
-  '@opentelemetry/core@1.30.1':
-    resolution: {integrity: sha512-OOCM2C/QIURhJMuKaekP3TRBxBKxG/TWWA0TL2J6nXUtDnuCtccy49LUJF8xPFXMX+0LMcxFpCo8M9cGY1W6rQ==}
-    engines: {node: '>=14'}
-    peerDependencies:
-      '@opentelemetry/api': '>=1.0.0 <1.10.0'
-
-  '@opentelemetry/resources@1.30.1':
-    resolution: {integrity: sha512-5UxZqiAgLYGFjS4s9qm5mBVo433u+dSPUFWVWXmLAD4wB65oMCoXaJP1KJa9DIYYMeHu3z4BZcStG3LC593cWA==}
-    engines: {node: '>=14'}
-    peerDependencies:
-      '@opentelemetry/api': '>=1.0.0 <1.10.0'
-
-  '@opentelemetry/sdk-metrics@1.30.1':
-    resolution: {integrity: sha512-q9zcZ0Okl8jRgmy7eNW3Ku1XSgg3sDLa5evHZpCwjspw7E8Is4K/haRPDJrBcX3YSn/Y7gUvFnByNYEKQNbNog==}
-    engines: {node: '>=14'}
-    peerDependencies:
-      '@opentelemetry/api': '>=1.3.0 <1.10.0'
-
-  '@opentelemetry/semantic-conventions@1.28.0':
-    resolution: {integrity: sha512-lp4qAiMTD4sNWW4DbKLBkfiMZ4jbAboJIGOQr5DvciMRI494OapieI9qiODpOt0XBr1LjIDy1xAGAnVs5supTA==}
-    engines: {node: '>=14'}
-
  '@protobufjs/aspromise@1.1.2':
    resolution: {integrity: sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==}

@@ -4957,27 +4925,6 @@ snapshots:
    dependencies:
      '@octokit/openapi-types': 27.0.0

-  '@opentelemetry/api@1.9.1': {}
-
-  '@opentelemetry/core@1.30.1(@opentelemetry/api@1.9.1)':
-    dependencies:
-      '@opentelemetry/api': 1.9.1
-      '@opentelemetry/semantic-conventions': 1.28.0
-
-  '@opentelemetry/resources@1.30.1(@opentelemetry/api@1.9.1)':
-    dependencies:
-      '@opentelemetry/api': 1.9.1
-      '@opentelemetry/core': 1.30.1(@opentelemetry/api@1.9.1)
-      '@opentelemetry/semantic-conventions': 1.28.0
-
-  '@opentelemetry/sdk-metrics@1.30.1(@opentelemetry/api@1.9.1)':
-    dependencies:
-      '@opentelemetry/api': 1.9.1
-      '@opentelemetry/core': 1.30.1(@opentelemetry/api@1.9.1)
-      '@opentelemetry/resources': 1.30.1(@opentelemetry/api@1.9.1)
-
-  '@opentelemetry/semantic-conventions@1.28.0': {}
-
  '@protobufjs/aspromise@1.1.2':
    optional: true

@@ -12,7 +12,6 @@ mod header;
 mod index;
 mod iterator;
 pub mod merge;
-pub mod otel;
 pub mod permutation;
 mod query;
 pub mod remote;
@@ -3,7 +3,7 @@

 use std::time::Duration;

-use lancedb::{ipc::ipc_file_to_batches, table::merge::MergeInsertBuilder};
+use lancedb::{arrow::IntoArrow, ipc::ipc_file_to_batches, table::merge::MergeInsertBuilder};
 use napi::bindgen_prelude::*;
 use napi_derive::napi;

@@ -66,9 +66,11 @@ impl NativeMergeInsertBuilder {

    #[napi(catch_unwind)]
    pub async fn execute(&self, buf: Buffer) -> napi::Result<MergeResult> {
-        let data = ipc_file_to_batches(buf.to_vec()).map_err(|e| {
-            napi::Error::from_reason(format!("Failed to read IPC file: {}", convert_error(&e)))
-        })?;
+        let data = ipc_file_to_batches(buf.to_vec())
+            .and_then(IntoArrow::into_arrow)
+            .map_err(|e| {
+                napi::Error::from_reason(format!("Failed to read IPC file: {}", convert_error(&e)))
+            })?;

        let this = self.clone();

@@ -1,119 +0,0 @@
-// SPDX-License-Identifier: Apache-2.0
-// SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-//! Node.js bindings over [`lancedb::metrics_otel`].
-//!
-//! The aggregation, catalog, and histogram bucketing all live in the LanceDB
-//! core crate; this module only converts the core snapshot types into napi
-//! objects and exposes the three entry points to JavaScript, where
-//! `lancedb/otel.ts` bridges them into the user's OpenTelemetry `MeterProvider`.
-
-use std::collections::HashMap;
-
-use lancedb::metrics_otel::{MetricPoint as CoreMetricPoint, MetricValue};
-use napi_derive::napi;
-
-/// One cumulative histogram bucket: all samples with value `<= le`.
-#[napi(object)]
-pub struct MetricBucket {
-    /// The inclusive upper bound of the bucket, or `"+Inf"` for the final bucket.
-    pub le: String,
-    /// Cumulative number of samples less than or equal to `le`.
-    pub cumulative_count: f64,
-}
-
-/// One aggregated metric data point. For counters and gauges only `value` is
-/// set; for histograms `buckets` (cumulative `le` counts), `count`, and `sum`
-/// are set.
-#[napi(object)]
-pub struct MetricPoint {
-    pub name: String,
-    pub kind: String,
-    pub attributes: HashMap<String, String>,
-    pub value: Option<f64>,
-    pub buckets: Option<Vec<MetricBucket>>,
-    pub count: Option<f64>,
-    pub sum: Option<f64>,
-}
-
-impl From<CoreMetricPoint> for MetricPoint {
-    fn from(point: CoreMetricPoint) -> Self {
-        let kind = point.kind.as_str().to_string();
-        let (value, buckets, count, sum) = match point.value {
-            MetricValue::Scalar(v) => (Some(v), None, None, None),
-            MetricValue::Histogram {
-                buckets,
-                count,
-                sum,
-            } => (
-                None,
-                Some(
-                    buckets
-                        .into_iter()
-                        // Counts stay well within the f64-exact integer range
-                        // (2^53), so this cast is lossless in practice and keeps
-                        // the values plain JS numbers for OpenTelemetry.
-                        .map(|(le, cumulative_count)| MetricBucket {
-                            le,
-                            cumulative_count: cumulative_count as f64,
-                        })
-                        .collect(),
-                ),
-                Some(count as f64),
-                Some(sum),
-            ),
-        };
-        Self {
-            name: point.name,
-            kind,
-            attributes: point.attributes,
-            value,
-            buckets,
-            count,
-            sum,
-        }
-    }
-}
-
-/// A described metric, used by the JavaScript layer to create instruments up front.
-#[napi(object)]
-pub struct MetricDescription {
-    pub name: String,
-    pub kind: String,
-    pub unit: Option<String>,
-    pub description: String,
-}
-
-/// Install the LanceDB metrics recorder as the process-global `metrics` recorder.
-///
-/// Returns `true` if the recorder is installed (now or previously). Returns
-/// `false` if a *different* recorder is already installed — `metrics` allows
-/// only one global recorder per process, so LanceDB cannot coexist with another.
-#[napi]
-pub fn register_lancedb_metrics_recorder() -> bool {
-    lancedb::metrics_otel::register_metrics_recorder()
-}
-
-/// The catalog of described LanceDB metrics. Empty until the recorder is installed.
-#[napi]
-pub fn lancedb_metrics_catalog() -> Vec<MetricDescription> {
-    lancedb::metrics_otel::metrics_catalog()
-        .into_iter()
-        .map(|desc| MetricDescription {
-            name: desc.name,
-            kind: desc.kind.as_str().to_string(),
-            unit: desc.unit,
-            description: desc.description,
-        })
-        .collect()
-}
-
-/// A point-in-time snapshot of every recorded metric. Empty until the recorder
-/// is installed.
-#[napi]
-pub fn snapshot_lancedb_metrics() -> Vec<MetricPoint> {
-    lancedb::metrics_otel::snapshot_metrics()
-        .into_iter()
-        .map(MetricPoint::from)
-        .collect()
-}
@@ -16,7 +16,6 @@ pub struct SplitRandomOptions {
    pub counts: Option<Vec<i64>>,
    pub fixed: Option<i64>,
    pub seed: Option<i64>,
-    pub clump_size: Option<i64>,
    pub split_names: Option<Vec<String>>,
 }

@@ -126,15 +125,10 @@ impl PermutationBuilder {
        };

        let seed = options.seed.map(|s| s as u64);
-        let clump_size = options.clump_size.map(|c| c as u64);

        self.modify(|builder| {
            builder.with_split_strategy(
-                SplitStrategy::Random {
-                    seed,
-                    sizes,
-                    clump_size,
-                },
+                SplitStrategy::Random { seed, sizes },
                options.split_names.clone(),
            )
        })
@@ -411,16 +411,6 @@ impl Table {
            .default_error()
    }

-    #[napi(catch_unwind)]
-    pub async fn get_lsm_write_spec(&self) -> napi::Result<Option<LsmWriteSpec>> {
-        let spec = self
-            .inner_ref()?
-            .get_lsm_write_spec()
-            .await
-            .default_error()?;
-        Ok(spec.map(LsmWriteSpec::from))
-    }
-
    #[napi(catch_unwind)]
    pub async fn close_lsm_writers(&self) -> napi::Result<()> {
        self.inner_ref()?.close_lsm_writers().await.default_error()
@@ -738,47 +728,6 @@ impl TryFrom<LsmWriteSpec> for lancedb::table::LsmWriteSpec {
    }
 }

-impl From<lancedb::table::LsmWriteSpec> for LsmWriteSpec {
-    fn from(spec: lancedb::table::LsmWriteSpec) -> Self {
-        use lancedb::table::LsmWriteSpec as Native;
-        match spec {
-            Native::Bucket {
-                column,
-                num_buckets,
-                maintained_indexes,
-                writer_config_defaults,
-            } => Self {
-                spec_type: "bucket".to_string(),
-                column: Some(column),
-                num_buckets: Some(num_buckets),
-                maintained_indexes: Some(maintained_indexes),
-                writer_config_defaults: Some(writer_config_defaults),
-            },
-            Native::Identity {
-                column,
-                maintained_indexes,
-                writer_config_defaults,
-            } => Self {
-                spec_type: "identity".to_string(),
-                column: Some(column),
-                num_buckets: None,
-                maintained_indexes: Some(maintained_indexes),
-                writer_config_defaults: Some(writer_config_defaults),
-            },
-            Native::Unsharded {
-                maintained_indexes,
-                writer_config_defaults,
-            } => Self {
-                spec_type: "unsharded".to_string(),
-                column: None,
-                num_buckets: None,
-                maintained_indexes: Some(maintained_indexes),
-                writer_config_defaults: Some(writer_config_defaults),
-            },
-        }
-    }
-}
-
 /// Statistics about a compaction operation.
 #[napi(object)]
 #[derive(Clone, Debug)]
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.35.0-beta.1"
+current_version = "0.34.0-beta.4"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-python"
-version = "0.35.0-beta.1"
+version = "0.34.0-beta.4"
 publish = false
 edition.workspace = true
 description = "Python bindings for LanceDB"
@@ -47,6 +47,6 @@ pyo3-build-config = { version = "0.28", features = [
 ] }

 [features]
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/cos", "lancedb/goosefs", "lancedb/metrics-otel"]
+default = ["remote",  "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
 fp16kernels = ["lancedb/fp16kernels"]
 remote = ["lancedb/remote"]
@@ -1,135 +0,0 @@
-#!/usr/bin/env python3
-# SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-"""Benchmark for StreamingDataset throughput.
-
-Sweeps read_batch_size from 1 to 16384 to show how amortising the per-request
-overhead scales.  Each row at each chunk size is timed via the real
-StreamingDataset so the numbers reflect production code.
-
-Run with:
-    cd python
-    uv run --extra tests benchmarks/bench_streaming_dataloader.py
-
-Optional env vars:
-    BENCH_NUM_ROWS   — total rows in the table (default 49152 = 24 × 2048)
-    BENCH_NUM_SPLITS — number of splits (default 24)
-    BENCH_STEPS      — round-robin cycles to time per chunk size (default 100)
-    BENCH_ROW_BYTES  — approximate bytes per row padded with a binary column
-                       (default 4096, mimics a small embedding/image patch)
-"""
-
-import os
-import time
-import tempfile
-
-import pyarrow as pa
-import lancedb
-
-from lancedb.streaming import StreamingDataset
-
-# ---------------------------------------------------------------------------
-# Configuration
-# ---------------------------------------------------------------------------
-
-NUM_SPLITS = int(os.environ.get("BENCH_NUM_SPLITS", 24))
-# Default: 2048 rows per split so every chunk size up to 16Ki has ≥1 full
-# chunk (except 16Ki itself which gets a single full-split fetch — still valid).
-NUM_ROWS = int(os.environ.get("BENCH_NUM_ROWS", NUM_SPLITS * 2048))
-STEPS = int(os.environ.get("BENCH_STEPS", 100))
-ROW_BYTES = int(os.environ.get("BENCH_ROW_BYTES", 4096))
-
-assert NUM_ROWS % NUM_SPLITS == 0, "NUM_ROWS must be divisible by NUM_SPLITS"
-
-CHUNK_SIZES = [1, 4, 16, 64, 256, 1024, 4096, 16384]
-
-
-# ---------------------------------------------------------------------------
-# Table helpers
-# ---------------------------------------------------------------------------
-
-
-def make_table(db_path: str) -> lancedb.table.Table:
-    db = lancedb.connect(db_path)
-    payload = b"x" * ROW_BYTES
-    data = pa.table(
-        {
-            "id": pa.array(range(NUM_ROWS), type=pa.int32()),
-            "payload": pa.array([payload] * NUM_ROWS, type=pa.large_binary()),
-        }
-    )
-    return db.create_table("bench", data, mode="overwrite")
-
-
-# ---------------------------------------------------------------------------
-# Timing
-# ---------------------------------------------------------------------------
-
-
-def bench_chunk(table, chunk_size: int, steps: int) -> tuple[int, float]:
-    """Return (rows_drained, elapsed_seconds) for one timed run."""
-    total_rows = steps * NUM_SPLITS
-    ds = StreamingDataset(
-        table, num_splits=NUM_SPLITS, shuffle_seed=42, read_batch_size=chunk_size
-    )
-    count = 0
-    t0 = time.perf_counter()
-    for _ in ds:
-        count += 1
-        if count >= total_rows:
-            break
-    return count, time.perf_counter() - t0
-
-
-# ---------------------------------------------------------------------------
-# Main
-# ---------------------------------------------------------------------------
-
-
-def main() -> None:
-    rows_per_split = NUM_ROWS // NUM_SPLITS
-    print("Benchmark config:")
-    print(
-        f"  NUM_ROWS={NUM_ROWS}  NUM_SPLITS={NUM_SPLITS}  "
-        f"rows/split={rows_per_split}  STEPS={STEPS}  ROW_BYTES={ROW_BYTES}"
-    )
-    print(f"  ~{NUM_ROWS * ROW_BYTES / 1024 / 1024:.1f} MB total table size")
-    print()
-
-    with tempfile.TemporaryDirectory() as tmp:
-        print("Creating table...", flush=True)
-        table = make_table(tmp)
-
-        cols = (
-            f"{'chunk':>6}  {'rows':>6}  {'elapsed':>8}  {'rows/s':>10}  {'ms/step':>9}"
-        )
-        print(f"\n{cols}")
-        print("-" * 52)
-
-        for chunk in CHUNK_SIZES:
-            # Warm-up pass (one step's worth of rows)
-            warmup_ds = StreamingDataset(
-                table, num_splits=NUM_SPLITS, shuffle_seed=42, read_batch_size=chunk
-            )
-            warmup_count = 0
-            for _ in warmup_ds:
-                warmup_count += 1
-                if warmup_count >= NUM_SPLITS:
-                    break
-
-            drained, elapsed = bench_chunk(table, chunk, STEPS)
-            rows_per_sec = drained / elapsed if elapsed > 0 else float("inf")
-            ms_per_step = elapsed / STEPS * 1000
-
-            print(
-                f"{chunk:>6}  {drained:>6}  {elapsed:>7.3f}s  "
-                f"{rows_per_sec:>10.0f}  {ms_per_step:>8.1f}ms"
-            )
-
-    print()
-    print("Done.")
-
-
-if __name__ == "__main__":
-    main()
@@ -47,10 +47,6 @@ repository = "https://github.com/lancedb/lancedb"
 pylance = [
    "pylance>=5.0.0b5",
 ]
-# A library only needs the OpenTelemetry API; the application supplies and
-# configures the SDK (the actual exporter/reader). See
-# https://opentelemetry.io/docs/languages/python/instrumentation/
-otel = ["opentelemetry-api"]
 tests = [
    "aiohttp>=3.9.0",
    "boto3>=1.28.57",
@@ -65,7 +61,6 @@ tests = [
    "pylance>=5.0.0b5",
    "requests>=2.31.0",
    "datafusion>=52,<53",
-    "opentelemetry-sdk>=1.30.0",
 ]
 dev = [
    "ruff>=0.3.0",
@@ -17,6 +17,17 @@ from .db import AsyncConnection, DBConnection, LanceDBConnection
 from .remote import ClientConfig
 from .remote.db import RemoteDBConnection
 from .expr import Expr, col, lit, func
+from .udf import (
+    udf,
+    table_udf,
+    Udf,
+    JobHandle,
+    JobFailedError,
+    MaterializedView,
+    AsyncJobHandle,
+    AsyncMaterializedView,
+)
+from .lineage import Lineage, Node, Edge, FunctionRef
 from .schema import vector
 from .table import AsyncTable, Table
 from ._lancedb import Session
@@ -149,14 +160,8 @@ def connect(

    For object storage, use a URI prefix:

-    >>> db = lancedb.connect(  # doctest: +SKIP
-    ...     "s3://my-bucket/lancedb",
-    ...     storage_options={
-    ...         "aws_access_key_id": "***",
-    ...         "aws_secret_access_key": "***",
-    ...         "aws_region": "us-east-1",
-    ...     },
-    ... )
+    >>> db = lancedb.connect("s3://my-bucket/lancedb",
+    ...                      storage_options={"aws_access_key_id": "***"})

    For tests and temporary data, use an in-memory database:

@@ -454,6 +459,18 @@ async def connect_async(


 __all__ = [
+    "udf",
+    "table_udf",
+    "Udf",
+    "JobHandle",
+    "JobFailedError",
+    "MaterializedView",
+    "AsyncJobHandle",
+    "AsyncMaterializedView",
+    "Lineage",
+    "Node",
+    "Edge",
+    "FunctionRef",
    "connect",
    "connect_async",
    "connect_namespace",
@@ -1,5 +1,4 @@
-from datetime import date, datetime, timedelta
-from decimal import Decimal
+from datetime import datetime, timedelta
 from typing import Dict, List, Optional, Tuple, Any, TypedDict, Union, Literal

 import pyarrow as pa
@@ -30,25 +29,6 @@ IvfHnswPq: type[HnswPq] = HnswPq
 IvfHnswSq: type[HnswSq] = HnswSq
 IvfHnswFlat: type[HnswFlat] = HnswFlat

-class MetricPoint:
-    name: str
-    kind: str
-    attributes: Dict[str, str]
-    value: Optional[float]
-    buckets: Optional[List[Tuple[str, int]]]
-    count: Optional[int]
-    sum: Optional[float]
-
-class MetricDescription:
-    name: str
-    kind: str
-    unit: Optional[str]
-    description: str
-
-def register_lancedb_metrics_recorder() -> bool: ...
-def lancedb_metrics_catalog() -> List[MetricDescription]: ...
-def snapshot_lancedb_metrics() -> List[MetricPoint]: ...
-
 class PyExpr:
    """A type-safe DataFusion expression node (Rust-side handle)."""

@@ -73,9 +53,7 @@ class PyExpr:
    def to_sql(self) -> str: ...

 def expr_col(name: str) -> PyExpr: ...
-def expr_lit(
-    value: Union[bool, int, float, str, bytes, date, datetime, Decimal],
-) -> PyExpr: ...
+def expr_lit(value: Union[bool, int, float, str, bytes]) -> PyExpr: ...
 def expr_func(name: str, args: List[PyExpr]) -> PyExpr: ...

 class Session:
@@ -248,7 +226,6 @@ class Table:
    async def set_unenforced_primary_key(self, columns: List[str]) -> None: ...
    async def set_lsm_write_spec(self, spec: LsmWriteSpec) -> None: ...
    async def unset_lsm_write_spec(self) -> None: ...
-    async def get_lsm_write_spec(self) -> Optional[LsmWriteSpec]: ...
    async def close_lsm_writers(self) -> None: ...
    @property
    def tags(self) -> Tags: ...
@@ -305,23 +282,6 @@ async def connect(
    namespace_client_properties: Optional[Dict[str, str]] = None,
    oauth_config: Optional[Any] = None,
 ) -> Connection: ...
-def connect_namespace(
-    namespace_client_impl: str,
-    namespace_client_properties: Dict[str, str],
-    read_consistency_interval: Optional[float] = None,
-    storage_options: Optional[Dict[str, str]] = None,
-    session: Optional[Session] = None,
-    namespace_client_pushdown_operations: Optional[List[str]] = None,
-) -> Connection: ...
-def connect_namespace_client(
-    namespace_client: Any,
-    read_consistency_interval: Optional[float] = None,
-    storage_options: Optional[Dict[str, str]] = None,
-    session: Optional[Session] = None,
-    namespace_client_pushdown_operations: Optional[List[str]] = None,
-    namespace_client_impl: Optional[str] = None,
-    namespace_client_properties: Optional[Dict[str, str]] = None,
-) -> Connection: ...

 class RecordBatchStream:
    @property
@@ -65,6 +65,7 @@ if TYPE_CHECKING:
    from .common import DATA, URI
    from .embeddings import EmbeddingFunctionConfig
    from ._lancedb import Session
+    from .udf import MaterializedView, AsyncMaterializedView

 from .namespace_utils import (
    _normalize_create_namespace_mode,
@@ -562,6 +563,259 @@ class DBConnection(EnforceOverrides):
        """
        raise NotImplementedError("serialize is not supported for this connection type")

+    # -- Derived compute: functions, materialized views, jobs -------------
+    # Server-backed features (LanceDB Enterprise / Cloud); local
+    # connections raise NotImplementedError for now.
+
+    def create_function(
+        self,
+        name,
+        language: str = "python",
+        return_type: Optional[str] = None,
+        body: Optional[str] = None,
+        options: Optional[Dict[str, str]] = None,
+        *,
+        replace: bool = False,
+    ):
+        """Register a UDF (CREATE FUNCTION).
+
+        Pass a ``@udf`` / ``@table_udf``-decorated function (preferred):
+
+            db.create_function(embed)
+
+        or the explicit fields:
+
+        Parameters
+        ----------
+        name: str or Udf
+            A decorated UDF object, or the function name.
+        language: str
+            Implementation language (currently "python").
+        return_type: str
+            SQL return type, e.g. "FLOAT", "FLOAT[1536]",
+            "STRUCT(a FLOAT, b VARCHAR)", "TABLE(chunk VARCHAR, idx INT)".
+        body: str
+            Function body: source text, or base64 cloudpickle bytes when
+            options["body_format"] == "cloudpickle".
+        options: dict, optional
+            input_columns, pip, num_gpus, batch_size, timeout,
+            error_policy, docker_image, body_format, ...
+        replace: bool
+            Drop an existing function of the same name first.
+        """
+        from .udf import Udf
+
+        if isinstance(name, Udf):
+            req = name.create_request()
+            name, language, return_type, body, options = (
+                req["name"],
+                req["language"],
+                req["return_type"],
+                req["body"],
+                req["options"],
+            )
+        if replace:
+            try:
+                self.drop_function(name)
+            except Exception:
+                pass
+        LOOP.run(self._conn.create_function(name, language, return_type, body, options))
+
+    def list_functions(self):
+        """List registered functions (SHOW FUNCTIONS)."""
+        return LOOP.run(self._conn.list_functions())
+
+    def drop_function(self, name: str):
+        """Drop a registered function (DROP FUNCTION)."""
+        LOOP.run(self._conn.drop_function(name))
+
+    def create_materialized_view(
+        self,
+        name: str,
+        source=None,
+        select=None,
+        *,
+        query: Optional[str] = None,
+        where: Optional[str] = None,
+        auto_refresh: bool = False,
+        with_no_data: bool = False,
+        replace: bool = False,
+        partition_by: Optional[str] = None,
+    ) -> "MaterializedView":
+        """Create a materialized view (CREATE MATERIALIZED VIEW); returns a
+        `MaterializedView` handle (``.wait()`` blocks until it is populated).
+
+        Two ways to specify the view body:
+
+        - ergonomic: pass ``source`` (a table name or table) and ``select``
+          items -- column names, expression strings ("embed(body)"),
+          (alias, expression) tuples, or ``@udf`` / ``@table_udf`` objects.
+          The SELECT is assembled and parsed server-side (one parser, shared
+          with SQL).
+        - raw: pass ``query=`` with a full SELECT, e.g.
+          "SELECT id, embed(body) AS vec FROM articles WHERE id > 1".
+
+        `partition_by` partitions the view's (single) table function on a source
+        column. If that column has an IVF vector index the server partitions by
+        its index clusters (image-dedup style); otherwise it groups by distinct
+        value. (Geneva's `partition_by` and `partition_by_indexed_column` unify
+        here -- the engine picks the strategy from the column.)
+        """
+        from .udf import build_view_query, MaterializedView
+
+        if query is None:
+            if source is None or select is None:
+                raise ValueError(
+                    "create_materialized_view needs either query= or both "
+                    "source and select"
+                )
+            query = build_view_query(source, select)
+        if where:
+            query += f" WHERE {where}"
+        if replace:
+            self._drop_view_if_exists(name)
+        job_id = LOOP.run(
+            self._conn.create_materialized_view(
+                name,
+                query=query,
+                auto_refresh=auto_refresh,
+                with_no_data=with_no_data,
+                partition_by=partition_by,
+            )
+        )
+        return MaterializedView(self, name, job_id=job_id)
+
+    def _drop_view_if_exists(self, name: str) -> None:
+        # `replace=True` is "drop if present"; only a not-found error is
+        # benign here. Anything else (perms, server fault) must surface rather
+        # than be masked by a later create failure.
+        try:
+            self.drop_materialized_view(name)
+        except Exception as e:
+            msg = str(e).lower()
+            if "not found" not in msg and "does not exist" not in msg:
+                raise
+
+    def job(self, job_id: str):
+        """A `JobHandle` for reconnecting to an inflight job by id -- e.g. an
+        id you stored, or one returned from the SQL / REST surface. Submit
+        methods (`refresh_column`, `MaterializedView.refresh`) already return a
+        handle directly, so you do not need this to wait on a fresh submission."""
+        from .udf import JobHandle
+
+        return JobHandle(self, job_id)
+
+    def lineage(
+        self,
+        table: str,
+        column: Optional[str] = None,
+        *,
+        direction: Optional[str] = None,
+        depth: Optional[int] = None,
+    ):
+        """Derived-compute lineage of a table/view, or one of its columns:
+        upstream sources, downstream dependents, and the function version +
+        location that produced each derived column (with a drift flag). Returns
+        a `Lineage`. `direction` is "upstream" | "downstream" | "both" (server
+        default both); `depth` limits column-hops (transitive when omitted)."""
+        # `self._conn` is the AsyncConnection; drive its async `lineage`
+        # (which parses the JSON) on the loop, mirroring create_materialized_view.
+        return LOOP.run(
+            self._conn.lineage(table, column, direction=direction, depth=depth)
+        )
+
+    def _refresh_materialized_view(
+        self,
+        name: str,
+        *,
+        full: bool = False,
+        src_version: Optional[int] = None,
+        num_workers: Optional[int] = None,
+        max_workers: Optional[int] = None,
+    ) -> str:
+        """Internal: submit a materialized-view refresh, return the job id.
+        The public surface is ``MaterializedView.refresh()`` (which returns a
+        `JobHandle`); this stays private so refresh is only reached through the
+        handle.
+
+        ``full=True`` forces a full rebuild (recompute and replace every row)
+        instead of the default incremental refresh.
+        """
+        return LOOP.run(
+            self._conn._refresh_materialized_view(
+                name,
+                full=full,
+                src_version=src_version,
+                num_workers=num_workers,
+                max_workers=max_workers,
+            )
+        )
+
+    def explain_refresh_materialized_view(
+        self,
+        name: str,
+        *,
+        full: bool = False,
+        src_version: Optional[int] = None,
+    ):
+        """Plan a refresh without running it (EXPLAIN REFRESH). Returns a
+        plan with .has_work / .source_version / .last_refreshed_version /
+        .full_refresh / .rebuild / .units_total. `full=True` plans a full
+        rebuild (incremental planning needs stable row IDs on the source)."""
+        return LOOP.run(
+            self._conn.explain_refresh_materialized_view(
+                name, full=full, src_version=src_version
+            )
+        )
+
+    def alter_materialized_view(self, name: str, *, auto_refresh: bool):
+        """Update a materialized view's options (ALTER MATERIALIZED VIEW)."""
+        LOOP.run(self._conn.alter_materialized_view(name, auto_refresh=auto_refresh))
+
+    def drop_materialized_view(self, name: str):
+        """Drop a materialized view definition (DROP MATERIALIZED VIEW)."""
+        LOOP.run(self._conn.drop_materialized_view(name))
+
+    def list_materialized_views(self):
+        """List registered materialized view definitions."""
+        return LOOP.run(self._conn.list_materialized_views())
+
+    def list_jobs(self):
+        """List inflight server-side jobs across the database's tables."""
+        return LOOP.run(self._conn.list_jobs())
+
+    def get_job(self, job_id: str, table: "str | None" = None):
+        """Look up one server-side job by id (the wait()/status poll path).
+
+        Passing ``table`` (the job's table) lets the server answer with an O(1)
+        single-node read instead of scanning the database's active jobs.
+        Returns the job's status, or None if it's unknown or no longer active.
+        """
+        return LOOP.run(self._conn.get_job(job_id, table))
+
+    def cancel_job(self, job_id: str) -> bool:
+        """Cancel an inflight server-side job by id (CANCEL JOB).
+
+        Returns True if a matching inflight job was found and flagged for
+        cancellation, False if none was inflight (already finished or
+        unknown id) -- cancellation is best-effort.
+        """
+        return LOOP.run(self._conn.cancel_job(job_id))
+
+    def job_history(self, job_id: "str | None" = None):
+        """Durable history of completed server-side jobs (SHOW JOB HISTORY).
+
+        Pass ``job_id`` to narrow to a single job. Unlike :meth:`list_jobs`
+        (live, inflight) these are the terminal records.
+        """
+        return LOOP.run(self._conn.job_history(job_id))
+
+    def errors(self, job_id: "str | None" = None, table: "str | None" = None):
+        """Per-row UDF errors recorded by ``error_policy=skip`` (SHOW ERRORS),
+        optionally filtered by ``job_id`` and/or ``table``.
+        """
+        return LOOP.run(self._conn.errors(job_id, table))
+

 class LanceDBConnection(DBConnection):
    """
@@ -1787,6 +2041,200 @@ class AsyncConnection(object):
        )
        return AsyncTable(table)

+    # -- Derived compute: functions, materialized views, jobs -------------
+    # Server-backed features (LanceDB Enterprise / Cloud); local
+    # connections raise NotImplementedError for now.
+
+    async def create_function(
+        self,
+        name,
+        language: str = "python",
+        return_type: Optional[str] = None,
+        body: Optional[str] = None,
+        options: Optional[Dict[str, str]] = None,
+        *,
+        replace: bool = False,
+    ):
+        """Register a UDF (CREATE FUNCTION). Accepts a ``@udf``/``@table_udf``
+        object (preferred) or the explicit (name, language, return_type, body,
+        options)."""
+        from .udf import Udf
+
+        if isinstance(name, Udf):
+            req = name.create_request()
+            name, language, return_type, body, options = (
+                req["name"],
+                req["language"],
+                req["return_type"],
+                req["body"],
+                req["options"],
+            )
+        if replace:
+            try:
+                await self.drop_function(name)
+            except Exception:
+                pass
+        await self._inner.create_function(name, language, return_type, body, options)
+
+    async def list_functions(self):
+        """List registered functions (SHOW FUNCTIONS)."""
+        return await self._inner.list_functions()
+
+    async def drop_function(self, name: str):
+        """Drop a registered function (DROP FUNCTION)."""
+        await self._inner.drop_function(name)
+
+    async def create_materialized_view(
+        self,
+        name: str,
+        source=None,
+        select=None,
+        *,
+        query: Optional[str] = None,
+        where: Optional[str] = None,
+        auto_refresh: bool = False,
+        with_no_data: bool = False,
+        replace: bool = False,
+        partition_by: Optional[str] = None,
+    ) -> "AsyncMaterializedView":
+        """Create a materialized view; returns an `AsyncMaterializedView`
+        handle (``.wait()`` blocks until populated). Pass either ``query=`` (a
+        full SELECT) or ``source`` + ``select`` items; `partition_by`
+        partitions the view's table function on a source column (index-cluster
+        if the column is IVF-indexed, else distinct-value). See the sync
+        method for the select grammar."""
+        from .udf import build_view_query, AsyncMaterializedView
+
+        if query is None:
+            if source is None or select is None:
+                raise ValueError(
+                    "create_materialized_view needs either query= or both "
+                    "source and select"
+                )
+            query = build_view_query(source, select)
+        if where:
+            query += f" WHERE {where}"
+        if replace:
+            try:
+                await self.drop_materialized_view(name)
+            except Exception as e:
+                msg = str(e).lower()
+                if "not found" not in msg and "does not exist" not in msg:
+                    raise
+        job_id = await self._inner.create_materialized_view(
+            name,
+            query,
+            auto_refresh=auto_refresh,
+            with_no_data=with_no_data,
+            partition_by=partition_by,
+        )
+        return AsyncMaterializedView(self, name, job_id=job_id)
+
+    def job(self, job_id: str):
+        """An `AsyncJobHandle` for reconnecting to an inflight job by id (a
+        stored id, or one from the SQL / REST surface). Submit methods already
+        return a handle, so this is only needed to re-attach to an existing
+        job."""
+        from .udf import AsyncJobHandle
+
+        return AsyncJobHandle(self, job_id)
+
+    async def lineage(
+        self,
+        table: str,
+        column: Optional[str] = None,
+        *,
+        direction: Optional[str] = None,
+        depth: Optional[int] = None,
+    ):
+        """Derived-compute lineage of a table/view (or column). See the sync
+        `Connection.lineage`. Returns a `Lineage`."""
+        from .lineage import Lineage
+
+        raw = await self._inner.table_lineage(table, column, direction, depth)
+        return Lineage.from_json(raw)
+
+    async def _refresh_materialized_view(
+        self,
+        name: str,
+        *,
+        full: bool = False,
+        src_version: Optional[int] = None,
+        num_workers: Optional[int] = None,
+        max_workers: Optional[int] = None,
+    ) -> str:
+        """Internal: submit a refresh, return the job id. The public surface is
+        ``AsyncMaterializedView.refresh()`` (returns an `AsyncJobHandle`).
+
+        ``full=True`` forces a full rebuild (recompute and replace every row)
+        instead of the default incremental refresh.
+        """
+        return await self._inner.refresh_materialized_view(
+            name,
+            full=full,
+            src_version=src_version,
+            num_workers=num_workers,
+            max_workers=max_workers,
+        )
+
+    async def explain_refresh_materialized_view(
+        self,
+        name: str,
+        *,
+        full: bool = False,
+        src_version: Optional[int] = None,
+    ):
+        """Plan a refresh without running it (EXPLAIN REFRESH)."""
+        return await self._inner.explain_refresh_materialized_view(
+            name, full=full, src_version=src_version
+        )
+
+    async def alter_materialized_view(self, name: str, *, auto_refresh: bool):
+        """Update a materialized view's options."""
+        await self._inner.alter_materialized_view(name, auto_refresh)
+
+    async def drop_materialized_view(self, name: str):
+        """Drop a materialized view definition."""
+        await self._inner.drop_materialized_view(name)
+
+    async def list_materialized_views(self):
+        """List registered materialized view definitions."""
+        return await self._inner.list_materialized_views()
+
+    async def list_jobs(self):
+        """List inflight server-side jobs across the database's tables."""
+        return await self._inner.list_jobs()
+
+    async def get_job(self, job_id: str, table: "str | None" = None):
+        """Look up one server-side job by id (the wait()/status poll path).
+        ``table`` (the job's table) enables an O(1) server-side lookup.
+        Returns the job's status, or None if unknown / no longer active."""
+        return await self._inner.get_job(job_id, table)
+
+    async def cancel_job(self, job_id: str) -> bool:
+        """Cancel an inflight server-side job by id (CANCEL JOB).
+
+        Returns True if a matching inflight job was found and flagged for
+        cancellation, False otherwise (best-effort).
+        """
+        return await self._inner.cancel_job(job_id)
+
+    async def job_history(self, job_id: "str | None" = None):
+        """Durable history of completed server-side jobs (SHOW JOB HISTORY).
+
+        Reads each table's durable job-history store. Pass ``job_id`` to narrow
+        to a single job. Unlike :meth:`list_jobs` (live, inflight) these are the
+        terminal records, with created/updated/completed timestamps.
+        """
+        return await self._inner.job_history(job_id)
+
+    async def errors(self, job_id: "str | None" = None, table: "str | None" = None):
+        """Per-row UDF errors recorded by ``error_policy=skip`` (SHOW ERRORS).
+
+        Optionally filtered by ``job_id`` and/or ``table``.
+        """
+        return await self._inner.errors(job_id, table)
+
    async def rename_table(
        self,
        cur_name: str,
@@ -2,7 +2,7 @@
 # SPDX-FileCopyrightText: Copyright The LanceDB Authors

 from functools import cached_property
-from typing import TYPE_CHECKING, Any, List, Optional, Sequence, Union
+from typing import TYPE_CHECKING, List, Optional, Sequence, Union

 import numpy as np

@@ -56,16 +56,6 @@ class OllamaEmbeddings(TextEmbeddingFunction):
        embeddings = self._compute_embedding(texts)
        return list(embeddings)

-    def __getstate__(self) -> dict[str, Any]:
-        state = super().__getstate__()
-        state["__dict__"] = {
-            k: v for k, v in state["__dict__"].items() if k != "_ollama_client"
-        }
-        return state
-
-    def __setstate__(self, state: dict[str, Any]) -> None:
-        super().__setstate__(state)
-
    @cached_property
    def _ollama_client(self) -> "ollama.Client":
        ollama = attempt_import_or_raise("ollama")
@@ -19,8 +19,6 @@ operators::

 from __future__ import annotations

-from datetime import date, datetime
-from decimal import Decimal
 from typing import Iterable, Union

 import pyarrow as pa
@@ -65,7 +63,7 @@ def _coerce(value: "ExprLike") -> "Expr":


 # Type alias used in annotations.
-ExprLike = Union["Expr", bool, int, float, str, bytes, date, datetime, Decimal]
+ExprLike = Union["Expr", bool, int, float, str, bytes]


 class Expr:
@@ -120,18 +118,10 @@ class Expr:
        """Logical AND (``expr_a & expr_b``)."""
        return Expr(self._inner.and_(_coerce(other)._inner))

-    def __rand__(self, other: ExprLike) -> "Expr":
-        """Right-hand logical AND (``True & expr``)."""
-        return Expr(_coerce(other)._inner.and_(self._inner))
-
    def __or__(self, other: "Expr") -> "Expr":
        """Logical OR (``expr_a | expr_b``)."""
        return Expr(self._inner.or_(_coerce(other)._inner))

-    def __ror__(self, other: ExprLike) -> "Expr":
-        """Right-hand logical OR (``False | expr``)."""
-        return Expr(_coerce(other)._inner.or_(self._inner))
-
    def __invert__(self) -> "Expr":
        """Logical NOT (``~expr``)."""
        return Expr(self._inner.not_())
@@ -276,14 +266,13 @@ def col(name: str) -> Expr:
    return Expr(expr_col(name))


-def lit(value: Union[bool, int, float, str, bytes, date, datetime, Decimal]) -> Expr:
+def lit(value: Union[bool, int, float, str, bytes]) -> Expr:
    """Create a literal (constant) value expression.

    Parameters
    ----------
    value:
-        A Python ``bool``, ``int``, ``float``, ``str``, ``bytes``, ``date``,
-        ``datetime``, or ``Decimal``.
+        A Python ``bool``, ``int``, ``float``, ``str``, or ``bytes``.

    Examples
    --------
@@ -291,9 +280,6 @@ def lit(value: Union[bool, int, float, str, bytes, date, datetime, Decimal]) ->
    >>> col("price") * lit(1.1)
    Expr((price * 1.1))
    """
-    if not isinstance(value, (bool, int, float, str, bytes, date, datetime, Decimal)):
-        raise TypeError(f"Unsupported literal type: {type(value).__name__}")
-
    return Expr(expr_lit(value))


@@ -0,0 +1,177 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+"""Client-side model of derived-compute lineage.
+
+`Connection.lineage()` / `Table.lineage()` / `MaterializedView.lineage()` return
+a `Lineage`: the graph of what a column or materialized view derives from
+(upstream), what derives from it (downstream), and -- for each derived column --
+the function that produced it, the version it was produced with, and whether
+that is stale relative to the function the registry now holds.
+
+The server returns this as JSON (the wire contract); these classes deserialize
+it. Nothing here talks to the server.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from typing import List, Optional, Union
+
+
+@dataclass
+class FunctionRef:
+    """The function that produced a derived column, with version + location."""
+
+    name: str
+    #: Version that produced the data (stamped at compute time), if known.
+    as_computed_version: Optional[str] = None
+    #: Version the registry currently holds for this function name.
+    current_version: Optional[str] = None
+    #: True when the column was produced by an older function than the registry
+    #: now holds -- i.e. silently stale; re-refresh to catch up.
+    stale_vs_current: bool = False
+    language: Optional[str] = None
+    docker_image: Optional[str] = None
+    env_digest: Optional[str] = None
+    code_uri: Optional[str] = None
+
+    @classmethod
+    def _from(cls, d: dict) -> "FunctionRef":
+        return cls(
+            name=d["name"],
+            as_computed_version=d.get("as_computed_version"),
+            current_version=d.get("current_version"),
+            stale_vs_current=d.get("stale_vs_current", False),
+            language=d.get("language"),
+            docker_image=d.get("docker_image"),
+            env_digest=d.get("env_digest"),
+            code_uri=d.get("code_uri"),
+        )
+
+
+@dataclass
+class Node:
+    """A lineage node: a table, view, column, or function."""
+
+    kind: str  # "table" | "view" | "column" | "function"
+    id: str  # "table", "table.column", or "fn:name@version"
+    table: Optional[str] = None
+    function: Optional[FunctionRef] = None
+
+    @classmethod
+    def _from(cls, d: dict) -> "Node":
+        fn = d.get("function")
+        return cls(
+            kind=d["kind"],
+            id=d["id"],
+            table=d.get("table"),
+            function=FunctionRef._from(fn) if fn else None,
+        )
+
+
+@dataclass
+class Edge:
+    """`downstream` depends on `upstream`, produced by `via` (a function name,
+    or None for a passthrough)."""
+
+    downstream: str
+    upstream: str
+    via: Optional[str] = None
+
+    @classmethod
+    def _from(cls, d: dict) -> "Edge":
+        return cls(downstream=d["downstream"], upstream=d["upstream"], via=d.get("via"))
+
+
+@dataclass
+class Lineage:
+    """A derived-compute lineage graph (nodes + labeled edges)."""
+
+    target: str
+    nodes: List[Node] = field(default_factory=list)
+    edges: List[Edge] = field(default_factory=list)
+
+    @classmethod
+    def from_json(cls, raw: Union[str, bytes, dict]) -> "Lineage":
+        d = json.loads(raw) if isinstance(raw, (str, bytes)) else raw
+        return cls(
+            target=d.get("target", ""),
+            nodes=[Node._from(n) for n in d.get("nodes", [])],
+            edges=[Edge._from(e) for e in d.get("edges", [])],
+        )
+
+    def functions(self) -> List[FunctionRef]:
+        """The function nodes in the graph."""
+        return [n.function for n in self.nodes if n.function is not None]
+
+    def stale(self) -> List[FunctionRef]:
+        """Functions whose as-computed version is behind the current registry
+        version -- the columns they produced are silently out of date."""
+        return [f for f in self.functions() if f.stale_vs_current]
+
+    def to_dict(self) -> dict:
+        def prune(d: dict) -> dict:
+            return {k: v for k, v in d.items() if v is not None}
+
+        return {
+            "target": self.target,
+            "nodes": [
+                prune(
+                    {
+                        "kind": n.kind,
+                        "id": n.id,
+                        "table": n.table,
+                        "function": prune(vars(n.function)) if n.function else None,
+                    }
+                )
+                for n in self.nodes
+            ],
+            "edges": [prune(vars(e)) for e in self.edges],
+        }
+
+    def to_graphviz(self) -> str:
+        """Graphviz DOT for the lineage DAG: columns/tables as nodes, function
+        names on edges, drift edges dashed + red."""
+        stale_names = {f.name for f in self.stale()}
+        out = [
+            "digraph lineage {",
+            "  rankdir=LR;",
+            '  node [fontname="monospace"];',
+        ]
+        for n in self.nodes:
+            if n.kind == "function":
+                continue
+            shape = "ellipse" if n.kind in ("table", "view") else "box"
+            out.append(f'  "{n.id}" [shape={shape}];')
+        for e in self.edges:
+            attrs = ""
+            if e.via:
+                if e.via in stale_names:
+                    attrs = f' [label="{e.via}" color=red style=dashed]'
+                else:
+                    attrs = f' [label="{e.via}"]'
+            out.append(f'  "{e.upstream}" -> "{e.downstream}"{attrs};')
+        out.append("}")
+        return "\n".join(out)
+
+    def _repr_html_(self) -> str:
+        warn = ""
+        drift = self.stale()
+        if drift:
+            names = ", ".join(sorted({f.name for f in drift}))
+            warn = (
+                f'<p style="color:#b00000"><b>stale vs current:</b> {names} '
+                "(re-refresh to catch up)</p>"
+            )
+        rows = "".join(
+            f"<tr><td><code>{e.downstream}</code></td>"
+            f"<td>&larr; {e.via or ''}</td>"
+            f"<td><code>{e.upstream}</code></td></tr>"
+            for e in self.edges
+        )
+        return (
+            f"<b>lineage: <code>{self.target}</code></b>{warn}"
+            "<table><tr><th>derived</th><th>via</th><th>from</th></tr>"
+            f"{rows}</table>"
+        )
@@ -51,15 +51,6 @@ class LanceMergeInsertBuilder(object):
        If there are multiple matches then the behavior is undefined.
        Currently this causes multiple copies of the row to be created
        but that behavior is subject to change.
-
-        Parameters
-        ----------
-        where: Optional[str], default None
-            An optional filter to limit which rows are updated. Column
-            references in this expression must be prefixed with "target."
-            to refer to the existing table data. For example, to only
-            update rows where the existing color is red, use:
-            ``where="target.color = 'red'"``
        """
        self._when_matched_update_all = True
        self._when_matched_update_all_condition = where
@@ -38,13 +38,15 @@ from lance_namespace_urllib3_client.models.query_table_request_vector import (
    QueryTableRequestVector,
 )
 from lance_namespace_urllib3_client.models.string_fts_query import StringFtsQuery
-from lance_namespace.errors import NamespaceNotEmptyError, TableNotFoundError
-from lancedb._lancedb import (
-    connect_namespace as _connect_namespace,
-    connect_namespace_client as _connect_namespace_client,
-)
+from lance_namespace.errors import TableNotFoundError
+from lancedb._lancedb import connect_namespace_client as _connect_namespace_client
 from lancedb.background_loop import LOOP
 from lancedb.db import AsyncConnection, DBConnection
+from lancedb.namespace_utils import (
+    _normalize_create_namespace_mode,
+    _normalize_drop_namespace_mode,
+    _normalize_drop_namespace_behavior,
+)
 from lance_namespace import (
    LanceNamespace,
    connect as namespace_connect,
@@ -53,6 +55,13 @@ from lance_namespace import (
    DropNamespaceResponse,
    ListNamespacesResponse,
    ListTablesResponse,
+    ListTablesRequest,
+    DescribeNamespaceRequest,
+    DropTableRequest,
+    RenameTableRequest,
+    ListNamespacesRequest,
+    CreateNamespaceRequest,
+    DropNamespaceRequest,
 )
 from lancedb.table import AsyncTable, LanceTable, Table
 from lancedb.util import validate_table_name
@@ -377,10 +386,6 @@ def _builds_namespace_natively(
    return namespace_client_impl == "rest" and bool(namespace_client_properties)


-def _supports_native_namespace(namespace_client_impl: str) -> bool:
-    return namespace_client_impl in {"dir", "rest"}
-
-
 class LanceNamespaceDBConnection(DBConnection):
    """
    A LanceDB connection that uses a namespace for table management.
@@ -391,7 +396,7 @@ class LanceNamespaceDBConnection(DBConnection):

    def __init__(
        self,
-        namespace_client: Optional[LanceNamespace] = None,
+        namespace_client: LanceNamespace,
        *,
        read_consistency_interval: Optional[timedelta] = None,
        storage_options: Optional[Dict[str, str]] = None,
@@ -399,7 +404,6 @@ class LanceNamespaceDBConnection(DBConnection):
        namespace_client_pushdown_operations: Optional[List[str]] = None,
        namespace_client_impl: Optional[str] = None,
        namespace_client_properties: Optional[Dict[str, str]] = None,
-        _inner: Optional[AsyncConnection] = None,
    ):
        """
        Initialize a namespace-based LanceDB connection.
@@ -441,36 +445,30 @@ class LanceNamespaceDBConnection(DBConnection):
        )
        self._namespace_client_impl = namespace_client_impl
        self._namespace_client_properties = namespace_client_properties
-        # When the namespace connection or client is built natively in Rust, the
-        # underlying Rust table performs QueryTable pushdown through the
-        # read-freshness context provider, which the pure-Python ``query_table``
-        # path bypasses.
-        self._route_pushdown_to_rust = _inner is not None or _builds_namespace_natively(
+        # When the namespace client is built natively (see Rust
+        # ``build_namespace_natively``), the underlying Rust table performs
+        # QueryTable pushdown through the read-freshness context provider, which
+        # the pure-Python ``query_table`` path bypasses.
+        self._route_pushdown_to_rust = _builds_namespace_natively(
            namespace_client_impl, namespace_client_properties
        )
-        if _inner is not None:
-            self._inner = _inner
-        else:
-            if namespace_client is None:
-                raise ValueError("namespace_client is required without a native _inner")
-            self._inner = AsyncConnection(
-                _connect_namespace_client(
-                    namespace_client,
-                    read_consistency_interval=(
-                        read_consistency_interval.total_seconds()
-                        if read_consistency_interval is not None
-                        else None
-                    ),
-                    storage_options=self.storage_options or None,
-                    session=session,
-                    namespace_client_pushdown_operations=(
-                        list(self._namespace_client_pushdown_operations)
-                    ),
-                    namespace_client_impl=namespace_client_impl,
-                    namespace_client_properties=namespace_client_properties,
-                )
+        self._inner = AsyncConnection(
+            _connect_namespace_client(
+                namespace_client,
+                read_consistency_interval=(
+                    read_consistency_interval.total_seconds()
+                    if read_consistency_interval is not None
+                    else None
+                ),
+                storage_options=self.storage_options or None,
+                session=session,
+                namespace_client_pushdown_operations=(
+                    list(self._namespace_client_pushdown_operations)
+                ),
+                namespace_client_impl=namespace_client_impl,
+                namespace_client_properties=namespace_client_properties,
            )
-        self._uri = self._inner.uri
+        )

    @override
    def serialize(self) -> str:
@@ -516,11 +514,11 @@ class LanceNamespaceDBConnection(DBConnection):
        )
        if namespace_path is None:
            namespace_path = []
-        return LOOP.run(
-            self._inner.table_names(
-                namespace_path=namespace_path, start_after=page_token, limit=limit
-            )
+        request = ListTablesRequest(
+            id=namespace_path, page_token=page_token, limit=limit
        )
+        response = self._namespace_client.list_tables(request)
+        return response.tables if response.tables else []

    @override
    def create_table(
@@ -591,8 +589,8 @@ class LanceNamespaceDBConnection(DBConnection):
                    index_cache_size=index_cache_size,
                )
            )
-        except (RuntimeError, ValueError) as e:
-            if "Table not found" in str(e) or "was not found" in str(e):
+        except RuntimeError as e:
+            if "Table not found" in str(e):
                table_id = namespace_path + [name]
                raise TableNotFoundError(f"Table not found: {'$'.join(table_id)}")
            raise
@@ -614,9 +612,12 @@ class LanceNamespaceDBConnection(DBConnection):

    @override
    def drop_table(self, name: str, namespace_path: Optional[List[str]] = None):
+        # Use namespace drop_table directly
        if namespace_path is None:
            namespace_path = []
-        LOOP.run(self._inner.drop_table(name, namespace_path=namespace_path))
+        table_id = namespace_path + [name]
+        request = DropTableRequest(id=table_id)
+        self._namespace_client.drop_table(request)

    @override
    def rename_table(
@@ -630,19 +631,14 @@ class LanceNamespaceDBConnection(DBConnection):
            cur_namespace_path = []
        if new_namespace_path is None:
            new_namespace_path = []
-        try:
-            LOOP.run(
-                self._inner.rename_table(
-                    cur_name,
-                    new_name,
-                    cur_namespace_path=cur_namespace_path,
-                    new_namespace_path=new_namespace_path,
-                )
-            )
-        except RuntimeError as e:
-            if "rename_table not implemented" in str(e):
-                raise NotImplementedError("rename_table not implemented") from e
-            raise
+        cur_table_id = cur_namespace_path + [cur_name]
+        new_namespace_id = new_namespace_path if new_namespace_path else None
+        request = RenameTableRequest(
+            id=cur_table_id,
+            new_table_name=new_name,
+            new_namespace_id=new_namespace_id,
+        )
+        self._namespace_client.rename_table(request)

    @override
    def drop_database(self):
@@ -654,7 +650,8 @@ class LanceNamespaceDBConnection(DBConnection):
    def drop_all_tables(self, namespace_path: Optional[List[str]] = None):
        if namespace_path is None:
            namespace_path = []
-        LOOP.run(self._inner.drop_all_tables(namespace_path=namespace_path))
+        for table_name in self.table_names(namespace_path=namespace_path):
+            self.drop_table(table_name, namespace_path=namespace_path)

    @override
    def list_namespaces(
@@ -684,10 +681,13 @@ class LanceNamespaceDBConnection(DBConnection):
        """
        if namespace_path is None:
            namespace_path = []
-        return LOOP.run(
-            self._inner.list_namespaces(
-                namespace_path=namespace_path, page_token=page_token, limit=limit
-            )
+        request = ListNamespacesRequest(
+            id=namespace_path, page_token=page_token, limit=limit
+        )
+        response = self._namespace_client.list_namespaces(request)
+        return ListNamespacesResponse(
+            namespaces=response.namespaces if response.namespaces else [],
+            page_token=response.page_token,
        )

    @override
@@ -715,12 +715,14 @@ class LanceNamespaceDBConnection(DBConnection):
        CreateNamespaceResponse
            Response containing the properties of the created namespace.
        """
-        return LOOP.run(
-            self._inner.create_namespace(
-                namespace_path=namespace_path,
-                mode=mode,
-                properties=properties,
-            )
+        request = CreateNamespaceRequest(
+            id=namespace_path,
+            mode=_normalize_create_namespace_mode(mode),
+            properties=properties,
+        )
+        response = self._namespace_client.create_namespace(request)
+        return CreateNamespaceResponse(
+            properties=response.properties if hasattr(response, "properties") else None
        )

    @override
@@ -748,18 +750,20 @@ class LanceNamespaceDBConnection(DBConnection):
        DropNamespaceResponse
            Response containing properties and transaction_id if applicable.
        """
-        try:
-            return LOOP.run(
-                self._inner.drop_namespace(
-                    namespace_path=namespace_path,
-                    mode=mode,
-                    behavior=behavior,
-                )
-            )
-        except RuntimeError as e:
-            if "Namespace not empty" in str(e):
-                raise NamespaceNotEmptyError(str(e)) from e
-            raise
+        request = DropNamespaceRequest(
+            id=namespace_path,
+            mode=_normalize_drop_namespace_mode(mode),
+            behavior=_normalize_drop_namespace_behavior(behavior),
+        )
+        response = self._namespace_client.drop_namespace(request)
+        return DropNamespaceResponse(
+            properties=(
+                response.properties if hasattr(response, "properties") else None
+            ),
+            transaction_id=(
+                response.transaction_id if hasattr(response, "transaction_id") else None
+            ),
+        )

    @override
    def describe_namespace(
@@ -778,7 +782,11 @@ class LanceNamespaceDBConnection(DBConnection):
        DescribeNamespaceResponse
            Response containing the namespace properties.
        """
-        return LOOP.run(self._inner.describe_namespace(namespace_path))
+        request = DescribeNamespaceRequest(id=namespace_path)
+        response = self._namespace_client.describe_namespace(request)
+        return DescribeNamespaceResponse(
+            properties=response.properties if hasattr(response, "properties") else None
+        )

    @override
    def list_tables(
@@ -808,10 +816,13 @@ class LanceNamespaceDBConnection(DBConnection):
        """
        if namespace_path is None:
            namespace_path = []
-        return LOOP.run(
-            self._inner.list_tables(
-                namespace_path=namespace_path, page_token=page_token, limit=limit
-            )
+        request = ListTablesRequest(
+            id=namespace_path, page_token=page_token, limit=limit
+        )
+        response = self._namespace_client.list_tables(request)
+        return ListTablesResponse(
+            tables=response.tables if response.tables else [],
+            page_token=response.page_token,
        )

    def _lance_table_from_uri(
@@ -867,18 +878,6 @@ class LanceNamespaceDBConnection(DBConnection):
        LanceNamespace
            The namespace client for this connection.
        """
-        if self._namespace_client is None:
-            if (
-                self._namespace_client_impl is None
-                or self._namespace_client_properties is None
-            ):
-                raise ValueError(
-                    "Cannot construct a Python namespace client without "
-                    "namespace implementation properties"
-                )
-            self._namespace_client = namespace_connect(
-                self._namespace_client_impl, self._namespace_client_properties
-            )
        return self._namespace_client


@@ -892,7 +891,7 @@ class AsyncLanceNamespaceDBConnection:

    def __init__(
        self,
-        namespace_client: Optional[LanceNamespace] = None,
+        namespace_client: LanceNamespace,
        *,
        read_consistency_interval: Optional[timedelta] = None,
        storage_options: Optional[Dict[str, str]] = None,
@@ -900,7 +899,6 @@ class AsyncLanceNamespaceDBConnection:
        namespace_client_pushdown_operations: Optional[List[str]] = None,
        namespace_client_impl: Optional[str] = None,
        namespace_client_properties: Optional[Dict[str, str]] = None,
-        _inner: Optional[AsyncConnection] = None,
    ):
        """
        Initialize an async namespace-based LanceDB connection.
@@ -942,35 +940,29 @@ class AsyncLanceNamespaceDBConnection:
        )
        self._namespace_client_impl = namespace_client_impl
        self._namespace_client_properties = namespace_client_properties
-        # See LanceNamespaceDBConnection: when Rust owns the namespace
-        # connection/client, its table performs QueryTable pushdown through the
-        # read-freshness provider, so defer to it rather than the urllib3 client
-        # path (which omits x-lancedb-min-timestamp).
-        self._route_pushdown_to_rust = _inner is not None or _builds_namespace_natively(
+        # See LanceNamespaceDBConnection: when built natively the Rust table runs
+        # QueryTable pushdown through the read-freshness provider, so defer to it
+        # rather than the urllib3 client (which omits x-lancedb-min-timestamp).
+        self._route_pushdown_to_rust = _builds_namespace_natively(
            namespace_client_impl, namespace_client_properties
        )
-        if _inner is not None:
-            self._inner = _inner
-        else:
-            if namespace_client is None:
-                raise ValueError("namespace_client is required without a native _inner")
-            self._inner = AsyncConnection(
-                _connect_namespace_client(
-                    namespace_client,
-                    read_consistency_interval=(
-                        read_consistency_interval.total_seconds()
-                        if read_consistency_interval is not None
-                        else None
-                    ),
-                    storage_options=self.storage_options or None,
-                    session=session,
-                    namespace_client_pushdown_operations=(
-                        list(self._namespace_client_pushdown_operations)
-                    ),
-                    namespace_client_impl=namespace_client_impl,
-                    namespace_client_properties=namespace_client_properties,
-                )
+        self._inner = AsyncConnection(
+            _connect_namespace_client(
+                namespace_client,
+                read_consistency_interval=(
+                    read_consistency_interval.total_seconds()
+                    if read_consistency_interval is not None
+                    else None
+                ),
+                storage_options=self.storage_options or None,
+                session=session,
+                namespace_client_pushdown_operations=(
+                    list(self._namespace_client_pushdown_operations)
+                ),
+                namespace_client_impl=namespace_client_impl,
+                namespace_client_properties=namespace_client_properties,
            )
+        )

    async def table_names(
        self,
@@ -994,9 +986,11 @@ class AsyncLanceNamespaceDBConnection:
        )
        if namespace_path is None:
            namespace_path = []
-        return await self._inner.table_names(
-            namespace_path=namespace_path, start_after=page_token, limit=limit
+        request = ListTablesRequest(
+            id=namespace_path, page_token=page_token, limit=limit
        )
+        response = self._namespace_client.list_tables(request)
+        return response.tables if response.tables else []

    async def create_table(
        self,
@@ -1059,8 +1053,8 @@ class AsyncLanceNamespaceDBConnection:
                storage_options=storage_options,
                index_cache_size=index_cache_size,
            )
-        except (RuntimeError, ValueError) as e:
-            if "Table not found" in str(e) or "was not found" in str(e):
+        except RuntimeError as e:
+            if "Table not found" in str(e):
                table_id = namespace_path + [name]
                raise TableNotFoundError(f"Table not found: {'$'.join(table_id)}")
            raise
@@ -1081,7 +1075,9 @@ class AsyncLanceNamespaceDBConnection:
        """Drop a table from the namespace."""
        if namespace_path is None:
            namespace_path = []
-        await self._inner.drop_table(name, namespace_path=namespace_path)
+        table_id = namespace_path + [name]
+        request = DropTableRequest(id=table_id)
+        self._namespace_client.drop_table(request)

    async def rename_table(
        self,
@@ -1095,17 +1091,14 @@ class AsyncLanceNamespaceDBConnection:
            cur_namespace_path = []
        if new_namespace_path is None:
            new_namespace_path = []
-        try:
-            await self._inner.rename_table(
-                cur_name,
-                new_name,
-                cur_namespace_path=cur_namespace_path,
-                new_namespace_path=new_namespace_path,
-            )
-        except RuntimeError as e:
-            if "rename_table not implemented" in str(e):
-                raise NotImplementedError("rename_table not implemented") from e
-            raise
+        cur_table_id = cur_namespace_path + [cur_name]
+        new_namespace_id = new_namespace_path if new_namespace_path else None
+        request = RenameTableRequest(
+            id=cur_table_id,
+            new_table_name=new_name,
+            new_namespace_id=new_namespace_id,
+        )
+        self._namespace_client.rename_table(request)

    async def drop_database(self):
        """Deprecated method."""
@@ -1117,7 +1110,9 @@ class AsyncLanceNamespaceDBConnection:
        """Drop all tables in the namespace."""
        if namespace_path is None:
            namespace_path = []
-        await self._inner.drop_all_tables(namespace_path=namespace_path)
+        table_names = await self.table_names(namespace_path=namespace_path)
+        for table_name in table_names:
+            await self.drop_table(table_name, namespace_path=namespace_path)

    async def list_namespaces(
        self,
@@ -1146,8 +1141,13 @@ class AsyncLanceNamespaceDBConnection:
        """
        if namespace_path is None:
            namespace_path = []
-        return await self._inner.list_namespaces(
-            namespace_path=namespace_path, page_token=page_token, limit=limit
+        request = ListNamespacesRequest(
+            id=namespace_path, page_token=page_token, limit=limit
+        )
+        response = self._namespace_client.list_namespaces(request)
+        return ListNamespacesResponse(
+            namespaces=response.namespaces if response.namespaces else [],
+            page_token=response.page_token,
        )

    async def create_namespace(
@@ -1174,11 +1174,15 @@ class AsyncLanceNamespaceDBConnection:
        CreateNamespaceResponse
            Response containing the properties of the created namespace.
        """
-        return await self._inner.create_namespace(
-            namespace_path=namespace_path,
-            mode=mode,
+        request = CreateNamespaceRequest(
+            id=namespace_path,
+            mode=_normalize_create_namespace_mode(mode),
            properties=properties,
        )
+        response = self._namespace_client.create_namespace(request)
+        return CreateNamespaceResponse(
+            properties=response.properties if hasattr(response, "properties") else None
+        )

    async def drop_namespace(
        self,
@@ -1204,16 +1208,20 @@ class AsyncLanceNamespaceDBConnection:
        DropNamespaceResponse
            Response containing properties and transaction_id if applicable.
        """
-        try:
-            return await self._inner.drop_namespace(
-                namespace_path=namespace_path,
-                mode=mode,
-                behavior=behavior,
-            )
-        except RuntimeError as e:
-            if "Namespace not empty" in str(e):
-                raise NamespaceNotEmptyError(str(e)) from e
-            raise
+        request = DropNamespaceRequest(
+            id=namespace_path,
+            mode=_normalize_drop_namespace_mode(mode),
+            behavior=_normalize_drop_namespace_behavior(behavior),
+        )
+        response = self._namespace_client.drop_namespace(request)
+        return DropNamespaceResponse(
+            properties=(
+                response.properties if hasattr(response, "properties") else None
+            ),
+            transaction_id=(
+                response.transaction_id if hasattr(response, "transaction_id") else None
+            ),
+        )

    async def describe_namespace(
        self, namespace_path: List[str]
@@ -1231,7 +1239,11 @@ class AsyncLanceNamespaceDBConnection:
        DescribeNamespaceResponse
            Response containing the namespace properties.
        """
-        return await self._inner.describe_namespace(namespace_path)
+        request = DescribeNamespaceRequest(id=namespace_path)
+        response = self._namespace_client.describe_namespace(request)
+        return DescribeNamespaceResponse(
+            properties=response.properties if hasattr(response, "properties") else None
+        )

    async def list_tables(
        self,
@@ -1260,8 +1272,13 @@ class AsyncLanceNamespaceDBConnection:
        """
        if namespace_path is None:
            namespace_path = []
-        return await self._inner.list_tables(
-            namespace_path=namespace_path, page_token=page_token, limit=limit
+        request = ListTablesRequest(
+            id=namespace_path, page_token=page_token, limit=limit
+        )
+        response = self._namespace_client.list_tables(request)
+        return ListTablesResponse(
+            tables=response.tables if response.tables else [],
+            page_token=response.page_token,
        )

    async def namespace_client(self) -> LanceNamespace:
@@ -1275,18 +1292,6 @@ class AsyncLanceNamespaceDBConnection:
        LanceNamespace
            The namespace client for this connection.
        """
-        if self._namespace_client is None:
-            if (
-                self._namespace_client_impl is None
-                or self._namespace_client_properties is None
-            ):
-                raise ValueError(
-                    "Cannot construct a Python namespace client without "
-                    "namespace implementation properties"
-                )
-            self._namespace_client = namespace_connect(
-                self._namespace_client_impl, self._namespace_client_properties
-            )
        return self._namespace_client


@@ -1337,32 +1342,6 @@ def connect_namespace(
    LanceNamespaceDBConnection
        A namespace-based connection to LanceDB
    """
-    if _supports_native_namespace(namespace_client_impl):
-        inner = AsyncConnection(
-            _connect_namespace(
-                namespace_client_impl,
-                namespace_client_properties,
-                read_consistency_interval=(
-                    read_consistency_interval.total_seconds()
-                    if read_consistency_interval is not None
-                    else None
-                ),
-                storage_options=storage_options,
-                session=session,
-                namespace_client_pushdown_operations=namespace_client_pushdown_operations,
-            )
-        )
-        return LanceNamespaceDBConnection(
-            namespace_client=None,
-            read_consistency_interval=read_consistency_interval,
-            storage_options=storage_options,
-            session=session,
-            namespace_client_pushdown_operations=namespace_client_pushdown_operations,
-            namespace_client_impl=namespace_client_impl,
-            namespace_client_properties=namespace_client_properties,
-            _inner=inner,
-        )
-
    namespace_client = namespace_connect(
        namespace_client_impl, namespace_client_properties
    )
@@ -1438,32 +1417,6 @@ def connect_namespace_async(
    ...     tables = await db.table_names()
    ...     table = await db.create_table("my_table", schema=schema)
    """
-    if _supports_native_namespace(namespace_client_impl):
-        inner = AsyncConnection(
-            _connect_namespace(
-                namespace_client_impl,
-                namespace_client_properties,
-                read_consistency_interval=(
-                    read_consistency_interval.total_seconds()
-                    if read_consistency_interval is not None
-                    else None
-                ),
-                storage_options=storage_options,
-                session=session,
-                namespace_client_pushdown_operations=namespace_client_pushdown_operations,
-            )
-        )
-        return AsyncLanceNamespaceDBConnection(
-            namespace_client=None,
-            read_consistency_interval=read_consistency_interval,
-            storage_options=storage_options,
-            session=session,
-            namespace_client_pushdown_operations=namespace_client_pushdown_operations,
-            namespace_client_impl=namespace_client_impl,
-            namespace_client_properties=namespace_client_properties,
-            _inner=inner,
-        )
-
    namespace_client = namespace_connect(
        namespace_client_impl, namespace_client_properties
    )
@@ -1,170 +0,0 @@
-# SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-"""Bridge LanceDB's internal metrics into OpenTelemetry.
-
-LanceDB (through Lance core) publishes metrics (currently object store request
-counts, bytes, latency, errors, and throttles) through the Rust ``metrics``
-facade. This module installs a process-global recorder that aggregates them and
-registers OpenTelemetry observable instruments that report the aggregated values
-into the user's ``MeterProvider``.
-
-The bridge is generic: every metric LanceDB describes is surfaced automatically,
-with no per-metric Python code. Histograms have no asynchronous OpenTelemetry
-instrument, so each is exported Prometheus-style as cumulative ``le`` buckets
-plus ``_count`` and ``_sum`` observable counters.
-"""
-
-from __future__ import annotations
-
-import warnings
-from typing import TYPE_CHECKING, Optional
-
-from ._lancedb import (
-    lancedb_metrics_catalog,
-    register_lancedb_metrics_recorder,
-    snapshot_lancedb_metrics,
-)
-
-if TYPE_CHECKING:
-    from opentelemetry.metrics import MeterProvider
-
-_INSTRUMENTED = False
-
-
-def instrument_lancedb_metrics(
-    meter_provider: Optional["MeterProvider"] = None,
-) -> bool:
-    """Register LanceDB metrics as OpenTelemetry observable instruments.
-
-    Installs a process-global metrics recorder and creates one observable
-    instrument per LanceDB metric on the given (or global) ``MeterProvider``. The
-    user's configured ``MetricReader`` then collects them on its own schedule.
-
-    Counters and gauges map directly to observable counters/gauges. Each
-    histogram is exported as cumulative ``le`` bucket counts (``<name>_bucket``,
-    with an ``le`` attribute) plus ``<name>_count`` and ``<name>_sum``.
-
-    Parameters
-    ----------
-    meter_provider : opentelemetry.metrics.MeterProvider, optional
-        The provider to register instruments on. Defaults to the global provider
-        from ``opentelemetry.metrics.get_meter_provider()``.
-
-    Returns
-    -------
-    bool
-        ``True`` if the recorder is installed and instruments are registered.
-        ``False`` if a different ``metrics`` recorder is already installed in
-        this process (``metrics`` permits only one global recorder), in which
-        case a warning is emitted and no instruments are created.
-
-    Notes
-    -----
-    Requires the OpenTelemetry API (``pip install lancedb[otel]``) and, to
-    actually export, an OpenTelemetry SDK (``pip install opentelemetry-sdk``)
-    configured by the application. Calling this more than once is safe;
-    instruments are created only on the first successful call.
-    """
-    global _INSTRUMENTED
-
-    try:
-        from opentelemetry.metrics import Observation, get_meter_provider
-    except ImportError as exc:
-        raise ImportError(
-            "instrument_lancedb_metrics requires the OpenTelemetry API/SDK. "
-            "Install it with `pip install lancedb[otel]` or "
-            "`pip install opentelemetry-sdk`."
-        ) from exc
-
-    if not register_lancedb_metrics_recorder():
-        warnings.warn(
-            "Could not install the LanceDB metrics recorder: another `metrics` "
-            "recorder is already installed in this process. LanceDB metrics will "
-            "not be exported via OpenTelemetry.",
-            stacklevel=2,
-        )
-        return False
-
-    if _INSTRUMENTED:
-        return True
-
-    provider = meter_provider or get_meter_provider()
-    meter = provider.get_meter("lancedb")
-
-    def scalar_callback(metric_name: str):
-        def callback(_options):
-            return [
-                Observation(point.value, point.attributes)
-                for point in snapshot_lancedb_metrics()
-                if point.name == metric_name and point.value is not None
-            ]
-
-        return callback
-
-    def bucket_callback(metric_name: str):
-        def callback(_options):
-            observations = []
-            for point in snapshot_lancedb_metrics():
-                if point.name != metric_name or point.buckets is None:
-                    continue
-                for le, cumulative in point.buckets:
-                    attributes = dict(point.attributes)
-                    attributes["le"] = le
-                    observations.append(Observation(cumulative, attributes))
-            return observations
-
-        return callback
-
-    def field_callback(metric_name: str, field: str):
-        def callback(_options):
-            observations = []
-            for point in snapshot_lancedb_metrics():
-                if point.name != metric_name:
-                    continue
-                value = getattr(point, field)
-                if value is not None:
-                    observations.append(Observation(value, point.attributes))
-            return observations
-
-        return callback
-
-    for desc in lancedb_metrics_catalog():
-        unit = desc.unit or ""
-        if desc.kind == "counter":
-            meter.create_observable_counter(
-                desc.name,
-                callbacks=[scalar_callback(desc.name)],
-                unit=unit,
-                description=desc.description,
-            )
-        elif desc.kind == "gauge":
-            meter.create_observable_gauge(
-                desc.name,
-                callbacks=[scalar_callback(desc.name)],
-                unit=unit,
-                description=desc.description,
-            )
-        elif desc.kind == "histogram":
-            # `_bucket` and `_count` observe cumulative sample counts, not the
-            # histogram's measured quantity, so they are unitless; only `_sum`
-            # carries the histogram's unit.
-            meter.create_observable_counter(
-                f"{desc.name}_bucket",
-                callbacks=[bucket_callback(desc.name)],
-                description=f"{desc.description} (cumulative buckets)",
-            )
-            meter.create_observable_counter(
-                f"{desc.name}_count",
-                callbacks=[field_callback(desc.name, "count")],
-                description=f"{desc.description} (count)",
-            )
-            meter.create_observable_counter(
-                f"{desc.name}_sum",
-                callbacks=[field_callback(desc.name, "sum")],
-                unit=unit,
-                description=f"{desc.description} (sum)",
-            )
-
-    _INSTRUMENTED = True
-    return True
@@ -11,7 +11,7 @@ import pyarrow as pa
 from ._lancedb import async_permutation_builder, PermutationReader
 from .table import LanceTable, Table
 from .background_loop import LOOP
-from .util import batch_to_tensor, batch_to_tensor_dict, batch_to_tensor_rows
+from .util import batch_to_tensor, batch_to_tensor_rows
 from typing import Any, Callable, Iterator, Literal, Optional, TYPE_CHECKING, Union

 if TYPE_CHECKING:
@@ -65,7 +65,6 @@ class PermutationBuilder:
        counts: Optional[list[int]] = None,
        fixed: Optional[int] = None,
        seed: Optional[int] = None,
-        clump_size: Optional[int] = None,
        split_names: Optional[list[str]] = None,
    ) -> "PermutationBuilder":
        """
@@ -88,9 +87,6 @@ class PermutationBuilder:
        Rows will be randomly assigned to splits.  The optional seed can be provided to
        make the assignment deterministic.

-        If clump_size is provided, rows are shuffled as contiguous groups of that size,
-        preserving I/O locality while still randomising the split assignment.
-
        The optional split_names can be provided to name the splits.  If not provided,
        the splits can only be referenced by their index.
        """
@@ -99,7 +95,6 @@ class PermutationBuilder:
            counts=counts,
            fixed=fixed,
            seed=seed,
-            clump_size=clump_size,
            split_names=split_names,
        )
        return self
@@ -946,7 +941,6 @@ class Permutation:
            "pandas",
            "arrow",
            "torch",
-            "torch_row",
            "torch_col",
            "polars",
        ],
@@ -962,19 +956,15 @@ class Permutation:
        - "python_col" - the batch will be a dict of lists (one entry per column)
        - "pandas" - the batch will be a pandas DataFrame
        - "arrow" - the batch will be a pyarrow RecordBatch
-        - "torch" - the batch will be a list of per-row dicts mapping column
-          name to a 0-D torch tensor. Works with the default
-          ``torch.utils.data.DataLoader`` collate, which stacks the per-row
-          dicts back into a dict of batched tensors.
-        - "torch_row" - the batch will be a list of tensors, one per row
+        - "torch" - the batch will be a list of tensors, one per row
        - "torch_col" - the batch will be a 2D torch tensor (first dim indexes columns)
        - "polars" - the batch will be a polars DataFrame

        Conversion may or may not involve a data copy.  Lance uses Arrow internally
        and so it is able to zero-copy to the arrow and polars formats.

-        Conversion to torch and torch_col will be zero-copy but will only support a
-        subset of data types (numeric types).
+        Conversion to torch_col will be zero-copy but will only support a subset of data
+        types (numeric types).

        Conversion to numpy and/or pandas will typically be zero-copy for numeric
        types.  Conversion of strings, lists, and structs will require creating python
@@ -995,8 +985,6 @@ class Permutation:
        elif format == "arrow":
            return self.with_transform(Transforms.arrow2arrow)
        elif format == "torch":
-            return self.with_transform(batch_to_tensor_dict)
-        elif format == "torch_row":
            return self.with_transform(batch_to_tensor_rows)
        elif format == "torch_col":
            return self.with_transform(batch_to_tensor)
@@ -119,27 +119,6 @@ def _filter_to_sql(filter: Optional[Union[str, Expr]]) -> Optional[str]:
    return filter


-def _combine_where(
-    existing: Optional[Union[str, Expr]], new: Union[str, Expr]
-) -> Union[str, Expr]:
-    """Combine a new filter with an existing one using a logical AND.
-
-    Calling ``where`` more than once composes the filters with AND instead of
-    replacing the previous filter. Two :class:`~lancedb.expr.Expr` filters are
-    combined as an expression; otherwise both filters are lowered to SQL strings
-    and combined as SQL.
-    """
-    if existing is None:
-        return new
-    existing_is_expr = isinstance(existing, Expr)
-    new_is_expr = isinstance(new, Expr)
-    if existing_is_expr and new_is_expr:
-        return existing & new
-    existing_sql = existing.to_sql() if existing_is_expr else existing
-    new_sql = new.to_sql() if new_is_expr else new
-    return f"({existing_sql}) AND ({new_sql})"
-
-
 def _projection_to_scanner_kwargs(
    columns: Optional[
        Union[
@@ -1169,13 +1148,8 @@ class LanceQueryBuilder(ABC):
        -------
        LanceQueryBuilder
            The LanceQueryBuilder object.
-
-        Notes
-        -----
-        Calling this multiple times combines the filters with a logical AND
-        rather than replacing the previous filter.
        """
-        self._where = _combine_where(self._where, where)
+        self._where = where
        self._postfilter = not prefilter
        return self

@@ -1719,13 +1693,8 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
        -------
        LanceQueryBuilder
            The LanceQueryBuilder object.
-
-        Notes
-        -----
-        Calling this multiple times combines the filters with a logical AND
-        rather than replacing the previous filter.
        """
-        self._where = _combine_where(self._where, where)
+        self._where = where
        if prefilter is not None:
            self._postfilter = not prefilter
        return self
@@ -2925,9 +2894,6 @@ class AsyncStandardQuery(AsyncQueryBase):

        Filtering performance can often be improved by creating a scalar index
        on the filter column(s).
-
-        Calling this multiple times combines the filters with a logical AND
-        rather than replacing the previous filter.
        """
        if isinstance(predicate, Expr):
            self._inner.where_expr(predicate._inner)
@@ -13,10 +13,14 @@ from typing import (
    Iterable,
    List,
    Optional,
+    TYPE_CHECKING,
    Union,
    Literal,
    overload,
 )
+
+if TYPE_CHECKING:
+    from ..udf import JobHandle
 import warnings

 from lancedb import __version__
@@ -884,8 +888,142 @@ class RemoteTable(Table):
    def count_rows(self, filter: Optional[str] = None) -> int:
        return LOOP.run(self._table.count_rows(filter))

-    def add_columns(self, transforms: Dict[str, str]) -> AddColumnsResult:
-        return LOOP.run(self._table.add_columns(transforms))
+    def add_columns(
+        self,
+        transforms: Optional[Dict[str, str]] = None,
+        *,
+        computed: Optional[Dict[str, tuple]] = None,
+    ) -> Optional[AddColumnsResult]:
+        result = None
+        if transforms is not None:
+            result = LOOP.run(self._table.add_columns(transforms))
+        if computed:
+            LOOP.run(self._table.add_columns(computed=computed))
+        return result
+
+    def refresh_column(
+        self,
+        columns,
+        *,
+        where: Optional[str] = None,
+        num_workers: Optional[int] = None,
+        max_workers: Optional[int] = None,
+        batch_size: Optional[int] = None,
+        priority: Optional[str] = None,
+    ) -> "JobHandle":
+        """Trigger recompute of computed columns (REFRESH COLUMN).
+
+        The expression is resolved server-side from each column's stored
+        binding; columns bound to the same struct-returning function
+        refresh together. Returns a `JobHandle` to wait on, poll, or cancel
+        (``tbl.refresh_column("c").wait()``). Server-backed feature
+        (LanceDB Enterprise / Cloud).
+
+        num_workers / max_workers / batch_size / priority are per-refresh
+        scheduling knobs (how to run THIS refresh) and override any default
+        the function carries. `priority` is a Kueue tier
+        (training | interactive | backfill).
+        """
+        from ..udf import JobHandle
+
+        if isinstance(columns, str):
+            columns = [columns]
+        job_id = LOOP.run(
+            self._table.refresh_column(
+                list(columns),
+                where=where,
+                num_workers=num_workers,
+                max_workers=max_workers,
+                batch_size=batch_size,
+                priority=priority,
+            )
+        )
+        return JobHandle(self._job_conn(), job_id)
+
+    def lineage(self, column=None, *, direction=None, depth=None):
+        """Derived-compute lineage of this table, or one of its columns:
+        upstream sources, downstream dependents, and the function version +
+        location that produced each derived column (with a drift flag). Returns
+        a `Lineage`. See `Connection.lineage`."""
+        return self._job_conn().lineage(
+            self._name, column, direction=direction, depth=depth
+        )
+
+    def _job_conn(self):
+        """A client connection for polling jobs this table spawns. Built lazily
+        from the table's serialized connection state and cached (not pickled --
+        a forked/unpickled table rebuilds it on next use)."""
+        from lancedb import deserialize_conn
+
+        conn = getattr(self, "_job_conn_cache", None)
+        if conn is None:
+            conn = deserialize_conn(self._serialized_connection_state())
+            self._job_conn_cache = conn
+        return conn
+
+    def load_columns(
+        self,
+        source: Union[str, Iterable[str]],
+        pk: str,
+        columns: Union[Iterable[str], Dict[str, str]],
+        *,
+        source_format: str = "parquet",
+        source_pk: Optional[str] = None,
+        on_missing: str = "carry",
+        source_storage_options: Optional[Dict[str, str]] = None,
+        num_workers: Optional[int] = None,
+        max_workers: Optional[int] = None,
+        batch_size: Optional[int] = None,
+        commit_granularity: Optional[int] = None,
+        priority: Optional[str] = None,
+    ) -> str:
+        """Fill existing columns from an external source by primary-key join.
+
+        The distributed-job equivalent of Geneva's ``Table.load_columns()``:
+        imports precomputed values (e.g. embeddings) from Parquet/Lance/IPC into
+        this table, matching on a primary key. Returns the load job id.
+        Server-backed feature (LanceDB Enterprise / Cloud).
+
+        Parameters
+        ----------
+        source: str | list[str]
+            One source URI or a list of URIs.
+        pk: str
+            Destination primary-key column. Also the source key unless
+            ``source_pk`` is given.
+        columns: list[str] | dict[str, str]
+            Value columns to load. A list loads same-named columns; a dict maps
+            ``{target: source}``.
+        source_format: str
+            ``"parquet"`` (default), ``"lance"``, or ``"ipc"``.
+        source_pk: str, optional
+            Source primary-key column when it differs from ``pk``.
+        on_missing: str
+            Behavior for destination rows with no source match:
+            ``"carry"`` (default, keep existing), ``"null"``, or ``"error"``.
+        """
+        if isinstance(source, str):
+            source = [source]
+        if isinstance(columns, dict):
+            mappings = [(target, src) for target, src in columns.items()]
+        else:
+            mappings = [(c, None) for c in columns]
+        return LOOP.run(
+            self._table.load_columns(
+                list(source),
+                source_format,
+                pk,
+                mappings,
+                source_key=source_pk,
+                source_storage_options=source_storage_options,
+                on_missing=on_missing,
+                num_workers=num_workers,
+                max_workers=max_workers,
+                batch_size=batch_size,
+                commit_granularity=commit_granularity,
+                priority=priority,
+            )
+        )

    def alter_columns(
        self, *alterations: Iterable[Dict[str, str]]
@@ -912,10 +1050,6 @@ class RemoteTable(Table):
        """Not supported on LanceDB Cloud."""
        return LOOP.run(self._table.unset_lsm_write_spec())

-    def get_lsm_write_spec(self) -> Optional["LsmWriteSpec"]:
-        """Read the installed LsmWriteSpec, or ``None``."""
-        return LOOP.run(self._table.get_lsm_write_spec())
-
    def close_lsm_writers(self) -> None:
        """No-op on LanceDB Cloud (no local shard writers)."""
        return LOOP.run(self._table.close_lsm_writers())
@@ -156,16 +156,9 @@ class MRRReranker(Reranker):
                reciprocal_rank = 1.0 / rank
                mrr_score_map[result_id].append(reciprocal_rank)

-        # MRR averages the reciprocal rank across *all* ranking systems, treating
-        # a system in which a document does not appear as a reciprocal rank of 0.
-        # We therefore divide by the total number of systems, not by the number of
-        # systems the document happens to appear in -- otherwise a document found
-        # by a single ranking would outrank one ranked highly by every system,
-        # defeating the purpose of fusing the rankings.
-        num_systems = len(vector_results)
        final_mrr_scores = {}
        for result_id, reciprocal_ranks in mrr_score_map.items():
-            mean_rr = float(np.sum(reciprocal_ranks)) / num_systems
+            mean_rr = np.mean(reciprocal_ranks)
            final_mrr_scores[result_id] = mean_rr

        combined = pa.concat_tables(vector_results, **self._concat_tables_args)
@@ -1,607 +0,0 @@
-# SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-"""Elastic streaming dataloader for PyTorch.
-
-Provides StreamingDataset, a PyTorch IterableDataset that guarantees:
-
- **Elastic determinism**: for a fixed (num_splits, shuffle_seed, epoch) the set
-  of samples that forms each global training step is identical regardless of
-  world_size or num_workers.
- **Resumability**: state_dict / load_state_dict capture per-split consumption
-  counts so training can resume from an exact mid-epoch position even when the
-  distributed topology changes between runs.
-"""
-
-import ctypes
-import logging
-import os
-import random
-import threading
-import time
-from collections import deque
-from concurrent.futures import ThreadPoolExecutor
-from multiprocessing import RawArray
-from typing import Any, Callable, Iterator, Optional
-
-from torch.utils.data import IterableDataset, get_worker_info
-
-from .permutation import (
-    Permutation,
-    Transforms,
-    permutation_builder,
-    _table_from_pickle_state,
-    _table_to_pickle_state,
-)
-
-logger = logging.getLogger(__name__)
-
-# Multiplier used to combine shuffle_seed and epoch into a single permutation
-# seed.  Chosen to be a large prime so different (seed, epoch) pairs produce
-# distinct seeds for any practically encountered epoch count.
-_EPOCH_PRIME = 100003
-
-DEFAULT_READ_BATCH_SIZE = 64
-DEFAULT_PREFETCH_BATCHES = 4
-
-
-class StreamingDataset(IterableDataset):
-    """An elastic, resumable PyTorch IterableDataset backed by a LanceDB table.
-
-    The table is partitioned into ``num_splits`` fixed splits using a
-    deterministic random shuffle controlled by ``shuffle_seed`` and ``epoch``.
-    Each rank is assigned a contiguous block of splits, and within a rank each
-    DataLoader worker is assigned a contiguous sub-block.  Samples are yielded
-    by round-robining over the assigned splits, one sample per split per cycle.
-
-    Internally ``__iter__`` runs a two-stage pipeline:
-
-    - **Stage 1 (I/O)**: one thread pool with ``num_splits * prefetch_batches``
-      workers fetches raw ``RecordBatch`` objects from LanceDB in parallel
-      across all splits and places them in a per-split raw-batch queue.
-    - **Stage 2 (transform)**: a second thread pool with ``os.cpu_count()``
-      workers picks up raw batches, applies the transform, and places the
-      results in a per-split cooked-row queue.
-
-    The main thread round-robins over the cooked queues, yielding one row per
-    split per cycle.
-
-    Parameters
-    ----------
-    table:
-        LanceDB table to stream from.
-    num_splits:
-        Number of fixed splits to partition the table into.  Must be divisible
-        by ``world_size``.  When used with DataLoader workers it must also be
-        divisible by ``world_size * num_workers``.  Defaults to ``world_size``.
-        If the row count (after any ``filter``) is not evenly divisible by
-        ``num_splits``, the surplus rows — at most ``num_splits - 1`` per epoch
-        — are silently dropped to keep all splits the same length.
-    shuffle:
-        Whether to randomly assign rows to splits.  When ``True`` (the
-        default) rows are shuffled using ``shuffle_seed`` and ``epoch``.
-        When ``False`` rows are divided into splits sequentially in storage
-        order, which can be useful for deterministic debugging or evaluation.
-    shuffle_seed:
-        Base seed for the random permutation.  Combined with ``epoch`` so
-        each epoch produces a different ordering.  Pass ``None`` to generate
-        a random seed at construction time.
-    epoch:
-        Current training epoch.  Combined with ``shuffle_seed`` so that each
-        epoch produces a different sample ordering.
-    rank:
-        This process's rank in the distributed training group.
-    world_size:
-        Total number of processes in the distributed training group.
-    read_batch_size:
-        Number of rows fetched from each split in a single ``take_offsets``
-        call.  Larger values amortise per-request overhead (critical on object
-        storage) at the cost of higher memory usage per split buffer.  Defaults
-        to ``DEFAULT_READ_BATCH_SIZE`` (64).
-    prefetch_batches:
-        Number of I/O batches to keep in flight per split.  Higher values
-        overlap storage latency with transform and training compute at the cost
-        of more memory and threads.  Defaults to ``DEFAULT_PREFETCH_BATCHES``
-        (4).
-    columns:
-        Optional list of column names to read.  When set, only those columns
-        are fetched from storage; all others are omitted.  ``None`` (the
-        default) reads every column.
-    shuffle_clump_size:
-        When set, rows are shuffled in contiguous groups of this size rather
-        than individually.  Larger clumps improve I/O locality (important on
-        object storage) at the cost of reduced randomness.  ``None`` (the
-        default) shuffles rows individually.
-    filter:
-        Optional SQL filter expression (e.g. ``"label = 'dog'"``).  Only rows
-        that satisfy the predicate are included in the permutation.  The filter
-        is applied during permutation construction so split sizes reflect the
-        filtered row count.
-    transform:
-        Optional callable applied to each ``pyarrow.RecordBatch`` before rows
-        are yielded.  Receives one batch at a time and must return an iterable
-        whose length equals the number of rows in the batch.  When ``None``
-        (the default) rows are returned as plain Python dicts.
-    worker_info_override:
-        If set, used in place of ``torch.utils.data.get_worker_info()`` to
-        determine the DataLoader worker assignment.  Intended for unit tests
-        that need to simulate multiple workers without spawning real processes.
-        If both this and the real worker info are non-None a warning is logged
-        and the override takes precedence.
-    """
-
-    def __init__(
-        self,
-        table,
-        *,
-        num_splits: Optional[int] = None,
-        shuffle: bool = True,
-        shuffle_seed: Optional[int] = 0,
-        epoch: int = 0,
-        rank: int = 0,
-        world_size: int = 1,
-        read_batch_size: int = DEFAULT_READ_BATCH_SIZE,
-        prefetch_batches: int = DEFAULT_PREFETCH_BATCHES,
-        columns: Optional[list[str]] = None,
-        shuffle_clump_size: Optional[int] = None,
-        filter: Optional[str] = None,
-        transform: Optional[Callable] = None,
-        connection_factory: Optional[Callable[[str], Any]] = None,
-        worker_info_override=None,
-    ):
-        super().__init__()
-        if num_splits is None:
-            num_splits = world_size
-        if shuffle_seed is None:
-            shuffle_seed = random.randrange(2**32)
-        if num_splits % world_size != 0:
-            raise ValueError(
-                f"num_splits ({num_splits}) must be divisible by "
-                f"world_size ({world_size})"
-            )
-
-        self._table = table
-        self._num_splits = num_splits
-        self._shuffle = shuffle
-        self._shuffle_seed = shuffle_seed
-        self._epoch = epoch
-        self._rank = rank
-        self._world_size = world_size
-        self._read_batch_size = read_batch_size
-        self._prefetch_batches = prefetch_batches
-        self._columns = columns
-        self._shuffle_clump_size = shuffle_clump_size
-        self._filter = filter
-        self._transform = transform
-        self._connection_factory = connection_factory
-        self._worker_info_override = worker_info_override
-
-        # Live references to pipeline state, set only while __iter__ is running
-        # in the same process.  Used by the observability properties when the
-        # DataLoader runs with num_workers=0.
-        self._raw_batches_ref: Optional[list[deque]] = None
-        self._cooked_ref: Optional[list[deque]] = None
-        self._fetch_head_ref: Optional[list[int]] = None
-        self._split_sizes_ref: Optional[list[int]] = None
-        self._local_consumed_ref: Optional[list[int]] = None
-
-        # Shared-memory counters written by __iter__ (which may run in a
-        # DataLoader worker process) and read by the observability properties
-        # in the main process.  RawArray is picklable via the forkserver
-        # reduction protocol so it survives the dataset pickle round-trip.
-        # Layout: [unscanned_rows, raw_rows, cooked_rows, consumed_rows,
-        #          bytes_loaded, fetch_time_us, transform_time_us]
-        self._worker_stats: RawArray = RawArray(ctypes.c_int64, 7)
-
-        # Cumulative bytes of Arrow buffer data fetched across all iterations.
-        self._bytes_loaded: int = 0
-        # Cumulative seconds spent in LanceDB I/O and in transform functions.
-        self._fetch_time: float = 0.0
-        self._transform_time: float = 0.0
-
-        # Number of samples each split has already been consumed.  At global
-        # step boundaries all splits have consumed this many samples, so a
-        # single scalar captures the topology-independent checkpoint state.
-        self._resume_offset: int = 0
-
-        # Build the permutation table once, deterministically.
-        builder = permutation_builder(table)
-        if filter is not None:
-            builder = builder.filter(filter)
-        if shuffle:
-            perm_seed = shuffle_seed + epoch * _EPOCH_PRIME
-            self._perm_table = builder.split_random(
-                fixed=num_splits, seed=perm_seed, clump_size=shuffle_clump_size
-            ).execute()
-        else:
-            self._perm_table = builder.split_sequential(fixed=num_splits).execute()
-
-        # Contiguous block of global split indices assigned to this rank.
-        splits_per_rank = num_splits // world_size
-        rank_start = rank * splits_per_rank
-        self._rank_splits: list[int] = list(
-            range(rank_start, rank_start + splits_per_rank)
-        )
-
-    def _resolve_my_splits(self) -> list[int]:
-        """Return the split indices this instance should read in __iter__."""
-        torch_worker_info = get_worker_info()
-        if self._worker_info_override is not None:
-            if torch_worker_info is not None:
-                logger.warning(
-                    "worker_info_override is set but get_worker_info() also returned a "
-                    "non-None value; ignoring the real torch worker info and using the "
-                    "override instead.  This may lead to duplicated or incorrect data "
-                    "from the dataset."
-                )
-            worker_info = self._worker_info_override
-        else:
-            worker_info = torch_worker_info
-
-        if worker_info is None:
-            return self._rank_splits
-
-        num_workers: int = worker_info.num_workers
-        worker_id: int = worker_info.id
-        n_rank_splits = len(self._rank_splits)
-        if n_rank_splits % num_workers != 0:
-            raise ValueError(
-                f"Number of rank splits ({n_rank_splits}) must be divisible by "
-                f"num_workers ({num_workers})"
-            )
-        splits_per_worker = n_rank_splits // num_workers
-        start = worker_id * splits_per_worker
-        return self._rank_splits[start : start + splits_per_worker]
-
-    def __iter__(self) -> Iterator[dict[str, Any]]:
-        if self._raw_batches_ref is not None:
-            raise RuntimeError(
-                "StreamingDataset does not support concurrent iteration. "
-                "Only one active iterator per dataset instance is allowed."
-            )
-        my_splits = self._resolve_my_splits()
-        if not my_splits:
-            return
-
-        # Set identity transform on each Permutation so __getitems__ returns
-        # the raw RecordBatch.  Stage 2 applies the real transform.
-        permutations: list[Permutation] = []
-        for split_idx in my_splits:
-            perm = Permutation.from_tables(
-                self._table, self._perm_table, split=split_idx
-            )
-            if self._columns is not None:
-                perm = perm.select_columns(self._columns)
-            perm = perm.with_transform(lambda batch: batch)
-            if self._resume_offset > 0:
-                perm = perm.with_skip(self._resume_offset)
-            permutations.append(perm)
-
-        n = len(permutations)
-        split_sizes = [perm.num_rows for perm in permutations]
-        initial_offset = self._resume_offset
-        local_consumed = [0] * n
-
-        batch_size = self._read_batch_size
-        max_prefetch = self._prefetch_batches
-        cpu_workers = os.cpu_count() or 1
-        final_transform = (
-            self._transform if self._transform is not None else Transforms.arrow2python
-        )
-
-        # Per-split pipeline state.
-        fetch_head = [0] * n
-        io_pending = [deque() for _ in range(n)]  # Future[RecordBatch]
-        raw_batches = [deque() for _ in range(n)]  # RecordBatch — fetched, awaiting tx
-        tx_pending = [deque() for _ in range(n)]  # Future[list[Any]]
-        cooked = [deque() for _ in range(n)]  # rows ready to yield
-
-        # Limit simultaneous transforms to cpu_workers across all splits.
-        tx_semaphore = threading.Semaphore(cpu_workers)
-
-        # ── Stage 1 helpers ───────────────────────────────────────────────────
-
-        def _io_call(perm, indices):
-            t0 = time.perf_counter()
-            batch = perm.__getitems__(indices)
-            self._bytes_loaded += batch.nbytes
-            self._fetch_time += time.perf_counter() - t0
-            return batch
-
-        def _submit_io(i: int) -> None:
-            remaining = split_sizes[i] - fetch_head[i]
-            if remaining <= 0:
-                return
-            fetch = min(batch_size, remaining)
-            start = fetch_head[i]
-            fetch_head[i] += fetch
-            perm_i = permutations[i]
-            indices = list(range(start, start + fetch))
-            io_pending[i].append(io_pool.submit(_io_call, perm_i, indices))
-
-        def _fill_io(i: int) -> None:
-            while len(io_pending[i]) < max_prefetch and fetch_head[i] < split_sizes[i]:
-                _submit_io(i)
-
-        def _drain_io(i: int) -> None:
-            """Move completed I/O futures into raw_batches non-blockingly."""
-            while io_pending[i] and io_pending[i][0].done():
-                raw_batches[i].append(io_pending[i].popleft().result())
-
-        # ── Stage 2 helpers ───────────────────────────────────────────────────
-
-        def _tx_call_guarded(batch):
-            try:
-                t0 = time.perf_counter()
-                result = final_transform(batch)
-                self._transform_time += time.perf_counter() - t0
-                return result
-            finally:
-                tx_semaphore.release()
-
-        def _try_submit_tx(i: int) -> None:
-            """Submit transforms for raw_batches[i] up to available capacity."""
-            while raw_batches[i] and tx_semaphore.acquire(blocking=False):
-                batch = raw_batches[i].popleft()
-                tx_pending[i].append(tx_pool.submit(_tx_call_guarded, batch))
-
-        def _drain_tx(i: int) -> None:
-            """Move completed transform futures into cooked non-blockingly."""
-            while tx_pending[i] and tx_pending[i][0].done():
-                cooked[i].extend(tx_pending[i].popleft().result())
-
-        # ── Combined advance ──────────────────────────────────────────────────
-
-        def _advance(i: int) -> None:
-            """Non-blocking pipeline pump for split i."""
-            _drain_io(i)
-            _drain_tx(i)
-            _try_submit_tx(i)
-            _fill_io(i)
-
-        def _ensure_cooked(i: int) -> None:
-            """Ensure cooked[i] has at least one row, blocking if necessary."""
-            _advance(i)
-            while not cooked[i]:
-                if tx_pending[i]:
-                    # Wait for the oldest in-flight transform.
-                    cooked[i].extend(tx_pending[i].popleft().result())
-                    _advance(i)
-                elif raw_batches[i]:
-                    # Acquire a transform slot (may block briefly if all
-                    # cpu_workers are busy with other splits).
-                    tx_semaphore.acquire()
-                    batch = raw_batches[i].popleft()
-                    tx_pending[i].append(tx_pool.submit(_tx_call_guarded, batch))
-                elif io_pending[i]:
-                    # Block on the oldest in-flight I/O fetch.
-                    raw_batches[i].append(io_pending[i].popleft().result())
-                    _advance(i)
-                else:
-                    break  # split exhausted
-
-        # ── Main loop ─────────────────────────────────────────────────────────
-
-        with ThreadPoolExecutor(max_workers=n * max_prefetch) as io_pool:
-            with ThreadPoolExecutor(max_workers=cpu_workers) as tx_pool:
-                self._raw_batches_ref = raw_batches
-                self._cooked_ref = cooked
-                self._fetch_head_ref = fetch_head
-                self._split_sizes_ref = split_sizes
-                self._local_consumed_ref = local_consumed
-                try:
-                    for i in range(n):
-                        _fill_io(i)
-
-                    while True:
-                        # Stop when any split is exhausted (all exhaust
-                        # simultaneously: equal split sizes + round-robin).
-                        if any(local_consumed[i] >= split_sizes[i] for i in range(n)):
-                            break
-
-                        for i in range(n):
-                            _ensure_cooked(i)
-                            row = cooked[i].popleft()
-                            local_consumed[i] += 1
-                            _advance(i)
-
-                            # After the last split in each cycle: update the
-                            # global offset and refresh the shared-memory stats
-                            # so the main process can observe pipeline depth
-                            # even when __iter__ runs in a worker process.
-                            if i == n - 1:
-                                self._resume_offset = initial_offset + local_consumed[i]
-                                ws = self._worker_stats
-                                ws[0] = sum(
-                                    split_sizes[j] - fetch_head[j] for j in range(n)
-                                )
-                                ws[1] = sum(
-                                    batch.num_rows for q in raw_batches for batch in q
-                                )
-                                ws[2] = sum(len(q) for q in cooked)
-                                ws[3] = sum(local_consumed)
-                                ws[4] = self._bytes_loaded
-                                ws[5] = int(self._fetch_time * 1_000_000)
-                                ws[6] = int(self._transform_time * 1_000_000)
-
-                            yield row
-                finally:
-                    self._raw_batches_ref = None
-                    self._cooked_ref = None
-                    self._fetch_head_ref = None
-                    self._split_sizes_ref = None
-                    self._local_consumed_ref = None
-
-    @property
-    def bytes_loaded(self) -> int:
-        """Cumulative bytes of raw Arrow buffer data fetched from storage.
-
-        Measured on the ``RecordBatch`` before any transform is applied, so
-        the value reflects actual I/O rather than the size of transformed
-        output.  Accumulates across multiple iterations of the same dataset
-        instance and is never reset automatically.
-        """
-        if self._raw_batches_ref is not None:
-            return self._bytes_loaded
-        return int(self._worker_stats[4])
-
-    @property
-    def fetch_time(self) -> float:
-        """Cumulative seconds spent waiting for data from LanceDB.
-
-        Measured per batch in the Stage 1 I/O threads as the total elapsed
-        time of the ``take_offsets`` call.  Accumulates across all splits and
-        all iterations.
-        """
-        if self._raw_batches_ref is not None:
-            return self._fetch_time
-        return self._worker_stats[5] / 1_000_000
-
-    @property
-    def transform_time(self) -> float:
-        """Cumulative seconds spent applying the transform.
-
-        Measured per batch in the Stage 2 transform threads as the elapsed
-        time inside the transform callable (or the default ``arrow2python``
-        conversion when no transform is set).  Accumulates across all splits
-        and all iterations.
-        """
-        if self._raw_batches_ref is not None:
-            return self._transform_time
-        return self._worker_stats[6] / 1_000_000
-
-    @property
-    def raw_queue_depth(self) -> int:
-        """Number of raw rows waiting for a transform thread across all splits.
-
-        A persistently non-zero value means Stage 2 (transform) is the
-        bottleneck: I/O is completing faster than transforms can consume
-        batches.  Returns 0 when not iterating.
-        """
-        if self._raw_batches_ref is not None:
-            return sum(batch.num_rows for q in self._raw_batches_ref for batch in q)
-        return int(self._worker_stats[1])
-
-    @property
-    def prefetch_queue_depth(self) -> int:
-        """Number of rows transformed and ready to yield across all splits.
-
-        Counts rows whose transform has completed and are sitting in memory
-        waiting for the main thread — rows that can be handed off with no
-        I/O or CPU wait.  Returns 0 when not iterating.
-        """
-        if self._cooked_ref is not None:
-            return sum(len(q) for q in self._cooked_ref)
-        return int(self._worker_stats[2])
-
-    @property
-    def unscanned_rows(self) -> int:
-        """Number of rows not yet submitted to the I/O stage across all splits.
-
-        Decreases as the I/O stage submits fetch requests.  When this reaches
-        zero all data has been requested from storage (though it may not have
-        arrived yet).  Returns 0 when not iterating.
-        """
-        if self._fetch_head_ref is not None:
-            return sum(
-                size - head
-                for size, head in zip(self._split_sizes_ref, self._fetch_head_ref)
-            )
-        return int(self._worker_stats[0])
-
-    @property
-    def consumed_rows(self) -> int:
-        """Number of rows already yielded to the caller across all splits.
-
-        Monotonically increases throughout iteration.  Returns 0 when not
-        iterating.
-        """
-        if self._local_consumed_ref is not None:
-            return sum(self._local_consumed_ref)
-        return int(self._worker_stats[3])
-
-    def __getstate__(self):
-        """Support pickling for multi-worker DataLoader (forkserver / spawn).
-
-        The live LanceDB table object contains non-picklable connection state
-        (sockets, Rust-backed PyO3 objects).  If a ``connection_factory`` was
-        supplied only the table name is serialised; the factory is called in
-        the worker to reopen the connection without embedding any credentials.
-        Without a factory the table's own picklable reopen state is captured
-        via ``_table_to_pickle_state`` (mirrors the ``Permutation`` approach).
-        """
-        state = self.__dict__.copy()
-        # _table: replace with reconnect info (credentials must not be embedded).
-        state["_table_name"] = self._table.name
-        if self._connection_factory is not None:
-            state["_table"] = None
-        else:
-            state["_table"] = _table_to_pickle_state(self._table)
-        # _perm_table: always in-memory; serialise as Arrow data (mirrors
-        # how Permutation.__getstate__ handles its permutation_table).
-        state["_perm_table"] = (
-            self._perm_table.name,
-            self._perm_table.to_arrow(),
-        )
-        for key in (
-            "_raw_batches_ref",
-            "_cooked_ref",
-            "_fetch_head_ref",
-            "_split_sizes_ref",
-            "_local_consumed_ref",
-        ):
-            state[key] = None
-        return state
-
-    def __setstate__(self, state):
-        """Reconnect to LanceDB after unpickling in a worker process."""
-        from . import connect as _connect
-
-        table_name = state.pop("_table_name")
-        table_state = state.pop("_table")
-        perm_name, perm_data = state.pop("_perm_table")
-        self.__dict__.update(state)
-        if self._connection_factory is not None:
-            self._table = self._connection_factory(table_name)
-        else:
-            self._table = _table_from_pickle_state(table_state)
-        self._perm_table = _connect("memory://").create_table(perm_name, perm_data)
-
-    def state_dict(self) -> dict:
-        """Snapshot the dataset's consumption state.
-
-        The returned dict is topology-independent: at global step boundaries
-        every split has been consumed the same number of times (by the
-        round-robin design), so the per-split count is a single uniform value
-        that is identical across all ranks and DataLoader workers.
-        """
-        return {
-            "shuffle_seed": self._shuffle_seed,
-            "num_splits": self._num_splits,
-            "epoch": self._epoch,
-            "samples_consumed_per_split": [self._resume_offset] * self._num_splits,
-        }
-
-    def load_state_dict(self, state: dict) -> None:
-        """Resume from a previously snapshotted state.
-
-        Raises ``ValueError`` if ``num_splits`` or ``shuffle_seed`` differ
-        from the checkpoint, since a different split structure or shuffle order
-        makes mid-epoch resumption meaningless.
-        """
-        if state["num_splits"] != self._num_splits:
-            raise ValueError(
-                f"num_splits mismatch: checkpoint has {state['num_splits']}, "
-                f"current dataset has {self._num_splits}"
-            )
-        if state["shuffle_seed"] != self._shuffle_seed:
-            raise ValueError(
-                f"shuffle_seed mismatch: checkpoint has {state['shuffle_seed']}, "
-                f"current dataset has {self._shuffle_seed}"
-            )
-        consumed = state["samples_consumed_per_split"]
-        # All entries are equal at step boundaries; use the first.
-        if isinstance(consumed, list):
-            self._resume_offset = consumed[0] if consumed else 0
-        else:
-            self._resume_offset = int(consumed)
@@ -651,16 +651,6 @@ def _append_vector_columns(
                    col_data = func.compute_source_embeddings_with_retry(
                        batch[conf.source_column]
                    )
-                    # Replace vectors with wrong length (including empty lists
-                    # returned for inputs like empty strings) with None so that
-                    # _handle_bad_vectors can process them according to the
-                    # on_bad_vectors policy instead of crashing when PyArrow
-                    # tries to cast them into a fixed-size list array.
-                    expected_ndims = conf.function.ndims()
-                    col_data = [
-                        v if v is not None and len(v) == expected_ndims else None
-                        for v in col_data
-                    ]
                    if no_vector_column:
                        batch = batch.append_column(
                            schema.field(vector_column),
@@ -712,6 +702,24 @@ def _normalize_progress(progress):
    return progress, False


+def _computed_groups(computed):
+    """Group computed columns by expression, preserving declaration order
+    (struct-returning functions need their columns adjacent so schema order
+    matches field order). Accepts the ergonomic forms -- `fn("col")` values
+    and tuple keys for struct fan-out -- via `_normalize_computed`."""
+    from .udf import _normalize_computed
+
+    groups = []
+    for name, (sql_type, expression) in _normalize_computed(computed).items():
+        for expr, cols in groups:
+            if expr == expression:
+                cols.append((name, sql_type))
+                break
+        else:
+            groups.append((expression, [(name, sql_type)]))
+    return groups
+
+
 class Table(ABC):
    """
    A Table is a collection of Records in a LanceDB Database.
@@ -817,6 +825,59 @@ class Table(ABC):
        """The number of rows in this Table"""
        return self.count_rows(None)

+    def add_computed_column(
+        self,
+        columns,
+        fn,
+        args: Optional[List[str]] = None,
+        types=None,
+    ) -> None:
+        """Declare computed column(s) bound to a UDF -- no compute happens
+        here (the agent fills them lazily, or refresh_column() triggers a run).
+
+        .. deprecated::
+            A computed column is an expression over a registered function, so
+            bind it as one: ``add_columns(computed={"vec": embed("data")})``.
+            ``embed("data")`` applies the function to the `data` column and
+            infers the type from the function's return signature -- the
+            function never couples to a particular column. Prefer that form.
+        """
+        import warnings
+
+        warnings.warn(
+            "add_computed_column is deprecated; use add_columns(computed="
+            '{"vec": embed("data")}).',
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        from .udf import Udf, struct_field_types
+
+        multi = isinstance(columns, (tuple, list))
+        if isinstance(fn, Udf):
+            expr = fn.expression(*(args or []))
+            if types is None:
+                if multi:
+                    if not fn.returns.upper().startswith("STRUCT"):
+                        raise ValueError(
+                            "several columns need a STRUCT-returning function"
+                        )
+                    types = struct_field_types(fn.returns)
+                else:
+                    types = fn.returns
+        else:
+            if types is None:
+                raise ValueError("pass types= when fn is a name string")
+            expr = f"{fn}({', '.join(args or [])})"
+        if multi:
+            if len(types) != len(columns):
+                raise ValueError(
+                    f"{len(columns)} columns but {len(types)} output types"
+                )
+            computed = {c: (t, expr) for c, t in zip(columns, types)}
+        else:
+            computed = {columns: (types, expr)}
+        self.add_columns(computed=computed)
+
    @property
    @abstractmethod
    def embedding_functions(self) -> Dict[str, EmbeddingFunctionConfig]:
@@ -2152,19 +2213,12 @@ class LanceTable(Table):

        branch = self.current_branch()
        version = None if branch is not None else self.version
-        namespace_client = self._namespace_client
-        if namespace_client is None:
-            conn_uri = getattr(self._conn, "uri", "")
-            if get_uri_scheme(conn_uri) == "namespace":
-                namespace_client = self._conn.namespace_client()
-                self._namespace_client = namespace_client
-
-        if namespace_client is not None:
+        if self._namespace_client is not None:
            table_id = self._namespace_path + [self.name]
            ds = lance.dataset(
                version=version,
                storage_options=self._conn.storage_options,
-                namespace_client=namespace_client,
+                namespace_client=self._namespace_client,
                table_id=table_id,
                **kwargs,
            )
@@ -3727,9 +3781,68 @@ class LanceTable(Table):
        return LOOP.run(self._table.index_stats(index_name))

    def add_columns(
-        self, transforms: Dict[str, str] | pa.field | List[pa.field] | pa.Schema
-    ) -> AddColumnsResult:
-        return LOOP.run(self._table.add_columns(transforms))
+        self,
+        transforms: Dict[str, str]
+        | pa.field
+        | List[pa.field]
+        | pa.Schema
+        | None = None,
+        *,
+        computed: Optional[Dict] = None,
+    ) -> Optional[AddColumnsResult]:
+        result = None
+        if transforms is not None:
+            result = LOOP.run(self._table.add_columns(transforms))
+        if computed:
+            # computed binds an expression over a registered function to a
+            # column: {col: fn("input_col")} -- fn("input_col") yields the
+            # expression and carries the inferred type; a tuple key fans a
+            # STRUCT return out to several columns. Declares the binding only;
+            # the server fills the values (server-backed). The legacy
+            # {col: (sql_type, expression)} tuple form is still accepted.
+            result_unused = LOOP.run(self._table.add_columns(computed=computed))
+            del result_unused
+        return result
+
+    def refresh_column(
+        self,
+        columns,
+        *,
+        where: Optional[str] = None,
+        num_workers: Optional[int] = None,
+        max_workers: Optional[int] = None,
+        batch_size: Optional[int] = None,
+        priority: Optional[str] = None,
+    ) -> "JobHandle":
+        """Trigger recompute of computed columns (REFRESH COLUMN).
+
+        The expression is resolved server-side from each column's stored
+        binding; columns bound to the same struct-returning function
+        refresh together. Returns a `JobHandle` to wait on, poll, or cancel
+        (``tbl.refresh_column("col").wait()``) -- mirrors
+        `MaterializedView.refresh()`. Server-backed feature (LanceDB
+        Enterprise / Cloud).
+
+        num_workers / max_workers / batch_size / priority are per-refresh
+        scheduling knobs (how to run THIS refresh) and override any default
+        the function carries. `priority` is a Kueue tier
+        (training | interactive | backfill).
+        """
+        from .udf import JobHandle
+
+        if isinstance(columns, str):
+            columns = [columns]
+        job_id = LOOP.run(
+            self._table.refresh_column(
+                list(columns),
+                where=where,
+                num_workers=num_workers,
+                max_workers=max_workers,
+                batch_size=batch_size,
+                priority=priority,
+            )
+        )
+        return JobHandle(self._conn, job_id, table=self.name)

    def alter_columns(
        self, *alterations: Iterable[Dict[str, str]]
@@ -3759,11 +3872,6 @@ class LanceTable(Table):
        [`AsyncTable.unset_lsm_write_spec`][lancedb.AsyncTable.unset_lsm_write_spec]."""
        return LOOP.run(self._table.unset_lsm_write_spec())

-    def get_lsm_write_spec(self) -> Optional["LsmWriteSpec"]:
-        """Read the installed LsmWriteSpec, or ``None``. See
-        [`AsyncTable.get_lsm_write_spec`][lancedb.AsyncTable.get_lsm_write_spec]."""
-        return LOOP.run(self._table.get_lsm_write_spec())
-
    def close_lsm_writers(self) -> None:
        """Close cached MemWAL shard writers. See
        [`AsyncTable.close_lsm_writers`][lancedb.AsyncTable.close_lsm_writers]."""
@@ -4035,16 +4143,7 @@ def _handle_bad_vector_column(
        dim = _infer_vector_dim(vec_arr)
        if dim is None:
            return data
-
-    is_null = pc.is_null(vec_arr)
-    # pc.list_value_length returns null for null list entries, so
-    # pc.not_equal(null, dim) also returns null. Use or_kleene so that
-    # True OR null = True (Kleene three-valued logic), ensuring null vectors
-    # are counted as wrong-dim.
-    has_wrong_dim = pc.or_kleene(
-        is_null,
-        pc.not_equal(pc.list_value_length(vec_arr), dim),
-    )
+    has_wrong_dim = pc.not_equal(pc.list_value_length(vec_arr), dim)

    has_bad_vectors = pc.any(has_nan).as_py() or pc.any(has_wrong_dim).as_py()

@@ -4422,17 +4521,6 @@ class AsyncTable:
        """
        await self._inner.unset_lsm_write_spec()

-    async def get_lsm_write_spec(self) -> Optional["LsmWriteSpec"]:
-        """Read the LsmWriteSpec currently installed on this table.
-
-        Returns ``None`` when the MemWAL LSM write path is not enabled (no
-        spec has been set, or it was removed with `unset_lsm_write_spec`).
-        The returned spec — including its ``maintained_indexes`` and
-        ``writer_config_defaults`` — mirrors what was passed to
-        `set_lsm_write_spec`.
-        """
-        return await self._inner.get_lsm_write_spec()
-
    async def close_lsm_writers(self) -> None:
        """Drain and close any cached MemWAL shard writers for this table.

@@ -5432,9 +5520,44 @@ class AsyncTable:

        return await self._inner.update(updates_sql, where)

+    async def refresh_column(
+        self,
+        columns,
+        *,
+        where: Optional[str] = None,
+        num_workers: Optional[int] = None,
+        max_workers: Optional[int] = None,
+        batch_size: Optional[int] = None,
+        priority: Optional[str] = None,
+    ) -> str:
+        """Trigger recompute of computed columns (REFRESH COLUMN).
+        Returns the refresh job id. Server-backed feature.
+
+        num_workers / max_workers / batch_size / priority are per-refresh
+        scheduling knobs (how to run THIS refresh); they override any default
+        the function carries. `priority` is a Kueue tier
+        (training | interactive | backfill)."""
+        if isinstance(columns, str):
+            columns = [columns]
+        return await self._inner.refresh_column(
+            list(columns),
+            where_clause=where,
+            num_workers=num_workers,
+            max_workers=max_workers,
+            batch_size=batch_size,
+            priority=priority,
+        )
+
    async def add_columns(
-        self, transforms: dict[str, str] | pa.field | List[pa.field] | pa.Schema
-    ) -> AddColumnsResult:
+        self,
+        transforms: dict[str, str]
+        | pa.field
+        | List[pa.field]
+        | pa.Schema
+        | None = None,
+        *,
+        computed: Optional[Dict] = None,
+    ) -> Optional[AddColumnsResult]:
        """
        Add new columns with defined values.

@@ -5453,6 +5576,7 @@ class AsyncTable:
            version: the new version number of the table after adding columns.

        """
+        result = None
        if isinstance(transforms, pa.Field):
            transforms = [transforms]
        if isinstance(transforms, list) and all(
@@ -5460,9 +5584,69 @@ class AsyncTable:
        ):
            transforms = pa.schema(transforms)
        if isinstance(transforms, pa.Schema):
-            return await self._inner.add_columns_with_schema(transforms)
+            result = await self._inner.add_columns_with_schema(transforms)
+        elif transforms is not None:
+            result = await self._inner.add_columns(list(transforms.items()))
+        if computed:
+            # computed binds an expression over a registered function to a
+            # column: {col: fn("input_col")} -- fn("input_col") yields the
+            # expression and carries the inferred type; a tuple key fans a
+            # STRUCT return out to several columns. Declares the binding only;
+            # the server fills the values (server-backed). The legacy
+            # {col: (sql_type, expression)} tuple form is still accepted.
+            for expression, cols in _computed_groups(computed):
+                await self._inner.add_computed_columns(cols, expression)
+        return result
+
+    async def add_computed_column(
+        self,
+        columns,
+        fn,
+        args: Optional[List[str]] = None,
+        types=None,
+    ) -> None:
+        """Declare computed column(s) bound to a UDF (async).
+
+        .. deprecated::
+            Use ``add_columns(computed={"col": fn("input_col")})`` -- a computed
+            column is an expression over a registered function, so bind it that
+            way instead of coupling the UDF to the column here.
+        """
+        import warnings
+
+        warnings.warn(
+            "add_computed_column is deprecated; use add_columns(computed="
+            '{"col": fn("input_col")}).',
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        from .udf import Udf, struct_field_types
+
+        multi = isinstance(columns, (tuple, list))
+        if isinstance(fn, Udf):
+            expr = fn.expression(*(args or []))
+            if types is None:
+                if multi:
+                    if not fn.returns.upper().startswith("STRUCT"):
+                        raise ValueError(
+                            "several columns need a STRUCT-returning function"
+                        )
+                    types = struct_field_types(fn.returns)
+                else:
+                    types = fn.returns
        else:
-            return await self._inner.add_columns(list(transforms.items()))
+            if types is None:
+                raise ValueError("pass types= when fn is a name string")
+            expr = f"{fn}({', '.join(args or [])})"
+        if multi:
+            if len(types) != len(columns):
+                raise ValueError(
+                    f"{len(columns)} columns but {len(types)} output types"
+                )
+            computed = {c: (t, expr) for c, t in zip(columns, types)}
+        else:
+            computed = {columns: (types, expr)}
+        await self.add_columns(computed=computed)

    async def alter_columns(
        self, *alterations: Iterable[dict[str, Any]]
@@ -0,0 +1,753 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+"""UDF authoring for LanceDB derived compute (server-backed).
+
+`@udf` / `@table_udf` turn a plain Python function into a registrable
+server-side UDF: a cloudpickled (or source) body, a SQL signature inferred
+from type hints, and the runtime options (pip deps, GPUs, batching, ...).
+Register and use them through the existing connection/table API:
+
+    import lancedb
+    from lancedb import udf, table_udf
+
+    db = lancedb.connect("db://my_db", api_key="...", host_override="...")
+
+    @udf(pip=["torch>=2.0"], num_gpus=1)
+    def embed(text: str) -> list[float]:
+        return model.encode(text).tolist()
+
+    db.create_function(embed)                 # CREATE FUNCTION (once)
+    tbl = db.open_table("docs")
+    tbl.add_columns(computed={"vec": embed("text")})  # bind embed(text) -> vec
+    tbl.refresh_column("vec").wait()          # materialize (returns a JobHandle)
+    view = db.create_materialized_view("chunks", tbl, ["id", chunk_fn])
+
+`embed("text")` applies the registered function to the `text` column and yields
+the expression `embed(text)`; the function itself stays decoupled from any
+column, so the same `embed` works on any column or table.
+
+These operations are server-backed (LanceDB Enterprise / Cloud); the
+decorator itself works locally (define + call), only registration needs a
+remote connection.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import dataclasses
+import functools
+import inspect
+import re
+import sys
+import textwrap
+import time
+import typing
+
+# -- type hints -> SQL type strings -------------------------------------
+
+_SCALARS = {
+    int: "BIGINT",
+    # Pragmatic default for ML workloads: python float maps to FLOAT
+    # (Float32). Use an explicit `returns=` for DOUBLE.
+    float: "FLOAT",
+    str: "VARCHAR",
+    bool: "BOOLEAN",
+    bytes: "BLOB",
+}
+
+
+class TypeInferenceError(TypeError):
+    pass
+
+
+def sql_type(hint) -> str:
+    """SQL type string for a python type hint."""
+    if hint in _SCALARS:
+        return _SCALARS[hint]
+    origin = typing.get_origin(hint)
+    if origin in (list, typing.List):
+        (item,) = typing.get_args(hint) or (None,)
+        if item in _SCALARS:
+            return f"{_SCALARS[item]}[]"
+        raise TypeInferenceError(
+            f"unsupported list item type {item!r}; use an explicit returns="
+        )
+    fields = _struct_fields(hint)
+    if fields is not None:
+        inner = ", ".join(f"{name} {sql_type(h)}" for name, h in fields)
+        return f"STRUCT({inner})"
+    raise TypeInferenceError(
+        f"cannot infer a SQL type for {hint!r}; pass an explicit type string"
+    )
+
+
+def _struct_fields(hint):
+    """(name, hint) pairs for a TypedDict or dataclass, else None."""
+    if dataclasses.is_dataclass(hint):
+        return [(f.name, f.type) for f in dataclasses.fields(hint)]
+    # TypedDict detection: a dict subclass with __annotations__.
+    if (
+        isinstance(hint, type)
+        and issubclass(hint, dict)
+        and typing.get_type_hints(hint)
+    ):
+        return list(typing.get_type_hints(hint).items())
+    return None
+
+
+def return_type(fn, override: "str | None", table: bool) -> str:
+    """SQL return type for a function: explicit override wins, else the
+    return annotation. Table functions render as TABLE(...) and accept
+    struct-shaped hints (TypedDict/dataclass, optionally list-wrapped)."""
+    if override is not None:
+        s = override.strip()
+        if table and not s.upper().startswith("TABLE"):
+            if s.upper().startswith("STRUCT"):
+                return "TABLE" + s[len("STRUCT") :]
+            raise TypeInferenceError(
+                "a table function's returns= must be TABLE(...) or STRUCT(...)"
+            )
+        return s
+
+    hints = typing.get_type_hints(fn)
+    ret = hints.get("return")
+    if ret is None:
+        raise TypeInferenceError(
+            f"function {fn.__name__!r} needs a return annotation or returns="
+        )
+    if table:
+        # Accept list[Row] / Row where Row is a TypedDict or dataclass.
+        if typing.get_origin(ret) in (list, typing.List):
+            (ret,) = typing.get_args(ret)
+        fields = _struct_fields(ret)
+        if fields is None:
+            raise TypeInferenceError(
+                "a table function must return rows shaped as a TypedDict or "
+                "dataclass (optionally list-wrapped); or pass returns=..."
+            )
+        inner = ", ".join(f"{name} {sql_type(h)}" for name, h in fields)
+        return f"TABLE({inner})"
+    return sql_type(ret)
+
+
+def param_types(fn) -> "list[tuple[str, str]]":
+    """(name, sql type) per parameter, from annotations. Each UDF
+    parameter binds to a source column of the same name by default."""
+    hints = typing.get_type_hints(fn)
+    out = []
+    for name, p in inspect.signature(fn).parameters.items():
+        if p.kind in (p.VAR_POSITIONAL, p.VAR_KEYWORD):
+            raise TypeInferenceError("*args/**kwargs are not supported in UDFs")
+        hint = hints.get(name)
+        if hint is None:
+            raise TypeInferenceError(
+                f"parameter {name!r} of {fn.__name__!r} needs a type annotation"
+            )
+        out.append((name, sql_type(hint)))
+    return out
+
+
+# -- column expressions -------------------------------------------------
+
+
+class ColumnExpr(str):
+    """A computed-column expression produced by applying a registered
+    function to column names, e.g. ``embed("data") -> "embed(data)"``.
+
+    It IS the expression string everywhere a string is expected (views, SQL,
+    logging), and additionally carries the function's declared return type so
+    ``add_columns(computed=...)`` can declare the column without a hand-written
+    type. ``field_types`` holds the per-field SQL types of a STRUCT return, for
+    fanning one expression out to several columns.
+    """
+
+    data_type: "str | None"
+    field_types: "list[str] | None"
+
+    def __new__(cls, expr: str, data_type=None, field_types=None):
+        obj = super().__new__(cls, expr)
+        obj.data_type = data_type
+        obj.field_types = field_types
+        return obj
+
+
+def _normalize_computed(computed: dict) -> dict:
+    """Normalize the user-facing ``computed=`` mapping to the canonical
+    ``{name: (sql_type, expression)}`` form.
+
+    Accepts, per entry:
+      - value is a `ColumnExpr` (from ``fn("col")``): the column's SQL type
+        comes from the function's return type -- no hand-written type needed. A
+        tuple key (``("chunk", "idx")``) fans a STRUCT return out to one
+        (type, expression) entry per field, in declared order.
+      - value is a legacy ``(sql_type, expression)`` tuple: passed through (the
+        escape hatch, e.g. bare-name function strings).
+    """
+    out: dict = {}
+    for key, val in computed.items():
+        if isinstance(val, ColumnExpr):
+            expr = str(val)
+            if isinstance(key, (tuple, list)):
+                if not val.field_types:
+                    raise ValueError(
+                        f"columns {tuple(key)} need a STRUCT-returning function; "
+                        f"{expr} returns a single value"
+                    )
+                if len(val.field_types) != len(key):
+                    raise ValueError(
+                        f"{len(key)} columns but {len(val.field_types)} struct fields "
+                        f"in {expr}"
+                    )
+                for name, t in zip(key, val.field_types):
+                    out[name] = (t, expr)
+            else:
+                if val.data_type is None:
+                    raise ValueError(f"cannot infer a type for {expr}; pass types=")
+                out[key] = (val.data_type, expr)
+        else:
+            out[key] = val
+    return out
+
+
+# -- the @udf / @table_udf decorators -----------------------------------
+
+
+class Udf:
+    def __init__(
+        self,
+        fn,
+        *,
+        returns: "str | None" = None,
+        table: bool = False,
+        name: "str | None" = None,
+        pip: "list[str] | None" = None,
+        pip_index_url: "str | None" = None,
+        pip_extra_index_urls: "list[str] | None" = None,
+        find_links: "list[str] | None" = None,
+        requirements: "str | list[str] | None" = None,
+        conda: "list[str] | None" = None,
+        conda_channels: "list[str] | None" = None,
+        env: "dict[str, str] | list[str] | None" = None,
+        num_cpus: "int | None" = None,
+        num_gpus: "int | None" = None,
+        batch_size: "int | None" = None,
+        timeout: "float | None" = None,
+        error_policy: "str | None" = None,
+        max_skip_ratio: "float | None" = None,
+        retries: "int | None" = None,
+        docker_image: "str | None" = None,
+        description: "str | None" = None,
+        prefer_source: bool = False,
+    ):
+        functools.update_wrapper(self, fn)
+        self.fn = fn
+        self.name = name or fn.__name__
+        self.table = table
+        self.params = param_types(fn)
+        self.returns = return_type(fn, returns, table)
+        self.prefer_source = prefer_source
+        self.options: "dict[str, str]" = {}
+        if conda and (pip or requirements):
+            raise ValueError("pass conda or pip/requirements, not both")
+        if conda_channels and not conda:
+            raise ValueError("conda_channels requires conda")
+        if pip:
+            self.options["pip"] = ",".join(pip)
+        if pip_extra_index_urls:
+            self.options["pip_extra_index_urls"] = ",".join(pip_extra_index_urls)
+        if find_links:
+            self.options["find_links"] = ",".join(find_links)
+        if requirements:
+            self.options["requirements"] = _format_requirements(requirements)
+        if conda:
+            self.options["conda"] = ",".join(conda)
+        if conda_channels:
+            self.options["conda_channels"] = ",".join(conda_channels)
+        if env:
+            self.options["env"] = _format_env(env)
+        for key, val in [
+            ("pip_index_url", pip_index_url),
+            ("num_cpus", num_cpus),
+            ("num_gpus", num_gpus),
+            ("batch_size", batch_size),
+            ("timeout", timeout),
+            ("error_policy", error_policy),
+            ("max_skip_ratio", max_skip_ratio),
+            ("retries", retries),
+            ("docker_image", docker_image),
+        ]:
+            if val is not None:
+                self.options[key] = str(val)
+        # Keep the source in the description (when available) so the
+        # catalog stays inspectable even for pickled bodies.
+        if description is not None:
+            self.options["description"] = description
+        else:
+            try:
+                self.options["description"] = textwrap.dedent(inspect.getsource(fn))
+            except (OSError, TypeError):
+                pass
+
+    def __call__(self, *args, **kwargs):
+        """Call with real values to run locally; call with column-name
+        strings to build an expression for backfills and views, e.g.
+        ``embed("data")`` -> the expression ``embed(data)`` (a `ColumnExpr`
+        carrying the function's return type for `add_columns(computed=...)`)."""
+        if args and all(isinstance(a, str) for a in args) and not kwargs:
+            return self.expression(*args)
+        return self.fn(*args, **kwargs)
+
+    def expression(self, *columns: str) -> ColumnExpr:
+        """The expression applying this function to `columns` (default: the
+        function's own parameter names). Returns a `ColumnExpr` -- a string
+        that also carries the declared return type (and struct field types)."""
+        cols = columns or [p for p, _ in self.params]
+        expr = f"{self.name}({', '.join(cols)})"
+        field_types = None
+        if self.returns.upper().startswith("STRUCT"):
+            field_types = struct_field_types(self.returns)
+        return ColumnExpr(expr, data_type=self.returns, field_types=field_types)
+
+    def _body(self) -> "tuple[str, str]":
+        """(body literal, body_format). Source when requested and
+        retrievable; cloudpickle otherwise (handles closures)."""
+        if self.prefer_source:
+            try:
+                src = textwrap.dedent(inspect.getsource(self.fn))
+                # Strip the decorator line(s) so the stored body is a
+                # plain function definition.
+                lines = src.splitlines(keepends=True)
+                while lines and lines[0].lstrip().startswith("@"):
+                    lines.pop(0)
+                return "".join(lines), "source"
+            except (OSError, TypeError):
+                pass
+        import cloudpickle
+
+        raw = cloudpickle.dumps(self.fn)
+        return base64.b64encode(raw).decode("ascii"), "cloudpickle"
+
+    def _body_and_options(self) -> "tuple[str, dict[str, str]]":
+        """The body literal plus the finalized options (body_format /
+        python_version / cloudpickle-pip bookkeeping for a non-source
+        body)."""
+        body, body_format = self._body()
+        options = dict(self.options)
+        if body_format != "source":
+            options["body_format"] = body_format
+            # Pickled code objects only load under the same interpreter
+            # minor version; record ours so the worker can fail with a
+            # clear message instead of a bytecode error.
+            options["python_version"] = self.pickle_environment()
+            # The worker deserializes the body with cloudpickle; make sure
+            # the job's pip environment provides it. Conda bakes inject
+            # cloudpickle server-side, so do not create an invalid pip+conda
+            # declaration here.
+            if "conda" not in options:
+                pip = [d for d in options.get("pip", "").split(",") if d]
+                if not any(d.startswith("cloudpickle") for d in pip):
+                    pip.append("cloudpickle")
+                options["pip"] = ",".join(pip)
+        return body, options
+
+    def create_request(self) -> dict:
+        """Keyword arguments for `connection.create_function`."""
+        body, options = self._body_and_options()
+        return {
+            "name": self.name,
+            "language": "python",
+            "return_type": self.returns,
+            "body": body,
+            "options": options,
+        }
+
+    def create_statement(self) -> str:
+        """The equivalent `CREATE FUNCTION` SQL (for SQL-surface callers)."""
+        params = ", ".join(f"{n} {t}" for n, t in self.params)
+        body, options = self._body_and_options()
+        with_clause = ""
+        if options:
+            rendered = ", ".join(
+                f"{k} = '{_escape(v)}'" for k, v in sorted(options.items())
+            )
+            with_clause = f" WITH ({rendered})"
+        return (
+            f"CREATE FUNCTION {self.name}({params}) RETURNS {self.returns} "
+            f"LANGUAGE python AS '{_escape_body(body)}'{with_clause}"
+        )
+
+    def pickle_environment(self) -> str:
+        """Python version the body pickles under -- workers should match
+        the minor version for cloudpickle compatibility."""
+        return f"{sys.version_info.major}.{sys.version_info.minor}"
+
+
+def _escape(s: str) -> str:
+    return str(s).replace("'", "''")
+
+
+def _format_requirements(requirements: "str | list[str]") -> str:
+    if isinstance(requirements, str):
+        return requirements
+    return "\n".join(str(req) for req in requirements)
+
+
+def _format_env(env: "dict[str, str] | list[str]") -> str:
+    if isinstance(env, dict):
+        return "; ".join(f"{key}={value}" for key, value in env.items())
+    return "; ".join(str(entry) for entry in env)
+
+
+def _escape_body(body: str) -> str:
+    # The server unescapes \n / \t in single-quoted bodies; encode real
+    # newlines accordingly and escape quotes.
+    return (
+        body.replace("\\", "\\\\")
+        .replace("'", "''")
+        .replace("\n", "\\n")
+        .replace("\t", "\\t")
+    )
+
+
+def udf(fn=None, **kwargs):
+    """Decorate a function as a scalar (or struct-returning) UDF.
+
+    @udf
+    def doubled(val: int) -> float: ...
+
+    @udf(pip=["torch>=2"], num_gpus=1)
+    def embed(body: str) -> list[float]: ...
+    """
+    if fn is not None:
+        return Udf(fn, **kwargs)
+    return lambda f: Udf(f, **kwargs)
+
+
+def table_udf(fn=None, **kwargs):
+    """Decorate a table function (UDTF): each input row may emit zero or
+    more output rows. Only usable in materialized views.
+
+        class Chunk(TypedDict):
+            chunk: str
+            chunk_idx: int
+
+        @table_udf
+        def chunker(body: str) -> list[Chunk]: ...
+    """
+    kwargs["table"] = True
+    if fn is not None:
+        return Udf(fn, **kwargs)
+    return lambda f: Udf(f, **kwargs)
+
+
+# -- view / job handles (thin references over a connection) -------------
+
+
+def struct_field_types(returns: str) -> "list[str]":
+    """Field type strings of a STRUCT(...) SQL type, in declared order."""
+    inner = returns.strip()[len("STRUCT(") : -1]
+    fields, depth, start = [], 0, 0
+    for i, c in enumerate(inner):
+        if c in "([":
+            depth += 1
+        elif c in ")]":
+            depth -= 1
+        elif c == "," and depth == 0:
+            fields.append(inner[start:i].strip())
+            start = i + 1
+    fields.append(inner[start:].strip())
+    # Each field is "name TYPE"; drop the name.
+    return [f.split(None, 1)[1] for f in fields]
+
+
+def build_view_query(source, select) -> str:
+    """Assemble a view SELECT from a source (name or table) and select
+    items: a column name, an expression string, a (alias, expression)
+    tuple, or a @udf/@table_udf object."""
+    src = source.name if hasattr(source, "name") else source
+    items = []
+    for item in select:
+        if isinstance(item, Udf):
+            items.append(item.expression())
+        elif isinstance(item, tuple):
+            alias, expr = item
+            expr = expr.expression() if isinstance(expr, Udf) else expr
+            items.append(f"{expr} AS {alias}")
+        else:
+            items.append(item)
+    return f"SELECT {', '.join(items)} FROM {src}"
+
+
+def _job_id_matches(handle_id: str, listed_id: str) -> bool:
+    # The refresh/backfill endpoints return the submission id (a uuid), but
+    # the agent names the manifest job "<table>-<type>-<first 8 of the
+    # submission id>" -- which is what list_jobs and cancel report. Match the
+    # canonical id directly, or by that submission prefix.
+    if listed_id == handle_id:
+        return True
+    prefix = handle_id[:8]
+    return len(prefix) >= 4 and prefix in listed_id
+
+
+class MaterializedView:
+    """A reference to a materialized view (name + connection). Operations are
+    server-backed connection calls bound to the name.
+
+    ``create_materialized_view`` returns one of these; ``job_id`` is the
+    initial-population job (None when the view was created with no data), so
+    ``db.create_materialized_view(...).wait()`` blocks until it is populated.
+    """
+
+    def __init__(self, conn, name: str, job_id: "str | None" = None):
+        self.conn = conn
+        self.name = name
+        #: initial-population job id from create, or None (with_no_data).
+        self.job_id = job_id
+
+    def wait(self, timeout: float = 3600.0, poll: float = 2.0) -> str:
+        """Block until the initial-population job (from create) finishes.
+        A no-op when the view was created with no data."""
+        if self.job_id is None:
+            return "finished"
+        return JobHandle(self.conn, self.job_id, table=self.name).wait(
+            timeout=timeout, poll=poll
+        )
+
+    def refresh(self, full: bool = False) -> "JobHandle":
+        """Refresh the materialized view; returns a `JobHandle` to wait on,
+        poll, or cancel (``view.refresh().wait()``).
+
+        ``full=True`` forces a full rebuild (recompute and replace every row)
+        instead of the default incremental refresh. A full rebuild preserves
+        the view's indexes -- they are reindexed by the distributed indexer.
+        """
+        job_id = self.conn._refresh_materialized_view(self.name, full=full)
+        return JobHandle(self.conn, job_id, table=self.name)
+
+    def explain_refresh(self, full: bool = False):
+        """Plan a refresh without running it (EXPLAIN REFRESH)."""
+        return self.conn.explain_refresh_materialized_view(self.name, full=full)
+
+    def alter(self, auto_refresh: bool) -> None:
+        self.conn.alter_materialized_view(self.name, auto_refresh=auto_refresh)
+
+    def drop(self) -> None:
+        self.conn.drop_materialized_view(self.name)
+
+    # A materialized view is a first-class table: it can be indexed and
+    # searched like any other. These open the materialized dataset by name and
+    # delegate. Indexes declared this way are recorded against the view, so the
+    # engine re-applies them after a full refresh rebuilds the dataset (a full
+    # refresh overwrites the dataset, which would otherwise drop its indices).
+    def _table(self):
+        return self.conn.open_table(self.name)
+
+    def create_index(self, *args, **kwargs):
+        """Build an index on the materialized view (see Table.create_index)."""
+        return self._table().create_index(*args, **kwargs)
+
+    def create_scalar_index(self, *args, **kwargs):
+        """Build a scalar index on the materialized view."""
+        return self._table().create_scalar_index(*args, **kwargs)
+
+    def create_fts_index(self, *args, **kwargs):
+        """Build a full-text-search index on the materialized view."""
+        return self._table().create_fts_index(*args, **kwargs)
+
+    def search(self, *args, **kwargs):
+        """Search the materialized view (vector / FTS / hybrid)."""
+        return self._table().search(*args, **kwargs)
+
+    def lineage(self, column=None, *, direction=None, depth=None):
+        """Lineage of the materialized view (or one of its columns). Delegates
+        to the backing table; the server already includes the view's sources
+        and downstream dependents. Returns a `Lineage`."""
+        return self._table().lineage(column, direction=direction, depth=depth)
+
+
+_PROGRESS = re.compile(r"(\d+)/(\d+)")
+
+
+class JobFailedError(RuntimeError):
+    """Raised by ``JobHandle.wait()`` when the server reports the job ``failed``.
+
+    Carries the server-side error so a doomed backfill (e.g. a multi-column
+    ``REFRESH COLUMN`` of a scalar UDF) surfaces its real cause promptly,
+    instead of the caller blocking until ``wait()``'s timeout.
+    """
+
+    def __init__(self, job_id: str, error: "str | None"):
+        self.job_id = job_id
+        self.error = error
+        super().__init__(f"job {job_id} failed: {error or 'unknown error'}")
+
+
+class JobHandle:
+    """A reference to an inflight server-side job, with polling helpers."""
+
+    #: How long an unseen job is treated as still materializing (submission
+    #: -> agent cycle -> manifest write is async).
+    GRACE_SECONDS = 20.0
+
+    def __init__(self, conn, job_id: str, table: "str | None" = None):
+        self.conn = conn
+        self.id = job_id
+        #: The job's table, when known (refresh_column / MV refresh). Lets the
+        #: server resolve this job with an O(1) single-node read; without it the
+        #: lookup scans the database's active jobs (still correct).
+        self.table = table
+        self._created = time.monotonic()
+        self._seen = False
+
+    def _job(self):
+        # Poll by id (one job), not list_jobs (every active job): the server
+        # matches the submission/manifest id and reads just this table's node.
+        return self.conn.get_job(self.id, self.table)
+
+    def status(self) -> str:
+        """pending / running / cancelling / stale, or 'finished' once the
+        job has left the inflight listing."""
+        job = self._job()
+        if job is not None:
+            self._seen = True
+            return job.state
+        if not self._seen and time.monotonic() - self._created < self.GRACE_SECONDS:
+            return "pending"
+        return "finished"
+
+    def progress(self) -> "tuple[int, int] | None":
+        """(units_done, units_total) while running, else None."""
+        job = self._job()
+        if job is not None and job.units_total is not None:
+            return job.units_done or 0, job.units_total
+        return None
+
+    def wait(self, timeout: float = 3600.0, poll: float = 2.0) -> str:
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            state = self.status()
+            if state in ("finished", "stale"):
+                return state
+            if state == "failed":
+                # Terminal failure -- surface the server error now, don't block
+                # until `timeout`. `finalize` wrote it to the job's status node.
+                job = self._job()
+                raise JobFailedError(self.id, job.error if job is not None else None)
+            if state == "pending":
+                time.sleep(min(poll, 0.5))
+                continue
+            job = self._job()
+            if job is not None and job.committed:
+                return "finished"
+            time.sleep(poll)
+        raise TimeoutError(f"job {self.id} still {self.status()} after {timeout}s")
+
+    def cancel(self) -> None:
+        # Cancel by the canonical manifest id (what cancel matches), found
+        # via the submission prefix; fall back to the raw id.
+        job = self._job()
+        self.conn.cancel_job(job.job_id if job is not None else self.id)
+
+
+class AsyncMaterializedView:
+    """Async reference to a materialized view (name + async connection)."""
+
+    def __init__(self, conn, name: str, job_id: "str | None" = None):
+        self.conn = conn
+        self.name = name
+        #: initial-population job id from create, or None (with_no_data).
+        self.job_id = job_id
+
+    async def wait(self, timeout: float = 3600.0, poll: float = 2.0) -> str:
+        """Block until the initial-population job (from create) finishes.
+        A no-op when the view was created with no data."""
+        if self.job_id is None:
+            return "finished"
+        return await AsyncJobHandle(self.conn, self.job_id, table=self.name).wait(
+            timeout=timeout, poll=poll
+        )
+
+    async def refresh(self, full: bool = False) -> "AsyncJobHandle":
+        """Refresh the materialized view; returns an `AsyncJobHandle` to wait
+        on, poll, or cancel.
+
+        ``full=True`` forces a full rebuild instead of an incremental refresh
+        (indexes are preserved and reindexed by the distributed indexer).
+        """
+        job_id = await self.conn._refresh_materialized_view(self.name, full=full)
+        return AsyncJobHandle(self.conn, job_id, table=self.name)
+
+    async def explain_refresh(self, full: bool = False):
+        return await self.conn.explain_refresh_materialized_view(self.name, full=full)
+
+    async def alter(self, auto_refresh: bool) -> None:
+        await self.conn.alter_materialized_view(self.name, auto_refresh=auto_refresh)
+
+    async def drop(self) -> None:
+        await self.conn.drop_materialized_view(self.name)
+
+    async def lineage(self, column=None, *, direction=None, depth=None):
+        """Lineage of the materialized view (or column). Returns a `Lineage`."""
+        return await self.conn.lineage(
+            self.name, column, direction=direction, depth=depth
+        )
+
+
+class AsyncJobHandle:
+    """Async reference to an inflight server-side job, with polling helpers."""
+
+    GRACE_SECONDS = 20.0
+
+    def __init__(self, conn, job_id: str, table: "str | None" = None):
+        self.conn = conn
+        self.id = job_id
+        #: See JobHandle.table -- enables an O(1) by-id lookup when known.
+        self.table = table
+        self._created = time.monotonic()
+        self._seen = False
+
+    async def _job(self):
+        # Poll by id, not list_jobs (see JobHandle._job).
+        return await self.conn.get_job(self.id, self.table)
+
+    async def status(self) -> str:
+        job = await self._job()
+        if job is not None:
+            self._seen = True
+            return job.state
+        if not self._seen and time.monotonic() - self._created < self.GRACE_SECONDS:
+            return "pending"
+        return "finished"
+
+    async def progress(self) -> "tuple[int, int] | None":
+        job = await self._job()
+        if job is not None and job.units_total is not None:
+            return job.units_done or 0, job.units_total
+        return None
+
+    async def wait(self, timeout: float = 3600.0, poll: float = 2.0) -> str:
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            state = await self.status()
+            if state in ("finished", "stale"):
+                return state
+            if state == "failed":
+                # Terminal failure -- surface the server error now, don't block
+                # until `timeout`. `finalize` wrote it to the job's status node.
+                job = await self._job()
+                raise JobFailedError(self.id, job.error if job is not None else None)
+            if state == "pending":
+                await asyncio.sleep(min(poll, 0.5))
+                continue
+            job = await self._job()
+            if job is not None and job.committed:
+                return "finished"
+            await asyncio.sleep(poll)
+        raise TimeoutError(
+            f"job {self.id} still {await self.status()} after {timeout}s"
+        )
+
+    async def cancel(self) -> None:
+        job = await self._job()
+        await self.conn.cancel_job(job.job_id if job is not None else self.id)
@@ -177,10 +177,7 @@ def flatten_columns(tbl: pa.Table, flatten: Optional[Union[int, bool]] = None):
                continue
            else:
                break
-    # `bool` is a subclass of `int`, so guard against it explicitly: `flatten=False`
-    # (and `None`) must mean "do not flatten" rather than falling into the integer
-    # branch and raising on the `flatten <= 0` check.
-    elif isinstance(flatten, int) and not isinstance(flatten, bool):
+    elif isinstance(flatten, int):
        if flatten <= 0:
            raise ValueError(
                "Please specify a positive integer for flatten or the boolean "
@@ -518,38 +515,3 @@ def batch_to_tensor_rows(batch: pa.RecordBatch):
    stacked = torch.tensor(numpy.column_stack(columns))
    rows = list(stacked.unbind(dim=0))
    return rows
-
-
-def batch_to_tensor_dict(batch: pa.RecordBatch):
-    """
-    Convert a PyArrow RecordBatch to a list of per-row dicts of PyTorch Tensors.
-
-    Each column is first converted to a 1-D tensor (zero-copy via DLPack), then
-    sliced per row. The result is a list whose length is ``batch.num_rows`` and
-    whose items are dicts keyed by column name. This shape composes directly
-    with PyTorch's default ``DataLoader`` collate, which stacks the per-row
-    dicts back into a dict of batched tensors.
-
-    Fails if torch is not installed.
-    Fails if a column's data type is not supported by PyTorch.
-
-    Parameters
-    ----------
-    batch : pa.RecordBatch
-        The record batch to convert.
-
-    Returns
-    -------
-    list[dict[str, torch.Tensor]]
-        One per-row dict per row in the batch. Each dict maps column name to a
-        0-D tensor view into the column.
-    """
-    torch = attempt_import_or_raise("torch", "torch")
-    tensors = {
-        name: torch.from_dlpack(col)
-        for name, col in zip(batch.schema.names, batch.columns)
-    }
-    return [
-        {name: tensor[i] for name, tensor in tensors.items()}
-        for i in range(batch.num_rows)
-    ]
@@ -2,7 +2,6 @@
 # SPDX-FileCopyrightText: Copyright The LanceDB Authors

 import os
-import pickle
 from typing import List, Optional, Union
 from unittest.mock import MagicMock, patch

@@ -243,49 +242,6 @@ def test_embedding_with_bad_results(tmp_path):
    assert tbl["vector"].null_count == 1


-def test_embedding_with_empty_output_vectors(tmp_path):
-    """Regression test for issue #1672.
-
-    When an embedding function returns an empty list (e.g. for empty-string
-    inputs), _append_vector_columns used to crash because PyArrow cannot cast
-    [] into a fixed-size list element.  The fix replaces wrong-length vectors
-    with None before building the Arrow array so that _handle_bad_vectors can
-    process them normally.
-    """
-
-    @register("empty-vec-embedding")
-    class EmptyVecEmbeddingFunction(TextEmbeddingFunction):
-        def ndims(self):
-            return 128
-
-        def generate_embeddings(self, texts: Union[List[str], np.ndarray]) -> list:
-            # Simulate a model that returns an empty list for blank inputs
-            return [
-                [] if text.strip() == "" else np.random.randn(self.ndims()).tolist()
-                for text in texts
-            ]
-
-    db = lancedb.connect(tmp_path)
-    registry = EmbeddingFunctionRegistry.get_instance()
-    model = registry.get("empty-vec-embedding").create()
-
-    class Schema(LanceModel):
-        text: str = model.SourceField()
-        vector: Vector(model.ndims()) = model.VectorField()
-
-    table = db.create_table("test_empty_vec", schema=Schema, mode="overwrite")
-
-    # Should not crash; the row with the empty string should be dropped
-    table.add(
-        [{"text": "hello world"}, {"text": ""}, {"text": "foo"}],
-        on_bad_vectors="drop",
-    )
-
-    assert len(table) == 2
-    texts = table.to_arrow()["text"].to_pylist()
-    assert "" not in texts
-
-
 def test_with_existing_vectors(tmp_path):
    @register("mock-embedding")
    class MockEmbeddingFunction(TextEmbeddingFunction):
@@ -592,22 +548,6 @@ def test_openai_no_retry_on_401(mock_sleep):
    assert mock_sleep.call_count == 0


-def test_ollama_embeddings_pickle():
-    """OllamaEmbeddings must pickle even after the cached client is created."""
-    registry = get_registry()
-    model = registry.get("ollama").create(name="nomic-embed-text")
-
-    # Simulate accessing the cached client, which stores it on the instance.
-    model.__dict__["_ollama_client"] = MagicMock()
-
-    pickled = pickle.dumps(model)
-    restored = pickle.loads(pickled)
-
-    assert restored.name == "nomic-embed-text"
-    assert restored.host == "http://localhost:11434"
-    assert "_ollama_client" not in restored.__dict__
-
-
 def test_url_retrieve_downloads_image():
    """
    Embedding functions like open-clip, siglip, and jinaai use url_retrieve()
@@ -0,0 +1,92 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+"""JobHandle.wait() terminal-state handling.
+
+Regression coverage for the cluster backfill-failure hang: the server reports a
+doomed job as ``state="failed"`` within seconds, but ``wait()`` used to ignore
+``failed`` and block until its (default 3600s) timeout. These tests pin that a
+``failed`` job raises ``JobFailedError`` promptly, carrying the server error.
+"""
+
+import asyncio
+import time
+
+import pytest
+
+from lancedb.udf import JobHandle, AsyncJobHandle, JobFailedError
+
+
+class FakeJobInfo:
+    """Mirror of the pyo3 builtins.JobInfo fields wait()/status() read."""
+
+    def __init__(self, state, error=None, committed=False, units_total=None):
+        self.state = state
+        self.error = error
+        self.committed = committed
+        self.units_total = units_total
+        self.units_done = None
+        self.job_id = "job-1"
+
+
+class FakeConn:
+    """get_job() walks a scripted list of JobInfo (or None) snapshots, holding
+    the last one once exhausted, so wait() polls a deterministic timeline."""
+
+    def __init__(self, snapshots):
+        self._snaps = list(snapshots)
+        self.calls = 0
+
+    def get_job(self, job_id, table=None):
+        snap = self._snaps[min(self.calls, len(self._snaps) - 1)]
+        self.calls += 1
+        return snap
+
+
+class AsyncFakeConn(FakeConn):
+    async def get_job(self, job_id, table=None):
+        return FakeConn.get_job(self, job_id, table)
+
+
+def test_wait_raises_on_failed_promptly():
+    # pending -> failed: wait() must raise the server error, not TimeoutError.
+    conn = FakeConn(
+        [None, FakeJobInfo("failed", error="multi-column backfill needs a STRUCT")]
+    )
+    jh = JobHandle(conn, "job-1", table="t")
+    t0 = time.monotonic()
+    with pytest.raises(JobFailedError) as exc:
+        jh.wait(timeout=30, poll=0.01)
+    assert time.monotonic() - t0 < 5  # prompt, nowhere near the 30s timeout
+    assert "STRUCT" in str(exc.value)
+    assert exc.value.error == "multi-column backfill needs a STRUCT"
+    assert exc.value.job_id == "job-1"
+
+
+def test_wait_returns_finished_on_success():
+    # running -> finished (job left the inflight listing) returns normally.
+    conn = FakeConn([FakeJobInfo("running", units_total=2), None])
+    jh = JobHandle(conn, "job-1", table="t")
+    jh._seen = True  # already observed, so a None now means "finished" not grace
+    assert jh.wait(timeout=30, poll=0.01) == "finished"
+
+
+def test_wait_returns_finished_on_committed():
+    # A committed job that is still listed resolves to finished.
+    conn = FakeConn([FakeJobInfo("running", committed=True, units_total=2)])
+    jh = JobHandle(conn, "job-1", table="t")
+    jh._seen = True
+    assert jh.wait(timeout=30, poll=0.01) == "finished"
+
+
+def test_async_wait_raises_on_failed_promptly():
+    conn = AsyncFakeConn([None, FakeJobInfo("failed", error="boom")])
+    jh = AsyncJobHandle(conn, "job-1", table="t")
+
+    async def run():
+        t0 = time.monotonic()
+        with pytest.raises(JobFailedError) as exc:
+            await jh.wait(timeout=30, poll=0.01)
+        assert time.monotonic() - t0 < 5
+        assert exc.value.error == "boom"
+
+    asyncio.run(run())
@@ -11,7 +11,6 @@ import lancedb
 import pyarrow as pa
 import pytest
 from lancedb._lancedb import LsmWriteSpec
-from lancedb.index import BTree

 SCHEMA = pa.schema(
    [
@@ -137,76 +136,3 @@ def test_lsm_write_spec_identity_and_writer_config_defaults():
    s = s.with_writer_config_defaults({"durable_write": "false"})
    assert s.writer_config_defaults == {"durable_write": "false"}
    assert "durable_write" in repr(s)
-
-
-def test_get_lsm_write_spec(tmp_path):
-    _db, table = _make_table(tmp_path)
-    table.set_unenforced_primary_key("id")
-
-    # None when nothing is installed.
-    assert table.get_lsm_write_spec() is None
-
-    # A real scalar index is needed to name it as a maintained index.
-    table.create_index("id", config=BTree())
-    idx_name = table.list_indices()[0].name
-
-    # Bucket spec round-trips, including maintained indexes and writer config
-    # defaults.
-    table.set_lsm_write_spec(
-        LsmWriteSpec.bucket("id", 4)
-        .with_maintained_indexes([idx_name])
-        .with_writer_config_defaults({"durable_write": "false"})
-    )
-    spec = table.get_lsm_write_spec()
-    assert spec is not None
-    assert spec.spec_type == "bucket"
-    assert spec.column == "id"
-    assert spec.num_buckets == 4
-    assert spec.maintained_indexes == [idx_name]
-    assert spec.writer_config_defaults == {"durable_write": "false"}
-
-    # After unset, None again.
-    table.unset_lsm_write_spec()
-    assert table.get_lsm_write_spec() is None
-
-    # Identity round-trips (column recovered from the schema).
-    table.set_lsm_write_spec(LsmWriteSpec.identity("id"))
-    spec = table.get_lsm_write_spec()
-    assert spec.spec_type == "identity"
-    assert spec.column == "id"
-    table.unset_lsm_write_spec()
-
-    # Unsharded round-trips (no routing column).
-    table.set_lsm_write_spec(LsmWriteSpec.unsharded())
-    spec = table.get_lsm_write_spec()
-    assert spec.spec_type == "unsharded"
-    assert spec.column is None
-
-
-@pytest.mark.asyncio
-async def test_async_get_lsm_write_spec(tmp_path):
-    db = await lancedb.connect_async(
-        tmp_path, read_consistency_interval=timedelta(seconds=0)
-    )
-    table = await db.create_table(
-        "t",
-        pa.RecordBatchReader.from_batches(SCHEMA, [_batch(["seed"], [0])]),
-    )
-
-    assert await table.get_lsm_write_spec() is None
-
-    # A real scalar index is needed to name it as a maintained index.
-    await table.create_index("id", config=BTree())
-    idx_name = (await table.list_indices())[0].name
-
-    await table.set_lsm_write_spec(
-        LsmWriteSpec.bucket("id", 8).with_maintained_indexes([idx_name])
-    )
-    spec = await table.get_lsm_write_spec()
-    assert spec is not None
-    assert spec.spec_type == "bucket"
-    assert spec.column == "id"
-    assert spec.num_buckets == 8
-    assert spec.maintained_indexes == [idx_name]
-    await table.unset_lsm_write_spec()
-    assert await table.get_lsm_write_spec() is None
@@ -5,7 +5,6 @@

 import tempfile
 import shutil
-import importlib
 import pytest
 import pyarrow as pa
 import lancedb
@@ -104,40 +103,6 @@ class TestNamespaceConnection:
        assert isinstance(db, lancedb.LanceNamespaceDBConnection)
        assert len(list(db.table_names())) == 0

-    def test_sync_builtin_namespace_uses_rust_without_python_client(self, monkeypatch):
-        """Built-in sync namespace connections should not construct or call the
-        Python namespace client for normal namespace/table management."""
-        namespace_module = importlib.import_module("lancedb.namespace")
-
-        def fail_namespace_connect(*args, **kwargs):
-            raise AssertionError("Python namespace client should not be constructed")
-
-        monkeypatch.setattr(
-            namespace_module, "namespace_connect", fail_namespace_connect
-        )
-
-        db = lancedb.connect_namespace("dir", {"root": self.temp_dir})
-        assert isinstance(db, lancedb.LanceNamespaceDBConnection)
-        assert db._namespace_client is None
-        assert db._route_pushdown_to_rust is True
-
-        db.create_namespace(["test_ns"])
-        assert "test_ns" in db.list_namespaces().namespaces
-
-        schema = pa.schema([pa.field("id", pa.int64())])
-        table = db.create_table("test_table", schema=schema, namespace_path=["test_ns"])
-        assert table.namespace == ["test_ns"]
-        assert "test_table" in db.table_names(namespace_path=["test_ns"])
-        assert "test_table" in db.list_tables(namespace_path=["test_ns"]).tables
-
-        opened = db.open_table("test_table", namespace_path=["test_ns"])
-        assert opened.namespace == ["test_ns"]
-
-        db.drop_table("test_table", namespace_path=["test_ns"])
-        assert db.list_tables(namespace_path=["test_ns"]).tables == []
-        db.drop_namespace(["test_ns"])
-        assert "test_ns" not in db.list_namespaces().namespaces
-
    def test_create_table_through_namespace(self):
        """Test creating a table through namespace."""
        db = lancedb.connect_namespace("dir", {"root": self.temp_dir})
@@ -599,61 +564,6 @@ class TestAsyncNamespaceConnection:
        table_names = await db.table_names()
        assert len(list(table_names)) == 0

-    async def test_async_builtin_namespace_uses_rust_without_python_client(
-        self, monkeypatch
-    ):
-        """Built-in async namespace connections should not construct or call the
-        Python namespace client for normal namespace/table management."""
-        namespace_module = importlib.import_module("lancedb.namespace")
-
-        def fail_namespace_connect(*args, **kwargs):
-            raise AssertionError("Python namespace client should not be constructed")
-
-        monkeypatch.setattr(
-            namespace_module, "namespace_connect", fail_namespace_connect
-        )
-
-        db = lancedb.connect_namespace_async("dir", {"root": self.temp_dir})
-        assert isinstance(db, lancedb.AsyncLanceNamespaceDBConnection)
-        assert db._namespace_client is None
-        assert db._route_pushdown_to_rust is True
-
-        await db.create_namespace(["test_ns"])
-        assert "test_ns" in (await db.list_namespaces()).namespaces
-
-        schema = pa.schema([pa.field("id", pa.int64())])
-        table = await db.create_table(
-            "test_table", schema=schema, namespace_path=["test_ns"]
-        )
-        assert table._namespace_path == ["test_ns"]
-        assert table._namespace_client is None
-        assert table._route_pushdown_to_rust is True
-        assert "test_table" in await db.table_names(namespace_path=["test_ns"])
-        assert "test_table" in (await db.list_tables(namespace_path=["test_ns"])).tables
-
-        opened = await db.open_table("test_table", namespace_path=["test_ns"])
-        assert opened._namespace_path == ["test_ns"]
-
-        await db.drop_table("test_table", namespace_path=["test_ns"])
-        assert (await db.list_tables(namespace_path=["test_ns"])).tables == []
-        await db.drop_namespace(["test_ns"])
-        assert "test_ns" not in (await db.list_namespaces()).namespaces
-
-    async def test_async_namespace_client_is_lazy(self):
-        """namespace_client() should still return the backing client on demand."""
-        pytest.importorskip("lance")
-        from lance.namespace import DirectoryNamespace
-
-        db = lancedb.connect_namespace_async("dir", {"root": self.temp_dir})
-        assert db._namespace_client is None
-
-        ns_client = await db.namespace_client()
-
-        assert isinstance(ns_client, DirectoryNamespace)
-        namespace_id = ns_client.namespace_id().replace("\\\\", "\\")
-        assert str(self.temp_dir) in namespace_id
-        assert db._namespace_client is ns_client
-
    # Async connect via namespace helper is not enabled yet.

    async def test_create_table_async(self):
@@ -908,11 +818,10 @@ class TestPushdownOperations:
        )
        assert db._route_pushdown_to_rust is True

-    def test_route_pushdown_to_rust_for_native_dir(self):
-        """The sync dir connection is natively built and defers QueryTable
-        pushdown to Rust."""
+    def test_route_pushdown_to_rust_false_for_dir(self):
+        """A non-native (dir) connection keeps the Python pushdown path."""
        db = lancedb.connect_namespace("dir", {"root": self.temp_dir})
-        assert db._route_pushdown_to_rust is True
+        assert db._route_pushdown_to_rust is False

    def test_async_route_pushdown_to_rust_for_native_rest(self):
        """The async connection must not silently bypass the read-freshness fix:
@@ -925,11 +834,10 @@ class TestPushdownOperations:
        )
        assert db._route_pushdown_to_rust is True

-    def test_async_route_pushdown_to_rust_for_native_dir(self):
-        """The async dir connection is natively built and defers QueryTable
-        pushdown to Rust."""
+    def test_async_route_pushdown_to_rust_false_for_dir(self):
+        """The async non-native (dir) connection keeps the Python pushdown path."""
        db = lancedb.connect_namespace_async("dir", {"root": self.temp_dir})
-        assert db._route_pushdown_to_rust is True
+        assert db._route_pushdown_to_rust is False

    def test_lance_table_to_arrow_uses_query_pushdown(self):
        namespace_client = _NamespaceClient()
@@ -935,41 +935,14 @@ def test_transform_fn(mem_db):
    try:
        import torch

-        # "torch" returns a list of per-row dicts. Default DataLoader collate
-        # stacks the per-row dicts back into a dict of batched tensors.
-        torch_perm = permutation.with_format("torch")
-        torch_batch = list(torch_perm.iter(10, skip_last_batch=False))[0]
-        assert isinstance(torch_batch, list)
-        assert len(torch_batch) == 10
-        assert isinstance(torch_batch[0], dict)
-        assert set(torch_batch[0].keys()) == {"id", "value"}
-        assert isinstance(torch_batch[0]["id"], torch.Tensor)
-        assert torch_batch[0]["id"].dtype == torch.int64
-
-        rows = torch_perm.__getitems__([0, 1, 2])
-        assert isinstance(rows, list)
-        assert len(rows) == 3
-        assert isinstance(rows[0], dict)
-        assert set(rows[0].keys()) == {"id", "value"}
-        assert isinstance(rows[0]["id"], torch.Tensor)
-
-        # "torch_row" returns a list of tensors, one per row.
-        torch_rows = list(
-            permutation.with_format("torch_row").iter(10, skip_last_batch=False)
+        torch_result = list(
+            permutation.with_format("torch").iter(10, skip_last_batch=False)
        )[0]
-        assert isinstance(torch_rows, list)
-        assert len(torch_rows) == 10
-        assert isinstance(torch_rows[0], torch.Tensor)
-        assert torch_rows[0].shape == (2,)
-        assert torch_rows[0].dtype == torch.int64
-
-        # "torch_col" stacks columns into a single 2D tensor.
-        torch_col = list(
-            permutation.with_format("torch_col").iter(10, skip_last_batch=False)
-        )[0]
-        assert isinstance(torch_col, torch.Tensor)
-        assert torch_col.shape == (2, 10)
-        assert torch_col.dtype == torch.int64
+        assert isinstance(torch_result, list)
+        assert len(torch_result) == 10
+        assert isinstance(torch_result[0], torch.Tensor)
+        assert torch_result[0].shape == (2,)
+        assert torch_result[0].dtype == torch.int64
    except ImportError:
        # Skip check if torch is not installed
        pass
@@ -502,61 +502,6 @@ def test_with_row_id(table: lancedb.table.Table):
    assert rs["_rowid"].to_pylist() == [0, 1]


-def test_where_repeated_combines_with_and(table: lancedb.table.Table):
-    # Calling where() more than once should AND the filters together instead of
-    # silently replacing the previous one (regression test for #2649).
-    builder = table.search().where("id >= 1").where("id < 2")
-    assert builder._where == "(id >= 1) AND (id < 2)"
-
-    ids = [row["id"] for row in builder.limit(10).to_list()]
-    assert ids == [1]
-
-
-def test_where_repeated_combines_expr(table: lancedb.table.Table):
-    from lancedb.expr import col, lit
-
-    builder = table.search().where(col("id") >= lit(1)).where(col("id") < lit(2))
-    ids = [row["id"] for row in builder.limit(10).to_list()]
-    assert ids == [1]
-
-
-def test_where_mixed_filter_kinds_combines(table: lancedb.table.Table):
-    # Mixing a SQL string filter with an expression filter lowers the
-    # expression to SQL and combines them as SQL strings.
-    from lancedb.expr import col, lit
-
-    builder = table.search().where("id >= 1").where(col("id") < lit(2))
-    ids = [row["id"] for row in builder.limit(10).to_list()]
-    assert ids == [1]
-
-
-@pytest.mark.asyncio
-async def test_where_repeated_combines_with_and_async(table_async: AsyncTable):
-    ids = [
-        row["id"]
-        for row in (
-            await table_async.query().where("id >= 1").where("id < 2").to_list()
-        )
-    ]
-    assert ids == [1]
-
-
-@pytest.mark.asyncio
-async def test_where_mixed_filter_kinds_combines_async(table_async: AsyncTable):
-    from lancedb.expr import col, lit
-
-    ids = [
-        row["id"]
-        for row in (
-            await table_async.query()
-            .where("id >= 1")
-            .where(col("id") < lit(2))
-            .to_list()
-        )
-    ]
-    assert ids == [1]
-
-
 def test_distance_range(table: lancedb.table.Table):
    q = [0, 0]
    rs = table.search(q).to_arrow()
@@ -412,12 +412,10 @@ def test_remote_permutation_is_picklable():
            content_len = int(request.headers.get("Content-Length"))
            body = json.loads(request.rfile.read(content_len))
            if "filter" in body:
-                match = re.search(
-                    r"_rowoffset\s+in\s+\((.*?)\)", body["filter"], re.IGNORECASE
-                )
-                offsets = [int(o.strip()) for o in match.group(1).split(",")]
+                match = re.search(r"_rowoffset in \((.*?)\)", body["filter"])
+                offsets = [int(offset.strip()) for offset in match.group(1).split(",")]
            else:
-                offsets = list(range(len(rows)))
+                offsets = rows
            table = pa.table({"a": [rows[offset] for offset in offsets]})

            request.send_response(200)
@@ -350,38 +350,6 @@ def test_mrr_reranker_empty_input():
        reranker.rerank_multivector([])


-def test_mrr_multivector_rewards_consensus():
-    # Reciprocal ranks must be averaged across *all* ranking systems, treating a
-    # missing system as 0. A document ranked first by every system must outrank a
-    # document ranked first by only one of them.
-    reranker = MRRReranker()
-
-    def ranking(row_ids):
-        return pa.table({"_rowid": pa.array(row_ids, type=pa.int64())})
-
-    # Doc 1 is rank 1 in only the first system; doc 2 is rank 1 in two systems
-    # and rank 2 in the third (strong cross-system consensus).
-    rs1 = ranking([1, 2, 3])
-    rs2 = ranking([2, 3, 4])
-    rs3 = ranking([2, 5, 6])
-
-    result = reranker.rerank_multivector([rs1, rs2, rs3])
-    scores = {
-        row_id: score
-        for row_id, score in zip(
-            result["_rowid"].to_pylist(),
-            result["_relevance_score"].to_pylist(),
-        )
-    }
-
-    # sum of reciprocal ranks / number of systems
-    assert scores[1] == pytest.approx(1.0 / 3)
-    assert scores[2] == pytest.approx((0.5 + 1.0 + 1.0) / 3)
-    assert scores[2] > scores[1]
-    # The consensus document ranks first overall.
-    assert result["_rowid"].to_pylist()[0] == 2
-
-
 def test_rrf_reranker_distance():
    data = pa.table(
        {
@@ -7,8 +7,7 @@ Tests for S3 bucket names containing dots.
 Related issue: https://github.com/lancedb/lancedb/issues/1898

 These tests validate the early error checking for S3 bucket names with dots.
-When validation succeeds, eager namespace initialization may still fail later
-because these tests intentionally do not provide real S3 credentials.
+No actual S3 connection is made - validation happens before connection.
 """

 import pytest
@@ -21,13 +20,6 @@ BUCKET_WITH_DOTS_AND_AWS_REGION = ("s3://my.bucket.name", {"aws_region": "us-eas
 BUCKET_WITHOUT_DOTS = "s3://my-bucket/path"


-def assert_not_rejected_for_bucket_dots(connect):
-    try:
-        connect()
-    except ValueError as err:
-        assert "contains dots" not in str(err)
-
-
 class TestS3BucketWithDotsSync:
    """Tests for connect()."""

@@ -35,22 +27,19 @@ class TestS3BucketWithDotsSync:
        with pytest.raises(ValueError, match="contains dots"):
            lancedb.connect(BUCKET_WITH_DOTS)

-    def test_bucket_with_dots_and_region_is_not_rejected(self):
+    def test_bucket_with_dots_and_region_passes(self):
        uri, opts = BUCKET_WITH_DOTS_AND_REGION
-        assert_not_rejected_for_bucket_dots(
-            lambda: lancedb.connect(uri, storage_options=opts)
-        )
+        db = lancedb.connect(uri, storage_options=opts)
+        assert db is not None

-    def test_bucket_with_dots_and_aws_region_is_not_rejected(self):
+    def test_bucket_with_dots_and_aws_region_passes(self):
        uri, opts = BUCKET_WITH_DOTS_AND_AWS_REGION
-        assert_not_rejected_for_bucket_dots(
-            lambda: lancedb.connect(uri, storage_options=opts)
-        )
+        db = lancedb.connect(uri, storage_options=opts)
+        assert db is not None

-    def test_bucket_without_dots_is_not_rejected(self):
-        assert_not_rejected_for_bucket_dots(
-            lambda: lancedb.connect(BUCKET_WITHOUT_DOTS)
-        )
+    def test_bucket_without_dots_passes(self):
+        db = lancedb.connect(BUCKET_WITHOUT_DOTS)
+        assert db is not None


 class TestS3BucketWithDotsAsync:
@@ -62,24 +51,18 @@ class TestS3BucketWithDotsAsync:
            await lancedb.connect_async(BUCKET_WITH_DOTS)

    @pytest.mark.asyncio
-    async def test_bucket_with_dots_and_region_is_not_rejected(self):
+    async def test_bucket_with_dots_and_region_passes(self):
        uri, opts = BUCKET_WITH_DOTS_AND_REGION
-        try:
-            await lancedb.connect_async(uri, storage_options=opts)
-        except ValueError as err:
-            assert "contains dots" not in str(err)
+        db = await lancedb.connect_async(uri, storage_options=opts)
+        assert db is not None

    @pytest.mark.asyncio
-    async def test_bucket_with_dots_and_aws_region_is_not_rejected(self):
+    async def test_bucket_with_dots_and_aws_region_passes(self):
        uri, opts = BUCKET_WITH_DOTS_AND_AWS_REGION
-        try:
-            await lancedb.connect_async(uri, storage_options=opts)
-        except ValueError as err:
-            assert "contains dots" not in str(err)
+        db = await lancedb.connect_async(uri, storage_options=opts)
+        assert db is not None

    @pytest.mark.asyncio
-    async def test_bucket_without_dots_is_not_rejected(self):
-        try:
-            await lancedb.connect_async(BUCKET_WITHOUT_DOTS)
-        except ValueError as err:
-            assert "contains dots" not in str(err)
+    async def test_bucket_without_dots_passes(self):
+        db = await lancedb.connect_async(BUCKET_WITHOUT_DOTS)
+        assert db is not None
@@ -1137,16 +1137,6 @@ def test_namespace_open_table_with_branch_version(tmp_path):
    assert db.open_table("t", namespace_path=["ns1"], branch="exp").count_rows() == 3


-def test_namespace_root_table_to_lance_uses_namespace_client(tmp_path):
-    pytest.importorskip("lance")  # "dir" impl is lance.namespace.DirectoryNamespace
-    db = lancedb.connect_namespace("dir", {"root": str(tmp_path)})
-    table = db.create_table("t", [{"i": 0}])
-
-    assert table._namespace_client is None
-    assert table.to_lance().count_rows() == 1
-    assert table._namespace_client is not None
-
-
@pytest.mark.asyncio
 async def test_async_namespace_open_table_with_branch_version(tmp_path):
    pytest.importorskip("lance")  # "dir" impl is lance.namespace.DirectoryNamespace
@@ -148,47 +148,15 @@ def test_permutation_dataloader(mem_db):
    for batch in dataloader:
        assert batch["a"].size(0) == 10

-    # "torch" produces a list of per-row dicts per batch. The default
-    # DataLoader collate stacks the per-row dicts back into a batched dict.
-    torch_perm = permutation.with_format("torch")
-    batch = next(torch_perm.iter(10, skip_last_batch=False))
-    assert isinstance(batch, list)
-    assert len(batch) == 10
-    assert isinstance(batch[0], dict)
-    assert isinstance(batch[0]["a"], torch.Tensor)
-    rows = torch_perm.__getitems__([0, 1, 2])
-    assert isinstance(rows, list)
-    assert len(rows) == 3
-    assert isinstance(rows[0], dict)
-    assert isinstance(rows[0]["a"], torch.Tensor)
-    dataloader = torch.utils.data.DataLoader(torch_perm, batch_size=10, shuffle=True)
-    for batch in dataloader:
-        assert isinstance(batch, dict)
-        assert batch["a"].shape == (10,)
-    # Spawn-based workers exercise the pickle round-trip path: the new
-    # transform-as-list shape must survive pickling so workers produce the
-    # same per-row dicts the parent does.
-    spawn_loader = torch.utils.data.DataLoader(
-        torch_perm,
-        batch_size=10,
-        num_workers=2,
-        multiprocessing_context="spawn",
-    )
-    for batch in spawn_loader:
-        assert isinstance(batch, dict)
-        assert batch["a"].shape == (10,)
-
-    # "torch_row" returns a list of row tensors. Works with the default
-    # DataLoader collate (stacks rows into 2D).
-    row_perm = permutation.with_format("torch_row")
-    dataloader = torch.utils.data.DataLoader(row_perm, batch_size=10, shuffle=True)
+    permutation = permutation.with_format("torch")
+    dataloader = torch.utils.data.DataLoader(permutation, batch_size=10, shuffle=True)
    for batch in dataloader:
        assert batch.size(0) == 10
        assert batch.size(1) == 1

-    col_perm = permutation.with_format("torch_col")
+    permutation = permutation.with_format("torch_col")
    dataloader = torch.utils.data.DataLoader(
-        col_perm, collate_fn=lambda x: x, batch_size=10, shuffle=True
+        permutation, collate_fn=lambda x: x, batch_size=10, shuffle=True
    )
    for batch in dataloader:
        assert batch.size(0) == 1
@@ -25,42 +25,10 @@ import pandas as pd
 import polars as pl
 import pytest
 import lancedb
-from lancedb.util import flatten_columns, get_uri_scheme, join_uri, value_to_sql
+from lancedb.util import get_uri_scheme, join_uri, value_to_sql
 from utils import exception_output


-def _struct_table() -> pa.Table:
-    return pa.table(
-        {
-            "id": [1, 2],
-            "nested": pa.array([{"a": 1, "b": 2}, {"a": 3, "b": 4}]),
-        }
-    )
-
-
-def test_flatten_columns():
-    tbl = _struct_table()
-
-    # None / False mean "do not flatten": the struct column is preserved.
-    # `False` is a regression guard: because bool is a subclass of int it used
-    # to fall into the integer branch and raise ValueError (see issue).
-    for no_flatten in (None, False):
-        result = flatten_columns(tbl, no_flatten)
-        assert result.column_names == ["id", "nested"]
-
-    # True flattens all nested levels.
-    flattened = flatten_columns(tbl, True)
-    assert flattened.column_names == ["id", "nested.a", "nested.b"]
-
-    # A positive integer flattens up to that depth.
-    flattened = flatten_columns(tbl, 1)
-    assert flattened.column_names == ["id", "nested.a", "nested.b"]
-
-    # Non-positive integers are still rejected.
-    with pytest.raises(ValueError):
-        flatten_columns(tbl, 0)
-
-
 def test_normalize_uri():
    uris = [
        "relative/path",
@@ -18,7 +18,10 @@ use lancedb::{
    connection::Connection as LanceConnection,
    connection::NamespaceClientPushdownOperation,
    database::namespace::LanceNamespaceDatabase,
-    database::{CreateTableMode, Database, ReadConsistency},
+    database::{
+        CreateFunctionRequest, CreateMaterializedViewRequest, CreateTableMode, Database,
+        ReadConsistency, RefreshMaterializedViewRequest, TableLineageRequest,
+    },
 };
 use pyo3::{
    Bound, FromPyObject, Py, PyAny, PyRef, PyResult, Python,
@@ -27,6 +30,92 @@ use pyo3::{
    types::{PyDict, PyDictMethods},
 };

+/// A registered function, as returned by `list_functions`.
+#[pyclass(get_all)]
+#[derive(Clone)]
+pub struct FunctionInfo {
+    pub name: String,
+    pub language: String,
+    pub return_type: String,
+    pub description: String,
+}
+
+/// A registered materialized view definition.
+#[pyclass(get_all)]
+#[derive(Clone)]
+pub struct MaterializedViewInfo {
+    pub name: String,
+    pub source_table: String,
+    pub projection: Vec<String>,
+    pub udf_columns: Vec<String>,
+    pub filter: Option<String>,
+    pub auto_refresh: bool,
+}
+
+/// One inflight server-side job.
+#[pyclass(get_all)]
+#[derive(Clone)]
+pub struct JobInfo {
+    pub table: String,
+    pub job_id: String,
+    pub job_type: String,
+    pub state: String,
+    pub column: Option<String>,
+    pub age_seconds: Option<i64>,
+    pub command: Option<String>,
+    pub units_done: Option<i64>,
+    pub units_total: Option<i64>,
+    pub committed: bool,
+    pub rows_skipped: u64,
+    pub error: Option<String>,
+}
+
+/// One durable, completed/terminal server-side job record (SHOW JOB HISTORY).
+#[pyclass(get_all)]
+#[derive(Clone)]
+pub struct JobHistoryEntry {
+    pub table: String,
+    pub job_id: String,
+    pub job_type: String,
+    pub state: String,
+    pub column: Option<String>,
+    pub created_ms: i64,
+    pub updated_ms: i64,
+    pub completed_ms: Option<i64>,
+    pub rows_processed: Option<i64>,
+    pub rows_skipped: Option<i64>,
+    pub error: Option<String>,
+    pub events: Option<String>,
+}
+
+/// One per-row UDF error recorded by `error_policy=skip` (SHOW ERRORS).
+#[pyclass(get_all)]
+#[derive(Clone)]
+pub struct JobErrorEntry {
+    pub job_id: String,
+    pub table: String,
+    pub column: String,
+    pub error_type: String,
+    pub error_message: String,
+    pub fragment_id: Option<i64>,
+    pub source_row_id: Option<i64>,
+    pub table_version: Option<i64>,
+    pub age_seconds: Option<i64>,
+}
+
+/// The plan a REFRESH MATERIALIZED VIEW would execute (EXPLAIN REFRESH).
+#[pyclass(get_all)]
+#[derive(Clone)]
+pub struct MvRefreshPlan {
+    pub table_name: String,
+    pub has_work: bool,
+    pub source_version: u64,
+    pub last_refreshed_version: Option<u64>,
+    pub full_refresh: bool,
+    pub rebuild: bool,
+    pub units_total: u64,
+}
+
 #[pyclass]
 pub struct Connection {
    inner: Option<LanceConnection>,
@@ -310,6 +399,308 @@ impl Connection {
        })
    }

+    #[pyo3(signature = (name, language, return_type, body, options=None))]
+    pub fn create_function(
+        self_: PyRef<'_, Self>,
+        name: String,
+        language: String,
+        return_type: String,
+        body: String,
+        options: Option<HashMap<String, String>>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner
+                .create_function(CreateFunctionRequest {
+                    name,
+                    language,
+                    return_type,
+                    body,
+                    options: options.unwrap_or_default(),
+                })
+                .await
+                .infer_error()
+        })
+    }
+
+    pub fn list_functions(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            let functions = inner.list_functions().await.infer_error()?;
+            Ok(functions
+                .into_iter()
+                .map(|f| FunctionInfo {
+                    name: f.name,
+                    language: f.language,
+                    return_type: f.return_type,
+                    description: f.description,
+                })
+                .collect::<Vec<_>>())
+        })
+    }
+
+    pub fn drop_function(self_: PyRef<'_, Self>, name: String) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner.drop_function(&name).await.infer_error()
+        })
+    }
+
+    #[pyo3(signature = (name, query, auto_refresh=false, with_no_data=false, partition_by=None))]
+    pub fn create_materialized_view(
+        self_: PyRef<'_, Self>,
+        name: String,
+        query: String,
+        auto_refresh: bool,
+        with_no_data: bool,
+        partition_by: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner
+                .create_materialized_view(CreateMaterializedViewRequest {
+                    name,
+                    query,
+                    auto_refresh,
+                    with_no_data,
+                    partition_by,
+                })
+                .await
+                .infer_error()
+        })
+    }
+
+    #[pyo3(signature = (name, full=false, src_version=None, num_workers=None, max_workers=None))]
+    pub fn refresh_materialized_view(
+        self_: PyRef<'_, Self>,
+        name: String,
+        full: bool,
+        src_version: Option<u64>,
+        num_workers: Option<u32>,
+        max_workers: Option<u32>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner
+                .refresh_materialized_view(RefreshMaterializedViewRequest {
+                    name,
+                    full,
+                    src_version,
+                    num_workers,
+                    max_workers,
+                })
+                .await
+                .infer_error()
+        })
+    }
+
+    /// Derived-compute lineage of a table/view (or column), returned as the
+    /// server's lineage JSON string (the Python layer parses it).
+    pub fn table_lineage(
+        self_: PyRef<'_, Self>,
+        name: String,
+        column: Option<String>,
+        direction: Option<String>,
+        depth: Option<u32>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner
+                .table_lineage(TableLineageRequest {
+                    name,
+                    column,
+                    direction,
+                    depth,
+                })
+                .await
+                .infer_error()
+        })
+    }
+
+    #[pyo3(signature = (name, full=false, src_version=None))]
+    pub fn explain_refresh_materialized_view(
+        self_: PyRef<'_, Self>,
+        name: String,
+        full: bool,
+        src_version: Option<u64>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            let p = inner
+                .explain_refresh_materialized_view(&name, full, src_version)
+                .await
+                .infer_error()?;
+            Ok(MvRefreshPlan {
+                table_name: p.table_name,
+                has_work: p.has_work,
+                source_version: p.source_version,
+                last_refreshed_version: p.last_refreshed_version,
+                full_refresh: p.full_refresh,
+                rebuild: p.rebuild,
+                units_total: p.units_total,
+            })
+        })
+    }
+
+    pub fn alter_materialized_view(
+        self_: PyRef<'_, Self>,
+        name: String,
+        auto_refresh: bool,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner
+                .alter_materialized_view(&name, auto_refresh)
+                .await
+                .infer_error()
+        })
+    }
+
+    pub fn drop_materialized_view(
+        self_: PyRef<'_, Self>,
+        name: String,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner.drop_materialized_view(&name).await.infer_error()
+        })
+    }
+
+    pub fn list_materialized_views(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            let views = inner.list_materialized_views().await.infer_error()?;
+            Ok(views
+                .into_iter()
+                .map(|v| MaterializedViewInfo {
+                    name: v.name,
+                    source_table: v.source_table,
+                    projection: v.projection,
+                    udf_columns: v.udf_columns,
+                    filter: v.filter,
+                    auto_refresh: v.auto_refresh,
+                })
+                .collect::<Vec<_>>())
+        })
+    }
+
+    pub fn list_jobs(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            let jobs = inner.list_jobs().await.infer_error()?;
+            Ok(jobs
+                .into_iter()
+                .map(|j| JobInfo {
+                    table: j.table,
+                    job_id: j.job_id,
+                    job_type: j.job_type,
+                    state: j.state,
+                    column: j.column,
+                    age_seconds: j.age_seconds,
+                    command: j.command,
+                    units_done: j.units_done,
+                    units_total: j.units_total,
+                    committed: j.committed,
+                    rows_skipped: j.rows_skipped,
+                    error: j.error,
+                })
+                .collect::<Vec<_>>())
+        })
+    }
+
+    pub fn cancel_job(self_: PyRef<'_, Self>, job_id: String) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            inner.cancel_job(&job_id).await.infer_error()
+        })
+    }
+
+    #[pyo3(signature = (job_id, table=None))]
+    pub fn get_job(
+        self_: PyRef<'_, Self>,
+        job_id: String,
+        table: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            let job = inner
+                .get_job(&job_id, table.as_deref())
+                .await
+                .infer_error()?;
+            Ok(job.map(|j| JobInfo {
+                table: j.table,
+                job_id: j.job_id,
+                job_type: j.job_type,
+                state: j.state,
+                column: j.column,
+                age_seconds: j.age_seconds,
+                command: j.command,
+                units_done: j.units_done,
+                units_total: j.units_total,
+                committed: j.committed,
+                rows_skipped: j.rows_skipped,
+                error: j.error,
+            }))
+        })
+    }
+
+    #[pyo3(signature = (job_id=None))]
+    pub fn job_history(
+        self_: PyRef<'_, Self>,
+        job_id: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            let rows = inner.job_history(job_id.as_deref()).await.infer_error()?;
+            Ok(rows
+                .into_iter()
+                .map(|r| JobHistoryEntry {
+                    table: r.table,
+                    job_id: r.job_id,
+                    job_type: r.job_type,
+                    state: r.state,
+                    column: r.column,
+                    created_ms: r.created_ms,
+                    updated_ms: r.updated_ms,
+                    completed_ms: r.completed_ms,
+                    rows_processed: r.rows_processed,
+                    rows_skipped: r.rows_skipped,
+                    error: r.error,
+                    events: r.events,
+                })
+                .collect::<Vec<_>>())
+        })
+    }
+
+    #[pyo3(signature = (job_id=None, table=None))]
+    pub fn errors(
+        self_: PyRef<'_, Self>,
+        job_id: Option<String>,
+        table: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.get_inner()?.clone();
+        future_into_py(self_.py(), async move {
+            let rows = inner
+                .errors(job_id.as_deref(), table.as_deref())
+                .await
+                .infer_error()?;
+            Ok(rows
+                .into_iter()
+                .map(|e| JobErrorEntry {
+                    job_id: e.job_id,
+                    table: e.table,
+                    column: e.column,
+                    error_type: e.error_type,
+                    error_message: e.error_message,
+                    fragment_id: e.fragment_id,
+                    source_row_id: e.source_row_id,
+                    table_version: e.table_version,
+                    age_seconds: e.age_seconds,
+                })
+                .collect::<Vec<_>>())
+        })
+    }
+
    #[pyo3(signature = (cur_name, new_name, cur_namespace_path=None, new_namespace_path=None))]
    pub fn rename_table(
        self_: PyRef<'_, Self>,
@@ -655,46 +1046,6 @@ pub fn connect_namespace_client(
    )))
 }

-#[pyfunction]
-#[pyo3(signature = (
-    namespace_client_impl,
-    namespace_client_properties,
-    read_consistency_interval=None,
-    storage_options=None,
-    session=None,
-    namespace_client_pushdown_operations=None,
-))]
-#[allow(clippy::too_many_arguments)]
-pub fn connect_namespace(
-    namespace_client_impl: String,
-    namespace_client_properties: HashMap<String, String>,
-    read_consistency_interval: Option<f64>,
-    storage_options: Option<HashMap<String, String>>,
-    session: Option<crate::session::Session>,
-    namespace_client_pushdown_operations: Option<Vec<String>>,
-) -> PyResult<Connection> {
-    let read_consistency_interval = read_consistency_interval.map(Duration::from_secs_f64);
-    let namespace_client_pushdown_operations =
-        parse_namespace_client_pushdown_operations(namespace_client_pushdown_operations)?;
-
-    let mut builder =
-        lancedb::connect_namespace(&namespace_client_impl, namespace_client_properties)
-            .pushdown_operations(namespace_client_pushdown_operations);
-    if let Some(storage_options) = storage_options {
-        builder = builder.storage_options(storage_options);
-    }
-    if let Some(read_consistency_interval) = read_consistency_interval {
-        builder = builder.read_consistency_interval(read_consistency_interval);
-    }
-    if let Some(session) = session {
-        builder = builder.session(session.inner.clone());
-    }
-
-    Ok(Connection::new(
-        crate::runtime::block_on(builder.execute()).infer_error()?,
-    ))
-}
-
 /// Whether to build the namespace natively (from impl + properties) instead of
 /// wrapping a pre-built client. Native construction is required for the
 /// read-freshness provider to be installed
@@ -7,14 +7,12 @@
 //! build type-safe filter / projection expressions that map directly to
 //! DataFusion [`Expr`] nodes, bypassing SQL string parsing.

-use std::ops::{Add, Div, Mul, Not, Sub};
-
 use arrow::{datatypes::DataType, pyarrow::PyArrowType};
 use datafusion_common::ScalarValue;
 use lancedb::expr::{
    DfExpr, col as ldb_col, contains, expr_cast, is_in, lit as df_lit, lower, upper,
 };
-use pyo3::types::{PyBytes, PyDate, PyDateTime};
+use pyo3::types::PyBytes;
 use pyo3::{Bound, PyAny, PyResult, exceptions::PyValueError, prelude::*, pyfunction};

 /// A type-safe DataFusion expression.
@@ -65,30 +63,30 @@ impl PyExpr {
        Self(self.0.clone().or(other.0.clone()))
    }

-    /// Logical NOT.
    fn not_(&self) -> Self {
+        use std::ops::Not;
        Self(self.0.clone().not())
    }

    // ── arithmetic ───────────────────────────────────────────────────────────

-    /// Add expressions.
    fn add(&self, other: &Self) -> Self {
+        use std::ops::Add;
        Self(self.0.clone().add(other.0.clone()))
    }

-    /// Subtract expressions.
    fn sub(&self, other: &Self) -> Self {
+        use std::ops::Sub;
        Self(self.0.clone().sub(other.0.clone()))
    }

-    /// Multiply expressions.
    fn mul(&self, other: &Self) -> Self {
+        use std::ops::Mul;
        Self(self.0.clone().mul(other.0.clone()))
    }

-    /// Divide expressions.
    fn div(&self, other: &Self) -> Self {
+        use std::ops::Div;
        Self(self.0.clone().div(other.0.clone()))
    }

@@ -155,8 +153,7 @@ pub fn expr_col(name: &str) -> PyExpr {

 /// Create a literal value expression.
 ///
-/// Supported Python types: `bool`, `int`, `float`, `str`, `bytes`, `date`,
-/// `datetime`, `Decimal`.
+/// Supported Python types: `bool`, `int`, `float`, `str`, `bytes`.
 #[pyfunction]
 pub fn expr_lit(value: Bound<'_, PyAny>) -> PyResult<PyExpr> {
    // bool must be checked before int because bool is a subclass of int in Python
@@ -166,19 +163,6 @@ pub fn expr_lit(value: Bound<'_, PyAny>) -> PyResult<PyExpr> {
    if let Ok(i) = value.extract::<i64>() {
        return Ok(PyExpr(df_lit(i)));
    }
-    // Decimal must be checked before f64: Python's Decimal implements __float__,
-    // so value.extract::<f64>() would succeed and silently truncate the value to
-    // f64, losing precision. Build a Decimal128 scalar to preserve it instead.
-    if value.get_type().name()? == "Decimal" {
-        let s = value.call_method0("__str__")?.extract::<String>()?;
-        // Parse the decimal string into an i128 value, precision, and scale.
-        let (val, precision, scale) = parse_decimal(&s)?;
-        return Ok(PyExpr(df_lit(ScalarValue::Decimal128(
-            Some(val),
-            precision,
-            scale,
-        ))));
-    }
    if let Ok(f) = value.extract::<f64>() {
        return Ok(PyExpr(df_lit(f)));
    }
@@ -189,48 +173,12 @@ pub fn expr_lit(value: Bound<'_, PyAny>) -> PyResult<PyExpr> {
        let bytes = value.extract::<Vec<u8>>()?;
        return Ok(PyExpr(df_lit(ScalarValue::Binary(Some(bytes)))));
    }
-
-    // datetime.datetime is a subclass of datetime.date, so it must be checked first.
-    if let Ok(dt) = value.cast::<PyDateTime>() {
-        let ts: f64 = dt.call_method0("timestamp")?.extract()?;
-        let micros = (ts * 1_000_000.0).round() as i64;
-        return Ok(PyExpr(df_lit(ScalarValue::TimestampMicrosecond(
-            Some(micros),
-            None,
-        ))));
-    }
-    if let Ok(d) = value.cast::<PyDate>() {
-        let ordinal: i32 = d.call_method0("toordinal")?.extract()?;
-        let days = ordinal - 719163; // Unix epoch is 1970-01-01
-        return Ok(PyExpr(df_lit(ScalarValue::Date32(Some(days)))));
-    }
-
    Err(PyValueError::new_err(format!(
-        "unsupported literal type: {}. Supported: bool, int, float, str, bytes, date, datetime, Decimal",
+        "unsupported literal type: {}. Supported: bool, int, float, str, bytes",
        value.get_type().name()?
    )))
 }

-fn parse_decimal(s: &str) -> PyResult<(i128, u8, i8)> {
-    let s = s.trim();
-    let dot_pos = s.find('.');
-    let scale = if let Some(pos) = dot_pos {
-        (s.len() - pos - 1) as i8
-    } else {
-        0
-    };
-
-    let digits = s.replace('.', "");
-    let val = digits
-        .parse::<i128>()
-        .map_err(|e| PyValueError::new_err(format!("failed to parse decimal digits: {}", e)))?;
-
-    // Precision is total number of digits
-    let precision = digits.trim_start_matches('-').len() as u8;
-
-    Ok((val, precision, scale))
-}
-
 /// Call an arbitrary registered SQL function by name.
 ///
 /// See `lancedb::expr::func` for the list of supported function names.
@@ -2,7 +2,7 @@
 // SPDX-FileCopyrightText: Copyright The LanceDB Authors

 use arrow::RecordBatchStream;
-use connection::{Connection, connect, connect_namespace, connect_namespace_client};
+use connection::{Connection, connect, connect_namespace_client};
 use env_logger::Env;
 use expr::{PyExpr, expr_col, expr_func, expr_lit};
 use index::IndexConfig;
@@ -27,7 +27,6 @@ pub mod header;
 pub mod index;
 pub mod namespace;
 pub mod oauth;
-pub mod otel;
 pub mod permutation;
 pub mod query;
 pub mod runtime;
@@ -42,6 +41,11 @@ pub fn _lancedb(_py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
        .write_style("LANCEDB_LOG_STYLE");
    env_logger::init_from_env(env);
    m.add_class::<Connection>()?;
+    m.add_class::<connection::FunctionInfo>()?;
+    m.add_class::<connection::MaterializedViewInfo>()?;
+    m.add_class::<connection::JobInfo>()?;
+    m.add_class::<connection::JobHistoryEntry>()?;
+    m.add_class::<connection::JobErrorEntry>()?;
    m.add_class::<Session>()?;
    m.add_class::<Table>()?;
    m.add_class::<IndexConfig>()?;
@@ -62,17 +66,7 @@ pub fn _lancedb(_py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_class::<PyAsyncPermutationBuilder>()?;
    m.add_class::<PyPermutationReader>()?;
    m.add_class::<PyExpr>()?;
-    // OpenTelemetry metrics bridge
-    m.add_class::<otel::PyMetricPoint>()?;
-    m.add_class::<otel::PyMetricDescription>()?;
-    m.add_function(wrap_pyfunction!(
-        otel::register_lancedb_metrics_recorder,
-        m
-    )?)?;
-    m.add_function(wrap_pyfunction!(otel::lancedb_metrics_catalog, m)?)?;
-    m.add_function(wrap_pyfunction!(otel::snapshot_lancedb_metrics, m)?)?;
    m.add_function(wrap_pyfunction!(connect, m)?)?;
-    m.add_function(wrap_pyfunction!(connect_namespace, m)?)?;
    m.add_function(wrap_pyfunction!(connect_namespace_client, m)?)?;
    m.add_function(wrap_pyfunction!(permutation::async_permutation_builder, m)?)?;
    m.add_function(wrap_pyfunction!(util::validate_table_name, m)?)?;
@@ -1,96 +0,0 @@
-// SPDX-License-Identifier: Apache-2.0
-// SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-//! Python-facing wrappers over [`lancedb::metrics_otel`].
-//!
-//! The aggregation, catalog, and histogram bucketing all live in the LanceDB
-//! core crate; this module only converts the core snapshot types into PyO3
-//! classes and exposes the three entry points to Python, where
-//! `lancedb/otel.py` bridges them into the user's OpenTelemetry `MeterProvider`.
-
-use std::collections::HashMap;
-
-use lancedb::metrics_otel::{MetricPoint, MetricValue};
-use pyo3::prelude::*;
-
-/// One metric data point exposed to Python. For counters and gauges only
-/// `value` is set; for histograms `buckets` (cumulative `le` counts), `count`,
-/// and `sum` are set.
-#[pyclass(name = "MetricPoint", get_all)]
-pub struct PyMetricPoint {
-    name: String,
-    kind: String,
-    attributes: HashMap<String, String>,
-    value: Option<f64>,
-    buckets: Option<Vec<(String, u64)>>,
-    count: Option<u64>,
-    sum: Option<f64>,
-}
-
-impl From<MetricPoint> for PyMetricPoint {
-    fn from(point: MetricPoint) -> Self {
-        let kind = point.kind.as_str().to_string();
-        let (value, buckets, count, sum) = match point.value {
-            MetricValue::Scalar(v) => (Some(v), None, None, None),
-            MetricValue::Histogram {
-                buckets,
-                count,
-                sum,
-            } => (None, Some(buckets), Some(count), Some(sum)),
-        };
-        Self {
-            name: point.name,
-            kind,
-            attributes: point.attributes,
-            value,
-            buckets,
-            count,
-            sum,
-        }
-    }
-}
-
-/// A described metric, used by the Python layer to create instruments up front.
-#[pyclass(name = "MetricDescription", get_all)]
-pub struct PyMetricDescription {
-    name: String,
-    kind: String,
-    unit: Option<String>,
-    description: String,
-}
-
-/// Install the LanceDB metrics recorder as the process-global `metrics` recorder.
-///
-/// Returns `True` if the recorder is installed (now or previously). Returns
-/// `False` if a *different* recorder is already installed — `metrics` allows
-/// only one global recorder per process, so LanceDB cannot coexist with another.
-#[pyfunction]
-pub fn register_lancedb_metrics_recorder() -> bool {
-    lancedb::metrics_otel::register_metrics_recorder()
-}
-
-/// The catalog of described LanceDB metrics. Empty until the recorder is installed.
-#[pyfunction]
-pub fn lancedb_metrics_catalog() -> Vec<PyMetricDescription> {
-    lancedb::metrics_otel::metrics_catalog()
-        .into_iter()
-        .map(|desc| PyMetricDescription {
-            name: desc.name,
-            kind: desc.kind.as_str().to_string(),
-            unit: desc.unit,
-            description: desc.description,
-        })
-        .collect()
-}
-
-/// A point-in-time snapshot of every recorded metric. Empty until the recorder
-/// is installed.
-///
-/// The read is lock-free but not O(1): it walks every registered series and
-/// allocates owned copies of their names and labels. The GIL is released across
-/// that work so a periodic collection doesn't stall other Python threads.
-#[pyfunction]
-pub fn snapshot_lancedb_metrics(py: Python<'_>) -> Vec<PyMetricPoint> {
-    let points = py.detach(lancedb::metrics_otel::snapshot_metrics);
-    points.into_iter().map(PyMetricPoint::from).collect()
-}
@@ -79,14 +79,13 @@ impl PyAsyncPermutationBuilder {

 #[pymethods]
 impl PyAsyncPermutationBuilder {
-    #[pyo3(signature = (*, ratios=None, counts=None, fixed=None, seed=None, clump_size=None, split_names=None))]
+    #[pyo3(signature = (*, ratios=None, counts=None, fixed=None, seed=None, split_names=None))]
    pub fn split_random(
        slf: PyRefMut<'_, Self>,
        ratios: Option<Vec<f64>>,
        counts: Option<Vec<u64>>,
        fixed: Option<u64>,
        seed: Option<u64>,
-        clump_size: Option<u64>,
        split_names: Option<Vec<String>>,
    ) -> PyResult<Self> {
        // Check that exactly one split type is provided
@@ -112,14 +111,7 @@ impl PyAsyncPermutationBuilder {
        };

        slf.modify(|builder| {
-            builder.with_split_strategy(
-                SplitStrategy::Random {
-                    seed,
-                    sizes,
-                    clump_size,
-                },
-                split_names,
-            )
+            builder.with_split_strategy(SplitStrategy::Random { seed, sizes }, split_names)
        })
    }

@@ -17,8 +17,8 @@ use arrow::{
    pyarrow::{FromPyArrow, PyArrowType, ToPyArrow},
 };
 use lancedb::table::{
-    AddDataMode, ColumnAlteration, Duration, FieldMetadataUpdate, NewColumnTransform,
-    OptimizeAction, OptimizeOptions, Ref, Table as LanceDbTable,
+    AddDataMode, ColumnAlteration, Duration, FieldMetadataUpdate, LoadColumnsRequest,
+    NewColumnTransform, OptimizeAction, OptimizeOptions, Ref, Table as LanceDbTable,
 };
 use pyo3::{
    Bound, FromPyObject, Py, PyAny, PyRef, PyResult, Python,
@@ -322,12 +322,6 @@ impl From<LsmWriteSpec> for lancedb::table::LsmWriteSpec {
    }
 }

-impl From<lancedb::table::LsmWriteSpec> for LsmWriteSpec {
-    fn from(inner: lancedb::table::LsmWriteSpec) -> Self {
-        Self { inner }
-    }
-}
-
 #[pyclass(get_all, from_py_object)]
 #[derive(Clone, Debug)]
 pub struct AddColumnsResult {
@@ -1035,14 +1029,6 @@ impl Table {
        })
    }

-    pub fn get_lsm_write_spec(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
-        let inner = self_.inner_ref()?.clone();
-        future_into_py(self_.py(), async move {
-            let spec = inner.get_lsm_write_spec().await.infer_error()?;
-            Ok(spec.map(LsmWriteSpec::from))
-        })
-    }
-
    pub fn close_lsm_writers(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
        let inner = self_.inner_ref()?.clone();
        future_into_py(self_.py(), async move {
@@ -1074,6 +1060,83 @@ impl Table {
        })
    }

+    pub fn add_computed_columns(
+        self_: PyRef<'_, Self>,
+        columns: Vec<(String, String)>,
+        expression: String,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        future_into_py(self_.py(), async move {
+            inner
+                .add_computed_columns(&columns, &expression)
+                .await
+                .infer_error()
+        })
+    }
+
+    #[pyo3(signature = (columns, where_clause=None, num_workers=None, max_workers=None, batch_size=None, priority=None))]
+    pub fn refresh_column(
+        self_: PyRef<'_, Self>,
+        columns: Vec<String>,
+        where_clause: Option<String>,
+        num_workers: Option<u32>,
+        max_workers: Option<u32>,
+        batch_size: Option<u32>,
+        priority: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        future_into_py(self_.py(), async move {
+            inner
+                .refresh_column(
+                    &columns,
+                    where_clause,
+                    num_workers,
+                    max_workers,
+                    batch_size,
+                    priority,
+                )
+                .await
+                .infer_error()
+        })
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    #[pyo3(signature = (source_uris, source_format, target_key, columns, source_key=None, source_storage_options=None, on_missing=None, num_workers=None, max_workers=None, batch_size=None, commit_granularity=None, priority=None))]
+    pub fn load_columns(
+        self_: PyRef<'_, Self>,
+        source_uris: Vec<String>,
+        source_format: String,
+        target_key: String,
+        columns: Vec<(String, Option<String>)>,
+        source_key: Option<String>,
+        source_storage_options: Option<std::collections::HashMap<String, String>>,
+        on_missing: Option<String>,
+        num_workers: Option<u32>,
+        max_workers: Option<u32>,
+        batch_size: Option<u32>,
+        commit_granularity: Option<u32>,
+        priority: Option<String>,
+    ) -> PyResult<Bound<'_, PyAny>> {
+        let inner = self_.inner_ref()?.clone();
+        let request = LoadColumnsRequest {
+            source_uris,
+            source_format,
+            source_storage_options,
+            target_key,
+            source_key,
+            columns,
+            on_missing,
+            num_workers,
+            max_workers,
+            batch_size,
+            commit_granularity,
+            priority,
+        };
+        future_into_py(self_.py(), async move {
+            inner.load_columns(request).await.infer_error()
+        })
+    }
+
    pub fn add_columns(
        self_: PyRef<'_, Self>,
        definitions: Vec<(String, String)>,
@@ -3,14 +3,10 @@

 """Tests for the type-safe expression builder API."""

-from datetime import date, datetime, timedelta, timezone
-from decimal import Decimal
-
-import pyarrow as pa
 import pytest
-
+import pyarrow as pa
 import lancedb
-from lancedb.expr import Expr, col, func, lit
+from lancedb.expr import Expr, col, lit, func


 # ── unit tests for Expr construction ─────────────────────────────────────────
@@ -58,28 +54,6 @@ class TestExprConstruction:
        with pytest.raises(Exception):
            func("not_a_real_function", col("x"))

-    def test_lit_date(self):
-        e = lit(date(2024, 1, 1))
-        assert isinstance(e, Expr)
-
-    def test_lit_datetime(self):
-        # Naive datetime
-        e = lit(datetime(2024, 1, 1, 10, 0))
-        assert isinstance(e, Expr)
-
-    def test_lit_datetime_tz(self):
-        # Timezone-aware datetime
-        tz = timezone(timedelta(hours=5))
-        dt = datetime(2024, 1, 1, 10, 0, tzinfo=tz)
-        e = lit(dt)
-        assert isinstance(e, Expr)
-
-    def test_lit_decimal_precision(self):
-        # High precision Decimal that would be rounded if converted to float
-        d = Decimal("1.234567890123456789")
-        e = lit(d)
-        assert isinstance(e, Expr)
-

 class TestExprOperators:
    def test_eq_operator(self):
@@ -168,20 +142,6 @@ class TestExprOperators:
        assert isinstance(e, Expr)
        assert e.to_sql() == "(name = 'alice')"

-    def test_reflexive_comparisons(self):
-        # 10 < col("age") swaps to col("age") > 10
-        assert (10 < col("age")).to_sql() == "(age > 10)"
-        assert (10 <= col("age")).to_sql() == "(age >= 10)"
-        assert (10 > col("age")).to_sql() == "(age < 10)"
-        assert (10 >= col("age")).to_sql() == "(age <= 10)"
-        assert (10 == col("age")).to_sql() == "(age = 10)"
-        assert (10 != col("age")).to_sql() == "(age <> 10)"
-
-    def test_reflexive_logical(self):
-        # True & Expr calls Expr.__rand__(True)
-        assert (True & (col("age") > 18)).to_sql() == "(true AND (age > 18))"
-        assert (False | (col("age") > 18)).to_sql() == "(false OR (age > 18))"
-

 class TestExprBytesLiteral:
    def test_bytes_to_sql(self):
@@ -322,40 +282,6 @@ class TestExprRepr:
            {e: 1}


-class TestExprReflexive:
-    def test_reflexive_eq(self):
-        e = 1 == col("x")
-        assert isinstance(e, Expr)
-        assert e.to_sql() == "(x = 1)"
-
-    def test_reflexive_ne(self):
-        e = 1 != col("x")
-        assert isinstance(e, Expr)
-        assert e.to_sql() == "(x <> 1)"
-
-    def test_reflexive_lt(self):
-        # 1 < x  =>  (x > 1)
-        e = 1 < col("x")
-        assert isinstance(e, Expr)
-        assert e.to_sql() == "(x > 1)"
-
-    def test_reflexive_gt(self):
-        # 1 > x  =>  (x < 1)
-        e = 1 > col("x")
-        assert isinstance(e, Expr)
-        assert e.to_sql() == "(x < 1)"
-
-    def test_reflexive_and(self):
-        e = True & col("active")
-        assert isinstance(e, Expr)
-        assert e.to_sql() == "(true AND active)"
-
-    def test_reflexive_or(self):
-        e = False | col("inactive")
-        assert isinstance(e, Expr)
-        assert e.to_sql() == "(false OR inactive)"
-
-
 # ── integration tests: end-to-end query against a real table ─────────────────


@@ -506,72 +432,6 @@ class TestColNamingIntegration:
        assert sorted(result["upper_name"].to_pylist()) == ["ALICE", "BOB", "CHARLIE"]


-@pytest.fixture
-def type_check_table(tmp_path):
-    """Fixture that creates a table with Date32 and Decimal128 columns."""
-    db = lancedb.connect(str(tmp_path))
-    schema = pa.schema(
-        [
-            ("date", pa.date32()),
-            ("decimal", pa.decimal128(10, 2)),
-            ("binary", pa.binary()),
-        ]
-    )
-    data = pa.table(
-        {
-            "date": [date(2024, 1, 1), date(2024, 1, 2)],
-            "decimal": [Decimal("10.50"), Decimal("20.75")],
-            "binary": [b"\x01", b"\x02"],
-        },
-        schema=schema,
-    )
-    return db.create_table("extended_types", data)
-
-
-class TestExtendedTypeIntegration:
-    """Integration tests verifying that typed literals work correctly in filters."""
-
-    def test_date_integration(self, type_check_table):
-        """Verify that Date32 literals are correctly parsed and filtered."""
-        result = (
-            type_check_table.search()
-            .where(col("date") == lit(date(2024, 1, 1)))
-            .to_arrow()
-        )
-        assert result.num_rows == 1
-        assert result["date"][0].as_py() == date(2024, 1, 1)
-
-    def test_decimal_integration(self, tmp_path):
-        """A Decimal literal must retain full 128-bit precision in a filter.
-
-        1.234567890123456789 and 1.234567890123456790 differ only in the last
-        digit and are indistinguishable once truncated to f64.  The filter
-        therefore returns the single expected row only if ``lit(Decimal)``
-        produces a true ``Decimal128`` scalar rather than being coerced to f64.
-        """
-        low = Decimal("1.234567890123456789")
-        high = Decimal("1.234567890123456790")
-
-        db = lancedb.connect(str(tmp_path / "decimal_precision"))
-        schema = pa.schema([("val", pa.decimal128(19, 18))])
-        table = db.create_table(
-            "decimal_precision",
-            pa.table({"val": [low, high]}, schema=schema),
-        )
-
-        result = table.search().where(col("val") < lit(high)).to_arrow()
-        assert result.num_rows == 1
-        assert result["val"][0].as_py() == low
-
-    def test_binary_integration(self, type_check_table):
-        """Verify that Binary literals are correctly filtered."""
-        result = (
-            type_check_table.search().where(col("binary") == lit(b"\x01")).to_arrow()
-        )
-        assert result.num_rows == 1
-        assert result["binary"][0].as_py() == b"\x01"
-
-
 # ── bytes / binary column integration tests ───────────────────────────────────


@@ -1,101 +0,0 @@
-# SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-import lancedb
-import pyarrow as pa
-import pytest
-
-# The metrics recorder is process-global and installed once, so the whole
-# bridge is exercised in a single test to avoid cross-test global-state coupling.
-
-
-def _metrics_by_name(reader):
-    data = reader.get_metrics_data()
-    result = {}
-    for resource_metrics in data.resource_metrics:
-        for scope_metrics in resource_metrics.scope_metrics:
-            for metric in scope_metrics.metrics:
-                result[metric.name] = metric
-    return result
-
-
-def test_instrument_lancedb_metrics_exports_object_store_metrics(tmp_path):
-    pytest.importorskip("opentelemetry.sdk.metrics")
-    from lancedb.otel import instrument_lancedb_metrics
-    from opentelemetry.sdk.metrics import MeterProvider
-    from opentelemetry.sdk.metrics.export import InMemoryMetricReader
-
-    reader = InMemoryMetricReader()
-    provider = MeterProvider(metric_readers=[reader])
-    assert instrument_lancedb_metrics(provider)
-
-    # The catalog is populated once the recorder is installed.
-    from lancedb._lancedb import lancedb_metrics_catalog
-
-    catalog = {desc.name: desc for desc in lancedb_metrics_catalog()}
-    # Every metric kind emitted by the object store must be described so it is
-    # surfaced by the bridge (counter, histogram, and gauge).
-    assert catalog["lance_object_store_requests_total"].kind == "counter"
-    assert catalog["lance_object_store_request_duration_seconds"].kind == "histogram"
-    assert catalog["lance_object_store_in_flight_requests"].kind == "gauge"
-    assert catalog["lance_object_store_retryable_responses_total"].kind == "counter"
-
-    # Generate object store activity on the local filesystem (scheme "file").
-    db = lancedb.connect(str(tmp_path))
-    table = db.create_table("t", pa.table({"id": pa.array(range(256))}))
-    assert table.count_rows() == 256
-    assert table.to_arrow().num_rows == 256
-
-    metrics = _metrics_by_name(reader)
-
-    requests = metrics["lance_object_store_requests_total"]
-    points = list(requests.data.data_points)
-    assert points, "expected at least one request data point"
-    # Object store metrics are labelled by `operation` and `base` (the store
-    # scheme, e.g. "file", by default).
-    assert all("base" in p.attributes and "operation" in p.attributes for p in points)
-    assert sum(p.value for p in points) > 0
-
-    # Histograms are decomposed into bucket / count / sum observable counters.
-    bucket = metrics["lance_object_store_request_duration_seconds_bucket"]
-    bucket_points = list(bucket.data.data_points)
-    assert bucket_points
-    assert all("le" in p.attributes for p in bucket_points)
-    # The implicit +Inf bucket must be present and is the cumulative maximum.
-    assert any(p.attributes["le"] == "+Inf" for p in bucket_points)
-
-    count = metrics["lance_object_store_request_duration_seconds_count"]
-    assert sum(p.value for p in count.data.data_points) > 0
-
-    # The `_sum` instrument must also be wired and report positive latency.
-    duration_sum = metrics["lance_object_store_request_duration_seconds_sum"]
-    assert sum(p.value for p in duration_sum.data.data_points) > 0
-
-    # Unit handling: only `_sum` keeps the histogram's unit (seconds); `_bucket`
-    # and `_count` observe cumulative counts and are unitless.
-    assert duration_sum.unit == "s"
-    assert bucket.unit == ""
-    assert count.unit == ""
-
-
-def test_snapshot_empty_before_install_is_safe():
-    # snapshot is callable regardless of installation state and never raises.
-    from lancedb._lancedb import snapshot_lancedb_metrics
-
-    assert isinstance(snapshot_lancedb_metrics(), list)
-
-
-def test_instrument_warns_when_recorder_unavailable(monkeypatch):
-    # A foreign `metrics` recorder already installed -> register returns False;
-    # instrument_lancedb_metrics must warn and return False without instrumenting.
-    pytest.importorskip("opentelemetry.sdk.metrics")
-    import lancedb.otel as otel
-    from opentelemetry.sdk.metrics import MeterProvider
-    from opentelemetry.sdk.metrics.export import InMemoryMetricReader
-
-    monkeypatch.setattr(otel, "register_lancedb_metrics_recorder", lambda: False)
-
-    reader = InMemoryMetricReader()
-    provider = MeterProvider(metric_readers=[reader])
-    with pytest.warns(UserWarning, match="recorder"):
-        assert otel.instrument_lancedb_metrics(provider) is False
@@ -780,7 +780,7 @@ name = "cuda-bindings"
 version = "13.3.1"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "cuda-pathfinder", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
+    { name = "cuda-pathfinder" },
 ]
 wheels = [
    { url = "https://files.pythonhosted.org/packages/a9/21/8464d133752951c154feafb3b65c297e7d80f301183d220bec4c830f1441/cuda_bindings-13.3.1-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:120fcc53d57903df529c3486962c56528cba5b7d6c57c99537320ed9922c8b86", size = 6073403, upload-time = "2026-05-29T23:11:36.22Z" },
@@ -815,37 +815,37 @@ wheels = [

 [package.optional-dependencies]
 cublas = [
-    { name = "nvidia-cublas", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-cublas", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 cudart = [
-    { name = "nvidia-cuda-runtime", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-cuda-runtime", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 cufft = [
-    { name = "nvidia-cufft", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-cufft", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 cufile = [
    { name = "nvidia-cufile", marker = "sys_platform == 'linux'" },
 ]
 cupti = [
-    { name = "nvidia-cuda-cupti", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-cuda-cupti", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 curand = [
-    { name = "nvidia-curand", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-curand", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 cusolver = [
-    { name = "nvidia-cusolver", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-cusolver", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 cusparse = [
-    { name = "nvidia-cusparse", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-cusparse", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 nvjitlink = [
-    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-nvjitlink", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 nvrtc = [
-    { name = "nvidia-cuda-nvrtc", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-cuda-nvrtc", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]
 nvtx = [
-    { name = "nvidia-nvtx", marker = "(python_full_version < '3.14' and sys_platform == 'win32') or sys_platform == 'linux'" },
+    { name = "nvidia-nvtx", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
 ]

 [[package]]
@@ -1909,9 +1909,6 @@ embeddings = [
    { name = "sentencepiece" },
    { name = "torch" },
 ]
-otel = [
-    { name = "opentelemetry-api" },
-]
 pylance = [
    { name = "pylance" },
 ]
@@ -1926,7 +1923,6 @@ tests = [
    { name = "boto3" },
    { name = "datafusion" },
    { name = "duckdb" },
-    { name = "opentelemetry-sdk" },
    { name = "pandas", version = "2.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
    { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.14'" },
    { name = "pandas", version = "3.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.14'" },
@@ -1967,8 +1963,6 @@ requires-dist = [
    { name = "open-clip-torch", marker = "extra == 'clip'" },
    { name = "open-clip-torch", marker = "extra == 'embeddings'", specifier = ">=2.20.0" },
    { name = "openai", marker = "extra == 'embeddings'", specifier = ">=1.6.1" },
-    { name = "opentelemetry-api", marker = "extra == 'otel'" },
-    { name = "opentelemetry-sdk", marker = "extra == 'tests'", specifier = ">=1.30.0" },
    { name = "overrides", marker = "python_full_version < '3.12'", specifier = ">=0.7" },
    { name = "packaging", specifier = ">=23.0" },
    { name = "pandas", marker = "extra == 'tests'", specifier = ">=1.4" },
@@ -2000,7 +1994,7 @@ requires-dist = [
    { name = "transformers", marker = "extra == 'siglip'", specifier = ">=4.41.0" },
    { name = "typing-extensions", marker = "python_full_version < '3.11' and extra == 'dev'", specifier = ">=4.0.0" },
 ]
-provides-extras = ["azure", "clip", "dev", "docs", "embeddings", "otel", "pylance", "siglip", "tests"]
+provides-extras = ["azure", "clip", "dev", "docs", "embeddings", "pylance", "siglip", "tests"]

 [[package]]
 name = "lomond"
@@ -2781,7 +2775,7 @@ name = "nvidia-cudnn-cu13"
 version = "9.19.0.56"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-cublas", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
+    { name = "nvidia-cublas" },
 ]
 wheels = [
    { url = "https://files.pythonhosted.org/packages/f1/84/26025437c1e6b61a707442184fa0c03d083b661adf3a3eecfd6d21677740/nvidia_cudnn_cu13-9.19.0.56-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:6ed29ffaee1176c612daf442e4dd6cfeb6a0caa43ddcbeb59da94953030b1be4", size = 433781201, upload-time = "2026-02-03T20:40:53.805Z" },
@@ -2793,7 +2787,7 @@ name = "nvidia-cufft"
 version = "12.0.0.61"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
+    { name = "nvidia-nvjitlink" },
 ]
 wheels = [
    { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
@@ -2823,9 +2817,9 @@ name = "nvidia-cusolver"
 version = "12.0.4.66"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-cublas", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
-    { name = "nvidia-cusparse", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
-    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
+    { name = "nvidia-cublas" },
+    { name = "nvidia-cusparse" },
+    { name = "nvidia-nvjitlink" },
 ]
 wheels = [
    { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
@@ -2837,7 +2831,7 @@ name = "nvidia-cusparse"
 version = "12.6.3.3"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "nvidia-nvjitlink", marker = "(python_full_version < '3.14' and sys_platform == 'emscripten') or (python_full_version < '3.14' and sys_platform == 'win32') or (sys_platform != 'emscripten' and sys_platform != 'win32')" },
+    { name = "nvidia-nvjitlink" },
 ]
 wheels = [
    { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
@@ -2940,45 +2934,6 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/f6/46/180e14be801a75bc13f234cb1b594b232adeb9c84e60a9ab1832e8333591/openai-2.40.0-py3-none-any.whl", hash = "sha256:2b205637ff214477f9ce9ab035e9f494db0e3fa8f1e599008953735fbf6ff1ff", size = 1350935, upload-time = "2026-06-01T21:48:21.462Z" },
 ]

-[[package]]
-name = "opentelemetry-api"
-version = "1.43.0"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "typing-extensions" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/ae/cc/e4c9584181f86494df0f6bdec1a4f3280c50db44704dc2a407e994fc87bb/opentelemetry_api-1.43.0.tar.gz", hash = "sha256:107d0d03857ea8fc7c5fcbbbd83f800c281f0d560553d61c1d675fccfd1761c1", size = 73476, upload-time = "2026-06-24T15:19:55.323Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/17/83/6dba32b85f31868400440dc7ad2ca1eab94cbbf3a7b0459ed39f8311a9e2/opentelemetry_api-1.43.0-py3-none-any.whl", hash = "sha256:20acf45e9b21851926835292e4045d290acade1edd2ff3de86d2f069687ba1fd", size = 61912, upload-time = "2026-06-24T15:19:35.434Z" },
-]
-
-[[package]]
-name = "opentelemetry-sdk"
-version = "1.43.0"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "opentelemetry-api" },
-    { name = "opentelemetry-semantic-conventions" },
-    { name = "typing-extensions" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/3e/eb/5041074274ac0956b03637cc039d434569112468e875eddfcc9a0674ce06/opentelemetry_sdk-1.43.0.tar.gz", hash = "sha256:d8187c81c162df9913e4003dd6485f7390d9a24fc17026ec7387b8b8218b08e9", size = 254744, upload-time = "2026-06-24T15:20:08.467Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/49/e3/b17be23af124201c9f52eececd4cc8ddfed1597d37b4ee771895d325805c/opentelemetry_sdk-1.43.0-py3-none-any.whl", hash = "sha256:d1323a547c1ce69d6a069a17a44b7da82bb8b332051ecb074041f87642c86823", size = 178852, upload-time = "2026-06-24T15:19:52.169Z" },
-]
-
-[[package]]
-name = "opentelemetry-semantic-conventions"
-version = "0.64b0"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "opentelemetry-api" },
-    { name = "typing-extensions" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/5a/30/5f26df29509eccd86b99b481ac9ffa39da49ba9577cc69071c552ae30447/opentelemetry_semantic_conventions-0.64b0.tar.gz", hash = "sha256:72f76fb2d1582d9d033dd1fcd84532e961e6ff3d90d24ba6fabc72975a83864c", size = 148340, upload-time = "2026-06-24T15:20:09.267Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/f2/ca/23ba87a221b574a7c5a99d48849d80bfe8b047624681357e2b002e566187/opentelemetry_semantic_conventions-0.64b0-py3-none-any.whl", hash = "sha256:ea77e85e354b8f604ddbe5f3d9135216f982fa4d77e5859ac30f6d8a50505aa6", size = 203713, upload-time = "2026-06-24T15:19:53.339Z" },
-]
-
 [[package]]
 name = "overrides"
 version = "7.7.0"
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb"
-version = "0.32.0-beta.0"
+version = "0.31.0-beta.4"
 edition.workspace = true
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license.workspace = true
@@ -14,7 +14,6 @@ rust-version.workspace = true
 ahash = { workspace = true }
 arrow = { workspace = true }
 arrow-array = { workspace = true }
-arrow-buffer = { workspace = true }
 arrow-data = { workspace = true }
 arrow-schema = { workspace = true }
 arrow-select = { workspace = true }
@@ -49,8 +48,6 @@ lance-encoding = { workspace = true }
 lance-arrow = { workspace = true }
 lance-namespace = { workspace = true }
 lance-namespace-impls = { workspace = true }
-metrics = { workspace = true, optional = true }
-metrics-util = { workspace = true, optional = true }
 moka = { workspace = true }
 pin-project = { workspace = true }
 tokio = { version = "1.23", features = ["rt-multi-thread", "sync"] }
@@ -110,8 +107,6 @@ http-body = "1"                                        # Matching reqwest
 rstest = "0.23.0"
 test-log = "0.2"
 serial_test = "3"
-[target.'cfg(unix)'.dev-dependencies]
-pprof = { version = "0.14", features = ["flamegraph"] }


 [features]
@@ -130,12 +125,6 @@ azure = [
    "lance-namespace-impls/dir-azure",
    "lance-namespace-impls/credential-vendor-azure",
 ]
-cos = ["lance/tencent", "lance-io/tencent"]
-goosefs = [
-    "lance/goosefs",
-    "lance-io/goosefs",
-    "lance-namespace-impls/dir-goosefs",
-]
 huggingface = [
    "lance/huggingface",
    "lance-io/huggingface",
@@ -149,15 +138,6 @@ remote = [
    "lance-namespace-impls/rest",
    "lance-namespace-impls/rest-adapter",
 ]
-# Publish LanceDB's internal metrics (currently object store request counts,
-# bytes, latency, errors, and throttles) through the `metrics` crate facade,
-# and re-export the `metrics` crate as `lancedb::metrics`. Install any
-# `metrics`-compatible recorder to collect them.
-metrics = ["dep:metrics", "lance/metrics", "lance-io/metrics"]
-# Additional adapter on top of `metrics` that installs a process-global recorder
-# and exposes a pull-based snapshot/catalog API (see `lancedb::metrics_otel`)
-# for bridging metrics into OpenTelemetry or other pull-based exporters.
-metrics-otel = ["metrics", "dep:metrics-util"]
 fp16kernels = ["lance-linalg/fp16kernels"]
 s3-test = []
 bedrock = ["dep:aws-sdk-bedrockruntime"]
@@ -183,16 +163,9 @@ required-features = ["sentence-transformers"]
 name = "bedrock"
 required-features = ["bedrock"]

-[[example]]
-name = "bench_streaming_dataloader"
-
 [[example]]
 name = "simple"

-[[example]]
-name = "polars"
-required-features = ["polars"]
-
 [[example]]
 name = "full_text_search"

@@ -1,272 +0,0 @@
-// SPDX-License-Identifier: Apache-2.0
-// SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-//! Benchmark + CPU profiler for the PermutationReader used by the elastic
-//! streaming dataloader.
-//!
-//! Normal sweep:
-//!   cargo run --release --example bench_streaming_dataloader
-//!
-//! Flamegraph (self-contained, no perf/dtrace needed):
-//!   BENCH_PROFILE=1 BENCH_CHUNK=64 cargo run --release \
-//!       --example bench_streaming_dataloader
-//!   # writes flamegraph.svg in the current directory
-//!
-//! Environment variables:
-//!   BENCH_NUM_ROWS   – total rows (default 49152 = 24 × 2048)
-//!   BENCH_NUM_SPLITS – number of splits (default 24)
-//!   BENCH_STEPS      – round-robin cycles per chunk-size trial (default 200)
-//!   BENCH_ROW_BYTES  – bytes of payload per row (default 4096)
-//!   BENCH_CHUNK      – restrict sweep to this single chunk size
-//!   BENCH_PROFILE    – if set to "1", capture a pprof flamegraph SVG
-
-use std::{sync::Arc, time::Instant};
-
-use arrow_array::{Int32Array, LargeBinaryArray, RecordBatch};
-use arrow_schema::{DataType, Field, Schema};
-use lancedb::{
-    Result, Table,
-    arrow::{SendableRecordBatchStream, SimpleRecordBatchStream},
-    connect,
-    dataloader::permutation::{
-        builder::{PermutationBuilder, ShuffleStrategy},
-        reader::PermutationReader,
-        split::{SplitSizes, SplitStrategy},
-    },
-    query::Select,
-};
-
-fn env_usize(key: &str, default: usize) -> usize {
-    std::env::var(key)
-        .ok()
-        .and_then(|v| v.parse().ok())
-        .unwrap_or(default)
-}
-
-// ---------------------------------------------------------------------------
-// Table creation
-// ---------------------------------------------------------------------------
-
-async fn make_base_table(num_rows: usize, row_bytes: usize) -> Result<Table> {
-    let schema = Arc::new(Schema::new(vec![
-        Field::new("id", DataType::Int32, false),
-        Field::new("payload", DataType::LargeBinary, false),
-    ]));
-    let payload = vec![0u8; row_bytes];
-    let ids: Int32Array = (0..num_rows as i32).collect();
-    let payloads: LargeBinaryArray = (0..num_rows).map(|_| Some(payload.as_slice())).collect();
-    let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(ids), Arc::new(payloads)])?;
-    let stream: SendableRecordBatchStream = Box::pin(SimpleRecordBatchStream::new(
-        futures::stream::once(std::future::ready(Ok(batch))),
-        schema,
-    ));
-    let db = connect("memory:///").execute().await?;
-    db.create_table("base", stream).execute().await
-}
-
-async fn make_permutation_table(base: &Table, num_splits: usize) -> Result<Table> {
-    PermutationBuilder::new(base.clone())
-        .with_split_strategy(
-            SplitStrategy::Random {
-                seed: Some(42),
-                sizes: SplitSizes::Fixed(num_splits as u64),
-                clump_size: None,
-            },
-            None,
-        )
-        .with_shuffle_strategy(ShuffleStrategy::Random {
-            seed: Some(42),
-            clump_size: None,
-        })
-        .build()
-        .await
-}
-
-// ---------------------------------------------------------------------------
-// Round-robin hot loop (mirrors StreamingDataset.__iter__)
-// ---------------------------------------------------------------------------
-
-async fn run_hot_loop(
-    readers: &[PermutationReader],
-    chunk_size: usize,
-    steps: usize,
-) -> Result<(usize, f64)> {
-    let n = readers.len();
-    let split_sizes: Vec<usize> = readers.iter().map(|r| r.count_rows() as usize).collect();
-
-    struct SplitBuf {
-        batch: Option<RecordBatch>,
-        row_in_batch: usize,
-        consumed: usize,
-    }
-    let mut bufs: Vec<SplitBuf> = (0..n)
-        .map(|_| SplitBuf {
-            batch: None,
-            row_in_batch: 0,
-            consumed: 0,
-        })
-        .collect();
-
-    // Pre-fill
-    for i in 0..n {
-        let fetch = chunk_size.min(split_sizes[i]);
-        if fetch > 0 {
-            let offsets: Vec<u64> = (0..fetch as u64).collect();
-            bufs[i].batch = Some(readers[i].take_offsets(&offsets, Select::All).await?);
-        }
-    }
-
-    let mut total_rows = 0usize;
-    let t0 = Instant::now();
-
-    'outer: for _step in 0..steps {
-        for i in 0..n {
-            if bufs[i].consumed >= split_sizes[i] {
-                break 'outer;
-            }
-            let need_refill = bufs[i]
-                .batch
-                .as_ref()
-                .map(|b| bufs[i].row_in_batch >= b.num_rows())
-                .unwrap_or(true);
-            if need_refill {
-                let start = bufs[i].consumed as u64;
-                let remaining = (split_sizes[i] - bufs[i].consumed) as u64;
-                let fetch = chunk_size.min(remaining as usize);
-                let offsets: Vec<u64> = (start..start + fetch as u64).collect();
-                bufs[i].batch = Some(readers[i].take_offsets(&offsets, Select::All).await?);
-                bufs[i].row_in_batch = 0;
-            }
-            bufs[i].row_in_batch += 1;
-            bufs[i].consumed += 1;
-            total_rows += 1;
-        }
-    }
-
-    Ok((total_rows, t0.elapsed().as_secs_f64()))
-}
-
-// ---------------------------------------------------------------------------
-// Main
-// ---------------------------------------------------------------------------
-
-#[tokio::main]
-async fn main() -> Result<()> {
-    let num_splits = env_usize("BENCH_NUM_SPLITS", 24);
-    let num_rows = env_usize("BENCH_NUM_ROWS", num_splits * 2048);
-    let steps = env_usize("BENCH_STEPS", 200);
-    let row_bytes = env_usize("BENCH_ROW_BYTES", 4096);
-    let single_chunk: Option<usize> = std::env::var("BENCH_CHUNK")
-        .ok()
-        .and_then(|v| v.parse().ok());
-    let do_profile = std::env::var("BENCH_PROFILE")
-        .map(|v| v == "1")
-        .unwrap_or(false);
-
-    assert_eq!(
-        num_rows % num_splits,
-        0,
-        "NUM_ROWS must be divisible by NUM_SPLITS"
-    );
-
-    println!("Benchmark config:");
-    println!(
-        "  num_rows={}  num_splits={}  rows/split={}  steps={}  row_bytes={}",
-        num_rows,
-        num_splits,
-        num_rows / num_splits,
-        steps,
-        row_bytes,
-    );
-    println!(
-        "  ~{:.1} MB total",
-        (num_rows * row_bytes) as f64 / (1024.0 * 1024.0)
-    );
-    println!();
-
-    print!("Building base table... ");
-    let _ = std::io::Write::flush(&mut std::io::stdout());
-    let base = make_base_table(num_rows, row_bytes).await?;
-    println!("done");
-
-    print!("Building permutation table... ");
-    let _ = std::io::Write::flush(&mut std::io::stdout());
-    let perm = make_permutation_table(&base, num_splits).await?;
-    println!("done");
-
-    print!("Building {} PermutationReaders... ", num_splits);
-    let _ = std::io::Write::flush(&mut std::io::stdout());
-    let base_inner = base.base_table().clone();
-    let perm_inner = perm.base_table().clone();
-    let mut readers = Vec::with_capacity(num_splits);
-    for split in 0..num_splits {
-        readers.push(
-            PermutationReader::try_from_tables(
-                base_inner.clone(),
-                perm_inner.clone(),
-                split as u64,
-            )
-            .await?,
-        );
-    }
-    println!("done ({} rows/split)", readers[0].count_rows());
-    println!();
-
-    let chunk_sizes: Vec<usize> = if let Some(c) = single_chunk {
-        vec![c]
-    } else {
-        vec![1, 4, 16, 64, 256, 1024, 4096, 16384]
-    };
-
-    if do_profile {
-        #[cfg(unix)]
-        {
-            let chunk = chunk_sizes[0];
-            println!("Profiling chunk={chunk} for {steps} steps...");
-            // Warm-up outside the profiler window
-            let _ = run_hot_loop(&readers, chunk, 1).await?;
-
-            let guard = pprof::ProfilerGuardBuilder::default()
-                .frequency(1000)
-                .build()
-                .unwrap();
-
-            let (rows, elapsed) = run_hot_loop(&readers, chunk, steps).await?;
-
-            if let Ok(report) = guard.report().build() {
-                let svg_path = "flamegraph.svg";
-                let file = std::fs::File::create(svg_path).unwrap();
-                report.flamegraph(file).unwrap();
-                println!("Flamegraph written to {svg_path}");
-            }
-
-            let rows_per_sec = rows as f64 / elapsed;
-            println!("chunk={chunk}  {rows} rows  {elapsed:.3}s  {rows_per_sec:.0} rows/s");
-        }
-        #[cfg(not(unix))]
-        {
-            println!("Flamegraph profiling (BENCH_PROFILE=1) is not supported on this platform.");
-            println!("Run without BENCH_PROFILE to get throughput numbers.");
-        }
-    } else {
-        println!(
-            "{:>6}  {:>7}  {:>8}  {:>11}  {:>10}",
-            "chunk", "rows", "elapsed", "rows/s", "ms/step"
-        );
-        println!("{}", "-".repeat(52));
-
-        for &chunk in &chunk_sizes {
-            let _ = run_hot_loop(&readers, chunk, 1).await?;
-            let (rows, elapsed) = run_hot_loop(&readers, chunk, steps).await?;
-            let rows_per_sec = rows as f64 / elapsed;
-            let ms_per_step = elapsed / steps as f64 * 1000.0;
-            println!(
-                "{:>6}  {:>7}  {:>7.3}s  {:>11.0}  {:>9.1}ms",
-                chunk, rows, elapsed, rows_per_sec, ms_per_step,
-            );
-        }
-    }
-
-    println!("\nDone.");
-    Ok(())
-}
@@ -1,47 +0,0 @@
-// SPDX-License-Identifier: Apache-2.0
-// SPDX-FileCopyrightText: Copyright The LanceDB Authors
-
-//! This example demonstrates ingesting a Polars DataFrame into LanceDB and
-//! reading it back out as a Polars DataFrame.
-
-use lancedb::arrow::IntoPolars;
-use lancedb::query::ExecutableQuery;
-use lancedb::{Result, connect};
-use polars::prelude::{DataFrame, NamedFrom, Series};
-
-fn make_dataframe() -> DataFrame {
-    let ids = Series::new("id", &[1i32, 2, 3, 4, 5]);
-    let names = Series::new("name", &["Alice", "Bob", "Carol", "Dave", "Eve"]);
-    let scores = Series::new("score", &[9.5f64, 8.1, 7.3, 9.0, 6.5]);
-    DataFrame::new(vec![ids, names, scores]).unwrap()
-}
-
-#[tokio::main]
-async fn main() -> Result<()> {
-    let tmp = tempfile::tempdir().unwrap();
-    let db = connect(tmp.path().to_str().unwrap()).execute().await?;
-
-    // Ingest a Polars DataFrame directly — DataFrame now implements Scannable.
-    let df = make_dataframe();
-    println!("Input DataFrame:\n{df}");
-
-    let table = db.create_table("people", df).execute().await?;
-
-    // Append more rows.
-    let more = DataFrame::new(vec![
-        Series::new("id", &[6i32, 7]),
-        Series::new("name", &["Frank", "Grace"]),
-        Series::new("score", &[7.8f64, 8.9]),
-    ])
-    .unwrap();
-    table.add(more).execute().await?;
-
-    // Read back as a Polars DataFrame.
-    let result_df = table.query().execute().await?.into_polars().await?;
-
-    println!(
-        "\nRound-tripped DataFrame ({} rows):\n{result_df}",
-        result_df.height()
-    );
-    Ok(())
-}
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Jack Ye	bff911a65d	chore: point lance dependency at jack branch	2026-06-29 22:46:07 -07:00
Wyatt Alt	3a4cdb7aff	fix(udf): JobHandle.wait() terminates on failed jobs wait() (sync + async) only stopped on finished/stale/committed, so a job the server already reported as state=failed was polled until the (default 3600s) timeout, then raised a misleading TimeoutError instead of the real cause. A doomed backfill -- e.g. a multi-column REFRESH COLUMN of a scalar UDF -- hung the client even though get_job surfaced the failure within ~3s. Add a terminal failed branch that raises JobFailedError carrying the server error, exported from the package. Verified end-to-end against the cluster: raises in 3.6s instead of hanging. Unit-tested with a mock conn (sync+async, failure + success + committed paths). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:35 -07:00
Wyatt Alt	142ac835d3	client: job_history() and errors() over REST (SHOW JOB HISTORY / SHOW ERRORS) The client exposed list_jobs/get_job/cancel_job but not the durable job history or the per-row UDF errors, so those SQL/REST surfaces had no SDK equivalent. Add job_history(job_id=None) and errors(job_id=None, table=None) through every layer: - Database trait + Connection API (JobHistoryInfo, JobErrorInfo types). - Remote REST impl: GET /v1/job/history (?job=) and GET /v1/job/errors (?job=&table=), with serde response types + From mappings. - pyo3 bindings + pyclasses JobHistoryEntry / JobErrorEntry, registered. - Python sync + async db.py wrappers. Mirrors the existing list_jobs plumbing exactly. Remote-handler test asserts the GET paths, query filters, and response parsing for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:35 -07:00
Wyatt Alt	3f44f93e92	job wait(): poll by id via get_job (point access) instead of list_jobs JobHandle/AsyncJobHandle now poll conn.get_job(id, table) -- one job -- instead of list_jobs() + client-side filter over every active job. The job's table is threaded in from refresh_column / MV refresh as an O(1) lookup hint. Plumbs get_job through the Database trait (default not_supported), RemoteDatabase (GET /v1/job/{id}?table=...), the Connection wrapper, and the pyo3 binding + db.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:35 -07:00
Wyatt Alt	9dfa43a9de	fix: sync Connection.lineage delegates to AsyncConnection.lineage self._conn on a remote sync connection is an AsyncConnection (python), which exposes `lineage` (parses the JSON), not the pyo3 `table_lineage`. The sync wrapper was calling self._conn.table_lineage -> AttributeError. Drive self._conn.lineage on the loop instead, mirroring create_materialized_view. (table_lineage stays the pyo3 method the async path calls via self._inner.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	03e895fa5c	Merge integration submodule (refresh_column JobHandle / create_mv fix) into lineage	2026-06-29 22:35:34 -07:00
Wyatt Alt	c31e53088e	client: slice 4 -- Python lineage surface - new lineage.py: Lineage / Node / Edge / FunctionRef dataclasses that parse the server's lineage JSON, with to_dict(), to_graphviz() (drift edges dashed+red), and _repr_html_(); plus .functions() / .stale() helpers. - Connection.lineage(table, column=, direction=, depth=) (sync + async) calls the pyo3 table_lineage binding and deserializes into Lineage. - Table.lineage(column=, ...) via the table's job connection; MaterializedView / AsyncMaterializedView .lineage() delegate to the backing table (the server already includes the view's sources + downstream dependents). - export the new types. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	434a5be187	feat(client): Table.refresh_column returns a JobHandle (like MaterializedView.refresh) refresh_column returned the bare job-id str, so callers had to wrap it: db.job(tbl.refresh_column("c")).wait(). Mirror MaterializedView.refresh() and return a JobHandle directly, so tbl.refresh_column("c").wait() / .status() / .id work without the wrapper. (db.job(job_id) stays for reconnecting by a stored id.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	78aa005093	client: slice 3 -- thread table_lineage through the remote client + pyo3 A new Database::table_lineage(TableLineageRequest) -> Result<String> threaded end to end: default not_supported in the trait; the remote impl issues GET /v1/table/{name}/lineage with column/direction/depth query params and returns the body verbatim; connection.rs exposes a pub wrapper; the pyo3 binding hands the JSON string to Python. The lineage payload is carried as opaque JSON on purpose: the open-source lancedb client must not depend on the sophon-internal derived_jobs crate that defines the lineage schema, so the wire format is the contract and the Python layer deserializes it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	6191542cfe	fix(mv): MaterializedView.refresh calls the async _refresh (underscore) The sync _refresh_materialized_view called self._conn.refresh_materialized_view (no underscore); the async method is _refresh_materialized_view, so MaterializedView.refresh() raised AttributeError. Add the underscore. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	6af3088b91	client: make refresh_materialized_view private (reach it via the handle) Refresh is a submit-a-job verb, so its only public surface should be MaterializedView.refresh() / AsyncMaterializedView.refresh() (which return a job handle). Rename the connection methods to _refresh_materialized_view and have the handles call that, so the raw by-name refresh is no longer advertised on the connection. The pyo3 native binding is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	e73d4618d8	fix(mv): create_materialized_view passes query as keyword, not positional The sync RemoteDBConnection.create_materialized_view assembled the SELECT but called the async create_materialized_view with the query as the 2nd positional arg, which binds to `source=` (query= is keyword-only). Every call then failed the "needs either query= or both source and select" validation. Pass query=query. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	3d92106394	client: split create_view into create_materialized_view; return job handles - create_materialized_view now takes either query= or source+select (folds in the old create_view builder) and returns a MaterializedView handle whose .wait() blocks on initial population. create_view is removed -- it was misnamed (it built a materialized view, while CREATE VIEW means the plain non-materialized view the engine also supports). - MaterializedView.refresh() and the remote Table.refresh_column() now return a JobHandle directly, so tbl.refresh_column("c").wait() needs no db.job(...) wrapper. db.job(id) is narrowed to reconnect-by-id (stored id / SQL / REST). - rename View/AsyncView -> MaterializedView/AsyncMaterializedView (+ exports). - tighten the replace path: only a not-found error on the pre-drop is benign; real failures (perms/server) now surface instead of being swallowed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	5810974b37	feat(client): Table.load_columns() REST client for LOAD COLUMNS Geneva Table.load_columns() parity on the REST-only client. Fills existing columns from an external Parquet/Lance/IPC source by primary-key join. - BaseTable::load_columns default (NotSupported) + public Table::load_columns, taking a LoadColumnsRequest (source uris/format/storage_options, target/source key, (target, source?) column mappings, on_missing, worker/batch/commit knobs). - Remote impl POSTs to /v1/table/{id}/load_columns with the matching body; mock test asserts the request shape. - PyO3 binding + Python remote Table.load_columns(source, pk, columns, *, source_format, source_pk, on_missing, ...) accepting a column list or {target: source} dict. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	8b38500b07	feat(view): full=True force-rebuild on refresh_materialized_view View.refresh(full=True) (sync + async) now works -- it previously raised NotImplementedError. Thread the flag through the client: RefreshMaterialized- ViewRequest.full -> the REST body (RemoteRefreshMaterializedViewRequest.full); pyo3 refresh_materialized_view(full=...); Connection.refresh_materialized_view( name, full=) sync + async. A full refresh forces a recompute-and-replace and preserves the view's indexes (reindexed by the distributed indexer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	fd0a3b97d0	feat(view): materialized views are first-class indexable + searchable Add View.create_index / create_scalar_index / create_fts_index / search as pass-throughs to open_table(name). A materialized view is a real Lance dataset; these let it be indexed and searched like any other table, closing the parity gap with Geneva (whose create_materialized_view returns a first-class Table). The server-side create_index handler records indexes declared on a view so they survive a full refresh (which overwrites the dataset, dropping its indices); that re-apply is wired in the sophon engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	b9f33ba1c9	feat(refresh): priority as a per-refresh knob; fix batch_size on RemoteTable Thread priority (Kueue tier) through refresh_column at every layer (Python sync+async + RemoteTable -> pyo3 -> Rust client trait/public/remote -> REST body), mirroring num_workers/batch_size. The function keeps its priority as a default; the per-refresh value overrides. Also adds the previously-missed batch_size to RemoteTable.refresh_column (the REST sync path). cargo check (lancedb --features remote --tests, lancedb-python) + ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	d4f4fef3ba	feat(refresh): batch_size is a per-refresh knob (refresh_column), not a function-only option batch_size / num_workers / max_workers are invocation concerns (how to schedule THIS refresh), so expose batch_size on refresh_column through every layer (Python sync+async -> pyo3 -> Rust client -> the REST RefreshColumnRequest.batch_size, which the handler already forwards into the backfill). num_workers/max_workers were already invocation- placed; batch_size was the gap. The function may still carry a default; the refresh override wins (extends the batch_size_override model). Both crates cargo-check clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	fbe6a5a3fd	feat(udf): computed columns as expressions -- add_columns(computed={col: fn("input")}) A computed column is an expression over a registered function applied to input columns, not a UDF coupled to a column. fn("data") already returned the expression string "fn(data)"; make it a ColumnExpr (a str subclass) that also carries the function's return type, so add_columns(computed={"vec": embed("data")}) declares the column with no hand-written type. _normalize_computed handles the new form (and tuple keys for STRUCT fan-out) and keeps the legacy {col: (sql_type, expression)} tuple. add_computed_column is deprecated (delegates, with a DeprecationWarning). The function stays decoupled from columns -- register once, apply anywhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	127054069a	feat(mv): partition_by option on create_materialized_view / create_view Thread an optional partition_by through the client: CreateMaterializedViewRequest -> REST body -> pyo3 binding -> Python create_materialized_view/create_view kwarg (sync + async). The server partitions the view's table function by the named source column -- by IVF index clusters if the column is indexed (image-dedup), else by distinct value. Unifies Geneva's partition_by + partition_by_indexed_column into one knob. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	b20931b8f7	feat: async UDF client ergonomics (AsyncConnection/AsyncTable + AsyncView/AsyncJobHandle) Mirrors the sync ergonomics on the async surface: AsyncConnection create_function(udf, replace=)/create_view/job; AsyncTable.add_computed_column; AsyncView + AsyncJobHandle (await + asyncio.sleep; shared submission-prefix matcher with the sync JobHandle). Decorator + REST routes are shared/already validated; this is the async wrapper layer. Exported from the package root. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	396d68e490	fix: JobHandle resolves the manifest job id from the submission id db.job(id) gets the submission id the refresh/backfill endpoints return, but list_jobs / cancel report the agent's manifest id (<table>-<type>-<first 8 of submission id>). JobHandle now matches that (exact id or submission prefix) so wait()/progress() truly track, and cancel() cancels by the resolved canonical id instead of the unusable submission uuid. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	ad37f87387	feat: fold UDF authoring into lancedb (udf module + connection/table ergonomics) Brings the @udf/@table_udf decorator + type inference into lancedb as lancedb.udf (Apache-2.0), and adds the ergonomic glue to the existing connection/table so there's no separate object model: - create_function() accepts a Udf (and a replace= flag) - Table.add_computed_column(column, udf) - create_view(name, source, select, ...) -> View (assembles the SELECT) - Connection.job(job_id) -> JobHandle - View / JobHandle are thin references over a connection Exports udf/table_udf/Udf/JobHandle/View from the package root. The operations stay the existing remote-only methods (enterprise/cloud); the decorator works locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	e93476f0e0	feat: explain_refresh_materialized_view over REST (EXPLAIN REFRESH SDK) Database trait gains explain_refresh_materialized_view (default NotSupported) returning an MvRefreshPlan; RemoteDatabase POSTs /v1/materialized_view/{name}/explain_refresh; Connection method; pyo3 MvRefreshPlan pyclass + binding; sync+async python wrappers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	2b41fce033	feat: cancel_job over REST (Database::cancel_job + remote impl + pyo3 + python) Exposes the existing server-side CANCEL JOB (CoordinatorCatalog::cancel_job) as a REST-backed SDK method: Database trait default NotSupported, RemoteDatabase POSTs /v1/job/{id}/cancel, pyo3 binding, sync+async python wrappers. Best-effort: a missing job returns false, not an error. Mock-HTTP unit test in test_derived_compute_routes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 22:35:34 -07:00
Wyatt Alt	04948fc4f6	feat: computed columns as a param on add_columns Per the interface design: computed columns are parameters on the existing add_columns operation, not a separate method. - BaseTable::add_computed_columns((name, sql_type) pairs + a f(args) expression) -- default NotSupported; RemoteTable posts 'computed' entries to the existing /v1/table/{id}/add_columns route. - python add_columns gains computed= on LanceTable, RemoteTable, and AsyncTable: tbl.add_columns(computed={'doubled': ('FLOAT', 'double_it(val)')}); grouped by expression so struct-returning functions' columns land adjacently.	2026-06-29 22:35:34 -07:00
Wyatt Alt	ff3c7111b9	feat: SDK surface for functions, materialized views, jobs, refresh_column Adds the derived-compute interface to the SDK: - Database trait: create/list/drop_function, create/refresh/alter/ drop/list_materialized_view, list_jobs -- default implementations return Error::NotSupported (NotImplementedError in python), so existing Database impls are unaffected; local single-node implementations are planned. BaseTable gains refresh_column with the same default. - RemoteDatabase/RemoteTable implement them against the server REST routes (/v1/function/, /v1/materialized_view/, /v1/job/list, /v1/table/{id}/refresh_column), with mock-HTTP unit tests. - Connection/Table public methods, pyo3 bindings (FunctionInfo, MaterializedViewInfo, JobInfo pyclasses), and python wrappers: sync on the DBConnection base (shared by local and remote connections), async on AsyncConnection; refresh_column on LanceTable, RemoteTable, and AsyncTable.	2026-06-29 22:35:34 -07:00