fix: support ST_DWithin filters

test: cover blob null/empty preservation across Table::optimize (#3774 )
## Description `Table::optimize()` compacts through `lance::dataset::optimize::compact_files` (`rust/lancedb/src/table/optimize.rs:155`). Until lance-format/lance#7965 that rewrite corrupted blob columns holding null or empty values, which is what #3744 reports: - **storage 2.0** (legacy v1 `lance-encoding:blob` descriptors): every payload following a null or empty row in the same fragment was rewritten as `{position: 0, size: 0}`, so it read back as `b""` and the new fragment no longer referenced the bytes — silent payload loss, unrecoverable once the pre-optimize versions are pruned. - **storage 2.2** (blob v2): a valid empty value was rewritten as null, destroying the null-vs-empty distinction. Both manifestations share one root cause: `is_inline_null_blob` classified any inline blob with `position == 0 && size == 0` as null, which is also exactly what a *valid empty value* looks like. Such rows were dropped from `blob_read_addrs`, misaligning every payload that followed. The behaviour is already correct on `main`: the vendored lance crate first carried the fix at `v10.0.0-beta.3` (#3710) and is now `v10.1.0-beta.1` (#3757). What was missing is coverage — nothing in this repo exercised a blob column containing a null or empty value through `optimize()`, which is why this shipped unnoticed. This PR adds that guard. ## Tests Two tests in `rust/lancedb/tests/blob_integration.rs`, reusing the file's existing 64 KiB dedicated-blob helpers and a delete-triggered fragment rewrite. After `id IN (1, 4)` is deleted the surviving rows are `2` (null), `3` (valid empty), `5` and `6` (payloads) — payloads sit immediately after the null/empty, which is where the misalignment landed. - `optimize_preserves_v1_blob_payloads_with_null_and_empty` — storage 2.0; asserts the **payload bytes** are unchanged across `OptimizeAction::All` (what the Python/Node `optimize()` bindings invoke). Payloads are read through `lance::Dataset::take_blobs`, since `Table::fetch_blobs` rejects legacy v1 columns. The before/after descriptors are reported on failure but deliberately *not* asserted: compaction repacks the blob file, so they shift legitimately (id 5 `(131072, 65536)` → `(0, 65536)`, id 6 `(196608, 65536)` → `(65536, 65536)`). Note that a post-compaction `position: 0` is both the legitimate first-payload offset and the bug's signature, so asserting descriptors would be actively misleading. - `optimize_preserves_blob_v2_null_and_empty_distinction` — storage >= 2.2; asserts a null stays null and a valid empty value stays non-null empty. Both assert the pre-optimize state first, so a setup change that stops producing the null/empty/payload mix fails loudly instead of passing vacuously. Both also assert the returned `CompactionMetrics` show a fragment was actually rewritten. These tests depend on `delete("id IN (1, 4)")` pushing the fragment past lance's `materialize_deletions_threshold` (0.1 by default; 2 of 6 rows here). That coupling is invisible and unasserted otherwise: against a forced no-op (`materialize_deletions_threshold: 1.5`) the metrics come back all zeroes and *every payload assertion still passes*. Since the whole point of these tests is to survive dependency changes, they check that the rewrite happened rather than trusting the planner to keep selecting the fragment. Guard verified against a pre-fix lance: with the published `lancedb==0.36.0` wheel (vendors lance 9.0.0), `Table.optimize()` on the same data rewrites the descriptors of the two rows following the null/empty from `(131072, 65536)` and `(196608, 65536)` to `(0, 0)`, and the payloads read back empty. Against the pinned `v10.1.0-beta.1`, all 39 tests in the file pass, adding roughly 10–20 ms to the file's runtime. ## Not addressed here - **No released artifact has the fix yet.** PyPI `lancedb` 0.36.0 (2026-07-29) vendors lance 9.0.0; npm `@lancedb/lancedb` 0.37.1-beta.0 predates the bump. No 9.x lance tag carries the fix: `v10.0.0-beta.3` is the first tag containing it, every `v9.1.0-beta.1`…`beta.8` is behind it, and `v9.0.0` / `v9.0.1-rc.1` sit on a diverged branch without it. A stable lancedb release needs a stable lance >= 10. - **The version skew #3744 flagged is still live.** `python/pyproject.toml` pins `pylance==9.0.0rc1` for the `tests` extra against a vendored `10.1.0-beta.1`, so Python CI still cannot observe this class of divergence. - **Only the single-fragment rewrite shape is covered.** Both tests rewrite one fragment by materializing deletions. lance's own `test_compact_blob_v1/v2_preserves_null_empty_and_payload_order` cover the multi-fragment merge shape (3 fragments → 1) at unit level, so this PR is complementary rather than redundant — it covers the binding-level path through `Table::optimize` — but it would not catch a regression that only appears when *merging* fragments. `multi_fragment_dedicated_blob_table` in the same file makes that a cheap follow-up. Closes #3744 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 14:52:25 +00:00 · 2026-08-05 19:43:33 +00:00 · 2026-08-04 12:23:03 -07:00 · 2026-08-04 11:18:22 -07:00 · 2026-08-03 15:49:52 -07:00 · 2026-08-03 12:54:41 -07:00
20 changed files with 2568 additions and 190 deletions
@@ -601,7 +601,7 @@ dependencies = [
 "bytes",
 "fastrand",
 "hex",
- "http 1.4.2",
+ "http 1.5.0",
 "sha1 0.10.6",
 "time",
 "tokio",
@@ -664,7 +664,7 @@ dependencies = [
 "bytes-utils",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 0.4.6",
 "http-body 1.1.0",
 "percent-encoding",
@@ -694,7 +694,7 @@ dependencies = [
 "bytes",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body-util",
 "regex-lite",
 "tracing",
@@ -719,7 +719,7 @@ dependencies = [
 "bytes",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "regex-lite",
 "tracing",
 ]
@@ -743,7 +743,7 @@ dependencies = [
 "bytes",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "regex-lite",
 "tracing",
 ]
@@ -773,7 +773,7 @@ dependencies = [
 "hex",
 "hmac 0.13.0",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "lru",
 "percent-encoding",
@@ -802,7 +802,7 @@ dependencies = [
 "bytes",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "regex-lite",
 "tracing",
 ]
@@ -826,7 +826,7 @@ dependencies = [
 "bytes",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "regex-lite",
 "tracing",
 ]
@@ -851,7 +851,7 @@ dependencies = [
 "aws-types",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "regex-lite",
 "tracing",
 ]
@@ -873,7 +873,7 @@ dependencies = [
 "hex",
 "hmac 0.13.0",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "p256",
 "percent-encoding",
 "ring",
@@ -906,7 +906,7 @@ dependencies = [
 "bytes",
 "crc-fast",
 "hex",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "md-5 0.11.0",
@@ -940,7 +940,7 @@ dependencies = [
 "bytes-utils",
 "futures-core",
 "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "percent-encoding",
@@ -961,7 +961,7 @@ dependencies = [
 "h2 0.3.27",
 "h2 0.4.14",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 0.4.6",
 "hyper 0.14.32",
 "hyper 1.9.0",
@@ -1023,7 +1023,7 @@ dependencies = [
 "bytes",
 "fastrand",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 0.4.6",
 "http-body 1.1.0",
 "http-body-util",
@@ -1044,7 +1044,7 @@ dependencies = [
 "aws-smithy-types",
 "bytes",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "pin-project-lite",
 "tokio",
 "tracing",
@@ -1070,7 +1070,7 @@ checksum = "7442cb268338f0eb8278140a107c046756aa01093d8ef5e99628d34ae09c94f5"
 dependencies = [
 "aws-smithy-runtime-api",
 "aws-smithy-types",
- "http 1.4.2",
+ "http 1.5.0",
 ]

 [[package]]
@@ -1084,7 +1084,7 @@ dependencies = [
 "bytes-utils",
 "futures-core",
 "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 0.4.6",
 "http-body 1.1.0",
 "http-body-util",
@@ -1132,7 +1132,7 @@ dependencies = [
 "axum-core",
 "bytes",
 "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "hyper 1.9.0",
@@ -1165,7 +1165,7 @@ dependencies = [
 "async-trait",
 "bytes",
 "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "mime",
@@ -3081,6 +3081,16 @@ version = "0.1.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e1d926b4d407d372f141f93bb444696142c29d32962ccbd3531117cf3aa0bfa9"

+[[package]]
+name = "earcutr"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "79127ed59a85d7687c409e9978547cffb7dc79675355ed22da6b66fd5f6ead01"
+dependencies = [
+ "itertools 0.11.0",
+ "num-traits",
+]
+
 [[package]]
 name = "ecdsa"
 version = "0.14.8"
@@ -3380,6 +3390,12 @@ dependencies = [
 "rand_distr 0.5.1",
 ]

+[[package]]
+name = "float_next_after"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8bf7cc16383c4b8d58b9905a8509f02926ce3058053c056376248d958c9df1e8"
+
 [[package]]
 name = "fnv"
 version = "1.0.7"
@@ -3690,6 +3706,129 @@ dependencies = [
 "version_check",
 ]

+[[package]]
+name = "geo"
+version = "0.31.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2fc1a1678e54befc9b4bcab6cd43b8e7f834ae8ea121118b0fd8c42747675b4a"
+dependencies = [
+ "earcutr",
+ "float_next_after",
+ "geo-types",
+ "geographiclib-rs",
+ "i_overlay",
+ "log",
+ "num-traits",
+ "robust",
+ "rstar",
+ "spade",
+]
+
+[[package]]
+name = "geo-traits"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2e7c353d12a704ccfab1ba8bfb1a7fe6cb18b665bf89d37f4f7890edcd260206"
+dependencies = [
+ "geo-types",
+]
+
+[[package]]
+name = "geo-types"
+version = "0.7.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "94776032c45f950d30a13af6113c2ad5625316c9abfbccee4dd5a6695f8fe0f5"
+dependencies = [
+ "approx",
+ "num-traits",
+ "rayon",
+ "rstar",
+ "serde",
+]
+
+[[package]]
+name = "geoarrow-array"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dafe7b7de3fab1a8b7099fd6a6434ca955fa65065f9c19f0f8a133693f3c2b0e"
+dependencies = [
+ "arrow-array",
+ "arrow-buffer",
+ "arrow-schema",
+ "geo-traits",
+ "geoarrow-schema",
+ "num-traits",
+ "wkb",
+ "wkt",
+]
+
+[[package]]
+name = "geoarrow-expr-geo"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e4a62ac19c86827c6ec81ea584594b3ee96db5a8119b9774d3466c6b373c434"
+dependencies = [
+ "arrow-array",
+ "arrow-buffer",
+ "geo",
+ "geo-traits",
+ "geoarrow-array",
+ "geoarrow-schema",
+]
+
+[[package]]
+name = "geoarrow-schema"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4d4a7edb2a1d87024a93805332a9c8184a0354836271d42c0d18cf628a5e3cd0"
+dependencies = [
+ "arrow-schema",
+ "geo-traits",
+ "serde",
+ "serde_json",
+ "thiserror 1.0.69",
+]
+
+[[package]]
+name = "geodatafusion"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fecbdd00d0fff2b04635c1b1e4129c217908f0c2d17539e0a2275308afce2552"
+dependencies = [
+ "arrow-arith",
+ "arrow-array",
+ "arrow-buffer",
+ "arrow-schema",
+ "datafusion",
+ "geo",
+ "geo-traits",
+ "geoarrow-array",
+ "geoarrow-expr-geo",
+ "geoarrow-schema",
+ "geohash",
+ "thiserror 1.0.69",
+ "wkt",
+]
+
+[[package]]
+name = "geographiclib-rs"
+version = "0.2.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c5a7f08910fd98737a6eda7568e7c5e645093e073328eeef49758cfe8b0489c7"
+dependencies = [
+ "libm",
+]
+
+[[package]]
+name = "geohash"
+version = "0.13.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7f58890382f70caccc5fa388981f7ac80c913795042afce9f3e065695d8f7464"
+dependencies = [
+ "geo-types",
+ "libm",
+]
+
 [[package]]
 name = "getrandom"
 version = "0.2.17"
@@ -3842,7 +3981,7 @@ dependencies = [
 "fnv",
 "futures-core",
 "futures-sink",
- "http 1.4.2",
+ "http 1.5.0",
 "indexmap 2.14.0",
 "slab",
 "tokio",
@@ -3865,6 +4004,15 @@ dependencies = [
 "zerocopy",
 ]

+[[package]]
+name = "hash32"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606"
+dependencies = [
+ "byteorder",
+]
+
 [[package]]
 name = "hashbrown"
 version = "0.12.3"
@@ -3921,6 +4069,16 @@ version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0049b265b7f201ca9ab25475b22b47fe444060126a51abe00f77d986fc5cc52e"

+[[package]]
+name = "heapless"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad"
+dependencies = [
+ "hash32",
+ "stable_deref_trait",
+]
+
 [[package]]
 name = "heck"
 version = "0.4.1"
@@ -3953,7 +4111,7 @@ checksum = "629d8f3bbeda9d148036d6b0de0a3ab947abd08ce90626327fc3547a49d59d97"
 dependencies = [
 "dirs",
 "futures",
- "http 1.4.2",
+ "http 1.5.0",
 "indicatif",
 "libc",
 "log",
@@ -3976,7 +4134,7 @@ checksum = "430b33fa84f92796d4d263070b6c0d3ca219df7b9a0e1853ee431029b1612bcd"
 dependencies = [
 "async-trait",
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 "more-asserts",
 "serde",
 "thiserror 2.0.18",
@@ -4041,9 +4199,9 @@ dependencies = [

 [[package]]
 name = "http"
-version = "1.4.2"
+version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425"
+checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0"
 dependencies = [
 "bytes",
 "itoa",
@@ -4067,7 +4225,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c"
 dependencies = [
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 ]

 [[package]]
@@ -4078,7 +4236,7 @@ checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a"
 dependencies = [
 "bytes",
 "futures-core",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "pin-project-lite",
 ]
@@ -4145,7 +4303,7 @@ dependencies = [
 "futures-channel",
 "futures-core",
 "h2 0.4.14",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "httparse",
 "httpdate",
@@ -4177,7 +4335,7 @@ version = "0.27.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
 dependencies = [
- "http 1.4.2",
+ "http 1.5.0",
 "hyper 1.9.0",
 "hyper-util",
 "rustls 0.23.40",
@@ -4211,7 +4369,7 @@ dependencies = [
 "bytes",
 "futures-channel",
 "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "hyper 1.9.0",
 "ipnet",
@@ -4235,6 +4393,49 @@ dependencies = [
 "serde",
 ]

+[[package]]
+name = "i_float"
+version = "1.15.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "010025c2c532c8d82e42d0b8bb5184afa449fa6f06c709ea9adcb16c49ae405b"
+dependencies = [
+ "libm",
+]
+
+[[package]]
+name = "i_key_sort"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9190f86706ca38ac8add223b2aed8b1330002b5cdbbce28fb58b10914d38fc27"
+
+[[package]]
+name = "i_overlay"
+version = "4.0.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "413183068e6e0289e18d7d0a1f661b81546e6918d5453a44570b9ab30cbed1b3"
+dependencies = [
+ "i_float",
+ "i_key_sort",
+ "i_shape",
+ "i_tree",
+ "rayon",
+]
+
+[[package]]
+name = "i_shape"
+version = "1.14.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1ea154b742f7d43dae2897fcd5ead86bc7b5eefcedd305a7ebf9f69d44d61082"
+dependencies = [
+ "i_float",
+]
+
+[[package]]
+name = "i_tree"
+version = "0.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "35e6d558e6d4c7b82bc51d9c771e7a927862a161a7d87bf2b0541450e0e20915"
+
 [[package]]
 name = "iana-time-zone"
 version = "0.1.65"
@@ -4971,6 +5172,7 @@ dependencies = [
 "lance-arrow",
 "lance-core",
 "lance-datagen",
+ "lance-geo",
 "log",
 "pin-project",
 "prost",
@@ -5075,6 +5277,21 @@ dependencies = [
 "tracing",
 ]

+[[package]]
+name = "lance-geo"
+version = "10.1.0-beta.1"
+source = "git+https://github.com/lance-format/lance.git?tag=v10.1.0-beta.1#68f4d4c1d0c4871b067557c61fc405078f1ab3b7"
+dependencies = [
+ "datafusion",
+ "geo-traits",
+ "geo-types",
+ "geoarrow-array",
+ "geoarrow-schema",
+ "geodatafusion",
+ "lance-core",
+ "serde",
+]
+
 [[package]]
 name = "lance-index"
 version = "10.1.0-beta.1"
@@ -5102,6 +5319,9 @@ dependencies = [
 "dirs",
 "fst",
 "futures",
+ "geo-types",
+ "geoarrow-array",
+ "geoarrow-schema",
 "half",
 "itertools 0.14.0",
 "jieba-rs",
@@ -5114,6 +5334,7 @@ dependencies = [
 "lance-datagen",
 "lance-encoding",
 "lance-file",
+ "lance-geo",
 "lance-index-core",
 "lance-io",
 "lance-linalg",
@@ -5182,7 +5403,7 @@ dependencies = [
 "chrono",
 "futures",
 "goosefs-sdk",
- "http 1.4.2",
+ "http 1.5.0",
 "io-uring",
 "lance-arrow",
 "lance-core",
@@ -5414,7 +5635,7 @@ dependencies = [
 "goosefs-sdk",
 "half",
 "hf-hub",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "lance",
 "lance-arrow",
@@ -6080,15 +6301,15 @@ dependencies = [

 [[package]]
 name = "napi-build"
-version = "2.3.2"
+version = "2.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c9c366d2c8c60b86fa632df75f745509b52f9128f91a6bad4c796e44abb505e1"
+checksum = "5282704fbe8d49b0cf8b08e3f33233416a528658f205c7e5ace63b582de0b11c"

 [[package]]
 name = "napi-derive"
-version = "3.6.0"
+version = "3.6.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a49c513341a61a16a10af6efcce46b30d0822ba2d4fb197d24d33dfc199c78d5"
+checksum = "4d5c9c02556ea6dc99dffd36c1ce60141411657438501a125b675776d011ce92"
 dependencies = [
 "convert_case",
 "ctor 1.0.5",
@@ -6100,9 +6321,9 @@ dependencies = [

 [[package]]
 name = "napi-derive-backend"
-version = "6.0.0"
+version = "6.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4747005fa3e2c9989ac45a723a514c5db2411238b72981a3cda4c701a9dfea17"
+checksum = "d60b5d773ad46c698c8cc2cd9fde0b283d39cbb7f71c04bee633c7bdba4423bd"
 dependencies = [
 "convert_case",
 "proc-macro2",
@@ -6299,6 +6520,28 @@ dependencies = [
 "libc",
 ]

+[[package]]
+name = "num_enum"
+version = "0.7.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5d0bca838442ec211fa11de3a8b0e0e8f3a4522575b5c4c06ed722e005036f26"
+dependencies = [
+ "num_enum_derive",
+ "rustversion",
+]
+
+[[package]]
+name = "num_enum_derive"
+version = "0.7.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8"
+dependencies = [
+ "proc-macro-crate",
+ "proc-macro2",
+ "quote",
+ "syn 2.0.117",
+]
+
 [[package]]
 name = "number_prefix"
 version = "0.4.0"
@@ -6356,7 +6599,7 @@ dependencies = [
 "futures-channel",
 "futures-core",
 "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body-util",
 "httparse",
 "humantime",
@@ -6477,7 +6720,7 @@ dependencies = [
 "base64 0.22.1",
 "bytes",
 "futures",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "jiff",
 "log",
@@ -6502,7 +6745,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0d6f81ba6960e3fae1882f253b114b21d7e444e1534f209c7737a79f6243eb6f"
 dependencies = [
 "futures",
- "http 1.4.2",
+ "http 1.5.0",
 "mea",
 "opendal-core",
 ]
@@ -6546,7 +6789,7 @@ checksum = "0030644366ef5d8cbe3a4a5822bf99a4aafddc1666e9d24b44d158d9062fc76a"
 dependencies = [
 "base64 0.22.1",
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "opendal-core",
 "opendal-service-azure-common",
@@ -6567,7 +6810,7 @@ checksum = "6dea4908d490143a9b0b7f7a790e139ff829b06a023f670455ed3d44f664b361"
 dependencies = [
 "base64 0.22.1",
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "opendal-core",
 "opendal-service-azure-common",
@@ -6585,7 +6828,7 @@ version = "0.57.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "9b489f13c42e69d69bdd72952b634356ec43a7881a20259b38b540fcecdf4051"
 dependencies = [
- "http 1.4.2",
+ "http 1.5.0",
 "opendal-core",
 ]

@@ -6596,7 +6839,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "aa8cafe9729213375c7331019b0cb756ad3e1aff7f45cd32c45eae91ebde8901"
 dependencies = [
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "opendal-core",
 "quick-xml 0.39.4",
@@ -6614,7 +6857,7 @@ checksum = "48de101aac565ed06af4b47903c24eafd249075553ec1fb18256751c45148d47"
 dependencies = [
 "async-trait",
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "opendal-core",
 "percent-encoding",
@@ -6649,7 +6892,7 @@ checksum = "c4922661976a1d40794a2adfbdb888cc3c23097690f825a92f773af38908a848"
 dependencies = [
 "bytes",
 "hf-xet",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "opendal-core",
 "percent-encoding",
@@ -6665,7 +6908,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "328fa55e8888cbdfe00826bfea2a79042422b720e8369e9e021e46121dea5ace"
 dependencies = [
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "opendal-core",
 "quick-xml 0.39.4",
@@ -6684,7 +6927,7 @@ dependencies = [
 "base64 0.22.1",
 "bytes",
 "crc32c",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "md-5 0.11.0",
 "opendal-core",
@@ -7583,7 +7826,7 @@ version = "0.14.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "343d3bd7056eda839b03204e68deff7d1b13aba7af2b2fd16890697274262ee7"
 dependencies = [
- "heck 0.5.0",
+ "heck 0.4.1",
 "itertools 0.14.0",
 "log",
 "multimap",
@@ -8226,7 +8469,7 @@ checksum = "57ac2757f3140aa2e213b554148ae0b52733e624fc6723f0cc6bb3d440176c95"
 dependencies = [
 "anyhow",
 "form_urlencoded",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "percent-encoding",
 "reqsign-core",
@@ -8244,7 +8487,7 @@ dependencies = [
 "anyhow",
 "bytes",
 "form_urlencoded",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "percent-encoding",
 "quick-xml 0.39.4",
@@ -8266,7 +8509,7 @@ dependencies = [
 "base64 0.22.1",
 "bytes",
 "form_urlencoded",
- "http 1.4.2",
+ "http 1.5.0",
 "jsonwebtoken",
 "log",
 "pem",
@@ -8291,7 +8534,7 @@ dependencies = [
 "futures",
 "hex",
 "hmac 0.12.1",
- "http 1.4.2",
+ "http 1.5.0",
 "jiff",
 "log",
 "percent-encoding",
@@ -8318,7 +8561,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "35cc609b49c69e76ecaceb775a03f792d1ed3e7755ab3548d4534fd801e3242e"
 dependencies = [
 "form_urlencoded",
- "http 1.4.2",
+ "http 1.5.0",
 "jsonwebtoken",
 "log",
 "percent-encoding",
@@ -8338,7 +8581,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e128f19525861dbded59e1e7c17653a8ed63d573ca04aed708d552dbef5bb32a"
 dependencies = [
 "anyhow",
- "http 1.4.2",
+ "http 1.5.0",
 "log",
 "percent-encoding",
 "reqsign-core",
@@ -8358,7 +8601,7 @@ dependencies = [
 "futures-core",
 "futures-util",
 "h2 0.4.14",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "hyper 1.9.0",
@@ -8402,7 +8645,7 @@ dependencies = [
 "bytes",
 "futures-core",
 "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "hyper 1.9.0",
@@ -8456,7 +8699,7 @@ checksum = "199dda04a536b532d0cc04d7979e39b1c763ea749bf91507017069c00b96056f"
 dependencies = [
 "anyhow",
 "async-trait",
- "http 1.4.2",
+ "http 1.5.0",
 "reqwest 0.13.3",
 "thiserror 2.0.18",
 "tower-service",
@@ -8498,9 +8741,9 @@ dependencies = [

 [[package]]
 name = "rkyv"
-version = "0.8.16"
+version = "0.8.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "73389e0c99e664f919275ab5b5b0471391fe9a8de61e1dff9b1eaf56a90f16e3"
+checksum = "815cc8a37159a463064825246cadb07961e25cd9885908606f6d08a98d8f8874"
 dependencies = [
 "bytecheck",
 "bytes",
@@ -8517,9 +8760,9 @@ dependencies = [

 [[package]]
 name = "rkyv_derive"
-version = "0.8.16"
+version = "0.8.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5d2ed0b54125315fb36bd021e82d314d1c126548f871634b483f46b31d13cac6"
+checksum = "c0ed1a78a1b19d184b0daa629dd9a024573173ec7d485b287cb369fb3607cc1c"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -8536,6 +8779,12 @@ dependencies = [
 "byteorder",
 ]

+[[package]]
+name = "robust"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4e27ee8bb91ca0adcf0ecb116293afa12d393f9c2b9b9cd54d33e8078fe19839"
+
 [[package]]
 name = "rsa"
 version = "0.9.10"
@@ -8557,6 +8806,17 @@ dependencies = [
 "zeroize",
 ]

+[[package]]
+name = "rstar"
+version = "0.12.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "421400d13ccfd26dfa5858199c30a5d76f9c54e0dba7575273025b43c5175dbb"
+dependencies = [
+ "heapless",
+ "num-traits",
+ "smallvec",
+]
+
 [[package]]
 name = "rstest"
 version = "0.23.0"
@@ -9277,7 +9537,7 @@ version = "0.8.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451"
 dependencies = [
- "heck 0.5.0",
+ "heck 0.4.1",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -9289,7 +9549,7 @@ version = "0.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "54254b8531cafa275c5e096f62d48c81435d1015405a91198ddb11e967301d40"
 dependencies = [
- "heck 0.5.0",
+ "heck 0.4.1",
 "proc-macro2",
 "quote",
 "syn 2.0.117",
@@ -9326,6 +9586,18 @@ dependencies = [
 "winapi",
 ]

+[[package]]
+name = "spade"
+version = "2.15.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9699399fd9349b00b184f5635b074f9ec93afffef30c853f8c875b32c0f8c7fa"
+dependencies = [
+ "hashbrown 0.16.1",
+ "num-traits",
+ "robust",
+ "smallvec",
+]
+
 [[package]]
 name = "spin"
 version = "0.9.9"
@@ -9730,7 +10002,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
 dependencies = [
 "fastrand",
- "getrandom 0.4.2",
+ "getrandom 0.3.4",
 "once_cell",
 "rustix",
 "windows-sys 0.61.2",
@@ -10058,7 +10330,7 @@ dependencies = [
 "base64 0.22.1",
 "bytes",
 "h2 0.4.14",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "hyper 1.9.0",
@@ -10114,7 +10386,7 @@ checksum = "1e9cd434a998747dd2c4276bc96ee2e0c7a2eadf3cae88e52be55a05fa9053f5"
 dependencies = [
 "bitflags 2.11.1",
 "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "pin-project-lite",
@@ -10134,7 +10406,7 @@ dependencies = [
 "bytes",
 "futures-core",
 "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
 "pin-project-lite",
@@ -11127,6 +11399,31 @@ dependencies = [
 "wasmparser",
 ]

+[[package]]
+name = "wkb"
+version = "0.9.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a120b336c7ad17749026d50427c23d838ecb50cd64aaea6254b5030152f890a9"
+dependencies = [
+ "byteorder",
+ "geo-traits",
+ "num_enum",
+ "thiserror 1.0.69",
+]
+
+[[package]]
+name = "wkt"
+version = "0.14.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "efb2b923ccc882312e559ffaa832a055ba9d1ac0cc8e86b3e25453247e4b81d7"
+dependencies = [
+ "geo-traits",
+ "geo-types",
+ "log",
+ "num-traits",
+ "thiserror 1.0.69",
+]
+
 [[package]]
 name = "writeable"
 version = "0.6.3"
@@ -11155,7 +11452,7 @@ dependencies = [
 "clap",
 "crc32fast",
 "futures",
- "http 1.4.2",
+ "http 1.5.0",
 "hyper 1.9.0",
 "lazy_static",
 "more-asserts",
@@ -11229,7 +11526,7 @@ dependencies = [
 "chrono",
 "clap",
 "gearhash",
- "http 1.4.2",
+ "http 1.5.0",
 "itertools 0.14.0",
 "lazy_static",
 "more-asserts",
@@ -13,7 +13,7 @@ categories = ["database-implementations"]
 rust-version = "1.91.0"

 [workspace.dependencies]
-lance = { "version" = "=10.1.0-beta.1", default-features = false, "tag" = "v10.1.0-beta.1", "git" = "https://github.com/lance-format/lance.git" }
+lance = { "version" = "=10.1.0-beta.1", default-features = false, features = ["geo"], "tag" = "v10.1.0-beta.1", "git" = "https://github.com/lance-format/lance.git" }
 lance-core = { "version" = "=10.1.0-beta.1", "tag" = "v10.1.0-beta.1", "git" = "https://github.com/lance-format/lance.git" }
 lance-datagen = { "version" = "=10.1.0-beta.1", "tag" = "v10.1.0-beta.1", "git" = "https://github.com/lance-format/lance.git" }
 lance-file = { "version" = "=10.1.0-beta.1", "tag" = "v10.1.0-beta.1", "git" = "https://github.com/lance-format/lance.git" }
@@ -339,7 +339,9 @@ impl Table {
        let transforms = NewColumnTransform::SqlExpressions(transforms);
        let res = self
            .inner_ref()?
-            .add_columns(transforms, None)
+            .add_columns()
+            .transform(transforms)
+            .execute()
            .await
            .default_error()?;
        Ok(res.into())
@@ -356,7 +358,9 @@ impl Table {
        let transforms = NewColumnTransform::AllNulls(schema);
        let res = self
            .inner_ref()?
-            .add_columns(transforms, None)
+            .add_columns()
+            .transform(transforms)
+            .execute()
            .await
            .default_error()?;
        Ok(res.into())
@@ -92,8 +92,10 @@ class LanceMergeInsertBuilder(object):
        self._when_not_matched_by_source_delete = True
        if isinstance(condition, Expr):
            self._when_not_matched_by_source_condition_expr = condition._inner
-        elif condition is not None:
+            self._when_not_matched_by_source_condition = None
+        else:
            self._when_not_matched_by_source_condition = condition
+            self._when_not_matched_by_source_condition_expr = None
        return self

    def use_index(self, use_index: bool) -> LanceMergeInsertBuilder:
@@ -2697,7 +2697,7 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
            self._fts_query.phrase_query(True)
        if self._distance_type:
            self._vector_query.metric(self._distance_type)
-        if self._minimum_nprobes:
+        if self._minimum_nprobes is not None:
            self._vector_query.minimum_nprobes(self._minimum_nprobes)
        if self._maximum_nprobes is not None:
            self._vector_query.maximum_nprobes(self._maximum_nprobes)
@@ -1595,8 +1595,8 @@ class Table(ABC):
        Prefer this over :meth:`fetch_blobs` for large payloads. ``row_ids`` is
        a ``list[int]`` or a query ``pyarrow.Table`` carrying row identity via
        ``_rowid`` or a ``_lance_row_id`` field on the blob descriptor. Null
-        rows are ``None``. Unsupported on LanceDB Cloud, where
-        :meth:`fetch_blobs` returns full bytes instead.
+        rows are ``None``. Remote tables require LanceDB Cloud server 0.5.0 or
+        newer.
        """

    @abstractmethod
@@ -123,6 +123,19 @@ async def test_async_hybrid_query_default_limit(table: AsyncTable):
    assert texts.count("a") == 1


+def test_hybrid_query_minimum_nprobes_zero_raises(sync_table: Table):
+    # minimum_nprobes(0) must raise the same validation error a plain vector
+    # query raises, not silently no-op because 0 is falsy.
+    with pytest.raises(ValueError, match="minimum_nprobes must be greater than 0"):
+        (
+            sync_table.search(query_type="hybrid")
+            .vector([0.0, 0.4])
+            .text("dog")
+            .minimum_nprobes(0)
+            .to_arrow()
+        )
+
+
 def test_hybrid_query_distance_range(sync_table: Table):
    reranker = RRFReranker(return_score="all")
    result = (
@@ -2055,6 +2055,24 @@ def blob_remote_table(*, server_version=Version("0.5.0")):
            request.send_header("phalanx-version", str(server_version))
            request.end_headers()
            request.wfile.write(json.dumps(BLOB_DESCRIBE_RESPONSE).encode())
+        elif request.path.startswith("/v1/table/test/blob/image/"):
+            path = request.path.partition("?")[0]
+            row_id = int(path.split("/")[-2])
+            payload = {10: b"alpha", 20: None, 30: b"gamma"}[row_id]
+            if payload is None:
+                request.send_response(204)
+                request.end_headers()
+                return
+            byte_range = request.headers["Range"].removeprefix("bytes=")
+            start_text, end_text = byte_range.split("-", maxsplit=1)
+            start = int(start_text)
+            end = int(end_text) if end_text else len(payload) - 1
+            chunk = payload[start : end + 1]
+            request.send_response(206)
+            request.send_header("Content-Range", f"bytes {start}-{end}/{len(payload)}")
+            request.send_header("Content-Length", str(len(chunk)))
+            request.end_headers()
+            request.wfile.write(chunk)
        elif request.path == "/v1/table/test/query/":
            content_len = int(request.headers.get("Content-Length", 0))
            body = json.loads(request.rfile.read(content_len))
@@ -2092,8 +2110,21 @@ def test_remote_blob_columns_and_fetch():
        assert table.blob_columns() == ["image"]
        blobs = table.fetch_blobs("image", [10, 20, 30])
        assert blobs.to_pylist() == [b"alpha", None, b"gamma"]
-        with pytest.raises(NotImplementedError, match="Use fetch_blobs for full bytes"):
-            table.fetch_blob_files("image", [10, 20, 30])
+
+
+def test_remote_blob_files_are_lazy_seekable_handles():
+    with blob_remote_table() as table:
+        files = table.fetch_blob_files("image", [10, 20, 30])
+
+        assert len(files) == 3
+        alpha, null_row, gamma = files
+        assert null_row is None
+        assert alpha is not None
+        assert gamma is not None
+        assert alpha.size() == 5
+        assert alpha.read_range(1, 3) == b"lph"
+        gamma.seek(2)
+        assert gamma.read() == b"mma"


 def test_remote_blob_fetch_accepts_query_table():
@@ -2364,6 +2364,29 @@ def test_merge_insert_by_source_delete_expr(mem_db: DBConnection):
    assert table.to_arrow().sort_by("a") == expected


+def test_merge_insert_by_source_delete_reconfigure(mem_db: DBConnection):
+    # Calling when_not_matched_by_source_delete() again with no condition must
+    # widen the delete to unconditional, not keep the earlier condition around.
+    table = mem_db.create_table(
+        "my_table",
+        data=pa.table({"a": [1, 2, 3], "b": ["a", "b", "c"]}),
+    )
+    new_data = pa.table({"a": [2, 4], "b": ["x", "z"]})
+
+    merge_insert_res = (
+        table.merge_insert("a")
+        .when_matched_update_all()
+        .when_not_matched_insert_all()
+        .when_not_matched_by_source_delete("a > 2")
+        .when_not_matched_by_source_delete()
+        .execute(new_data)
+    )
+    assert merge_insert_res.num_deleted_rows == 2
+
+    expected = pa.table({"a": [2, 4], "b": ["x", "z"]})
+    assert table.to_arrow().sort_by("a") == expected
+
+
@pytest.mark.asyncio
 async def test_merge_insert_by_source_delete_expr_async(
    mem_db_async: AsyncConnection,
@@ -426,9 +426,11 @@ pub struct PyBlobFile {
 impl PyBlobFile {
    fn read_bytes(self_: PyRef<'_, Self>) -> PyResult<Py<PyBytes>> {
        let inner = self_.inner.clone();
-        let bytes = block_on(async move { inner.read().await })
+        let py = self_.py();
+        let bytes = py
+            .detach(move || block_on(async move { inner.read().await }))
            .map_err(|e| PyRuntimeError::new_err(format!("blob read failed: {e}")))?;
-        Ok(PyBytes::new(self_.py(), bytes.as_ref()).unbind())
+        Ok(PyBytes::new(py, bytes.as_ref()).unbind())
    }

    pub fn read(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
@@ -444,24 +446,32 @@ impl PyBlobFile {

    fn close(self_: PyRef<'_, Self>) -> PyResult<()> {
        let inner = self_.inner.clone();
-        block_on(async move { inner.close().await })
+        self_
+            .py()
+            .detach(move || block_on(async move { inner.close().await }))
            .map_err(|e| PyRuntimeError::new_err(format!("blob close failed: {e}")))
    }

    fn is_closed(self_: PyRef<'_, Self>) -> bool {
        let inner = self_.inner.clone();
-        block_on(async move { inner.is_closed().await })
+        self_
+            .py()
+            .detach(move || block_on(async move { inner.is_closed().await }))
    }

    fn seek(self_: PyRef<'_, Self>, position: u64) -> PyResult<()> {
        let inner = self_.inner.clone();
-        block_on(async move { inner.seek(position).await })
+        self_
+            .py()
+            .detach(move || block_on(async move { inner.seek(position).await }))
            .map_err(|e| PyRuntimeError::new_err(format!("blob seek failed: {e}")))
    }

    fn tell(self_: PyRef<'_, Self>) -> PyResult<u64> {
        let inner = self_.inner.clone();
-        block_on(async move { inner.tell().await })
+        self_
+            .py()
+            .detach(move || block_on(async move { inner.tell().await }))
            .map_err(|e| PyRuntimeError::new_err(format!("blob tell failed: {e}")))
    }

@@ -475,16 +485,20 @@ impl PyBlobFile {
            .checked_add(length as u64)
            .ok_or_else(|| PyValueError::new_err("offset + length overflowed"))?;
        let inner = self_.inner.clone();
-        let bytes = block_on(async move { inner.read_range(offset..end).await })
+        let py = self_.py();
+        let bytes = py
+            .detach(move || block_on(async move { inner.read_range(offset..end).await }))
            .map_err(|e| PyRuntimeError::new_err(format!("blob read_range failed: {e}")))?;
-        Ok(PyBytes::new(self_.py(), bytes.as_ref()).unbind())
+        Ok(PyBytes::new(py, bytes.as_ref()).unbind())
    }

    fn read_up_to(self_: PyRef<'_, Self>, length: usize) -> PyResult<Py<PyBytes>> {
        let inner = self_.inner.clone();
-        let bytes = block_on(async move { inner.read_up_to(length).await })
-            .map_err(|e| PyRuntimeError::new_err(format!("blob read failed: {e}")))?;
-        Ok(PyBytes::new(self_.py(), bytes.as_ref()).unbind())
+        let py = self_.py();
+        let bytes = py
+            .detach(move || block_on(async move { inner.read_up_to(length).await }))
+            .map_err(|e| PyRuntimeError::new_err(format!("blob read_up_to failed: {e}")))?;
+        Ok(PyBytes::new(py, bytes.as_ref()).unbind())
    }
 }

@@ -1361,7 +1375,12 @@ impl Table {

        let inner = self_.inner_ref()?.clone();
        future_into_py(self_.py(), async move {
-            let result = inner.add_columns(definitions, None).await.infer_error()?;
+            let result = inner
+                .add_columns()
+                .transform(definitions)
+                .execute()
+                .await
+                .infer_error()?;
            Ok(AddColumnsResult::from(result))
        })
    }
@@ -1375,7 +1394,12 @@ impl Table {

        let inner = self_.inner_ref()?.clone();
        future_into_py(self_.py(), async move {
-            let result = inner.add_columns(transform, None).await.infer_error()?;
+            let result = inner
+                .add_columns()
+                .transform(transform)
+                .execute()
+                .await
+                .infer_error()?;
            Ok(AddColumnsResult::from(result))
        })
    }
@@ -9,6 +9,7 @@
 //!
 //! Blob tables require Lance file format >= 2.2 and stable row ids at create.

+use std::ops::Range;
 use std::sync::Arc;

 use arrow_array::LargeBinaryArray;
@@ -17,10 +18,202 @@ use arrow_schema::{DataType, Field, Schema};
 use lance::dataset::{BlobRangeRequest as LanceBlobRangeRequest, Dataset, WriteParams};
 use lance_arrow::FieldExt;
 use lance_encoding::version::LanceFileVersion;
+use lance_io::object_store::ObjectStore;
+use object_store::path::Path;

 use crate::error::{Error, Result};

-pub use lance::dataset::BlobFile;
+/// Seekable handle for one blob value, backed by local storage or a remote
+/// HTTP byte-range endpoint.
+#[derive(Debug)]
+pub struct BlobFile {
+    inner: BlobFileInner,
+}
+
+#[derive(Debug)]
+enum BlobFileInner {
+    Native(lance::dataset::BlobFile),
+    #[cfg(feature = "remote")]
+    Remote(Box<crate::remote::table::blobs::RemoteBlobFile>),
+}
+
+impl From<lance::dataset::BlobFile> for BlobFile {
+    fn from(value: lance::dataset::BlobFile) -> Self {
+        Self {
+            inner: BlobFileInner::Native(value),
+        }
+    }
+}
+
+#[cfg(feature = "remote")]
+impl From<crate::remote::table::blobs::RemoteBlobFile> for BlobFile {
+    fn from(value: crate::remote::table::blobs::RemoteBlobFile) -> Self {
+        Self {
+            inner: BlobFileInner::Remote(Box::new(value)),
+        }
+    }
+}
+
+impl BlobFile {
+    /// Inline reader over a data-file slice.
+    pub fn new_inline(
+        object_store: Arc<ObjectStore>,
+        path: Path,
+        position: u64,
+        size: u64,
+    ) -> Self {
+        lance::dataset::BlobFile::new_inline(object_store, path, position, size).into()
+    }
+
+    /// Dedicated sidecar-file reader.
+    pub fn new_dedicated(object_store: Arc<ObjectStore>, path: Path, size: u64) -> Self {
+        lance::dataset::BlobFile::new_dedicated(object_store, path, size).into()
+    }
+
+    /// Packed reader for a slice in a shared sidecar.
+    pub fn new_packed(
+        object_store: Arc<ObjectStore>,
+        path: Path,
+        position: u64,
+        size: u64,
+    ) -> Self {
+        lance::dataset::BlobFile::new_packed(object_store, path, position, size).into()
+    }
+
+    /// External reader at a resolved object location.
+    pub fn new_external(
+        object_store: Arc<ObjectStore>,
+        path: Path,
+        uri: String,
+        position: u64,
+        size: u64,
+    ) -> Self {
+        lance::dataset::BlobFile::new_external(object_store, path, uri, position, size).into()
+    }
+
+    /// Close the handle.
+    pub async fn close(&self) -> lance_core::Result<()> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.close().await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.close().await,
+        }
+    }
+
+    /// Whether the handle is closed.
+    pub async fn is_closed(&self) -> bool {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.is_closed().await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.is_closed(),
+        }
+    }
+
+    /// Read a range without moving the cursor.
+    pub async fn read_range(&self, range: Range<u64>) -> lance_core::Result<bytes::Bytes> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.read_range(range).await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.read_range(range).await,
+        }
+    }
+
+    /// Read ranges without moving the cursor.
+    pub async fn read_ranges(
+        &self,
+        ranges: &[Range<u64>],
+    ) -> lance_core::Result<Vec<bytes::Bytes>> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.read_ranges(ranges).await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.read_ranges(ranges).await,
+        }
+    }
+
+    /// Read from the cursor to the end.
+    pub async fn read(&self) -> lance_core::Result<bytes::Bytes> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.read().await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.read().await,
+        }
+    }
+
+    /// Read up to `len` bytes and advance the cursor.
+    pub async fn read_up_to(&self, len: usize) -> lance_core::Result<bytes::Bytes> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.read_up_to(len).await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.read_up_to(len).await,
+        }
+    }
+
+    /// Move the cursor to `new_cursor`.
+    pub async fn seek(&self, new_cursor: u64) -> lance_core::Result<()> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.seek(new_cursor).await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.seek(new_cursor).await,
+        }
+    }
+
+    /// Current cursor position.
+    pub async fn tell(&self) -> lance_core::Result<u64> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.tell().await,
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.tell().await,
+        }
+    }
+
+    /// Blob length in bytes.
+    pub fn size(&self) -> u64 {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.size(),
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(file) => file.size(),
+        }
+    }
+
+    /// Physical byte offset in the data file. `None` on remote handles. The
+    /// Cloud byte-range route does not expose storage layout.
+    pub fn position(&self) -> Option<u64> {
+        match &self.inner {
+            BlobFileInner::Native(file) => Some(file.position()),
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(_) => None,
+        }
+    }
+
+    /// Path of the data file holding the blob. `None` on remote handles. The
+    /// Cloud byte-range route does not expose storage layout.
+    pub fn data_path(&self) -> Option<&Path> {
+        match &self.inner {
+            BlobFileInner::Native(file) => Some(file.data_path()),
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(_) => None,
+        }
+    }
+
+    /// Native storage layout. `None` on remote handles. The Cloud byte-range
+    /// route does not expose layout.
+    pub fn kind(&self) -> Option<lance_core::datatypes::BlobKind> {
+        match &self.inner {
+            BlobFileInner::Native(file) => Some(file.kind()),
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(_) => None,
+        }
+    }
+
+    /// External URI for native handles. Remote handles do not expose storage URIs.
+    pub fn uri(&self) -> Option<&str> {
+        match &self.inner {
+            BlobFileInner::Native(file) => file.uri(),
+            #[cfg(feature = "remote")]
+            BlobFileInner::Remote(_) => None,
+        }
+    }
+}

 /// One row-specific blob range read request.
 ///
@@ -264,7 +457,10 @@ pub(crate) async fn take_blob_files_aligned(

    let handles = dataset.take_blobs(row_ids, column).await?;
    ensure_all_row_ids_resolved(column, row_ids.len(), handles.len())?;
-    Ok(handles)
+    Ok(handles
+        .into_iter()
+        .map(|handle| handle.map(Into::into))
+        .collect())
 }

 #[cfg(test)]
@@ -32,6 +32,7 @@ use crate::{
 };

 mod hybrid;
+mod sql;

 pub(crate) const DEFAULT_TOP_K: usize = 10;

@@ -561,8 +562,8 @@ impl<T: HasQuery> QueryBase for T {
    }

    fn only_if(mut self, filter: impl AsRef<str>) -> Self {
-        self.mut_query()
-            .add_filter(QueryFilter::Sql(filter.as_ref().to_string()));
+        let filter = sql::rewrite_st_dwithin(filter.as_ref());
+        self.mut_query().add_filter(QueryFilter::Sql(filter));
        self
    }

@@ -1648,8 +1649,8 @@ mod tests {
    use super::*;
    use arrow::{array::downcast_array, compute::concat_batches, datatypes::Int32Type};
    use arrow_array::{
-        FixedSizeListArray, Float32Array, Int32Array, RecordBatch, StringArray, cast::AsArray,
-        types::Float32Type,
+        FixedSizeListArray, Float32Array, Float64Array, Int32Array, RecordBatch, StringArray,
+        cast::AsArray, types::Float32Type,
    };
    use arrow_schema::{DataType, Field as ArrowField, Schema as ArrowSchema};
    use futures::{StreamExt, TryStreamExt};
@@ -2024,6 +2025,46 @@ mod tests {
        assert!(result.is_err());
    }

+    #[tokio::test]
+    async fn test_st_dwithin_filter() {
+        let tmp_dir = tempdir().unwrap();
+        let uri = tmp_dir.path().to_str().unwrap();
+        let batch = RecordBatch::try_from_iter([
+            ("id", Arc::new(Int32Array::from_iter_values([1, 2, 3])) as _),
+            (
+                "x",
+                Arc::new(Float64Array::from_iter_values([0.0, 1.0, 3.0])) as _,
+            ),
+            (
+                "y",
+                Arc::new(Float64Array::from_iter_values([0.0, 1.0, 4.0])) as _,
+            ),
+        ])
+        .unwrap();
+
+        let table = connect(uri)
+            .execute()
+            .await
+            .unwrap()
+            .create_table("points", batch)
+            .execute()
+            .await
+            .unwrap();
+        let batches = table
+            .query()
+            .only_if("ST_DWithin(ST_Point(x, y), ST_Point(0.0, 0.0), 2.0)")
+            .execute()
+            .await
+            .unwrap()
+            .try_collect::<Vec<_>>()
+            .await
+            .unwrap();
+        let batch = concat_batches(&batches[0].schema(), &batches).unwrap();
+        let ids = batch["id"].as_primitive::<Int32Type>();
+
+        assert_eq!(ids.values(), &[1, 2]);
+    }
+
    fn make_non_empty_batches() -> Box<dyn arrow_array::RecordBatchReader + Send> {
        let vec = Box::new(RandomVector::new().named("vector".to_string()));
        let id = Box::new(IncrementingInt32::new().named("id".to_string()));
@@ -0,0 +1,150 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+//! SQL compatibility helpers for query filters.
+
+use std::{any::TypeId, ops::ControlFlow};
+
+use datafusion_sql::sqlparser::{
+    ast::{
+        BinaryOperator, Expr, FunctionArg, FunctionArgExpr, FunctionArguments, Ident, ObjectName,
+        SelectItem, SetExpr, Statement, visit_expressions_mut,
+    },
+    dialect::{Dialect, GenericDialect},
+    parser::Parser,
+    tokenizer::{Token, Tokenizer},
+};
+
+#[derive(Debug, Default)]
+struct LanceDialect(GenericDialect);
+
+impl Dialect for LanceDialect {
+    fn dialect(&self) -> TypeId {
+        self.0.dialect()
+    }
+
+    fn is_identifier_start(&self, ch: char) -> bool {
+        self.0.is_identifier_start(ch)
+    }
+
+    fn is_identifier_part(&self, ch: char) -> bool {
+        self.0.is_identifier_part(ch)
+    }
+
+    fn is_delimited_identifier_start(&self, ch: char) -> bool {
+        ch == '`'
+    }
+}
+
+/// Rewrite `ST_DWithin(a, b, distance)` as `ST_Distance(a, b) <= distance`.
+///
+/// GeoDataFusion does not currently expose `ST_DWithin`, but its definition is
+/// exactly this distance comparison. Invalid SQL and unsupported function
+/// shapes are left untouched so the query planner can report the usual error.
+pub(super) fn rewrite_st_dwithin(filter: &str) -> String {
+    if !filter.to_ascii_lowercase().contains("st_dwithin") {
+        return filter.to_string();
+    }
+
+    let Some(mut expr) = parse_filter(filter) else {
+        return filter.to_string();
+    };
+    let mut rewritten = false;
+    let _ = visit_expressions_mut(&mut expr, |expr| {
+        let Expr::Function(function) = expr else {
+            return ControlFlow::<()>::Continue(());
+        };
+        if !function.name.to_string().eq_ignore_ascii_case("st_dwithin") {
+            return ControlFlow::Continue(());
+        }
+        let FunctionArguments::List(arguments) = &function.args else {
+            return ControlFlow::Continue(());
+        };
+        let [
+            FunctionArg::Unnamed(FunctionArgExpr::Expr(first)),
+            FunctionArg::Unnamed(FunctionArgExpr::Expr(second)),
+            FunctionArg::Unnamed(FunctionArgExpr::Expr(distance)),
+        ] = arguments.args.as_slice()
+        else {
+            return ControlFlow::Continue(());
+        };
+
+        let mut st_distance = function.clone();
+        st_distance.name = ObjectName::from(vec![Ident::new("ST_Distance")]);
+        if let FunctionArguments::List(arguments) = &mut st_distance.args {
+            arguments.args = vec![
+                FunctionArg::Unnamed(FunctionArgExpr::Expr(first.clone())),
+                FunctionArg::Unnamed(FunctionArgExpr::Expr(second.clone())),
+            ];
+        }
+        *expr = Expr::BinaryOp {
+            left: Box::new(Expr::Function(st_distance)),
+            op: BinaryOperator::LtEq,
+            right: Box::new(distance.clone()),
+        };
+        rewritten = true;
+        ControlFlow::Continue(())
+    });
+
+    if rewritten {
+        expr.to_string()
+    } else {
+        filter.to_string()
+    }
+}
+
+fn parse_filter(filter: &str) -> Option<Expr> {
+    let statement = format!("SELECT 1 FROM t WHERE {filter}");
+    let dialect = LanceDialect::default();
+    let mut token_iter = Tokenizer::new(&dialect, &statement)
+        .tokenize()
+        .ok()?
+        .into_iter();
+    let mut previous = token_iter.next()?;
+    let mut tokens = Vec::new();
+
+    // Match Lance's support for `==` as an equality operator.
+    for next in token_iter {
+        if let (Token::Eq, Token::Eq) = (&previous, &next) {
+            continue;
+        }
+        tokens.push(std::mem::replace(&mut previous, next));
+    }
+    tokens.push(previous);
+
+    let statement = Parser::new(&dialect)
+        .with_tokens(tokens)
+        .parse_statement()
+        .ok()?;
+    if let Statement::Query(query) = statement
+        && let SetExpr::Select(select) = *query.body
+        && let Some(expr) = select.selection
+        && matches!(select.projection.as_slice(), [SelectItem::UnnamedExpr(_)])
+    {
+        Some(expr)
+    } else {
+        None
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn rewrites_nested_and_case_insensitive_dwithin() {
+        assert_eq!(
+            rewrite_st_dwithin("id == 1 AND st_dwithin(ST_Point(x, y), ST_Point(0, 0), radius)"),
+            "id = 1 AND ST_Distance(ST_Point(x, y), ST_Point(0, 0)) <= radius"
+        );
+    }
+
+    #[test]
+    fn leaves_other_filters_unchanged() {
+        assert_eq!(rewrite_st_dwithin("id == 1"), "id == 1");
+        assert_eq!(
+            rewrite_st_dwithin("ST_DWithin(point, origin)"),
+            "ST_DWithin(point, origin)"
+        );
+    }
+}
@@ -1,7 +1,7 @@
 // SPDX-License-Identifier: Apache-2.0
 // SPDX-FileCopyrightText: Copyright The LanceDB Authors

-mod blobs;
+pub mod blobs;
 pub mod insert;

 use self::insert::{RemoteWriteExec, WriteOp};
@@ -3089,10 +3089,12 @@ mod tests {
            Box::pin(table.delete("false").map_ok(|_| ())),
            Box::pin(
                table
-                    .add_columns(
-                        NewColumnTransform::SqlExpressions(vec![("x".into(), "y".into())]),
-                        None,
-                    )
+                    .add_columns()
+                    .transform(NewColumnTransform::SqlExpressions(vec![(
+                        "x".into(),
+                        "y".into(),
+                    )]))
+                    .execute()
                    .map_ok(|_| ()),
            ),
            Box::pin(async {
@@ -4300,32 +4302,9 @@ mod tests {
            "fetch_blobs",
        );

-        let message = table
-            .fetch_blob_files("image", &[1])
-            .await
-            .unwrap_err()
-            .to_string();
-        assert!(
-            message.contains("fetch_blob_files is not supported on LanceDB Cloud"),
-            "got: {message}"
-        );
-        assert!(
-            !message.contains("Use fetch_blobs"),
-            "old server must not be told to use fetch_blobs, got: {message}"
-        );
-    }
-
-    #[tokio::test]
-    async fn test_blob_files_point_at_fetch_blobs_on_a_blob_capable_server() {
-        let table = Table::new_with_handler_version(
-            "my_table",
-            semver::Version::new(0, 5, 0),
-            |_| -> http::Response<String> { panic!("fetch_blob_files must not reach the server") },
-        );
-
        assert_not_supported_error(
            table.fetch_blob_files("image", &[1]).await.unwrap_err(),
-            "Use fetch_blobs for full bytes",
+            "requires LanceDB Cloud server 0.5.0 or newer",
        );
    }

@@ -6411,13 +6390,12 @@ mod tests {
        });

        let result = table
-            .add_columns(
-                NewColumnTransform::SqlExpressions(vec![
-                    ("b".into(), "a + 1".into()),
-                    ("x".into(), "cast(NULL as int32)".into()),
-                ]),
-                None,
-            )
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![
+                ("b".into(), "a + 1".into()),
+                ("x".into(), "cast(NULL as int32)".into()),
+            ]))
+            .execute()
            .await
            .unwrap();

@@ -7142,10 +7120,12 @@ mod tests {
            }
            "add_columns" => {
                let _ = table
-                    .add_columns(
-                        NewColumnTransform::SqlExpressions(vec![("c".into(), "a + 1".into())]),
-                        None,
-                    )
+                    .add_columns()
+                    .transform(NewColumnTransform::SqlExpressions(vec![(
+                        "c".into(),
+                        "a + 1".into(),
+                    )]))
+                    .execute()
                    .await;
            }
            "drop_columns" => {
@@ -9903,10 +9883,12 @@ mod tests {
            .await
            .unwrap();
        branch
-            .add_columns(
-                NewColumnTransform::SqlExpressions(vec![("b".into(), "a + 1".into())]),
-                None,
-            )
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![(
+                "b".into(),
+                "a + 1".into(),
+            )]))
+            .execute()
            .await
            .unwrap();
        branch
@@ -3,6 +3,7 @@

 //! LanceDB Table APIs

+use crate::blob::BlobFile;
 use arrow_array::{LargeBinaryArray, RecordBatch, RecordBatchReader};
 use arrow_schema::{Schema, SchemaRef};
 use async_trait::async_trait;
@@ -12,7 +13,6 @@ use datafusion_physical_plan::ExecutionPlan;
 use datafusion_physical_plan::display::DisplayableExecutionPlan;
 use futures::StreamExt;
 use futures::stream::FuturesUnordered;
-use lance::dataset::BlobFile;
 pub use lance::dataset::ColumnAlteration;
 pub use lance::dataset::NewColumnTransform;
 pub use lance::dataset::ReadParams;
@@ -65,6 +65,7 @@ use crate::utils::{PatchReadParam, PatchWriteParam, resolve_arrow_field_path};
 use self::dataset::DatasetConsistencyWrapper;
 use self::merge::MergeInsertBuilder;

+pub mod add_columns;
 mod add_data;
 pub mod branch_merge;
 mod create_index;
@@ -79,6 +80,7 @@ pub mod schema_evolution;
 pub mod update;
 pub mod write_progress;
 use crate::index::waiter::wait_for_index;
+pub use add_columns::AddColumnsBuilder;
 #[cfg(feature = "remote")]
 pub(crate) use add_data::PreprocessingOutput;
 pub use add_data::{AddDataBuilder, AddDataMode, AddResult, NaNVectorBehavior};
@@ -1620,12 +1622,8 @@ impl Table {
    }

    /// Add new columns to the table, providing values to fill in.
-    pub async fn add_columns(
-        &self,
-        transforms: NewColumnTransform,
-        read_columns: Option<Vec<String>>,
-    ) -> Result<AddColumnsResult> {
-        self.inner.add_columns(transforms, read_columns).await
+    pub fn add_columns(&self) -> AddColumnsBuilder {
+        AddColumnsBuilder::new(self.inner.clone())
    }

    /// Change a column's name or nullability.
@@ -0,0 +1,161 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+//! Builder for adding columns to a table.
+
+use std::sync::Arc;
+
+use lance::dataset::NewColumnTransform;
+
+use super::BaseTable;
+use super::schema_evolution::AddColumnsResult;
+use crate::{Error, Result};
+
+/// Adds columns to a table. See [`Table::add_columns`](super::Table::add_columns).
+pub struct AddColumnsBuilder {
+    parent: Arc<dyn BaseTable>,
+    transform: Option<NewColumnTransform>,
+    read_columns: Option<Vec<String>>,
+}
+
+impl std::fmt::Debug for AddColumnsBuilder {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("AddColumnsBuilder")
+            .field("parent", &self.parent)
+            .field("has_transform", &self.transform.is_some())
+            .field("read_columns", &self.read_columns)
+            .finish()
+    }
+}
+
+impl AddColumnsBuilder {
+    pub(crate) fn new(parent: Arc<dyn BaseTable>) -> Self {
+        Self {
+            parent,
+            transform: None,
+            read_columns: None,
+        }
+    }
+
+    /// Set how the new columns' values are produced. Required.
+    pub fn transform(mut self, transform: NewColumnTransform) -> Self {
+        self.transform = Some(transform);
+        self
+    }
+
+    /// Limit which existing columns a [`NewColumnTransform::BatchUDF`] mapper
+    /// receives. Every other transform determines what it reads, so setting
+    /// this alongside one is an error rather than a silent no-op.
+    pub fn read_columns(mut self, columns: impl IntoIterator<Item = impl Into<String>>) -> Self {
+        self.read_columns = Some(columns.into_iter().map(Into::into).collect());
+        self
+    }
+
+    /// Add the columns.
+    pub async fn execute(self) -> Result<AddColumnsResult> {
+        let Self {
+            parent,
+            transform,
+            read_columns,
+        } = self;
+
+        let Some(transform) = transform else {
+            return Err(Error::InvalidInput {
+                message: "add_columns requires a transform".into(),
+            });
+        };
+
+        if read_columns.is_some() && !matches!(transform, NewColumnTransform::BatchUDF(_)) {
+            return Err(Error::InvalidInput {
+                message: "read_columns applies only to a BatchUDF transform; \
+                          every other transform determines what it reads"
+                    .into(),
+            });
+        }
+
+        parent.add_columns(transform, read_columns).await
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Arc;
+
+    use arrow_array::{Int32Array, RecordBatch, record_batch};
+    use arrow_schema::{DataType, Field, Schema};
+    use lance::dataset::{BatchUDF, NewColumnTransform};
+
+    use crate::Table;
+    use crate::connect;
+
+    async fn table_with_two_columns(name: &str) -> Table {
+        let conn = connect("memory://").execute().await.unwrap();
+        let batch = record_batch!(("x", Int32, [1, 2, 3]), ("y", Int32, [10, 20, 30])).unwrap();
+        conn.create_table(name, batch).execute().await.unwrap()
+    }
+
+    #[tokio::test]
+    async fn test_requires_a_transform() {
+        let table = table_with_two_columns("no_transform").await;
+        let err = table.add_columns().execute().await.unwrap_err();
+        assert!(
+            err.to_string().contains("requires a transform"),
+            "got: {err}"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_read_columns_with_sql_expressions_is_rejected() {
+        let table = table_with_two_columns("read_cols_sql").await;
+        let err = table
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![(
+                "doubled".into(),
+                "x * 2".into(),
+            )]))
+            .read_columns(["x"])
+            .execute()
+            .await
+            .unwrap_err();
+        assert!(err.to_string().contains("BatchUDF"), "got: {err}");
+
+        let schema = table.schema().await.unwrap();
+        assert!(
+            schema.field_with_name("doubled").is_err(),
+            "a rejected call must not commit"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_read_columns_limits_what_a_batch_udf_sees() {
+        let table = table_with_two_columns("read_cols_udf").await;
+
+        let output_schema = Arc::new(Schema::new(vec![Field::new("sum", DataType::Int32, true)]));
+        let mapper_schema = output_schema.clone();
+        let udf = BatchUDF {
+            mapper: Box::new(move |batch: &RecordBatch| {
+                assert!(batch.column_by_name("x").is_some());
+                assert!(batch.column_by_name("y").is_none(), "y was not requested");
+                let x = batch["x"].as_any().downcast_ref::<Int32Array>().unwrap();
+                let doubled: Int32Array = x.iter().map(|v| v.map(|v| v * 2)).collect();
+                Ok(RecordBatch::try_new(
+                    mapper_schema.clone(),
+                    vec![Arc::new(doubled)],
+                )?)
+            }),
+            output_schema,
+            result_checkpoint: None,
+        };
+
+        table
+            .add_columns()
+            .transform(NewColumnTransform::BatchUDF(udf))
+            .read_columns(["x"])
+            .execute()
+            .await
+            .unwrap();
+
+        let schema = table.schema().await.unwrap();
+        assert!(schema.field_with_name("sum").is_ok());
+    }
+}
@@ -576,10 +576,12 @@ mod tests {

        // Add a new physical column AFTER the embedding column.
        table
-            .add_columns(
-                NewColumnTransform::SqlExpressions(vec![("score".into(), "42.0".into())]),
-                None,
-            )
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![(
+                "score".into(),
+                "42.0".into(),
+            )]))
+            .execute()
            .await
            .unwrap();

@@ -683,7 +685,9 @@ mod tests {
            true,
        )]));
        table
-            .add_columns(NewColumnTransform::AllNulls(nested_schema), None)
+            .add_columns()
+            .transform(NewColumnTransform::AllNulls(nested_schema))
+            .execute()
            .await
            .unwrap();

@@ -193,10 +193,12 @@ mod tests {

        // Add a computed column
        let result = table
-            .add_columns(
-                NewColumnTransform::SqlExpressions(vec![("doubled".into(), "id * 2".into())]),
-                None,
-            )
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![(
+                "doubled".into(),
+                "id * 2".into(),
+            )]))
+            .execute()
            .await
            .unwrap();

@@ -251,13 +253,12 @@ mod tests {

        // Add multiple columns at once
        table
-            .add_columns(
-                NewColumnTransform::SqlExpressions(vec![
-                    ("y".into(), "x + 1".into()),
-                    ("z".into(), "x * x".into()),
-                ]),
-                None,
-            )
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![
+                ("y".into(), "x + 1".into()),
+                ("z".into(), "x * x".into()),
+            ]))
+            .execute()
            .await
            .unwrap();

@@ -283,10 +284,12 @@ mod tests {

        // Add a column with a constant value
        table
-            .add_columns(
-                NewColumnTransform::SqlExpressions(vec![("constant".into(), "42".into())]),
-                None,
-            )
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![(
+                "constant".into(),
+                "42".into(),
+            )]))
+            .execute()
            .await
            .unwrap();

@@ -659,10 +662,12 @@ mod tests {

        // Add column increments version
        let add_result = table
-            .add_columns(
-                NewColumnTransform::SqlExpressions(vec![("c".into(), "a + b".into())]),
-                None,
-            )
+            .add_columns()
+            .transform(NewColumnTransform::SqlExpressions(vec![(
+                "c".into(),
+                "a + b".into(),
+            )]))
+            .execute()
            .await
            .unwrap();
        assert!(add_result.version > v1);
@@ -9,14 +9,17 @@ use arrow_array::{
 };
 use arrow_schema::{DataType, Field, Fields, Schema};
 use futures::TryStreamExt;
+use lance::Dataset;
 use lance_encoding::version::LanceFileVersion;
 use lancedb::{
    Connection, Error, Result, Table,
    blob::{BlobRangeRequest, blob},
    connect, connect_namespace,
-    database::listing::OPT_NEW_TABLE_ENABLE_STABLE_ROW_IDS,
+    database::listing::{
+        ListingDatabaseOptions, NewTableConfig, OPT_NEW_TABLE_ENABLE_STABLE_ROW_IDS,
+    },
    query::{ExecutableQuery, QueryBase},
-    table::{AddDataMode, CompactionOptions, OptimizeAction},
+    table::{AddDataMode, CompactionOptions, OptimizeAction, OptimizeStats},
 };
 use tempfile::tempdir;

@@ -1075,3 +1078,252 @@ async fn fetch_blob_files_aligns_across_fragments_with_nulls_and_dups() -> Resul
    }
    Ok(())
 }
+
+/// Rows exercising the null/empty interleavings from
+/// <https://github.com/lancedb/lancedb/issues/3744>: a payload, a null, a valid
+/// empty value, then payloads whose descriptors a fragment rewrite used to zero.
+fn null_empty_input_batch() -> RecordBatch {
+    let owned = [
+        Some(dedicated_blob_bytes(1)),
+        None,
+        Some(Vec::new()),
+        Some(dedicated_blob_bytes(4)),
+        Some(dedicated_blob_bytes(5)),
+        Some(dedicated_blob_bytes(6)),
+    ];
+    let payloads: Vec<Option<&[u8]>> = owned.iter().map(|payload| payload.as_deref()).collect();
+    binary_input_batch(&[1, 2, 3, 4, 5, 6], &payloads)
+}
+
+/// One `(id, Some((payload length, first byte)))` per live row, or `(id, None)`
+/// for a null blob.  Comparing lengths and first bytes keeps failure output
+/// readable where comparing whole payloads would not.
+type BlobSummary = Vec<(i64, Option<(usize, Option<u8>)>)>;
+
+/// The rows [`null_empty_input_batch`] leaves behind after `id IN (1, 4)` is
+/// deleted: a null, a valid empty value, and the two payloads that follow them.
+fn expected_null_empty_survivors() -> BlobSummary {
+    vec![
+        (2, None),
+        (3, Some((0, None))),
+        (5, Some((DEDICATED_BLOB_LEN, Some(5)))),
+        (6, Some((DEDICATED_BLOB_LEN, Some(6)))),
+    ]
+}
+
+/// `optimize()` only rewrites a fragment when lance's compaction planner selects
+/// it — here because the delete pushes the fragment past
+/// `materialize_deletions_threshold` (0.1 by default; these tests delete 2 of 6
+/// rows). Without this check, a planner or threshold change upstream would leave
+/// both regression tests green while no rewrite happened at all.
+fn assert_compacted(stats: &OptimizeStats) {
+    let metrics = stats
+        .compaction
+        .as_ref()
+        .expect("OptimizeAction::All runs compaction");
+    assert!(
+        metrics.fragments_removed >= 1,
+        "optimize() rewrote no fragment, so this test proves nothing: {metrics:?}"
+    );
+}
+
+fn summarize(rows: &[(i64, Option<Vec<u8>>)]) -> BlobSummary {
+    rows.iter()
+        .map(|(id, payload)| {
+            (
+                *id,
+                payload
+                    .as_ref()
+                    .map(|bytes| (bytes.len(), bytes.first().copied())),
+            )
+        })
+        .collect()
+}
+
+async fn sorted_id_rowid(table: &Table) -> Result<Vec<(i64, u64)>> {
+    let mut pairs = collect_id_rowid(table).await?;
+    pairs.sort_by_key(|(id, _)| *id);
+    Ok(pairs)
+}
+
+/// `{position, size}` descriptors of a legacy v1 blob column, keyed by `id`.
+async fn v1_blob_descriptors(table: &Table) -> Result<Vec<(i64, Option<(u64, u64)>)>> {
+    let batches = table
+        .query()
+        .execute()
+        .await?
+        .try_collect::<Vec<_>>()
+        .await?;
+    let batch = arrow_select::concat::concat_batches(&batches[0].schema(), &batches).unwrap();
+    let ids = batch
+        .column_by_name("id")
+        .unwrap()
+        .as_any()
+        .downcast_ref::<Int64Array>()
+        .unwrap();
+    let descriptors = batch
+        .column_by_name("image")
+        .unwrap()
+        .as_any()
+        .downcast_ref::<StructArray>()
+        .expect("v1 blob column reads back as a descriptor struct");
+    let position = descriptors
+        .column_by_name("position")
+        .unwrap()
+        .as_any()
+        .downcast_ref::<UInt64Array>()
+        .unwrap();
+    let size = descriptors
+        .column_by_name("size")
+        .unwrap()
+        .as_any()
+        .downcast_ref::<UInt64Array>()
+        .unwrap();
+    let mut rows: Vec<(i64, Option<(u64, u64)>)> = (0..batch.num_rows())
+        .map(|row| {
+            let descriptor =
+                (!descriptors.is_null(row)).then(|| (position.value(row), size.value(row)));
+            (ids.value(row), descriptor)
+        })
+        .collect();
+    rows.sort_by_key(|(id, _)| *id);
+    Ok(rows)
+}
+
+/// Payload bytes of every live row of a legacy v1 blob column, keyed by `id`.
+/// [`Table::fetch_blobs`] rejects v1 columns, so read them through lance.
+async fn v1_blob_payloads(dataset_uri: &str, table: &Table) -> Result<Vec<(i64, Option<Vec<u8>>)>> {
+    let pairs = sorted_id_rowid(table).await?;
+    let row_ids: Vec<u64> = pairs.iter().map(|(_, row_id)| *row_id).collect();
+    let dataset = Arc::new(Dataset::open(dataset_uri).await?);
+    let files = dataset.take_blobs(&row_ids, "image").await?;
+    assert_eq!(
+        files.len(),
+        pairs.len(),
+        "take_blobs returned {} handles for {} live rows",
+        files.len(),
+        pairs.len()
+    );
+    let mut rows = Vec::with_capacity(pairs.len());
+    for ((id, _), file) in pairs.iter().zip(files) {
+        let payload = match file {
+            Some(file) => Some(file.read().await?.to_vec()),
+            None => None,
+        };
+        rows.push((*id, payload));
+    }
+    Ok(rows)
+}
+
+/// Length and first byte of every live blob v2 value, keyed by `id`.
+async fn blob_v2_values(table: &Table) -> Result<BlobSummary> {
+    let pairs = sorted_id_rowid(table).await?;
+    let row_ids: Vec<u64> = pairs.iter().map(|(_, row_id)| *row_id).collect();
+    let bytes = table.fetch_blobs("image", &row_ids).await?;
+    Ok(pairs
+        .iter()
+        .enumerate()
+        .map(|(slot, (id, _))| {
+            let value = (!bytes.is_null(slot))
+                .then(|| (bytes.value(slot).len(), bytes.value(slot).first().copied()));
+            (*id, value)
+        })
+        .collect())
+}
+
+/// Regression test for [#3744]: on storage 2.0 (legacy v1 descriptors),
+/// compaction rewrote every payload following a null or empty value in the same
+/// fragment as `{position: 0, size: 0}`, so the payload bytes read back as `b""`
+/// and the new fragment no longer referenced them at all.
+///
+/// [#3744]: https://github.com/lancedb/lancedb/issues/3744
+#[tokio::test]
+async fn optimize_preserves_v1_blob_payloads_with_null_and_empty() -> Result<()> {
+    let tmp = tempdir().unwrap();
+    let db_uri = tmp.path().to_str().unwrap().to_string();
+    let db = connect(&db_uri)
+        .database_options(&ListingDatabaseOptions {
+            new_table_config: NewTableConfig {
+                data_storage_version: Some(LanceFileVersion::V2_0),
+                ..Default::default()
+            },
+            ..Default::default()
+        })
+        .execute()
+        .await?;
+    let legacy = Field::new("image", DataType::LargeBinary, true).with_metadata(
+        std::collections::HashMap::from([("lance-encoding:blob".to_string(), "true".to_string())]),
+    );
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("id", DataType::Int64, false),
+        legacy,
+    ]));
+    let table = db.create_empty_table("t", schema).execute().await?;
+    table.add(null_empty_input_batch()).execute().await?;
+    assert_eq!(
+        storage_format_version(&table).await,
+        LanceFileVersion::V2_0.resolve(),
+        "v1 blob descriptors only exist below storage 2.2"
+    );
+    let dataset_uri = table.uri().await?;
+
+    // Any rewrite triggers it; deleting rows is the shape from the issue.
+    table.delete("id IN (1, 4)").await?;
+    let descriptors_before = v1_blob_descriptors(&table).await?;
+    let before = v1_blob_payloads(&dataset_uri, &table).await?;
+    assert_eq!(
+        summarize(&before),
+        expected_null_empty_survivors(),
+        "test setup no longer produces the null/empty/payload mix"
+    );
+
+    let stats = table.optimize(OptimizeAction::All).await?;
+    assert_compacted(&stats);
+
+    let descriptors_after = v1_blob_descriptors(&table).await?;
+    let after = v1_blob_payloads(&dataset_uri, &table).await?;
+    assert_eq!(
+        summarize(&after),
+        summarize(&before),
+        "optimize() lost blob payloads; descriptors before={descriptors_before:?} after={descriptors_after:?}"
+    );
+    assert!(after == before, "optimize() changed blob payload bytes");
+    Ok(())
+}
+
+/// Regression test for the blob v2 half of [#3744]: compaction rewrote a valid
+/// empty value as null, destroying the null-vs-empty distinction.
+///
+/// [#3744]: https://github.com/lancedb/lancedb/issues/3744
+#[tokio::test]
+async fn optimize_preserves_blob_v2_null_and_empty_distinction() -> Result<()> {
+    let tmp = tempdir().unwrap();
+    let db = connect(tmp.path().to_str().unwrap()).execute().await?;
+    let table = db
+        .create_empty_table("t", blob_table_schema())
+        .execute()
+        .await?;
+    table.add(null_empty_input_batch()).execute().await?;
+    assert!(
+        storage_format_version(&table).await >= LanceFileVersion::V2_2,
+        "blob v2 columns require storage >= 2.2"
+    );
+
+    table.delete("id IN (1, 4)").await?;
+    let before = blob_v2_values(&table).await?;
+    assert_eq!(
+        before,
+        expected_null_empty_survivors(),
+        "test setup no longer produces the null/empty/payload mix"
+    );
+
+    let stats = table.optimize(OptimizeAction::All).await?;
+    assert_compacted(&stats);
+
+    assert_eq!(
+        blob_v2_values(&table).await?,
+        before,
+        "optimize() changed blob v2 values"
+    );
+    Ok(())
+}