feat(listing): soft-delete root tables via an embedded V1 namespace

Root-table lifecycle in a ListingDatabase now flows through an embedded V1 (manifest-disabled) DirectoryNamespace, so a root drop is a soft-delete rather than an immediate remove_dir_all: - drop_table writes a delete marker and leaves the data for a later purge (TTL). - create_table on a soft-deleted name revives it (clears the marker under the namespace's lifecycle lock, then overwrites via the native create path). - open_table / table_names / list_tables treat soft-deleted tables as absent. - table listing is now a single O(1) read_dir in the namespace instead of a per-table probe here. A ListingDatabase now holds two embedded namespaces: the existing manifest-backed `namespace_database` for child namespaces (multi-level table ids), and a new `root_namespace_database` (V1, manifest-off) that owns root soft-delete/purge/ table_status. `namespace_client()` still returns the manifest namespace so child namespace ops are unaffected. Also preserves TableNotFound through LanceNamespaceDatabase::drop_table instead of flattening it to a generic Runtime error, so dropping a missing table reports the right error. Removes the now-dead native drop_tables / commit-handler path and the object_store/base_path fields it needed. Depends on lance-format/lance#7541. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
chore: temporarily pin lance to soft-delete PR rev
2026-07-01 01:50:39 +00:00 · 2026-06-30 17:08:16 -07:00 · 2026-06-30 17:08:07 -07:00 · 2026-06-24 23:29:23 +00:00 · 2026-06-24 15:30:00 -07:00 · 2026-06-24 13:25:59 -07:00
26 changed files with 475 additions and 218 deletions
--- a/.bumpversion.toml
+++ b/.bumpversion.toml
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.31.0-beta.1"
+current_version = "0.31.0-beta.2"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -3432,8 +3432,8 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"

 [[package]]
 name = "fsst"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "rand 0.9.4",
@@ -4735,8 +4735,8 @@ checksum = "e037a2e1d8d5fdbd49b16a4ea09d5d6401c1f29eca5ff29d03d3824dba16256a"

 [[package]]
 name = "lance"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arc-swap",
 "arrow",
@@ -4771,7 +4771,7 @@ dependencies = [
 "futures",
 "half",
 "humantime",
- "itertools 0.13.0",
+ "itertools 0.14.0",
 "lance-arrow",
 "lance-core",
 "lance-datafusion",
@@ -4810,8 +4810,8 @@ dependencies = [

 [[package]]
 name = "lance-arrow"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4832,7 +4832,7 @@ dependencies = [
 [[package]]
 name = "lance-arrow-scalar"
 version = "58.0.0"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4846,7 +4846,7 @@ dependencies = [
 [[package]]
 name = "lance-arrow-stats"
 version = "58.0.0"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "arrow-schema",
@@ -4855,8 +4855,8 @@ dependencies = [

 [[package]]
 name = "lance-bitpacking"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrayref",
 "paste",
@@ -4865,8 +4865,8 @@ dependencies = [

 [[package]]
 name = "lance-core"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -4878,7 +4878,7 @@ dependencies = [
 "datafusion-common",
 "datafusion-sql",
 "futures",
- "itertools 0.13.0",
+ "itertools 0.14.0",
 "lance-arrow",
 "lance-derive",
 "libc",
@@ -4904,8 +4904,8 @@ dependencies = [

 [[package]]
 name = "lance-datafusion"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -4935,8 +4935,8 @@ dependencies = [

 [[package]]
 name = "lance-datagen"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -4953,8 +4953,8 @@ dependencies = [

 [[package]]
 name = "lance-derive"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -4963,8 +4963,8 @@ dependencies = [

 [[package]]
 name = "lance-encoding"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-arith",
 "arrow-array",
@@ -4980,7 +4980,7 @@ dependencies = [
 "futures",
 "hex",
 "hyperloglogplus",
- "itertools 0.13.0",
+ "itertools 0.14.0",
 "lance-arrow",
 "lance-bitpacking",
 "lance-core",
@@ -4999,8 +4999,8 @@ dependencies = [

 [[package]]
 name = "lance-file"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-arith",
 "arrow-array",
@@ -5030,8 +5030,8 @@ dependencies = [

 [[package]]
 name = "lance-index"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arc-swap",
 "arrow",
@@ -5056,7 +5056,7 @@ dependencies = [
 "fst",
 "futures",
 "half",
- "itertools 0.13.0",
+ "itertools 0.14.0",
 "jieba-rs",
 "jsonb",
 "lance-arrow",
@@ -5096,8 +5096,8 @@ dependencies = [

 [[package]]
 name = "lance-io"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow",
 "arrow-arith",
@@ -5138,8 +5138,8 @@ dependencies = [

 [[package]]
 name = "lance-linalg"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5155,8 +5155,8 @@ dependencies = [

 [[package]]
 name = "lance-namespace"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow",
 "async-trait",
@@ -5168,8 +5168,8 @@ dependencies = [

 [[package]]
 name = "lance-namespace-impls"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow",
 "arrow-ipc",
@@ -5223,15 +5223,15 @@ dependencies = [

 [[package]]
 name = "lance-select"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
 "arrow-schema",
 "byteorder",
 "bytes",
- "itertools 0.13.0",
+ "itertools 0.14.0",
 "lance-core",
 "roaring",
 "tracing",
@@ -5239,8 +5239,8 @@ dependencies = [

 [[package]]
 name = "lance-table"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow",
 "arrow-array",
@@ -5279,8 +5279,8 @@ dependencies = [

 [[package]]
 name = "lance-testing"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "arrow-array",
 "arrow-schema",
@@ -5293,8 +5293,8 @@ dependencies = [

 [[package]]
 name = "lance-tokenizer"
-version = "9.0.0-beta.2"
-source = "git+https://github.com/lance-format/lance.git?tag=v9.0.0-beta.2#23211989de648fefc4454f5eee09ec176f0a465b"
+version = "9.0.0-beta.8"
+source = "git+https://github.com/lance-format/lance.git?rev=4acefffd5d38f88003fce681ae1d0871077ce5e7#4acefffd5d38f88003fce681ae1d0871077ce5e7"
 dependencies = [
 "icu_segmenter",
 "jieba-rs",
@@ -5307,7 +5307,7 @@ dependencies = [

 [[package]]
 name = "lancedb"
-version = "0.31.0-beta.1"
+version = "0.31.0-beta.2"
 dependencies = [
 "ahash",
 "anyhow",
@@ -5390,7 +5390,7 @@ dependencies = [

 [[package]]
 name = "lancedb-nodejs"
-version = "0.31.0-beta.1"
+version = "0.31.0-beta.2"
 dependencies = [
 "arrow-array",
 "arrow-buffer",
@@ -5415,7 +5415,7 @@ dependencies = [

 [[package]]
 name = "lancedb-python"
-version = "0.34.0-beta.1"
+version = "0.34.0-beta.2"
 dependencies = [
 "arrow",
 "async-trait",
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -13,20 +13,20 @@ categories = ["database-implementations"]
 rust-version = "1.91.0"

 [workspace.dependencies]
-lance = { "version" = "=9.0.0-beta.2", default-features = false, "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-core = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-datagen = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-file = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-io = { "version" = "=9.0.0-beta.2", default-features = false, "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-index = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-linalg = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-namespace-impls = { "version" = "=9.0.0-beta.2", default-features = false, "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-table = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-testing = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-datafusion = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-encoding = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
-lance-arrow = { "version" = "=9.0.0-beta.2", "tag" = "v9.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
+lance = { "version" = "=9.0.0-beta.8", default-features = false, "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-core = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-datagen = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-file = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-io = { "version" = "=9.0.0-beta.8", default-features = false, "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-index = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-linalg = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-namespace-impls = { "version" = "=9.0.0-beta.8", default-features = false, "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-table = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-testing = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-datafusion = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-encoding = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
+lance-arrow = { "version" = "=9.0.0-beta.8", "rev" = "4acefffd5d38f88003fce681ae1d0871077ce5e7", "git" = "https://github.com/lance-format/lance.git" }
 ahash = "0.8"
 # Note that this one does not include pyarrow
 arrow = { version = "58.0.0", optional = false }
--- a/docs/src/java/java.md
+++ b/docs/src/java/java.md
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
 <dependency>
    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-core</artifactId>
-    <version>0.31.0-beta.1</version>
+    <version>0.31.0-beta.2</version>
 </dependency>
 ```

--- a/java/lancedb-core/pom.xml
+++ b/java/lancedb-core/pom.xml
@@ -8,7 +8,7 @@
    <parent>
      <groupId>com.lancedb</groupId>
      <artifactId>lancedb-parent</artifactId>
-      <version>0.31.0-beta.1</version>
+      <version>0.31.0-beta.2</version>
      <relativePath>../pom.xml</relativePath>
    </parent>

--- a/java/pom.xml
+++ b/java/pom.xml
@@ -6,7 +6,7 @@

    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-parent</artifactId>
-    <version>0.31.0-beta.1</version>
+    <version>0.31.0-beta.2</version>
    <packaging>pom</packaging>
    <name>${project.artifactId}</name>
    <description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <arrow.version>15.0.0</arrow.version>
-        <lance-core.version>9.0.0-beta.2</lance-core.version>
+        <lance-core.version>9.0.0-beta.8</lance-core.version>
        <spotless.skip>false</spotless.skip>
        <spotless.version>2.30.0</spotless.version>
        <spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
--- a/nodejs/Cargo.toml
+++ b/nodejs/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "lancedb-nodejs"
 edition.workspace = true
-version = "0.31.0-beta.1"
+version = "0.31.0-beta.2"
 publish = false
 license.workspace = true
 description.workspace = true
--- a/nodejs/npm/darwin-arm64/package.json
+++ b/nodejs/npm/darwin-arm64/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.31.0-beta.1",
+	"version": "0.31.0-beta.2",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",
--- a/nodejs/npm/linux-arm64-gnu/package.json
+++ b/nodejs/npm/linux-arm64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.31.0-beta.1",
+	"version": "0.31.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",
--- a/nodejs/npm/linux-arm64-musl/package.json
+++ b/nodejs/npm/linux-arm64-musl/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-musl",
-	"version": "0.31.0-beta.1",
+	"version": "0.31.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-musl.node",
--- a/nodejs/npm/linux-x64-gnu/package.json
+++ b/nodejs/npm/linux-x64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.31.0-beta.1",
+	"version": "0.31.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",
--- a/nodejs/npm/linux-x64-musl/package.json
+++ b/nodejs/npm/linux-x64-musl/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-musl",
-	"version": "0.31.0-beta.1",
+	"version": "0.31.0-beta.2",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-musl.node",
--- a/nodejs/npm/win32-arm64-msvc/package.json
+++ b/nodejs/npm/win32-arm64-msvc/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@lancedb/lancedb-win32-arm64-msvc",
-  "version": "0.31.0-beta.1",
+  "version": "0.31.0-beta.2",
  "os": [
    "win32"
  ],
--- a/nodejs/npm/win32-x64-msvc/package.json
+++ b/nodejs/npm/win32-x64-msvc/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.31.0-beta.1",
+	"version": "0.31.0-beta.2",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",
--- a/nodejs/package-lock.json
+++ b/nodejs/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "@lancedb/lancedb",
-  "version": "0.31.0-beta.1",
+  "version": "0.31.0-beta.2",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@lancedb/lancedb",
-      "version": "0.31.0-beta.1",
+      "version": "0.31.0-beta.2",
      "cpu": [
        "x64",
        "arm64"
--- a/nodejs/package.json
+++ b/nodejs/package.json
@@ -11,7 +11,7 @@
    "ann"
  ],
  "private": false,
-  "version": "0.31.0-beta.1",
+  "version": "0.31.0-beta.2",
  "main": "dist/index.js",
  "exports": {
    ".": "./dist/index.js",
--- a/python/python/lancedb/embeddings/colpali.py
+++ b/python/python/lancedb/embeddings/colpali.py
@@ -81,6 +81,7 @@ class ColPaliEmbeddings(EmbeddingFunction):
            warnings.warn(
                "use_token_pooling is deprecated, use pooling_strategy=None instead",
                DeprecationWarning,
+                stacklevel=2,
            )
            self.pooling_strategy = None

--- a/python/python/lancedb/remote/db.py
+++ b/python/python/lancedb/remote/db.py
@@ -124,6 +124,7 @@ class RemoteDBConnection(DBConnection):
                "request_thread_pool is no longer used and will be removed in "
                "a future release.",
                DeprecationWarning,
+                stacklevel=2,
            )

        if connection_timeout is not None:
@@ -132,6 +133,7 @@ class RemoteDBConnection(DBConnection):
                "release. Please use client_config.timeout_config.connect_timeout "
                "instead.",
                DeprecationWarning,
+                stacklevel=2,
            )
            client_config.timeout_config.connect_timeout = timedelta(
                seconds=connection_timeout
@@ -142,6 +144,7 @@ class RemoteDBConnection(DBConnection):
                "read_timeout is deprecated and will be removed in a future release. "
                "Please use client_config.timeout_config.read_timeout instead.",
                DeprecationWarning,
+                stacklevel=2,
            )
            client_config.timeout_config.read_timeout = timedelta(seconds=read_timeout)

--- a/python/python/lancedb/remote/table.py
+++ b/python/python/lancedb/remote/table.py
@@ -845,7 +845,8 @@ class RemoteTable(Table):
        """
        warnings.warn(
            "cleanup_old_versions() is a no-op on LanceDB Cloud. "
-            "Tables are automatically cleaned up and optimized."
+            "Tables are automatically cleaned up and optimized.",
+            stacklevel=2,
        )
        pass

@@ -857,7 +858,8 @@ class RemoteTable(Table):
        """
        warnings.warn(
            "compact_files() is a no-op on LanceDB Cloud. "
-            "Tables are automatically compacted and optimized."
+            "Tables are automatically compacted and optimized.",
+            stacklevel=2,
        )
        pass

@@ -874,7 +876,8 @@ class RemoteTable(Table):
        """
        warnings.warn(
            "optimize() is a no-op on LanceDB Cloud. "
-            "Indices are optimized automatically."
+            "Indices are optimized automatically.",
+            stacklevel=2,
        )
        pass

--- a/python/python/lancedb/table.py
+++ b/python/python/lancedb/table.py
@@ -3409,18 +3409,20 @@ class LanceTable(Table):

        if data_storage_version is not None:
            warnings.warn(
-                "setting data_storage_version directly on create_table is deprecated. ",
+                "setting data_storage_version directly on create_table is deprecated. "
                "Use database_options instead.",
                DeprecationWarning,
+                stacklevel=2,
            )
            if storage_options is None:
                storage_options = {}
            storage_options["new_table_data_storage_version"] = data_storage_version
        if enable_v2_manifest_paths is not None:
            warnings.warn(
-                "setting enable_v2_manifest_paths directly on create_table is ",
+                "setting enable_v2_manifest_paths directly on create_table is "
                "deprecated. Use database_options instead.",
                DeprecationWarning,
+                stacklevel=2,
            )
            if storage_options is None:
                storage_options = {}
@@ -5662,6 +5664,7 @@ class AsyncTable:
                "The 'retrain' parameter is deprecated and will be removed in a "
                "future version.",
                DeprecationWarning,
+                stacklevel=2,
            )

        return await self._inner.optimize(
--- a/rust/lancedb/Cargo.toml
+++ b/rust/lancedb/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb"
-version = "0.31.0-beta.1"
+version = "0.31.0-beta.2"
 edition.workspace = true
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license.workspace = true
--- a/rust/lancedb/src/connection.rs
+++ b/rust/lancedb/src/connection.rs
@@ -576,6 +576,9 @@ impl Connection {
    /// For LanceNamespaceDatabase, it is the underlying LanceNamespace.
    /// For ListingDatabase, it is the equivalent DirectoryNamespace.
    /// For RemoteDatabase, it is the equivalent RestNamespace.
+    ///
+    /// Remote connections using dynamic headers forward them through the
+    /// namespace client's per-request context provider.
    pub async fn namespace_client(&self) -> Result<Arc<dyn lance_namespace::LanceNamespace>> {
        self.internal.namespace_client().await
    }
@@ -584,6 +587,9 @@ impl Connection {
    /// Returns (impl_type, properties) where:
    /// - impl_type: "dir" for DirectoryNamespace, "rest" for RestNamespace
    /// - properties: configuration properties for the namespace
+    ///
+    /// Remote connections using dynamic headers cannot be exported because the
+    /// namespace client config only carries static headers.
    pub async fn namespace_client_config(
        &self,
    ) -> Result<(String, std::collections::HashMap<String, String>)> {
--- a/rust/lancedb/src/database/listing.rs
+++ b/rust/lancedb/src/database/listing.rs
@@ -14,7 +14,6 @@ use lance::io::{ObjectStore, ObjectStoreParams, WrappingObjectStore};
 use lance_datafusion::utils::StreamingWriteSource;
 use lance_encoding::version::LanceFileVersion;
 use lance_io::object_store::{StorageOptionsAccessor, StorageOptionsProvider};
-use lance_table::io::commit::commit_handler_from_url;
 use object_store::local::LocalFileSystem;
 use snafu::ResultExt;

@@ -235,11 +234,9 @@ impl ListingDatabaseOptionsBuilder {
 /// We will have two tables named `table1` and `table2`.
 #[derive(Debug)]
 pub struct ListingDatabase {
-    object_store: Arc<ObjectStore>,
    query_string: Option<String>,

    pub(crate) uri: String,
-    pub(crate) base_path: object_store::path::Path,

    // the object store wrapper to use on write path
    pub(crate) store_wrapper: Option<Arc<dyn WrappingObjectStore>>,
@@ -258,8 +255,13 @@ pub struct ListingDatabase {
    // Session for object stores and caching
    session: Arc<lance::session::Session>,

-    // Namespace-backed database for child namespace operations
+    // Namespace-backed database for child namespace operations (manifest mode).
    namespace_database: Arc<LanceNamespaceDatabase>,
+
+    // V1 (manifest-disabled) directory namespace for root table lifecycle, so root
+    // drops are soft-deletes and purge/table_status are available. Shares the same root
+    // as `namespace_database` but in directory mode.
+    root_namespace_database: Arc<LanceNamespaceDatabase>,
 }

 impl std::fmt::Display for ListingDatabase {
@@ -280,7 +282,6 @@ impl std::fmt::Display for ListingDatabase {
    }
 }

-const LANCE_EXTENSION: &str = "lance";
 const ENGINE: &str = "engine";
 const MIRRORED_STORE: &str = "mirroredStore";

@@ -342,6 +343,39 @@ impl ListingDatabase {
        ))
    }

+    /// Build the V1 (manifest-disabled) directory namespace used for *root* table
+    /// lifecycle ops.
+    ///
+    /// Root tables in a listing database are flat `<name>.lance` directories; soft-delete
+    /// (drop/purge/TTL) is a V1-only mechanism, so root ops go through this namespace.
+    /// Child namespaces are manifest-backed and handled by the separate
+    /// (manifest-enabled) `namespace_database`.
+    async fn connect_root_namespace_database(
+        uri: &str,
+        storage_options: HashMap<String, String>,
+        namespace_client_properties: HashMap<String, String>,
+        read_consistency_interval: Option<std::time::Duration>,
+        session: Arc<lance::session::Session>,
+    ) -> Result<Arc<LanceNamespaceDatabase>> {
+        let mut ns_properties = Self::build_namespace_client_properties(
+            uri,
+            &storage_options,
+            namespace_client_properties,
+        );
+        ns_properties.insert("manifest_enabled".to_string(), "false".to_string());
+        Ok(Arc::new(
+            LanceNamespaceDatabase::connect(
+                "dir",
+                ns_properties,
+                storage_options,
+                read_consistency_interval,
+                Some(session),
+                HashSet::new(),
+            )
+            .await?,
+        ))
+    }
+
    async fn prepare_namespace_root(
        uri: &str,
        storage_options: &HashMap<String, String>,
@@ -548,7 +582,7 @@ impl ListingDatabase {
                    },
                    ..Default::default()
                };
-                let (object_store, base_path) = ObjectStore::from_uri_and_params(
+                let (object_store, _base_path) = ObjectStore::from_uri_and_params(
                    session.store_registry(),
                    &plain_uri,
                    &os_params,
@@ -577,12 +611,18 @@ impl ListingDatabase {
                    session.clone(),
                )
                .await?;
+                let root_namespace_database = Self::connect_root_namespace_database(
+                    &table_base_uri,
+                    options.storage_options.clone(),
+                    request.namespace_client_properties.clone(),
+                    request.read_consistency_interval,
+                    session.clone(),
+                )
+                .await?;

                Ok(Self {
                    uri: table_base_uri,
                    query_string,
-                    base_path,
-                    object_store,
                    store_wrapper: write_store_wrapper,
                    read_consistency_interval: request.read_consistency_interval,
                    storage_options: options.storage_options,
@@ -590,6 +630,7 @@ impl ListingDatabase {
                    new_table_config: options.new_table_config,
                    session,
                    namespace_database,
+                    root_namespace_database,
                })
            }
            Err(_) => {
@@ -613,7 +654,7 @@ impl ListingDatabase {
        session: Option<Arc<lance::session::Session>>,
    ) -> Result<Self> {
        let session = session.unwrap_or_else(|| Arc::new(lance::session::Session::default()));
-        let (object_store, base_path) = ObjectStore::from_uri_and_params(
+        let (object_store, _base_path) = ObjectStore::from_uri_and_params(
            session.store_registry(),
            path,
            &ObjectStoreParams::default(),
@@ -624,6 +665,14 @@ impl ListingDatabase {
        }

        let namespace_database = Self::connect_namespace_database(
+            path,
+            HashMap::new(),
+            namespace_client_properties.clone(),
+            read_consistency_interval,
+            session.clone(),
+        )
+        .await?;
+        let root_namespace_database = Self::connect_root_namespace_database(
            path,
            HashMap::new(),
            namespace_client_properties,
@@ -635,8 +684,6 @@ impl ListingDatabase {
        Ok(Self {
            uri: path.to_string(),
            query_string: None,
-            base_path,
-            object_store,
            store_wrapper: None,
            read_consistency_interval,
            storage_options: HashMap::new(),
@@ -644,6 +691,7 @@ impl ListingDatabase {
            new_table_config,
            session,
            namespace_database,
+            root_namespace_database,
        })
    }

@@ -705,42 +753,10 @@ impl ListingDatabase {
        self.namespace_database.clone()
    }

-    async fn drop_tables(&self, names: Vec<String>) -> Result<()> {
-        let object_store_params = ObjectStoreParams {
-            storage_options_accessor: if self.storage_options.is_empty() {
-                None
-            } else {
-                Some(Arc::new(StorageOptionsAccessor::with_static_options(
-                    self.storage_options.clone(),
-                )))
-            },
-            ..Default::default()
-        };
-        let mut uri = self.uri.clone();
-        if let Some(query_string) = &self.query_string {
-            uri.push_str(&format!("?{}", query_string));
-        }
-        let commit_handler = commit_handler_from_url(&uri, &Some(object_store_params)).await?;
-        for name in names {
-            let dir_name = format!("{}.{}", name, LANCE_EXTENSION);
-            let full_path = self.base_path.clone().join(dir_name.clone());
-
-            commit_handler.delete(&full_path).await?;
-
-            self.object_store
-                .remove_dir_all(full_path.clone())
-                .await
-                .map_err(|err| match err {
-                    // this error is not lance::Error::DatasetNotFound, as the method
-                    // `remove_dir_all` may be used to remove something not be a dataset
-                    lance::Error::NotFound { .. } => Error::TableNotFound {
-                        name: name.clone(),
-                        source: Box::new(err),
-                    },
-                    _ => Error::from(err),
-                })?;
-        }
-        Ok(())
+    /// The V1 directory namespace used for root table lifecycle (soft-delete drop, purge,
+    /// table_status, O(1) listing).
+    fn root_namespace_database(&self) -> Arc<LanceNamespaceDatabase> {
+        self.root_namespace_database.clone()
    }

    /// Inherit storage options from the connection into the target map
@@ -946,88 +962,43 @@ impl Database for ListingDatabase {
        if !request.namespace_path.is_empty() {
            return self.namespace_database().table_names(request).await;
        }
-        let mut f = self
-            .object_store
-            .read_dir(self.base_path.clone())
-            .await?
-            .iter()
-            .map(Path::new)
-            .filter(|path| {
-                let is_lance = path
-                    .extension()
-                    .and_then(|e| e.to_str())
-                    .map(|e| e == LANCE_EXTENSION);
-                is_lance.unwrap_or(false)
-            })
-            .filter_map(|p| p.file_stem().and_then(|s| s.to_str().map(String::from)))
-            .collect::<Vec<String>>();
-        f.sort();
-        if let Some(start_after) = request.start_after {
-            let index = f
-                .iter()
-                .position(|name| name.as_str() > start_after.as_str())
-                .unwrap_or(f.len());
-            f.drain(0..index);
-        }
-        if let Some(limit) = request.limit {
-            f.truncate(limit as usize);
-        }
-        Ok(f)
+        // Root tables: the V1 namespace lists them in a single read_dir (O(1) requests)
+        // and excludes soft-deleted tables, instead of a per-table probe here.
+        self.root_namespace_database().table_names(request).await
    }

    async fn list_tables(&self, request: ListTablesRequest) -> Result<ListTablesResponse> {
        if request.id.as_ref().map(|v| !v.is_empty()).unwrap_or(false) {
            return self.namespace_database().list_tables(request).await;
        }
-        let mut f = self
-            .object_store
-            .read_dir(self.base_path.clone())
-            .await?
-            .iter()
-            .map(Path::new)
-            .filter(|path| {
-                let is_lance = path
-                    .extension()
-                    .and_then(|e| e.to_str())
-                    .map(|e| e == LANCE_EXTENSION);
-                is_lance.unwrap_or(false)
-            })
-            .filter_map(|p| p.file_stem().and_then(|s| s.to_str().map(String::from)))
-            .collect::<Vec<String>>();
-        f.sort();
-
-        // Handle pagination with page_token
-        if let Some(ref page_token) = request.page_token {
-            let index = f
-                .iter()
-                .position(|name| name.as_str() > page_token.as_str())
-                .unwrap_or(f.len());
-            f.drain(0..index);
-        }
-
-        // Determine if there's a next page
-        let next_page_token = if let Some(limit) = request.limit {
-            if f.len() > limit as usize {
-                let token = f[limit as usize].clone();
-                f.truncate(limit as usize);
-                Some(token)
-            } else {
-                None
-            }
-        } else {
-            None
-        };
-
-        Ok(ListTablesResponse {
-            tables: f,
-            page_token: next_page_token,
-        })
+        self.root_namespace_database().list_tables(request).await
    }

    async fn create_table(&self, request: CreateTableRequest) -> Result<Arc<dyn BaseTable>> {
        if !request.namespace_path.is_empty() {
            return self.namespace_database().create_table(request).await;
        }
+        let mut request = request;
+        // Re-creating a soft-deleted table is a revive: clear the delete marker (via the
+        // V1 root namespace, under its lifecycle lock so a concurrent purge can't race),
+        // making the table live again, then overwrite its data through the native create
+        // path below (preserving lineage as a new version). A plain native create would
+        // leave the marker in place, keeping the table hidden.
+        if matches!(
+            self.root_namespace_database()
+                .namespace_client()
+                .await?
+                .table_status(Some(vec![request.name.clone()]))
+                .await?,
+            lance_namespace::TableLifecycle::SoftDeleted { .. }
+        ) {
+            self.root_namespace_database()
+                .namespace_client()
+                .await?
+                .undelete_table(Some(vec![request.name.clone()]))
+                .await?;
+            request.mode = CreateTableMode::Overwrite;
+        }
        // Use provided location if available, otherwise derive from table name
        let table_uri = request
            .location
@@ -1146,6 +1117,19 @@ impl Database for ListingDatabase {
        if !request.namespace_path.is_empty() {
            return self.namespace_database().open_table(request).await;
        }
+        // A soft-deleted (dropped-but-not-purged) table must read as absent even though
+        // its data still exists on disk. Consult the V1 root namespace (which owns the
+        // marker); if soft-deleted, route to it so the open surfaces TableNotFound.
+        if matches!(
+            self.root_namespace_database()
+                .namespace_client()
+                .await?
+                .table_status(Some(vec![request.name.clone()]))
+                .await?,
+            lance_namespace::TableLifecycle::SoftDeleted { .. }
+        ) {
+            return self.root_namespace_database().open_table(request).await;
+        }
        // Use provided location if available, otherwise derive from table name
        let table_uri = request
            .location
@@ -1245,20 +1229,23 @@ impl Database for ListingDatabase {
                .drop_table(name, namespace_path)
                .await;
        }
-        self.drop_tables(vec![name.to_string()]).await
+        // Root table: route through the V1 namespace so the drop is a soft-delete (writes
+        // a marker, leaves data for later purge) rather than an immediate remove_dir_all.
+        self.root_namespace_database()
+            .drop_table(name, namespace_path)
+            .await
    }

-    #[allow(deprecated)]
    async fn drop_all_tables(&self, namespace_path: &[String]) -> Result<()> {
-        // Check if namespace parameter is provided
        if !namespace_path.is_empty() {
            return self
                .namespace_database()
                .drop_all_tables(namespace_path)
                .await;
        }
-        let tables = self.table_names(TableNamesRequest::default()).await?;
-        self.drop_tables(tables).await
+        self.root_namespace_database()
+            .drop_all_tables(namespace_path)
+            .await
    }

    fn as_any(&self) -> &dyn std::any::Any {
@@ -1266,6 +1253,9 @@ impl Database for ListingDatabase {
    }

    async fn namespace_client(&self) -> Result<Arc<dyn lance_namespace::LanceNamespace>> {
+        // Returns the manifest-backed namespace so callers can operate on child
+        // namespaces (multi-level table ids) through the client. Root-table soft-delete
+        // lifecycle (table_status/purge) is reached via the V1 root namespace internally.
        self.namespace_database.namespace_client().await
    }

@@ -2615,4 +2605,67 @@ mod tests {
            .unwrap();
        assert!(post_drop.tables.is_empty());
    }
+
+    /// Root-table drop is a soft-delete routed through the V1 namespace: the table is
+    /// hidden from listing/open but its data survives until purged, and re-creating it
+    /// revives it. Verifies the consolidation end-to-end at the ListingDatabase level.
+    #[tokio::test]
+    async fn test_root_table_soft_delete_lifecycle() {
+        let (_tempdir, db) = setup_database().await;
+        let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int32, false)]));
+        let create = |name: &str| CreateTableRequest {
+            name: name.to_string(),
+            namespace_path: vec![],
+            data: Box::new(RecordBatch::new_empty(schema.clone())) as Box<dyn Scannable>,
+            mode: CreateTableMode::Create,
+            write_options: Default::default(),
+            location: None,
+            namespace_client: None,
+        };
+        let open = |name: &str| OpenTableRequest {
+            name: name.to_string(),
+            namespace_path: vec![],
+            index_cache_size: None,
+            lance_read_params: None,
+            location: None,
+            namespace_client: None,
+            managed_versioning: None,
+        };
+
+        db.create_table(create("t")).await.unwrap();
+        db.drop_table("t", &[]).await.unwrap();
+
+        // Hidden from listing and not openable...
+        #[allow(deprecated)]
+        let names = db.table_names(TableNamesRequest::default()).await.unwrap();
+        assert!(!names.contains(&"t".to_string()));
+        assert!(matches!(
+            db.open_table(open("t")).await,
+            Err(Error::TableNotFound { .. })
+        ));
+
+        // ...but data survives: it shows up as purgable via the V1 root namespace.
+        let root_ns = db
+            .root_namespace_database()
+            .namespace_client()
+            .await
+            .unwrap();
+        let purgable = root_ns.list_purgable_tables(None).await.unwrap();
+        assert_eq!(purgable.len(), 1);
+        assert_eq!(purgable[0].id, vec!["t".to_string()]);
+
+        // Re-creating revives it.
+        db.create_table(create("t")).await.unwrap();
+        db.open_table(open("t")).await.unwrap();
+        #[allow(deprecated)]
+        let names = db.table_names(TableNamesRequest::default()).await.unwrap();
+        assert!(names.contains(&"t".to_string()));
+        assert!(root_ns.list_purgable_tables(None).await.unwrap().is_empty());
+
+        // Drop then purge reclaims it for good.
+        db.drop_table("t", &[]).await.unwrap();
+        let purged = root_ns.purge_tables(None).await.unwrap();
+        assert_eq!(purged, vec![vec!["t".to_string()]]);
+        assert!(root_ns.list_purgable_tables(None).await.unwrap().is_empty());
+    }
 }
--- a/rust/lancedb/src/database/namespace.rs
+++ b/rust/lancedb/src/database/namespace.rs
@@ -583,9 +583,9 @@ impl Database for LanceNamespaceDatabase {
        self.namespace
            .drop_table(drop_request)
            .await
-            .map_err(|e| Error::Runtime {
-                message: format!("Failed to drop table: {}", e),
-            })?;
+            // Preserve TableNotFound (e.g. dropping a non-existent table) rather than
+            // flattening every failure to a generic Runtime error.
+            .map_err(|e| map_namespace_lance_error(e, name))?;

        Ok(())
    }
--- a/rust/lancedb/src/remote/client.rs
+++ b/rust/lancedb/src/remote/client.rs
@@ -459,12 +459,14 @@ impl<S: HttpSend> RestfulLanceDbClient<S> {
        config: &ClientConfig,
    ) -> Result<HeaderMap> {
        let mut headers = HeaderMap::new();
-        headers.insert(
-            HeaderName::from_static("x-api-key"),
-            HeaderValue::from_str(api_key).map_err(|_| Error::InvalidInput {
-                message: "non-ascii api key provided".to_string(),
-            })?,
-        );
+        if !api_key.is_empty() {
+            headers.insert(
+                HeaderName::from_static("x-api-key"),
+                HeaderValue::from_str(api_key).map_err(|_| Error::InvalidInput {
+                    message: "non-ascii api key provided".to_string(),
+                })?,
+            );
+        }
        if region == "local" {
            let host = format!("{}.local.api.lancedb.com", db_name);
            headers.insert(
@@ -1005,6 +1007,33 @@ mod tests {
        assert!(!config_tls.assert_hostname);
    }

+    #[test]
+    fn test_default_headers_skip_empty_api_key() {
+        let headers = RestfulLanceDbClient::<Sender>::default_headers(
+            "",
+            "us-east-1",
+            "db-name",
+            false,
+            &RemoteOptions::default(),
+            None,
+            &ClientConfig::default(),
+        )
+        .unwrap();
+        assert!(!headers.contains_key("x-api-key"));
+
+        let headers = RestfulLanceDbClient::<Sender>::default_headers(
+            "api-key",
+            "us-east-1",
+            "db-name",
+            false,
+            &RemoteOptions::default(),
+            None,
+            &ClientConfig::default(),
+        )
+        .unwrap();
+        assert_eq!(headers.get("x-api-key").unwrap(), "api-key");
+    }
+
    // Test implementation of HeaderProvider
    #[derive(Debug, Clone)]
    struct TestHeaderProvider {
--- a/rust/lancedb/src/remote/db.rs
+++ b/rust/lancedb/src/remote/db.rs
@@ -7,6 +7,7 @@ use std::sync::Arc;
 use async_trait::async_trait;
 use http::StatusCode;
 use lance_io::object_store::StorageOptions;
+use lance_namespace_impls::{DynamicContextProvider, OperationInfo};
 use moka::future::Cache;
 use reqwest::header::CONTENT_TYPE;

@@ -26,7 +27,9 @@ use crate::remote::util::stream_as_body;
 use crate::table::BaseTable;

 use super::ARROW_STREAM_CONTENT_TYPE;
-use super::client::{ClientConfig, HttpSend, RequestResultExt, RestfulLanceDbClient, Sender};
+use super::client::{
+    ClientConfig, HeaderProvider, HttpSend, RequestResultExt, RestfulLanceDbClient, Sender,
+};
 use super::table::RemoteTable;
 use super::util::parse_server_version;

@@ -194,10 +197,66 @@ pub struct RemoteDatabase<S: HttpSend = Sender> {
    uri: String,
    /// Headers to pass to the namespace client for authentication
    namespace_headers: HashMap<String, String>,
+    namespace_context_provider: Option<Arc<dyn DynamicContextProvider>>,
    /// TLS configuration for mTLS support
    tls_config: Option<super::client::TlsConfig>,
 }

+#[derive(Clone)]
+struct NamespaceHeaderProviderContext {
+    header_provider: Arc<dyn HeaderProvider>,
+}
+
+impl std::fmt::Debug for NamespaceHeaderProviderContext {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("NamespaceHeaderProviderContext")
+            .field("header_provider", &"Some(...)")
+            .finish()
+    }
+}
+
+impl DynamicContextProvider for NamespaceHeaderProviderContext {
+    fn provide_context(&self, _info: &OperationInfo) -> HashMap<String, String> {
+        let header_provider = Arc::clone(&self.header_provider);
+        let handle = match std::thread::Builder::new()
+            .name("lancedb-namespace-headers".to_string())
+            .spawn(move || {
+                tokio::runtime::Builder::new_current_thread()
+                    .enable_all()
+                    .build()
+                    .map_err(|e| Error::Runtime {
+                        message: format!(
+                            "Failed to create runtime for namespace header provider: {e}"
+                        ),
+                    })?
+                    .block_on(header_provider.get_headers())
+            }) {
+            Ok(handle) => handle,
+            Err(err) => {
+                log::warn!("Failed to spawn dynamic namespace header provider thread: {err}");
+                return HashMap::new();
+            }
+        };
+
+        let headers = handle.join();
+
+        match headers {
+            Ok(Ok(headers)) => headers
+                .into_iter()
+                .map(|(key, value)| (format!("headers.{key}"), value))
+                .collect(),
+            Ok(Err(err)) => {
+                log::warn!("Failed to get dynamic namespace headers: {err}");
+                HashMap::new()
+            }
+            Err(_) => {
+                log::warn!("Dynamic namespace header provider panicked");
+                HashMap::new()
+            }
+        }
+    }
+}
+
 impl RemoteDatabase {
    pub fn try_new(
        uri: &str,
@@ -228,6 +287,16 @@ impl RemoteDatabase {
            })
            .collect();

+        let namespace_context_provider =
+            client_config
+                .header_provider
+                .as_ref()
+                .map(|header_provider| {
+                    Arc::new(NamespaceHeaderProviderContext {
+                        header_provider: Arc::clone(header_provider),
+                    }) as Arc<dyn DynamicContextProvider>
+                });
+
        let client = RestfulLanceDbClient::try_new(
            &parsed,
            region,
@@ -247,6 +316,7 @@ impl RemoteDatabase {
            table_cache,
            uri: uri.to_owned(),
            namespace_headers,
+            namespace_context_provider,
            tls_config: client_config.tls_config,
        })
    }
@@ -271,6 +341,7 @@ mod test_utils {
                table_cache: Cache::new(0),
                uri: "http://localhost".to_string(),
                namespace_headers: HashMap::new(),
+                namespace_context_provider: None,
                tls_config: None,
            }
        }
@@ -281,11 +352,18 @@ mod test_utils {
            T: Into<reqwest::Body>,
        {
            let client = client_with_handler_and_config(handler, config.clone());
+            let namespace_context_provider =
+                config.header_provider.as_ref().map(|header_provider| {
+                    Arc::new(NamespaceHeaderProviderContext {
+                        header_provider: Arc::clone(header_provider),
+                    }) as Arc<dyn DynamicContextProvider>
+                });
            Self {
                client,
                table_cache: Cache::new(0),
                uri: "http://localhost".to_string(),
                namespace_headers: config.extra_headers.clone(),
+                namespace_context_provider,
                tls_config: config.tls_config.clone(),
            }
        }
@@ -759,9 +837,12 @@ impl<S: HttpSend> Database for RemoteDatabase<S> {
        // Create a RestNamespace pointing to the same remote host with the same authentication headers
        let mut builder = lance_namespace_impls::RestNamespaceBuilder::new(self.client.host())
            .delimiter(&self.client.id_delimiter)
-            // TODO: support header provider
            .headers(self.namespace_headers.clone());

+        if let Some(context_provider) = &self.namespace_context_provider {
+            builder = builder.context_provider(Arc::clone(context_provider));
+        }
+
        // Apply mTLS configuration if present
        if let Some(tls_config) = &self.tls_config {
            if let Some(cert_file) = &tls_config.cert_file {
@@ -781,6 +862,14 @@ impl<S: HttpSend> Database for RemoteDatabase<S> {
    }

    async fn namespace_client_config(&self) -> Result<(String, HashMap<String, String>)> {
+        if self.namespace_context_provider.is_some() {
+            return Err(Error::NotSupported {
+                message:
+                    "Cannot export a namespace client config when dynamic headers are configured; use LanceDB connection namespace methods instead"
+                        .to_string(),
+            });
+        }
+
        let mut properties = HashMap::new();
        properties.insert("uri".to_string(), self.client.host().to_string());
        properties.insert("delimiter".to_string(), self.client.id_delimiter.clone());
@@ -832,12 +921,13 @@ impl From<StorageOptions> for RemoteOptions {

 #[cfg(test)]
 mod tests {
-    use super::build_cache_key;
+    use super::{NamespaceHeaderProviderContext, build_cache_key};
    use std::collections::HashMap;
    use std::sync::{Arc, OnceLock};

    use arrow_array::{Int32Array, RecordBatch};
    use arrow_schema::{DataType, Field, Schema};
+    use lance_namespace_impls::{DynamicContextProvider, OperationInfo};

    use crate::connection::ConnectBuilder;
    use crate::{
@@ -1702,6 +1792,75 @@ mod tests {
        assert!(namespace_client.is_ok());
    }

+    #[test]
+    fn test_namespace_header_provider_context_maps_headers() {
+        #[derive(Debug)]
+        struct TestHeaderProvider;
+
+        #[async_trait::async_trait]
+        impl HeaderProvider for TestHeaderProvider {
+            async fn get_headers(&self) -> crate::Result<HashMap<String, String>> {
+                Ok(HashMap::from([(
+                    "authorization".to_string(),
+                    "Bearer token".to_string(),
+                )]))
+            }
+        }
+
+        let context_provider = NamespaceHeaderProviderContext {
+            header_provider: Arc::new(TestHeaderProvider) as Arc<dyn HeaderProvider>,
+        };
+
+        let context =
+            context_provider.provide_context(&OperationInfo::new("list_tables", "namespace"));
+
+        assert_eq!(
+            context.get("headers.authorization"),
+            Some(&"Bearer token".to_string())
+        );
+    }
+
+    #[tokio::test]
+    async fn test_namespace_client_supports_dynamic_headers() {
+        #[derive(Debug)]
+        struct TestHeaderProvider;
+
+        #[async_trait::async_trait]
+        impl HeaderProvider for TestHeaderProvider {
+            async fn get_headers(&self) -> crate::Result<HashMap<String, String>> {
+                Ok(HashMap::from([(
+                    "authorization".to_string(),
+                    "Bearer token".to_string(),
+                )]))
+            }
+        }
+
+        let client_config = ClientConfig {
+            header_provider: Some(Arc::new(TestHeaderProvider) as Arc<dyn HeaderProvider>),
+            ..Default::default()
+        };
+
+        let conn = Connection::new_with_handler_and_config(
+            |_| {
+                http::Response::builder()
+                    .status(200)
+                    .body(r#"{"tables": []}"#)
+                    .unwrap()
+            },
+            client_config,
+        );
+
+        let namespace_client = conn.namespace_client().await;
+        assert!(namespace_client.is_ok());
+
+        match conn.namespace_client_config().await {
+            Err(Error::NotSupported { message })
+                if message.contains("dynamic headers are configured") => {}
+            Err(err) => panic!("expected NotSupported, got {err:?}"),
+            Ok(_) => panic!("expected namespace_client_config to reject dynamic headers"),
+        }
+    }
+
    /// Integration tests using RestAdapter to run RemoteDatabase against a real namespace server
    mod rest_adapter_integration {
        use super::*;
Author	SHA1	Message	Date
Will Jones	2d7b8f6173	feat(listing): soft-delete root tables via an embedded V1 namespace Root-table lifecycle in a ListingDatabase now flows through an embedded V1 (manifest-disabled) DirectoryNamespace, so a root drop is a soft-delete rather than an immediate remove_dir_all: - drop_table writes a delete marker and leaves the data for a later purge (TTL). - create_table on a soft-deleted name revives it (clears the marker under the namespace's lifecycle lock, then overwrites via the native create path). - open_table / table_names / list_tables treat soft-deleted tables as absent. - table listing is now a single O(1) read_dir in the namespace instead of a per-table probe here. A ListingDatabase now holds two embedded namespaces: the existing manifest-backed `namespace_database` for child namespaces (multi-level table ids), and a new `root_namespace_database` (V1, manifest-off) that owns root soft-delete/purge/ table_status. `namespace_client()` still returns the manifest namespace so child namespace ops are unaffected. Also preserves TableNotFound through LanceNamespaceDatabase::drop_table instead of flattening it to a generic Runtime error, so dropping a missing table reports the right error. Removes the now-dead native drop_tables / commit-handler path and the object_store/base_path fields it needed. Depends on lance-format/lance#7541. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 17:08:16 -07:00
Will Jones	99a1c04d2a	chore: temporarily pin lance to soft-delete PR rev Points the lance workspace deps at lance-format/lance rev 4acefffd5 (PR lance-format/lance#7541), which adds the soft-delete lifecycle trait methods this change depends on. Revert to a release tag once a new lance beta is published. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 17:08:07 -07:00
lancedb automation	027f60a8b2	chore: update lance dependency to v9.0.0-beta.8	2026-06-24 23:29:23 +00:00
Jack Ye	fe287dc98c	fix(remote): support namespace clients with dynamic headers Bridge LanceDB dynamic header providers into Lance Namespace dynamic context providers for live remote namespace clients.	2026-06-24 15:30:00 -07:00
Jack Ye	411568b72c	fix(remote): omit empty api key header (#3573 ) ## Summary Skip inserting the x-api-key header when the configured API key is empty. This lets bearer-token or other dynamic-header authentication avoid sending an empty static API key header alongside the real auth header.	2026-06-24 13:25:59 -07:00
LanceDB Robot	ebf8d55ede	chore: update lance dependency to v9.0.0-beta.4 (#3570 ) Bumps the Lance dependencies to v9.0.0-beta.4 and refreshes the generated lockfile metadata. No compatibility fixes were required beyond the dependency updates. Triggered by https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.4	2026-06-24 10:16:29 -05:00
Raphael Malikian	0ba70d96c3	fix: add missing stacklevel=2 to warnings.warn() and fix broken message concatenation (Fixes #3563 ) (#3564 ) Fixes #3563 ## Summary - Add `stacklevel=2` to 10 `warnings.warn()` calls across 4 files - Fix broken message concatenation in `table.py` where the second string was incorrectly passed as the `category` parameter ## Problem Multiple `warnings.warn()` calls in the `python/lancedb/` codebase were missing the `stacklevel` parameter. Without `stacklevel=2`, warnings point to library internals instead of the caller's code, making it impossible for users to identify which of their function calls triggered the warning. Additionally, two calls in `table.py` (lines 3411 and 3420) had a more serious bug: the deprecation message was split across two separate string arguments, causing the second string to be passed as the `category` parameter instead of being concatenated with the first string. This would cause `TypeError` when the warning was triggered. ## Changes \| File \| Fixes \| Description \| \|------\|-------\|-------------\| \| `embeddings/colpali.py` \| 1 \| Add `stacklevel=2` to `use_token_pooling` deprecation warning \| \| `remote/db.py` \| 3 \| Add `stacklevel=2` to `request_thread_pool`, `connection_timeout`, `read_timeout` deprecation warnings \| \| `remote/table.py` \| 3 \| Add `stacklevel=2` to `cleanup_old_versions`, `compact_files`, `optimize` no-op warnings \| \| `table.py` \| 3 \| Fix broken message concatenation for `data_storage_version` and `enable_v2_manifest_paths` deprecation warnings + add `stacklevel=2` to `retrain` deprecation warning \| ## Verification ```python # All warnings.warn() calls now have stacklevel python3 -c "import ast, os; ..." # Result: All warnings.warn() calls now have stacklevel! ``` ## Changelog \| Date \| Change \| Author \| \|------\|--------\|--------\| \| 2026-06-20 \| Fix missing stacklevel=2 in 10 warnings.warn() calls + fix broken message concatenation \| rtmalikian \| ### Files Changed - `python/python/lancedb/embeddings/colpali.py` — Add stacklevel=2 - `python/python/lancedb/remote/db.py` — Add stacklevel=2 to 3 deprecation warnings - `python/python/lancedb/remote/table.py` — Add stacklevel=2 to 3 no-op warnings - `python/python/lancedb/table.py` — Fix broken message concatenation + add stacklevel=2 ### Verification - AST-based audit confirms all `warnings.warn()` calls now include `stacklevel=2` - Syntax check passes for all 4 modified files --- About the Author: Raphael Malikian — Clinical AI Solutions Architect. I specialise in building and fixing AI/ML systems for healthcare, including vector databases, RAG pipelines, and clinical NLP. If you need help with your project or think I can add value to your organisation, feel free to reach out — I'd love to connect. 📧 rtmalikian@gmail.com 🔗 GitHub: https://github.com/rtmalikian 🔗 LinkedIn: http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a --- Disclosure: This code was developed with assistance from Hermes Agent (Nous Research). All changes were reviewed, tested against the actual codebase, and verified for correctness. Signed-off-by: rtmalikian <rtmalikian@gmail.com>	2026-06-23 13:42:59 -07:00
Lance Release	0749532c3c	Bump version: 0.31.0-beta.1 → 0.31.0-beta.2	2026-06-23 16:23:08 +00:00