chore: update lance dependency to v15.0.0-beta.6 (#4742)

Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.

Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.

Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.

---------

Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

python/python/tests/test_table.py

@@ -1417,6 +1417,7 @@ def test_branches_preserve_namespace(tmp_path):
         "lance"
     )  # namespace_path routes through lance's DirectoryNamespace
     db = lancedb.connect(tmp_path)
+    db.create_namespace(["ns1"])
     table = db.create_table("t", [{"id": 1}], namespace_path=["ns1"])
     assert table.namespace == ["ns1"]