mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 12:08:35 +00:00
9707966943ec2925f8f7c5572997ec4e00fccb34
2766 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lancedb-gatefixer[bot]
|
9707966943 |
test(rust): cover named memory databases on Windows (#3839)
## Summary\n\n- add a create-table regression for a named database\n- assert that the derived table URI uses URL separators\n- restore the four query tests that were moved to temporary files for #1051\n\n## Root cause\n\n historically joined table names with . On Windows this inserted a backslash into , so Lance interpreted the URI as an invalid local filename. The production URI builder now preserves forward slashes for URI schemes; this change restores the issue-specific tests and adds direct regression coverage for table creation and the derived URI.\n\n## Validation\n\n- \n- \n- (passes with four pre-existing warnings in unrelated remote-table code)\n- running 814 tests ....................................................................................... 87/814 .....................................i................................................. 174/814 ....................................................................................... 261/814 ....................................................................................... 348/814 ....................................................................................... 435/814 ....................................................................................... 522/814 ....................................................................................... 609/814 ....................................................................................... 696/814 ....................................................................................... 783/814 ............................... test result: ok. 813 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 7.76s running 39 tests ....................................... test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s running 6 tests ...... test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s running 5 tests ..... test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 2 tests .. test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 2 tests .. test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s (867 passed, 1 ignored)\n- focused named-memory create and restored query tests\n\nFixes #1051\n\n<!-- lance-gatekeeper-fix:v1 agent=5ddf7a9520292b4cbaa58b9ea5a1fe76 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Wyatt Alt
|
62fe413a52 |
fix: percent-encode index names in per-index remote REST paths (#3840)
Nothing validates index names, so a `/` in one is reachable, and the remote client interpolates it straight into the URL, splitting the path so the router 404s. The index then reads back as missing and cannot be dropped, while `create_index` keeps succeeding because it sends the name in the body. Encode at the three affected sites, mirroring `fetch_blob_files`. The shared Rust client covers all bindings. |
||
|
lancedb-gatefixer[bot]
|
1493ece3de |
test(node): cover remote table server errors (#3841)
## Summary - add a public Node API regression test for JSON server errors from remote table operations - verify countRows reports the server message instead of an ArrayBuffer decoding TypeError ## Root cause and fix The former TypeScript remote HTTP client passed an Axios-decoded JSON error object to TextDecoder, which masked the server response with an ArrayBuffer TypeError. The current Rust-backed remote client consumes non-success response bodies as text and propagates them through the Node error chain. This test exercises that corrected path through countRows and prevents the original failure from regressing. ## Validation - pnpm build - pnpm lint-ci - pnpm test --runInBand __test__/remote.test.ts - pnpm run docs Fixes #825 <!-- lance-gatekeeper-fix:v1 agent=91591c3d6b065796e6166664ef638aa7 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
e6444ecc05 |
fix(rust): handle missing mirrored copy sources (#3843)
## Summary - treat `NotFound` from the mirrored secondary copy as a cache miss while preserving every other secondary error - perform the durable primary copy after either a successful secondary copy or a secondary cache miss - cover both an initially missing secondary manifest and eviction immediately before the secondary copy ## Root cause Readers can use process-local secondary stores that do not contain a staging manifest written by another process, or that evict it before finalization. `MirroringObjectStore::copy_opts` propagated that secondary `NotFound`, so older object_store versions could loop indefinitely and the locked version aborted before performing the durable primary copy. ## Validation - `cargo fmt --all -- --check` - `cargo test --quiet --features remote -p lancedb io::object_store::test::test_copy_when -- --nocapture` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` Fixes #1176 <!-- lance-gatekeeper-fix:v1 agent=636210af9dcd25b6dceadebd2fcafc6f generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
cc0139c136 |
test(node): cover foreign Float64 vector schema workflow (#3844)
## Summary - add an end-to-end regression for schemas created by a different Apache Arrow package instance - cover seeded table creation, filtered scanning, and Float64 vector search across Arrow 15–18 ## Root cause Apache Arrow's runtime identity checks historically rejected schemas created by another installed Arrow instance, producing the constructor failures reported in the issue. LanceDB's peer dependency and foreign-schema sanitization now handle that boundary, but the complete reported workflow was only covered by separate unit tests. This regression keeps the repaired behavior protected end to end. ## Validation - `pnpm exec jest --runInBand __test__/table.test.ts` (281 passed) - `pnpm lint-ci` - `pnpm build` - `pnpm run docs` Fixes #882 <!-- lance-gatekeeper-fix:v1 agent=43b19dea581cfbc83ee1e9ed21a335a6 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
b20696ef9c |
fix(remote): validate cloud DNS hostnames (#3845)
## Summary - validate the generated LanceDB Cloud hostname during connection setup - return a clear invalid-input error for empty, overlong, or oversized DNS names before network resolution - add Rust and Python regression coverage for malformed `db://` authorities ## Root cause The `db://` authority and region were interpolated into the Cloud API hostname without DNS length validation. Empty or overlong labels therefore reached the resolver and surfaced as an opaque IDNA `UnicodeError` instead of a useful connection error. ## Validation - `cargo test --quiet --features remote -p lancedb test_rejects_invalid_cloud_dns_hostname --lib` - `cargo check --quiet --features remote --tests --examples` - `uv run --no-sync --extra tests pytest python/tests/test_remote_db.py::test_async_remote_db python/tests/test_remote_db.py::test_connect_rejects_invalid_cloud_dns_hostname -q` - `cargo fmt --all -- --check` - `ruff check .` - `ruff format --check python/python/tests/test_remote_db.py` Fixes #799 <!-- lance-gatekeeper-fix:v1 agent=4d1597b3d244b58f0603ed40a8a59cf9 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
772bdeced8 |
fix(rust): prevent vector optimize regression after deletes (#3848)
## Summary - Adds a LanceDB regression for stable row IDs, scattered deletes, IVF_RQ, and default index optimization. - Verifies optimization completes and preserves the expected live-row count. ## Root cause Lance 3.0.1 built the stable-row-ID address list by dropping deleted IDs while retaining the original ID list. The subsequent positional zip misaligned IDs and addresses, so vector partition joins requested deleted rows and failed with batch.num_rows() != chunk.len(). Lance PR https://github.com/lance-format/lance/pull/7704 corrected the generic filter, and the LanceDB dependency currently pinned on main contains that correction. ## Fix Add regression coverage at the Rust Table optimize surface using the IVF_RQ configuration from the report. This locks the upstream correction into the LanceDB workflow that originally crashed. ## Validation - cargo fmt --all -- --check - cargo test --quiet --features remote -p lancedb table::optimize::tests (14 passed) - cargo check --quiet --features remote --tests --examples - cargo clippy --quiet --features remote --tests --examples -p lancedb Fixes #3330 <!-- lance-gatekeeper-fix:v1 agent=4c2c25373942aab9ba9f7444977de7e3 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
c1a3fa7f51 |
fix(python): preserve repeated indexed merge inserts (#3850)
## Summary - add a Python regression test for two partial-schema merge inserts against the same BTree-indexed rows - verify repeated updates retain one copy of every row and the final update values ## Root cause Lance 4.0, used by LanceDB 0.30.2, removed a rewritten fragment from the index bitmap while stale BTree entries for that fragment remained searchable. The next merge found each target through both the stale index and the unindexed-fragment scan, producing the ambiguous-match error. Lance fixed the root cause in lance-format/lance#6563 by applying the fragment-bitmap allow-list to index results, and the Lance release pinned by current LanceDB includes that fix. This test preserves the corrected behavior through the Python API. ## Validation - `cd python && uv run --extra tests pytest python/tests/test_table.py -k merge_insert -q` (9 passed) - `cd python && uv run --extra tests --extra dev ruff format --check python/tests/test_table.py` - `cd python && uv run --extra tests --extra dev ruff check python/tests/test_table.py` Repository-wide Ruff also reports 20 pre-existing violations in untouched CI and plugin scripts. Fixes #3280 <!-- lance-gatekeeper-fix:v1 agent=ee6b9565f9780712026076930566f116 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
0ba82873c5 |
fix(python): cover nullable list v2.2 decoding (#3853)
## Summary - add a minimized regression for mostly-null `list<float32>` data at the v2.2 structural page boundary - verify scans preserve all 64,885 rows, including 64,668 null list values ## Root cause Lance 3.0.0 sliced repetition/definition state using top-level row offsets in the complex all-null decoder. At this page boundary, the list and validity children were materialized at different lengths. The current Lance dependency contains the upstream decoder repair; this test locks that behavior into the LanceDB Python suite without duplicating decoder logic. ## Validation - reproduced the attached 1,892,466-row case on `lancedb==0.30.0` with `expected 1024 got 285` - verified the full attachment reads on the current branch - `python/.venv/bin/ruff format --check python/python/tests/test_table.py` - `python/.venv/bin/ruff check .` - `cd python && uv run --extra tests pytest python/tests/test_table.py::test_read_mostly_null_list_v2_2_page_boundary -q` - `cd python && uv run --extra tests pytest python/tests/test_table.py -q` (137 passed) Fixes #3194 <!-- lance-gatekeeper-fix:v1 agent=0445adc5303a3302152cea3d2110bed1 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
3af51541a0 |
test(rust): cover fixed-size binary merge insert regression (#3854)
## Summary - add a LanceDB regression for `merge_insert` with a non-nullable `FixedSizeBinary` column - exercise matched updates, unmatched inserts, and source-missing deletes - assert the exact merge statistics and final row count ## Root cause The Arrow `take` kernel previously ignored nulls in the index array for `FixedSizeBinary`. DataFusion uses that kernel while constructing outer-join results, so the join behind `when_not_matched_by_source_delete` could place invalid values into non-nullable columns. The current Arrow dependency contains the upstream fix; this test locks the corrected behavior at the LanceDB API boundary. ## Validation - `cargo fmt --all -- --check` - `cargo test --quiet --features remote --tests` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` Fixes #2869 <!-- lance-gatekeeper-fix:v1 agent=e275446044185ef4e8cf88da6af3e70b generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
2c06a48bd8 |
test(python): cover Arrow buffer release after add (#3860)
## Summary - add deterministic regression coverage that `Table.add()` releases backing Arrow buffers without cyclic garbage collection - track the foreign buffer owner rather than RSS, separating live input retention from allocator high-water behavior - preserve the bounded-lifetime behavior of the Scannable writer that superseded the historical preprocessing path ## Root cause The historical Python preprocessing/write path produced a high allocator RSS while ingesting very wide IPC batches. The current Scannable writer releases each input buffer when `Table.add()` completes; remaining RSS is allocator high-water rather than a live Arrow reference. The resolved behavior had no regression coverage, so a future native lifetime regression could silently reintroduce the original failure mode. ## Validation - `uv run --extra tests --extra dev maturin develop` - `uv run --project python --extra tests pytest python/python/tests/test_table.py::test_add python/python/tests/test_table.py::test_add_releases_arrow_buffers_without_gc -q` - `uv run --project python --extra dev ruff format --check python/python/tests/test_table.py` - `uv run --project python --extra dev ruff check .` Fixes #2512 <!-- lance-gatekeeper-fix:v1 agent=29226408a8d07da592daf341d5384e37 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
ac8b28c010 |
fix(python): support nullable pandas merge input (#3864)
## Summary - add an end-to-end Python regression for pandas DataFrame inputs merged into a table created from a Pydantic model - verify reordered, nullable Arrow source fields can update and insert into a non-nullable target schema when the values contain no nulls ## Root cause Lance merge_insert previously compared source schema nullability with the target, unlike add. The upstream fix now pinned by LanceDB ignores declared nullability during schema compatibility and validates actual null values at write time. LanceDB lacked regression coverage for the full pandas-to-Pydantic path, so this test locks in the correct behavior without falsifying the input schema nullability. ## Validation - 5 focused merge-insert tests passed - Ruff lint passed for the repository - Ruff format check passed for the changed file - git diff --check passed Fixes #2366 <!-- lance-gatekeeper-fix:v1 agent=f897fccfa206620c8a2acdc3bcd1c21f generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
173f889d2a |
test(python): cover stale scalar prefilters in hybrid search (#3865)
## Summary - capture the stale-index state behind the reported fixed-size-binary panic: the vector and FTS indices cover newer fragments while the BTree prefilter does not - verify vector, FTS, and hybrid searches return matches from both scalar-indexed and unindexed fragments without panicking - preserve binding-level coverage for the Lance fix in https://github.com/lance-format/lance/pull/3768, which restricts incomplete scalar prefilters when search indices are further ahead The production root cause is in Lance and the current LanceDB dependency already contains that fix, so this change adds the missing LanceDB Python regression coverage. ## Validation - `cd python && uv run --no-sync pytest python/tests/test_hybrid_query.py::test_hybrid_query_with_stale_fixed_size_binary_prefilter -q` - `cd python && uv run --no-sync pytest python/tests/test_hybrid_query.py -q` - `python/.venv/bin/ruff check .` - `python/.venv/bin/ruff format --check python/python/tests/test_hybrid_query.py` Fixes #2370 <!-- lance-gatekeeper-fix:v1 agent=5d16e59b9e513fd9247e0698732fa283 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
03b52e5877 |
test(node): cover fixed-size list schemas with typed arrays (#3866)
## Summary - cover explicit FixedSizeList schemas populated from Float32Array values - verify the original vector.0 failure stays fixed across Arrow 15, 16, 17, and 18 ## Root cause and fix In v0.16, schema subset inference treated typed-array vectors as nested objects and looked up numeric paths such as vector.0, which do not exist in a FixedSizeList schema. Current typed-array handling correctly recognizes ArrayBuffer views as vector values instead of traversing their elements. This change adds the missing regression coverage for the reported explicit-schema path so that behavior cannot regress unnoticed. ## Validation - pnpm test __test__/arrow.test.ts --runInBand - pnpm lint - pnpm build - pnpm run docs - pnpm test --runInBand (681 passed, 5 skipped) Fixes #2134 <!-- lance-gatekeeper-fix:v1 agent=1d548cb70f6df110ce0a5b119395b52a generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
798e5364fb |
test(python): cover VoyageAI text source routing (#3872)
## Summary - add fast regression coverage for VoyageAI `voyage-3` source embeddings - verify table text uses `client.embed` and never `client.multimodal_embed` ## Root cause The original VoyageAI source-embedding path treated table source values as images and always invoked the multimodal API. Production routing was corrected by later merged changes, but the table regression was covered only by API-gated slow tests. This test locks the corrected text routing into the regular unit suite. ## Validation - `cd python && uv run --extra tests pytest python/tests/test_voyageai_embeddings.py -q` - `uv run --project python --extra tests --extra dev ruff format --check python/python/tests/test_voyageai_embeddings.py` - `uv run --project python --extra tests --extra dev ruff check .` Fixes #2059 <!-- lance-gatekeeper-fix:v1 agent=49b9e2daeed95a78ce827e2bf90abda0 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
f1f34dfdd3 |
fix(python): instruct dimension probe for instructor embeddings (#3874)
## Summary - pass an Instructor-compatible `[instruction, text]` pair when detecting embedding dimensions - add a regression test that verifies the dimension probe uses the configured source instruction ## Root cause `InstructorEmbeddingFunction.ndims()` encoded a bare string even though Instructor models require instruction/text pairs. With affected `sentence-transformers` versions, the bare input omitted `instruction_mask` and raised `KeyError` while defining the LanceDB schema. ## Validation - `uv run --extra tests pytest python/tests/test_embeddings.py -q` (`14 passed, 9 skipped`) - `uv run --project python --extra tests --extra dev ruff format --check python/python/lancedb/embeddings/instructor.py python/python/tests/test_embeddings.py` - `uv run --project python --extra tests --extra dev ruff check .` Fixes #2041 <!-- lance-gatekeeper-fix:v1 agent=4b05e0d9f3eef17bccfb446e788294f4 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
123c921c4f |
test(python): cover sliced nullable table search (#3875)
## Summary - add a Python regression for vector search over a sliced Arrow table with nullable scalar columns - verify the nearest row retains its non-null score values after the table is written ## Root cause Lance 0.19.2 deep-copied a validity bitmap without preserving its non-zero bit offset. For a sliced nullable table, scalar values and vectors began at the slice while the copied validity bitmap began at the parent table's first row. That made valid score values appear null even though the corresponding vector stayed intact. The upstream Lance repair is already present in the current dependency; this adds a LanceDB-level guard for the reported create/search path. ## Validation - reproduced on Python 3.12 with LanceDB 0.16.0, pylance 0.19.2, PyArrow 18.0.0, and Polars 1.14.0 - `uv run --project python --extra dev ruff format --check python/python/tests/test_table.py` - `uv run --project python --extra dev ruff check .` - `cd python && uv run --extra tests pytest python/tests/test_table.py::test_search_preserves_nulls_from_sliced_arrow_table -q` Fixes #1879 <!-- lance-gatekeeper-fix:v1 agent=bfa0551793f8e3cf3980cf64ad89908a generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
99a68db78c |
test(rust): cover concurrent appends during compaction (#3878)
## Summary - add a LanceDB core regression for compaction overlapping appends through separate table handles - verify concurrent commits preserve fragment ID order on an indexed table - run the follow-up compaction that exposed the original row-ID ordering failure and verify all rows remain ## Root cause Older Lance versions could reserve fragment IDs for compaction, allow concurrent appends to commit later IDs, and then commit the reserved compaction fragments at the end of the manifest. A later compaction could consequently receive row IDs out of order. Current Lance sorts fragments at the transaction boundary; this adds the missing LanceDB-level regression coverage for the Node-visible concurrency contract. ## Validation - `cargo fmt --all` - focused regression passed once with output and 20 repeated runs - `cargo test --quiet --features remote -p lancedb table::optimize::tests` (14 passed) - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` (867 passed, 1 ignored) Fixes #1498 <!-- lance-gatekeeper-fix:v1 agent=93aaefb15507dca52d064e15388773d7 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
9e73d440a3 |
test(python): cover schema-only vector table creation (#3882)
## Summary - make the existing #1968 regression explicitly assert that schema-only table creation succeeds - verify the new table has zero rows and preserves the requested fixed-size vector schema before accepting subsequent data ## Root cause In v0.16.0, schema-only table creation sent an empty table through vector sanitization, which calculated a remainder using `len(data)` and raised `ZeroDivisionError`. Later refactors removed that runtime path, but the issue-specific regression only asserted the final row count after a subsequent add. This change makes the reported operation and its expected empty-table state explicit so the original defect remains directly covered. ## Validation - `uv run --extra tests pytest python/tests/test_table.py::test_create_table_without_data_with_vector_schema -q` - `uv --project python run --extra tests --extra dev ruff format --check python/python/tests/test_table.py` - `uv --project python run --extra tests --extra dev ruff check .` - `git diff --check` Fixes #1968 <!-- lance-gatekeeper-fix:v1 agent=b8ec6f40f4bba2f9beeaaae12233e5c4 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
3956d9dbfa |
fix(python): prevent OpenSSL linkage in Linux wheels (#3877)
## Summary - select rustls with native certificate roots explicitly for LanceDB's remote HTTP client - add a Linux regression test that rejects `libssl` or `libcrypto` dependencies in the built Python extension ## Root cause The Python remote client originally enabled reqwest's native TLS backend. During manylinux wheel repair, that caused OpenSSL 1.1 libraries to be bundled into the wheel. Loading those libraries on RHEL 9 with FIPS enabled aborts during the OpenSSL self-test before `import lancedb` can complete. LanceDB has since moved away from native TLS, but its own reqwest dependency relied on transitive rustls feature selection and the built extension had no regression guard. This change makes rustls selection explicit and tests the produced Linux native module's dynamic dependencies. ## Validation - `uv run --no-sync pytest python/tests/test_import.py -q` - `ruff format --check python` - `ruff check .` - `cargo fmt --all -- --check` - `cargo check --quiet --features remote --tests --examples` - `ldd python/lancedb/_lancedb.abi3.so` (no `libssl` or `libcrypto` dependency) - verified the resolved Python Rust dependency graph contains rustls and no `openssl-sys` or `native-tls` Fixes #1884 <!-- lance-gatekeeper-fix:v1 agent=31f916c7ac5c072bbbd54f3539d24f71 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
16e1967efc |
fix(python): align wheel ABI with supported versions (#3884)
## Summary - align the PyO3 runtime and build ABI floor with the declared Python 3.10 minimum - add a regression test that keeps both ABI features synchronized with `requires-python` ## Root cause The Python 3.10 support-floor update originally changed PyO3 to `abi3-py310`, but a later dependency update reverted both PyO3 features to `abi3-py39`. Published Windows wheels were consequently tagged `cp39-abi3` while importing `PyCMethod_New`, a stable-ABI procedure absent from CPython 3.9.0 and 3.9.1. Windows reports that mismatch as “The specified procedure could not be found” while loading `_lancedb`. Restoring `abi3-py310` makes the wheel tag and native imports agree with the package metadata and prevents future wheels from advertising unsupported Python 3.9 compatibility. ## Validation - `uv run --extra tests pytest python/tests/test_package_metadata.py -q` - `uv run --extra tests --extra dev ruff format --check .` - `uv run --extra tests --extra dev ruff check .` - `cargo fmt --all` - `cargo check --quiet -p lancedb-python` - `uvx --from maturin==1.12.4 maturin build --profile ci` (built `lancedb-0.37.1b0-cp310-abi3-manylinux_2_34_x86_64.whl`) Fixes #2051 <!-- lance-gatekeeper-fix:v1 agent=d66c984498190d2207d1c5126cba5047 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
27dd92c67e |
test(python): cover debugger-safe connection inspection (#3880)
## Summary - expand the synchronous debugger regression to enumerate every exposed connection attribute while the Python background loop is unavailable - retain direct representation checks for connections and tables ## Root cause VS Code debugpy suspends Python threads at a breakpoint and inspects local variables. Connection representation and property access previously dispatched asynchronous work to LanceDBBackgroundEventLoop and waited for the suspended loop thread, deadlocking the debugger. The production safeguards landed in #3620 and #3788; this regression exercises debugger-style whole-object expansion so a newly exposed property cannot reintroduce the original failure. ## Validation - uv run --no-sync pytest python/tests/test_db.py::test_sync_debugger_inspection_does_not_use_background_loop python/tests/test_db.py::test_read_consistency_interval_does_not_use_background_loop -q (2 passed) - uv run --no-sync pytest python/tests/test_db.py -q (48 passed) - python/.venv/bin/ruff format --check python/python/tests/test_db.py - python/.venv/bin/ruff check . - git diff --check Fixes #3611 <!-- lance-gatekeeper-fix:v1 agent=cdf4b39b2ce2ccb3eb5fe501acae77bb generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
9e2e711c7a |
test(python): cover OpenAI registry variable round-trip (#3863)
## Summary - replace the synthetic registry-variable metadata test with the OpenAI embedding function reported in #2387 - verify the resolved API key survives table metadata reconstruction - assert the OpenAI client receives the resolved key while serialized metadata retains the variable reference ## Root cause LanceDB 0.22.0 reconstructed embedding functions from table metadata with the model constructor, bypassing EmbeddingFunction.create and leaving the literal $var:api_key placeholder in OpenAI configuration. The production path was corrected for duplicate #2181 by #2640; this change gives that fix direct, network-free OpenAI regression coverage for #2387. ## Validation - uv run --extra tests pytest python/tests/test_embeddings.py -q (13 passed, 9 skipped) - uv run --project python --extra dev ruff check . - uv run --project python --extra dev ruff format --check python/python/tests/test_embeddings.py - git diff --check Fixes #2387 <!-- lance-gatekeeper-fix:v1 agent=d453b1b9b2a298a776f2e4ea1b1449b5 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
c3176a47ce |
fix(python): report unsplittable IVF partition errors (#3846)
## Summary - add Python regression coverage for an IVF build that cannot form all requested non-empty partitions - verify hierarchical k-means returns an actionable RuntimeError instead of panicking or silently creating a degenerate index - exercise the current Lance v10.1.0-beta.1 dependency, which contains the upstream error-return fix ## Root cause Hierarchical k-means previously guarded a shortfall in generated clusters with only a debug assertion. Debug builds panicked, while release builds could silently publish an index with many empty partitions. The upstream Lance fix now returns a descriptive error and is already included in the dependency pinned on main; this test locks in propagation through the LanceDB Python API. ## Validation - uv run --extra tests pytest python/tests/test_index.py -q (24 passed) - uv run --extra tests pytest python/tests/test_index.py::test_create_ivf_index_reports_unsplittable_partitions -q (1 passed) - python/.venv/bin/ruff format python/python/tests/test_index.py - python/.venv/bin/ruff check . - git diff --check Fixes #3649 <!-- lance-gatekeeper-fix:v1 agent=a4d34448a9d350a3e2e659f33f5db6f2 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
7357d63e87 |
fix(python): guard concurrent table deletes (#3787)
<!-- lance-gatekeeper-fix:v1 agent=5c80c44c083b3b8ad0da595419d468fc generation=1 --> ## Root cause The legacy synchronous Python table called `delete` on a shared, mutable `lance.Dataset`. Concurrent table operations could hold a PyO3 borrow while delete requested an exclusive borrow, producing `RuntimeError: Already borrowed`. The current async-backed binding fixes this by cloning its thread-safe Rust table handle before awaiting, but that concurrency contract had no regression coverage. ## Fix - Document why delete must clone the Rust table handle before entering its async future. - Add a barrier-synchronized regression test that deletes distinct rows through one shared table from eight Python threads. - Verify every delete commits exactly one row, every commit gets a distinct version, and no rows remain. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo fmt --all -- --check` - `uv run --extra tests --extra dev ruff format --check python/tests/test_table.py` - `uv run --extra tests --extra dev ruff check python/tests/test_table.py` - `uv run --extra tests --extra dev pytest python/tests/test_table.py::test_concurrent_deletes_are_thread_safe python/tests/test_table.py::test_delete python/tests/test_table.py::test_delete_expr python/tests/test_table.py::test_delete_expr_async -q` (4 passed) - Manual stress reproduction: 100 concurrent deletes on one table completed at versions 2–101 with zero rows remaining. Fixes #530 Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
624a75edf7 |
fix(python): avoid debugger deadlock during connection inspection (#3788)
## Summary - cache the immutable read consistency interval on synchronous connection wrappers - keep debugger property expansion from dispatching to the background event loop - cover direct connections and wrappers reconstructed from native connections ## Root cause The debugger expands connection variables by evaluating properties after suspending all Python threads. `LanceDBConnection.read_consistency_interval` dispatched a coroutine to `LanceDBBackgroundEventLoop` and synchronously waited for it, but that loop thread was also suspended, causing a deadlock. ## Validation - `uv run --no-sync pytest python/tests/test_db.py -q` (48 passed) - `ruff format --check python/python/lancedb/db.py python/python/tests/test_db.py` - `ruff check .` - `git diff --check` Fixes #3773 <!-- lance-gatekeeper-fix:v1 agent=e2e612236d722d926f64245d3f682bbc generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Justin Miller
|
c7ea91f3ea |
test: cover blob null/empty preservation across Table::optimize (#3774)
## Description `Table::optimize()` compacts through `lance::dataset::optimize::compact_files` (`rust/lancedb/src/table/optimize.rs:155`). Until lance-format/lance#7965 that rewrite corrupted blob columns holding null or empty values, which is what #3744 reports: - **storage 2.0** (legacy v1 `lance-encoding:blob` descriptors): every payload following a null or empty row in the same fragment was rewritten as `{position: 0, size: 0}`, so it read back as `b""` and the new fragment no longer referenced the bytes — silent payload loss, unrecoverable once the pre-optimize versions are pruned. - **storage 2.2** (blob v2): a valid empty value was rewritten as null, destroying the null-vs-empty distinction. Both manifestations share one root cause: `is_inline_null_blob` classified any inline blob with `position == 0 && size == 0` as null, which is also exactly what a *valid empty value* looks like. Such rows were dropped from `blob_read_addrs`, misaligning every payload that followed. The behaviour is already correct on `main`: the vendored lance crate first carried the fix at `v10.0.0-beta.3` (#3710) and is now `v10.1.0-beta.1` (#3757). What was missing is coverage — nothing in this repo exercised a blob column containing a null or empty value through `optimize()`, which is why this shipped unnoticed. This PR adds that guard. ## Tests Two tests in `rust/lancedb/tests/blob_integration.rs`, reusing the file's existing 64 KiB dedicated-blob helpers and a delete-triggered fragment rewrite. After `id IN (1, 4)` is deleted the surviving rows are `2` (null), `3` (valid empty), `5` and `6` (payloads) — payloads sit immediately after the null/empty, which is where the misalignment landed. - `optimize_preserves_v1_blob_payloads_with_null_and_empty` — storage 2.0; asserts the **payload bytes** are unchanged across `OptimizeAction::All` (what the Python/Node `optimize()` bindings invoke). Payloads are read through `lance::Dataset::take_blobs`, since `Table::fetch_blobs` rejects legacy v1 columns. The before/after descriptors are reported on failure but deliberately *not* asserted: compaction repacks the blob file, so they shift legitimately (id 5 `(131072, 65536)` → `(0, 65536)`, id 6 `(196608, 65536)` → `(65536, 65536)`). Note that a post-compaction `position: 0` is both the legitimate first-payload offset and the bug's signature, so asserting descriptors would be actively misleading. - `optimize_preserves_blob_v2_null_and_empty_distinction` — storage >= 2.2; asserts a null stays null and a valid empty value stays non-null empty. Both assert the pre-optimize state first, so a setup change that stops producing the null/empty/payload mix fails loudly instead of passing vacuously. Both also assert the returned `CompactionMetrics` show a fragment was actually rewritten. These tests depend on `delete("id IN (1, 4)")` pushing the fragment past lance's `materialize_deletions_threshold` (0.1 by default; 2 of 6 rows here). That coupling is invisible and unasserted otherwise: against a forced no-op (`materialize_deletions_threshold: 1.5`) the metrics come back all zeroes and *every payload assertion still passes*. Since the whole point of these tests is to survive dependency changes, they check that the rewrite happened rather than trusting the planner to keep selecting the fragment. Guard verified against a pre-fix lance: with the published `lancedb==0.36.0` wheel (vendors lance 9.0.0), `Table.optimize()` on the same data rewrites the descriptors of the two rows following the null/empty from `(131072, 65536)` and `(196608, 65536)` to `(0, 0)`, and the payloads read back empty. Against the pinned `v10.1.0-beta.1`, all 39 tests in the file pass, adding roughly 10–20 ms to the file's runtime. ## Not addressed here - **No released artifact has the fix yet.** PyPI `lancedb` 0.36.0 (2026-07-29) vendors lance 9.0.0; npm `@lancedb/lancedb` 0.37.1-beta.0 predates the bump. No 9.x lance tag carries the fix: `v10.0.0-beta.3` is the first tag containing it, every `v9.1.0-beta.1`…`beta.8` is behind it, and `v9.0.0` / `v9.0.1-rc.1` sit on a diverged branch without it. A stable lancedb release needs a stable lance >= 10. - **The version skew #3744 flagged is still live.** `python/pyproject.toml` pins `pylance==9.0.0rc1` for the `tests` extra against a vendored `10.1.0-beta.1`, so Python CI still cannot observe this class of divergence. - **Only the single-fragment rewrite shape is covered.** Both tests rewrite one fragment by materializing deletions. lance's own `test_compact_blob_v1/v2_preserves_null_empty_and_payload_order` cover the multi-fragment merge shape (3 fragments → 1) at unit level, so this PR is complementary rather than redundant — it covers the binding-level path through `Table::optimize` — but it would not catch a regression that only appears when *merging* fragments. `multi_fragment_dedicated_blob_table` in the same file makes that a cheap follow-up. Closes #3744 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Wyatt Alt
|
8e24dd3828 |
feat(rust)!: make add_columns a builder (#3778)
Table::add_columns now takes no arguments and returns AddColumnsBuilder, so calls become .add_columns().transform(t).execute(). read_columns was the second positional argument but reaches only one of the five transform variants. In lance's add_columns_to_fragments only BatchUDF receives the caller's value: SqlExpressions replaces it with the columns its expressions reference, Stream and Reader pass None, and AllNulls reads nothing. So it was mandatory on every call -- all eighteen call sites here passed None -- and silently discarded four times out of five. As a builder method it is optional, and setting it where lance would discard it is now an error, which does reject a call that previously succeeded while ignoring the argument. Matches the builders add, update, and merge_insert already use. |
||
 Adityaj0
|
f79dc017c4 |
fix: when_not_matched_by_source_delete() doesn't reset a previously-set condition (#3771)
## Summary `LanceMergeInsertBuilder.when_not_matched_by_source_delete()` didn't clear a previously-set condition when called again with no argument (or a different condition type). Per the docstring, `condition=None` means "delete all unmatched rows," but if the builder had already been configured with a string/Expr condition, a later no-arg call left the stale condition in place instead of widening the delete to unconditional. Fixes #3767 ## Change Each call now unconditionally sets both `_when_not_matched_by_source_condition` and `_when_not_matched_by_source_condition_expr` (one to the new value, the other to `None`), so the latest call always wins — consistent with every other setter on this builder (e.g. `when_matched_update_all(where=...)`). ## Test plan - [x] New regression test `test_merge_insert_by_source_delete_reconfigure` in `python/python/tests/test_table.py` - [x] `uv run --extra tests pytest python/tests/test_table.py::test_merge_insert_by_source_delete_reconfigure python/tests/test_table.py::test_merge_insert_by_source_delete_expr python/tests/test_table.py::test_merge_insert_by_source_delete_expr_async -vv` — 3 passed - [x] `uv run --extra dev ruff format` / `ruff check` — clean Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Adityaj0
|
e6ae93f52a |
fix: hybrid search minimum_nprobes(0) silently no-ops instead of raising (#3770)
## Summary `LanceHybridQueryBuilder._create_query_builders()` checked `self._minimum_nprobes` for truthiness instead of `is not None` — the very next line correctly checks `is not None` for `self._maximum_nprobes`. Since `0` is falsy in Python, `.minimum_nprobes(0)` on a hybrid query silently dropped the value instead of forwarding it to the vector sub-query, where it would raise the same `ValueError` a plain vector query raises for the same input (`minimum_nprobes must be greater than 0`, validated in `rust/lancedb/src/query.rs` and covered for the plain-query path by `test_invalid_nprobes_sync`). Fixes #3766 ## Change One-line fix: `if self._minimum_nprobes:` → `if self._minimum_nprobes is not None:`, matching the existing `maximum_nprobes` check right below it. ## Test plan - [x] New regression test `test_hybrid_query_minimum_nprobes_zero_raises` in `python/python/tests/test_hybrid_query.py` - [x] `uv run --extra tests pytest python/tests/test_hybrid_query.py -vv` — 13 passed - [x] `uv run --extra dev ruff format` / `ruff check` — clean Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Drew Gallardo
|
3dd9c598e9 |
feat(remote): add seekable blob range reads (#3750)
## Summary - Implements Cloud `fetch_blob_files`: returns real seekable `BlobFile` handles over HTTP Range instead of `NotSupported`. - Completes the second Cloud blob read verb after #3684 (`fetch_blobs` = eager whole bytes; this = lazy / partial / sequential reads). - Same public handle API as local (`read_range`, `read_up_to`, `seek`, `tell`, `close`), so one code path works for local and Cloud. Large blobs (video, audio, PDFs) should not require downloading the whole object to inspect a header or stream a slice. After search, callers open a handle and read only what they need: ```python hits = table.search(vec).select(["id", "video"]).limit(5).to_arrow() with table.fetch_blob_files("video", hits)[0] as f: header = f.read_range(0, 256) f.seek(keyframe_offset) chunk = f.read_up_to(1 << 20) ``` ### Behavior - Handle creation probes size with `bytes=0-0` (bounded concurrency, input order preserved). - `204` → null (`None`); `416` with `bytes */0` → valid empty blob; other `416` → error. - `read_range` validates `Content-Range` and body length; OOB ranges fail with `invalid_input` before the request (aligned with Lance). - `read_up_to` reuses one open-ended Range response across sequential reads; `seek` drops it. - Servers older than 0.5.0 get a clear `NotSupported` (does not suggest `fetch_blobs`, which they also lack). ## Testing - `cargo test --features remote -p lancedb remote_blob` - `cargo test --features remote -p lancedb test_blob` - `cargo clippy --features remote --tests --examples` (no new warnings from this change) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
 dependabot[bot]
|
9e26bf3fba |
chore(deps): bump the rust-minor-patch group with 3 updates (#3758)
Bumps the rust-minor-patch group with 3 updates: [http](https://github.com/hyperium/http), [napi-derive](https://github.com/napi-rs/napi-rs) and [napi-build](https://github.com/napi-rs/napi-rs). Updates `http` from 1.4.2 to 1.5.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/hyperium/http/releases">http's releases</a>.</em></p> <blockquote> <h2>v1.5.0</h2> <h2>What's Changed</h2> <ul> <li>feat(method): add QUERY method by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/798">hyperium/http#798</a></li> <li>fix(uri): allow empty paths in uri::Builder by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/853">hyperium/http#853</a></li> <li>perf(header,uri): faster value validation, URI parse/format, map inserts by <a href="https://github.com/geeknoid"><code>@geeknoid</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/852">hyperium/http#852</a></li> <li>fix(uri): enforce max length in PathAndQuery by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/856">hyperium/http#856</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/geeknoid"><code>@geeknoid</code></a> made their first contribution in <a href="https://redirect.github.com/hyperium/http/pull/852">hyperium/http#852</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/hyperium/http/compare/v1.4.2...v1.5.0">https://github.com/hyperium/http/compare/v1.4.2...v1.5.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/hyperium/http/blob/master/CHANGELOG.md">http's changelog</a>.</em></p> <blockquote> <h1>1.5.0 (July 29, 2026)</h1> <ul> <li>Add <code>Method::QUERY</code> constant for the new QUERY method defined in RFC 10008.</li> <li>Fix <code>uri::Builder::path_and_query()</code> to allow empty strings to mean no path.</li> <li>Fix <code>uri::PathAndQuery</code> parsing to enforce URI max length.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/hyperium/http/commit/16fc9a7b840c2181e7f8b37397c107b0ffcd050d"><code>16fc9a7</code></a> v1.5.0</li> <li><a href="https://github.com/hyperium/http/commit/e559023f67e3fad6ecc3ee91307be178e0f13626"><code>e559023</code></a> fix(uri): enforce max length in PathAndQuery (<a href="https://redirect.github.com/hyperium/http/issues/856">#856</a>)</li> <li><a href="https://github.com/hyperium/http/commit/2178e175c4e247a33ba5f6ca3503afb1afbaabba"><code>2178e17</code></a> perf(header,uri): faster value validation, URI parse/format, map inserts (<a href="https://redirect.github.com/hyperium/http/issues/852">#852</a>)</li> <li><a href="https://github.com/hyperium/http/commit/03c8cd7faeddfad00873b4d58a45ecdf74ebebe6"><code>03c8cd7</code></a> fix(uri): allow empty paths in uri::Builder (<a href="https://redirect.github.com/hyperium/http/issues/853">#853</a>)</li> <li><a href="https://github.com/hyperium/http/commit/bb8705b25cdb6e29081edf9ade2ea124f6783e18"><code>bb8705b</code></a> feat(method): add QUERY method (<a href="https://redirect.github.com/hyperium/http/issues/798">#798</a>)</li> <li>See full diff in <a href="https://github.com/hyperium/http/compare/v1.4.2...v1.5.0">compare view</a></li> </ul> </details> <br /> Updates `napi-derive` from 3.6.0 to 3.6.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-derive's releases</a>.</em></p> <blockquote> <h2>napi-derive-v3.6.1</h2> <h3>Other</h3> <ul> <li>updated the following local packages: napi-derive-backend</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/58bd87fa524a837a7c962ab4103e5588557ccd81"><code>58bd87f</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3414">#3414</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/9da87236dbc4fef99f066b7a130f4d0377308d44"><code>9da8723</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/8d22196aa98a1e6e70584561f5446d117d9c802c"><code>8d22196</code></a> chore(deps): update dependency oxc-parser to ^0.142.0 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3422">#3422</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/abc30fbafc2e3967d499cef970c68b3edfefd850"><code>abc30fb</code></a> build(deps): bump postcss from 8.5.17 to 8.5.23 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3421">#3421</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/55421392cbaa24d4df69419e4c6d4958fbcb6a12"><code>5542139</code></a> build(deps): bump fast-xml-parser from 5.9.3 to 5.10.1 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3418">#3418</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/dc4ee8c89cc27ce30e239482199b3b3d786bf8b6"><code>dc4ee8c</code></a> build(deps): bump fast-uri from 3.1.3 to 3.1.4 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3419">#3419</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/050d985196174b4be830cdb813d09e2705258455"><code>050d985</code></a> feat(async-runtime): drain-linger surface + lock-free scheduler internals (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3">#3</a>...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/e0b87086eefe0e7efeea6d269e9403c4be4ba9aa"><code>e0b8708</code></a> chore(deps): update dependency oxc-parser to ^0.141.0 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3417">#3417</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/fc8494010697d078a93a528c3180271f6f187504"><code>fc84940</code></a> chore(deps): update actions/setup-node action to v7 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3413">#3413</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/ee598db45985ef11e18c7340801c28bb2452b688"><code>ee598db</code></a> build(deps): bump protobufjs from 7.6.4 to 7.6.5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3410">#3410</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.0...napi-derive-v3.6.1">compare view</a></li> </ul> </details> <br /> Updates `napi-build` from 2.3.2 to 2.4.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-build's releases</a>.</em></p> <blockquote> <h2>napi-build-v2.4.0</h2> <h3>Added</h3> <ul> <li><em>(cli)</em> support non-threaded WASI targets (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3353">#3353</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/58bd87fa524a837a7c962ab4103e5588557ccd81"><code>58bd87f</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3414">#3414</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/9da87236dbc4fef99f066b7a130f4d0377308d44"><code>9da8723</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/8d22196aa98a1e6e70584561f5446d117d9c802c"><code>8d22196</code></a> chore(deps): update dependency oxc-parser to ^0.142.0 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3422">#3422</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/abc30fbafc2e3967d499cef970c68b3edfefd850"><code>abc30fb</code></a> build(deps): bump postcss from 8.5.17 to 8.5.23 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3421">#3421</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/55421392cbaa24d4df69419e4c6d4958fbcb6a12"><code>5542139</code></a> build(deps): bump fast-xml-parser from 5.9.3 to 5.10.1 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3418">#3418</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/dc4ee8c89cc27ce30e239482199b3b3d786bf8b6"><code>dc4ee8c</code></a> build(deps): bump fast-uri from 3.1.3 to 3.1.4 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3419">#3419</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/050d985196174b4be830cdb813d09e2705258455"><code>050d985</code></a> feat(async-runtime): drain-linger surface + lock-free scheduler internals (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3">#3</a>...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/e0b87086eefe0e7efeea6d269e9403c4be4ba9aa"><code>e0b8708</code></a> chore(deps): update dependency oxc-parser to ^0.141.0 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3417">#3417</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/fc8494010697d078a93a528c3180271f6f187504"><code>fc84940</code></a> chore(deps): update actions/setup-node action to v7 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3413">#3413</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/ee598db45985ef11e18c7340801c28bb2452b688"><code>ee598db</code></a> build(deps): bump protobufjs from 7.6.4 to 7.6.5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3410">#3410</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.3.2...napi-build-v2.4.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
 Will Jones
|
93354baf34 |
chore: upgrade rust toolchain to 1.97.0 (#3643)
Bumps the pinned Rust toolchain from 1.95.0 to the latest stable (1.97.0). Rust 1.97's clippy adds `useless_borrows_in_formatting`, which flags a redundant `&` in `format!`/`debug!` arguments in a few places. This PR removes those to keep `cargo clippy` clean. No behavior change; the MSRV (`rust-version = "1.91.0"`) is unchanged. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 LuQQiu
|
05602ec7d5 |
chore: update lance dependency to v10.1.0-beta.1 (#3757)
Updates the Lance Rust workspace dependencies and Java lance-core version to v10.1.0-beta.1. Includes a compatibility fix for the Lance file writer API by using the explicit V2_1 writer creation path for permutation shuffle spill files. Triggered by https://github.com/lance-format/lance/releases/tag/v10.1.0-beta.1 |
||
|
Wyatt Alt
|
e3b472c212 |
feat: connection-level job operations (#3755)
Adds job operations to the connection surface, building on the Job handle from #3742: job(id), list_jobs, get_job, cancel_job, and job_history, plus a non-blocking Job.status(). Implemented on the Database trait (defaulting to NotSupported), the remote backend (/v1/jobs), and the Python and Node bindings; job_history returns Arrow batches. errors() and progress() are not included. Tested with mocked endpoints in all three languages. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Wyatt Alt
|
a6418b6cb9 |
feat: create_index returns a Job handle (#3742)
IndexBuilder::execute now returns a Job with wait and cancel methods. Local tables build the index synchronously and return an already-done job. Remote tables read the job id the server returns from create_index and track it through the /v1/jobs API: wait polls describe until the job reaches a terminal state and cancel posts a cancellation. Servers that return no job id yield a done job, so behavior against older servers is unchanged. The job id is not exposed on the handle. The Python and TypeScript bindings keep their current signatures and discard the handle; exposing Job there is left to follow-ups. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
 Cohen Karnell
|
dd2b11eda2 |
fix(python): log when storage_options is ignored in RemoteDBConnection.open_table (#3743)
`RemoteDBConnection.open_table` accepts `storage_options` and never uses
it:
```python
def open_table(
self,
name: str,
*,
namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None,
index_cache_size: Optional[int] = None,
...
) -> Table:
...
if index_cache_size is not None:
logging.info("index_cache_size is ignored in LanceDb Cloud ...")
table = LOOP.run(self._conn.open_table(name, namespace_path=namespace_path))
```
The value is never passed down and never mentioned. `index_cache_size`
is ignored on Cloud in the
same way, but it says so.
I checked this at runtime on 0.34.0, not just by reading it: swapping
the inner connection for a
recorder, `open_table("t", storage_options={...})` hands the layer below
`['namespace_path']` and
nothing else, no log record is emitted, and the same probe shows
`index_cache_size` producing its
message as expected.
This adds the matching log line, so the two ignored parameters behave
the same way. `ruff check` and
`ruff format --check` are clean on the file.
A note on severity. This is not a security hole and nothing is exposed.
Someone passing credentials
there gets silence instead of an error, and finds out later.
One thing I am unsure about, and it changes the fix. I have assumed
per-table storage options are
meaningless on Cloud, which is what the `index_cache_size` line next to
it implies about managed
storage. If they are supposed to work, then the right change is to pass
them through to
`self._conn.open_table` instead and this patch is the wrong one. Happy
to redo it that way.
I did not check whether `create_table` or the async connection have the
same gap.
|
||
|
Will Jones
|
5a1015ba72 |
docs(python): fill gaps in the Python API reference (#3746)
`docs/src/python/python.md` is the whole Python API reference, but it is maintained by hand and had drifted from the public API. Anything not listed there simply doesn't get rendered, so a number of public, documented, tested APIs were invisible to users — most notably branch management, where `diff` and `merge` live. I audited every public symbol reachable from `lancedb` and its subpackages against the `:::` directives on the page. This adds the missing ones: - **Branching** — `Branches`, `AsyncBranches` (`list` / `create` / `checkout` / `delete` / `diff` / `merge`) - **Tables** — `TableStatistics` (returned by `Table.stats()`; the fragment-level stats classes were already listed) - **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`, `BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`, `Occur` - **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`, `AsyncTakeQuery` - **Indices** — `Fm` (the FM-index for substring search), `IndexConfig` - **Blobs** — `blob`, `BlobType`, `BlobFile` - **Namespaces** — `connect_namespace`, `connect_namespace_async`, and both namespace connection classes - **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`, `OAuthFlowType` - **Rerankers** — the `Reranker` base class plus `JinaReranker`, `RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`, `VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed) - **Embeddings** — `get_registry`, `register`, and the 14 embedding functions that were missing (3 of 17 were listed) - **PyTorch** — `StreamingDataset` and the permutation API it is built on - **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`, `pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two exception types It also repairs cross-references in docstrings that no longer resolve: links into guide pages that have since moved to lancedb.com (`querying-an-ann-index`, `experimental-full-text-search`), `lance.dataset` references with no inventory behind them, and the relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`. Deliberately left out: concrete implementation classes reached through their abstract base (`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already covered by `inherited_members: true`, and internal plumbing such as `FullTextSearchQuery` and `ColumnOrdering`. ## Testing The docs job only runs on pushes to `main`, so I built the site locally and compared against a build of `upstream/main`: every added entry resolves, and no symbol that was rendered before stopped being rendered when the four packages moved to automodule. `mkdocs build --strict` exits 0 on this branch, against 61 warnings on `main`. ## Also in this PR `lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and `lancedb.rerankers` are now rendered by a single mkdocstrings directive each, driven by the module's `__all__`, rather than a hand-maintained list. These four are where most of the drift was, and `__all__` is harder to forget than a docs page. `lancedb.embeddings` had no `__all__`; without one mkdocstrings renders no members at all for a re-export package, so one is added. AGENTS.md gains a section on how the page is wired up and how to build the docs locally. Rendering all that code for the first time surfaced ~100 more build warnings, which would have made #3707 (turning on `mkdocs build --strict`) harder to land, so the warning backlog is cleared here too. 97 of the 158 warnings were one systematic false positive — griffe cannot see the generated `__init__` of a pydantic dataclass, so every documented parameter looks unknown — switched off via `warn_unknown_params`. The remaining 61 came from 15 docstrings with real bugs: prose trailing a `Parameters` section (we were rendering parameters called `The`, `you` and `To`), types dropped because numpydoc needs spaces around the colon, `num_partitions, default sqrt(num_rows)` parsing as a list of names and inventing a `default` parameter, and one parameter indented five spaces. `mkdocs build --strict` now exits 0. --- #3747 (the coverage test that keeps this from happening again) is stacked on this branch, so review it after this one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 Farmer.Chillax
|
48945d0658 |
feat(python): add namespace/table exist support (#3460)
In the current LanceDB usage implementation, there is no way to check whether a table or namespace already exists. This PR introduces the namespace_exists and table_exists methods to determine the existence of tables and namespaces. useage like this: ``` # check table exists db.table_exists(table_id=['xxx']) # check namespace exists db.namespace_exists(namespace_id=['xxx']) ``` fixes: #3419 --------- Signed-off-by: farmer <farmerchillax@outlook.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Drew Gallardo
|
77208fd464 |
feat(remote): add RemoteTable fetch_blobs HTTP client (#3684)
Remote half of the blob read path. #3578 did local Python. This makes `RemoteTable` hit the server. - `fetch_blobs(column, row_ids or hits)` → bytes over `POST /v1/table/{id}/fetch_blobs/` - `blob_columns()` from the cached schema (describe already has the metadata, no extra route) - search then `fetch_blobs` works. row identity rides inside the blob descriptor so you do not need a public `_rowid` - `fetch_blob_files` still `NotSupported` on remote. use `fetch_blobs` for full bytes for now. Range is a follow up Accepts Binary / LargeBinary / BinaryView on the way back. Empty `row_ids` short-circuits. Version + branch go in the request body same as other read calls. ### Example ```python db = lancedb.connect(uri="db://my-project", api_key=...) table = db.open_table("clips") hits = table.search(query_vec).select(["id", "video"]).limit(10).to_arrow() # hits is just id + video. row ids are stashed on the descriptor blobs = table.fetch_blobs("video", hits) # null-aligned, same length as hits ``` Or pass ids yourself: ```python blobs = table.fetch_blobs("video", [10, 20, 30]) ``` ### Testing - `cargo test -p lancedb --features remote --lib` - `cargo test -p lancedb --features remote --test blob_integration` - `pytest python/tests/test_remote_db.py -k remote_blob` - live e2e against a local 0.5.0 remote server (search → fetch, nulls, nested path, old server gate) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 Joaquin Hui
|
b505dc1315 |
fix: distinguish corrupt table from missing in open_table (#3731)
`table_names()` lists any `*.lance` directory, but `open_table()` maps every `DatasetNotFound` to `TableNotFound`, so a corrupt or partially-written table looks identical to one that never existed (#3127). This takes the issue's Option 2: on `DatasetNotFound`, check the parent listing for the table's `.lance` entry — the same predicate `table_names()` uses — and return a new `TableCorrupted` error when the directory is present. The check runs only on the error path, and any failure in the recheck falls back to the previous `TableNotFound` behavior. Tests cover the reporter's empty-dir repro, a deleted-manifest case, true absence (still `TableNotFound`), and an end-to-end list-then-open assertion; the three new corrupt-case tests fail without the src change. `cargo test -p lancedb --lib` 732 passed, clippy/fmt clean, `cargo check --workspace --all-targets` clean (both language bindings end in wildcard error arms). Two notes for review: `Error` isn't `#[non_exhaustive]`, so the new variant is technically semver-breaking for exhaustive matchers (pre-1.0, and the alternative — changing `TableNotFound`'s shape — breaks more); and on the Python side corrupt tables now surface as `RuntimeError` rather than `ValueError`, which is the intended distinction but worth a maintainer's eye. `open_from_namespace` was left unchanged since namespace listings come from a server-side registry, not directory globbing. Closes #3127 |
||
 sanskar singh bhardwaj
|
7dfdfe6401 |
fix(remote): surface masked merge_insert stream errors under HTTP2 (#2339) (#3711)
## Summary Fixes #2339. `merge_insert()` on the remote client could mask the real cause of a mid-stream input error, reporting only: > stream error sent by user: unexpected internal error ## Root cause There were two divergent streaming-write code paths in the remote client: - `add()` uses `RemoteInsertExec`, which streams the request body through a `tokio::sync::oneshot` error side-channel and drains it before reporting the HTTP result. If the input stream errors mid-body, the original error is recovered. - `merge_insert()` used a legacy path (`send_streaming` -> `reader_as_body`) that piped arrow `Some(Err(e))` straight into the HTTP2 request body. Hyper swallows body-stream errors under HTTP2 (see hyperium/hyper#2547), so the original error was lost and only the generic transport error surfaced. ## Fix Consolidate both write paths onto the side-channel mechanism instead of patching the legacy path: - Generalize `RemoteInsertExec` into `RemoteWriteExec`, carrying a `WriteOp` enum (`Insert { overwrite }` | `MergeInsert { query, timeout }`) that selects the endpoint, query params, request-timeout header, and response parsing. The executor returns a `WriteResult` enum (`Add` | `Merge`) with typed accessors, and `with_new_children` still resets the result so the rescannable retry loop is unaffected. - Route `merge_insert()` through `RemoteWriteExec`. The public API only accepts a `RecordBatchReader` (not rescannable), so the reader is buffered into a `Vec<RecordBatch>` before the retry loop to preserve the previous retry-on-retryable-status behaviour. This mirrors what the old `send_streaming(with_retry=true)` path already did. - Remove the now-unused `send_streaming` / `reader_as_body` / `buffer_reader` / `make_reader` helpers. Multipart stays insert-only (the server has no multipart merge_insert endpoint), so that hot path is behaviorally unchanged. ## Testing - Added `test_merge_insert_input_error_surfaces_original`, which drives an erroring input through the single-request `merge_insert` path and asserts the original error (`boom`) is surfaced rather than the masked HTTP error. Confirmed it fails without the side-channel drain (it then reports a masked `500 ... request or response body error`). - Full suite green: `cargo test -p lancedb --lib --features remote` -> 694 passed, 0 failed. Includes the existing `test_merge_insert_retries_on_409`, confirming retry behaviour is preserved. |
||
 heart
|
4dc2d9a0f2 |
fix(python): avoid async work in sync reprs (#3620)
## Summary - keep the existing synchronous `connect()` path unchanged - make `LanceDBConnection.__repr__` and `LanceTable.__repr__` side-effect-free - add a regression test that verifies sync reprs do not call the Python background loop ## Root cause The freeze is caused by debugger rendering, not by `connect()` itself: 1. debugpy stops at a breakpoint and suspends all Python threads. 2. The debugger renders the new `db_connection` local by calling `repr()`. 3. `LanceDBConnection.__repr__` reads `read_consistency_interval`. 4. That property calls `LOOP.run(...).result()`. 5. The `LanceDBBackgroundEventLoop` thread is suspended by the debugger, so `repr()` waits for a thread that cannot run. This explains why the symptom appears immediately after `connect()`: it is the first point where a connection object exists in locals and is automatically rendered. `LanceTable.__repr__` had the same problem because it also read the connection's consistency interval. This follows the same principle as #3411: `__repr__` must not trigger async work or I/O that a debugger assumes is lightweight. ## Evidence I reproduced the behavior with the real LanceDB classes and debugpy 1.8.21 using a DAP client: - latest `main` (`ff6ff099`): the debugger reported `allThreadsStopped: true`, and evaluating `repr(db_connection)` timed out - this branch (`5755a5ba`): the same evaluation returned `LanceDBConnection(uri='/tmp/lancedb-debug-repro')` immediately - setting `PYDEVD_UNBLOCK_THREADS_TIMEOUT=0` also allowed the original repr path to complete, independently confirming that it was waiting on a suspended thread The regression test creates a connection and table, replaces `LOOP.run` with a function that fails, and verifies that both reprs still work. ## Validation - `maturin develop --manifest-path python/Cargo.toml` - `python -m pytest python/python/tests/test_db.py::test_sync_repr_does_not_use_background_loop python/python/tests/test_table.py::test_consistency -q` (`4 passed`) - `ruff check .` - `ruff format --check python/python/lancedb/db.py python/python/lancedb/table.py python/python/tests/test_db.py python/python/tests/test_table.py` - `git diff --check` Refs #3611. |
||
 LanceDB Robot
|
1ad6ce3a4e |
chore: update lance dependency to v10.0.0-beta.7 (#3745)
Updates the Rust workspace Lance dependencies and Java lance-core dependency to v10.0.0-beta.7. No compatibility fixes were required; full workspace clippy passed with warnings denied. Lance tag: https://github.com/lance-format/lance/releases/tag/v10.0.0-beta.7 --------- Co-authored-by: Lu Qiu <luqiujob@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Will Jones
|
03b26d585b |
fix: deflake test_read_consistency_interval (#3713)
`test_read_consistency_interval` asserted that a table opened with a
100ms `read_consistency_interval` still read stale data immediately
after a concurrent write. The cache timestamp is set when the table is
opened and reads within the interval do not refresh it, so that
assertion only held if the intervening open/count/commit/count sequence
finished within 100ms of real wall-clock time. On a loaded CI runner it
did not: the TTL expired, `count_rows` refreshed synchronously, and the
test failed with `left: 1, right: 0`. This broke the Rust workflow on
`main` at
|
||
 Yang Cen
|
f7feed48c3 |
feat(fts): support custom stop-word lists (#3734)
## What Expose custom FTS stop-word lists in the Python and TypeScript public APIs, including their standalone tokenize helpers and remote index creation. This PR supports concrete string lists only. It does not add file or LanceDB-table stop-word sources. ## Why Rust already exposes Lance's custom stop-word list option. The Python and TypeScript APIs did not pass it through, and local index details did not retain the full tokenizer parameters needed by index-backed tokenization after reopening a table. ## How - Add `custom_stop_words` / `customStopWords` to the Python and TypeScript FTS and tokenize options. - Preserve `None` / `undefined`, empty lists, and list contents without normalization. - Load the persisted FTS segment parameters when returning local index details. - Serialize the concrete list in remote create-index requests. - Keep Python and TypeScript tests thin; behavior, persistence, query tokenization, and remote JSON coverage live primarily in Rust. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` - Python extension rebuild with `uv` and `maturin` - Targeted Python tests: 4 passed - Python `ruff format --check` and `ruff check` - TypeScript build, typecheck, Biome lint, generated docs, and targeted tests --------- Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local> |
||
 Lance Release
|
e5f489818b | Bump version: 0.37.0-beta.0 → 0.37.1-beta.0 | ||
 buduoqiu
|
98a52267a2 |
feat(python): configure streaming transform parallelism (#3699)
## Summary - add a keyword-only `transform_parallelism` option to `StreamingDataset` - preserve CPU auto-detection by default and fall back to one worker when unavailable - apply the configured limit to both the transform executor and concurrency semaphore - document and test explicit, default, fallback, and invalid values ## Testing - `uv run --extra tests --with torch pytest python/tests/test_elastic_dataloader.py -q` (`136 passed`) - `uvx ruff check python/lancedb/streaming.py python/tests/test_elastic_dataloader.py` - `uvx ruff format --check python/lancedb/streaming.py python/tests/test_elastic_dataloader.py` - `git diff --check origin/main...HEAD` Closes #3695 Co-authored-by: buduoqiu <yaodong-shen@users.noreply.github.com> |
||
|
Will Jones
|
ff50e698cf |
ci: cut Actions cost by moving builds to free runners and fixing caches (#3735)
Standard GitHub-hosted runners are free on public repos, so all Actions spend here is on the `*-8x-*` / `4x` larger runners. Measured over 30 days at current (post-Jan-2026) larger-runner rates, that is ~$1,400/mo, and `npm-publish` is ~70% of it. ## Changes **Fat LTO was forcing builds onto large runners.** `[profile.release]` in `.cargo/config.toml` sets `lto = "fat"` with `codegen-units = 1`, which is single-threaded and the peak-memory step. The macOS `npm-publish` build was 111 of its 113 minutes in one `napi build` step, making it the critical path of the whole publish pipeline. The ThinLTO override already applied to Windows now covers macOS too, and both Windows builds move from `windows-2025-8x-x64` to the free standard `windows-2025`. **The npm-publish cargo cache never existed.** There are zero caches with its key prefix. The key was static, so `actions/cache` (which only writes on a miss) could never refresh it, and a multi-GB release `target/` per target could never fit the repo's 10 GB budget anyway. Now caches only the crate registry, keyed on `Cargo.lock`. The docker builds also mounted `.cargo/registry/*` while the cache saved `.cargo-cache`, so containers re-downloaded the registry every run. **Cache eviction thrash.** Repo cache usage is 10.4 GB against GitHub's 10 GB cap, so every PR run evicted main's warm entries. `rust.yml` and `nodejs.yml` now restore everywhere but only save from `main`. **npm-publish moves to nightly + tags** instead of every push to main (~90/month). The cross-compiled targets do need watching, so `report-failure` now fires on scheduled runs, and dedupes onto an existing open issue rather than filing one per night. **rust.yml aarch64-pc-windows-msvc** cross-compiled its tests and then skipped them, paying full codegen and link cost for a compile check. `windows-11-arm` is now GA and free on public repos, so it builds and tests natively. Its test step also passes `--target` — without it cargo used `target/ci/` rather than `target/<triple>/ci/` and rebuilt the entire dependency graph a second time. **pypi-publish.yml had no concurrency group**, so force-pushes left a ~74 minute Windows job running. ## What is cost vs. wall-clock | Change | Cost | Wall-clock | |---|---|---| | Windows npm-publish → free runners | **−$570/mo** | slower per job (8→4 cores) | | npm-publish nightly | **−$125/mo** | — | | pypi-publish concurrency | small | — | | macOS ThinLTO | $0 (already free) | **−~50 min** per release | | rust aarch64 Windows native | $0 (already free) | **−~25 min** | | rust `--target` on test step | $0 | large, avoids a second full build | | rust-cache `save-if` | small | faster via real cache hits | ## Risks - The two Windows builds now have 4 cores instead of 8 and ~14 GB of free disk. If they fail, it is most likely disk rather than memory; fallback is `windows-2025-4x-x64`, which still halves that line. - `windows-11-arm` has a thinner toolset (choco/vcpkg/protoc under emulation) and this enables a test step that has never run, so it may surface real aarch64 failures. That is the point, but it is the change most likely to need iteration. - ThinLTO applies to published macOS and Windows binaries, typically within a few percent of fat LTO. Linux release builds are untouched. ## Follow-ups - `python.yml` `pydantic1x` (37 min) and `Doctest` (33 min) each rebuild the extension from source via `pip install -e .` with no Rust cache; they should consume the wheel the `linux` job already builds. Worth ~$235/mo and ~70 min of compute per run. Separate PR. - The three `ubuntu-2404-8x-x64` npm-publish builds (~$420/mo at the old cadence) are the remaining large-runner spend; `aarch64-unknown-linux-gnu` could run natively on free `ubuntu-24.04-arm`. Worth doing after this lands so the ThinLTO change can be validated first. - The wheel composite actions declare `python-minor-version` as required but never use it, and every caller omits it (actionlint warns). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 kid
|
b799ebaa69 |
fix(node): reject non-string Arrow metadata (#3728)
## Summary - validate Arrow metadata keys and values independently at runtime - reject malformed foreign schemas before constructing a local Arrow schema - cover valid and invalid metadata entries across Arrow 15–18 ## Testing - `node_modules/.bin/jest --runInBand __test__/arrow.test.ts -t "schema metadata"` - `node_modules/.bin/jest --runInBand __test__/arrow.test.ts` - `node node_modules/@biomejs/biome/bin/biome format --write lancedb/sanitize.ts __test__/arrow.test.ts` - `pnpm lint` - `pnpm build` - `pnpm run docs` Fixes #3729 |