3085 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Aviral Saxena	5f0bb98e90	fix(python): use exact integer math for list_versions timestamps (#4426) ... Refs #4404. AsyncTable.list_versions split the nanosecond timestamp with float division/modulo (`ts // 1e9`, `ts % 1e9`). At ~1.8e18 a float can't hold every integer, so the microseconds drifted (.274000 came back as .273999). Use `divmod(ts, 1_000_000_000)` and integer // 1000 instead. Only the precision fix is included here. Timezone-aware UTC (option 1 in #4404) is on hold for now, and I can add it here or in a follow-up depending on what they prefer. Added test_list_versions_timestamp_precision using the value from the issue (1790989429274000000), expecting microsecond == 274000.	2026-10-05 11:15:46 -07:00
Xuanwo	61a285f590	fix: preserve Function dependency errors in column operations (#4434) ... Function column declarations and refreshes can return HTTP 404 when the referenced Function has been deleted. The SDK currently translates those responses into `TableNotFound`, hiding the server diagnostic and making an existing table appear missing. Preserve the HTTP error, status, and request ID for these two operations while retaining schema-cache invalidation. Ordinary table operations keep their existing `TableNotFound` behavior.	2026-10-05 14:34:32 +08:00
lancedb-gatefixer[bot]	7f5933594f	fix(python): reject empty query vectors before column inference (#4415) ... Empty list and NumPy vector queries were validated too late: local search inferred a vector column first, producing misleading schema errors, while remote search indexed `vector[0]` and raised `IndexError`. Add a shared empty-vector check before column inference and query creation, and validate direct vector-builder inputs. Local and remote sync search now raise `ValueError: Query vector must not be empty` for these inputs. Validation: - All 26 new regression cases failed on the original code and pass with the fix, covering list/NumPy inputs, automatic/explicit vector search, inferred/explicit column names, one/two local vector columns, and remote search before query submission. - After merging current `main`, all 32 focused cases passed, covering these regressions and upstream async searches without a query. - The query, table, remote, utility, hybrid-query, and database suites passed in the rebuilt uv environment: 541 passed, 1 skipped, 1 slow test deselected. - `ruff check .`, `ruff format .`, and `cargo fmt --all` passed from the repository root. Fixes #4389 <!-- lance-gatekeeper-fix:v1 agent=f1f4875090ef391bb1aa912dd54026dc generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 21:30:13 -07:00
lancedb-gatefixer[bot]	3e0a1a8780	fix: reject zero vector search probes across SDKs (#4418) ... `nprobes(0)` bypassed the validation used by the minimum and maximum probe setters, allowing zero into query requests and producing inconsistent local and remote results. Sync Python could additionally interpret the zero maximum as an unbounded search. Reject zero in the Rust vector-query builder and propagate its input error through the Python vector/hybrid and Node bindings. Validate nonpositive counts in the sync Python vector and hybrid builders so every SDK and backend reports `nprobes must be greater than 0` before execution. Update the Rust examples and document the positive-count requirement. Regression tests cover local and remote vector/hybrid builders, rejection before remote query requests, promised TypeScript query vectors, and positive counts replacing both probe bounds. Validation: - Rust probe regression tests: 2 passed. - Focused Python query, remote, hybrid, and serialization tests: 22 passed. The four sync regression cases fail with the original setters. - Focused TypeScript indexed local and mocked remote tests: 2 passed. - Workspace `cargo check --features remote --tests --examples` and Clippy passed. - Rust formatting, Python formatting/lint, and TypeScript build/lint/docs passed. Repository-wide `ruff check .` reports 29 existing errors in unchanged CI scripts; `ruff check python` passes. Fixes #4388 BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>` instead of `Self`; callers must handle the result, as with the existing minimum/maximum probe setters. <!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 21:29:47 -07:00
Chi-Wei LeeandWill Jones	0be3ae960e	fix(python): support exist_ok in local create_table (#4421) ... Synchronous local Python `create_table` callers using `mode="exist_ok"` received `ValueError` instead of creating or opening a table. Related to [#2900](https://github.com/lancedb/lancedb/issues/2900). The local wrapper rejected this mode before the shared Rust parser could validate it. Removing that guard delegates mode validation to the parser, and the sync and async docs now describe `exist_ok`. Remote option gaps and the TypeScript table type gap remain outside this change. The added tests retain both calls from [@wjones127's comment](https://github.com/lancedb/lancedb/issues/2900#issuecomment-5964167206). This builds on the code @wjones127 posted there. Three regression tests fail before the change, pass after it, fail with the original `db.py` restored, then pass again on HEAD. Tested on Linux x86_64 with Python 3.14.7. From the Python directory, `uv run --no-sync --extra tests --extra dev python -m pytest python/tests/test_db.py -q` reports 53 passed. `uv run --no-sync --extra tests --extra dev python -m pytest python/tests -q -rs -m "not slow and not s3_test"` reports 1428 passed, 34 skipped, 68 deselected, and one failure: the existing `test_get_lsm_write_spec`, also failing on clean base (1425 passed, one failed). Ruff 0.9.9 `ruff format --check .` and `ruff check .` pass. Tests used worktree Python code with the released `lancedb==0.40.0b12` native wheel. No native source build was run. Co-authored-by: Will Jones <willjones127@gmail.com>	2026-10-03 10:12:29 -05:00
lancedb-gatefixer[bot]	9c3e1cbae7	fix: reject cloning onto an existing local table (#4416) ... Local `clone_table` passed existing targets to Lance's clone transaction, which entered the manifest-update path and failed with `Clone operation should not enter build_manifest`. Check the target dataset using the connection's storage options and session before cloning, and return `TableAlreadyExists` when it exists. Propagate other read errors. Add regression coverage for cloning onto the source itself and onto a different existing table, with both manifest layouts, verifying that rows and version histories remain unchanged. Validation: - All four regression cases reproduced the reported internal error before the fix. - `cargo test --locked --quiet --profile ci --features remote -p lancedb --lib test_clone -j 12`: 24 passed. - `cargo fmt --all -- --check` and `git diff --check` passed. - `cargo clippy --locked --quiet --profile ci --features remote -p lancedb --tests --examples -j 12` passed. Fixes #4387 <!-- lance-gatekeeper-fix:v1 agent=522c80785eeaaef146e8179c848b903e generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 06:16:31 -05:00
lancedb-gatefixer[bot]	de19ecbf63	fix: reject empty column alterations and unknown Python keys (#4417) ... Fixes #4390 Lance accepts column alterations with every change field unset, so a path-only Python call could commit a new local version without changing the schema. The Python binding also silently discarded unknown keys such as `nulable`. Validate every alteration in the shared Rust `Table` API before dispatching to either backend, and reject unknown dictionary keys in the Python binding while preserving the legacy `name` alias. Invalid batches leave local schema and version unchanged and send no remote request. Validation: - Reproduced both reported calls on the original extension: versions advanced from 1 to 2 to 3 with an unchanged schema; all 13 new invalid-input Python cases failed before the fix. - Rebuilt the extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`; all 18 focused Python cases pass across sync/async and local/mocked remote tables. - `cargo test --quiet --features remote -p lancedb --lib alter_column`: all eight tests pass, including both new regressions. - `cargo clippy --quiet --features remote --tests --examples` passes across the workspace. - `cargo fmt --all`, `ruff check .`, and `ruff format .` pass. - `typos` 1.26.0 passes across the repository; intentional misspellings in regression fixtures use the existing line-specific spelling exemptions. <!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 06:15:15 -05:00
lancedb-gatefixer[bot]	57a952135b	fix(python): support async search without a query (#4419) ... `AsyncTable.search()` and `search(None)` reached vector search without assigning `vector_query`, raising `UnboundLocalError` on both local and remote tables. Return a plain `AsyncQuery` before inferring the search type, equivalent to `table.query()`, and update the overloads and documentation to describe that return value. Add six regression cases for omitted and explicit `None` queries on local tables with and without vectors and on a mocked remote endpoint. They verify scans, limits, filters, column projection, and the remote request body. Validation: all six regressions reproduced the original `UnboundLocalError`; all 11 focused query tests pass with the fix, including existing vector, full-text, hybrid, and sync scan coverage. The uv development bootstrap, locked Ruff formatting/lint checks, and focused Pyright return-type assertions also pass. Fixes #4385 <!-- lance-gatekeeper-fix:v1 agent=bb9a7509f6dadde030d8113b172abc9a generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 06:04:29 -05:00
lancedb-gatefixer[bot]	6851133a7c	fix(python): serialize deprecated v2 manifest option (#4420) ... Local synchronous `create_table(..., enable_v2_manifest_paths=True/False)` failed with `TypeError` because the deprecated argument was forwarded as a boolean in `storage_options`, while the Rust binding expects string values. Convert the argument to lowercase `"true"` or `"false"` before forwarding it, preserving the deprecation warning. Add regression coverage for both values with populated and schema-only tables, checking stored data and the manifest mode after reopening. Validation: - All four regression cases reproduced the original `TypeError` before the fix and pass afterward. - `uv run --no-sync --extra tests --extra dev pytest python/tests/test_db.py -q --tb=short -m 'not slow and not s3_test'`: 54 passed. - Repository-wide `ruff format .` and `ruff check .` completed successfully. Fixes #4384 <!-- lance-gatekeeper-fix:v1 agent=9c14955b90d0c12d6a46605ec5ef32c0 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 06:02:55 -05:00
lancedb-gatefixer[bot]	8c2c39b3c4	fix(python): repair sync manifest path migration (#4422) ... Fixes #4383 `LanceTable.migrate_v2_manifest_paths()` raised `AttributeError` because it called a nonexistent `AsyncTable.migrate_v2_manifest_paths()` method. Delegate to the existing async method, `migrate_manifest_paths_v2()`, while preserving the public sync API. Add a local regression test for tables starting with either v1 or v2 manifest paths. It checks repeated migration, v2 manifest filenames on disk, and preservation of data and version history after reopening the table. Validation: - Both new regression cases failed with the reported `AttributeError` before the fix. - `cd python && uv run --extra tests pytest python/tests/test_table.py::test_migrate_v2_manifest_paths python/tests/test_db.py::test_create_table_v2_manifest_paths_async -q` — 3 passed. - Repository-wide `ruff check .` and `ruff format .` passed using Ruff 0.15.20 from `python/uv.lock`. - `git diff --check` passed. <!-- lance-gatekeeper-fix:v1 agent=bb500f0d1ddefeba3c7d7d8390a2fb74 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 05:57:56 -05:00
lancedb-gatefixer[bot]	7e979c1416	fix(node): preserve schemas for empty query results (#4423) ... Empty local TypeScript queries returned a zero-row Arrow table with no columns because `toArrow()` built the table solely from record batches. Local streams can yield no batches, while remote streams often supply an empty batch carrying the schema. Expose the executed stream's schema through the native iterator and decode it into an empty Arrow table when no batches arrive. This preserves projected and search columns, types, nullability, and metadata without executing another query. Document the empty-result behavior. Validation: - Eleven regression cases failed with the original `toArrow()` implementation; all twelve now pass, including remote schema-only and empty-batch responses. - All 1,075 tests in the query, remote, Arrow, and table suites pass. - Native build, TypeScript compilation, Rust formatting and Clippy (`-D warnings`), TypeScript lint/formatting, and documentation generation pass. Fixes #4386 <!-- lance-gatekeeper-fix:v1 agent=7d589da1f7db5c7383f9600490e111a9 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 05:56:50 -05:00
lancedb-gatefixer[bot]	cc6b4aa4b4	fix(remote): preserve schema for empty merge insert sources (#4425) ... Remote `merge_insert` buffered its reader into a `Vec<RecordBatch>`, losing the schema when the reader yielded no batches. Scanning that vector failed before the request could reach the server. Preserve the reader's schema in an empty record batch when buffering produces no batches. The source remains replayable for retries, and empty-source merges still reach the server so `when_not_matched_by_source_delete` can apply deletes. The regression tests decode the outgoing Arrow IPC and verify the source schema, zero rows, merge options, and result statistics on an initial request and a retry after HTTP 409. They cover readers with no batches and readers with an empty batch, in insert and delete modes, including an ID-only delete source. Validation: - Before the fix, both no-batch cases reproduced `Cannot scan an empty Vec<RecordBatch>`; the empty-batch cases passed. - `cargo test --quiet --profile ci --features remote -p lancedb --lib remote::table::tests::test_merge_insert` — 9 passed. - `cargo fmt --all` and `cargo fmt --all -- --check`. - `cargo clippy --quiet --profile ci --features remote -p lancedb --tests --examples`. Fixes #4380 <!-- lance-gatekeeper-fix:v1 agent=a48c9fc83772479990d76124f3a2b3be generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-03 05:53:00 -05:00
lancedb-gatefixer[bot]	3d8c9e2ddc	fix(python): close synchronous database connections (#4424) ... Calling `close()` on a synchronous remote connection returned an unawaited coroutine and left the connection usable. Local synchronous connections had no close or context-manager API. Add synchronous `close()` and `is_open()` to the connection interface, delegate them to the existing connection handle for local, remote, and namespace connections, and provide shared context-manager cleanup. Closing is idempotent, and subsequent database operations use the existing `Connection is closed` error. Document the lifecycle with examples. Validation: - Reproduced the reported coroutine warning and missing local lifecycle API on the original code; all 15 new regression cases failed before the fix. - All 168 tests in `test_db.py`, `test_remote_db.py`, and `test_namespace.py` passed through the uv-managed environment, including local, in-memory, namespace, native-derived, and mocked remote lifecycle coverage. - All 6 synchronous connection doctests passed. - Repository-wide Ruff 0.9.9 lint and formatting passed; `git diff --check` passed. Fixes #4382 <!-- lance-gatekeeper-fix:v1 agent=a743139af98198f1d4c8934c9cc1d0c9 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-10-02 22:19:24 -05:00
Will Jones	b0c0d8ee92	perf(remote): size blob file handles from one descriptor take (#4349)	2026-10-02 19:59:24 -05:00
3 people	2734e10767	fix(node): update arrow to 21 (#4362) ... Thanks @vincentkoc for starting this! I was just looking into this myself. I started with your PR and added: - make arrow 21 the default - widen the peer range to <22 so if they release 21.3 that'll be included - have to do skipLibCheck in tsconfig.json, because of flatbuffers 25 which needs TypeScript 5.7 - fix override in Symbol.hasInstance - support new types in Arrow 20-21 (LargeList, Utf8View, BinaryView, IntervalMonthDayNano) - arrow-compat job in nodejs.yml that runs tests with arrow 19 and 20 (adds 2 sec) - put the list of arrow types in one place - update NODEJS_THIRD_PARTY_LICENSES.md - regenerated typescript docs (sorry about reorderings, one time cost) - reformat arrow.test.ts (sorry, almost all whitespace, thanks to biome. git diff -w shows 110 lines edited) - fixed Sliced Utf8View / BinaryView serialization error, by just converting them to Utf8 and Binary (as our rust code would anyway) --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-02 15:56:46 -05:00
dependabot[bot]	f72779ebb5	build(deps): bump the rust-minor-patch group across 1 directory with 7 updates (#4355) ... Bumps the rust-minor-patch group with 7 updates in the / directory: | Package | From | To | | --- | --- | --- | | [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.6.3` | `1.6.4` | | [object_store](https://github.com/apache/arrow-rs-object-store) | `0.14.1` | `0.14.2` | | [serde_with](https://github.com/jonasbb/serde_with) | `3.23.0` | `3.24.0` | | [napi](https://github.com/napi-rs/napi-rs) | `3.12.2` | `3.12.5` | | [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.5` | `3.6.9` | | [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.2` | `2.5.0` | | [arc-swap](https://github.com/vorner/arc-swap) | `1.9.1` | `1.9.2` | Updates `aws-smithy-types` from 1.6.3 to 1.6.4 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Updates `object_store` from 0.14.1 to 0.14.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/apache/arrow-rs-object-store/releases">object_store's releases</a>.</em></p> <blockquote> <h2>object_store v0.14.2</h2> <!-- raw HTML omitted --> <h1>Changelog</h1> <h2><a href="https://github.com/apache/arrow-rs-object-store/tree/v0.14.2">v0.14.2</a> (2026-09-11)</h2> <p><a href="https://github.com/apache/arrow-rs-object-store/compare/v0.14.1...v0.14.2">Full Changelog</a></p> <p><strong>Implemented enhancements:</strong></p> <ul> <li>Implement <code>Clone</code> for <code>MicrosoftAzure</code> <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/824">#824</a></li> <li>Support URLs for jurisdiction-restricted buckets on R2 <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/814">#814</a></li> <li>Lightweight DNS cache with shuffle to prevent DNS flooding <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/726">#726</a></li> <li>Allow presigning object_store multipart uploads <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/271">#271</a></li> <li>Allow specifying <code>Content-Type</code> when signing object_store PUT requests <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/270">#270</a></li> </ul> <p><strong>Fixed bugs:</strong></p> <ul> <li>Incomplete ASF license header in <code>src/client/s3.rs</code> (caused -1 vote on 0.14.2 RC1) <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/853">#853</a></li> <li>WriteMultipart::finish should abort after part upload failure <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/818">#818</a></li> <li>The 1.85 MSRV is inaccurate with all features <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/811">#811</a></li> <li>When the backend service returns an HTTP 500, the object store panics <a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/414">#414</a></li> </ul> <p><strong>Documentation updates:</strong></p> <ul> <li>Docs: point users uploading large objects at the multipart API <a href="https://redirect.github.com/apache/arrow-rs-object-store/pull/839">#839</a> (<a href="https://github.com/alamb">alamb</a>)</li> <li>[object-store]: update release schedule <a href="https://redirect.github.com/apache/arrow-rs-object-store/pull/834">#834</a> (<a href="https://github.com/alamb">alamb</a>)</li> <li>Add doc example for multipart upload to <code>GoogleCloudStorage::create_multipart</code> <a href="https://redirect.github.com/apache/arrow-rs-object-store/pull/803">#803</a> (<a href="https://github.com/alamb">alamb</a>)</li> <li>Add doc example for multipart upload to <code>MicrosoftAzure::create_multipart</code> <a href="https://redirect.github.com/apache/arrow-rs-object-store/pull/802">#802</a> (<a href="https://github.com/alamb">alamb</a>)</li> <li>Add doc example for multipart upload to AmazonS3::create_multipart <a href="https://redirect.github.com/apache/arrow-rs-object-store/pull/801">#801</a> (<a href="https://github.com/alamb">alamb</a>)</li> </ul> <p><strong>Closed issues:</strong></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/apache/arrow-rs-object-store/blob/main/CHANGELOG-old.md">object_store's changelog</a>.</em></p> <blockquote> <!-- raw HTML omitted --> <h1>Historical Changelog</h1> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/279572ea60f3a7a6e5237e066a6f4a35eee611e0"><code>279572e</code></a> Update CHANGELOG for latest main (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/855">#855</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/4ff7d048073c07bcc1f1dca74436245b2ba20dda"><code>4ff7d04</code></a> Restore missing first line of ASF license header in src/client/s3.rs (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/854">#854</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/25fa079c41fecb8011d5b8c3019f20e09c9b0c74"><code>25fa079</code></a> Update version to 0.14.2 and add CHANGELOG (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/850">#850</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/c37db72b92b78608291c1b33feb79bb5bb6d2d1c"><code>c37db72</code></a> feat: retry failed multipart part uploads (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/849">#849</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/135984fdf91f813c48c47b6288586d9b0e205164"><code>135984f</code></a> Add doc hints pointing large-object uploads at the multipart API (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/839">#839</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/b07471e2bc341278f86e30cf80a850d56cbe2c67"><code>b07471e</code></a> [object-store]: update release schedule (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/834">#834</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/978fb48b9ea6ebe3f8f4833f656e793cde79ab15"><code>978fb48</code></a> Fix MSRV violation in <code>parse.rs</code> test, check <code>--all-targets</code> in MSRV CI (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/837">#837</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/e2a49349622aa7ad5db12b107a557d957ac01241"><code>e2a4934</code></a> Speed up MSRV CI check by installing cargo-msrv via taiki-e/install-action (#...</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/e28807de7338abebf14ecf2ab959294cb3096d83"><code>e28807d</code></a> feat: presigned URLs with extra query params and signed headers (SignedUrlOpt...</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/5c743637e97d2f3417b6fd54532bb1223c6547c1"><code>5c74363</code></a> Restore UnwindSafe/RefUnwindSafe on ClientOptions by bounding DnsResolver (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/836">#836</a>)</li> <li>Additional commits viewable in <a href="https://github.com/apache/arrow-rs-object-store/compare/v0.14.1...v0.14.2">compare view</a></li> </ul> </details> <br /> Updates `serde_with` from 3.23.0 to 3.24.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/jonasbb/serde_with/releases">serde_with's releases</a>.</em></p> <blockquote> <h2>serde_with v3.24.0</h2> <h3>Added</h3> <ul> <li>Implement <code>JsonSchemaAs</code> for <code>NoneAsEmptyString</code>, so fields using it compile with the <code>schemars_*</code> features (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1014">#1014</a>)</li> </ul> <h3>Changed</h3> <ul> <li>Make <code>with_prefix!</code> and <code>with_suffix!</code> available with the <code>alloc</code> feature, instead of only with <code>std</code> (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1008">#1008</a>)</li> </ul> <h3>Fixed</h3> <ul> <li>Handle potential panics in deserialization of chrono and time DateTimes. This was privately reported by <a href="https://github.com/NotAFlightRisk"><code>@​NotAFlightRisk</code></a>.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/jonasbb/serde_with/commit/35ac84c8faf34d6822991bbf2397056994c9208b"><code>35ac84c</code></a> Bump version to 3.24 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1016">#1016</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/7b8e561c709544b417b3c0f05f3c57a753e86417"><code>7b8e561</code></a> Bump version to 3.24</li> <li><a href="https://github.com/jonasbb/serde_with/commit/6e576c8d5cc5f31efd26cf783d76eac6607d784e"><code>6e576c8</code></a> Bump the github-actions group with 2 updates (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1012">#1012</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/7b1814b5cf8f2ce4ef7844105ab26760371af0cf"><code>7b1814b</code></a> Add schemars support for NoneAsEmptyString (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1014">#1014</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/92eeedd1e5be0f176bd9a0532eb8ab1503396bb2"><code>92eeedd</code></a> Handle potential panic during deserialization for large datetimes (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1015">#1015</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/3404642aae13a3dfcf4ea0f47749f759b36a85db"><code>3404642</code></a> Handle potential panic during deserialization for large datetimes</li> <li><a href="https://github.com/jonasbb/serde_with/commit/c7073b98f150d60c22e1ccdb49f1c7151dd8595f"><code>c7073b9</code></a> Add schemars support for NoneAsEmptyString</li> <li><a href="https://github.com/jonasbb/serde_with/commit/6b32290267fa18064894e2fa67bf4bed7fb4bd5f"><code>6b32290</code></a> Run test with only the alloc feature (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1013">#1013</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/3361c69817d02b079b15f71d1281f573594cc948"><code>3361c69</code></a> Run test with only the alloc feature</li> <li><a href="https://github.com/jonasbb/serde_with/commit/b43bb81cd8f02f06c2bf4bcd057c83e71442a515"><code>b43bb81</code></a> Reduce required feature for with_{prefix,suffix} from std to alloc (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1008">#1008</a>)</li> <li>Additional commits viewable in <a href="https://github.com/jonasbb/serde_with/compare/v3.23.0...v3.24.0">compare view</a></li> </ul> </details> <br /> Updates `napi` from 3.12.2 to 3.12.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi's releases</a>.</em></p> <blockquote> <h2>napi-v3.12.5</h2> <h3>Fixed</h3> <ul> <li>fatal_exception noop in optimized builds (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3509">#3509</a>)</li> </ul> <h2>napi-v3.12.4</h2> <h3>Security</h3> <ul> <li><code>Object::unwrap</code> and <code>Object::remove_wrapped</code> (and the deprecated <code>Env::unwrap</code> / <code>Env::drop_wrapped</code>) read a <code>TypeId</code> out of a payload that is not a <code>TaggedObject</code>, giving a heap out-of-bounds read and a JavaScript-forgeable out-of-bounds write; payload provenance is now validated before dereferencing. See <a href="https://github.com/napi-rs/napi-rs/security/advisories/GHSA-rhpj-pggq-896v">GHSA-rhpj-pggq-896v</a>, fixed in <a href="https://redirect.github.com/napi-rs/napi-rs/pull/3506">#3506</a>.</li> <li>The serde deserializer reached <code>unreachable!()</code> on ordinary JavaScript values and aborted the process instead of returning an error. See <a href="https://github.com/napi-rs/napi-rs/security/advisories/GHSA-f334-75xc-qxv3">GHSA-f334-75xc-qxv3</a>, fixed in <a href="https://redirect.github.com/napi-rs/napi-rs/pull/3507">#3507</a>.</li> <li><code>ArrayBuffer::from_external</code>, <code>TypedArraySlice::from_external</code> and <code>Uint8ClampedSlice::from_external</code> (plus the compat-mode <code>Env::create_arraybuffer_with_borrowed_data</code>) built their slice from freed memory on runtimes without external buffers. See <a href="https://github.com/napi-rs/napi-rs/security/advisories/GHSA-32mm-r9wp-hrvc">GHSA-32mm-r9wp-hrvc</a>, fixed in <a href="https://redirect.github.com/napi-rs/napi-rs/pull/3503">#3503</a>.</li> </ul> <h3>Fixed</h3> <ul> <li><em>(napi)</em> return errors from the serde deserializer for unexpected JS value shapes (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3507">#3507</a>)</li> <li><em>(napi)</em> validate wrapped payload provenance in Object::unwrap/remove_wrapped (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3506">#3506</a>)</li> <li><em>(napi)</em> point from_external slices at the engine-owned copy after finalize reclaims the source (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3503">#3503</a>)</li> </ul> <h2>napi-v3.12.3</h2> <h3>Fixed</h3> <ul> <li>CurrentThread waker-stack deadlocks and threadless-wasm Buffer detachment (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3489">#3489</a>)</li> <li><em>(napi)</em> catch panics in closure trampolines instead of aborting (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3473">#3473</a>)</li> </ul> <h3>Other</h3> <ul> <li><em>(napi)</em> keep anyhow error cause when converting (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3477">#3477</a>)</li> <li><em>(napi)</em> outline threadsafe callback dispatch (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3465">#3465</a>)</li> <li><em>(napi)</em> outline class instance construction (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3464">#3464</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/43100baf28a3e5709641e35f892be4da5d62dcb2"><code>43100ba</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3508">#3508</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/bb7b75a7d53b0757f99813c7fef0537cef044adf"><code>bb7b75a</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/89d7a0b3636b4454265363dab58df471030b4f85"><code>89d7a0b</code></a> chore(deps): update dependency oxfmt to ^0.68.0 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3519">#3519</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/55545910c2487da38453e6171330e8fc9133213c"><code>5554591</code></a> fix(cli): rate-limit failed incarnation probes and let reconciliation tests i...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/6d10d826c34f5d0009eb1ab47efaa4a2e34f4555"><code>6d10d82</code></a> feat(cli): make the deferred workerd loader usable by host-driven addons (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3520">#3520</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/a1bcecbd1bef2f51f1047f5800d5a312eb2eba3b"><code>a1bcecb</code></a> feat(cli): add <code>napi.wasm.threadlessInitialMemory</code> for threadless WASI loader...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/0999babc1e37ce869b79f59db14f9eb12f6647df"><code>0999bab</code></a> feat(cli): export <code>__napiBindingTarget</code> from generated binding loaders (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3515">#3515</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/30ea504c3d65e1cb178d3124068257d8df8f3835"><code>30ea504</code></a> fix(cli): bound symlink-ancestor reconciliation guards to the anchor's parent...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/ae8f34a8dc5f629b53d136442e8f614f2d3c17b6"><code>ae8f34a</code></a> fix(cli): run the wasm env cleanup barrier on a raw emnapi `Context.destroy()...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/ccd314bbb5201182311b74dddd61f60b323318f4"><code>ccd314b</code></a> fix(cli): never wedge reconciliation locks when host identity probes fail (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3">#3</a>...</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-v3.12.2...napi-v3.12.5">compare view</a></li> </ul> </details> <br /> Updates `napi-derive` from 3.6.5 to 3.6.9 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-derive's releases</a>.</em></p> <blockquote> <h2>napi-derive-v3.6.9</h2> <h3>Other</h3> <ul> <li>update Cargo.toml dependencies</li> </ul> <h2>napi-derive-v3.6.8</h2> <h3>Fixed</h3> <ul> <li><em>(napi)</em> guard AsyncTask completion against env teardown (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3536">#3536</a>)</li> </ul> <h2>napi-derive-v3.6.7</h2> <h3>Other</h3> <ul> <li>update Cargo.toml dependencies</li> </ul> <h2>napi-derive-v3.6.6</h2> <h3>Other</h3> <ul> <li>update Cargo.toml dependencies</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/2763e12efc855748485129952a6ccb97ac991c06"><code>2763e12</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3538">#3538</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/5ae5cf604809ed6b9868c8e67b201bc9eb68d9bf"><code>5ae5cf6</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/0da138ec836906da5bc3a996521299e1d284bdda"><code>0da138e</code></a> feat(cli): self-sign OpenHarmony artifacts with --ohos-sign (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3539">#3539</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/2499a6311148492bc01ae316621b514a9345e4e7"><code>2499a63</code></a> feat(async-runtime): offer the MultiThread flavor on wasm32-wasip1-threads (#...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/886bd071972e09010ee901d1a1bb669a26d3e0dc"><code>886bd07</code></a> chore(deps): update dependency oxc-parser to ^0.151.0 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3542">#3542</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/edb0a295a14371b2f943c2a89bacf385eb61a038"><code>edb0a29</code></a> fix(napi): validate native payload provenance for External and instance data ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/38162bb0eb324ae24b402982bad3d4ef3f24c90a"><code>38162bb</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3529">#3529</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/ec3d8ead66c18e022d2e3fb6832e52e5204ffcc3"><code>ec3d8ea</code></a> fix(napi): guard AsyncTask completion against env teardown (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3536">#3536</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/96ed2672820b5468b6a451080fc27e6eef594d3d"><code>96ed267</code></a> chore(deps): update release-plz/action action to v0.5.139 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3537">#3537</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/b2c9f3b610cb167d2c05b8d6c748c120926ffd73"><code>b2c9f3b</code></a> chore(deps): update release-plz/action action to v0.5.138 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3533">#3533</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.5...napi-derive-v3.6.9">compare view</a></li> </ul> </details> <br /> Updates `napi-build` from 2.4.2 to 2.5.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-build's releases</a>.</em></p> <blockquote> <h2>napi-build-v2.5.0</h2> <h3>Added</h3> <ul> <li><em>(async-runtime)</em> offer the MultiThread flavor on wasm32-wasip1-threads (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3541">#3541</a>)</li> </ul> <h2>napi-build-v2.4.4</h2> <h3>Fixed</h3> <ul> <li><em>(napi)</em> guard AsyncTask completion against env teardown (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3536">#3536</a>)</li> </ul> <h2>napi-build-v2.4.3</h2> <h3>Fixed</h3> <ul> <li><em>(cli)</em> drain outstanding async work before disposing a WASI binding (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3528">#3528</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/2763e12efc855748485129952a6ccb97ac991c06"><code>2763e12</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3538">#3538</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/5ae5cf604809ed6b9868c8e67b201bc9eb68d9bf"><code>5ae5cf6</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/0da138ec836906da5bc3a996521299e1d284bdda"><code>0da138e</code></a> feat(cli): self-sign OpenHarmony artifacts with --ohos-sign (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3539">#3539</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/2499a6311148492bc01ae316621b514a9345e4e7"><code>2499a63</code></a> feat(async-runtime): offer the MultiThread flavor on wasm32-wasip1-threads (#...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/886bd071972e09010ee901d1a1bb669a26d3e0dc"><code>886bd07</code></a> chore(deps): update dependency oxc-parser to ^0.151.0 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3542">#3542</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/edb0a295a14371b2f943c2a89bacf385eb61a038"><code>edb0a29</code></a> fix(napi): validate native payload provenance for External and instance data ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/38162bb0eb324ae24b402982bad3d4ef3f24c90a"><code>38162bb</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3529">#3529</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/ec3d8ead66c18e022d2e3fb6832e52e5204ffcc3"><code>ec3d8ea</code></a> fix(napi): guard AsyncTask completion against env teardown (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3536">#3536</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/96ed2672820b5468b6a451080fc27e6eef594d3d"><code>96ed267</code></a> chore(deps): update release-plz/action action to v0.5.139 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3537">#3537</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/b2c9f3b610cb167d2c05b8d6c748c120926ffd73"><code>b2c9f3b</code></a> chore(deps): update release-plz/action action to v0.5.138 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3533">#3533</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.2...napi-build-v2.5.0">compare view</a></li> </ul> </details> <br /> Updates `arc-swap` from 1.9.1 to 1.9.2 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/vorner/arc-swap/blob/master/CHANGELOG.md">arc-swap's changelog</a>.</em></p> <blockquote> <h1>1.9.2</h1> <ul> <li>Document RefCnt must not panic (<a href="https://redirect.github.com/vorner/arc-swap/issues/208">#208</a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/vorner/arc-swap/commits">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-10-02 07:49:06 -05:00
XY Zhan	723a2394e5	feat: maintain every index a table has, including ones created later (#4342) ... **The maintained set is an intent, not a snapshot.** An unnamed set is stored as the intent to maintain every index, which lance re-resolves whenever it builds a MemTable, so an index created after the spec installs is maintained too. A named set is checked here, where the caller can be told which names exist; an index of a kind the MemWAL cannot mirror fails a named set and is skipped by an unnamed one. **A newly created index reaches a writer that is already open.** lancedb caches a `ShardWriter` per shard and reuses it, so `create_index` installs the current set on the cached writer. A failure there is reported rather than propagated: the index is committed durably by then, and the writer this process holds may be fenced or poisoned, which is no reason to fail creating an index. **A table carries one spec.** `set_lsm_write_spec` fails while one is installed and names `unset_lsm_write_spec` in the error, because the generations already written were homed under that spec. Accepting a repeat that changed only the maintained set would hand a caller who also passed a different sharding a success that changed nothing they asked for. **The read path asks what is actually carried.** An unnamed set leaves the stored list empty, so the list alone cannot decide whether an index is maintained. The read asks the spec *and* every resident MemTable whether it carries the index, so a MemTable built before the index joined the set refuses the read rather than answering without its rows. Both spec routes report an unnamed set as `null` rather than `[]`: maintain whatever the table gains, and maintain nothing, are different settings. ## Known gap The guard above covers resident MemTables. Once a writer flushes and closes none is resident, so a retained generation written before the index existed still gets an indexed read that omits its rows. This window is **wider than on main**: a set resolved at install time excluded an index created later, so the read was refused; an intent includes it, and the refusal only holds while a MemTable is resident. Closing it needs the generation's own coverage, which this layer cannot see — its snapshot carries a generation and a path — so it belongs in lance's scanner or flush path. ## Testing `cargo nextest run -p lancedb --lib --features remote` — **1479 passed, 3 skipped**. `cargo fmt --all --check`, and `cargo clippy --features remote --lib --tests --examples -- -D warnings`, both clean. Covered: the round trip of an unnamed set, a set over an installed spec refused, a spec replaced through unset, a read refused while a resident MemTable predates the index, and an index created mid-write reaching the open writer. Two Python binding tests move with the semantics: an unnamed set now reads back as `None` rather than the list it resolves to, and the refusal is matched on `already set`. Neither was run locally — no Python build here — so CI is the check on them.	2026-10-02 00:11:44 -04:00
3 people	857e455361	chore: update lance dependency to v13.1.0-beta.1 (#4358) ... Updates the Rust workspace and Java Lance dependencies to [v13.1.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.1.0-beta.1) and refreshes Cargo.lock. Removes a redundant `pub(crate)` declaration in the private Node.js blob module to satisfy the required Clippy check. Validation: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, `pnpm build`, and `pnpm test __test__/blob.test.ts --runInBand` (26 tests passed) in `nodejs`. --------- Co-authored-by: Daniel Rammer <hamersaw@protonmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-01 09:13:08 -05:00
Lance Release	4730f9036b	Bump version: 0.40.0-beta.11 → 0.40.0-beta.12	2026-09-30 13:32:13 +00:00
Jonathan HsiehandClaude Opus 5.5	82d39485a4	feat(functions): address Functions by namespace path (#4341) ... Adds namespace addressing to Functions, matching what Secrets already have (#4150). ## API `create_function` / `create_function_async`, `get_function`, `list_functions`, `drop_function` / `drop_function_async` take `namespace_path` — keyword-only in Python (sync, async and remote connections) and a trailing `&[String]` in Rust — defaulting to the root. The namespace is addressed in the route identifier (`/v1/function/analytics$features$embed/describe`, `/v1/namespace/analytics$features/function/list`); no body carries it. `FunctionVersion` and `FunctionVersionRef` name their Function **by parts** — a short `name` plus `namespace_path` — the same shape as `SecretReference`, so a returned version reopens from its own fields: ```python fv = db.create_function(embed, namespace_path=["analytics", "features"]) fv.name, fv.namespace_path # "embed", ("analytics", "features") db.get_function(fv.name, version=fv.version, namespace_path=list(fv.namespace_path)) ``` An empty path is omitted from the wire, so root documents keep their shape. `namespace_path` is passed straight to the binding, so a bare string (`namespace_path="prod"`) is a `TypeError` rather than four one-character segments; segments are checked for addressability before a request is built. The binding and application allow-lists now share one field list, `FUNCTION_VERSION_REF_FIELDS`, which admits `namespace_path` — without it a namespaced binding or application read as coming from a newer contract. ## Service side The Sophon half serves and persists FunctionVersion / FunctionVersionRef documents in this split form, keeping the joined id as its internal catalog key. It lands in sophon with the submodule bump. ## Tests - Rust: every Function verb addresses a namespace in the path; an unaddressable segment never reaches the transport; `FunctionVersion` round-trips a namespace and `reference()` carries it; a namespaced binding and application are known shapes. - Python: sync and async verbs address the namespace; a created Function reopens from its own `name`/`namespace_path`; a bare-string path is refused before any request. Draft: compiles, `cargo fmt --check` and clippy `-D warnings` clean; lancedb lib Function tests (116) and `first_class_function_slice2` pass on this base. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-29 14:07:14 -07:00
Bilal Atique	0a1c1a04c7	feat(node): accept an AbortSignal to cancel blob reads (#4350) ... Closes #4288. Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange` and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When the signal aborts, the call rejects with `signal.reason` (an `AbortError` unless the caller gave a reason) and the native future is dropped, so the read stops instead of running to completion. A signal that is already aborted rejects before any read starts. How it works: - On the Rust side, these five methods are now sync `#[napi]` methods that return `env.spawn_future(...)`. The future is wrapped in `futures::future::abortable`, and the abort handle is registered with napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so it can't be an argument of an `async fn`, which is why the methods changed shape. The TypeScript signatures are unchanged apart from the new optional argument. - napi-rs installs its own `onabort` on the signal it receives, which would overwrite a handler the caller set. So the TypeScript wrapper passes a private `AbortController` signal to the native call for each request, forwards the caller's abort to it with `addEventListener(..., { once: true })`, and removes the listener when the call settles. Tests in `table.test.ts`: - an already aborted signal rejects on all five methods, and the handle still reads afterwards - a custom abort reason is what the call rejects with - a read with a signal that never aborts returns normally and leaves no `abort` listener on the signal - aborting a 32 MiB `readRange` right after it starts rejects with `AbortError`, and the same handle reads again afterwards The Node docs under `docs/src/js` are regenerated. I didn't add a test for where the `read()` cursor ends up after a mid-flight abort on a local table. The remote `BlobFile` documents that a cancelled read leaves the cursor unchanged, but I haven't confirmed the same for the native Lance handle, so the docs don't claim it.	2026-09-29 14:01:17 -05:00
lancedb-gatefixer[bot]	ad79112b8e	fix(node): support LargeUtf8 and LargeBinary Arrow columns (#4287) ... ## Cause The Arrow type sanitizer handled Utf8 and Binary but omitted LargeUtf8 and LargeBinary, so Arrow tables from another Arrow installation failed schema sanitization before reaching LanceDB. ## Fix Reconstruct both large Arrow types during sanitization. Regression tests cover createTable and add with Arrow 15–18 and adding a LargeBinary Arrow column to a blob table. ## Validation - Confirmed the new table test failed before the fix with `Unrecognized type id in schema: 20` on all four Arrow versions. - The table, sanitizer, and Arrow Jest suites passed: 604 tests. - `pnpm lint`, `pnpm build`, and `pnpm run docs` passed. Fixes #4266 <!-- lance-gatekeeper-fix:v1 agent=1090de3c0ddeca587956632b79f0193c generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-09-29 12:25:12 -05:00
lancedb-gatefixer[bot]	85b0eba176	fix(python): support typed embedding vector fields (#3826) ... ## Summary - infer fixed-size float32 Arrow vectors from `VectorField` metadata when a model uses the standard `list[float]` annotation - document the mypy-compatible embedding-field declaration while preserving the existing `Vector(dim)` runtime API - add regression coverage for schema conversion and model construction ## Root cause `Vector(dim)` is a runtime constrained-type factory, but Python's static typing rules do not permit call expressions in annotation positions. Mypy's suggested `Vector[...]` spelling also cannot represent a dimension computed at runtime and fails because `Vector` is a function. `VectorField` already carries the embedding function and its dimension, but schema conversion previously ignored that metadata for a statically valid `list[float]` annotation. The runtime factory remains available for dimension-constrained Pydantic validation. Embedding-backed models that require mypy compatibility can now use `list[float] = embedding.VectorField()`; LanceDB derives the same nullable fixed-size float32 Arrow type from the field metadata. ## Validation - `cd python && uv run --extra tests pytest python/tests/test_pydantic.py -q` (23 passed) - focused mypy check reproducing the original error and accepting the new `LanceModel` declaration - isolated schema checks with Pydantic 1.10.24 and 2.11.7 - `ruff format .` - `ruff check .` Fixes #1982 <!-- lance-gatekeeper-fix:v1 agent=877facbc659cdc9424bd367e12426366 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-09-29 12:00:52 -05:00
Bilal Atique	4e3ec7cc27	feat(node): expose seek, tell, bounded read, readRanges and close on BlobFile (#4348) ... Closes #4300. The TypeScript `BlobFile` only had `size()`, `read()` and `readRange()`, while the Rust `BlobFile` behind it also has `seek`, `tell`, `read_up_to`, `read_ranges` and `close`. This exposes those on the Node binding: - `read(maxBytes?)` reads at most `maxBytes` from the cursor (Rust `read_up_to`). Without the argument it reads to the end, as before. - `readRanges([{ start, end }, ...])` reads several half-open ranges and returns one buffer per range, in order. Like `readRange`, it does not move the cursor. - `seek(position)` and `tell()` move and report the cursor. - `close()` releases the handle, and `isClosed()` reports it. Every read, `seek` and `tell` after `close()` fails with "blob file is already closed". A second `close()` does nothing. The binding keeps its own `closed` flag, the same way the Python `BlobFile` does. Offsets and lengths are `bigint`, like `readRange` and `size()`. `BlobRange` is exported as a type. The new tests in `table.test.ts` cover chunked reads with `read(maxBytes)`, `seek` and `tell`, a negative seek position, `readRanges` order and cursor, a `readRanges` end past the blob size, and reads after `close()`. The Node docs under `docs/src/js` are regenerated with `pnpm run docs`. I left the `ReadableStream`/async iterator idea from the issue out of this PR. The AbortSignal work in #4288 touches the same methods, so it can follow on top of this.	2026-09-29 11:32:30 -05:00
Xuanwo	ea223c4ec4	chore: update lance dependency to v13.0.0-beta.18 (#4339) ... Updates the Rust workspace dependencies, Cargo lockfile, and Java `lance-core` to [Lance v13.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.18). It includes lance-format/lance#9574, which bounds the memory an IVF partition split uses when it re-reads raw vectors during `optimize_indices`, and lance-format/lance#9582, which keeps the mTLS REST namespace client on rustls. Lance's new combined-fields FTS query variant (introduced by v13.0.0-beta.17) is handled the same way as in #4336: the remote capability check and Rust/Node bindings accept it, with a regression test for the remote request payload, and the Python conversion reports that this query type is not yet supported. This PR supersedes #4336.	2026-09-29 23:19:59 +08:00
Bilal Atique	166c114710	fix(rust): list indices when an FTS index's files are missing (#4330) ... Fixes #4250. `list_indices()` reads each FTS index's files to fill `index_details`. When those files were missing, the error failed the whole call, so the caller could not list any index, including the damaged one it needs to rebuild. Scalar indexes were not affected because their listing reads only the manifest. Now a failed read logs a warning and leaves that index's `index_details` as `None`. The other fields still come from the manifest, and `tokenize()` already returns a clear error when an FTS index has no details. The new test `test_list_indices_with_missing_fts_index_files` creates an FTS index and a BTree index, deletes the FTS index directory, and reopens the table. It then checks that both indexes are still listed. Without the change, it fails with the same `Not found` error as the issue.	2026-09-29 10:11:34 -05:00
lancedb-gatefixer[bot]	629f39d4a5	fix(python): preserve mixed blob values before Arrow conversion (#4304) ... ## Cause Python list and pandas rows were converted to Arrow before the target blob v2 schema was applied. In a mixed bytes and URI batch, Arrow inferred a binary column and encoded the URI as inline bytes; dict and `lance.Blob` values could fail conversion. ## Fix Build blob v2 columns from the original Python values using the target field's storage type before converting the rest of each row. This preserves inline bytes, external URIs, descriptor dictionaries, blob objects, and nulls for `add`, `merge_insert`, and `create_table`. ## Validation - `python/tests/test_blob.py`: 76 passed, 1 skipped. - Python project Ruff check passed. - Repository-wide Ruff check still reports 29 existing findings in `ci/` files outside this change. Fixes #4271 <!-- lance-gatekeeper-fix:v1 agent=54c95f06f11c88187b07a430aac6be2f generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-09-29 08:38:00 -05:00
lancedb-gatefixer[bot]	51ea02ea8c	fix: pin remote BlobFile reads to the opened version (#4295) ... ## Cause On latest-tracking remote tables, `fetch_blob_files` probed a concrete table version but retained `None` on each handle. Later range reads resolved the latest version again, so an update or delete could make an opened handle fail. ## Fix For nonempty blobs, capture `x-lancedb-version` from the 206 size probe and include it on every later range request. Reject a nonempty probe without a valid version. Accept an empty-blob 416 probe with or without that header, because an empty handle makes no later data range request. The Python remote mock now covers both response shapes. ## Limitation A table dropped and recreated with the same name can reuse version numbers and row addresses. Version pinning does not identify the table itself, so an old handle can still return data from the recreated table. That identity case needs a separate fix. Callers that replace a table should discard its old blob handles. ## Validation - `cargo test --quiet --features remote -p lancedb --lib remote::table::blobs::tests` (24 passed) - Rebuilt the Python extension; `uv run --extra tests pytest python/tests/test_remote_db.py -q -k remote_blob` (6 passed) - `cargo clippy --quiet --features remote -p lancedb --lib --tests --examples` - `cargo fmt --all -- --check`, `ruff check .`, and `ruff format .` Fixes #4276 <!-- lance-gatekeeper-fix:v1 agent=e309bdf75b1a87f6492a6dc3ee1bfd81 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-09-29 08:35:19 -05:00
Yang Cen	b8af687286	feat(sql): expose composable native vector relations (#4340) ... ## Summary Native vector dedup currently materializes through the native MV refresh protocol. Add composable DataFusion pair and retained-row relations so a host can use its existing SQL job and MV writer instead. `VectorQueryTable` binds a retained snapshot, selects all/first/random nonempty IVF scopes before dispatch, and exposes `DuplicatePairsExec` as its physical child. Pair queries can include segment, partition and snapshot columns and fetch a scalar application key for each endpoint without reading source vectors for scoring. The default primitive schema remains unchanged. `VectorDedupExec` consumes the pair stream, reuses the spillable greedy direct-representative selection, and scans retained original rows from that snapshot. A–B–C without A–C retains A and C. This query path has no durable partition receipts or writes; publication and failure belong to the enclosing SQL sink. Existing native MV APIs remain available. `BaseTableAdapter::vector_query_snapshot(None)` captures latest without changing a shared table handle's pin; `Some(version)` selects a retained version. A host can retain unbound SQL and resolve latest again for each refresh. Single-segment random sampling matches Sophon's `ivf_partition` SplitMix64 policy; multi-segment sampling ranks canonical nonempty scopes. Host integration and SQL syntax: https://github.com/lancedb/sophon/pull/8135. The host authorizes sources, binds named arguments, distributes the pair child, and uses its existing SQL Job Registry and atomic MV replacement. ## Validation All manual validation ran on an AWS r8i.8xlarge VM, not the developer laptop. - Focused Rust integration suite: 2 passed, including multiple segments/partitions, empty results, threshold boundaries, 8,193 rows per segment, deterministic sampling, mapped application IDs, and native-oracle pair equality. - Additional compacted stable/nonstable row-ID and blob test: 1 passed, including application-key mapping, null vectors, snapshot deletions and explicit/latest version capture. - Rust formatting and focused Clippy with warnings denied passed. - Dependent Sophon SQL-job E2E passes with two worker PEs: pairs, sampling, direct representatives, latest/fixed snapshot refreshes, and blob payloads. Killing a SQL worker after partial staging preserves the old MV; full rerun commits the complete replacement. Performance measurement is in progress in the dependent PR; no performance claim is made here.	2026-09-29 13:24:21 +08:00
lancedb-gatefixer[bot]	8e3606fbe3	fix: handle remote blob fetch request caps (#4309) ... ## Cause Remote `fetch_blobs` sent every row ID in one Cloud request, exceeding the server's 1024-row or 64 MiB byte cap. A single blob over the byte cap could not be read through this method. ## Fix - Send at most 1024 row IDs per request and split batches further when the server reports the blob-byte cap, preserving input order and nulls. - Resolve one exact dataset version before multi-request reads. If an initial single request hits the byte cap, resolve the version before any successful split or Range read; use that version for every resulting request. - Read an individual oversized blob through the existing Range route. - Document the per-request caps and transparent handling in the Rust, Python, and TypeScript APIs and generated JS docs. ## Validation - Focused Rust `fetch_blobs` tests: 26 passed, including row-count, byte-cap, Range fallback, and exact-version assertions. - Rust clippy and formatting passed. - Focused Python blob tests: 13 passed; Ruff lint and formatting passed. - Node build, lint, docs generation, and focused `fetchBlobs` test passed. Fixes #4275 <!-- lance-gatekeeper-fix:v1 agent=6ae94d49bf1d4cae057bba72584e7b90 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-09-28 20:31:36 -05:00
brekenandbreken-ai	c119f0c8e5	fix(python): rank each result list separately in RRF multivector rerank (#4325) ... ## What's wrong `RRFReranker.rerank_multivector` concatenates every vector result list and passes the concatenation to `rerank_hybrid` as a single list. `rerank_hybrid` then scores each row by its position in that concatenation, so a row's rank depends on how many rows came before it in *earlier* lists. With two searches of 3 rows each (`K=60`): | row | rank in its own list | score today | RRF score | |---|---|---|---| | 1 (top of list 1) | 1 | 1/61 | 1/61 | | 3 (third of list 1) | 3 | 1/63 | 1/63 | | 4 (top of list 2) | 1 | **1/64** | 1/61 | | 5 (second of list 2) | 2 | **1/65** | 1/62 | So the best hit of the second search ranks below the worst hit of the first, and the order of the lists passed in decides the fused ranking. `MRRReranker.rerank_multivector` already scores each list on its own ("MRR semantics require treating each vector result as a separate ranking system"). RRF is defined the same way. ## Fix Score each result list with its own ranks (`1 / (rank + K)`, summed across lists for rows found by more than one), then dedupe the concatenation by `_rowid`, attach `_relevance_score`, and sort. The `return_score` handling is unchanged. ## Tests `test_rrf_multivector_ranks_each_result_list` in `python/python/tests/test_rerankers.py`: - On `main`, it fails with `assert 0.015625 == 0.01639344262295082` (the top hit of list 2 got `1/64` instead of `1/61`). - With the fix, it passes, and so do the other tests in `test_rerankers.py` (16 passed, 12 skipped for missing API keys or models). - `ruff format --check` and `ruff check` are clean. The fix was found and written by an AI agent (Breken) and checked against the tests above. Co-authored-by: breken-ai <312387581+breken-ai@users.noreply.github.com>	2026-09-28 20:19:51 -05:00
ningsh7	ef11e44adc	feat(node): expose allowExternalBlobOutsideBases in Table.add (#4317) ... ## Summary - Add `allowExternalBlobOutsideBases` to the Node `AddDataOptions` and forward it through N-API to the Rust `AddDataBuilder`. - Document its default and local-only external blob behavior. - Cover URI strings and URI objects, option scoping, progress callbacks, overwrite mode, and remote rejection. ## Validation - `pnpm test --runInBand` (21 suites passed, 1 skipped; 908 tests passed, 6 skipped) - `pnpm lint-ci` - `pnpm tsc` - `pnpm run docs` - `cargo fmt --all -- --check` - `CC=clang cargo clippy --profile ci -p lancedb-nodejs --all-features -- -D warnings` Fixes #4267	2026-09-28 20:19:36 -05:00
brekenandbreken-ai	70a1123404	fix(python): write NaN and infinity as SQL literals in table.update (#4326) ... ## What's wrong `value_to_sql` renders floats with `str()`, so `float("nan")` becomes `nan` and `float("inf")` becomes `inf`. SQL reads those as column names: ```python table.update(where="id = 1", values={"price": float("nan")}) # ValueError: Invalid input, Schema error: No field named nan. Valid fields are id, price. ``` The same happens for `math.inf`, `-math.inf`, `np.nan` and `np.float32("inf")`, which go through the same float path. The only workaround today is `values_sql`. ## Fix Emit `CAST('NaN' AS DOUBLE)`, `CAST('Infinity' AS DOUBLE)` and `CAST('-Infinity' AS DOUBLE)` for these values. Finite floats render as before. The cast also lands correctly in `float32` columns. ## Tests `test_value_to_sql_nan_and_infinity` in `python/python/tests/test_util.py` updates `float64` and `float32` columns to NaN, +inf and -inf, then reads them back: - On `main`, it fails with `ValueError: Invalid input, Schema error: No field named nan`. - With the fix, it passes. All of `test_util.py` passes (75 passed, 1 skipped), and so do the update tests in `test_table.py` (`-k update`, 8 passed). - `ruff format --check` and `ruff check` are clean. NaN inside a list value (for example a vector) still fails, because Lance only accepts literals inside array values. This PR covers scalar columns only. The fix was found and written by an AI agent (Breken) and checked against the tests above. Co-authored-by: breken-ai <312387581+breken-ai@users.noreply.github.com>	2026-09-28 20:16:26 -05:00
Mohammad Hijjawi	580240278f	fix(python): apply the default limit of 10 to sync hybrid queries (#4344) ... ## Summary A synchronous hybrid query without an explicit `.limit()` can return up to 20 rows. The documented default is 10, and `AsyncHybridQuery` already falls back to `DEFAULT_HYBRID_LIMIT` (10). In `LanceHybridQueryBuilder`, each sub-query (vector and FTS) falls back to its own default of 10 rows. The fused result was then sliced with `limit=None`, i.e. not truncated at all. When the two legs match different rows, the caller gets the union. t = db.create_table("t", [{"text": ("dog" if i >= 20 else "cat") + f" {i}", "vector": [float(i), 0.0]} for i in range(40)]) t.create_index("text", config=FTS()) len(t.search(query_type="hybrid").vector([0.0, 0.0]).text("dog").to_arrow()) # before: 20, after: 10 For the same reason, `.offset(n)` without `.limit()` did not raise the sub-query limits to cover the skipped prefix; that only happened inside `if self._limit:`. ## Changes - Use `DEFAULT_HYBRID_LIMIT` when no limit is set, both for the sub-query limits (`limit + offset`) and for the final slice of the reranked results, matching `AsyncHybridQuery`. ## Test plan - [x] New `test_hybrid_query_default_limit` in `test_hybrid_query.py`. It is the sync counterpart of `test_async_hybrid_query_default_limit`, plus an `offset()`-without-`limit()` check. It fails before the fix (20 rows) and passes after. - [x] `pytest python/tests/test_hybrid_query.py python/tests/test_rerankers.py` - [x] `ruff format --check` / `ruff check` I have no local Rust toolchain, so I ran the tests against the released 0.39.0 native module with the changed Python files overlaid. The changed functions are identical between 0.39.0 and main.	2026-09-28 20:12:00 -05:00
3 people	1a463f3633	fix(python): warn when optimize cleans up all old versions (#4337) ... ## Why `optimize(cleanup_older_than=timedelta(0))` deletes the data files of every version except the latest. Any other handle still on an older version then fails with a missing-file error, and nothing tells the caller this can happen. #2470 asks for a docs note and a runtime warning. Fixes #2470 ## Scope - `AsyncTable.optimize` emits a `UserWarning` when `cleanup_older_than` rounds to 0 ms or less. `LanceTable.optimize` runs through it, so sync callers get the warning too. - The `cleanup_older_than` docstring on `Table.optimize`, `LanceTable.optimize`, and `AsyncTable.optimize` gets a `.. warning::` block, in the same style as the existing `delete_unverified` one. - Removed the function-local `import warnings` in `AsyncTable.optimize`. `warnings` is already imported at module level, and the local import would shadow it for the new check (`UnboundLocalError`). - `test_optimize` and `test_optimize_delete_unverified` intentionally use a 0 cleanup, so they now assert the warning with `pytest.warns`. Nodejs `cleanupOlderThan` is out of scope, since the issue is filed against the Python SDK. ## Tradeoffs Like the existing `retrain` deprecation warning, the warning is raised inside the async coroutine. For sync callers it is attributed to `asyncio/events.py` rather than the caller's line. Warning from both the sync and async entry points would duplicate the check for a small gain. ## Verification - Repro: a second handle checked out at an older version, then `optimize(cleanup_older_than=timedelta(0))` on the first. The second handle's read fails with `LanceError(IO) ... Not found: .../data/<file>.lance`. Before this change no warning is emitted. After it, the `UserWarning` fires. - New tests `test_optimize_warns_on_zero_cleanup` (async, and checks that `timedelta(days=1)` and the default don't warn) and `test_optimize_warns_on_zero_cleanup_sync` fail on main (`DID NOT WARN`) and pass with the fix. The failing tests are in their own commit before the fix. - `pytest python/tests/test_table.py -m "not slow and not s3_test"`: all pass except `test_merge_insert`. That test's `timeout=timedelta(0)` assertion fails the same way on unmodified main on my machine. - `ruff format --check` and `ruff check` are clean on the changed files. Tests ran against the source tree with the `_lancedb` extension from the matching `0.40.0b11` wheel. The change touches no Rust. This PR was written by an AI coding agent (Claude Code) and reviewed before submission. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Jay Zhou <190699450+jayzhou2309@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 20:11:47 -05:00
brekenandbreken-ai	d8063545d9	fix(node): write NaN and Infinity as SQL literals in table.update (#4327) ... ## What's wrong `toSQL` renders numbers with `toString()`, so `NaN` becomes `NaN` and `Infinity` becomes `Infinity`. SQL reads those as column names: ```ts await table.update({ values: { price: NaN }, where: "id = 1" }); // Error: Invalid input, Schema error: No field named "NaN". Valid fields are id, price. ``` `Infinity` and `-Infinity` fail the same way. The only workaround today is `valuesSql`. ## Fix Emit `CAST('NaN' AS DOUBLE)`, `CAST('Infinity' AS DOUBLE)` and `CAST('-Infinity' AS DOUBLE)` for these values. Finite numbers render as before. ## Tests - `__test__/util.test.ts` adds `toSQL` cases for `NaN`, `Infinity` and `-Infinity`. - `__test__/table.test.ts` adds "should let me update float values to NaN and Infinity", which updates a float column and reads the values back. - On `main`, the new cases fail with `No field named "NaN"`. - With the fix, all of `util.test.ts` and `table.test.ts` pass (344 tests). These were run against the published `@lancedb/lancedb@0.40.0-beta.11` native binary. - `biome check` is clean on the changed files. The Python SDK has the same issue in `value_to_sql`. That fix is in #4326. The fix was found and written by an AI agent (Breken) and checked against the tests above. Co-authored-by: breken-ai <312387581+breken-ai@users.noreply.github.com>	2026-09-28 20:11:39 -05:00
3 people	0327618a62	feat(sql): add native vector dedup and partitioned mv refresh (#4303) ... ## Summary Materialize a source-preserving dedup result from an indexed, pinned snapshot with a short declaration. The default greedy policy keeps the smallest unassigned snapshot row ID and removes only its directly matching neighbors: For A < B < C, an A–B–C chain without an A–C edge keeps A and C. ```python from lancedb import vector_dedup view = db.create_materialized_view( "images_clean", vector_dedup("images", column="phash", distance_threshold=4), ) cleaned = view.table ``` The Python source expression captures the source version once at creation and waits for the materialized result by default. Use `dataset_version=...` for an explicit snapshot, `with_no_data=True` for declaration only, or `create_materialized_view_async(...)` for a job handle. AsyncConnection supports the same expression. It uses the existing MV REST endpoints and registry jobs, without opening a SQL query connection. Source names are exact root-namespace table names. Additional select/where/limit options are rejected for this source. The SQL declaration remains available: ```sql CREATE MATERIALIZED VIEW images_clean AS SELECT * FROM vector_dedup('images', 7, 'phash', 4); REFRESH MATERIALIZED VIEW images_clean; ``` This is the host-facing SQL entry point. The MV builder now accepts a typed native vector source; the Rust SDK also supports the same definition through `MaterializedViewDefinition::from_sql` and `prepare_definition`; `view.refresh().execute()` executes it locally. Hosts distribute its units using their existing job infrastructure. The raw `vector_duplicate_pairs` source remains available separately when callers want the pair relation. ## Implementation and contract - Use actual Rust Lance `13.0.0-beta.15` (`db53dcf367`); the native partition primitive computes pairs from index representations without source-vector aggregation or a Python pair loop. - Extend partitioned MV refresh with two dependent waves. Physical segment/partition tasks stream pairs through a bounded external sort, select direct representatives and write per-source-fragment bitmap artifacts. Source-fragment tasks start after every selection receipt completes and stream retained original rows, including blob payloads, into uncommitted output fragments. - Reuse normal blob write coercion and row-mask scans, including stable row-ID translation after compaction. Null vectors, isolated rows and source deletions are preserved correctly. Conflicting representative decisions across overlapping index scopes fail before publication. - Expose `dependency_units()` and `write_refresh_partition_with_inputs` alongside the existing plan/commit API. Receipts bind snapshot/configuration/policy/destination generation. Only a complete accepted set can publish; unfinished units restart, completed units can be reused. There is no pair-position checkpoint. - Use MV format 4 for dedup results and format 3 for native pairs; Python and TypeScript metadata readers recognize both. No source deletion or source row-ID migration occurs. Only same-segment/same-partition edges are covered. Quantized scores use symmetric represented-vector distances. Result materialization requires V2 data storage and is initially a native MV source, not an ordinary SELECT table function. Compressed vertex masks stay in memory; edge sorting spills to local disk. Unreferenced artifacts use normal Lance retention/cleanup and must be retained for the job's recovery window. ## Validation AWS Linux validation: - Native pair/result integration matrix passed: multiple segments/partitions, exact thresholds, empty output, cross-batch data, single-process pair parity, source preservation, lost receipts and publication guards. - Blob/null-vector/pinned-deletion tests passed with both row-ID modes, including compaction that changes physical row addresses. - Representative policy, forced external-sort spill and conflicting/corrupt artifact tests passed. - Full MV regression suite: 153 passed, 2 pre-existing ignored. Rust formatting and Clippy passed. - Python MV suite: 47 passed; TypeScript MV suite: 11 passed. Native bindings were rebuilt in project-local environments; formatting, lint and TypeScript docs passed. - Rust declaration doctests: 3 passed, including the complete local dedup example. - Companion host validation: 41 MV job tests passed (1 pre-existing ignored), full workspace Clippy passed, and the complete process-level CLI E2E suite passed (3 tests). Recovery exercises both selection and materialization waves. - A real CIFAR-10 pHash run used five actual registry-worker owners, completed 32 selection + 24 materialization tasks, and retained 49,846 of 50,000 rows with identical original values and image payloads. Host workflow performance measurements are recorded in the companion host PR. Cross-platform CI is still running. The previous remote-test bootstrap failed before Cargo tests with `401 Bad credentials`; the workflow fix described below is being validated by CI. No deployment or merge is performed. ## Python source validation AWS tests cover automatic and explicit snapshots, physical/stable row IDs, local and remote creation, sync/job/async-connection APIs, delayed refresh, strict options and identifier quoting. The native Python extension was rebuilt through the project's uv/maturin backend. Ruff, Pyright and the full MkDocs API-reference build passed; both new public symbols are rendered. A real 50,000-image FEv2 run disabled both Python SQL entry points and supplied an unreachable SQL endpoint. The new API still materialized the same 49,846 retained rows and original image payloads. Three recorded creation/refresh jobs completed 56 tasks each on all five registry-worker owners, and remained on source snapshot 3 after source latest advanced to 4. No worker or server wire-protocol changes were needed for this client entry point. Previous CI authentication failure: https://github.com/lancedb/lancedb/actions/runs/36334679276/job/108663151265. ## Review and CI fixes - Validate all JSON merge source fields before projection, rejecting unknown and duplicate names at every struct level. Missing/reordered fields and ordinary JSON merges remain supported. A checked-in 12-case integration matrix covers root fields, structs, lists, large lists, fixed-size lists and map values; rejected writes leave the table version and row count unchanged. - Match namespace connection MV method signatures to `MaterializedViewSource` in all four sync/async methods, fixing the Python 3.10 import failure from runtime override checks. - Restore the remote-test script's encrypted-CLI path: use the Actions token for artifact download and pass `SOPHON_READ_TOKEN` under the variable the decryption script consumes. The previous workflow omitted that variable and attempted the private release path, failing with HTTP 401. On the final head, remote CI passes: 1,464 Rust unit tests, the integration targets and doctests pass (3 unit tests ignored). AWS validation for these fixes: both review reproducers failed before their fixes and pass afterward; 55 merge tests, 29 cast tests and the standalone 7-case admission regression pass. Python 3.10.21 imports successfully; 90 MV/namespace tests, 14 JSON tests and four real namespace dedup cases pass. After `794b97a0`, the native binding was rebuilt again and the JSON suite rerun. Rust formatting/Clippy, Ruff and Pyright pass. Sophon's locked test compilation passes against the final SDK source for all seven affected crates. Extended runners, logs and benchmark artifacts remain local. Gatekeeper recommends approval on `794b97a0`, and the `breaking-change` label is set. Final CI: **49 checks passed, 3 expected skips, no failures or pending checks**, including remote tests, Python 3.10, minimum dependencies and Windows ARM Rust tests. All wheel builds pass after retrying the Linux matrix affected by transient AlmaLinux repository-metadata 404s. All code-review threads are resolved. The PR remains open for maintainer review and merge. --------- Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local> Co-authored-by: Lance Release <lance-dev@lancedb.com>	2026-09-28 18:06:55 +08:00
Ayush Chaurasia	139382b5f7	feat(python): pass batch_size and other CrossEncoder kwargs (#4328)	2026-09-25 21:50:55 +05:30
LanceDB Robot	d0fb43b0b4	chore: update lance dependency to v13.0.0-beta.14 (#4324) ... Update the Rust workspace Lance dependencies and Java lance-core from v13.0.0-beta.12 to [v13.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.14), including the refreshed Cargo lockfile. No compatibility fixes were required; `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed.	2026-09-25 09:02:00 -05:00
Dan RammerandClaude Opus 5	2216586023	feat(query): dedup hybrid search by primary key on MemWAL tables (#4268) ... Implementing a PK-based dedup to support WAL in hybrid search. Basically, the existing code just calls a vector query + FTS query and dedups locally using the row ID. Since row ID is unsupported in WAL (currently), we update the algorithm to use the tables PK (which is required in WAL, and uniquely linked to a PK on updates). To enable PK-dedup branch, we've chosen to fallback on error rather than fire an exploratory query. Specifically, we issue the regular hybrid query, using row IDs, and when that fails with "WAL does not support row IDs" we fallback to the PK-based variant, and then cache the fallback for future calls from the same client. The alternative is that we chec if WAL is enabled before issuing the query. This would require a `get_lsm_write_spec` call, slowing down all the existing non-WAL paths unnecessarily. The ultimate goal is to get row ID supported in WAL, but that is pending stable row ID GA and subsequent integration into the enterprise stack. Once that lands, we will rip out the PK-based dedup that is only for WAL-enabled tables and use the existing row ID code. This is the main reason for building this adaptively. I had spec-ed out a refactor so that rather than duplicating code between python and rust we build this in rust and have python call that through PyO3. However, that made the reverse path (ie. ) more difficult. So I think the tradeoff of maintaining duplicate code in the short term for the ease of reverting this in the medium term (hopefully 2-3 months). --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-09-24 22:02:04 -05:00
Wyatt Alt	604b1b583b	feat: pause and resume jobs from the Python SDK (#4112) ... Cancellation is the only job control the SDK exposes, and it is terminal, so a long-running server-side job cannot be parked and picked up again. This adds `pause_job` and `resume_job` to the Rust core connection and the Python bindings (sync and async), posting to the server's `/v1/jobs/pause` and `/v1/jobs/resume` endpoints. A pause parks the job until it is resumed: its workers drain and stop. The outcome strings mirror the server's answers — a job finalizing its results reports `"committing"` and cannot be parked, and a resume before the drain is confirmed reports `"still_pausing"`; both are retried rather than failed. Resuming re-queues the job and its workers pick their work back up from checkpoints. Local connections report the operations as unsupported, like the rest of the jobs API.	2026-09-24 23:47:00 +08:00
Lance Release	11c637252f	Bump version: 0.40.0-beta.10 → 0.40.0-beta.11	2026-09-24 10:05:13 +00:00
Xuanwo	0b18dcdaec	fix(python): warn instead of failing on local imports in @udf source (#4318) ... Since 0.40.0b8 (#4254), `@udf` raises when the packaged source imports a module from a local source tree that is neither shipped with `code=` nor named by a declared pip/conda package. That turned registrations that used to succeed into errors: - Callers that rewrite the packaged source before registering it now fail at decoration. Sophon's datagen integration suite inlines its test-module helpers this way, and `test_registered_udf_source_stands_alone_on_a_worker` fails on 0.40.0b8. - The check cannot know which modules a distribution installs. A declared package whose distribution name differs from its module (for example an internal library installed editable) is reported as local, and there was no way to proceed. This keeps the diagnosis but makes it a `UserWarning`, so registration behaves as it did before #4254 and still tells the author, at registration time, why the worker is likely to fail.	2026-09-24 18:02:25 +08:00
Taylor	a6dcbba49e	feat(python): add opt-in request batching to TypeSafeReranker (#4316) ... Adds opt-in `batch_size=40`: 80 non-null candidates use **2 requests instead of 80**. Default `batch_size=1` preserves the existing payload. Each batched question sees only its document and the shared query; instructions/criteria stay unchanged. Prompts referencing `state.document` need adaptation. Concurrency still limits requests; retries remain SDK-managed. Batch-local IDs and ordered results preserve duplicate documents; invalid IDs/probabilities raise without fallback. Validation: **124 automated tests passed**, plus live search integrations in both modes. A **100-query live subset test of the initial implementation** (6,851 candidate pairs; `jev-1.13.0`, SDK 0.7.1, concurrency 32) passed: unbatched/batched calls **6,851/199**, median **877/351 ms**, p95 **9,446/598 ms**. Hybrid Hit@5/Hit@10 was **83%/89% vs 84%/90%**; vector/FTS results and exact configuration are in the [test report](https://github.com/lancedb/lancedb/blob/typesafe-request-batching/python/benchmarks/typesafe_batching.md). Ruff and MkDocs passed. The simplified batching flow also passed a fresh four-query live check: 292 candidate pairs, with 292 unbatched versus 8 batched calls. Builds on #4209; motivated by [research #7](https://github.com/lancedb/research/pull/7).	2026-09-24 13:53:52 +05:30
Lance Release	b3f0e12e32	Bump version: 0.40.0-beta.9 → 0.40.0-beta.10	2026-09-24 07:50:23 +00:00
Jack Ye	e4d0f9e2da	feat: return a cleanup job from drop_view (#4262) ... Dropping a view unbinds its name and leaves the definition dataset to a server-side cleanup job, so the two are separate events a caller may want to wait on. `drop_view_async` returns that job — the same shape `drop_materialized_view_async` and `drop_function_async` already use: a `202` carries the job id, a `200` (nothing was bound to the name) yields an already-finished job with no id, and any other success status is an error rather than a silent no-op. ## `drop_view` waits `drop_view` now awaits the job before returning, so a caller who does not want to think about cleanup gets the stronger guarantee: when it returns, the definition really is deleted. That is deliberately **different** from `drop_materialized_view` and `drop_function`, which return as soon as the name is unbound and document that content may still be deleting. The view API is the newer one, and waiting is the semantic worth having; the other two are left alone rather than changing behaviour already released. ## Surfaces `Database` trait, the remote client, `Connection`, and the Python and Node bindings — matching where `drop_materialized_view_async` is already exposed. Four client tests cover the accepted case reporting its job id, the nothing-bound case reporting a finished job, a `202` without a usable `job_id`, and an unexpected success status.	2026-09-24 00:27:20 -07:00
Lance Release	85c9cf6240	Bump version: 0.40.0-beta.8 → 0.40.0-beta.9	2026-09-23 23:40:53 +00:00
3 people	53a2afdffa	chore: update lance dependency to v13.0.0-beta.12 (#4265) ... Update the Rust workspace Lance dependencies and Java lance-core from v13.0.0-beta.8 to [v13.0.0-beta.12](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.12), refreshing Cargo.lock. No compatibility fixes were required; validation passed with `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. --------- Co-authored-by: Lu Qiu <luqiujob@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-23 14:31:35 -07:00
lancedb-gatefixer[bot]	85daeb27cb	fix: isolate remote table checkout state per handle (#4310) ... ## Cause `RemoteDatabase` cached and returned the same `Arc<RemoteTable>` for every open of a table on one connection. Checkout mutates version and freshness state on that shared object, so pinning one handle also pinned the others. ## Fix Cache the table's server version rather than its mutable handle. Each `open_table` now creates its own `RemoteTable` with independent checkout, schema cache, and freshness state. ## Validation - Added a cache-enabled regression test that failed before the fix: pinning one handle changed another handle's reported version. It now verifies a write through the independent handle succeeds. - `cargo test --quiet --features remote -p lancedb --lib remote::db::tests` (90 passed) - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo fmt --all` Fixes #4296 <!-- lance-gatekeeper-fix:v1 agent=38593bfbce0279286da9850fe61050e0 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-09-23 14:27:44 -07:00
lancedb-gatefixer[bot]	75e71176cb	fix(python): prevent BlobFile finalizer panics on Tokio workers (#4291) ... ## Cause `io.RawIOBase.__del__` checks `closed` and calls `close()` during GC. Both PyO3 methods entered `block_on`, which panics when GC runs on a LanceDB Tokio worker. ## Fix Track Python blob handle closure atomically so `closed` is synchronous. `close()` marks the handle closed immediately and, when called on a Tokio thread, schedules the underlying async cleanup without blocking that thread. Reads and seeks after close fail immediately; ordinary callers still wait for cleanup. ## Validation - Rebuilt the Python extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`. - Ran the blob test file and remote blob handle test: 69 passed, 1 skipped. - Ran `cargo fmt --all`, `ruff format .`, and `ruff check .`. Fixes #4283 <!-- lance-gatekeeper-fix:v1 agent=15a7bd545e63dea4b69faa0429f0a684 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>	2026-09-23 14:24:04 -07:00