rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-12 23:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,105 Commits 374 Branches 556 Tags
refactor/split-remote-module
Commit Graph
3105 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Claude bd3a16db32 refactor(rust): split remote table module into submodules 
Move RemoteTable helpers, freshness tracking, tags, describe/IPC helpers,
multipart writes, index parsing and MergeInsertRequest into submodules, and
move the branch, LSM and column-operation bodies of the BaseTable impl into
inherent methods so every file under remote/ stays below 1500 lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BgkiPYS2eTmAWVyqn6KBnN
 2026-10-07 07:11:39 +00:00
Claude 8fcb10b339 refactor(rust): split remote db module into submodules 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BgkiPYS2eTmAWVyqn6KBnN
 2026-10-07 06:41:35 +00:00
Claude fb76471a57 refactor(rust): split remote sql and oauth modules into submodules 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BgkiPYS2eTmAWVyqn6KBnN
 2026-10-07 06:36:03 +00:00
Claude 6dde01de33 refactor(rust): move remote module unit tests into separate files 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BgkiPYS2eTmAWVyqn6KBnN
 2026-10-07 06:23:00 +00:00
lancedb-gatefixer[bot] bbc4aebe55 fix(python): infer vector columns for sync remote search (#4409) 
Sync remote string searches ignored the table's embedding function when
`vector_column_name` was omitted. `auto` silently fell back to full-text
search, while `vector` and `hybrid` failed with `No embedding function
for None`.

Resolve vector columns from the schema before creating the query
builder, using the same inference helper as local search. Automatic
string queries without embedding functions select full-text search
first, so text-only tables and tables with multiple unconfigured vector
columns remain searchable. Vector and hybrid searches still reject
ambiguous columns, infer numeric queries by dimension, and honor
explicit column selection. Structured full-text queries and empty
queries retain their existing behavior.

Regression coverage uses embedding metadata returned by a mocked remote
server rather than a mocked embedding-function lookup. It also checks
automatic and explicit full-text queries on schemas with zero, one, or
multiple vector columns and no embedding functions.

Validation after merging `main` at `87ecfeac`:

- Confirmed the original embedding and ambiguity failures before the
fix, and reproduced the automatic FTS inference failures before
restoring the fallback.
- Remote, query, hybrid, and embedding test files through the uv-managed
environment: 240 passed, 4 skipped, 9 deselected (`not slow and not
s3_test`).
- Python extension built with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`.
- `ruff check .` and `ruff format .` from the repository root, using
CI's Ruff version 0.9.9.
- `cargo fmt --all`.

Fixes #4377

<!-- lance-gatekeeper-fix:v1 agent=c167c6c89d857342e5d7415b08142589
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-06 19:59:21 -07:00
lancedb-gatefixer[bot] a8baecb306 fix: reject non-finite query vectors without panicking (#4410) 
Remote vector serialization unwrapped `serde_json::Number::from_f64`,
which returns `None` for NaN and infinity, causing a Tokio worker panic.

Validate vectors after conversion to Float32 in the shared `nearest_to`
/ `add_query_vector` builder so local and remote queries consistently
return `Error::InvalidInput` (mapped to `ValueError` by the Python
bindings). This also rejects finite Float64 inputs that overflow
Float32. Make remote serialization fallible to protect requests
constructed directly, including single-vector queries and both remote
batch-query paths.

Validation:

- All 116 query-related Rust tests passed with the `remote` feature,
including the three new regression cases.
- The three regression cases fail with the original production code and
pass with the fix. Coverage includes NaN, positive and negative
infinity, Float16/32/64 Arrow inputs, Float64 overflow, and rejection
before any HTTP request.
- `cargo fmt --all -- --check`, `cargo check --profile ci --locked -p
lancedb --features remote --tests --examples`, and `cargo clippy
--profile ci --locked -p lancedb --features remote --tests --examples --
-D warnings` passed.

Fixes #4375

<!-- lance-gatekeeper-fix:v1 agent=97d6925d4af7cd09767da8b552c9bcf3
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-06 16:26:25 -07:00
lancedb-gatefixer[bot] 87ecfeaccd fix(python): preserve embedding functions in sync remote table creation (#4408) 
Sync remote `create_table(..., embedding_functions=...)` discarded the
supplied configuration and only logged an unsupported-feature warning.
Schema-only creation therefore lost the metadata needed by `add()` to
embed text, while text-only initial data produced no vector column.

Forward the configuration to the existing async creation path, remove
the stale warning, and document the parameter. The shared path stores
the embedding metadata and generates vectors for initial data and
subsequent additions.

Add a regression test against the mocked remote HTTP API for both
schema-only and text-only creation. It checks the outgoing Arrow schema
metadata, the metadata returned by the remote table, inferred vector
types, and generated vectors during creation and `add()`.

Validation:

- Both regression cases failed before the fix because the outgoing
embedding metadata was missing, then passed after the fix.
- `uv run --extra tests --extra dev pytest
python/tests/test_remote_db.py python/tests/test_embeddings.py -q -m
'not slow and not s3_test'`: 82 passed, 4 skipped, 8 deselected.
- Repository-wide `ruff check .` and `ruff format .` passed using the
uv-managed environment; `git diff --check` passed.

Fixes #4376

<!-- lance-gatekeeper-fix:v1 agent=a6816b1f3335438bd8ad192f36395213
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-06 16:18:10 -07:00
lancedb-gatefixer[bot] 3385dc55dc fix(remote): validate schema for create_table with exist_ok (#4407) 
Remote `create_table` with `exist_ok` opened an existing table without
validating the supplied schema, so Python and TypeScript silently
accepted mismatches.

Capture the request schema before consuming its data stream, compare it
with the opened table's schema, and return the same `Error::Schema` and
message as the local backend when they differ. The existing open-table
callback continues to run before validation.

Regression coverage exercises matching schemas and differences in field
name, type, and nullability, for both empty and data-backed creation
with cached and uncached table opens. All 12 mismatch cases failed
against the original implementation; all 16 cases now pass.

Validation:

- `cargo test --profile ci --features remote -p lancedb --lib
remote::db::tests`: 113 passed.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`: passed.
- `cargo fmt --all` and `git diff --check`: passed.

Fixes #4378

<!-- lance-gatekeeper-fix:v1 agent=5781a39ba124cf9094d36715730342b4
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-06 16:17:01 -07:00
lancedb-gatefixer[bot] 02ed5cf129 fix: require checkout before restoring remote tables (#4406) 
Remote handles at the latest version have no checkout version, so
`restore()` sent `{"version":null}` and surfaced a server
deserialization error. Validate the checkout version before constructing
the request and return `Error::InvalidInput` with the same message as
local tables. This shared Rust fix covers Python sync/async and
TypeScript callers.

Add a regression test proving that restore sends no request on a fresh
handle, after `checkout_latest()`, or after a successful restore clears
checkout state. It also verifies that a valid restore sends the
checked-out version. Update the schema-cache and branch restore tests to
check out a version before restoring and require success.

Validation:

- The new regression failed against the original implementation when the
mock received a null version.
- `cargo test --locked --profile ci --features remote -p lancedb --lib
restore` — 6 tests passed.
- `cargo fmt --all` and `git diff --check` passed.
- `cargo clippy --quiet --locked --profile ci --features remote -p
lancedb --tests --examples` passed.

Fixes #4381

<!-- lance-gatekeeper-fix:v1 agent=cc67544dfcbb46d631762752d55b39e3
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-06 16:16:13 -07:00
lancedb-gatefixer[bot] 6b6d55e6c7 fix: accept updated_rows in remote update responses (#4444) 
Remote update responses report their count as `updated_rows`, but the
client only recognized `rows_updated` and silently defaulted the count
to zero. Python sync/async and TypeScript both use this Rust result.

Accept `updated_rows` as a Serde alias while retaining `rows_updated`
and compatibility with older responses. Extend the remote mock test to
cover both keys, zero and missing counts, returned versions, and empty
responses, with and without a predicate.

Validation:

- Both nonzero `updated_rows` regression cases failed before the fix (0
instead of 5); all 11 focused remote tests pass after it.
- All 3 native update tests pass.
- `cargo fmt --all`
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`

Fixes #4443

<!-- lance-gatekeeper-fix:v1 agent=423249073212c9840d96ccf6d4452905
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-06 16:14:57 -07:00
XY Zhan 2bf70758a7 fix: read back the maintained-index selection a table was given (#4447) 
`maintained_indexes` on a MemWAL table has three values: a named list,
`[]` for none, and `null` for every index the table has, including ones
created later. This PR makes every client read all three back as they
were set.

- **Rust remote client** (also used by the Python and Node SDKs) reads
`null` as `None`.
- **Java client** reads `null` as `null`, not as an empty list.
- **Docs** for the setter, the field and the getter describe the three
values the same way in Rust, Python, Node and Java. The generated JS
docs are regenerated to match.

## Testing

- `rust/lancedb`: 73 LSM tests pass, including one case per value. With
the decoder typed as a list, the `null` case fails.
- Java: two cases for `null` and `[]` on the read path, run by the Java
CI job.
- `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean.
 2026-10-06 19:07:10 -04:00
Wyatt Alt dbd86ed9b7 feat: let CountPushdown see through MetadataEraserExec (#4458) 
BaseTableAdapter wraps every scan in MetadataEraserExec, which reported
an unknown cardinality effect. Lance's CountPushdown only looks through
nodes that keep every row, so a SQL COUNT(*) over the adapter always
scanned, even with the rule registered. The eraser now reports Equal.

with_new_children kept the old schema and panicked when the new child's
schema differed; it now rebuilds from the child.
 2026-10-06 12:02:47 -07:00
LanceDB RobotandWyatt Alt 17e9a657e6 chore: update lance dependency to v14.0.0-beta.1 (#4445) 
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v14.0.0-beta.1](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.1).
No compatibility fixes were required.

Validation passed: `cargo clippy --quiet --workspace --tests
--all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff
--check`.

---------

Co-authored-by: Wyatt Alt <wyatt.alt@gmail.com>
 2026-10-06 08:34:45 -07:00
Lance Release 81d52a98f8 Bump version: 0.40.0-beta.12 → 0.40.0-beta.13 2026-10-06 04:36:52 +00:00
Jack Ye 7e8f0d4982 refactor: always fully refresh materialized views (#4373) 
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.

## Current layout

Local materialized views currently maintain incremental-refresh state
alongside the materialized values:

- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.

Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.

## New layout

An MV is an ordinary Lance backing table containing only:

- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.

There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.

Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.

## New remote create workflow

```text
Connection::create_materialized_view(...)
        -> builder renders the defining SELECT
        -> remote Database receives a CreateMaterializedViewRequest
        -> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
        -> retain the SQL Query handle while the Job is running
        -> consume the durable SQL-job result
        -> open the created table and read mv.definition
        -> return MaterializedView
```

Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.

## API changes

- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.

The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
 2026-10-05 21:33:34 -07:00
Colin Patrick McCabe d864e7198f feat!: list_databases returns an iterator (#4442) 
CatalogConnection.list_databases should return an iterator so that users
don't have to manually re-implement pagination. For those who do want
more control over pagination, add num_page_results and page_token to the
Iterator object, and also allow the iterator object to be created at a
specific page token.
 2026-10-05 19:41:53 -07:00
ccmao1130 f4544628ba fix: add UTM to reverie link, remove cloud in banner (#4446) 
- Add UTM to Reverie link for tracking
- Remove "Try LanceDB Cloud" in banner
 2026-10-05 15:19:08 -07:00
ccmao1130 6bdfe73604 fix: add reverie banner to readme (#4346) 
Add Reverie banner to the top of README until event ends.

Additionally, I also removed the "LanceDB Cloud" skinny banner since
we're not promoting that right? Should we modify the larger LanceDB
modify to not mention "Try LanceDB Cloud"?
 2026-10-05 15:03:16 -07:00
Will JonesandClaude a68c7be202 feat(node): accept ArrayBuffer, Blob/File, and URL as blob column inputs (#4313) 
Closes #4270.

## Summary

Blob v2 columns now accept more of the containers TypeScript code
commonly has, both as the column value and inside `{ data }` / `{ uri
}`:

| Input | Handling |
|---|---|
| `ArrayBuffer` | `new Uint8Array(value)`, no copy |
| `URL` | stored as `value.href` |
| `Blob` / `File` | `await value.arrayBuffer()` in the async write
paths: `createTable`, `add`, `mergeInsert` |

`makeArrowTable` is synchronous, so it can't read a `Blob`. Given one,
it throws an error that names the field and row and points to the async
APIs or `new Uint8Array(await value.arrayBuffer())`. It does not write a
null blob.

### Types
`BlobInput` is now the accepted input type. It is widened to the types
above and exported from the package entry point with `BlobData` and
`BlobUri`, so callers can annotate rows and `tsc` catches bad inputs.
The struct forms are a union that requires exactly one of `data` /
`uri`. The old normalized `{ data, uri }` shape is renamed to
`BlobValue` (internal).

### Implementation
- `blob.ts`: `normalizeBlobInput` (sync: `ArrayBuffer`/`URL`),
`resolveBlobInput` (async: `Blob`/`File`). `coerceBlobValue` normalizes
first.
- `arrow.ts`: `mapBlobInputs` walks only the blob positions in the
schema, including blobs nested in structs and lists. It copies records
and arrays only where a value changed, so caller data is never mutated.
`resolveBlobInputs` runs before `makeArrowTable` in `convertToTable`
(used by `add` / `mergeInsert`) and in `createTable`. Each distinct
`Blob` is read once.
- `makeArrowTable` now coerces every blob value to the full `{ data, uri
}` struct **before** schema inference. This also fixes two bugs that
existed before this change:
  - Rows mixing plain bytes with `{ data }` failed inference.
- A `{ uri }`-only struct produced a blob struct missing its `data`
child.

  Blob validation errors now include the field path and row.
- Also fixed along the way:
- `add` / `mergeInsert` on a table with an embedding function rejected
most blob values, because `ensureNestedFieldsExist` filled blob columns
in like ordinary structs. It now skips them.
- With a schema, a struct field that is `null` in one row and a record
in a later row failed inference. Primitives and arrays given for a
struct field are now rejected instead of being written as an all-null
struct.
- Blob structs with keys other than `data` / `uri` are rejected, which
catches typos. Class instances with `data` / `uri` are handled like
plain objects.

## Testing
- Unit tests in `__test__/blob.test.ts`: coercion of the new types,
rejection and error messages for `Blob` in the sync path, nested lists
and structs, `resolveBlobInputs` (only blob columns touched, caller data
untouched, shared `Blob` read once).
- Table tests in `__test__/table.test.ts`:
- `createTable`, `add`, and `mergeInsert` round-trip `ArrayBuffer` /
`Blob` / `File` bytes through `fetchBlobs`.
- `URL` and `{ uri: URL }` are added to the external-blob round-trip
test from #4317 (`allowExternalBlobOutsideBases`).
- Full `jest` suite passes locally after the rebase (938 passed). `pnpm
lint` is clean and `pnpm run docs` has been regenerated.

Streaming inputs (`ReadableStream`, Node `Readable`, async iterables)
remain out of scope, as the issue says.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012U4KdHofT5edv3LcXHXbLv

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-05 12:35:00 -07:00
Will JonesandClaude Opus 5.5 11b679a2db ci: add critical fixes section to release notes (#4361) 
Adds a "🚨 Critical Fixes" section to the release notes, directly below
Highlights, for PRs labeled `critical-fix`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 12:25:47 -07:00
Aviral Saxena 5f0bb98e90 fix(python): use exact integer math for list_versions timestamps (#4426) 
Refs #4404.

AsyncTable.list_versions split the nanosecond timestamp with float
division/modulo (`ts // 1e9`, `ts % 1e9`).
At ~1.8e18 a float can't hold every integer, so the microseconds drifted
(.274000 came back as .273999).

Use `divmod(ts, 1_000_000_000)` and integer // 1000 instead.

Only the precision fix is included here. Timezone-aware UTC (option 1 in
#4404) is on hold for now,
and I can add it here or in a follow-up depending on what they prefer.

Added test_list_versions_timestamp_precision using the value from the 
issue (1790989429274000000), expecting microsecond == 274000.
 2026-10-05 11:15:46 -07:00
Xuanwo 61a285f590 fix: preserve Function dependency errors in column operations (#4434) 
Function column declarations and refreshes can return HTTP 404 when the
referenced Function has been deleted. The SDK currently translates those
responses into `TableNotFound`, hiding the server diagnostic and making
an existing table appear missing.

Preserve the HTTP error, status, and request ID for these two operations
while retaining schema-cache invalidation. Ordinary table operations
keep their existing `TableNotFound` behavior.
 2026-10-05 14:34:32 +08:00
lancedb-gatefixer[bot] 7f5933594f fix(python): reject empty query vectors before column inference (#4415) 
Empty list and NumPy vector queries were validated too late: local
search inferred a vector column first, producing misleading schema
errors, while remote search indexed `vector[0]` and raised `IndexError`.

Add a shared empty-vector check before column inference and query
creation, and validate direct vector-builder inputs. Local and remote
sync search now raise `ValueError: Query vector must not be empty` for
these inputs.

Validation:

- All 26 new regression cases failed on the original code and pass with
the fix, covering list/NumPy inputs, automatic/explicit vector search,
inferred/explicit column names, one/two local vector columns, and remote
search before query submission.
- After merging current `main`, all 32 focused cases passed, covering
these regressions and upstream async searches without a query.
- The query, table, remote, utility, hybrid-query, and database suites
passed in the rebuilt uv environment: 541 passed, 1 skipped, 1 slow test
deselected.
- `ruff check .`, `ruff format .`, and `cargo fmt --all` passed from the
repository root.

Fixes #4389

<!-- lance-gatekeeper-fix:v1 agent=f1f4875090ef391bb1aa912dd54026dc
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 21:30:13 -07:00
lancedb-gatefixer[bot] 3e0a1a8780 fix: reject zero vector search probes across SDKs (#4418) 
`nprobes(0)` bypassed the validation used by the minimum and maximum
probe setters, allowing zero into query requests and producing
inconsistent local and remote results. Sync Python could additionally
interpret the zero maximum as an unbounded search.

Reject zero in the Rust vector-query builder and propagate its input
error through the Python vector/hybrid and Node bindings. Validate
nonpositive counts in the sync Python vector and hybrid builders so
every SDK and backend reports `nprobes must be greater than 0` before
execution. Update the Rust examples and document the positive-count
requirement.

Regression tests cover local and remote vector/hybrid builders,
rejection before remote query requests, promised TypeScript query
vectors, and positive counts replacing both probe bounds.

Validation:

- Rust probe regression tests: 2 passed.
- Focused Python query, remote, hybrid, and serialization tests: 22
passed. The four sync regression cases fail with the original setters.
- Focused TypeScript indexed local and mocked remote tests: 2 passed.
- Workspace `cargo check --features remote --tests --examples` and
Clippy passed.
- Rust formatting, Python formatting/lint, and TypeScript
build/lint/docs passed. Repository-wide `ruff check .` reports 29
existing errors in unchanged CI scripts; `ruff check python` passes.

Fixes #4388

BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>`
instead of `Self`; callers must handle the result, as with the existing
minimum/maximum probe setters.

<!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 21:29:47 -07:00
Chi-Wei LeeandWill Jones 0be3ae960e fix(python): support exist_ok in local create_table (#4421) 
Synchronous local Python `create_table` callers using `mode="exist_ok"`
received `ValueError` instead of creating or opening a table.

Related to [#2900](https://github.com/lancedb/lancedb/issues/2900). The
local wrapper rejected this mode before the shared Rust parser could
validate it. Removing that guard delegates mode validation to the
parser, and the sync and async docs now describe `exist_ok`. Remote
option gaps and the TypeScript table type gap remain outside this
change.

The added tests retain both calls from [@wjones127's
comment](https://github.com/lancedb/lancedb/issues/2900#issuecomment-5964167206).
This builds on the code @wjones127 posted there. Three regression tests
fail before the change, pass after it, fail with the original `db.py`
restored, then pass again on HEAD.

Tested on Linux x86_64 with Python 3.14.7. From the Python directory,
`uv run --no-sync --extra tests --extra dev python -m pytest
python/tests/test_db.py -q` reports 53 passed. `uv run --no-sync --extra
tests --extra dev python -m pytest python/tests -q -rs -m "not slow and
not s3_test"` reports 1428 passed, 34 skipped, 68 deselected, and one
failure: the existing `test_get_lsm_write_spec`, also failing on clean
base (1425 passed, one failed). Ruff 0.9.9 `ruff format --check .` and
`ruff check .` pass. Tests used worktree Python code with the released
`lancedb==0.40.0b12` native wheel. No native source build was run.

Co-authored-by: Will Jones <willjones127@gmail.com>
 2026-10-03 10:12:29 -05:00
lancedb-gatefixer[bot] 9c3e1cbae7 fix: reject cloning onto an existing local table (#4416) 
Local `clone_table` passed existing targets to Lance's clone
transaction, which entered the manifest-update path and failed with
`Clone operation should not enter build_manifest`.

Check the target dataset using the connection's storage options and
session before cloning, and return `TableAlreadyExists` when it exists.
Propagate other read errors. Add regression coverage for cloning onto
the source itself and onto a different existing table, with both
manifest layouts, verifying that rows and version histories remain
unchanged.

Validation:

- All four regression cases reproduced the reported internal error
before the fix.
- `cargo test --locked --quiet --profile ci --features remote -p lancedb
--lib test_clone -j 12`: 24 passed.
- `cargo fmt --all -- --check` and `git diff --check` passed.
- `cargo clippy --locked --quiet --profile ci --features remote -p
lancedb --tests --examples -j 12` passed.

Fixes #4387

<!-- lance-gatekeeper-fix:v1 agent=522c80785eeaaef146e8179c848b903e
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 06:16:31 -05:00
lancedb-gatefixer[bot] de19ecbf63 fix: reject empty column alterations and unknown Python keys (#4417) 
Fixes #4390

Lance accepts column alterations with every change field unset, so a
path-only Python call could commit a new local version without changing
the schema. The Python binding also silently discarded unknown keys such
as `nulable`.

Validate every alteration in the shared Rust `Table` API before
dispatching to either backend, and reject unknown dictionary keys in the
Python binding while preserving the legacy `name` alias. Invalid batches
leave local schema and version unchanged and send no remote request.

Validation:

- Reproduced both reported calls on the original extension: versions
advanced from 1 to 2 to 3 with an unchanged schema; all 13 new
invalid-input Python cases failed before the fix.
- Rebuilt the extension with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`; all 18 focused Python cases pass across
sync/async and local/mocked remote tables.
- `cargo test --quiet --features remote -p lancedb --lib alter_column`:
all eight tests pass, including both new regressions.
- `cargo clippy --quiet --features remote --tests --examples` passes
across the workspace.
- `cargo fmt --all`, `ruff check .`, and `ruff format .` pass.
- `typos` 1.26.0 passes across the repository; intentional misspellings
in regression fixtures use the existing line-specific spelling
exemptions.

<!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 06:15:15 -05:00
lancedb-gatefixer[bot] 57a952135b fix(python): support async search without a query (#4419) 
`AsyncTable.search()` and `search(None)` reached vector search without
assigning `vector_query`, raising `UnboundLocalError` on both local and
remote tables. Return a plain `AsyncQuery` before inferring the search
type, equivalent to `table.query()`, and update the overloads and
documentation to describe that return value.

Add six regression cases for omitted and explicit `None` queries on
local tables with and without vectors and on a mocked remote endpoint.
They verify scans, limits, filters, column projection, and the remote
request body.

Validation: all six regressions reproduced the original
`UnboundLocalError`; all 11 focused query tests pass with the fix,
including existing vector, full-text, hybrid, and sync scan coverage.
The uv development bootstrap, locked Ruff formatting/lint checks, and
focused Pyright return-type assertions also pass.

Fixes #4385

<!-- lance-gatekeeper-fix:v1 agent=bb9a7509f6dadde030d8113b172abc9a
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 06:04:29 -05:00
lancedb-gatefixer[bot] 6851133a7c fix(python): serialize deprecated v2 manifest option (#4420) 
Local synchronous `create_table(...,
enable_v2_manifest_paths=True/False)` failed with `TypeError` because
the deprecated argument was forwarded as a boolean in `storage_options`,
while the Rust binding expects string values.

Convert the argument to lowercase `"true"` or `"false"` before
forwarding it, preserving the deprecation warning. Add regression
coverage for both values with populated and schema-only tables, checking
stored data and the manifest mode after reopening.

Validation:
- All four regression cases reproduced the original `TypeError` before
the fix and pass afterward.
- `uv run --no-sync --extra tests --extra dev pytest
python/tests/test_db.py -q --tb=short -m 'not slow and not s3_test'`: 54
passed.
- Repository-wide `ruff format .` and `ruff check .` completed
successfully.

Fixes #4384

<!-- lance-gatekeeper-fix:v1 agent=9c14955b90d0c12d6a46605ec5ef32c0
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 06:02:55 -05:00
lancedb-gatefixer[bot] 8c2c39b3c4 fix(python): repair sync manifest path migration (#4422) 
Fixes #4383

`LanceTable.migrate_v2_manifest_paths()` raised `AttributeError` because
it called a nonexistent `AsyncTable.migrate_v2_manifest_paths()` method.
Delegate to the existing async method, `migrate_manifest_paths_v2()`,
while preserving the public sync API.

Add a local regression test for tables starting with either v1 or v2
manifest paths. It checks repeated migration, v2 manifest filenames on
disk, and preservation of data and version history after reopening the
table.

Validation:
- Both new regression cases failed with the reported `AttributeError`
before the fix.
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_migrate_v2_manifest_paths
python/tests/test_db.py::test_create_table_v2_manifest_paths_async -q` —
3 passed.
- Repository-wide `ruff check .` and `ruff format .` passed using Ruff
0.15.20 from `python/uv.lock`.
- `git diff --check` passed.

<!-- lance-gatekeeper-fix:v1 agent=bb500f0d1ddefeba3c7d7d8390a2fb74
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 05:57:56 -05:00
lancedb-gatefixer[bot] 7e979c1416 fix(node): preserve schemas for empty query results (#4423) 
Empty local TypeScript queries returned a zero-row Arrow table with no
columns because `toArrow()` built the table solely from record batches.
Local streams can yield no batches, while remote streams often supply an
empty batch carrying the schema.

Expose the executed stream's schema through the native iterator and
decode it into an empty Arrow table when no batches arrive. This
preserves projected and search columns, types, nullability, and metadata
without executing another query. Document the empty-result behavior.

Validation:

- Eleven regression cases failed with the original `toArrow()`
implementation; all twelve now pass, including remote schema-only and
empty-batch responses.
- All 1,075 tests in the query, remote, Arrow, and table suites pass.
- Native build, TypeScript compilation, Rust formatting and Clippy (`-D
warnings`), TypeScript lint/formatting, and documentation generation
pass.

Fixes #4386

<!-- lance-gatekeeper-fix:v1 agent=7d589da1f7db5c7383f9600490e111a9
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 05:56:50 -05:00
lancedb-gatefixer[bot] cc6b4aa4b4 fix(remote): preserve schema for empty merge insert sources (#4425) 
Remote `merge_insert` buffered its reader into a `Vec<RecordBatch>`,
losing the schema when the reader yielded no batches. Scanning that
vector failed before the request could reach the server.

Preserve the reader's schema in an empty record batch when buffering
produces no batches. The source remains replayable for retries, and
empty-source merges still reach the server so
`when_not_matched_by_source_delete` can apply deletes.

The regression tests decode the outgoing Arrow IPC and verify the source
schema, zero rows, merge options, and result statistics on an initial
request and a retry after HTTP 409. They cover readers with no batches
and readers with an empty batch, in insert and delete modes, including
an ID-only delete source.

Validation:
- Before the fix, both no-batch cases reproduced `Cannot scan an empty
Vec<RecordBatch>`; the empty-batch cases passed.
- `cargo test --quiet --profile ci --features remote -p lancedb --lib
remote::table::tests::test_merge_insert` — 9 passed.
- `cargo fmt --all` and `cargo fmt --all -- --check`.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`.

Fixes #4380

<!-- lance-gatekeeper-fix:v1 agent=a48c9fc83772479990d76124f3a2b3be
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 05:53:00 -05:00
lancedb-gatefixer[bot] 3d8c9e2ddc fix(python): close synchronous database connections (#4424) 
Calling `close()` on a synchronous remote connection returned an
unawaited coroutine and left the connection usable. Local synchronous
connections had no close or context-manager API.

Add synchronous `close()` and `is_open()` to the connection interface,
delegate them to the existing connection handle for local, remote, and
namespace connections, and provide shared context-manager cleanup.
Closing is idempotent, and subsequent database operations use the
existing `Connection is closed` error. Document the lifecycle with
examples.

Validation:

- Reproduced the reported coroutine warning and missing local lifecycle
API on the original code; all 15 new regression cases failed before the
fix.
- All 168 tests in `test_db.py`, `test_remote_db.py`, and
`test_namespace.py` passed through the uv-managed environment, including
local, in-memory, namespace, native-derived, and mocked remote lifecycle
coverage.
- All 6 synchronous connection doctests passed.
- Repository-wide Ruff 0.9.9 lint and formatting passed; `git diff
--check` passed.

Fixes #4382

<!-- lance-gatekeeper-fix:v1 agent=a743139af98198f1d4c8934c9cc1d0c9
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-02 22:19:24 -05:00
Will Jones b0c0d8ee92 perf(remote): size blob file handles from one descriptor take (#4349) 2026-10-02 19:59:24 -05:00
Dan TasseVincent KocClaude Fable 5.1
 		2734e10767 fix(node): update arrow to 21 (#4362) 
Thanks @vincentkoc for starting this! I was just looking into this
myself. I started with your PR and added:
- make arrow 21 the default
- widen the peer range to <22 so if they release 21.3 that'll be
included
- have to do skipLibCheck in tsconfig.json, because of flatbuffers 25
which needs TypeScript 5.7
- fix override in Symbol.hasInstance
- support new types in Arrow 20-21 (LargeList, Utf8View, BinaryView,
IntervalMonthDayNano)
- arrow-compat job in nodejs.yml that runs tests with arrow 19 and 20
(adds 2 sec)
- put the list of arrow types in one place
- update NODEJS_THIRD_PARTY_LICENSES.md
- regenerated typescript docs (sorry about reorderings, one time cost)
- reformat arrow.test.ts (sorry, almost all whitespace, thanks to biome.
git diff -w shows 110 lines edited)
- fixed Sliced Utf8View / BinaryView serialization error, by just
converting them to Utf8 and Binary (as our rust code would anyway)

---------

Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 15:56:46 -05:00
dependabot[bot] f72779ebb5 build(deps): bump the rust-minor-patch group across 1 directory with 7 updates (#4355) 
Bumps the rust-minor-patch group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.6.3`
| `1.6.4` |
| [object_store](https://github.com/apache/arrow-rs-object-store) |
`0.14.1` | `0.14.2` |
| [serde_with](https://github.com/jonasbb/serde_with) | `3.23.0` |
`3.24.0` |
| [napi](https://github.com/napi-rs/napi-rs) | `3.12.2` | `3.12.5` |
| [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.5` | `3.6.9`
|
| [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.2` | `2.5.0` |
| [arc-swap](https://github.com/vorner/arc-swap) | `1.9.1` | `1.9.2` |


Updates `aws-smithy-types` from 1.6.3 to 1.6.4
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/smithy-lang/smithy-rs/commits">compare
view</a></li>
</ul>
</details>
<br />

Updates `object_store` from 0.14.1 to 0.14.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/apache/arrow-rs-object-store/releases">object_store's
releases</a>.</em></p>
<blockquote>
<h2>object_store v0.14.2</h2>
<!-- raw HTML omitted -->
<h1>Changelog</h1>
<h2><a
href="https://github.com/apache/arrow-rs-object-store/tree/v0.14.2">v0.14.2</a>
(2026-09-11)</h2>
<p><a
href="https://github.com/apache/arrow-rs-object-store/compare/v0.14.1...v0.14.2">Full
Changelog</a></p>
<p><strong>Implemented enhancements:</strong></p>
<ul>
<li>Implement <code>Clone</code> for <code>MicrosoftAzure</code> <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/824">#824</a></li>
<li>Support URLs for jurisdiction-restricted buckets on R2 <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/814">#814</a></li>
<li>Lightweight DNS cache with shuffle to prevent DNS flooding <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/726">#726</a></li>
<li>Allow presigning object_store multipart uploads <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/271">#271</a></li>
<li>Allow specifying <code>Content-Type</code> when signing object_store
PUT requests <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/270">#270</a></li>
</ul>
<p><strong>Fixed bugs:</strong></p>
<ul>
<li>Incomplete ASF license header in <code>src/client/s3.rs</code>
(caused -1 vote on 0.14.2 RC1) <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/853">#853</a></li>
<li>WriteMultipart::finish should abort after part upload failure <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/818">#818</a></li>
<li>The 1.85 MSRV is inaccurate with all features <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/811">#811</a></li>
<li>When the backend service returns an HTTP 500, the object store
panics <a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/414">#414</a></li>
</ul>
<p><strong>Documentation updates:</strong></p>
<ul>
<li>Docs: point users uploading large objects at the multipart API <a
href="https://redirect.github.com/apache/arrow-rs-object-store/pull/839">#839</a>
(<a href="https://github.com/alamb">alamb</a>)</li>
<li>[object-store]: update release schedule <a
href="https://redirect.github.com/apache/arrow-rs-object-store/pull/834">#834</a>
(<a href="https://github.com/alamb">alamb</a>)</li>
<li>Add doc example for multipart upload to
<code>GoogleCloudStorage::create_multipart</code> <a
href="https://redirect.github.com/apache/arrow-rs-object-store/pull/803">#803</a>
(<a href="https://github.com/alamb">alamb</a>)</li>
<li>Add doc example for multipart upload to
<code>MicrosoftAzure::create_multipart</code> <a
href="https://redirect.github.com/apache/arrow-rs-object-store/pull/802">#802</a>
(<a href="https://github.com/alamb">alamb</a>)</li>
<li>Add doc example for multipart upload to AmazonS3::create_multipart
<a
href="https://redirect.github.com/apache/arrow-rs-object-store/pull/801">#801</a>
(<a href="https://github.com/alamb">alamb</a>)</li>
</ul>
<p><strong>Closed issues:</strong></p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/apache/arrow-rs-object-store/blob/main/CHANGELOG-old.md">object_store's
changelog</a>.</em></p>
<blockquote>
<!-- raw HTML omitted -->
<h1>Historical Changelog</h1>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/279572ea60f3a7a6e5237e066a6f4a35eee611e0"><code>279572e</code></a>
Update CHANGELOG for latest main (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/855">#855</a>)</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/4ff7d048073c07bcc1f1dca74436245b2ba20dda"><code>4ff7d04</code></a>
Restore missing first line of ASF license header in src/client/s3.rs (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/854">#854</a>)</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/25fa079c41fecb8011d5b8c3019f20e09c9b0c74"><code>25fa079</code></a>
Update version to 0.14.2 and add CHANGELOG (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/850">#850</a>)</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/c37db72b92b78608291c1b33feb79bb5bb6d2d1c"><code>c37db72</code></a>
feat: retry failed multipart part uploads (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/849">#849</a>)</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/135984fdf91f813c48c47b6288586d9b0e205164"><code>135984f</code></a>
Add doc hints pointing large-object uploads at the multipart API (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/839">#839</a>)</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/b07471e2bc341278f86e30cf80a850d56cbe2c67"><code>b07471e</code></a>
[object-store]: update release schedule (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/834">#834</a>)</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/978fb48b9ea6ebe3f8f4833f656e793cde79ab15"><code>978fb48</code></a>
Fix MSRV violation in <code>parse.rs</code> test, check
<code>--all-targets</code> in MSRV CI (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/837">#837</a>)</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/e2a49349622aa7ad5db12b107a557d957ac01241"><code>e2a4934</code></a>
Speed up MSRV CI check by installing cargo-msrv via
taiki-e/install-action (#...</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/e28807de7338abebf14ecf2ab959294cb3096d83"><code>e28807d</code></a>
feat: presigned URLs with extra query params and signed headers
(SignedUrlOpt...</li>
<li><a
href="https://github.com/apache/arrow-rs-object-store/commit/5c743637e97d2f3417b6fd54532bb1223c6547c1"><code>5c74363</code></a>
Restore UnwindSafe/RefUnwindSafe on ClientOptions by bounding
DnsResolver (<a
href="https://redirect.github.com/apache/arrow-rs-object-store/issues/836">#836</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/apache/arrow-rs-object-store/compare/v0.14.1...v0.14.2">compare
view</a></li>
</ul>
</details>
<br />

Updates `serde_with` from 3.23.0 to 3.24.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/jonasbb/serde_with/releases">serde_with's
releases</a>.</em></p>
<blockquote>
<h2>serde_with v3.24.0</h2>
<h3>Added</h3>
<ul>
<li>Implement <code>JsonSchemaAs</code> for
<code>NoneAsEmptyString</code>, so fields using it compile with the
<code>schemars_*</code> features (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1014">#1014</a>)</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Make <code>with_prefix!</code> and <code>with_suffix!</code>
available with the <code>alloc</code> feature, instead of only with
<code>std</code> (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1008">#1008</a>)</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>Handle potential panics in deserialization of chrono and time
DateTimes.
This was privately reported by <a
href="https://github.com/NotAFlightRisk"><code>@​NotAFlightRisk</code></a>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/jonasbb/serde_with/commit/35ac84c8faf34d6822991bbf2397056994c9208b"><code>35ac84c</code></a>
Bump version to 3.24 (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1016">#1016</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/7b8e561c709544b417b3c0f05f3c57a753e86417"><code>7b8e561</code></a>
Bump version to 3.24</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/6e576c8d5cc5f31efd26cf783d76eac6607d784e"><code>6e576c8</code></a>
Bump the github-actions group with 2 updates (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1012">#1012</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/7b1814b5cf8f2ce4ef7844105ab26760371af0cf"><code>7b1814b</code></a>
Add schemars support for NoneAsEmptyString (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1014">#1014</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/92eeedd1e5be0f176bd9a0532eb8ab1503396bb2"><code>92eeedd</code></a>
Handle potential panic during deserialization for large datetimes (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1015">#1015</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/3404642aae13a3dfcf4ea0f47749f759b36a85db"><code>3404642</code></a>
Handle potential panic during deserialization for large datetimes</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/c7073b98f150d60c22e1ccdb49f1c7151dd8595f"><code>c7073b9</code></a>
Add schemars support for NoneAsEmptyString</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/6b32290267fa18064894e2fa67bf4bed7fb4bd5f"><code>6b32290</code></a>
Run test with only the alloc feature (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1013">#1013</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/3361c69817d02b079b15f71d1281f573594cc948"><code>3361c69</code></a>
Run test with only the alloc feature</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/b43bb81cd8f02f06c2bf4bcd057c83e71442a515"><code>b43bb81</code></a>
Reduce required feature for with_{prefix,suffix} from std to alloc (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/1008">#1008</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/jonasbb/serde_with/compare/v3.23.0...v3.24.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi` from 3.12.2 to 3.12.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi's
releases</a>.</em></p>
<blockquote>
<h2>napi-v3.12.5</h2>
<h3>Fixed</h3>
<ul>
<li>fatal_exception noop in optimized builds (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3509">#3509</a>)</li>
</ul>
<h2>napi-v3.12.4</h2>
<h3>Security</h3>
<ul>
<li><code>Object::unwrap</code> and <code>Object::remove_wrapped</code>
(and the deprecated <code>Env::unwrap</code> /
<code>Env::drop_wrapped</code>) read a <code>TypeId</code> out of a
payload that is not a <code>TaggedObject</code>, giving a heap
out-of-bounds read and a JavaScript-forgeable out-of-bounds write;
payload provenance is now validated before dereferencing. See <a
href="https://github.com/napi-rs/napi-rs/security/advisories/GHSA-rhpj-pggq-896v">GHSA-rhpj-pggq-896v</a>,
fixed in <a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3506">#3506</a>.</li>
<li>The serde deserializer reached <code>unreachable!()</code> on
ordinary JavaScript values and aborted the process instead of returning
an error. See <a
href="https://github.com/napi-rs/napi-rs/security/advisories/GHSA-f334-75xc-qxv3">GHSA-f334-75xc-qxv3</a>,
fixed in <a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3507">#3507</a>.</li>
<li><code>ArrayBuffer::from_external</code>,
<code>TypedArraySlice::from_external</code> and
<code>Uint8ClampedSlice::from_external</code> (plus the compat-mode
<code>Env::create_arraybuffer_with_borrowed_data</code>) built their
slice from freed memory on runtimes without external buffers. See <a
href="https://github.com/napi-rs/napi-rs/security/advisories/GHSA-32mm-r9wp-hrvc">GHSA-32mm-r9wp-hrvc</a>,
fixed in <a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3503">#3503</a>.</li>
</ul>
<h3>Fixed</h3>
<ul>
<li><em>(napi)</em> return errors from the serde deserializer for
unexpected JS value shapes (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3507">#3507</a>)</li>
<li><em>(napi)</em> validate wrapped payload provenance in
Object::unwrap/remove_wrapped (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3506">#3506</a>)</li>
<li><em>(napi)</em> point from_external slices at the engine-owned copy
after finalize reclaims the source (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3503">#3503</a>)</li>
</ul>
<h2>napi-v3.12.3</h2>
<h3>Fixed</h3>
<ul>
<li>CurrentThread waker-stack deadlocks and threadless-wasm Buffer
detachment (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3489">#3489</a>)</li>
<li><em>(napi)</em> catch panics in closure trampolines instead of
aborting (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3473">#3473</a>)</li>
</ul>
<h3>Other</h3>
<ul>
<li><em>(napi)</em> keep anyhow error cause when converting (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3477">#3477</a>)</li>
<li><em>(napi)</em> outline threadsafe callback dispatch (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3465">#3465</a>)</li>
<li><em>(napi)</em> outline class instance construction (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3464">#3464</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/43100baf28a3e5709641e35f892be4da5d62dcb2"><code>43100ba</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3508">#3508</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/bb7b75a7d53b0757f99813c7fef0537cef044adf"><code>bb7b75a</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/89d7a0b3636b4454265363dab58df471030b4f85"><code>89d7a0b</code></a>
chore(deps): update dependency oxfmt to ^0.68.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3519">#3519</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/55545910c2487da38453e6171330e8fc9133213c"><code>5554591</code></a>
fix(cli): rate-limit failed incarnation probes and let reconciliation
tests i...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/6d10d826c34f5d0009eb1ab47efaa4a2e34f4555"><code>6d10d82</code></a>
feat(cli): make the deferred workerd loader usable by host-driven addons
(<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3520">#3520</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/a1bcecbd1bef2f51f1047f5800d5a312eb2eba3b"><code>a1bcecb</code></a>
feat(cli): add <code>napi.wasm.threadlessInitialMemory</code> for
threadless WASI loader...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/0999babc1e37ce869b79f59db14f9eb12f6647df"><code>0999bab</code></a>
feat(cli): export <code>__napiBindingTarget</code> from generated
binding loaders (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3515">#3515</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/30ea504c3d65e1cb178d3124068257d8df8f3835"><code>30ea504</code></a>
fix(cli): bound symlink-ancestor reconciliation guards to the anchor's
parent...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ae8f34a8dc5f629b53d136442e8f614f2d3c17b6"><code>ae8f34a</code></a>
fix(cli): run the wasm env cleanup barrier on a raw emnapi
`Context.destroy()...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ccd314bbb5201182311b74dddd61f60b323318f4"><code>ccd314b</code></a>
fix(cli): never wedge reconciliation locks when host identity probes
fail (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3">#3</a>...</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-v3.12.2...napi-v3.12.5">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-derive` from 3.6.5 to 3.6.9
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-derive's
releases</a>.</em></p>
<blockquote>
<h2>napi-derive-v3.6.9</h2>
<h3>Other</h3>
<ul>
<li>update Cargo.toml dependencies</li>
</ul>
<h2>napi-derive-v3.6.8</h2>
<h3>Fixed</h3>
<ul>
<li><em>(napi)</em> guard AsyncTask completion against env teardown (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3536">#3536</a>)</li>
</ul>
<h2>napi-derive-v3.6.7</h2>
<h3>Other</h3>
<ul>
<li>update Cargo.toml dependencies</li>
</ul>
<h2>napi-derive-v3.6.6</h2>
<h3>Other</h3>
<ul>
<li>update Cargo.toml dependencies</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2763e12efc855748485129952a6ccb97ac991c06"><code>2763e12</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3538">#3538</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/5ae5cf604809ed6b9868c8e67b201bc9eb68d9bf"><code>5ae5cf6</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/0da138ec836906da5bc3a996521299e1d284bdda"><code>0da138e</code></a>
feat(cli): self-sign OpenHarmony artifacts with --ohos-sign (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3539">#3539</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2499a6311148492bc01ae316621b514a9345e4e7"><code>2499a63</code></a>
feat(async-runtime): offer the MultiThread flavor on
wasm32-wasip1-threads (#...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/886bd071972e09010ee901d1a1bb669a26d3e0dc"><code>886bd07</code></a>
chore(deps): update dependency oxc-parser to ^0.151.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3542">#3542</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/edb0a295a14371b2f943c2a89bacf385eb61a038"><code>edb0a29</code></a>
fix(napi): validate native payload provenance for External and instance
data ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/38162bb0eb324ae24b402982bad3d4ef3f24c90a"><code>38162bb</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3529">#3529</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ec3d8ead66c18e022d2e3fb6832e52e5204ffcc3"><code>ec3d8ea</code></a>
fix(napi): guard AsyncTask completion against env teardown (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3536">#3536</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/96ed2672820b5468b6a451080fc27e6eef594d3d"><code>96ed267</code></a>
chore(deps): update release-plz/action action to v0.5.139 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3537">#3537</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b2c9f3b610cb167d2c05b8d6c748c120926ffd73"><code>b2c9f3b</code></a>
chore(deps): update release-plz/action action to v0.5.138 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3533">#3533</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.5...napi-derive-v3.6.9">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-build` from 2.4.2 to 2.5.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-build's
releases</a>.</em></p>
<blockquote>
<h2>napi-build-v2.5.0</h2>
<h3>Added</h3>
<ul>
<li><em>(async-runtime)</em> offer the MultiThread flavor on
wasm32-wasip1-threads (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3541">#3541</a>)</li>
</ul>
<h2>napi-build-v2.4.4</h2>
<h3>Fixed</h3>
<ul>
<li><em>(napi)</em> guard AsyncTask completion against env teardown (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3536">#3536</a>)</li>
</ul>
<h2>napi-build-v2.4.3</h2>
<h3>Fixed</h3>
<ul>
<li><em>(cli)</em> drain outstanding async work before disposing a WASI
binding (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3528">#3528</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2763e12efc855748485129952a6ccb97ac991c06"><code>2763e12</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3538">#3538</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/5ae5cf604809ed6b9868c8e67b201bc9eb68d9bf"><code>5ae5cf6</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/0da138ec836906da5bc3a996521299e1d284bdda"><code>0da138e</code></a>
feat(cli): self-sign OpenHarmony artifacts with --ohos-sign (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3539">#3539</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2499a6311148492bc01ae316621b514a9345e4e7"><code>2499a63</code></a>
feat(async-runtime): offer the MultiThread flavor on
wasm32-wasip1-threads (#...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/886bd071972e09010ee901d1a1bb669a26d3e0dc"><code>886bd07</code></a>
chore(deps): update dependency oxc-parser to ^0.151.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3542">#3542</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/edb0a295a14371b2f943c2a89bacf385eb61a038"><code>edb0a29</code></a>
fix(napi): validate native payload provenance for External and instance
data ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/38162bb0eb324ae24b402982bad3d4ef3f24c90a"><code>38162bb</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3529">#3529</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ec3d8ead66c18e022d2e3fb6832e52e5204ffcc3"><code>ec3d8ea</code></a>
fix(napi): guard AsyncTask completion against env teardown (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3536">#3536</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/96ed2672820b5468b6a451080fc27e6eef594d3d"><code>96ed267</code></a>
chore(deps): update release-plz/action action to v0.5.139 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3537">#3537</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b2c9f3b610cb167d2c05b8d6c748c120926ffd73"><code>b2c9f3b</code></a>
chore(deps): update release-plz/action action to v0.5.138 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3533">#3533</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.2...napi-build-v2.5.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `arc-swap` from 1.9.1 to 1.9.2
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/vorner/arc-swap/blob/master/CHANGELOG.md">arc-swap's
changelog</a>.</em></p>
<blockquote>
<h1>1.9.2</h1>
<ul>
<li>Document RefCnt must not panic (<a
href="https://redirect.github.com/vorner/arc-swap/issues/208">#208</a>).</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/vorner/arc-swap/commits">compare view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-10-02 07:49:06 -05:00
XY Zhan 723a2394e5 feat: maintain every index a table has, including ones created later (#4342) 
**The maintained set is an intent, not a snapshot.** An unnamed set is
stored as
the intent to maintain every index, which lance re-resolves whenever it
builds a
MemTable, so an index created after the spec installs is maintained too.
A named
set is checked here, where the caller can be told which names exist; an
index of
a kind the MemWAL cannot mirror fails a named set and is skipped by an
unnamed
one.

**A newly created index reaches a writer that is already open.** lancedb
caches a
`ShardWriter` per shard and reuses it, so `create_index` installs the
current set
on the cached writer. A failure there is reported rather than
propagated: the
index is committed durably by then, and the writer this process holds
may be
fenced or poisoned, which is no reason to fail creating an index.

**A table carries one spec.** `set_lsm_write_spec` fails while one is
installed
and names `unset_lsm_write_spec` in the error, because the generations
already
written were homed under that spec. Accepting a repeat that changed only
the
maintained set would hand a caller who also passed a different sharding
a success
that changed nothing they asked for.

**The read path asks what is actually carried.** An unnamed set leaves
the stored
list empty, so the list alone cannot decide whether an index is
maintained. The
read asks the spec *and* every resident MemTable whether it carries the
index, so
a MemTable built before the index joined the set refuses the read rather
than
answering without its rows. Both spec routes report an unnamed set as
`null`
rather than `[]`: maintain whatever the table gains, and maintain
nothing, are
different settings.

## Known gap

The guard above covers resident MemTables. Once a writer flushes and
closes none
is resident, so a retained generation written before the index existed
still gets
an indexed read that omits its rows.

This window is **wider than on main**: a set resolved at install time
excluded an
index created later, so the read was refused; an intent includes it, and
the
refusal only holds while a MemTable is resident. Closing it needs the
generation's
own coverage, which this layer cannot see — its snapshot carries a
generation and
a path — so it belongs in lance's scanner or flush path.

## Testing

`cargo nextest run -p lancedb --lib --features remote` — **1479 passed,
3
skipped**. `cargo fmt --all --check`, and `cargo clippy --features
remote --lib
--tests --examples -- -D warnings`, both clean.

Covered: the round trip of an unnamed set, a set over an installed spec
refused,
a spec replaced through unset, a read refused while a resident MemTable
predates
the index, and an index created mid-write reaching the open writer.

Two Python binding tests move with the semantics: an unnamed set now
reads back
as `None` rather than the list it resolves to, and the refusal is
matched on
`already set`. Neither was run locally — no Python build here — so CI is
the
check on them.
 2026-10-02 00:11:44 -04:00
LanceDB RobotDaniel RammerClaude Opus 5.5
 		857e455361 chore: update lance dependency to v13.1.0-beta.1 (#4358) 
Updates the Rust workspace and Java Lance dependencies to
[v13.1.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.1.0-beta.1)
and refreshes Cargo.lock. Removes a redundant `pub(crate)` declaration
in the private Node.js blob module to satisfy the required Clippy check.

Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `pnpm build`, and `pnpm test
__test__/blob.test.ts --runInBand` (26 tests passed) in `nodejs`.

---------

Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-01 09:13:08 -05:00
Lance Release 4730f9036b Bump version: 0.40.0-beta.11 → 0.40.0-beta.12 2026-09-30 13:32:13 +00:00
Jonathan HsiehandClaude Opus 5.5 82d39485a4 feat(functions): address Functions by namespace path (#4341) 
Adds namespace addressing to Functions, matching what Secrets already
have (#4150).

## API

`create_function` / `create_function_async`, `get_function`,
`list_functions`, `drop_function` / `drop_function_async` take
`namespace_path` — keyword-only in Python (sync, async and remote
connections) and a trailing `&[String]` in Rust — defaulting to the
root. The namespace is addressed in the route identifier
(`/v1/function/analytics$features$embed/describe`,
`/v1/namespace/analytics$features/function/list`); no body carries it.

`FunctionVersion` and `FunctionVersionRef` name their Function **by
parts** — a short `name` plus `namespace_path` — the same shape as
`SecretReference`, so a returned version reopens from its own fields:

```python
fv = db.create_function(embed, namespace_path=["analytics", "features"])
fv.name, fv.namespace_path          # "embed", ("analytics", "features")
db.get_function(fv.name, version=fv.version, namespace_path=list(fv.namespace_path))
```

An empty path is omitted from the wire, so root documents keep their
shape. `namespace_path` is passed straight to the binding, so a bare
string (`namespace_path="prod"`) is a `TypeError` rather than four
one-character segments; segments are checked for addressability before a
request is built.

The binding and application allow-lists now share one field list,
`FUNCTION_VERSION_REF_FIELDS`, which admits `namespace_path` — without
it a namespaced binding or application read as coming from a newer
contract.

## Service side

The Sophon half serves and persists FunctionVersion / FunctionVersionRef
documents in this split form, keeping the joined id as its internal
catalog key. It lands in sophon with the submodule bump.

## Tests

- Rust: every Function verb addresses a namespace in the path; an
unaddressable segment never reaches the transport; `FunctionVersion`
round-trips a namespace and `reference()` carries it; a namespaced
binding and application are known shapes.
- Python: sync and async verbs address the namespace; a created Function
reopens from its own `name`/`namespace_path`; a bare-string path is
refused before any request.

Draft: compiles, `cargo fmt --check` and clippy `-D warnings` clean;
lancedb lib Function tests (116) and `first_class_function_slice2` pass
on this base.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 14:07:14 -07:00
Bilal Atique 0a1c1a04c7 feat(node): accept an AbortSignal to cancel blob reads (#4350) 
Closes #4288.

Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange`
and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an
optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When
the signal aborts, the call rejects with `signal.reason` (an
`AbortError` unless the caller gave a reason) and the native future is
dropped, so the read stops instead of running to completion. A signal
that is already aborted rejects before any read starts.

How it works:

- On the Rust side, these five methods are now sync `#[napi]` methods
that return `env.spawn_future(...)`. The future is wrapped in
`futures::future::abortable`, and the abort handle is registered with
napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so
it can't be an argument of an `async fn`, which is why the methods
changed shape. The TypeScript signatures are unchanged apart from the
new optional argument.
- napi-rs installs its own `onabort` on the signal it receives, which
would overwrite a handler the caller set. So the TypeScript wrapper
passes a private `AbortController` signal to the native call for each
request, forwards the caller's abort to it with `addEventListener(..., {
once: true })`, and removes the listener when the call settles.

Tests in `table.test.ts`:

- an already aborted signal rejects on all five methods, and the handle
still reads afterwards
- a custom abort reason is what the call rejects with
- a read with a signal that never aborts returns normally and leaves no
`abort` listener on the signal
- aborting a 32 MiB `readRange` right after it starts rejects with
`AbortError`, and the same handle reads again afterwards

The Node docs under `docs/src/js` are regenerated.

I didn't add a test for where the `read()` cursor ends up after a
mid-flight abort on a local table. The remote `BlobFile` documents that
a cancelled read leaves the cursor unchanged, but I haven't confirmed
the same for the native Lance handle, so the docs don't claim it.
 2026-09-29 14:01:17 -05:00
lancedb-gatefixer[bot] ad79112b8e fix(node): support LargeUtf8 and LargeBinary Arrow columns (#4287) 
## Cause

The Arrow type sanitizer handled Utf8 and Binary but omitted LargeUtf8
and LargeBinary, so Arrow tables from another Arrow installation failed
schema sanitization before reaching LanceDB.

## Fix

Reconstruct both large Arrow types during sanitization. Regression tests
cover createTable and add with Arrow 15–18 and adding a LargeBinary
Arrow column to a blob table.

## Validation

- Confirmed the new table test failed before the fix with `Unrecognized
type id in schema: 20` on all four Arrow versions.
- The table, sanitizer, and Arrow Jest suites passed: 604 tests.
- `pnpm lint`, `pnpm build`, and `pnpm run docs` passed.

Fixes #4266

<!-- lance-gatekeeper-fix:v1 agent=1090de3c0ddeca587956632b79f0193c
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-09-29 12:25:12 -05:00
lancedb-gatefixer[bot] 85b0eba176 fix(python): support typed embedding vector fields (#3826) 
## Summary

- infer fixed-size float32 Arrow vectors from `VectorField` metadata
when a model uses the standard `list[float]` annotation
- document the mypy-compatible embedding-field declaration while
preserving the existing `Vector(dim)` runtime API
- add regression coverage for schema conversion and model construction

## Root cause

`Vector(dim)` is a runtime constrained-type factory, but Python's static
typing rules do not permit call expressions in annotation positions.
Mypy's suggested `Vector[...]` spelling also cannot represent a
dimension computed at runtime and fails because `Vector` is a function.
`VectorField` already carries the embedding function and its dimension,
but schema conversion previously ignored that metadata for a statically
valid `list[float]` annotation.

The runtime factory remains available for dimension-constrained Pydantic
validation. Embedding-backed models that require mypy compatibility can
now use `list[float] = embedding.VectorField()`; LanceDB derives the
same nullable fixed-size float32 Arrow type from the field metadata.

## Validation

- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py -q` (23 passed)
- focused mypy check reproducing the original error and accepting the
new `LanceModel` declaration
- isolated schema checks with Pydantic 1.10.24 and 2.11.7
- `ruff format .`
- `ruff check .`

Fixes #1982

<!-- lance-gatekeeper-fix:v1 agent=877facbc659cdc9424bd367e12426366
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-09-29 12:00:52 -05:00
Bilal Atique 4e3ec7cc27 feat(node): expose seek, tell, bounded read, readRanges and close on BlobFile (#4348) 
Closes #4300.

The TypeScript `BlobFile` only had `size()`, `read()` and `readRange()`,
while the Rust `BlobFile` behind it also has `seek`, `tell`,
`read_up_to`, `read_ranges` and `close`. This exposes those on the Node
binding:

- `read(maxBytes?)` reads at most `maxBytes` from the cursor (Rust
`read_up_to`). Without the argument it reads to the end, as before.
- `readRanges([{ start, end }, ...])` reads several half-open ranges and
returns one buffer per range, in order. Like `readRange`, it does not
move the cursor.
- `seek(position)` and `tell()` move and report the cursor.
- `close()` releases the handle, and `isClosed()` reports it. Every
read, `seek` and `tell` after `close()` fails with "blob file is already
closed". A second `close()` does nothing. The binding keeps its own
`closed` flag, the same way the Python `BlobFile` does.

Offsets and lengths are `bigint`, like `readRange` and `size()`.
`BlobRange` is exported as a type.

The new tests in `table.test.ts` cover chunked reads with
`read(maxBytes)`, `seek` and `tell`, a negative seek position,
`readRanges` order and cursor, a `readRanges` end past the blob size,
and reads after `close()`. The Node docs under `docs/src/js` are
regenerated with `pnpm run docs`.

I left the `ReadableStream`/async iterator idea from the issue out of
this PR. The AbortSignal work in #4288 touches the same methods, so it
can follow on top of this.
 2026-09-29 11:32:30 -05:00
Xuanwo ea223c4ec4 chore: update lance dependency to v13.0.0-beta.18 (#4339) 
Updates the Rust workspace dependencies, Cargo lockfile, and Java
`lance-core` to [Lance
v13.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.18).
It includes lance-format/lance#9574, which bounds the memory an IVF
partition split uses when it re-reads raw vectors during
`optimize_indices`, and lance-format/lance#9582, which keeps the mTLS
REST namespace client on rustls.

Lance's new combined-fields FTS query variant (introduced by
v13.0.0-beta.17) is handled the same way as in #4336: the remote
capability check and Rust/Node bindings accept it, with a regression
test for the remote request payload, and the Python conversion reports
that this query type is not yet supported. This PR supersedes #4336.
 2026-09-29 23:19:59 +08:00
Bilal Atique 166c114710 fix(rust): list indices when an FTS index's files are missing (#4330) 
Fixes #4250.

`list_indices()` reads each FTS index's files to fill `index_details`.
When those files were missing, the error failed the whole call, so the
caller could not list any index, including the damaged one it needs to
rebuild. Scalar indexes were not affected because their listing reads
only the manifest.

Now a failed read logs a warning and leaves that index's `index_details`
as `None`. The other fields still come from the manifest, and
`tokenize()` already returns a clear error when an FTS index has no
details.

The new test `test_list_indices_with_missing_fts_index_files` creates an
FTS index and a BTree index, deletes the FTS index directory, and
reopens the table. It then checks that both indexes are still listed.
Without the change, it fails with the same `Not found` error as the
issue.
 2026-09-29 10:11:34 -05:00
lancedb-gatefixer[bot] 629f39d4a5 fix(python): preserve mixed blob values before Arrow conversion (#4304) 
## Cause
Python list and pandas rows were converted to Arrow before the target
blob v2 schema was applied. In a mixed bytes and URI batch, Arrow
inferred a binary column and encoded the URI as inline bytes; dict and
`lance.Blob` values could fail conversion.

## Fix
Build blob v2 columns from the original Python values using the target
field's storage type before converting the rest of each row. This
preserves inline bytes, external URIs, descriptor dictionaries, blob
objects, and nulls for `add`, `merge_insert`, and `create_table`.

## Validation
- `python/tests/test_blob.py`: 76 passed, 1 skipped.
- Python project Ruff check passed.
- Repository-wide Ruff check still reports 29 existing findings in `ci/`
files outside this change.

Fixes #4271

<!-- lance-gatekeeper-fix:v1 agent=54c95f06f11c88187b07a430aac6be2f
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-09-29 08:38:00 -05:00
lancedb-gatefixer[bot] 51ea02ea8c fix: pin remote BlobFile reads to the opened version (#4295) 
## Cause

On latest-tracking remote tables, `fetch_blob_files` probed a concrete
table version but retained `None` on each handle. Later range reads
resolved the latest version again, so an update or delete could make an
opened handle fail.

## Fix

For nonempty blobs, capture `x-lancedb-version` from the 206 size probe
and include it on every later range request. Reject a nonempty probe
without a valid version. Accept an empty-blob 416 probe with or without
that header, because an empty handle makes no later data range request.
The Python remote mock now covers both response shapes.

## Limitation

A table dropped and recreated with the same name can reuse version
numbers and row addresses. Version pinning does not identify the table
itself, so an old handle can still return data from the recreated table.
That identity case needs a separate fix. Callers that replace a table
should discard its old blob handles.

## Validation

- `cargo test --quiet --features remote -p lancedb --lib
remote::table::blobs::tests` (24 passed)
- Rebuilt the Python extension; `uv run --extra tests pytest
python/tests/test_remote_db.py -q -k remote_blob` (6 passed)
- `cargo clippy --quiet --features remote -p lancedb --lib --tests
--examples`
- `cargo fmt --all -- --check`, `ruff check .`, and `ruff format .`

Fixes #4276

<!-- lance-gatekeeper-fix:v1 agent=e309bdf75b1a87f6492a6dc3ee1bfd81
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-09-29 08:35:19 -05:00
Yang Cen b8af687286 feat(sql): expose composable native vector relations (#4340) 
## Summary

Native vector dedup currently materializes through the native MV refresh
protocol. Add composable DataFusion pair and retained-row relations so a
host can use its existing SQL job and MV writer instead.

`VectorQueryTable` binds a retained snapshot, selects all/first/random
nonempty IVF scopes before dispatch, and exposes `DuplicatePairsExec` as
its physical child. Pair queries can include segment, partition and
snapshot columns and fetch a scalar application key for each endpoint
without reading source vectors for scoring. The default primitive schema
remains unchanged.

`VectorDedupExec` consumes the pair stream, reuses the spillable greedy
direct-representative selection, and scans retained original rows from
that snapshot. A–B–C without A–C retains A and C. This query path has no
durable partition receipts or writes; publication and failure belong to
the enclosing SQL sink. Existing native MV APIs remain available.

`BaseTableAdapter::vector_query_snapshot(None)` captures latest without
changing a shared table handle's pin; `Some(version)` selects a retained
version. A host can retain unbound SQL and resolve latest again for each
refresh. Single-segment random sampling matches Sophon's `ivf_partition`
SplitMix64 policy; multi-segment sampling ranks canonical nonempty
scopes.

Host integration and SQL syntax:
https://github.com/lancedb/sophon/pull/8135. The host authorizes
sources, binds named arguments, distributes the pair child, and uses its
existing SQL Job Registry and atomic MV replacement.

## Validation

All manual validation ran on an AWS r8i.8xlarge VM, not the developer
laptop.

- Focused Rust integration suite: 2 passed, including multiple
segments/partitions, empty results, threshold boundaries, 8,193 rows per
segment, deterministic sampling, mapped application IDs, and
native-oracle pair equality.
- Additional compacted stable/nonstable row-ID and blob test: 1 passed,
including application-key mapping, null vectors, snapshot deletions and
explicit/latest version capture.
- Rust formatting and focused Clippy with warnings denied passed.
- Dependent Sophon SQL-job E2E passes with two worker PEs: pairs,
sampling, direct representatives, latest/fixed snapshot refreshes, and
blob payloads. Killing a SQL worker after partial staging preserves the
old MV; full rerun commits the complete replacement. Performance
measurement is in progress in the dependent PR; no performance claim is
made here.
 2026-09-29 13:24:21 +08:00
lancedb-gatefixer[bot] 8e3606fbe3 fix: handle remote blob fetch request caps (#4309) 
## Cause

Remote `fetch_blobs` sent every row ID in one Cloud request, exceeding
the server's 1024-row or 64 MiB byte cap. A single blob over the byte
cap could not be read through this method.

## Fix

- Send at most 1024 row IDs per request and split batches further when
the server reports the blob-byte cap, preserving input order and nulls.
- Resolve one exact dataset version before multi-request reads. If an
initial single request hits the byte cap, resolve the version before any
successful split or Range read; use that version for every resulting
request.
- Read an individual oversized blob through the existing Range route.
- Document the per-request caps and transparent handling in the Rust,
Python, and TypeScript APIs and generated JS docs.

## Validation

- Focused Rust `fetch_blobs` tests: 26 passed, including row-count,
byte-cap, Range fallback, and exact-version assertions.
- Rust clippy and formatting passed.
- Focused Python blob tests: 13 passed; Ruff lint and formatting passed.
- Node build, lint, docs generation, and focused `fetchBlobs` test
passed.

Fixes #4275

<!-- lance-gatekeeper-fix:v1 agent=6ae94d49bf1d4cae057bba72584e7b90
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-09-28 20:31:36 -05:00