mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-25 21:22:29 +00:00
00ae2cdf7a426da8b965e0551a9a6e3203f65dca
3316
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Gatefixer
|
00ae2cdf7a | docs: add Rust compaction tuning example | ||
|
Gatefixer
|
10541cc2d8 | Merge main into gatekeeper/fix-2325-1 | ||
 Jack Ye
|
dcfef7c5ab |
feat(python): open lazy blob files from SQL descriptors (#4668)
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
|
||
|
Jack Ye
|
9af7757053 |
fix: execute remote materialized view creation through SQL (#4665)
Remote materialized-view creation currently expects a REST response containing a registry job ID, while refresh already submits SQL. Route both through the SQL query backend and preserve Python's synchronous and asynchronous return types. ### Query to Job mapping `SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through the same backend as `execute_query_async()`. Once submission succeeds, it retains the returned `Query` and starts one client-side task that consumes its result reader to completion. `Job::from_query_task()` wraps that task, its cached outcome, a single ID and a cancellation callback. Python exposes this as `Job[T]` or `AsyncJob[T]`. The existing `job_type` property maps SQL-backed handles to the new `"query"` kind. The shared `id` contains that query's UUID string, so Python callers can inspect it with `db.describe_query(job.id)` on the submitting connection. No separate `query_id`, backend enum or `type()` API is added. | Handle | `job_type` | `id` | Available lookup | | --- | --- | --- | --- | | SQL-backed | `"query"`, known at submission | Connection-scoped query UUID string | `describe_query(job.id)` | | Registry-backed | Existing server-reported kind, after `status()` or `refresh()` | Registry job ID string | Registry job APIs, descriptions and events | | Local | `None` | `None` | The current in-process handle | | Job concept | SQL-backed behavior | | --- | --- | | Submission | Await initial SQL submission before returning the handle. Submission errors propagate immediately. | | `status()` | Tracks the adapter task: running during result consumption/finalization, finished after both succeed, failed if either fails, or cancelled after cancellation. | | `wait()` | Returns the operation's typed terminal result. One task consumes the query reader once; repeated waits reuse its cached outcome without resubmitting SQL or reopening the reader. | | `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported cancellation leaves the task running; an accepted request stops the task. Cancelling a finished operation is a no-op. | | Registry details | Query-backed handles expose no registry description, event history, pause/resume or reattachment after a client restart. `refresh()` updates cached lifecycle state. | The Job tracks command completion rather than exposing SQL batches. A wait timeout does not cancel the query. Query IDs remain scoped to the submitting connection and cannot be used with registry APIs or as durable resume tokens. Python `describe_query()` accepts both UUID objects and the string `id` exposed by query-backed jobs. Rust, Python and TypeScript expose the same classification through their existing job-type accessors. ### Materialized-view operations - **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace and optional auto-refresh settings. After command completion, resolve the job to `None`. Synchronous creation waits on the job, then opens and returns the materialized view. - **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same adapter. After command completion, read the refreshed table's version and row count to construct the existing typed refresh result. Synchronous refresh waits on the same path. Remote creation requires the SQL service. Execution placement remains a server decision. REST endpoints are unchanged. The job guide documents the ID/job-type contract and query-backed limitations. Regression coverage exercises SQL submission, namespaces, refresh options, job-type/ID preservation through typed result mapping, repeated waits, failures and cancellation rejection. No local builds or tests were run. |
||
|
Jack Ye
|
24c2669a50 |
feat(sql): follow adaptive streaming and polling responses (#4664)
SQL execution now starts with one Flight call. The server can stream
rows immediately or return a polling continuation, which the SDK follows
through the existing query handle and reader APIs. The same behavior
applies with and without parameters.
Both initial transports share direct-stream lifecycle handling. Handoffs
are explicitly negotiated; existing servers can continue returning
ordinary streams. Add coverage for streaming, deferred results,
cancellation, malformed decisions, and legacy responses.
```mermaid
flowchart TD
A[execute_query or execute_query_async] --> B{Parameters supplied?}
B -->|No| C[DoGet: SQL command ticket]
B -->|Yes| D[DoExchange: SQL and Arrow parameter row]
C --> E[Server chooses execution path]
D --> E
E -->|Stream| F[Initial response: streaming decision and Arrow batches]
E -->|Defer| G[Initial response: PollInfo continuation]
G --> H[SDK follows PollFlightInfo]
H -->|Still running| H
H -->|Result endpoints available| I[SDK fetches tickets with DoGet]
F --> J[Query reader yields Arrow batches]
I --> J
```
`execute_query` opens the reader; `execute_query_async` returns its
query handle. The caller does not select the transport or execution
path.
|
||
|
Jack Ye
|
6b6778e684 |
feat: infer lazy blob reader Function inputs (#4663)
Infer Blob V2 input access from the Python Function signature. `video:
BlobReader` requests a lazy, seekable reader; `video: bytes` requests
the complete payload. Users do not need a separate `blob_inputs`
registration option.
## Python experience
For a video table joined to a table of frame timestamps, the Function
can pass its reader directly to a media decoder:
```python
import io
import av
from lancedb import BlobReader, udf
@udf(pip=["av", "Pillow"])
def extract_frame(video: BlobReader, timestamp_s: float) -> bytes:
with av.open(video) as container:
stream = container.streams.video[0]
container.seek(int(timestamp_s / stream.time_base), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time >= timestamp_s:
output = io.BytesIO()
frame.to_image().save(output, format="JPEG")
return output.getvalue()
raise ValueError("No frame at or after the requested timestamp")
```
After registering `extract_frame`, the query keeps the usual Function
call:
```sql
SELECT f.video_id, f.timestamp_s,
extract_frame(v.video, f.timestamp_s) AS image
FROM videos AS v
JOIN frames AS f ON v.id = f.video_id
```
The returned JPEG can feed a subsequent image-model step. The reader
provides `read`, `readinto`, `seek`, `tell`, `size()`, and
`read_range(offset, length)`; range reads leave the sequential cursor
unchanged. Each row invocation owns its reader, which closes after the
call. Decoder seeks can request the parts of the MP4 needed to locate
and decode the requested frame; the amount read depends on the file
layout and codec.
Functions that need the whole value can keep a bytes signature:
```python
@udf
def video_length(video: bytes) -> int:
return len(video)
```
## Execution contract
- Signature inference maps `BlobReader` to a Blob V2 field and records
`blob_access="reader"`. `bytes` keeps its binary Arrow type and records
`blob_access="bytes"`; it can also bind to a Blob V2 source. Explicit
schemas support the same choice, including a Blob V2 field annotated as
`bytes`.
- Reader execution carries a compact blob reference into the Function
worker. Payload reads happen when Python calls the reader, with requests
split into at most 1 MiB ranges. The full video does not need to enter
the Arrow input batch just to invoke the Function.
- Bytes execution materializes the full source before the Function runs
and remains subject to eager-input memory admission. Calling
`reader.read()` without a size also materializes the remaining contents
in Python; lazy access depends on how the Function consumes it.
- Input access metadata survives Python/Rust Function serialization and
registration. Packaged Functions import the worker-provided reader
implementation. This PR defines the SDK contract; execution requires
corresponding server support.
- This changes blob inputs. The example still returns ordinary image
bytes; it does not add streamed outputs or run an image model.
Adds coverage for signature inference, explicit schemas, nullable
readers, signature round trips, seeking, bounded range reads, buffering,
and invalid responses.
|
||
  Shiduo LiandXuanwo
|
0ea932dafd |
fix(rust): compile jobs without remote feature (#4103)
## What Move typed job decode error construction behind a feature-aware helper so builds without the `remote` feature do not reference `Error::Http`. Local decode failures continue to surface as `Error::Runtime`, while remote builds retain HTTP errors with request IDs. Add a regression test for local typed-result decode failures. Closes #4096. ## Testing - `cargo fmt --all -- --check` - `cargo check -p lancedb --no-default-features` - `cargo test -p lancedb --no-default-features job::tests::local_typed_result_decode_errors_are_runtime_errors` Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Yeluru NitishandXuanwo
|
2c9f2e75a0 |
fix: keep FTS indexes in list_indices when their files are missing (#4251)
Fixes #4250 `list_indices()` loads each FTS index's segment params from the index files to fill in `index_details`. That call used `?`, so a single FTS index with missing files made the whole listing fail. A BTree or Bitmap index with missing files still lists fine, because it never takes that path. Now, if the params can't be loaded, it logs a warning and keeps the entry with the details from the manifest. Callers can still see the damaged index and rebuild or drop it. Added a Python test based on the repro in the issue. It fails on 0.39.0 with the same RuntimeError. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 
|
cf5c9b5453 |
fix: validate cached remote tables on open (#4557)
## Summary Fixes #4401. - Validate cached remote table entries with the existing `describe` endpoint before returning a handle. - Preserve cached server-version behavior after a successful validation. - Add a regression test for a table dropped by another connection after its entry has been cached. A cached entry can otherwise outlive an external drop, allowing `open_table()` to return a handle that fails only on its first later operation. ## Verification [Linux validation](https://github.com/Ganesh1729-ui/lancedb/actions/runs/39420560974) passed against the exact runtime and test sources at PR head `d39092b01f6b49921b5d66ccedf29709a7ddc23d`, using the repository's Rust toolchain and `ci` profile: - `cargo fmt --all -- --check` - `cargo test --profile ci --locked -p lancedb --features remote --lib remote::db::tests::test_open_table -- --nocapture`: **8 passed**, including cached-table deletion, independent handle checkout, branch/version handling, schema seeding and describe-response handling. - Removing only the cached-hit existence guard makes `test_open_table_checks_that_a_cached_table_still_exists` fail at its `TableNotFound` assertion; restoring the fix makes the same test pass again. - `git diff --check` The validation workflow is on a separate fork branch and is excluded from this PR. These are focused, mock-backed remote unit tests; the full upstream matrix and live Cloud integration tests still need upstream CI approval. Local Windows compilation was unavailable because the MSVC environment has no `link.exe`; the linked Linux run completed successfully. --------- Co-authored-by: Ganesh172919 <ganeshreddy17291122@gmail.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
  
|
a8305e0722 |
fix(rust): accept usize::MAX as a local query limit (#4567)
Fixes #2211. Local queries converted `limit`/`offset` with `as i64`, so `limit(usize::MAX)` wrapped to `-1` and Lance failed with `Limit must be non-negative`. The remote client already worked around this (#2354); this fixes the local path. - Saturate scanner offsets to `i64::MAX` and cap limits at `i64::MAX - offset`, keeping both conversion and full-text pagination addition safe. - Use `saturating_add` for vector `top_k = limit + offset`, then cap candidates at the dataset row count to avoid indexed-search capacity and refinement overflows. Keep a positive candidate bound for empty tables because Lance rejects `k = 0`. - Leave public APIs and remote query handling unchanged. ### Testing - Regression coverage includes plain scans, indexed IVF search, refinement, index bypass, empty vector tables, and indexed full-text pagination with finite and extreme limits/offsets. - Reproduced the indexed-search and FTS overflows before applying the follow-up fix. - `cargo test --quiet --features remote -p lancedb --lib query::` — 81 passed. - `cargo clippy --quiet --features remote -p lancedb --tests --examples` — clean. - `cargo fmt --all` and `git diff --check` — clean. Vector planning now reads the dataset row count (metadata fast path) to bound candidate allocation; this was not performance-benchmarked. --------- Co-authored-by: Romin Parekh <roparekh@adobe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
6777a40dc2 |
fix: support remote tables in storage root (#3881)
## Summary - allow `db://` to select the remote storage root when a host override supplies the endpoint - preserve the empty `x-lancedb-database` header used by the legacy clients for root storage - reject root-storage connections without a host override before constructing an invalid Cloud URL - document the supported URI and cover both accepted and rejected configurations ## Root cause The shared Rust remote URL parser required every `db://` URI to contain a host. That validation was introduced when the Node and Python clients moved to the Rust implementation, but the legacy Node client intentionally treated the empty host in `db://` as the storage-root selector. ## Validation - `cargo fmt --all` - `cargo test --quiet --features remote -p lancedb test_connect_remote_storage_root` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` Fixes #1944 <!-- lance-gatekeeper-fix:v1 agent=c71ca188791ad36fe5644c12960f758c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
dd4795028b |
fix(python): align sync remote methods with async behavior (#4411)
Sync remote table and connection methods inherited bare base-class stubs, so unsupported conversion and bulk-drop operations raised `NotImplementedError` with an empty message and differed from the async API. Delegate `to_lance()`, `drop_all_tables()`, and `drop_database()` to the async implementation. Sync and async remote Lance conversion share #4306's native-table guard and reject before requesting a URI or opening client storage. Conversion documentation directs remote reads through server queries. Add an explicit Polars error, suggest query conversions in Arrow/Pandas/Polars errors, and give the remaining base-class errors an operation name. Preserve upstream `blob_mode` validation for remote Arrow and Pandas conversion. The shared remote bulk-drop error suggests dropping tables individually. Regression tests cover sync/async rejection parity, accessible and missing advertised locations, pinned versions, and bulk-drop rejection without requests. Sync blob query tests verify that descriptions come from the server and unsupported modes cannot open client storage. Keep the remote closed-blob-handle assertion consistent with the upstream `ValueError` contract. Validation: - All 30 focused regression cases passed. - 222 Python tests passed with rebuilt bindings in the locked uv environment: the full remote and error suites, plus selected local conversion, blob, and deletion tests. - Python lint and formatting, Rust formatting, `cargo check --quiet --features remote --tests --examples`, and `cargo clippy --quiet --features remote --tests --examples` passed. - Root Ruff checking reports 49 existing errors in upstream `docs/web-tests`; those files are unchanged relative to `main`. Fixes #4392 <!-- lance-gatekeeper-fix:v1 agent=95db5dd7002043a2c48924ded18b51b2 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Kerman LauandXuanwo
|
0dd78656b9 |
fix(python): honor explicit OpenAI reranker API keys (#4515)
Fixes #4514 ## Summary Honor an explicitly supplied `OpenaiReranker(api_key=...)` before falling back to `OPENAI_API_KEY`, matching the constructor documentation. Previously the environment key always won when both were set, so an instance configured for a different account still used the process-wide account. ## Tests Added four cases using real OpenAI client construction without sending requests: explicit/environment conflict, explicit key alone, environment fallback, and missing credentials. The conflict case fails before the change; all four pass afterward. Clients are closed after each assertion. - Targeted source-module tests: 4 passed. - Ruff check and format check: passed for both changed files. - Full native source build was attempted but failed because the environment has no Rust toolchain; bootstrap then attempted a cache path outside the writable workspace. The targeted tests load the current Python reranker module against installed dependencies and the released LanceDB package; no database/native behavior is exercised. Full native/integration suite not run. This change and its tests were prepared with an AI coding agent. The diff was reviewed and the failure/pass behavior verified locally. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 alexcastgonzandXuanwo
|
8cf115b081 |
fix(python): replace the retired gpt-4-turbo-preview default in OpenaiReranker (#4613)
OpenAI shuts down `gpt-4-turbo` and its snapshots on 2026-10-23 (https://developers.openai.com/api/docs/deprecations), so `OpenaiReranker` fails for every user who relies on the default `gpt-4-turbo-preview` after that date. This changes the default to `gpt-5.6-sol`, the replacement OpenAI names for `gpt-4-turbo` on that page, in the class, its docstring and the integration docs page. `_rerank` also stops sending `temperature=0`: the GPT-5 models only accept the default temperature, so keeping it would fail the new default on its first call. The reranker still asks for `json_object` output and the rest of the request is unchanged. The new test calls `_rerank` with a mocked client and checks that no `temperature` is sent. It fails on main and passes with the change. No request was made to OpenAI. This change was made with Claude Code. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]
|
27b65064d0 | fix(python): support custom blob thresholds without pylance (#4634) | ||
 Joaquin Hui GomezandXuanwo
|
9da4658d57 |
fix(python): accept with_row_id in take_offsets/take_row_ids on every Table implementation (#4374)
`Table.take_offsets` and `Table.take_row_ids` declare a keyword-only `with_row_id: bool = False`, but none of the three implementations accept it — `LanceTable`, `AsyncTable` and `RemoteTable` all raise `TypeError: ... got an unexpected keyword argument 'with_row_id'` on the documented call. The builders they return already support it (`LanceTakeQueryBuilder.with_row_id()` / `AsyncTakeQuery.with_row_id()`), so this is a signature that never reached the implementations. The base docstrings also don't mention the parameter and still say `take_offsets` returns a `pa.RecordBatch` (it returns a `LanceTakeQueryBuilder`), so those are corrected too. Each implementation now accepts the kwarg and applies `.with_row_id()` to the builder it returns. Tests: sync and async assert `_rowid` appears in the result columns; the remote test uses the suite's request-capturing fixture and asserts `with_row_id: true` reaches the wire (and is absent by default). All three fail on `main` with the `TypeError`. ruff format/check clean. A related mismatch I left alone: the deprecated `LanceTable.create_scalar_index`/`create_fts_index` also drop a `wait_timeout` the base declares — deprecated since 0.25, so not worth touching here. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 LeonSGPandXuanwo
|
3321860e9f |
fix(python): preserve Arrow map scalars in updates (#4448)
Arrow map scalars returned by a table query cannot currently be passed back to `update`: the generic Arrow scalar conversion turns them into a list of tuples, which the SQL literal encoder cannot handle. Empty map scalars become an untyped SQL list and also fail to update a map column. Add a dedicated `MapScalar` conversion that renders its keys and values as a SQL map while preserving the existing null conversion. The regression tests exercise real synchronous and asynchronous table updates, quoted keys, nullable values, empty typed maps, and nested maps. Validation: the complete documented unit-test scope passed locally (1,515 passed, 37 skipped; unchanged baseline: 1,507 passed, 37 skipped). Full doctests passed (71 passed, 6 skipped), and repository-wide Python formatting/lint, Rust formatting/clippy, and Python 3.13 type checking passed. The locally built native wheel also passed all eight regression cases in a separate Python 3.13 environment. Full Linux/S3 integration and all platform matrices remain for CI. The SQL map constructor's existing backend constraints still apply: duplicate keys are rejected, and a nested map containing both nonempty and empty inner maps can fail SQL type unification. This change does not claim arbitrary Arrow map roundtrip support. AI assistance: implementation and local verification used Codex with gpt-6.1-sol agents; the diff and evidence were reviewed before submission. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
e74a7c15b5 |
fix: bump tokio to address RUSTSEC-2026-0386 (#4642)
Upgrade the locked Tokio version from 1.53.1 to 1.53.3 to fix [RUSTSEC-2026-0386](https://rustsec.org/advisories/RUSTSEC-2026-0386) and restore the Rust workflow's `deny` job. The diff changes only Tokio's version and checksum. Validation: - `cargo deny check advisories` — passed. - `cargo deny check advisories bans licenses sources` — passed with the repository's `deny.toml`. - `cargo check --quiet -p lancedb` — passed (dead-code warnings). - PR `deny` job — [passed](https://github.com/lancedb/lancedb/actions/runs/40000920646/job/120576455117). |
||
 KarthikSandXuanwo
|
702389654e |
fix(python): align pylance extra and preview install indexes (#4556)
Fixes #4248. The preview index still contains pylance 1.0.0b6. uv's default first-index resolution can select it, or backtrack to an old LanceDB release when installing the extra. Meanwhile, the extra still permits pylance 5 even though current main bundles Lance 14. Raise the optional dependency to `pylance>=14.0.0b12`, the version already pinned in the test extra, and keep the lockfile metadata consistent. Document both Fury indexes and uv's `unsafe-best-match` strategy in the Python README and the source-owned quickstart (`docs/web/quickstart.mdx`). Explain the index trust requirement and recommend installing the extra to enforce the version floor. Add CI checks that reject older Lance majors and ensure the pylance and tests extras remain compatible. Run these checks when either dependency manifest changes, so a future Lance major bump cannot silently leave the Python minimum behind. ### Validation - Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects `pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and the unexpected `blob_handling` scanner argument using a real blob table. - Resolved the modified `python/pyproject.toml` with the pylance extra against both indexes: selects `pylance==14.0.0b18`, matching the current Rust pin. - Ran both documented pip and uv installations in isolated environments: `lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both package dependency checks pass. - New regression fails on the original dependency bound (27 rejected-version cases incorrectly accepted), then passes with the fix. Full CI-script suite: 3 tests pass. - `uv lock --check` with both indexes passes; focused Ruff checks pass; `git diff --check` passes. - Repository-wide Ruff reports 49 existing errors in untouched `docs/web-tests` files. Unrelated formatting changes were discarded. Runtime smoke checks used released native wheels; source dependency resolution was tested directly from the modified pyproject. No Rust or Python runtime code changes. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
5a79c7d22d |
ci: run aarch64 Windows tests on main only (#4637)
Run the aarch64 Windows build/test cycle only on pushes to main so PRs wait for a single Windows platform (x86_64). Main keeps the same native aarch64 coverage as before. GitHub evaluates job-level `if` before the matrix strategy is applied, so the aarch64 target is split into its own `windows-arm` job instead of being filtered inside the existing matrix. The job keeps the historical check name. |
||
 LuQQiu
|
b762ec221b |
docs: add Enterprise distributed search reference page (#4625)
Adds the Enterprise reference page at `docs/web/enterprise/distributed-search.mdx`, following the same arrangement as Enterprise architecture. The page describes distributed search and the embedded execution model, with links to shared search guides. Enterprise architecture links to it, and the official sidebar lists it immediately after Architecture in the existing Deployment group. Adds stable anchors for the RaBitQ and `approx_mode` sections linked by the Enterprise guide. Companion: https://github.com/lancedb/sophon/pull/8986 supplies the detailed Enterprise overlay at the same path. Merge this reference-page PR first so the overlay has an existing route to replace. Validation: - Combined-site assembly with both PR branches passed (309 files). - Mintlify build validation and link, anchor, and redirect checks passed. |
||
|
Xuanwo
|
a9535684cc |
chore(deps): bump ctutils to 0.4.4 for RUSTSEC-2026-0385 (#4638)
Fixes RUSTSEC-2026-0385: `ctutils` < 0.4.4 generates masks with branches on secret values (`u128` mask generation / comparisons are not constant-time on some targets). - Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0385 - Affected: `ctutils 0.4.2` pinned in `Cargo.lock`, reached transitively via `digest 0.11` (RustCrypto) - Change: `cargo update -p ctutils --precise 0.4.4`; single lockfile-only change (patch bump, no API impact) The `deny` CI job started failing for every PR head after the advisory was published at 06:45Z today; `main`'s last `deny` run (05:51Z) predates it and does not yet cover this advisory. |
||
 Wyatt Alt
|
45c4b159ca |
fix: cap the remote job poll interval at one second (#4629)
job.wait() polled on a backoff that doubled up to 5 s, so a job's completion was seen 2.5 s late on average, as long as a small refresh itself takes. Cap the interval at 1 s; the first polls still back off from 200 ms, so a quick job is seen quickly and a long one costs one request a second. Linear: OSS-2704 https://linear.app/lancedb/issue/OSS-2704 |
||
|
Xuanwo
|
444f7f3bde |
ci: refresh apt lists before installing protobuf in docs build (#4636)
The docs deploy workflow installs protobuf-compiler without refreshing apt lists first. Ubuntu noble-updates replaced `protobuf 3.21.12-8.2ubuntu0.3` with `.0.4`, so stale package lists in older runner images make `apt install` fail with 404 and exit code 100. Run `apt update` before installing, matching the other workflows. |
||
 Colin Patrick McCabe
|
631ea41a63 |
feat!: return lazy iterators for table and namespace listings (#4599)
Make table and namespace listings consistent with other resource
listings across Rust, Python, and TypeScript. Callers can iterate over
individual names across all pages, with configurable page sizes and
continuation tokens.
In order to avoid breaking existing Python users, this change adds some
compatibility methods to the returned Python iterators. Code such as:
```
for t in db.list_tables().tables:
print(t)
```
will continue to work as expected because the iterator has a `tables`
field and a `page_token` field, which function as expected to reveal the
first page of results. In other words, it is never necessary to use the
iterator __as an iterator__. It may be used as if it were an old-style
response object. (As the iterator progresses, the `tables` list may
switch to the second, third, etc. page of results.) Similar
compatibility code allows the async python API to keep working as well.
|
||
 Christopher Krah
|
88203d2515 |
fix(python): keep a renamed Permutation column in its position (#4626)
The `Permutation.rename_column` docstring says it "only changes the name
of the column that is visible from this permutation". But
`permutation.py:868-870` adds the new name at the end of the selection
and then deletes the old key, so the renamed column also moves to the
last position. Hugging Face `datasets`, which `Permutation` mirrors
(#2725), keeps it in place. Positional formats feel it most:
`with_format("numpy")` swaps the feature order of every row.
Before, on `d97df0c`:
```
$ python repro.py <tree>/python/python # lancedb 0.41.0b12 extension, Python sources from the tree
['id', 'value']
['value', 'row_id']
[{'value': 0, 'row_id': 0}]
[[10 1]]
```
After:
```
$ python repro.py <tree>/python/python
['id', 'value']
['row_id', 'value']
[{'row_id': 0, 'value': 0}]
[[ 1 10]]
```
<details><summary>repro.py (its first output line,
<code>lancedb.__file__</code>, is cut above)</summary>
```python
import sys; sys.path.insert(0, sys.argv[1])
import tempfile
import lancedb
from lancedb.permutation import Permutation
print(lancedb.__file__)
db = lancedb.connect(tempfile.mkdtemp())
tbl = db.create_table("t", [{"id": i, "value": i * 10} for i in range(3)])
perm = Permutation.identity(tbl)
print(perm.column_names)
renamed = perm.rename_column("id", "row_id")
print(renamed.column_names)
print(renamed.__getitems__([0]))
print(renamed.with_format("numpy").__getitems__([1]))
```
</details>
For comparison, `datasets` 5.1.0 gives `['new_id', 'value']` for
`Dataset.from_dict({'id': [1, 2], 'value': [3, 4]}).rename_column('id',
'new_id').column_names`.
No issue exists for this. The fix rebuilds the selection dict in order.
The cost: code that renamed a column and then read it by position in
numpy batches (say `[:, -1]`) gets a different column after this, with
no error. `rename_columns` calls `rename_column` once per name, so it
gets the fix too. The tests that check schema order only rename `value`,
the last column, and `test_rename_rowid` checks membership but not
order. The new test renames `id`.
### Verification
```
# Python 3.12.6; lancedb 0.41.0b12 wheel from pypi.fury.io, its _lancedb.abi3.so copied into the tree; pylance 14.0.0-rc.1
$ PYTHONPATH=python python -m pytest python/tests/test_permutation.py -q -p no:cacheprovider
|
||
|
lancedb-gatefixer[bot]andXuanwo
|
f5f4463b15 |
fix(node): materialize nested struct blobs in bytes mode (#4611)
`blobMode: "bytes"` left blobs inside structs as descriptors because projection matching ignored parent structs and byte replacement only rebuilt top-level fields. Expand direct struct projections and aliases to their blob leaves, then recursively replace the corresponding Arrow fields and data with `LargeBinary`. Track literal output names separately from descendant paths so scalar aliases stay intact and every overlapping blob projection is converted. Preserve null structs, null and empty blobs, sibling fields, schema metadata, and the existing row ID and snapshot behavior. Blobs inside lists retain their descriptors. Update the API docs to describe struct support. Regression coverage includes full and parent-struct projections, leaf and struct aliases, dotted output names, overlapping projections in both orders, nested structs, nulls, empty blobs and results, multiple batches, list siblings, query reuse, row IDs, and concurrent writes. Regression tests reproduce the nested-blob and projection-collision failures before the corresponding changes. Validation: - `pnpm build` - `pnpm lint` - `pnpm run docs` - `pnpm test __test__/nested_blob.test.ts __test__/blob.test.ts __test__/query.test.ts --runInBand --verbose=false` — 142 passed - `pnpm test __test__/table.test.ts --runInBand --verbose=false -t 'when dealing with blob columns'` — 60 passed Fixes #4609 <!-- lance-gatekeeper-fix:v1 agent=76edaac77d044c468d569b58efa2104a generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Ishita Singh
|
e5873d9763 |
fix(node): support nested blobs with blobMode 'bytes' (#4610)
Closes #4609. `blobMode: "bytes"` only resolved top-level blob columns, so a blob nested in a struct came back as its descriptor. Now struct projections that contain blob columns are resolved too: the bytes are fetched per blob column and the struct is rebuilt with `LargeBinary` leaves, preserving struct validity so a null struct stays null. Sibling fields pass through untouched, aliases work for both `info.image AS image` and `info AS details`, and blobs nested in lists still keep their descriptors. Tested with `npx tsc`, `biome check`, and the full `table.test.ts` suite (580 tests, including new regressions for select-all, struct selection, aliases, null structs, null and empty blobs, sibling fields, empty results, and single-row batches). Not run against remote tables. |
||
 Captain
|
a6011dcd95 |
fix(python): accept NumPy text arrays in Ollama embeddings (#4605)
`OllamaEmbeddings.compute_source_embeddings(np.array(["hello", "world"]))` raises a validation error before sending a request. The embedding interface accepts NumPy arrays, but the adapter passes the array directly to Ollama's string-or-sequence request field. Convert NumPy text arrays to lists before the batched request. This preserves input order, the single request, model options and `keep_alive`. The tests cover Unicode/object/sliced arrays, empty input, existing list/Arrow/query inputs and API error propagation. The added input-contract tests also cover two-dimensional arrays and byte strings, with equivalent list inputs as controls. The real Ollama SDK rejects 2-D inputs and invalid UTF-8 locally, before transport. Valid UTF-8 bytes are decoded normally. No extra adapter validation is needed for these cases. Validation on Python 3.12.14 / macOS arm64: - Built the Rust/PyO3 bindings using the repository's uv/maturin environment. - The initial regression run had 5 failures and 4 passing controls on the base; all 9 passed with the fix. With the 6 additional input-contract cases, the related embedding and trust suites now total 55 passed / 11 skipped. - All Python Ruff lint and formatting checks pass. Root checks still report the same 49 lint errors and 11 documentation formatting files reproduced on the untouched base. - Separate review exercised 13 public API cases. Another 7 groups passed through a real loopback HTTP transport, including local validation, UTF-8 serialization and empty-batch model-error propagation. The regression tests use the real Ollama 0.6.2 SDK with HTTPX MockTransport. The loopback check uses an HTTP surrogate, not an Ollama daemon. No live model inference was run. A live Ollama server was unavailable locally. [The current server handler](https://github.com/ollama/ollama/blob/2cd63f7a15039ccd790e97b31edc091fa1707c56/server/routes.go#L1212) explicitly returns HTTP 200 with empty embeddings for empty input after model lookup and runner setup. The adapter preserves the existing empty-list request and its model/setup errors. This is a source check, not a live-server test. Developed with OpenAI Codex, with separate agent review and executed local checks. |
||
 Raphael Fakhri
|
566517be05 |
fix(python): apply pyarrow expression filters in PyarrowDatasetAdapter (#4592)
`PyarrowDatasetAdapter` lets DuckDB query a LanceDB table, but DuckDB pushes `WHERE` clauses into `Dataset.scanner(filter=...)` as `pyarrow.compute.Expression` objects. The adapter forwarded them to `Table.count_rows` and `LanceQueryBuilder.where`, which accept only SQL strings, so every filtered DuckDB query failed with `TypeError: argument 'filter': 'Expression' object is not an instance of 'str'`. This change: - Applies a `pyarrow.compute.Expression` filter locally with `pyarrow.dataset.Scanner.from_batches`. The scan reads all columns and pyarrow applies the filter and projection, because DuckDB can filter on a column it does not project. - Applies `head` limits after the filter, so the limit counts matching rows. - Routes `PyarrowDatasetAdapter.count_rows(filter)` through the scanner, so it accepts both filter types. SQL string filters keep the existing path and still run inside LanceDB. Fixes #4591 ### Testing - New `test_dataset_adapter_expression_filter` in `test_pyarrow.py` covers `count_rows`, `to_table`, `head` and a projection that excludes the filtered column. - New `test_query_with_filter` in `test_duckdb.py` runs DuckDB queries with a `WHERE` clause. - Both tests fail without the fix and pass with it. `ruff check` and `ruff format --check` pass. |
||
|
Captain
|
f2c6aa35c2 |
fix(python): normalize VoyageAI image modes before JPEG encoding (#4576)
PIL images, PNG bytes and local image paths in VoyageAI source embeddings are saved directly as JPEG. RGBA, grayscale-with-alpha (LA) and palette (P) images therefore raise `OSError: cannot write mode ... as JPEG` before the provider is called. Convert to RGB at all three JPEG encoding sites. The caller’s original PIL image remains unchanged. Document that alpha is discarded and callers should composite onto a chosen background when it matters. Validation: - New image regressions + existing VoyageAI tests: **18 failed / 26 passed before → 44 passed after**. Tests cover both multimodal models, PIL/bytes/Path, RGBA/LA/P/RGB, decoded JPEG pixels and invalid images. - Independent local LanceDB table ingestion smoke: three PNG modes stored with 1024-dimensional vectors, original bytes preserved. Uses the freshly built current native extension. - Adjacent local embedding tests: 15 passed, 1 skipped (no OpenAI key), 9 deselected. The separate Flickr download test failed with `RemoteDisconnected`; it is excluded from the local result. - Pyright: 0 errors; CI-pinned Ruff 0.9.9 lint and Python formatting passed. External VoyageAI calls are stubbed; real Pillow conversion and LanceDB ingestion execute locally. No live provider or full-repository test result is claimed. AI assistance was used for implementation and validation. |
||
 Srijan Keshri
|
118e606e8e |
fix(python): align vector parent indices after flattening null lists (#4572)
Arrow's `flatten()` skips child values hidden under null list parents, but `list_parent_indices()` can retain their indices. This makes `has_nan_values()` raise `ArrowInvalid` because the parent indices and NaN mask have different lengths. Filter out indices belonging to null parents before applying the mask. The regression test covers `ListArray` and `LargeListArray`, float16/32/64, and chunked inputs: 12 cases. NaN and null-component policies stay the same. The crash also affects #4503. This patch fixes it separately from that PR's changes to bad-vector policies. Validation: - All 12 regression cases failed before the fix. The utility suite now passes with 87 passed and 1 skipped. - Ruff lint and formatting pass for both changed files, as does `git diff --check`. - The patch applies cleanly to main at `ca275b6d`. The patched helper from that revision also passes all 12 cases. - A separate AI reviewer checked commit `343082a6` and found no blocking issues. They reran the utility suite and compared 4,059 generated cases plus 12 explicit null-parent cases with a row-by-row reference implementation. Those checks include slices, chunked inputs, and fixed-size lists. SDK tests used base `07e626ec` with the existing compiled extension. The broader Python run stopped after 474 passed and 23 skipped at `test_local_module_outside_code_warns_at_registration` with `ModuleNotFoundError`. That test also fails with the original vector helper restored. Repository-wide Ruff reports 49 lint errors and 11 files needing formatting, all in untouched documentation examples. Full SDK integration tests against current main require a newer native build and have not been run. |
||
|
Captain
|
0cb5bd0a5d |
fix(python): accept VoyageAI text batch containers (#4558)
Direct VoyageAI `compute_source_embeddings()` calls reject `list[str]` and NumPy text batches before reaching the provider, although both belong to the declared `TEXT` input type. For example, `["first", "second"]` raises `ValueError: Input type <class 'list'> not allowed with text model.` Normalize these containers to a list before the existing string-entry validation. The regression cases exercise the public embedding API with a stub provider: input order, list/NumPy batches, Arrow controls, empty batches, and rejection of non-text entries and ordinary two-dimensional arrays. The VoyageAI integration page now documents the accepted text containers. Verified on current main `7b3ba4c` with a locally compiled native extension (0.41.0b7, Python 3.12.12): - VoyageAI suite: 7 failed / 20 passed before the fix; 27 passed afterward. - Adjacent embedding suite: 13 passed, 4 skipped, 8 slow cases deselected. - Python CI's pinned Ruff 0.9.9 lint/format checks pass; configured Pyright reports zero errors/warnings; all applicable pre-commit hooks pass. The provider is stubbed; no live Voyage API request or model-quality evaluation was performed. AI disclosure: I used Codex for investigation, implementation, tests, review, and drafting this description. |
||
  Yash Raj PandeyandClaude Opus 5.5
|
2af7d88ec7 |
fix(python): keep OpenaiReranker scores on the rows they rate (#4549)
`OpenaiReranker._rerank` asks the chat model for a relevance score per document. The model can return the documents in any order, for example sorted by score. The reranker then replaces the `text` column with the documents in that order and appends the scores in that order. The other columns keep the input order. So after a rerank, the `text` and `_relevance_score` of a row belong to a different row. The `id`, the vector and every other column still describe the original row. If the model leaves out a document, `append_column` raises `ArrowInvalid`, because the new column has fewer rows than the table. ## Fix `_rerank` keeps the `text` column unchanged, so it also keeps its position and type. It maps each score back to its row by the document content. If the model leaves out a document, `_rerank` raises a `ValueError` that gives the number of documents with no score. ## Tests `python/python/tests/test_openai_reranker.py` replaces the chat client with a stub that returns a fixed reply. The tests do not need `openai` or an API key. - One test returns the documents sorted by score. It checks that each `id` keeps its own `text` and score. On `main`, the row with id 1 gets the text `cats purr`, which belongs to id 2. - One test returns 2 of 3 documents. It checks for the `ValueError` with the count of missing documents. On `main`, the call raises `ArrowInvalid: Added column's length must match table's length`. - Both tests fail on `main` and pass with the fix. - `test_rerankers.py`: 16 passed, 13 skipped, on the lancedb 0.40.0 wheel with and without the fix. The skipped tests need real API keys. - `ruff format --check` and `ruff check` (0.9.9) pass on the changed files. - I did not build the native extension. I ran the tests against the lancedb 0.40.0 wheel with the changed file copied over it. In that wheel, `rerankers/openai.py` and `rerankers/base.py` are byte-identical to `main`. The Node SDK has no OpenAI reranker, so there is no parity change. This change does not conflict with #4515, which edits `_client` in the same file. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Yash Raj PandeyandClaude Opus 5.5
|
353ce4d80c |
fix(python): give each JinaEmbeddings instance its own session (#4547)
`JinaEmbeddings` stores its `requests` session in a `ClassVar`. The first instance builds the session with its own API key, and every later instance reuses that session. So a second instance with a different `api_key` sends the first instance's key. ## Fix The session is now a per-instance `cached_property`, the same pattern `OpenAIEmbeddings` uses for `_openai_client`. `_init_client` and the `ClassVar` are gone. An instance with no `api_key` and no `JINA_API_KEY` now raises the missing-key error. Before, it reused the session of an earlier instance and sent that instance's key. ## Tests `test_jina_embeddings_use_their_own_api_key` creates two instances with different keys through the registry. It stubs `requests.Session.post` and records the `Authorization` header of each request. - On `main`, the test fails because both requests send `Bearer key-a`. With the fix, it passes. - `test_embeddings.py`: 18 passed, 8 skipped. - An instance still pickles after the session exists, and `safe_model_dump()` still returns only `api_key`. - On pydantic 2.7.4, the minimum version, a model with the same `cached_property` gets one session per instance and leaves it out of `model_dump()`. - `ruff format --check` and `ruff check` (0.9.9) pass on the changed files. - I did not build the native extension. I ran the tests against the lancedb 0.40.0 wheel with the changed file copied over it. In that wheel, `embeddings/jinaai.py` is byte-identical to `main` before this change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Yash Raj PandeyandClaude Opus 5.5
|
5fa30ae404 |
fix(python): honor explicit Cohere and VoyageAI reranker API keys (#4546)
`CohereReranker` and `VoyageAIReranker` read the environment variable before the `api_key` argument. When `COHERE_API_KEY` or `VOYAGE_API_KEY` is set, the reranker ignores an explicit `api_key` and uses the account from the environment. The docs say the opposite. `docs/web/integrations/reranking/cohere.mdx` and `voyageai.mdx` use the environment variable only when `api_key` is not provided. `JinaReranker` and `WatsonxReranker` already put the explicit key first, and #4515 makes the same change for `OpenaiReranker`. ## Fix `CohereReranker._client` and `VoyageAIReranker._client` now use `self.api_key` first and the environment variable as the fallback. ## Tests `python/python/tests/test_reranker_credentials.py` replaces the SDK module with a stub whose `Client` records the API key. It covers both keys set, each key alone, and no key. With the stub, the test runs without `cohere` or `voyageai` installed. - On `main`, the 2 cases with both keys set fail and the other 6 pass. With the fix, all 8 pass. - `test_rerankers.py` with the new file: 24 passed, 13 skipped (the skipped tests need real API keys). - `ruff format --check` and `ruff check` (0.9.9) pass on the changed files. - I did not build the native extension. I ran the tests against the lancedb 0.40.0 wheel with the changed files copied over it. In that wheel, both reranker files are byte-identical to `main` before this change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 Avanish Garg
|
bc4c32b779 |
fix: keep tagged versions when pruning in optimize (#4544)
`optimize()` fails once a tagged version is older than the cleanup cutoff, although the docs say tagged versions are exempt from cleanup, and the failed prune also skips the index-optimize step (repro and numbers in the issue). Change: `OptimizeAction::Prune` already exposes `error_if_tagged_old_versions: Option<bool>`, so no new API. `cleanup_old_versions` and `cleanup_old_versions_before` in `rust/lancedb/src/table/optimize.rs` now use `unwrap_or(false)`, which covers `OptimizeAction::Prune`, `OptimizeAction::All` and `optimize_prune_before`. The Python and Node bindings pass `None`, which now means "retain tagged versions"; I left `python/src/table.rs` and `nodejs/src/table.rs` untouched (this also avoids a conflict with #3870, which edits that region). `Some(true)` still opts in to the error. Doc comments updated; the sentence in `docs/web/management/table.mdx` is accurate once this lands. Tests: Rust `test_optimize_prune_retains_tagged_versions` (`Some(true)` still errors; the default retains the tag; `optimize_prune_before`; `checkout_tag` plus a data read), a Python sync test with a scalar index (index step runs, a second optimize works, untagged versions are pruned, `checkout("release-1")` works), a Python async test and a Node test. **What I could and could not verify (please read):** - I could NOT compile Rust here (not enough disk), so `cargo check`, `cargo test`, the Python/Node binding builds and therefore the fixed-path assertions of the new Rust, Python and Node tests have not been run. This is why the PR is a draft; I will mark it ready once CI is green and fix whatever it finds. - Ran: both new Python tests fail on the released 0.40.0 wheel with the exact error above (so they do catch the bug); pylance 13.0.0 confirms the semantics (`cleanup_old_versions(error_if_tagged_old_versions=False)` removes the untagged versions and keeps the tagged one, still readable with `lance.dataset(version="keep")`); in Lance v14.0.0-beta.7 `cleanup.rs` the policy field is `bool`, the builder default is `true`, and Lance's own `auto_cleanup_hook` uses `Some(false)`; `rustfmt --edition 2024 --check` passes on `optimize.rs`. - The Rust and Node tests rely on timestamp ordering (`nextMillisecond()`, `days(0)`/`now()`), so they could be flaky: tell me if you prefer another approach. Behaviour change: anyone who relied on the error to learn that a tag blocks cleanup now gets a silent retain. Not changed: the deprecated sync `LanceTable.cleanup_old_versions` calls pylance directly and still errors on tagged versions (the keyword may not exist in the oldest supported pylance); `freshness.rs:1377` and `refresh.rs:1077` still pass `None` (I did not check what those paths do with tagged versions). Fixes #4543 |
||
|
Avanish GargandXuanwo
|
2d54630a3c |
fix(python): stop retrying permanent embedding errors and cap the backoff delay (#4542)
Permanent embedding errors (HTTP 400/404/422 and other 4xx) were retried the full `max_retries` times with an uncapped, multiplicatively jittered backoff, so a typo'd model name looked like a hang of about an hour (median 50 min of sleeping with a 404; see the issue for the repro and numbers). Changes (`python/python/lancedb/embeddings/utils.py`, `embeddings/base.py`): - An HTTP status in 4xx other than 408/409/429 is raised immediately. The status is read from `status_code`, `http_status`, `response.status_code`, an integer `code` (google-api-core, urllib) or a botocore `response["ResponseMetadata"]["HTTPStatusCode"]`. AWS throttling reported as 400 stays retryable (matched by error code). `AuthenticationError` by name is unchanged. - Still retried: 408, 409, 429, >=500, and anything without a recognisable status (connection errors, timeouts, custom exceptions), so existing custom embedding functions behave as before. - New keyword `max_delay=60` (last in the signature, default keeps call sites working). `delay = min(delay * base, max_delay)` with equal jitter `delay/2 + random() * delay/2`. With defaults the waits are [1,2], [2,4], [4,8], [8,16], [16,32], [30,60], [30,60] s, worst case about 182 s instead of up to 2.5 h. - The "Maximum number of retries" exception now has `from e`. The `max_retries` comment in `base.py` describes the rules. Not changed: `embeddings/openai.py` (it swallows `BadRequestError`; open #4154/#4005/#3152 cover it), Retry-After handling (the code never used it), and the "C Data interface error" wrapping from the native layer (the original message is now included in it). Behaviour changes to be aware of: every 4xx except 408/409/429 fails fast, including 401/403 coming through `response.status_code` (requests/httpx), which used to be retried; a provider that signals transient failures with a 4xx and no AWS-style code is no longer retried; the first retry waits 1-2 s instead of 2-4 s. Tests (`python/python/tests/test_embeddings.py`): 48 new cases (23 test functions): permanent errors are not retried, transient errors are, botocore-style errors, the delay is capped, exact exponential growth without jitter, jitter bounds, no float overflow for a large `max_retries`, the original error is attached, and end to end through `table.add` and `table.search` with a stub function (one call and no sleep). On `main` 23 of them fail with the change stashed; the rest are guards that must not change. `test_embeddings.py`: 61 passed, 4 skipped. `ruff format`/`ruff check` are clean on the touched files. Verification caveat: `main`'s Python layer needs a newer native module than the 0.40.0 wheel, so the tests were run with the wheel's native module preloaded through a small shim (the same shim ran the baseline on `main`); a wider run of embeddings/pydantic/query/table tests has the same 36 failures before and after (that mismatch). Not run: real provider APIs (a fake OpenAI-style server was used), Node or Rust code. Fixes #4541 Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 adenzhou1350
|
9f4cd1618b |
fix(python): handle empty batch iterables with a schema (#4538)
## Summary `create_table(data=iter(()), schema=...)` currently raises `RuntimeError: coroutine raised StopIteration`, although an empty list with the same schema works. A tuple of batches also fails because the helper calls `next()` before obtaining an iterator. Normalize the iterable once and use the supplied schema for an empty reader. Without a schema, raise a clear `ValueError`. Nonempty schema inference and batch casting are unchanged. Tests cover synchronous/asynchronous creation, empty/nonempty iterators and tuples, and schema-aware `add()` sanitization. The default schema-less `add()` rejection is deliberately unchanged. ## Validation - Built the current Rust/PyO3 extension from base `9f37f392` with the repository's `ci` profile; both Python-only arms use the same verified extension and locked Python dependency versions. - Complete `python/tests/test_table.py` + `python/tests/test_scannable.py`: **268 passed, 0 skipped**. The unchanged production code with identical tests has **24 failures / 244 passes**. - Both arms used four-CPU affinity. An initial unrestricted-affinity run exhausted thread creation under the private memory limit; that failed attempt is not counted as a pass. - Package-wide Ruff check/format pass. Repository-wide checks still report pre-existing findings in unchanged `docs/web-tests`; those files are not modified here. AI-assisted change, reproduced against unchanged production code and validated with a native source build. This does not claim the full repository/platform matrix or a performance improvement. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
Christopher KrahandXuanwo
|
b6d15558ec |
fix(python): stop retrying a missing API key or provider package (#4521)
`retry_with_exponential_backoff` retries every exception except an auth
error (`python/python/lancedb/embeddings/utils.py:298-318`). For Cohere,
Gemini, Jina and VoyageAI a missing API key raises from
`api_key_not_found_help`, and a missing provider package raises
`ImportError` everywhere. Neither can succeed on a retry, yet both sleep
through all seven rounds and then surface wrapped in `Exception("Maximum
number of retries...")`.
Before, on `142d032`, a VoyageAI function with no key set (script below,
`time.sleep` recorded, `random.seed(0)`):
```
$ python lance_retry_deterministic.py python/python
raised: Exception('Maximum number of retries (7) exceeded.', ValueError('Please set the VOYAGEAI_API_KEY environment v
attempts: 8 sleeps: [4, 13, 37, 93, 280, 788, 2811]
slept 4026 s before a missing-API-key error surfaced
```
After:
```
$ python lance_retry_deterministic.py python/python
raised: MissingApiKeyError('Please set the VOYAGEAI_API_KEY environment variable.')
attempts: 1 sleeps: []
```
<details><summary>lance_retry_deterministic.py</summary>
```python
# loads the pure-Python embeddings package from the tree, native extension stubbed
import os, random, sys, time, types
TREE = sys.argv[1]
pkg = types.ModuleType("lancedb"); pkg.__path__ = [TREE + "/lancedb"]; sys.modules["lancedb"] = pkg
native = types.ModuleType("lancedb._lancedb"); native.__getattr__ = lambda name: None
sys.modules["lancedb._lancedb"] = native
import pyarrow as pa
from lancedb.embeddings import get_registry
random.seed(0)
os.environ.pop("VOYAGE_API_KEY", None)
slept = []
time.sleep = slept.append
func = get_registry().get("voyageai").create(name="voyage-3")
try:
func.compute_source_embeddings_with_retry(pa.array(["hello"]))
except Exception as e:
print("raised:", repr(e)[:110])
print("attempts:", len(slept) + 1, "sleeps:", [round(s) for s in slept])
print("slept %.0f s before a missing-API-key error surfaced" % sum(slept))
```
</details>
The message names the wrong variable: `voyageai.py:501` reads
`VOYAGE_API_KEY`. I left that for a separate fix.
`api_key_not_found_help` now raises `MissingApiKeyError`, a `ValueError`
subclass, so existing `except ValueError` callers still catch it. That
error and `ImportError` join the 401 exemption from #2995. A plain
`ValueError` is still retried, because `bedrock.py:171` wraps boto3
throttling in one. No issue exists for this. OpenAI is not covered: with
no key, `openai.OpenAI()` raises `OpenAIError`, which has no status
code, so it still retries. I can follow up on that.
### Verification
```
# lance_stub: an empty stand-in for lancedb._lancedb, loaded from the source tree
$ pytest -p lance_stub --noconftest python/tests/test_embeddings.py -k "retry or 401 or deterministic" -q
5 passed, 23 deselected in 0.59s
# utils.py from
|
||
 Aviral Saxena
|
d826b6115a |
fix(rust): explain client-side model files when remote tokenize fails (#4519)
Addresses option 2 of #4440. tokenize() builds the FTS analyzer on the client. For jieba/* and lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and when they’re missing you only got lance’s raw Invalid directory path: ... error, even though server-side search works. Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps the original cause and adds that tokenize() runs on the client, which directory the files need to be in, and that for remote tables they must match the server’s. I also updated the tokenize docs (Rust, Python, TS, generated JS docs) to say a mismatch can’t be detected, so tokens may differ from the server’s index. Behavior when the files exist is unchanged. Detecting a mismatch needs a server-side tokenize endpoint (option 1), so I left it out. The hint isn’t limited to remote tables, since Table::as_native is slated for removal and the wording fits local tables too. Tests: Rust test with mocked jieba/default index_details and an empty model dir, one checking other tokenizer errors don’t get the hint, and Python sync/async tests. |
||
|
adenzhou1350
|
93f054bcc5 |
fix(rust): finish query streams after a zero-duration timeout (#4506)
A positive query timeout emits one error and then ends, but `Duration::ZERO` leaves `TimeoutStream` in `NotStarted`, returning the same error on every subsequent poll. Transition the zero-duration path to `Completed` before returning its error. The initial error is unchanged. Extend both timeout tests to check that subsequent polls return `None`; the normal-completion test is unchanged. This affects callers that continue polling after an error, not `try_collect`, which already stops at the first error. Validation: - `cargo fmt --all` and `cargo fmt --all --check`: pass. - Mechanically extracted the unchanged implementation and three upstream timeout tests into a native Rust test binary using real cached Arrow 58.4.0, DataFusion 54.1.0, futures 0.3.32 and Tokio 1.53.1 libraries (matching the current lock). With the new assertions, baseline: 2 pass / 1 fail; fix: 3 pass. - Full-crate `cargo test --locked --offline --features remote -p lancedb --lib test_timeout_stream` could not start because this environment lacks the newly locked Lance `v14.0.0-beta.10` checkout. The component comparison is not a claim that the full Cargo suite passed; CI still needs to validate integration. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
3d883af674 |
fix(python): propagate invalid index wait_timeout errors (#4495)
Invalid `wait_timeout` values currently reach `Duration::extract(...).unwrap()` in both native index-creation bindings. A normal Python argument error becomes `pyo3_runtime.PanicException`, which is not caught by `except Exception`. Propagate the extraction error with `map(...).transpose()?` before constructing the index operation. Valid `None` and `timedelta` behavior is unchanged. Tests cover both `create_index` and `create_index_async`, three invalid types, valid `None`/`timedelta`, and no index created after rejection. Validation against main `0aa842b`, Python 3.12, uv-locked dependencies, freshly compiled PyO3 extension for each arm: - Unmodified source: 6 new regressions fail with `PanicException`; 6 controls pass. - Fixed source: 12/12 pass; broader index-creation regression selection: 18/18 pass. - `cargo fmt --all --check`, Ruff check/format, and diff check pass. The broader selection includes the 12 cases above. No cloud, full-suite, or performance claim. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
 adenzhou1350andaden
|
b205e7c628 |
fix(node): prevent table listing limits from wrapping (#4492)
## Summary Prevent large JavaScript table-listing limits from wrapping during the native `u32` conversion. With two tables, `limit: 2 ** 32` currently returns no tables and `2 ** 32 + 1` returns only one. Validate nonnegative safe integers using the existing helper and clamp valid limits above `u32::MAX` before calling the native binding. Apply the same normalization to `tableNames` and `listTables`, including namespace overloads. Undefined, zero, small limits and `u32::MAX` keep their existing behavior. ## Validation Built the real Node native extension from main `0aa842b17f13d1f8ffd2851bc57c222bc5a376ba` with the repository's `pnpm build`, using locked dependencies and reusable build caches. No mocked/released native extension or disabled TypeScript diagnostics. With identical regression tests and that native library in fresh Jest processes: - Baseline: 40 passed, 8 failed; candidate: **48 passed**, no failures. - Both arms: `pnpm run tsc` and `pnpm run docs` passed. - Changed-file Biome check passed. Tests cover wrapping limits, invalid numbers, zero/small/u32 limits, defaults and both overload forms. Validation is the focused connection suite, not the complete repository suite. No performance claim. AI-assisted implementation and verification; the patch and exact tested source bytes were reviewed before publication. Signed-off-by: aden <aden@example.com> Co-authored-by: aden <aden@example.com> |
||
|
adenzhou1350
|
e256b3c2cf |
fix(python): honor namespace table storage options (#4483)
Follow-up on current main `7b3ba4c4` (merge commit `0b0475c1`): resolved the test import conflict, preserving both upstream imports and the original `warnings` import. The final PR diff is still limited to the same storage-option helper/call sites and regression class. AST comparisons preserve every upstream test definition and the original regression class/helper; both changed files compile and pass Ruff 0.15.1 check/format. Fresh locked-environment runtime validation was attempted with `uv run --locked --extra tests --extra dev pytest python/tests/test_namespace.py -q`, but dependency setup failed downloading the locked `pylance==14.0.0b12` wheel due a network connection error. No current-main native test result is claimed. Repository-wide Ruff reports 49 findings and 11 formatting differences in unchanged upstream docs/web-tests files; these were left outside this PR. CI approval and a human review are still pending. --- Original implementation and historical validation (before the main merge): Namespace-backed `create_table` accepts `data_storage_version` and `enable_v2_manifest_paths`, but neither sync nor async creation forwards them. The requested format/manifest settings are silently ignored. Normalize these deprecated arguments into the existing `new_table_*` storage options, matching ordinary connections. Explicit arguments take precedence; the caller's dictionary is not mutated. No Rust or native interface changes. Regression tests inspect persisted tables, including sync/async creation, populated/empty tables, stable/legacy formats, both manifest settings and explicit-option precedence. Validation on Linux/Python 3.12, based on current main `d3390a9`: - Rebuilt the exact-main native extension and reused that identical extension in separate baseline/candidate environments; only the candidate Python namespace wrapper differs. Dependencies came from the unchanged `uv.lock`. - Same complete `test_namespace.py`: baseline 16 regression failures and 39 passing controls; candidate **55 passed**, no skips/errors. - Ruff check/format and diff whitespace checks passed. This validates the local directory-namespace path, not a live Cloud service or the complete repository CI suite. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
13d54db594 |
fix(node): preserve caller-owned storage options (#4480)
`getStorageOptions` writes typed creation options back into the caller's
`storageOptions` object. This makes frozen options throw and causes
overrides to leak into later calls that reuse the storage map.
For example, using `{ storageOptions: shared, enableV2ManifestPaths:
true }` changes `shared.newTableEnableV2ManifestPaths` to `"true"`. A
subsequent creation with just `{ storageOptions: shared }` then uses V2
manifest paths even if `shared` originally requested `"false"`.
Merge the typed overrides into a local map instead. Their existing
precedence, key normalization, and undefined/default behavior remain
unchanged. This affects both `createTable` and `createEmptyTable`; no
native/Rust changes.
Adds six native-database regressions covering frozen creation options,
frozen nested storage options with conflicting defaults, and cross-call
reuse for both APIs.
Validation (Linux x64 / Node 24.19.0 / official 0.41.0-beta.0 native
library):
- Before: all six new regressions fail; an existing namespace
storage-options control passes.
- After: the full connection test file passes, 43/43 (current TypeScript
source, transpile-only).
- Same fix on the matching 0.41.0-beta.0 source: TypeScript compilation,
strict ts-jest connection tests (43/43), and TypeDoc with
warnings-as-errors pass.
- Biome on changed files and whitespace checks pass.
A current-main native rebuild was not run. Released native declarations
predate six unrelated lazy-iterator signatures; current-main
typechecking reports the same six errors before and after this fix (only
downstream line coordinates shift).
### Connection entry-point follow-up
The same ownership problem occurs in `connect(uri, options)` and
`connectNamespace`: normalization assigns a new storage map into the
caller's options. Copy the options before normalization instead. The
`connect(options)` overload already copies its input and is retained as
a control.
Nine added native-database cases cover frozen outer options
(with/without storage options) and repeated reuse, including table
creation and row-count checks. Before this entry-point fix, the
URI/namespace cases fail (6), while the object-overload controls pass
(3). Afterward the complete connection file passes 52/52 on current
TypeScript with the official 0.41.0-beta.0 native library. Applying the
entry-point fix to matching-release source also passes strict
TypeScript, strict ts-jest (52/52), and TypeDoc. Biome passes on both
newly changed files.
The same six current-main/released-native lazy-iterator declaration
errors remain unchanged. No Rust/native rebuild or GPU claim is added.
---------
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
|
||
|
adenzhou1350andXuanwo
|
036f4ec53f |
fix(node): preserve options when namespace is omitted (#4479)
The namespace overloads allow `undefined` to select the root namespace,
but overload dispatch treats it as the legacy options argument and
discards the trailing options. For example, `createTable(name, data,
undefined, { mode: "overwrite" })` raises an already-exists error
instead of overwriting.
Preserve the trailing options for `createTable`, `createEmptyTable`,
`tableNames`, and `listTables` when the namespace is omitted. Legacy
options-object calls and explicit namespace arrays keep their existing
behavior.
Adds native-database regressions for overwrite, existOk, and both
listing APIs, parameterized over `undefined` and `[]`.
Validation on Windows / Node 24.19.0:
- Before the fix: all five omitted-namespace cases failed; all five
explicit-root controls passed.
- Current TypeScript source with official, integrity-verified
0.41.0-beta.0 native binaries: the entire connection test file passed,
47/47 (transpile-only).
- The same fix replayed onto the matching 0.41.0-beta.0 source:
TypeScript compilation, strict ts-jest connection tests (47/47), and
TypeDoc passed.
- Biome check/format on both changed files and diff whitespace checks
passed.
No Rust changes. A complete current-main native rebuild remains for CI:
released declarations predate the unrelated lazy-iterator APIs, so
current-main typechecking against those declarations reports six
pre-existing iterator signature errors.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
LeonSGP
|
57a6cce681 |
fix(python): omit excluded model fields from persistence (#4465)
LanceDB serializes Pydantic rows with `model_dump()`, which omits `Field(exclude=True)`, but currently constructs an Arrow schema containing those same fields. Creating a local table therefore fails when an excluded field is non-nullable, including defaulted transient fields and nested models. Filter unconditionally excluded fields when converting model schemas, including nested structs. Skip excluded source/vector fields when deriving embedding configuration so persistent metadata does not refer to absent columns. Keep `field_names()` as model-field reflection. Tests exercise implicit and explicit schemas, synchronous and asynchronous create/add/reopen/readback, nested fields, defaults and default factories, retained fields, and local deterministic embeddings. Excluded values are not persisted; `to_pydantic` restores declared defaults. Conditional exclusion and readback of required excluded fields without defaults are outside this change. Implemented with assistance from gpt-6.1-sol; final diff reviewed locally. Local validation reuses the native extension matching official base |
||
|
LeonSGP
|
694942a1e0 |
fix(python): convert tuple and NumPy datetime update values (#4464)
`Table.update` and `AsyncTable.update` convert Python values into SQL literals. Tuple arrays and `numpy.datetime64` values currently raise `NotImplementedError`, even when the same values can be ingested into the corresponding local Arrow columns. Register tuples with the existing recursive list converter and format NumPy timestamps as complete nanosecond ISO strings with an explicit UTC offset. `NaT` becomes SQL NULL; subnanosecond units raise an explicit error rather than silently losing precision. Regression tests exercise local synchronous and asynchronous create/update/Arrow reads, date and timestamp units, nanosecond precision, NULL, and existing list/ndarray/Python datetime behavior. Nested and empty tuple arrays are compared with equivalent lists on the actual SDK: successful results must match, and the matching d864 native engine's existing list limitations must produce the same stable error without changing either table. This does not change SQL map handling. Implemented with assistance from gpt-6.1-sol; final diff reviewed locally. Local verification uses Python 3.11.14 and 3.13.11 in existing read-only development environments with the native extension matching base |
||
|
LeonSGP
|
554dc01f4d |
fix(python): preserve custom embedding field metadata (#4463)
`SourceField(json_schema_extra={...})` and
`VectorField(json_schema_extra={...})` currently raise `TypeError`
because the helper supplies the same `Field` keyword twice. Explicit
`None` also fails. This prevents users from adding ordinary Pydantic
JSON schema metadata to embedding fields.
Copy and merge the supplied dictionary before attaching the helper's
internal embedding identity. Caller dictionaries remain unchanged, and
reserved embedding keys retain the real function. Other `Field`
arguments keep their existing behavior. This change covers dictionaries
and `None`; callable `json_schema_extra` remains unsupported by the
existing embedding metadata reader and is outside this fix.
The 11 regression cases exercise both helpers, empty/custom/None
metadata, reserved-key collisions, other Field arguments, and a
deterministic local embedding model through JSON schema generation,
table creation, reopening, and search. On the original source, 9 fail
and 2 controls pass; the candidate passes all 11 on Python 3.11 and
3.13.
Validation uses read-only reused environments and the native extension
built for
|