mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-23 12:12:27 +00:00
566517be05ff39d3fc6445c5031abccceecf11eb
3258
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Raphael Fakhri
|
566517be05 |
fix(python): apply pyarrow expression filters in PyarrowDatasetAdapter (#4592)
`PyarrowDatasetAdapter` lets DuckDB query a LanceDB table, but DuckDB pushes `WHERE` clauses into `Dataset.scanner(filter=...)` as `pyarrow.compute.Expression` objects. The adapter forwarded them to `Table.count_rows` and `LanceQueryBuilder.where`, which accept only SQL strings, so every filtered DuckDB query failed with `TypeError: argument 'filter': 'Expression' object is not an instance of 'str'`. This change: - Applies a `pyarrow.compute.Expression` filter locally with `pyarrow.dataset.Scanner.from_batches`. The scan reads all columns and pyarrow applies the filter and projection, because DuckDB can filter on a column it does not project. - Applies `head` limits after the filter, so the limit counts matching rows. - Routes `PyarrowDatasetAdapter.count_rows(filter)` through the scanner, so it accepts both filter types. SQL string filters keep the existing path and still run inside LanceDB. Fixes #4591 ### Testing - New `test_dataset_adapter_expression_filter` in `test_pyarrow.py` covers `count_rows`, `to_table`, `head` and a projection that excludes the filtered column. - New `test_query_with_filter` in `test_duckdb.py` runs DuckDB queries with a `WHERE` clause. - Both tests fail without the fix and pass with it. `ruff check` and `ruff format --check` pass. |
||
 Captain
|
f2c6aa35c2 |
fix(python): normalize VoyageAI image modes before JPEG encoding (#4576)
PIL images, PNG bytes and local image paths in VoyageAI source embeddings are saved directly as JPEG. RGBA, grayscale-with-alpha (LA) and palette (P) images therefore raise `OSError: cannot write mode ... as JPEG` before the provider is called. Convert to RGB at all three JPEG encoding sites. The caller’s original PIL image remains unchanged. Document that alpha is discarded and callers should composite onto a chosen background when it matters. Validation: - New image regressions + existing VoyageAI tests: **18 failed / 26 passed before → 44 passed after**. Tests cover both multimodal models, PIL/bytes/Path, RGBA/LA/P/RGB, decoded JPEG pixels and invalid images. - Independent local LanceDB table ingestion smoke: three PNG modes stored with 1024-dimensional vectors, original bytes preserved. Uses the freshly built current native extension. - Adjacent local embedding tests: 15 passed, 1 skipped (no OpenAI key), 9 deselected. The separate Flickr download test failed with `RemoteDisconnected`; it is excluded from the local result. - Pyright: 0 errors; CI-pinned Ruff 0.9.9 lint and Python formatting passed. External VoyageAI calls are stubbed; real Pillow conversion and LanceDB ingestion execute locally. No live provider or full-repository test result is claimed. AI assistance was used for implementation and validation. |
||
 Srijan Keshri
|
118e606e8e |
fix(python): align vector parent indices after flattening null lists (#4572)
Arrow's `flatten()` skips child values hidden under null list parents, but `list_parent_indices()` can retain their indices. This makes `has_nan_values()` raise `ArrowInvalid` because the parent indices and NaN mask have different lengths. Filter out indices belonging to null parents before applying the mask. The regression test covers `ListArray` and `LargeListArray`, float16/32/64, and chunked inputs: 12 cases. NaN and null-component policies stay the same. The crash also affects #4503. This patch fixes it separately from that PR's changes to bad-vector policies. Validation: - All 12 regression cases failed before the fix. The utility suite now passes with 87 passed and 1 skipped. - Ruff lint and formatting pass for both changed files, as does `git diff --check`. - The patch applies cleanly to main at `ca275b6d`. The patched helper from that revision also passes all 12 cases. - A separate AI reviewer checked commit `343082a6` and found no blocking issues. They reran the utility suite and compared 4,059 generated cases plus 12 explicit null-parent cases with a row-by-row reference implementation. Those checks include slices, chunked inputs, and fixed-size lists. SDK tests used base `07e626ec` with the existing compiled extension. The broader Python run stopped after 474 passed and 23 skipped at `test_local_module_outside_code_warns_at_registration` with `ModuleNotFoundError`. That test also fails with the original vector helper restored. Repository-wide Ruff reports 49 lint errors and 11 files needing formatting, all in untouched documentation examples. Full SDK integration tests against current main require a newer native build and have not been run. |
||
|
Captain
|
0cb5bd0a5d |
fix(python): accept VoyageAI text batch containers (#4558)
Direct VoyageAI `compute_source_embeddings()` calls reject `list[str]` and NumPy text batches before reaching the provider, although both belong to the declared `TEXT` input type. For example, `["first", "second"]` raises `ValueError: Input type <class 'list'> not allowed with text model.` Normalize these containers to a list before the existing string-entry validation. The regression cases exercise the public embedding API with a stub provider: input order, list/NumPy batches, Arrow controls, empty batches, and rejection of non-text entries and ordinary two-dimensional arrays. The VoyageAI integration page now documents the accepted text containers. Verified on current main `7b3ba4c` with a locally compiled native extension (0.41.0b7, Python 3.12.12): - VoyageAI suite: 7 failed / 20 passed before the fix; 27 passed afterward. - Adjacent embedding suite: 13 passed, 4 skipped, 8 slow cases deselected. - Python CI's pinned Ruff 0.9.9 lint/format checks pass; configured Pyright reports zero errors/warnings; all applicable pre-commit hooks pass. The provider is stubbed; no live Voyage API request or model-quality evaluation was performed. AI disclosure: I used Codex for investigation, implementation, tests, review, and drafting this description. |
||
  Yash Raj PandeyandClaude Opus 5.5
|
2af7d88ec7 |
fix(python): keep OpenaiReranker scores on the rows they rate (#4549)
`OpenaiReranker._rerank` asks the chat model for a relevance score per document. The model can return the documents in any order, for example sorted by score. The reranker then replaces the `text` column with the documents in that order and appends the scores in that order. The other columns keep the input order. So after a rerank, the `text` and `_relevance_score` of a row belong to a different row. The `id`, the vector and every other column still describe the original row. If the model leaves out a document, `append_column` raises `ArrowInvalid`, because the new column has fewer rows than the table. ## Fix `_rerank` keeps the `text` column unchanged, so it also keeps its position and type. It maps each score back to its row by the document content. If the model leaves out a document, `_rerank` raises a `ValueError` that gives the number of documents with no score. ## Tests `python/python/tests/test_openai_reranker.py` replaces the chat client with a stub that returns a fixed reply. The tests do not need `openai` or an API key. - One test returns the documents sorted by score. It checks that each `id` keeps its own `text` and score. On `main`, the row with id 1 gets the text `cats purr`, which belongs to id 2. - One test returns 2 of 3 documents. It checks for the `ValueError` with the count of missing documents. On `main`, the call raises `ArrowInvalid: Added column's length must match table's length`. - Both tests fail on `main` and pass with the fix. - `test_rerankers.py`: 16 passed, 13 skipped, on the lancedb 0.40.0 wheel with and without the fix. The skipped tests need real API keys. - `ruff format --check` and `ruff check` (0.9.9) pass on the changed files. - I did not build the native extension. I ran the tests against the lancedb 0.40.0 wheel with the changed file copied over it. In that wheel, `rerankers/openai.py` and `rerankers/base.py` are byte-identical to `main`. The Node SDK has no OpenAI reranker, so there is no parity change. This change does not conflict with #4515, which edits `_client` in the same file. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Yash Raj PandeyandClaude Opus 5.5
|
353ce4d80c |
fix(python): give each JinaEmbeddings instance its own session (#4547)
`JinaEmbeddings` stores its `requests` session in a `ClassVar`. The first instance builds the session with its own API key, and every later instance reuses that session. So a second instance with a different `api_key` sends the first instance's key. ## Fix The session is now a per-instance `cached_property`, the same pattern `OpenAIEmbeddings` uses for `_openai_client`. `_init_client` and the `ClassVar` are gone. An instance with no `api_key` and no `JINA_API_KEY` now raises the missing-key error. Before, it reused the session of an earlier instance and sent that instance's key. ## Tests `test_jina_embeddings_use_their_own_api_key` creates two instances with different keys through the registry. It stubs `requests.Session.post` and records the `Authorization` header of each request. - On `main`, the test fails because both requests send `Bearer key-a`. With the fix, it passes. - `test_embeddings.py`: 18 passed, 8 skipped. - An instance still pickles after the session exists, and `safe_model_dump()` still returns only `api_key`. - On pydantic 2.7.4, the minimum version, a model with the same `cached_property` gets one session per instance and leaves it out of `model_dump()`. - `ruff format --check` and `ruff check` (0.9.9) pass on the changed files. - I did not build the native extension. I ran the tests against the lancedb 0.40.0 wheel with the changed file copied over it. In that wheel, `embeddings/jinaai.py` is byte-identical to `main` before this change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Yash Raj PandeyandClaude Opus 5.5
|
5fa30ae404 |
fix(python): honor explicit Cohere and VoyageAI reranker API keys (#4546)
`CohereReranker` and `VoyageAIReranker` read the environment variable before the `api_key` argument. When `COHERE_API_KEY` or `VOYAGE_API_KEY` is set, the reranker ignores an explicit `api_key` and uses the account from the environment. The docs say the opposite. `docs/web/integrations/reranking/cohere.mdx` and `voyageai.mdx` use the environment variable only when `api_key` is not provided. `JinaReranker` and `WatsonxReranker` already put the explicit key first, and #4515 makes the same change for `OpenaiReranker`. ## Fix `CohereReranker._client` and `VoyageAIReranker._client` now use `self.api_key` first and the environment variable as the fallback. ## Tests `python/python/tests/test_reranker_credentials.py` replaces the SDK module with a stub whose `Client` records the API key. It covers both keys set, each key alone, and no key. With the stub, the test runs without `cohere` or `voyageai` installed. - On `main`, the 2 cases with both keys set fail and the other 6 pass. With the fix, all 8 pass. - `test_rerankers.py` with the new file: 24 passed, 13 skipped (the skipped tests need real API keys). - `ruff format --check` and `ruff check` (0.9.9) pass on the changed files. - I did not build the native extension. I ran the tests against the lancedb 0.40.0 wheel with the changed files copied over it. In that wheel, both reranker files are byte-identical to `main` before this change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 Avanish Garg
|
bc4c32b779 |
fix: keep tagged versions when pruning in optimize (#4544)
`optimize()` fails once a tagged version is older than the cleanup cutoff, although the docs say tagged versions are exempt from cleanup, and the failed prune also skips the index-optimize step (repro and numbers in the issue). Change: `OptimizeAction::Prune` already exposes `error_if_tagged_old_versions: Option<bool>`, so no new API. `cleanup_old_versions` and `cleanup_old_versions_before` in `rust/lancedb/src/table/optimize.rs` now use `unwrap_or(false)`, which covers `OptimizeAction::Prune`, `OptimizeAction::All` and `optimize_prune_before`. The Python and Node bindings pass `None`, which now means "retain tagged versions"; I left `python/src/table.rs` and `nodejs/src/table.rs` untouched (this also avoids a conflict with #3870, which edits that region). `Some(true)` still opts in to the error. Doc comments updated; the sentence in `docs/web/management/table.mdx` is accurate once this lands. Tests: Rust `test_optimize_prune_retains_tagged_versions` (`Some(true)` still errors; the default retains the tag; `optimize_prune_before`; `checkout_tag` plus a data read), a Python sync test with a scalar index (index step runs, a second optimize works, untagged versions are pruned, `checkout("release-1")` works), a Python async test and a Node test. **What I could and could not verify (please read):** - I could NOT compile Rust here (not enough disk), so `cargo check`, `cargo test`, the Python/Node binding builds and therefore the fixed-path assertions of the new Rust, Python and Node tests have not been run. This is why the PR is a draft; I will mark it ready once CI is green and fix whatever it finds. - Ran: both new Python tests fail on the released 0.40.0 wheel with the exact error above (so they do catch the bug); pylance 13.0.0 confirms the semantics (`cleanup_old_versions(error_if_tagged_old_versions=False)` removes the untagged versions and keeps the tagged one, still readable with `lance.dataset(version="keep")`); in Lance v14.0.0-beta.7 `cleanup.rs` the policy field is `bool`, the builder default is `true`, and Lance's own `auto_cleanup_hook` uses `Some(false)`; `rustfmt --edition 2024 --check` passes on `optimize.rs`. - The Rust and Node tests rely on timestamp ordering (`nextMillisecond()`, `days(0)`/`now()`), so they could be flaky: tell me if you prefer another approach. Behaviour change: anyone who relied on the error to learn that a tag blocks cleanup now gets a silent retain. Not changed: the deprecated sync `LanceTable.cleanup_old_versions` calls pylance directly and still errors on tagged versions (the keyword may not exist in the oldest supported pylance); `freshness.rs:1377` and `refresh.rs:1077` still pass `None` (I did not check what those paths do with tagged versions). Fixes #4543 |
||
 Avanish GargandXuanwo
|
2d54630a3c |
fix(python): stop retrying permanent embedding errors and cap the backoff delay (#4542)
Permanent embedding errors (HTTP 400/404/422 and other 4xx) were retried the full `max_retries` times with an uncapped, multiplicatively jittered backoff, so a typo'd model name looked like a hang of about an hour (median 50 min of sleeping with a 404; see the issue for the repro and numbers). Changes (`python/python/lancedb/embeddings/utils.py`, `embeddings/base.py`): - An HTTP status in 4xx other than 408/409/429 is raised immediately. The status is read from `status_code`, `http_status`, `response.status_code`, an integer `code` (google-api-core, urllib) or a botocore `response["ResponseMetadata"]["HTTPStatusCode"]`. AWS throttling reported as 400 stays retryable (matched by error code). `AuthenticationError` by name is unchanged. - Still retried: 408, 409, 429, >=500, and anything without a recognisable status (connection errors, timeouts, custom exceptions), so existing custom embedding functions behave as before. - New keyword `max_delay=60` (last in the signature, default keeps call sites working). `delay = min(delay * base, max_delay)` with equal jitter `delay/2 + random() * delay/2`. With defaults the waits are [1,2], [2,4], [4,8], [8,16], [16,32], [30,60], [30,60] s, worst case about 182 s instead of up to 2.5 h. - The "Maximum number of retries" exception now has `from e`. The `max_retries` comment in `base.py` describes the rules. Not changed: `embeddings/openai.py` (it swallows `BadRequestError`; open #4154/#4005/#3152 cover it), Retry-After handling (the code never used it), and the "C Data interface error" wrapping from the native layer (the original message is now included in it). Behaviour changes to be aware of: every 4xx except 408/409/429 fails fast, including 401/403 coming through `response.status_code` (requests/httpx), which used to be retried; a provider that signals transient failures with a 4xx and no AWS-style code is no longer retried; the first retry waits 1-2 s instead of 2-4 s. Tests (`python/python/tests/test_embeddings.py`): 48 new cases (23 test functions): permanent errors are not retried, transient errors are, botocore-style errors, the delay is capped, exact exponential growth without jitter, jitter bounds, no float overflow for a large `max_retries`, the original error is attached, and end to end through `table.add` and `table.search` with a stub function (one call and no sleep). On `main` 23 of them fail with the change stashed; the rest are guards that must not change. `test_embeddings.py`: 61 passed, 4 skipped. `ruff format`/`ruff check` are clean on the touched files. Verification caveat: `main`'s Python layer needs a newer native module than the 0.40.0 wheel, so the tests were run with the wheel's native module preloaded through a small shim (the same shim ran the baseline on `main`); a wider run of embeddings/pydantic/query/table tests has the same 36 failures before and after (that mismatch). Not run: real provider APIs (a fake OpenAI-style server was used), Node or Rust code. Fixes #4541 Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 adenzhou1350
|
9f4cd1618b |
fix(python): handle empty batch iterables with a schema (#4538)
## Summary `create_table(data=iter(()), schema=...)` currently raises `RuntimeError: coroutine raised StopIteration`, although an empty list with the same schema works. A tuple of batches also fails because the helper calls `next()` before obtaining an iterator. Normalize the iterable once and use the supplied schema for an empty reader. Without a schema, raise a clear `ValueError`. Nonempty schema inference and batch casting are unchanged. Tests cover synchronous/asynchronous creation, empty/nonempty iterators and tuples, and schema-aware `add()` sanitization. The default schema-less `add()` rejection is deliberately unchanged. ## Validation - Built the current Rust/PyO3 extension from base `9f37f392` with the repository's `ci` profile; both Python-only arms use the same verified extension and locked Python dependency versions. - Complete `python/tests/test_table.py` + `python/tests/test_scannable.py`: **268 passed, 0 skipped**. The unchanged production code with identical tests has **24 failures / 244 passes**. - Both arms used four-CPU affinity. An initial unrestricted-affinity run exhausted thread creation under the private memory limit; that failed attempt is not counted as a pass. - Package-wide Ruff check/format pass. Repository-wide checks still report pre-existing findings in unchanged `docs/web-tests`; those files are not modified here. AI-assisted change, reproduced against unchanged production code and validated with a native source build. This does not claim the full repository/platform matrix or a performance improvement. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
 Christopher KrahandXuanwo
|
b6d15558ec |
fix(python): stop retrying a missing API key or provider package (#4521)
`retry_with_exponential_backoff` retries every exception except an auth
error (`python/python/lancedb/embeddings/utils.py:298-318`). For Cohere,
Gemini, Jina and VoyageAI a missing API key raises from
`api_key_not_found_help`, and a missing provider package raises
`ImportError` everywhere. Neither can succeed on a retry, yet both sleep
through all seven rounds and then surface wrapped in `Exception("Maximum
number of retries...")`.
Before, on `142d032`, a VoyageAI function with no key set (script below,
`time.sleep` recorded, `random.seed(0)`):
```
$ python lance_retry_deterministic.py python/python
raised: Exception('Maximum number of retries (7) exceeded.', ValueError('Please set the VOYAGEAI_API_KEY environment v
attempts: 8 sleeps: [4, 13, 37, 93, 280, 788, 2811]
slept 4026 s before a missing-API-key error surfaced
```
After:
```
$ python lance_retry_deterministic.py python/python
raised: MissingApiKeyError('Please set the VOYAGEAI_API_KEY environment variable.')
attempts: 1 sleeps: []
```
<details><summary>lance_retry_deterministic.py</summary>
```python
# loads the pure-Python embeddings package from the tree, native extension stubbed
import os, random, sys, time, types
TREE = sys.argv[1]
pkg = types.ModuleType("lancedb"); pkg.__path__ = [TREE + "/lancedb"]; sys.modules["lancedb"] = pkg
native = types.ModuleType("lancedb._lancedb"); native.__getattr__ = lambda name: None
sys.modules["lancedb._lancedb"] = native
import pyarrow as pa
from lancedb.embeddings import get_registry
random.seed(0)
os.environ.pop("VOYAGE_API_KEY", None)
slept = []
time.sleep = slept.append
func = get_registry().get("voyageai").create(name="voyage-3")
try:
func.compute_source_embeddings_with_retry(pa.array(["hello"]))
except Exception as e:
print("raised:", repr(e)[:110])
print("attempts:", len(slept) + 1, "sleeps:", [round(s) for s in slept])
print("slept %.0f s before a missing-API-key error surfaced" % sum(slept))
```
</details>
The message names the wrong variable: `voyageai.py:501` reads
`VOYAGE_API_KEY`. I left that for a separate fix.
`api_key_not_found_help` now raises `MissingApiKeyError`, a `ValueError`
subclass, so existing `except ValueError` callers still catch it. That
error and `ImportError` join the 401 exemption from #2995. A plain
`ValueError` is still retried, because `bedrock.py:171` wraps boto3
throttling in one. No issue exists for this. OpenAI is not covered: with
no key, `openai.OpenAI()` raises `OpenAIError`, which has no status
code, so it still retries. I can follow up on that.
### Verification
```
# lance_stub: an empty stand-in for lancedb._lancedb, loaded from the source tree
$ pytest -p lance_stub --noconftest python/tests/test_embeddings.py -k "retry or 401 or deterministic" -q
5 passed, 23 deselected in 0.59s
# utils.py from
|
||
 Aviral Saxena
|
d826b6115a |
fix(rust): explain client-side model files when remote tokenize fails (#4519)
Addresses option 2 of #4440. tokenize() builds the FTS analyzer on the client. For jieba/* and lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and when they’re missing you only got lance’s raw Invalid directory path: ... error, even though server-side search works. Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps the original cause and adds that tokenize() runs on the client, which directory the files need to be in, and that for remote tables they must match the server’s. I also updated the tokenize docs (Rust, Python, TS, generated JS docs) to say a mismatch can’t be detected, so tokens may differ from the server’s index. Behavior when the files exist is unchanged. Detecting a mismatch needs a server-side tokenize endpoint (option 1), so I left it out. The hint isn’t limited to remote tables, since Table::as_native is slated for removal and the wording fits local tables too. Tests: Rust test with mocked jieba/default index_details and an empty model dir, one checking other tokenizer errors don’t get the hint, and Python sync/async tests. |
||
|
adenzhou1350
|
93f054bcc5 |
fix(rust): finish query streams after a zero-duration timeout (#4506)
A positive query timeout emits one error and then ends, but `Duration::ZERO` leaves `TimeoutStream` in `NotStarted`, returning the same error on every subsequent poll. Transition the zero-duration path to `Completed` before returning its error. The initial error is unchanged. Extend both timeout tests to check that subsequent polls return `None`; the normal-completion test is unchanged. This affects callers that continue polling after an error, not `try_collect`, which already stops at the first error. Validation: - `cargo fmt --all` and `cargo fmt --all --check`: pass. - Mechanically extracted the unchanged implementation and three upstream timeout tests into a native Rust test binary using real cached Arrow 58.4.0, DataFusion 54.1.0, futures 0.3.32 and Tokio 1.53.1 libraries (matching the current lock). With the new assertions, baseline: 2 pass / 1 fail; fix: 3 pass. - Full-crate `cargo test --locked --offline --features remote -p lancedb --lib test_timeout_stream` could not start because this environment lacks the newly locked Lance `v14.0.0-beta.10` checkout. The component comparison is not a claim that the full Cargo suite passed; CI still needs to validate integration. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
3d883af674 |
fix(python): propagate invalid index wait_timeout errors (#4495)
Invalid `wait_timeout` values currently reach `Duration::extract(...).unwrap()` in both native index-creation bindings. A normal Python argument error becomes `pyo3_runtime.PanicException`, which is not caught by `except Exception`. Propagate the extraction error with `map(...).transpose()?` before constructing the index operation. Valid `None` and `timedelta` behavior is unchanged. Tests cover both `create_index` and `create_index_async`, three invalid types, valid `None`/`timedelta`, and no index created after rejection. Validation against main `0aa842b`, Python 3.12, uv-locked dependencies, freshly compiled PyO3 extension for each arm: - Unmodified source: 6 new regressions fail with `PanicException`; 6 controls pass. - Fixed source: 12/12 pass; broader index-creation regression selection: 18/18 pass. - `cargo fmt --all --check`, Ruff check/format, and diff check pass. The broader selection includes the 12 cases above. No cloud, full-suite, or performance claim. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
 adenzhou1350andaden
|
b205e7c628 |
fix(node): prevent table listing limits from wrapping (#4492)
## Summary Prevent large JavaScript table-listing limits from wrapping during the native `u32` conversion. With two tables, `limit: 2 ** 32` currently returns no tables and `2 ** 32 + 1` returns only one. Validate nonnegative safe integers using the existing helper and clamp valid limits above `u32::MAX` before calling the native binding. Apply the same normalization to `tableNames` and `listTables`, including namespace overloads. Undefined, zero, small limits and `u32::MAX` keep their existing behavior. ## Validation Built the real Node native extension from main `0aa842b17f13d1f8ffd2851bc57c222bc5a376ba` with the repository's `pnpm build`, using locked dependencies and reusable build caches. No mocked/released native extension or disabled TypeScript diagnostics. With identical regression tests and that native library in fresh Jest processes: - Baseline: 40 passed, 8 failed; candidate: **48 passed**, no failures. - Both arms: `pnpm run tsc` and `pnpm run docs` passed. - Changed-file Biome check passed. Tests cover wrapping limits, invalid numbers, zero/small/u32 limits, defaults and both overload forms. Validation is the focused connection suite, not the complete repository suite. No performance claim. AI-assisted implementation and verification; the patch and exact tested source bytes were reviewed before publication. Signed-off-by: aden <aden@example.com> Co-authored-by: aden <aden@example.com> |
||
|
adenzhou1350
|
e256b3c2cf |
fix(python): honor namespace table storage options (#4483)
Follow-up on current main `7b3ba4c4` (merge commit `0b0475c1`): resolved the test import conflict, preserving both upstream imports and the original `warnings` import. The final PR diff is still limited to the same storage-option helper/call sites and regression class. AST comparisons preserve every upstream test definition and the original regression class/helper; both changed files compile and pass Ruff 0.15.1 check/format. Fresh locked-environment runtime validation was attempted with `uv run --locked --extra tests --extra dev pytest python/tests/test_namespace.py -q`, but dependency setup failed downloading the locked `pylance==14.0.0b12` wheel due a network connection error. No current-main native test result is claimed. Repository-wide Ruff reports 49 findings and 11 formatting differences in unchanged upstream docs/web-tests files; these were left outside this PR. CI approval and a human review are still pending. --- Original implementation and historical validation (before the main merge): Namespace-backed `create_table` accepts `data_storage_version` and `enable_v2_manifest_paths`, but neither sync nor async creation forwards them. The requested format/manifest settings are silently ignored. Normalize these deprecated arguments into the existing `new_table_*` storage options, matching ordinary connections. Explicit arguments take precedence; the caller's dictionary is not mutated. No Rust or native interface changes. Regression tests inspect persisted tables, including sync/async creation, populated/empty tables, stable/legacy formats, both manifest settings and explicit-option precedence. Validation on Linux/Python 3.12, based on current main `d3390a9`: - Rebuilt the exact-main native extension and reused that identical extension in separate baseline/candidate environments; only the candidate Python namespace wrapper differs. Dependencies came from the unchanged `uv.lock`. - Same complete `test_namespace.py`: baseline 16 regression failures and 39 passing controls; candidate **55 passed**, no skips/errors. - Ruff check/format and diff whitespace checks passed. This validates the local directory-namespace path, not a live Cloud service or the complete repository CI suite. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
13d54db594 |
fix(node): preserve caller-owned storage options (#4480)
`getStorageOptions` writes typed creation options back into the caller's
`storageOptions` object. This makes frozen options throw and causes
overrides to leak into later calls that reuse the storage map.
For example, using `{ storageOptions: shared, enableV2ManifestPaths:
true }` changes `shared.newTableEnableV2ManifestPaths` to `"true"`. A
subsequent creation with just `{ storageOptions: shared }` then uses V2
manifest paths even if `shared` originally requested `"false"`.
Merge the typed overrides into a local map instead. Their existing
precedence, key normalization, and undefined/default behavior remain
unchanged. This affects both `createTable` and `createEmptyTable`; no
native/Rust changes.
Adds six native-database regressions covering frozen creation options,
frozen nested storage options with conflicting defaults, and cross-call
reuse for both APIs.
Validation (Linux x64 / Node 24.19.0 / official 0.41.0-beta.0 native
library):
- Before: all six new regressions fail; an existing namespace
storage-options control passes.
- After: the full connection test file passes, 43/43 (current TypeScript
source, transpile-only).
- Same fix on the matching 0.41.0-beta.0 source: TypeScript compilation,
strict ts-jest connection tests (43/43), and TypeDoc with
warnings-as-errors pass.
- Biome on changed files and whitespace checks pass.
A current-main native rebuild was not run. Released native declarations
predate six unrelated lazy-iterator signatures; current-main
typechecking reports the same six errors before and after this fix (only
downstream line coordinates shift).
### Connection entry-point follow-up
The same ownership problem occurs in `connect(uri, options)` and
`connectNamespace`: normalization assigns a new storage map into the
caller's options. Copy the options before normalization instead. The
`connect(options)` overload already copies its input and is retained as
a control.
Nine added native-database cases cover frozen outer options
(with/without storage options) and repeated reuse, including table
creation and row-count checks. Before this entry-point fix, the
URI/namespace cases fail (6), while the object-overload controls pass
(3). Afterward the complete connection file passes 52/52 on current
TypeScript with the official 0.41.0-beta.0 native library. Applying the
entry-point fix to matching-release source also passes strict
TypeScript, strict ts-jest (52/52), and TypeDoc. Biome passes on both
newly changed files.
The same six current-main/released-native lazy-iterator declaration
errors remain unchanged. No Rust/native rebuild or GPU claim is added.
---------
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
|
||
|
adenzhou1350andXuanwo
|
036f4ec53f |
fix(node): preserve options when namespace is omitted (#4479)
The namespace overloads allow `undefined` to select the root namespace,
but overload dispatch treats it as the legacy options argument and
discards the trailing options. For example, `createTable(name, data,
undefined, { mode: "overwrite" })` raises an already-exists error
instead of overwriting.
Preserve the trailing options for `createTable`, `createEmptyTable`,
`tableNames`, and `listTables` when the namespace is omitted. Legacy
options-object calls and explicit namespace arrays keep their existing
behavior.
Adds native-database regressions for overwrite, existOk, and both
listing APIs, parameterized over `undefined` and `[]`.
Validation on Windows / Node 24.19.0:
- Before the fix: all five omitted-namespace cases failed; all five
explicit-root controls passed.
- Current TypeScript source with official, integrity-verified
0.41.0-beta.0 native binaries: the entire connection test file passed,
47/47 (transpile-only).
- The same fix replayed onto the matching 0.41.0-beta.0 source:
TypeScript compilation, strict ts-jest connection tests (47/47), and
TypeDoc passed.
- Biome check/format on both changed files and diff whitespace checks
passed.
No Rust changes. A complete current-main native rebuild remains for CI:
released declarations predate the unrelated lazy-iterator APIs, so
current-main typechecking against those declarations reports six
pre-existing iterator signature errors.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
 LeonSGP
|
57a6cce681 |
fix(python): omit excluded model fields from persistence (#4465)
LanceDB serializes Pydantic rows with `model_dump()`, which omits `Field(exclude=True)`, but currently constructs an Arrow schema containing those same fields. Creating a local table therefore fails when an excluded field is non-nullable, including defaulted transient fields and nested models. Filter unconditionally excluded fields when converting model schemas, including nested structs. Skip excluded source/vector fields when deriving embedding configuration so persistent metadata does not refer to absent columns. Keep `field_names()` as model-field reflection. Tests exercise implicit and explicit schemas, synchronous and asynchronous create/add/reopen/readback, nested fields, defaults and default factories, retained fields, and local deterministic embeddings. Excluded values are not persisted; `to_pydantic` restores declared defaults. Conditional exclusion and readback of required excluded fields without defaults are outside this change. Implemented with assistance from gpt-6.1-sol; final diff reviewed locally. Local validation reuses the native extension matching official base |
||
|
LeonSGP
|
694942a1e0 |
fix(python): convert tuple and NumPy datetime update values (#4464)
`Table.update` and `AsyncTable.update` convert Python values into SQL literals. Tuple arrays and `numpy.datetime64` values currently raise `NotImplementedError`, even when the same values can be ingested into the corresponding local Arrow columns. Register tuples with the existing recursive list converter and format NumPy timestamps as complete nanosecond ISO strings with an explicit UTC offset. `NaT` becomes SQL NULL; subnanosecond units raise an explicit error rather than silently losing precision. Regression tests exercise local synchronous and asynchronous create/update/Arrow reads, date and timestamp units, nanosecond precision, NULL, and existing list/ndarray/Python datetime behavior. Nested and empty tuple arrays are compared with equivalent lists on the actual SDK: successful results must match, and the matching d864 native engine's existing list limitations must produce the same stable error without changing either table. This does not change SQL map handling. Implemented with assistance from gpt-6.1-sol; final diff reviewed locally. Local verification uses Python 3.11.14 and 3.13.11 in existing read-only development environments with the native extension matching base |
||
|
LeonSGP
|
554dc01f4d |
fix(python): preserve custom embedding field metadata (#4463)
`SourceField(json_schema_extra={...})` and
`VectorField(json_schema_extra={...})` currently raise `TypeError`
because the helper supplies the same `Field` keyword twice. Explicit
`None` also fails. This prevents users from adding ordinary Pydantic
JSON schema metadata to embedding fields.
Copy and merge the supplied dictionary before attaching the helper's
internal embedding identity. Caller dictionaries remain unchanged, and
reserved embedding keys retain the real function. Other `Field`
arguments keep their existing behavior. This change covers dictionaries
and `None`; callable `json_schema_extra` remains unsupported by the
existing embedding metadata reader and is outside this fix.
The 11 regression cases exercise both helpers, empty/custom/None
metadata, reserved-key collisions, other Field arguments, and a
deterministic local embedding model through JSON schema generation,
table creation, reopening, and search. On the original source, 9 fail
and 2 controls pass; the candidate passes all 11 on Python 3.11 and
3.13.
Validation uses read-only reused environments and the native extension
built for
|
||
|
LeonSGP
|
8552fd2f5b |
fix(python): prevent embedding aliases from replacing registered functions (#4462)
Registering a different embedding class under an existing alias silently replaced the original class. A deterministic real local SDK table reproduction changed the top result for the same query from alpha to beta after reopening the persisted table. Check the resolved key before inserting a class. Preserve repeated registration of the same class under the same nonempty explicit alias as idempotent, as existing SDK tests rely on it. Repeated implicit class-name registration, including an empty alias, still raises KeyError. Failed registration preserves the mapping and class metadata. Existing embedding tests receive isolated registry mappings; test_util.py remains unchanged. Eight regression cases cover different-class alias collisions, alias/class-name collisions, implicit and empty-alias duplicates, equal class names under distinct aliases, explicit-alias idempotence, and real local creation/reopening/search. This is independent of #4356, which fixes no-alias serialization. Validation on |
||
|
LeonSGP
|
e1e87e1cae |
fix(python): preserve logical rows from Hugging Face datasets (#4461)
Hugging Face `select`, `filter`, and `shuffle` can retain a physical
Arrow table while storing logical row selection/order in an indices
table. Ingesting the physical table then silently writes excluded rows
or the wrong order and can lose duplicate selections.
Read indexed datasets through bounded Arrow-format iteration, and keep
the existing direct batch path for datasets without indices. Apply the
same conversion to sync/async table creation and additions, including
DatasetDict splits. The conversion does not mutate the user's formatting
settings.
Regression coverage includes reordered/duplicate selections, filtering,
shuffling, empty selections, projected numpy/pandas/Arrow formatting,
DatasetDicts, rescanning, and bounded batches.
Validation on base
|
||
|
adenzhou1350
|
9c161869e1 |
fix(node): respect transformers tokenizer option (#4457)
## Summary `tokenizer` is a documented option, but initialization always passed the model name to `AutoTokenizer.from_pretrained`. Preserve the resolved tokenizer option, fall back to the model name when it is omitted, and identify the selected tokenizer in loading errors. Add five initialization regressions covering default model/tokenizer selection, an explicit tokenizer, registry variables, and tokenizer loading errors. They run in native Node child processes so the real ESM import works without changing Jest/CI flags. Model loaders are mocked and local/remote model access is disabled; these tests do not download models or test inference. ## Validation - Before: three regressions fail and two default-selection controls pass. After: all five pass. - New tests plus the existing registry and embedding-registry suites: 48 passed. - TypeScript compilation, changed-file Biome checks, and `pnpm run docs` pass; generated tracked docs are unchanged. - Checked with Node 24.14.1 / pnpm 11.1.1 on Linux, both with default Jest flags and the Linux CI ESM flag. The native binding and declarations came from the official `0.40.0-beta.13` packages, with registry integrity verified. Rebuilding Rust from source timed out during Cargo metadata resolution, before compilation; no Rust code is changed. |
||
|
adenzhou1350
|
7a3709b376 |
fix(remote): preserve request IDs in multipart response errors (#4456)
Keep the request ID when a multipart create/complete response has a missing or incorrectly typed `upload_id`/`version`. These errors currently use an empty ID, unlike JSON parse errors, so callers lose the ID needed to correlate the failed request. Add six cases through `Table.add(...).write_parallelism(2).execute()`, capturing the actual outgoing `x-request-id`: four missing/wrong-type cases fail on unchanged main, while two invalid-JSON controls already pass. After the fix all six pass, and all 15 multipart tests pass. Native Linux validation (locked dependencies, `ci` profile): - `cargo test --offline --profile ci --locked -p lancedb --no-default-features --features remote --lib remote::table::tests::test_multipart_write`: 15 passed. - `cargo fmt --all -- --check`: passed. - `cargo clippy --offline --profile ci --locked -p lancedb --no-default-features --features remote --lib --tests -- -D warnings`: passed. The committed source bytes match the tested source. HTTP endpoints are mocked; this does not claim a live Cloud or full-repository test run. |
||
|
adenzhou1350
|
a12aef2a8c |
fix: return a Python error when querying a closed table (#4455)
`AsyncTable.query()` unwraps the native closed-table error, so querying a closed table raises `PanicException` instead of the documented `RuntimeError`. `AsyncTable.head()` reaches the same path. Propagate `inner_ref()`'s error through `PyResult` instead. Add regressions for closed `query()` and `head()`, with `count_rows()`, repeated close, and normal open-table queries as controls. Validation against separately built native bindings: - Before: 2 regression failures (`PanicException`), 2 controls passed. - After: all 4 regressions passed. - Existing query tests plus the basic database test: 102 passed, 1 skipped on each arm (Torch is not installed). - `cargo fmt --all -- --check`, Ruff, and `git diff --check` passed. The native build used the locked Cargo dependencies with `remote,lancedb/metrics-otel,pyo3/extension-module`. This is source-tree native testing, not a complete wheel-packaging or cloud-service test. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
b9ef048abb |
fix: validate Python read consistency intervals (#4454)
Negative `read_consistency_interval` values currently reach Rust's
panicking `Duration::from_secs_f64` conversion. For example:
```python
from datetime import timedelta
import lancedb
lancedb.connect("./db", read_consistency_interval=timedelta(seconds=-1))
```
Validate the interval with `Duration::try_from_secs_f64` at the three
native connection entry points and return `ValueError` before
constructing the connection or async future. `None`, zero and positive
intervals retain their behavior. The raw native binding also rejects
NaN, infinity and overflow; the public Python API still accepts
`timedelta`, not floats.
Tests cover public sync/async connections, native namespace connections,
real Python namespace clients, the three raw binding entry points and
valid-value controls.
Validation on Python 3.12 / Rust 1.97:
- Same 26 regression cases against separately built native baseline and
candidate: baseline 22 failed / 4 passed; candidate 26 passed.
- Existing basic connection and background-loop regression tests pass.
- `cargo fmt --all -- --check`, Ruff check/format and `git diff --check`
pass.
The native builds used the unchanged repository `ci` profile and
Cargo.lock with `--no-default-features --features
remote,lancedb/metrics-otel,pyo3/extension-module`. Python runtime and
focused test dependency versions came from `python/uv.lock`. Source-tree
tests used a minimal distribution-version metadata fixture derived from
the unchanged Cargo/pyproject manifests; both imported module origins
and native versions were checked in fresh processes. This is not a
complete Maturin packaging, cloud backend or full Python suite result.
No GPU was used.
AI-assisted diagnosis, implementation and validation.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
|
||
|
LeonSGP
|
92ada619d2 |
fix(python): preserve metadata during vector schema inference (#4451)
Creating a table from an Arrow `list` or `large_list` vector field infers a fixed-size vector type, but currently drops that field's metadata. Descriptions and tags such as `lancedb:description` and `lancedb:tag:model` disappear even though the input field carries them. Preserve the original field metadata while replacing its inferred type. Tests use public synchronous and asynchronous table creation, reopening, and reads for floating-point and integer inference, checking the complete schema and row values. Existing dtype inference and nullability remain unchanged. Validation on macOS arm64 with an official native extension built from main `d864e719` using Rust 1.97.0: - Python 3.11.14 and 3.13.11: full `pytest -m "not slow and not s3_test" -q -rs python/tests`: **1,533 passed, 37 skipped, 68 deselected** each, wall times 331.464 s and 416.308 s. - Added public SDK cases on unchanged main: **12 failed**; after the fix: **12 passed** on each runtime. Final installed distributions passed **8 public SDK round trips** each, including fixed-size fields and fields without metadata. - Documentation tests on Python 3.11.14: **77 passed, 6 skipped**. Ruff lint/format, Pyright (**0 errors**), applicable pre-commit hooks (**4 passed, 2 without matching files**), compile checks and `git diff --check` passed. - Comparable unchanged-main full baselines from the same day, native extension and dependency versions: **1,521 passed, 37 skipped, 68 deselected** on each runtime. These were reused rather than rerun. The final local wheel is an explicitly verified repack of the official main wheel, replacing only `table.py` and `RECORD`; all production Python files match the tested source and the native extension is unchanged. No candidate Rust rebuild was performed. Slow/S3 integration tests, the remaining OS/runtime and minimum-dependency matrices, and the workflow's Linux/Python 3.13 doctest job were not run locally. The existing Rust migration draft #2970 and issues #3216/#3222 do not fix this metadata loss in the current Python path; a future migration should preserve the same metadata. During final verification main advanced to `81d52a98` through #4373 and the beta.13 version bump. Neither modified the two files here; the local native and full-suite validation above remain explicitly based on `d864e719`, and the updated native changes have not been tested locally. AI assistance: developed and verified with **gpt-6.1-sol**. |
||
|
LeonSGP
|
fcd26c425a |
fix(python): preserve integer Vector and MultiVector values (#4450)
When `Vector` or `MultiVector` declares an integer Arrow value type, Pydantic currently validates every element as a float. This silently rounds integers above `2**53`, and converts `uint64`'s maximum value into an out-of-range float before table insertion. Use Pydantic's integer schema for integer Arrow value types in both validators. Floating-point types retain the existing validation. Tests exercise model serialization and real local table creation, insertion, and reads for signed and unsigned integer boundaries and floating-point controls, plus dimensions and optional/nullability behavior. Fractional floats are rejected for integer declarations; integral floats remain accepted. This does not extend ANN index support. Validation on macOS arm64, with the current main native extension built using Rust 1.97.0: - Python 3.11.14 and 3.13.11, Pydantic 2.13.5: full `pytest -m "not slow and not s3_test" -q python/tests` suite: **1,537 passed, 37 skipped, 68 deselected** on each runtime. Comparable unchanged-main baselines: **1,521 passed, 37 skipped, 68 deselected** each. - Added cases on unchanged main: **8 failed, 8 passed**; complete `test_pydantic.py` after the fix: **40 passed** on each runtime. - Documentation tests with the existing embeddings dependencies in an isolated environment, Python 3.11.14: **77 passed, 6 skipped**, matching the unchanged-main baseline. The workflow's Linux/Python 3.13 doctest matrix was not reproduced locally. - Repository Ruff lint/format, Pyright, all applicable pre-commit hooks, compile checks, and `git diff --check` passed. Eight public SDK table round trips passed on each final installed wheel runtime. Slow, credential-dependent S3 integration tests, the remaining OS/runtime matrix, and the minimum Pydantic/PyArrow dependency matrix were not run locally. The native extension was built from unchanged current main; the final local wheel repack contains the verified Python change with that identical native extension. A separate candidate maturin packaging attempt triggered Rust recompilation and was stopped, so an official candidate distribution build was not completed. The project requires Pydantic `>=2.7.4,<3`; Pydantic v1 support was removed in #3990. AI assistance: developed and verified with **gpt-6.1-sol**. |
||
|
LeonSGP
|
1e313d3c0d |
fix(python): parse decimal exponent literals exactly (#4449)
Decimal literals such as Decimal("1E-7") and Decimal("1E+3") currently
raise ValueError while building an expression. Parse the coefficient and
exponent exactly, and normalize representable negative scales to zero so
both typed filters and predicate.to_sql() queries preserve the value.
Checked arithmetic bounds the Decimal128 coefficient to 38 digits,
preserves existing scale limits, and rejects nonfinite or overflowing
values. This does not add Decimal256 literal support.
Added 21 cases covering actual Arrow-backed table queries, SQL queries,
zero, signed exponents, precision boundaries, and errors. Both Python
3.11 and 3.13 expression suites pass 123 tests; the baseline fails 10 of
the new cases. Python binding Rust tests pass 13; official make doctest
passes 77 with 6 skips; Ruff, Rustfmt, hooks, Pyright and
workspace/tests/all-features Clippy with -D warnings pass. A fresh
environment restored from the original dependency freeze passes the full
required suite: 1528 passed, 37 skipped, 68 deselected (143.51s); while
optional Torch suites have the same baseline remote mock-server ordering
failure.
Tested against f4544628; observed official main
|
||
  Preston ZhaoandPreston Zhao
|
13df747f18 |
fix(python): return the session supplied to LanceDBConnection (#4433)
LanceDBConnection.session read AsyncConnection.session, but AsyncConnection has no session attribute, so access raised AttributeError. Store the session passed to the constructor and return that object. Assisted by Cursor (Grok). Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com> |
||
|
Preston ZhaoandPreston Zhao
|
cc9f01ae26 |
fix(node): accept exist_ok in createTable (#4432)
TypeScript CreateTableOptions.mode omitted exist_ok even though createTable already forwards that mode to the native client. closes: #2900 --------- Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com> |
||
|
adenzhou1350
|
e489d0cd7b |
fix: return errors for unsupported remote alteration types (#4430)
RemoteTable::alter_columns currently unwraps the JsonDataType
conversion. A valid Arrow input that the remote wire format cannot
represent (for example Utf8View, BinaryView, or a struct containing
Utf8View) therefore panics instead of returning an error.
Propagate conversion/serialization failures as Error::InvalidInput,
including the column path. Unsupported inputs are rejected before any
HTTP request; supported alteration payloads are unchanged. This does not
add server support for view types or change server HTTP-error
classification.
Three new native Rust cases reproduce the panic on unchanged production
and return InvalidInput with the fix. The existing supported-type cases
remain passing.
Validation at head
|
||
|
adenzhou1350
|
4bdc8c3c00 |
fix: wait for the submitted remote index job (#4428)
Addresses the client-side waiting portion of #4402. When create_index returns a job ID, wait_timeout now waits for that submission rather than an already-ready index with the same name. It propagates JobFailed / JobCancelled and bounds job polling by the requested timeout. Older servers without job IDs keep name-based waiting, calls without wait_timeout remain fire-and-forget, and the existing two-hour maximum is preserved. The job wrapper is shared with create_index_async; no synchronous server-side config validation is added. Rebased on main |
||
 Mohammad Hijjawi
|
712a8b8699 |
fix(python): write plain Enum fields of a LanceModel as their values (#4357)
Since #3232, `pydantic_to_schema` maps an `Enum` field to the Arrow type of its values, including enums without a `str`/`int` mixin. Writing such models still failed: `model_to_dict()` returns `model.model_dump()`, which keeps the Enum members, and Arrow can only convert members that are also `str`/`int` instances. ```python class Color(Enum): RED = "red" class Item(LanceModel): color: Color tbl = db.create_table("items", schema=Item) tbl.add([Item(color=Color.RED)]) # pyarrow.lib.ArrowTypeError: Expected bytes, got a 'Color' object ``` `create_table(data=[...])` fails the same way, and int-valued enums raise `ArrowInvalid`. `model_to_dict()` now replaces Enum members with their values, including in nested models and lists. Lists of plain scalars (vectors in particular) are returned without being walked, so the per-row cost stays about the same as `model_dump()` alone (21.3µs vs 20.3µs for a 1536-dim model). Reading back is unchanged: Pydantic already validates the stored value into the Enum member. Added `test_plain_enum_values_are_written`, which covers str and int valued enums, `Optional` and `List` fields, and an Enum inside a nested model, and checks that `to_pydantic` round-trips the rows. It fails with the `ArrowTypeError` above before the change. `test_pydantic.py`, `test_embeddings.py`, `test_table.py` and `test_db.py` pass, and `ruff format --check` / `ruff check` are clean. |
||
|
Mohammad Hijjawi
|
5f9a8af27c |
fix(python): keep embedding functions registered without an alias readable (#4356)
`EmbeddingFunctionRegistry.register()` documents that the class name is
used when no alias is given, and it does register the class under that
name. But it stored the alias itself (`None`) as
`__embedding_function_registry_alias__`, which is what
`function_to_metadata()` writes into the table schema. A table using
such a function gets `"name": null` in its `embedding_functions`
metadata, and any later read of the functions fails:
```python
@get_registry().register()
class MyEmbed(TextEmbeddingFunction): ...
f = MyEmbed.create()
class Schema(LanceModel):
text: str = f.SourceField()
vector: Vector(4) = f.VectorField()
tbl = db.create_table("t", schema=Schema)
tbl.add([{"text": "hi"}]) # KeyError: None
tbl.search("hi") # KeyError: None
```
This stores the key the class is registered under instead, so the class
name round-trips. For aliased classes the key is the alias, so their
metadata is unchanged. The TypeScript registry already falls back to
`ctor.name` in the same way.
Added `test_embedding_function_registered_without_alias`, which creates
a table with such a function, reopens it, and adds a row. It fails with
`KeyError` before the change. `test_embeddings.py`, `test_pydantic.py`,
`test_table.py` and `test_db.py` pass, and `ruff format --check` / `ruff
check` are clean.
|
||
|
Mohammad HijjawiandXuanwo
|
a0946282ca |
fix(python): return original scores from hybrid search with normalize="rank" (#4343)
## Summary With `rerank(..., normalize="rank")`, a hybrid query returns ranks (1.0, 2.0, 3.0, ...) in the `_distance` and `_score` columns instead of the real vector distances and BM25 scores. `_combine_hybrid_results` converted both result sets to ranks first, and only then captured `original_distances` / `original_scores`. So the step that restores the original values after reranking put the ranks back. With `normalize="score"`, the real values are returned as expected. Repro (RRFReranker(return_score="all"), vector distances 0 / 25 / 100): | normalize | `_distance` returned | `_score` returned | | --- | --- | --- | | `"score"` | 0.0, 25.0, 100.0 | 0.47, 0.67, None | | `"rank"` (before) | 1.0, 2.0, 3.0 | 1.0, 2.0, None | | `"rank"` (after) | 0.0, 25.0, 100.0 | 0.47, 0.67, None | ## Changes - Capture the original `_distance` / `_score` columns before converting them to ranks, then normalize the ranked column. The reranker input is unchanged; only the values restored afterwards are fixed. `AsyncHybridQuery` uses the same function, so both sync and async are covered. ## Test plan - [x] New `test_hybrid_return_all_restores_original_scores` in `test_rerankers.py`, parametrized over `score` / `rank`: the `rank` case fails before the fix and passes after - [x] `pytest python/tests/test_rerankers.py python/tests/test_hybrid_query.py` - [x] `ruff format --check` / `ruff check` I have no local Rust toolchain, so I ran the tests against the released 0.39.0 native module with the changed Python files overlaid. The changed functions are identical between 0.39.0 and main. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
62d83f2ea2 |
fix(python): expect oidc credential-type header on refreshed tokens (#4632)
## Summary #3971 added `test_zero_lifetime_token_is_refreshed`, but its expected headers omit `x-lancedb-credential-type: oidc`, which `OAuthProvider.get_headers()` has emitted since #2631/#4070. The test fails against current main and breaks the PR CI matrix for every open PR. ## Changes - Align the two zap assertions in `test_zero_lifetime_token_is_refreshed` with the provider contract (see `test_token_refresh` and `test_sync_method`). ## Tests - `pytest python/tests/test_header_provider.py -q` |
||
 Prince Jain
|
7f932b8552 |
fix(nodejs): reject listNamespaces limit above i32::MAX instead of wrapping (#4584)
## Summary Fixes #4583. `Connection::list_namespaces` narrowed `Option<u32>` to `i32` with an unchecked `as` cast, so a JavaScript `limit` between 2^31 and 2^32 - 1 (e.g. `3000000000`) wrapped negative before reaching `ListNamespacesRequest`. The server received a negative page size instead of a clear client-side error. Replaced the cast with a `checked_limit` helper using `i32::try_from`, returning `napi::Error::from_reason` on overflow, matching the existing `try_from` validation used for `limit` elsewhere in the same file. ## Tests Unit tests cover the wrap case (`u32` values above `i32::MAX` reject), the boundary (`i32::MAX` passes), and `None`/zero passthrough. `cargo test -p lancedb-nodejs` passes 8/8 and `pnpm build` compiles the binding and TypeScript definitions cleanly. |
||
 Yingzi Zhang
|
55897d7c3b |
fix(python): preserve Arrow schema when merging reranker scores (#4332)
Fixes #4331 With `return_score="all"`, model rerankers reconstruct merged rows without their Arrow schema. This changes fixed-size float32 vectors into variable-length float64 lists, loses metadata, and can overflow valid uint64 row IDs. Unify the original input schemas before inserting placeholder scores, then use the unified schema when reconstructing the merged table. Row merging and reranking behavior stay the same. Capturing the schema first also preserves float64 scores without conflicting with the float32 null placeholders. Validation: - New regression cases: 5 failures and 5 passing controls before the fix; all 10 pass afterward. - Reranker, hybrid-query and TypeSafe reranker tests: 150 passed, 17 skipped. Optional live model/API tests were not run. - The fake-client tests that use random metadata passed again with seeds 7 and 41. - Repository-wide Ruff lint/format checks, configured Pyright check and applicable pre-commit hooks pass. - Tests used the locally built Rust/Python extension and the repository's locked uv environment. No Rust code changed. AI disclosure: I used Codex for investigation, implementation, tests, local checks, and drafting this description. |
||
 lancedb-gatefixer[bot]
|
624defd109 |
fix: preserve sliced child validity during struct merge (#4261)
## Root cause The pinned Lance Arrow crate rebuilt a sliced struct child with its original Arrow offset but supplied a validity bitmap containing only the visible rows. Arrow requires the bitmap to cover the offset as well, so filtered vector searches involving nullable nested fields could panic with `null_bit_buffer size too small`. ## Fix - Carry an offset-aligned validity bitmap when propagating parent nulls to a sliced child. - Vendor the pinned `lance-arrow` crate until a Lance release includes the fix; include it in Maturin source distributions. - Add unit regressions for sliced children and parent bitmaps, plus a record-batch merge regression. ## Validation - The issue's 8,600-row Python reproducer completes its filtered flat dot search and returns 8,448 rows without a panic. - The targeted Rust merge regression and two focused Python query tests pass. - Rust formatting, Ruff formatting and lint, the vendored crate tests, and the workspace Cargo checks passed. The Maturin source archive includes the patched crate. Fixes #4260 <!-- lance-gatekeeper-fix:v1 agent=e6045f701d9b708ee26528ba36353aba generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
66b5b5bbdc |
fix(python): clarify unavailable CUDA index acceleration (#4249)
## Summary - preflight CUDA availability and initialization before accelerated index training - direct unavailable or misconfigured CUDA environments to the explicit CPU path - cover both legacy and unified Python index APIs ## Root cause LanceDB delegated accelerator requests directly to PyLance. When PyTorch lacked CUDA support or CUDA initialization failed, users received a low-level assertion or runtime error with no indication that omitting `accelerator` or setting `accelerator=None` creates the index on CPU. The repair validates CUDA before training and preserves explicit accelerator semantics without silently changing the requested backend. ## Scope This PR addresses only the separately reproduced CUDA availability and initialization error. It does not close #1766; the reported query failure after successful CUDA index creation remains open pending reproducible evidence. ## Validation - `uv run --extra tests pytest python/tests/test_table.py -k create_index -q` (10 passed) - `ruff format .` - `ruff check .` - CPU-only integration check confirms the actionable fallback message Related to #1766 <!-- lance-gatekeeper-fix:v1 agent=d05c37237b04c2c8f88395a82e00bc67 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 SIDDARTHA REDDY
|
d37a6b76a4 |
fix: gate remote job error construction behind the remote feature (#4233)
## Summary Fixes #4142: `lancedb` failed to compile without the `remote` feature. `TerminalResult::decode` in `rust/lancedb/src/job.rs` constructed the `Error::Http` variant unconditionally, but that variant is declared behind `#[cfg(feature = "remote")]` in `rust/lancedb/src/error.rs`. Since the crate's default feature set is empty, a plain `lancedb = "0.38"` dependency failed with: ``` error[E0599]: no variant named `Http` found for enum `error::Error` --> rust/lancedb/src/job.rs:102:40 ``` The module itself is not remote-only (it carries local job support), so instead of gating `pub mod job`, both `Error::Http` constructions now go through a small `remote_job_error` helper that resolves to `Error::Http` with the `remote` feature and falls back to `Error::Runtime` without it. A `request_id` is only ever `Some` for remote jobs, so the fallback is unreachable in practice without the feature. ## Testing - Reproduced the reported failure on current `main`: `cargo check -p lancedb --no-default-features` failed with the exact E0599 errors from the issue before the change, and passes after it. - `cargo check -p lancedb --features remote` passes with no errors and no new warnings. - Added two regression tests in `rust/lancedb/src/job.rs` (`decode_missing_result_uses_feature_appropriate_error`, `decode_unparsable_result_uses_feature_appropriate_error`) asserting the error variant matches the feature configuration. The test code compiles under the fixed configuration (`cargo check --tests` passes), and the branching logic was executed standalone against both feature cfgs (3/3 green with and without `remote`). The full `cargo test` binary link could not complete on the small build VM available here (the linker stalled), so CI coverage of the new tests is appreciated. - `cargo fmt --all -- --check` is clean. ## Disclosure AI assistance was used while preparing this change (code search, editing, and verification commands were agent-driven); the diagnosis, fix design, and all test results above were verified by actually running the builds. |
||
 Joaquin Hui Gomez
|
82bfe7d38e |
fix: table_names should not list loose .lance files as tables (#4203)
`table_names()` enumerates with `read_dir`, which returns files as well
as directories, filtered only on the `.lance` extension — so a loose
`foo.lance` *file* at the database prefix is reported as a table.
`list_tables()` already applies the correct rule ("Only child
directories can be tables"), and
`test_listing_ignores_non_table_children` documents it, so the two
listing APIs currently disagree.
The practical damage is in `drop_all_tables()`, which is built on
`table_names`: with a loose `.lance` file present it deletes some real
tables, fails on the file with `os error 20 (NotADirectory)`, and then
fails the same way on every retry — the database can never be emptied
through the API, and the partial drop is not recoverable.
`table_names` now lists via `list_with_delimiter` and filters
`common_prefixes` through the same `table_name` helper `list_tables`
uses; the sort/`start_after`/`limit` handling is unchanged.
`drop_tables` additionally attempts every name before returning, still
reporting the first error unchanged (kept rather than an aggregate error
so the mapped exception types in the bindings stay exactly as they are).
Extended the non-table-children test to cover `table_names`, plus tests
for the drop behavior — all three fail without the fix. Full `cargo test
-p lancedb --lib --features remote` 1320 passed on the repo-pinned 1.97
toolchain; clippy/fmt clean.
|
||
 陈志谦
|
9a59091376 |
fix(python): add missing space in the hybrid search error message (#4202)
`LanceHybridQueryBuilder._validate_query`'s ValueError concatenated two adjacent string literals without a separator, rendering "...explicitly for hybrid search.**But** not both.". Added the trailing space. String-only change. |
||
|
陈志谦
|
d4b2290bab |
fix(python): correct the voyageai reranker env var docstring (#4201)
`VoyageAIReranker`'s `api_key` docstring said it falls back to `OPENAI_API_KEY`, but the class actually reads `VOYAGE_API_KEY` (and raises a "VOYAGE_API_KEY not set" error without it) — the docstring was copy-pasted from the OpenAI reranker. Docstring-only fix. |
||
 Mohd Quamar TyagiandXuanwo
|
79e256959f |
fix(python): accept pathlib.Path in get_uri_scheme and get_uri_location (#4132)
This PR converts the uri parameter to string before calling urlparse in get_uri_scheme and get_uri_location, allowing pathlib.Path objects to be passed seamlessly without raising a TypeError. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Madan Kumar
|
0c3a1c92c8 |
fix(python): don't crash inferring schema for an all-null vector column (#4034)
Fixes #4033 `create_table(data=...)` without an explicit schema raised `IndexError: index out of bounds` when a `vector`/`embedding`-named list column was all-null. `_modal_list_size` infers the dimension with `pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips nulls, so an all-null column produces a zero-length result and the `[0]` subscript blows up. The sibling `_infer_vector_dim` already guards the identical `pc.mode(...)` with a `len == 0` check and returns `None`; `_modal_list_size` just didn't. This makes `_modal_list_size` return `Optional[int]` and return `None` when every value is null, and the call site then leaves the column as a variable-length list instead of inventing a fixed size — the dimension is genuinely unknowable when there are no non-null rows. Added a regression test that creates a table from an all-null `list<float32>` vector column and asserts it succeeds, the column stays a `list<float32>`, and the rows are queryable. It fails on `main` with the `IndexError` and passes with this change. |
||
 Vinayak Vathare
|
ae97d43f4b |
fix(python): preserve valid embeddings when an OpenAI batch is rejected (#4005)
Retries individual inputs one at a time when the OpenAI API rejects a batch with BadRequestError, instead of discarding the whole batch. Valid inputs now still get embeddings; only the actually-bad input resolves to None, and the log names the exact failing input. Fixes #1677 |
||
 Zhewen TanandXuanwo
|
6d92307fc3 |
fix(python): accept tuple and Arrow vector queries (#3972)
## Summary Sync `Table.search()` declares tuple, `pyarrow.Array`, and `pyarrow.ChunkedArray` query vectors as supported, and the async query path already accepts them. The sync query builder only classified lists and NumPy arrays as vectors, however, so these inputs were routed through embedding/FTS resolution and then rejected. Arrow and tuple inputs also did not contribute a query dimension when selecting among multiple vector columns. This change: - recognizes the supported tuple and Arrow containers as sync vector queries for both `auto` and explicit `vector` query types; - converts them to the existing internal `float32` NumPy representation at the sync builder boundary; - infers dimensions from tuple, primitive Arrow, and list-valued Arrow queries so schemas with differently sized vector columns select the matching column. The regression test uses two vector columns with dimensions 2 and 3. It verifies tuple, Arrow array, and chunked Arrow queries through both sync query modes and confirms the dimension-2 column is selected. ## Test plan - `pytest python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers -q` (6 passed) - `pytest python/tests/test_util.py python/tests/test_query.py python/tests/test_table.py -q -m "not slow and not s3_test"` (308 passed, 1 skipped, 1 deselected) - `pytest python/tests -q -m "not slow and not s3_test" --ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected) - `ruff format --check python` - `ruff check python` - `cargo fmt --all --check` Co-authored-by: Xuanwo <github@xuanwo.io> |