mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-29 06:56:15 +00:00
dcac834233ded0d992affd67745d1a824a52dcdf
3314
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Jay PokaleandXuanwo
|
dcac834233 |
fix(python)!: reject tuple fields with mixed element types in pydantic models (#4679)
`pydantic_to_schema` maps a `Tuple[...]` field to an Arrow list of its
**first** element type. A tuple with different element types gets the
wrong schema, and the error only surfaces when a row is written:
```python
class Pair(LanceModel):
pair: Tuple[int, str]
pydantic_to_schema(Pair) # pair: list<item: int64>
db.create_table("t", schema=Pair).add([Pair(pair=(1, "a"))])
# pyarrow.lib.ArrowInvalid: Could not convert 'a' with type str: tried to convert to int64
```
An Arrow list has a single element type, so such a tuple cannot be
stored as one. This raises a `TypeError` when the schema is built,
saying the tuple elements must share one type and pointing to a nested
`LanceModel` for a fixed set of differently typed fields. Homogeneous
tuples are unchanged: `Tuple[int, int]` and `Tuple[float, ...]` still
map to lists.
**Breaking change:** a model with a mixed-type tuple field, which used
to get a (wrong) `list<...>` schema from `pydantic_to_schema`, now
raises `TypeError` when the schema is built. Such a schema could not
hold the model's values: writing a row failed with `ArrowInvalid`. The
requirement and the nested-`LanceModel` alternative are documented in
the `pydantic_to_schema` docstring.
Tests (in `test_pydantic.py`): `TypeError` for `Tuple[int, str]` (fails
on `main`), and the list mapping for `Tuple[int, int]` and `Tuple[float,
...]`. The whole file passes; `ruff format` and `ruff check` are clean.
I used an AI assistant (Claude) while working on this; I reviewed and
tested the change.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
 陈志谦andXuanwo
|
3575f972ac |
fix(python): accept tuples and pyarrow arrays as hybrid search vectors (#4662)
The synchronous hybrid query builder rejected vector containers that the
rest of the Python API accepts:
```python
table.search(query_type="hybrid").vector((1.0, 2.0)).text("hello").to_pandas() # ValueError
table.search(query_type="hybrid").vector(pa.array([1.0, 2.0], pa.float32()))...
# ValueError: Vector query must be either a string or a vector
```
`lancedb.common.VEC` documents `list[float] | tuple[float, ...] |
np.ndarray | pa.Array`, and the same two containers work for
`table.search(vector)` and for the async hybrid API
(`AsyncQuery.nearest_to(...).nearest_to_text(...)`).
`LanceHybridQueryBuilder._validate_query` only listed `str | list |
np.ndarray`, so a tuple or a pyarrow array was refused before reaching
the query planner.
This normalises the vector exactly the way `LanceQueryBuilder.create`
already does — list/tuple to a `np.float32` array, `np.ndarray` through
`astype(float32)`, `pa.Array`/`pa.ChunkedArray` through `np.array(...,
dtype=float32)`, strings passed through — and keeps the existing
`ValueError` for anything else.
`python/python/tests/test_hybrid_query.py::test_hybrid_query_accepts_every_vector_container`
is parametrised over tuple, pyarrow array and pyarrow chunked array, and
asserts the returned `_rowid`s match the `list` form: 3 failed on `main`
(`ValueError` from `query.py:2452`), 3 passed with the change.
Neighbouring suites (`test_hybrid_query.py`, `test_query.py`,
`test_rerankers.py`) are 220 passed / 9 skipped, and `test_table.py`,
`test_remote_db.py`, `test_fts.py`, `test_index.py`,
`test_permutation.py` are 724 passed / 5 skipped. `ruff` at the version
pinned in the repository's pre-commit config (0.9.9) reports no findings
and `ruff format --check` is clean.
One source file (`python/python/lancedb/query.py`), no behaviour change
for containers that already worked.
Prepared with AI assistance; every line was verified by running the
cases above.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
 lancedb-gatefixer[bot]andXuanwo
|
2cd635d41c |
fix: correct remote materialized view refresh results and options (#4672)
Remote materialized-view refreshes drained the SQL result and then fabricated `mode=rebuild`, `rows_written=<total view rows>`, and `source_version=0`. This made incremental refreshes report incorrect statistics. Make these three result fields optional across Rust, Python, and TypeScript, and leave them unset for remote SQL refreshes. Remove the post-refresh row count request. Local refreshes continue to populate all metrics, and `version` remains the view version observed after completion. Remote per-operation statistics remain unavailable until the SQL job adapter can retrieve them. Expose the forced-rebuild option through the Rust builder, synchronous/asynchronous Python refresh and job methods, and TypeScript refresh. Remote `full=true` sends `WITH (full = true)`; the default lets the server choose its refresh mode. Local source-version pinning and remote job cancellation are preserved. Local refresh regression tests respect the `experimental-functions` build gate, while remote refresh coverage also runs in default builds. Validation: - Rust: 20 materialized-view tests and 21 remote SQL tests passed with `remote,experimental-functions`; `cargo fmt --all` passed. - Python: rebuilt the extension in the uv-managed environment. The default build passed 23 tests with 27 feature-dependent skips. The opt-in build passed 61 tests with 3 expected default-build-only skips, covering the materialized-view suite and 14 async query-serialization cases from current main. Python Ruff lint and formatting checks passed. - TypeScript: default and opt-in native builds and TypeScript compilation passed. The default build passed 5 tests with 10 expected feature-dependent skips; the opt-in build passed all 14 materialized-view tests with the default-build-only test skipped. Lint/format checks and documentation generation passed. - Repository-wide `ruff check .` reports 48 pre-existing violations in unchanged `docs/web-tests` files; `ruff check python` passes. Fixes #4670 <!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
c167b99fa0 |
fix: read legacy blob columns through blob APIs (#4649)
Legacy blob v1 columns were rejected by LanceDB's blob validation and omitted from `blob_columns()`, even though the pinned Lance reader already supports them. Accept both blob versions in read validation and discovery so `fetch_blobs`, `fetch_blob_files`, and `fetch_blob_ranges` can read legacy payloads through the existing native reader. Keep list-nested blobs excluded from row-based reads and document how to fetch legacy blobs using explicit row IDs. Regression coverage checks sync and async APIs, independently written Lance datasets, ordering and duplicates across fragments, null and empty blobs, lazy seeking, ranges, and deleted rows. Fixes #4644 Validation: - Both new Python regressions failed on the original implementation with the reported legacy-column rejection and empty discovery result, then passed with the fix. - Python blob suite: 202 passed, 1 skipped. - Remote blob tests: 29 passed. - Focused Rust blob unit tests with `remote`: 13 passed. - Rust blob integration suite with `remote`: 59 passed, including legacy storage 2.0 and 2.1. - Rust compile and Clippy checks with `remote`, tests, and examples passed. - Python package Ruff lint and format checks passed; Rust formatting passed. - Repository-wide `ruff check .` reports 49 existing errors in unchanged `docs/web-tests` examples; the Python package is clean. <!-- lance-gatekeeper-fix:v1 agent=02b7edcbfa8ff0e7fad4615159249b3e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
83b7275137 |
fix(remote): make HTTP timeout errors actionable (#4650)
Remote `create_table` reader uploads can exceed the read deadline while waiting for response headers, but reqwest's main error message only says "error sending request". Make HTTP and retry errors explicitly identify timeouts and name the relevant `client_config.timeout_config` settings and environment variables, preserving typed sources, request metadata, and retry classification. Document that the read timeout covers waiting for headers during upload. This implements the issue's diagnostic option: creation still uses one streaming request, so large or slow readers may require a longer `client_config.timeout_config.read_timeout`. Validation: - 48 focused Rust client, error, and retry tests passed. Regression coverage holds an upload open until its read timeout and also exercises TLS connection timeouts, with and without retries. - 8 Python create-table and HTTP/retry error tests passed in the uv-managed environment after rebuilding the extension, including reader creation with read and overall timeouts. - `cargo check --quiet --features remote --tests --examples` passed. - `cargo clippy --quiet --features remote --tests --examples` passed. - Rust formatting and Python package Ruff lint/format checks passed. - Repository-wide `ruff check .` reports 49 existing findings under `docs/web-tests`; those files are unchanged. Fixes #4646 <!-- lance-gatekeeper-fix:v1 agent=ce1c040fa581f1748166ecb686d9c9c0 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
ad1212fb32 |
chore: update Lance to v15.0.0-beta.2 (#4690)
**Blocked on the corrected Lance release:** https://github.com/lance-format/lance/pull/10370 restores the `Send` contract required by downstream Function workers. Refresh this upgrade with that release before merging. Upgrade the Rust, Java, and Python Lance dependencies to https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.2. This brings in bounded vector-index consolidation without retraining (https://github.com/lance-format/lance/pull/10335) and deletion filtering for index entries whose fragment coverage was pruned (https://github.com/lance-format/lance/pull/10325). Synchronize the vendored `lance-arrow` with the release and retain its offset-safe validity patch. The existing `nullable_struct_merge` regression still panics with the unpatched release, so removing that patch would regress nullable nested queries. Refresh both lockfiles, including the `zstd-sys` version required by the new Lance encoding dependency. Update the custom memtable bitmap test adapter to train on row addresses, matching the new bitmap contract; remove its obsolete row-ID conversion. Preserve the Blob APIs' existing missing-row error messages when newer Lance rejects a missing fragment earlier in the read path. Retain upstream attribution in vendored sources while satisfying the repository license-header check. Validation of the compatibility fixes: all 58 Rust Blob integration tests and 219 Python Blob tests pass (one Python test skipped). Strict workspace Clippy, formatting, and diff checks pass. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
7fca4d666a |
fix(python): preserve async query search options during serialization (#4659)
Async `to_query_object()` dropped enabled `fast_search` on every query type and `postfilter` on plain and full-text queries. Re-executing a serialized fast search could therefore return unindexed rows. Copy enabled `fast_search` into the Python `Query` and serialize enabled plain/FTS postfilter from the Rust request's prefilter setting. Preserve the existing unset defaults: `None` for these options, with the vector query's existing `False` postfilter default. Validation: - Added 12 serialization cases across plain, FTS, and vector queries, covering both enabled and unset options, plus two FTS execution round-trip cases. Before the fix, nine cases failed, including returning an unindexed row after serialization. - Rebuilt the Python extension using the uv-managed environment. - Query tests and existing FTS fast-search tests: 192 passed, 1 slow test deselected. - Rust formatting and Python package lint/format checks pass. Repository-wide `ruff check .` reports 49 pre-existing errors in unchanged `docs/web-tests` files. Fixes #4658 <!-- lance-gatekeeper-fix:v1 agent=f828b24871a2dc928668be2de4f85078 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
1edfa6077b |
fix(python): allow phrase queries without a column (#4653)
Quoted async full-text queries without `columns` carry an unset phrase column, but converting them to Python unwrapped it and raised `PanicException`. Python also required `PhraseQuery.column`, preventing callers from constructing the equivalent query object. Make the Python column optional and preserve `Option<String>` in both Rust conversion directions, leaving column selection to query planning. Regression tests cover quoted strings and structured queries with omitted, `None`, and explicit columns, query reuse, JSON serialization, slop, and document granularity. Validation: - Reproduced both missing-column failures before the fix; explicit-column controls passed. - Rebuilt the extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`. - `cd python && uv run --no-sync --extra tests pytest python/tests/test_fts.py python/tests/test_query.py::test_query_serialization_async -q`: 85 passed. - `cargo fmt --all -- --check` and `ruff check python` pass; touched Python files pass formatting checks. - `cargo clippy --quiet -p lancedb-python --lib --features remote` passes. - Root-wide `ruff check .` reports 49 existing findings in unchanged documentation examples. Fixes #4651 <!-- lance-gatekeeper-fix:v1 agent=62d36fad70c8717236283b8e97dd6b30 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Avanish GargandXuanwo
|
8887c82789 |
fix(python)!: raise on bool(Expr) instead of silently dropping predicates (#4682)
Python's `and` / `or` / `not` and chained comparisons call `bool()` on
an `Expr`. `Expr` had no
`__bool__`, so that was always `True` and the filter silently lost a
predicate: `where(lit(2) <
col("id") < lit(5))` returned ids 0..4 (lower bound dropped),
`(col("id") > 7) and (col("g") == 0)`
returned all even ids, and `(a) or (b)` kept only `a` (see the issue for
the repro and output).
Changes (`python/python/lancedb/expr.py`,
`python/python/lancedb/query.py`):
- `Expr.__bool__` raises `TypeError("The truth value of an Expr is
ambiguous. ...")` with a hint to
use `&`, `|`, `~` with parentheses and `Expr.isin()` for membership.
`TypeError` rather than
`ValueError` because the object does not support truthiness at all
(polars `Expr` and SQLAlchemy
clauses raise `TypeError`; the codebase raises `TypeError` for
unsupported `lit()` values).
pandas, numpy and pyarrow `Expression` use `ValueError`.
- `LanceHybridQueryBuilder` tested `if self._where:`, which would have
started raising for every
`.where(Expr)` on a hybrid query. It is now `isinstance(self._where,
Expr) or self._where`, so
an Expr is always applied and an empty SQL string is still skipped as
before.
- Notes section in the `Expr` docstring.
Not changed: no other code in `python/python/lancedb` tests the
truthiness of a filter or of an
`Expr` (checked `where`/`filter`/`predicate`/`condition` handling in
`query.py`, `table.py`,
`remote/`, `merge.py`, `functions.py`, `pydantic.py`: they use `is None`
/ `isinstance`). Cross-SDK:
`nodejs` has no `Expr` class on `main` yet, and JavaScript cannot
overload `&&` / `||` / `!`
either, so there is nothing equivalent to port; the Rust core is not
affected.
Behaviour changes to be aware of: code that already used `and` / `or` /
`not` / chained
comparisons on an `Expr` (which returned wrong rows) now raises
`TypeError`. `x in [expr, ...]`,
`any([...])` and `all([...])` on Exprs, and `==` between two `Query`
objects whose `filter` are
different Expr objects (pydantic compares the fields), now raise instead
of returning a meaningless
answer; `Expr` stays unhashable and `isinstance` / `is None` checks are
unaffected.
Tests (`python/python/tests/test_expr.py`): `TestExprTruthiness`
(`bool`, `and`, `or`, `not`,
chained comparison, `if`, list membership, `any`/`all` raise with a
helpful message; `&`, `|`, `~`
still combine) and `TestExprWhereBuilders` (plain, vector, FTS, hybrid
and async `.where(Expr)`
return the right rows, hybrid with an SQL string and with an empty
string is unchanged, chained
comparison is rejected before any query runs). With the source change
stashed 11 of the new tests
fail; with only `__bool__` added and the hybrid change reverted the
hybrid test fails with the
new `TypeError`, so it guards the second change. `test_expr.py`: 144
passed (123 before this change).
Verification caveat: `main`'s Python layer is newer than the newest
published wheel, so the tests
were run against the 0.41.0b14 native module
(`python/python/lancedb/expr.py` and `query.py` are
identical to the wheel); the full suite (`pytest python/tests
--ignore=python/tests/docs -m "not slow and not s3_test"`)
has the identical 25 failures before and after (wheel/main skew in
remote options, remote db,
index and table tests; none in expr/query tests): 2604 passed before,
2625 passed after (the 21
new tests). `ruff format --check python` and `ruff check python` are
clean. Not run: Rust build, Node tests, remote (cloud)
tests.
Fixes #4681
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
Xuanwo
|
8a4effa06a |
fix: gate experimental local functions behind an opt-in feature (#4688)
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files. |
||
|
lancedb-gatefixer[bot]
|
dcbf5ce66a |
fix: preserve null blob descriptor fields in SQL (#4571)
SQL descriptor access returns NULL for every field of a NULL Blob v2 value, so `payload.size = 0` matches only empty blobs and `payload.size IS NULL` matches NULL blobs. DataFusion 54.1.0 extracts struct children without combining their validity with the parent struct. Lance descriptors keep nullness in the parent and can store zero-filled children beneath NULL blobs. The adapter derives the descriptor schema from Lance's default projection, marks descriptor children nullable, and masks them before SQL evaluation. Schema derivation requires no query request, and the mask handles struct and list nesting while preserving existing child nulls and value buffers without fetching blob bytes. Filter classification follows the referenced struct field path. Blob predicates stay above the mask; scalar siblings such as `extra.note` retain exact filter pushdown even when `extra.clip` is a blob. **Merge hold:** as requested in the [review discussion](https://github.com/lancedb/lancedb/pull/4571#issuecomment-6253443077), blob SQL INSERT compatibility remains pending a separate write-mode design. DataFusion uses the provider's single schema for reads and INSERT coercion, so the descriptor schema still rejects blob write structs. This follow-up does not restore those INSERTs; the merge should wait until the write contract is settled. Validation: - Reproduced the original NULL blob returning `kind = 0` before enabling the mask. - Reproduced both the extra remote query and lost nested sibling pushdown before the follow-up fixes. - All 441 DataFusion adapter tests passed after merging main at `e6789a41`, including SQL projections, filters, mixed predicates, LIMIT, COUNT, schema agreement, sliced arrays, nested structs, list variants, and EXPLAIN assertions for scalar sibling pushdown. The blob fixtures use the current four-field write format, and NULL-blob filtering is also covered with row IDs and row addresses enabled. - Remote constructor regression passed: one describe request and no query request, with nullable descriptor fields preserved. - `cargo fmt --all` - `cargo clippy --locked --profile ci -p lancedb --features remote --tests --examples -- -D warnings` Fixes #4569 <!-- lance-gatekeeper-fix:v1 agent=2ca7389efd48d11a8debe9111abeb00e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Lance Release
|
37b46f6b4f | Bump version: 0.41.0-beta.14 → 0.41.0-beta.15 | ||
 Wyatt Alt
|
fe07c25a0a |
fix: refuse Function-bound writes by column instead of by table (#4686)
A table with a Function binding rejected every update and merge_insert, whatever columns the write named: updating an unrelated column, or a merge that only inserts rows, failed with "not supported on a table with registered Function bindings". Expression computed columns on the same table accept those writes. Both paths now use the per-column guard that schema evolution already uses. An update is checked on its SET columns. A merge is checked on the columns a matched row would take from the source, every source column but the join keys; an insert-only merge is an append, and its new rows get their Function outputs from the next refresh. Writes to a binding's inputs or outputs are still refused, now naming the column. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
e3dce2cbad |
fix(rust): prevent oversized SQ remap panics (#3861)
## Summary - preflight IVF-SQ and IVF-HNSW-SQ storage segments touched by compaction - preserve unknown legacy fragment coverage as conservatively affected - cover both fixed-size vector and multivector SQ storage dimensions - reject only affected partitions whose fixed-size-list child exceeds Arrow's `u32` take-index limit - return an actionable `InvalidInput` error before Arrow can panic ## Root cause Arrow 58 builds fixed-size-list child indices with `u32`. For a large SQ partition, `partition_rows * vector_dimension` can exceed `u32::MAX`; the child range then wraps inside `take_fixed_size_list`, producing the reported `ArrayData::slice` assertion during index remapping. The preflight now reads raw segment metadata, mirrors Lance's fragment-bitmap selection (including `None` as affected/unknown), opens only affected vector segments, and checks their actual storage quantizer and partition sizes. It derives the stored dimension for both `FixedSizeList` vectors and flattened `List<FixedSizeList>` multivectors. Stable-row-ID datasets, deferred remapping, custom remappers, unaffected segments, non-SQ indexes, safe partitions, and no-op compactions are unchanged. Because the upstream Arrow kernel still has the limit, an affected index must be recreated with default IVF partitioning (or enough partitions to stay below the reported maximum) before compaction. This changes the failure from a process panic to a targeted recovery error; it does not silently alter index structure. ## Validation - `cargo fmt --all -- --check` - `cargo test --quiet --features aws,remote -p lancedb table::optimize::tests --lib` - `cargo test --quiet --features remote -p lancedb table::create_index::tests::test_create_index_ivf_hnsw_sq` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples -- -A clippy::redundant-pub-crate` (the allowance is for pre-existing Rust 1.97 warnings in remote-table code) Fixes #2866 <!-- lance-gatekeeper-fix:v1 agent=ba2f07b1ee6f9e68a0de1961c164f70e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
a9fb16ac44 |
fix: return cosine-scaled ANN distances (#3852)
## Summary - convert unrefined cosine ANN scores from normalized squared-L2 scale to the public cosine scale - translate public distance-range bounds to the internal ANN scale before execution - add regression coverage for indexed distances and range filtering while leaving refined, exact, and flat paths unchanged ## Root cause Cosine PQ, SQ, and RQ quantizers normalize vectors and search with squared L2 internally. The unrefined native query plan returned those internal scores directly, making the values twice the public cosine distance for unit-normalized embeddings. ## Validation - `cargo test --quiet --features remote --tests` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo fmt --all -- --check` - rebuilt the Python extension and reran the issue reproduction through the Python API Fixes #3530 <!-- lance-gatekeeper-fix:v1 agent=fa75cad2d924c92a97215fcab4e21b14 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
311ded06d0 |
fix(ci): acknowledge Arrow-pinned zstd advisories (#4683)
RustSec published four `zstd` advisories after #3819 merged, causing the `deny` job to fail on `main`. `arrow-ipc 58.x` pins `zstd ^0.13`, while the patched releases require `zstd >= 0.14.3`; moving to Arrow 59+ requires a coordinated DataFusion and Lance upgrade. This records temporary exceptions with their reachability rationale. LanceDB uses the `zstd::bulk` API through `arrow-ipc`; the affected `with_context`, seekable, and streaming-decoder APIs are not used, and the `zstd-sys` advisory is wasm32-only while wasm32 is not a shipped target. Validation: `cargo-deny 0.20.2` reports `advisories ok, bans ok, licenses ok, sources ok`. |
||
 Drew
|
436c209580 | docs: document Blob V2 in the multimodal guide (#4623) | ||
|
lancedb-gatefixer[bot]andXuanwo
|
7952ee2a71 |
fix(python): cover FP16 GPU index queries (#3819)
Keep FP16 vectors through IVF-PQ index creation and search in the synchronous and asynchronous quickstarts. Add native and conditional CUDA regression coverage using a float64 query against an explicitly typed FP16 column. This is test and documentation-example coverage for dtype preservation already present in Lance; it does not change production code. Related #2120. Validation on `bbad20940632b21a9ee85bdcb46b4cc19dbd65e3` with Python 3.12.12, Lance 15.0.0-beta.1, PyArrow 24.0.0, and Torch 2.14.1: - The native FP16 parameter, Arrow FP16 ingestion test, and both quickstarts passed: 4 passed, 1 CUDA-only skip. - The two affected test files passed: 386 passed, 1 CUDA-only skip (`-m "not slow and not s3_test"`). - Ruff 0.9.9 lint and formatting checks passed for the Python package. Real CUDA execution has not been performed. Torch is installed, but `torch.cuda.is_available()` is false; `test_create_f16_table[cuda]` skips with `CUDA not available`. The native CPU result does not establish that the original GPU failure is resolved. Issue #2120 remains open for verification on an actual CUDA device. Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 KarthikSandXuanwo
|
180eaaf68e |
fix!: reject unsupported remote connection and table options (#4486)
Fixes #4399. Remote connections currently discard local-only settings, including table write options that determine storage format. The synchronous Python client also binds the fourth positional `create_table` argument to `on_bad_vectors`, so an intended overwrite is sent as a create. This change aligns the remote Python signature with `DBConnection.create_table`, preserves the legacy remote timeout keywords, and raises `ValueError` for other unknown connection keywords. The shared Rust client rejects unsupported connection, create, and open options before sending HTTP requests; open validation also runs before a table-cache hit. Python reports these as `NotImplementedError`, including explicit `enable_v2_manifest_paths=False`. Supported Enterprise connection-level storage options remain available. This implements option 2 from the issue: explicit rejection. It deliberately does not add unverified server write parameters. Calls that previously supplied ignored options must omit them to use server-managed storage and caching. Database URI prefixes are already preserved on current main and are unchanged here. Regression tests cover Python sync/async, TypeScript create/open calls, and the Rust remote client, including empty-table creation, overwrite mode, and cached table handles. Python API documentation describes the behavior and corrected positional order. Reconciled with main at `0ea932da`, preserving upstream blob and error-handling tests and updating the Python test that previously expected open options to be ignored. Validation on the merged source: - Rebuilt the Python native extension; 304 tests passed across remote options, database, remote database, and session suites. - Rust remote database suite: 135 passed. Rebuilt the Node native extension; TypeScript remote suite: 62 passed. TypeScript compilation and documentation generation passed. - Full Python SDK lint and formatting pass (159 files); full TypeScript lint and formatting pass (66 files). Rust formatting and `git diff --check` pass. - Repository-wide Python lint/format reports existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust 1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci` build profile. Remote tests use HTTP fixtures; this does not claim validation against the original ARM64 single-node deployment. Documentation review follow-up: added a draft Enterprise migration note in `docs/web/basics/sdk.mdx`. It explains that `db://` connections reject local session/cache and per-table storage/write options, while supported connection-level Enterprise storage options remain available. Following the repository's release conventions, the note stays in an MDX comment below the migration list until the behavior is released. Vale 3.24.0 with the pinned LanceDB style package reports no findings; `git diff --check` passes. This follow-up changes documentation only. BREAKING CHANGE: Remote connections now reject unsupported local session/cache and per-table storage/write options instead of silently ignoring them. Callers must omit these options and use server-managed storage and caching; supported Enterprise connection-level storage options remain available. Integration with current main preserves its synchronous API signatures, including the remote `create_table(mode=None)` default, and the table compatibility defaults. The positional-order repair is already on main via #4608; this PR retains the distinct unsupported-option rejection behavior. CI follow-up: the inherited IVF search-effort test occasionally measured recall 0.375 against its unchanged 0.5 threshold after random index training. A matching local reproducer observed this once in 30 builds. The fixture now places the eight exact neighbors together, preserving the 16-partition setup, two writes, all probe/precedence assertions, and the recall threshold. The adjusted fixture returned recall 1.0 in 100 builds; the full query and remote suites passed (102 tests). This stabilizes test data and does not change search behavior or promise production recall. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
KarthikSandXuanwo
|
982b2a1229 |
fix(python): restore sync remote connection and table API parity (#4473)
Fixes #4400. Sync remote connections fail on `uri`, `len(db)`, and `name in db`, and remote tables lack `namespace`, `id`, and `replace_field_metadata`. Share the paginated connection helpers and table identifier contract through the base classes, and delegate remote metadata replacement to the existing async implementation. Unsupported remote session and read-consistency properties raise explicit `NotImplementedError`s. Reconciled with main at `3dd57978`, including the synchronous method signatures from #4498. The local session fix has already landed in #4433, so this PR preserves its supplied-session-or-None contract and adds no Rust/PyO3 session access. Namespace connections retain their writable properties. Existing custom `Table` subclasses remain instantiable: `namespace` defaults to a fresh empty list, and deprecated `replace_field_metadata` delegates to `update_field_metadata` with `replace=True` while retaining its `None` return. Native and remote overrides are preserved. Regression tests instantiate an old-style subclass through normal ABC class creation and verify identifiers, independent namespace lists, deprecation, and exact metadata delegation. Historical validation at `03c43f7b` before the latest main integration and subclass compatibility repair, Ubuntu x86-64, Python 3.12.13: - Rebuilt the native extension with the repository `ci` profile and locked Python dependencies. - 273 tests passed across `test_sync_api_parity.py`, `test_db.py`, `test_remote_db.py`, and `test_session.py`. - 84 namespace and catalog compatibility tests passed (357 tests total). - Python SDK lint and formatting pass (158 files); changed-file checks and `git diff --check` pass. - Repository-wide lint/format checks report existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). These are outside this patch. Remote tests use the existing HTTP fixtures through the rebuilt native client; the original aarch64 server deployment was not recreated. Validation after merging main `3dd57978` and the compatibility repair (macOS ARM64, uv-managed Python 3.12.12, rebuilt editable native extension): - 60 focused parity, signature, and error-contract tests passed. - 381 database, remote, session, namespace, catalog, and sync parity tests passed; 2 platform-dependent tests skipped. - The two new subclass regressions fail against the pre-repair implementation with the expected abstract-member instantiation error, then pass with the repair. - Python SDK Ruff lint/format passes (167 files); repository-wide findings remain confined to unchanged docs/web-tests (49 lint findings, 11 formatting files). --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
KarthikSandXuanwo
|
3dd579785e |
fix(python)!: align synchronous table method signatures (#4498)
Closes #4396. Calls written against `Table` can fail or bind different arguments when used with `RemoteTable`. Align the affected sync signatures, positional order, and defaults; preserve renamed remote keywords as deprecated aliases. Share legacy vector configuration and tokenizer inference, forward index tuning and timeout options, and accept `fast_search` consistently. Preserve current upstream conversion behavior: local `to_polars(batch_size=...)` returns a LazyFrame; remote tables accept the same argument signature but raise the documented unsupported-operation error and direct callers to `table.search().to_polars()`. The abstract `Table` method retains its unsupported-operation error, with the implementation confined to `LanceTable`. Regression coverage checks argument binding, deprecated aliases, option forwarding, native behavior through both wrappers, HTTP index requests, and remote conversion rejection. The merge preserves current tokenizer defaults, remote query defaults, HNSW-PQ support, and unsupported-operation handling. Runtime validation at `c4fb01f2` (Linux x86_64, uv-managed Python 3.12.15): - Focused Ruff lint and formatting passed for all four changed Python files. - Rebuilt and installed the current native extension successfully with the repository's `ci` Cargo profile. - Seven regression suites passed: `test_errors.py`, `test_table_signatures.py`, `test_remote_db.py`, `test_table.py`, `test_index.py`, `test_fts.py`, and `test_query.py`: **874 passed, 1 deselected**, with slow and S3 tests excluded. - CI caught an abstract-base conversion regression in the initial merge. The existing error-contract test reproduced it locally (1 failed, 7 passed); restoring the implementation to `LanceTable` fixes it while preserving signature parity. - Full-root Ruff reports 49 lint errors and 11 files needing formatting, all in unchanged upstream `docs/web-tests` files. The four changed Python files pass both checks. - Validated against upstream `197403e8`, using Lance 15.0.0-beta.1 and LanceDB 0.41.0-beta.14 bindings. Historical validation before this merge: 618 tests passed and one was deselected. That result does not establish correctness of the current merge. The original client-side reproduction produced nine argument-binding errors without contacting a server; the initial regression suite failed 32 cases before the fix and passed 39 after it. Documentation review follow-up: restored the `batch_size` parameter description on the abstract `Table.to_polars` method so it appears in the public API reference. This adds six docstring lines and changes no runtime behavior. After this edit, all 47 focused error-contract and signature tests pass; changed-file Ruff lint/format and `git diff --check` pass. Repository-wide Ruff still reports the same 49 findings and 11 formatting files in unchanged `docs/web-tests`. The full documentation site and seven-suite runtime run were not repeated for this documentation-only follow-up. The reported ARM64 production-server deployment has not been recreated. HTTP integration tests use the repository's local mock server. BREAKING CHANGE: Remote synchronous table methods now use the positional argument order and defaults documented by `Table`. Migrate calls using the former remote-only positional order to named arguments, particularly `create_index` and `search`; pass `index_type` by keyword to `create_scalar_index`. Remote scalar and vector index creation now default to `replace=True`; specify `replace=False` to retain rejection of an existing index. Renamed remote keywords remain supported as deprecated aliases, as documented above. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
66357275bb |
fix: inherit Lance defaults for IVF RQ quantization (#4585)
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding Lance's five-bit default. The shared native index creation path also substituted one bit when the Rust or Node builder left the value unset. Make Python's bit count optional and forward only explicit values. Use `RQBuildParams::default()` for an unset native bit count, and omit unset values when dispatching to pylance for accelerator builds. Update the Python, Rust, Node, and vector-index documentation. New indexes with an unset bit count now inherit Lance's default, currently five bits, across Python, Rust, and Node. Five-bit vector codes use roughly five times the storage of one-bit codes, plus other index overhead. This also changes reader compatibility: indexes created with `num_bits >= 2` use a newer on-disk layout that some older LanceDB versions cannot read. Upgrade readers to versions that support multi-bit IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)` in Python) when sharing indexes with readers limited to one-bit IVF_RQ. Validation: - Reproduced six regression failures before the fix: one-bit default metadata, rejected `None`, and incorrect remote/accelerator forwarding. - After merging `main` through `b762ec22`, rebuilt both native bindings against Lance 15.0.0-beta.1. Python validation passed with 106 tests and 15 optional Ollama skips. Coverage includes the full index suite, omitted and explicit bit counts, reading Lance-created IVF_RQ indexes, accelerator forwarding, CUDA-unavailability handling, timeout validation, remote requests, and table/namespace listing behavior. - Rust formatting and Python package Ruff lint/format checks pass. Node `pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run docs`, the full connection suite, and the existing IVF_RQ regression pass. Generated documentation adds no further changes. The merge preserves upstream's explicit RaBitQ heading anchor alongside the `IvfRq()` example and five-bit default/reader-compatibility guidance. Repository-wide `ruff check .` still reports 49 existing errors in unchanged documentation examples under `docs/web-tests`; root formatting also finds 11 unchanged documentation example files. The Python package checks pass. Fixes #4582 <!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 
|
22fc2a2087 |
fix(python)!: put OpenCLIP models in eval mode before embedding (#4536)
`OpenClipEmbeddings` loads its model with
`open_clip.create_model_and_transforms()`, which hands the model back in
training mode (the open_clip README calls `model.eval()` right after it
for that reason). `OpenClipEmbeddings` never switches it out of training
mode, so layers that behave differently during training stay active
while we embed: stochastic depth in the timm-based image towers
(ViTamin, ConvNeXt, MobileCLIP2-L-14), dropout in the Hugging Face text
towers (`xlm-roberta-*`, `roberta-ViT-B-32`, `nllb-clip-*`), and batch
norm in the ResNet towers loaded with non-OpenAI weights. The
`torch.no_grad()` blocks turn off autograd but don't change how these
layers behave.
The default `ViT-B-32` has none of those layers, which is probably why
nobody has run into this. With dropout or stochastic depth active, the
same input gives a different vector on every call, so the vector stored
at ingestion and the one computed for an identical query don't match.
With batch norm in training mode, an image's vector depends on the other
images in its batch:
```python
import numpy as np
from PIL import Image
from lancedb.embeddings import get_registry
func = get_registry().get("open-clip").create(name="ViTamin-S", pretrained="datacomp1b")
image = Image.new("RGB", (256, 256), "red")
a = func.generate_image_embedding(image)
b = func.generate_image_embedding(image)
print(func._model.training, np.array_equal(a, b))
# before: True False
# after: False True
```
The fix calls `model.eval()` once after loading, same as the SigLIP,
ColPali and ImageBind embedding functions already do.
The new test swaps in a stand-in for the `open_clip` module, so it runs
without torch or a model download, and checks that the model handed back
by `create_model_and_transforms` ends up in eval mode.
BREAKING CHANGE: Embeddings change for OpenCLIP models with
training-sensitive layers, including ViTamin, ConvNeXt, MobileCLIP2,
Hugging Face RoBERTa/XLM-RoBERTa/NLLB text towers, and ResNet models
loaded with non-OpenAI weights. For tables embedded with affected
models, regenerate the stored embeddings in eval mode and rebuild their
vector indexes before querying with the new embeddings. Rebuilding an
index alone does not regenerate embeddings. The default ViT-B-32 model
is unaffected.
Co-authored-by: devjoinedthechat <235875317+devjoinedthechat@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
lancedb-gatefixer[bot]andXuanwo
|
7ed03e1c40 |
fix(python): honor bad-vector policies for pandas NaNs (#4503)
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN mask and Rust's default append validator ignored null children, so `add()` silently stored incomplete vectors despite `on_bad_vectors="error"`. Reject null components in the existing Rust ingestion validator, and include null components in Python's bad-vector detection and filling. `drop` removes affected rows, `fill` repairs missing components while preserving valid ones, and `null` replaces affected vectors. Explicit null components in Arrow and list inputs follow the same policies. Rust allows whole null vectors and retains the explicit `NaNVectorBehavior::Keep` opt-out. The ingestion guide and synchronous/asynchronous `create_table()` option docs describe null components, all four policies, and filling that preserves valid components. Validation: - Reproduced the default pandas append failure with the regression test before changing the Rust validator. - Rebuilt the native Python extension from the merged sources. The Python table and utility suites passed: 481 passed, 1 skipped. Coverage includes create/append/overwrite, pandas/rows/Arrow inputs, every bad-vector policy, float16/32/64, sliced and chunked arrays, all-null components, and hidden children of null lists. - All 43 focused Rust tests passed: 7 validator tests and 36 ingestion tests. - Rust Clippy with `remote`, tests, and examples passed. Ruff lint passes for the Python package, formatting checks pass for modified Python files, and `cargo fmt --all` is clean. - Documentation follow-up: all 56 focused vector-policy tests passed; Python package lint and creation-docstring formatting checks passed. Repository-wide Ruff reports 49 existing errors in untouched documentation examples. Create/overwrite retain their existing `RuntimeError` wrapping for errors raised while consuming lazy Arrow readers. Fixes #4435 <!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Joaquin Hui GomezandXuanwo
|
256db55a51 |
fix(python)!: align RemoteDBConnection.create_table parameter order (#4608)
`RemoteDBConnection.create_table` orders its parameters `(name, data,
schema, on_bad_vectors, fill_value, mode, exist_ok, ...)`, while
`DBConnection`, `LanceDBConnection` and `LanceNamespaceDBConnection` all
use `(name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
...)`. Since `RemoteDBConnection` subclasses `DBConnection`, identical
positional calls bind to different parameters: switching a working
script from `lancedb.connect(path)` to `lancedb.connect("db://...")`
silently turns `mode="overwrite"` into `on_bad_vectors="overwrite"` and
`exist_ok=True` into `fill_value=True`, and nothing client-side
validates either. `inspect.signature(...).bind(None, "t", data, None,
"overwrite", True)` shows it deterministically, no cloud needed.
This reorders the remote signature to match the other three; every
parameter keeps its own existing default (including the remote's
`mode=None`, which its body relies on), and the body already forwards by
keyword so nothing else changes. The new test asserts positional parity
with `DBConnection` through `embedding_functions` and fails on `main`.
Blast radius, stated plainly: only remote callers passing four or more
positional arguments are affected — positions 4–7 move; positions 1–3
and 8 are unchanged and keyword callers are unaffected. No in-repo
caller passes more than two positionals. Anyone relying on the current
order was relying on behaviour that contradicts the documented base
contract and diverges from local connections. Not added here: the
`storage_options`/`data_storage_version`/`enable_v2_manifest_paths`
parameters the remote signature still lacks.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
lancedb-gatefixer[bot]
|
e6789a4118 |
fix: support system columns in DataFusion table adapters (#4667)
DataFusion plans SQL against the adapter's declared schema, which omitted `_rowid` and `_rowaddr` even though Lance can project both. SQL therefore rejected these columns before reaching the scan. Add `BaseTableAdapter::with_system_columns` to opt into either column using Lance's nullable `UInt64` fields. Scans without a projection now select the full declared column list so their output matches the provider schema. Default adapters retain data-only `SELECT *` and remain compatible with `INSERT INTO ... SELECT *`; opted-in adapters include the requested system columns in `SELECT *`. Regression tests compare SQL results with native queries and cover projection order, system-only projections, row-ID predicates, scans without a projection, duplicate/unsupported names, and default SQL inserts. Validation: - DataFusion unit suite: 436 tests passed with `cargo test --locked --quiet --profile ci -p lancedb --features remote --lib table::datafusion`. - `cargo clippy --locked --quiet --profile ci -p lancedb --features remote --tests --examples -- -D warnings` passed. - The new builder's documentation example passed its doctest. - `cargo fmt --all -- --check` and `git diff --check` passed. Fixes #4666 <!-- lance-gatekeeper-fix:v1 agent=58d267206580227b3820df65af15b0a7 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Jack YeandLance Release
|
197403e891 |
fix: honor downstream Lance dependency overrides (#4671)
Consumers that provide a local Lance checkout cannot resolve the dependency graph because the SDK directly selects a vendored `lance-arrow`, producing two path packages with the same name and version. The Rust core now declares its upstream Git dependency separately so the consuming workspace can override that source consistently. Python continues inheriting the vendored workspace dependency, keeping the sliced-struct validity fix in Maturin source distributions. Standalone SDK builds still select that same fix through the existing root patch. Adds a downstream consumer CI regression that resolves a local Lance override and checks that exactly one canonical `lance-arrow` package is selected. The Lance version updater also updates the core declaration in stable, preview and local modes. Public APIs and query behavior are unchanged. The workspace declaration lint excludes path dependencies because the Python binding intentionally uses the bundled source. Duplicate and unused external declarations remain errors; advisory, license, banned-package and source checks continue covering the full dependency graph. --------- Co-authored-by: Lance Release <lance-dev@lancedb.com> |
||
|
Lance Release
|
de0c5edc7d | Bump version: 0.41.0-beta.13 → 0.41.0-beta.14 | ||
|
Jack Ye
|
dcfef7c5ab |
feat(python): open lazy blob files from SQL descriptors (#4668)
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
|
||
|
Jack Ye
|
9af7757053 |
fix: execute remote materialized view creation through SQL (#4665)
Remote materialized-view creation currently expects a REST response containing a registry job ID, while refresh already submits SQL. Route both through the SQL query backend and preserve Python's synchronous and asynchronous return types. ### Query to Job mapping `SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through the same backend as `execute_query_async()`. Once submission succeeds, it retains the returned `Query` and starts one client-side task that consumes its result reader to completion. `Job::from_query_task()` wraps that task, its cached outcome, a single ID and a cancellation callback. Python exposes this as `Job[T]` or `AsyncJob[T]`. The existing `job_type` property maps SQL-backed handles to the new `"query"` kind. The shared `id` contains that query's UUID string, so Python callers can inspect it with `db.describe_query(job.id)` on the submitting connection. No separate `query_id`, backend enum or `type()` API is added. | Handle | `job_type` | `id` | Available lookup | | --- | --- | --- | --- | | SQL-backed | `"query"`, known at submission | Connection-scoped query UUID string | `describe_query(job.id)` | | Registry-backed | Existing server-reported kind, after `status()` or `refresh()` | Registry job ID string | Registry job APIs, descriptions and events | | Local | `None` | `None` | The current in-process handle | | Job concept | SQL-backed behavior | | --- | --- | | Submission | Await initial SQL submission before returning the handle. Submission errors propagate immediately. | | `status()` | Tracks the adapter task: running during result consumption/finalization, finished after both succeed, failed if either fails, or cancelled after cancellation. | | `wait()` | Returns the operation's typed terminal result. One task consumes the query reader once; repeated waits reuse its cached outcome without resubmitting SQL or reopening the reader. | | `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported cancellation leaves the task running; an accepted request stops the task. Cancelling a finished operation is a no-op. | | Registry details | Query-backed handles expose no registry description, event history, pause/resume or reattachment after a client restart. `refresh()` updates cached lifecycle state. | The Job tracks command completion rather than exposing SQL batches. A wait timeout does not cancel the query. Query IDs remain scoped to the submitting connection and cannot be used with registry APIs or as durable resume tokens. Python `describe_query()` accepts both UUID objects and the string `id` exposed by query-backed jobs. Rust, Python and TypeScript expose the same classification through their existing job-type accessors. ### Materialized-view operations - **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace and optional auto-refresh settings. After command completion, resolve the job to `None`. Synchronous creation waits on the job, then opens and returns the materialized view. - **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same adapter. After command completion, read the refreshed table's version and row count to construct the existing typed refresh result. Synchronous refresh waits on the same path. Remote creation requires the SQL service. Execution placement remains a server decision. REST endpoints are unchanged. The job guide documents the ID/job-type contract and query-backed limitations. Regression coverage exercises SQL submission, namespaces, refresh options, job-type/ID preservation through typed result mapping, repeated waits, failures and cancellation rejection. No local builds or tests were run. |
||
|
Jack Ye
|
24c2669a50 |
feat(sql): follow adaptive streaming and polling responses (#4664)
SQL execution now starts with one Flight call. The server can stream
rows immediately or return a polling continuation, which the SDK follows
through the existing query handle and reader APIs. The same behavior
applies with and without parameters.
Both initial transports share direct-stream lifecycle handling. Handoffs
are explicitly negotiated; existing servers can continue returning
ordinary streams. Add coverage for streaming, deferred results,
cancellation, malformed decisions, and legacy responses.
```mermaid
flowchart TD
A[execute_query or execute_query_async] --> B{Parameters supplied?}
B -->|No| C[DoGet: SQL command ticket]
B -->|Yes| D[DoExchange: SQL and Arrow parameter row]
C --> E[Server chooses execution path]
D --> E
E -->|Stream| F[Initial response: streaming decision and Arrow batches]
E -->|Defer| G[Initial response: PollInfo continuation]
G --> H[SDK follows PollFlightInfo]
H -->|Still running| H
H -->|Result endpoints available| I[SDK fetches tickets with DoGet]
F --> J[Query reader yields Arrow batches]
I --> J
```
`execute_query` opens the reader; `execute_query_async` returns its
query handle. The caller does not select the transport or execution
path.
|
||
|
Jack Ye
|
6b6778e684 |
feat: infer lazy blob reader Function inputs (#4663)
Infer Blob V2 input access from the Python Function signature. `video:
BlobReader` requests a lazy, seekable reader; `video: bytes` requests
the complete payload. Users do not need a separate `blob_inputs`
registration option.
## Python experience
For a video table joined to a table of frame timestamps, the Function
can pass its reader directly to a media decoder:
```python
import io
import av
from lancedb import BlobReader, udf
@udf(pip=["av", "Pillow"])
def extract_frame(video: BlobReader, timestamp_s: float) -> bytes:
with av.open(video) as container:
stream = container.streams.video[0]
container.seek(int(timestamp_s / stream.time_base), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time >= timestamp_s:
output = io.BytesIO()
frame.to_image().save(output, format="JPEG")
return output.getvalue()
raise ValueError("No frame at or after the requested timestamp")
```
After registering `extract_frame`, the query keeps the usual Function
call:
```sql
SELECT f.video_id, f.timestamp_s,
extract_frame(v.video, f.timestamp_s) AS image
FROM videos AS v
JOIN frames AS f ON v.id = f.video_id
```
The returned JPEG can feed a subsequent image-model step. The reader
provides `read`, `readinto`, `seek`, `tell`, `size()`, and
`read_range(offset, length)`; range reads leave the sequential cursor
unchanged. Each row invocation owns its reader, which closes after the
call. Decoder seeks can request the parts of the MP4 needed to locate
and decode the requested frame; the amount read depends on the file
layout and codec.
Functions that need the whole value can keep a bytes signature:
```python
@udf
def video_length(video: bytes) -> int:
return len(video)
```
## Execution contract
- Signature inference maps `BlobReader` to a Blob V2 field and records
`blob_access="reader"`. `bytes` keeps its binary Arrow type and records
`blob_access="bytes"`; it can also bind to a Blob V2 source. Explicit
schemas support the same choice, including a Blob V2 field annotated as
`bytes`.
- Reader execution carries a compact blob reference into the Function
worker. Payload reads happen when Python calls the reader, with requests
split into at most 1 MiB ranges. The full video does not need to enter
the Arrow input batch just to invoke the Function.
- Bytes execution materializes the full source before the Function runs
and remains subject to eager-input memory admission. Calling
`reader.read()` without a size also materializes the remaining contents
in Python; lazy access depends on how the Function consumes it.
- Input access metadata survives Python/Rust Function serialization and
registration. Packaged Functions import the worker-provided reader
implementation. This PR defines the SDK contract; execution requires
corresponding server support.
- This changes blob inputs. The example still returns ordinary image
bytes; it does not add streamed outputs or run an image model.
Adds coverage for signature inference, explicit schemas, nullable
readers, signature round trips, seeking, bounded range reads, buffering,
and invalid responses.
|
||
 Shiduo LiandXuanwo
|
0ea932dafd |
fix(rust): compile jobs without remote feature (#4103)
## What Move typed job decode error construction behind a feature-aware helper so builds without the `remote` feature do not reference `Error::Http`. Local decode failures continue to surface as `Error::Runtime`, while remote builds retain HTTP errors with request IDs. Add a regression test for local typed-result decode failures. Closes #4096. ## Testing - `cargo fmt --all -- --check` - `cargo check -p lancedb --no-default-features` - `cargo test -p lancedb --no-default-features job::tests::local_typed_result_decode_errors_are_runtime_errors` Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Yeluru NitishandXuanwo
|
2c9f2e75a0 |
fix: keep FTS indexes in list_indices when their files are missing (#4251)
Fixes #4250 `list_indices()` loads each FTS index's segment params from the index files to fill in `index_details`. That call used `?`, so a single FTS index with missing files made the whole listing fail. A BTree or Bitmap index with missing files still lists fine, because it never takes that path. Now, if the params can't be loaded, it logs a warning and keeps the entry with the details from the manifest. Callers can still see the damaged index and rebuild or drop it. Added a Python test based on the repro in the issue. It fails on 0.39.0 with the same RuntimeError. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 
|
cf5c9b5453 |
fix: validate cached remote tables on open (#4557)
## Summary Fixes #4401. - Validate cached remote table entries with the existing `describe` endpoint before returning a handle. - Preserve cached server-version behavior after a successful validation. - Add a regression test for a table dropped by another connection after its entry has been cached. A cached entry can otherwise outlive an external drop, allowing `open_table()` to return a handle that fails only on its first later operation. ## Verification [Linux validation](https://github.com/Ganesh1729-ui/lancedb/actions/runs/39420560974) passed against the exact runtime and test sources at PR head `d39092b01f6b49921b5d66ccedf29709a7ddc23d`, using the repository's Rust toolchain and `ci` profile: - `cargo fmt --all -- --check` - `cargo test --profile ci --locked -p lancedb --features remote --lib remote::db::tests::test_open_table -- --nocapture`: **8 passed**, including cached-table deletion, independent handle checkout, branch/version handling, schema seeding and describe-response handling. - Removing only the cached-hit existence guard makes `test_open_table_checks_that_a_cached_table_still_exists` fail at its `TableNotFound` assertion; restoring the fix makes the same test pass again. - `git diff --check` The validation workflow is on a separate fork branch and is excluded from this PR. These are focused, mock-backed remote unit tests; the full upstream matrix and live Cloud integration tests still need upstream CI approval. Local Windows compilation was unavailable because the MSVC environment has no `link.exe`; the linked Linux run completed successfully. --------- Co-authored-by: Ganesh172919 <ganeshreddy17291122@gmail.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
  
|
a8305e0722 |
fix(rust): accept usize::MAX as a local query limit (#4567)
Fixes #2211. Local queries converted `limit`/`offset` with `as i64`, so `limit(usize::MAX)` wrapped to `-1` and Lance failed with `Limit must be non-negative`. The remote client already worked around this (#2354); this fixes the local path. - Saturate scanner offsets to `i64::MAX` and cap limits at `i64::MAX - offset`, keeping both conversion and full-text pagination addition safe. - Use `saturating_add` for vector `top_k = limit + offset`, then cap candidates at the dataset row count to avoid indexed-search capacity and refinement overflows. Keep a positive candidate bound for empty tables because Lance rejects `k = 0`. - Leave public APIs and remote query handling unchanged. ### Testing - Regression coverage includes plain scans, indexed IVF search, refinement, index bypass, empty vector tables, and indexed full-text pagination with finite and extreme limits/offsets. - Reproduced the indexed-search and FTS overflows before applying the follow-up fix. - `cargo test --quiet --features remote -p lancedb --lib query::` — 81 passed. - `cargo clippy --quiet --features remote -p lancedb --tests --examples` — clean. - `cargo fmt --all` and `git diff --check` — clean. Vector planning now reads the dataset row count (metadata fast path) to bound candidate allocation; this was not performance-benchmarked. --------- Co-authored-by: Romin Parekh <roparekh@adobe.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
6777a40dc2 |
fix: support remote tables in storage root (#3881)
## Summary - allow `db://` to select the remote storage root when a host override supplies the endpoint - preserve the empty `x-lancedb-database` header used by the legacy clients for root storage - reject root-storage connections without a host override before constructing an invalid Cloud URL - document the supported URI and cover both accepted and rejected configurations ## Root cause The shared Rust remote URL parser required every `db://` URI to contain a host. That validation was introduced when the Node and Python clients moved to the Rust implementation, but the legacy Node client intentionally treated the empty host in `db://` as the storage-root selector. ## Validation - `cargo fmt --all` - `cargo test --quiet --features remote -p lancedb test_connect_remote_storage_root` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` Fixes #1944 <!-- lance-gatekeeper-fix:v1 agent=c71ca188791ad36fe5644c12960f758c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
dd4795028b |
fix(python): align sync remote methods with async behavior (#4411)
Sync remote table and connection methods inherited bare base-class stubs, so unsupported conversion and bulk-drop operations raised `NotImplementedError` with an empty message and differed from the async API. Delegate `to_lance()`, `drop_all_tables()`, and `drop_database()` to the async implementation. Sync and async remote Lance conversion share #4306's native-table guard and reject before requesting a URI or opening client storage. Conversion documentation directs remote reads through server queries. Add an explicit Polars error, suggest query conversions in Arrow/Pandas/Polars errors, and give the remaining base-class errors an operation name. Preserve upstream `blob_mode` validation for remote Arrow and Pandas conversion. The shared remote bulk-drop error suggests dropping tables individually. Regression tests cover sync/async rejection parity, accessible and missing advertised locations, pinned versions, and bulk-drop rejection without requests. Sync blob query tests verify that descriptions come from the server and unsupported modes cannot open client storage. Keep the remote closed-blob-handle assertion consistent with the upstream `ValueError` contract. Validation: - All 30 focused regression cases passed. - 222 Python tests passed with rebuilt bindings in the locked uv environment: the full remote and error suites, plus selected local conversion, blob, and deletion tests. - Python lint and formatting, Rust formatting, `cargo check --quiet --features remote --tests --examples`, and `cargo clippy --quiet --features remote --tests --examples` passed. - Root Ruff checking reports 49 existing errors in upstream `docs/web-tests`; those files are unchanged relative to `main`. Fixes #4392 <!-- lance-gatekeeper-fix:v1 agent=95db5dd7002043a2c48924ded18b51b2 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Kerman LauandXuanwo
|
0dd78656b9 |
fix(python): honor explicit OpenAI reranker API keys (#4515)
Fixes #4514 ## Summary Honor an explicitly supplied `OpenaiReranker(api_key=...)` before falling back to `OPENAI_API_KEY`, matching the constructor documentation. Previously the environment key always won when both were set, so an instance configured for a different account still used the process-wide account. ## Tests Added four cases using real OpenAI client construction without sending requests: explicit/environment conflict, explicit key alone, environment fallback, and missing credentials. The conflict case fails before the change; all four pass afterward. Clients are closed after each assertion. - Targeted source-module tests: 4 passed. - Ruff check and format check: passed for both changed files. - Full native source build was attempted but failed because the environment has no Rust toolchain; bootstrap then attempted a cache path outside the writable workspace. The targeted tests load the current Python reranker module against installed dependencies and the released LanceDB package; no database/native behavior is exercised. Full native/integration suite not run. This change and its tests were prepared with an AI coding agent. The diff was reviewed and the failure/pass behavior verified locally. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 alexcastgonzandXuanwo
|
8cf115b081 |
fix(python): replace the retired gpt-4-turbo-preview default in OpenaiReranker (#4613)
OpenAI shuts down `gpt-4-turbo` and its snapshots on 2026-10-23 (https://developers.openai.com/api/docs/deprecations), so `OpenaiReranker` fails for every user who relies on the default `gpt-4-turbo-preview` after that date. This changes the default to `gpt-5.6-sol`, the replacement OpenAI names for `gpt-4-turbo` on that page, in the class, its docstring and the integration docs page. `_rerank` also stops sending `temperature=0`: the GPT-5 models only accept the default temperature, so keeping it would fail the new default on its first call. The reranker still asks for `json_object` output and the rest of the request is unchanged. The new test calls `_rerank` with a mocked client and checks that no `temperature` is sent. It fails on main and passes with the change. No request was made to OpenAI. This change was made with Claude Code. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]
|
27b65064d0 | fix(python): support custom blob thresholds without pylance (#4634) | ||
|
Joaquin Hui GomezandXuanwo
|
9da4658d57 |
fix(python): accept with_row_id in take_offsets/take_row_ids on every Table implementation (#4374)
`Table.take_offsets` and `Table.take_row_ids` declare a keyword-only `with_row_id: bool = False`, but none of the three implementations accept it — `LanceTable`, `AsyncTable` and `RemoteTable` all raise `TypeError: ... got an unexpected keyword argument 'with_row_id'` on the documented call. The builders they return already support it (`LanceTakeQueryBuilder.with_row_id()` / `AsyncTakeQuery.with_row_id()`), so this is a signature that never reached the implementations. The base docstrings also don't mention the parameter and still say `take_offsets` returns a `pa.RecordBatch` (it returns a `LanceTakeQueryBuilder`), so those are corrected too. Each implementation now accepts the kwarg and applies `.with_row_id()` to the builder it returns. Tests: sync and async assert `_rowid` appears in the result columns; the remote test uses the suite's request-capturing fixture and asserts `with_row_id: true` reaches the wire (and is absent by default). All three fail on `main` with the `TypeError`. ruff format/check clean. A related mismatch I left alone: the deprecated `LanceTable.create_scalar_index`/`create_fts_index` also drop a `wait_timeout` the base declares — deprecated since 0.25, so not worth touching here. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 LeonSGPandXuanwo
|
3321860e9f |
fix(python): preserve Arrow map scalars in updates (#4448)
Arrow map scalars returned by a table query cannot currently be passed back to `update`: the generic Arrow scalar conversion turns them into a list of tuples, which the SQL literal encoder cannot handle. Empty map scalars become an untyped SQL list and also fail to update a map column. Add a dedicated `MapScalar` conversion that renders its keys and values as a SQL map while preserving the existing null conversion. The regression tests exercise real synchronous and asynchronous table updates, quoted keys, nullable values, empty typed maps, and nested maps. Validation: the complete documented unit-test scope passed locally (1,515 passed, 37 skipped; unchanged baseline: 1,507 passed, 37 skipped). Full doctests passed (71 passed, 6 skipped), and repository-wide Python formatting/lint, Rust formatting/clippy, and Python 3.13 type checking passed. The locally built native wheel also passed all eight regression cases in a separate Python 3.13 environment. Full Linux/S3 integration and all platform matrices remain for CI. The SQL map constructor's existing backend constraints still apply: duplicate keys are rejected, and a nested map containing both nonempty and empty inner maps can fail SQL type unification. This change does not claim arbitrary Arrow map roundtrip support. AI assistance: implementation and local verification used Codex with gpt-6.1-sol agents; the diff and evidence were reviewed before submission. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
e74a7c15b5 |
fix: bump tokio to address RUSTSEC-2026-0386 (#4642)
Upgrade the locked Tokio version from 1.53.1 to 1.53.3 to fix [RUSTSEC-2026-0386](https://rustsec.org/advisories/RUSTSEC-2026-0386) and restore the Rust workflow's `deny` job. The diff changes only Tokio's version and checksum. Validation: - `cargo deny check advisories` — passed. - `cargo deny check advisories bans licenses sources` — passed with the repository's `deny.toml`. - `cargo check --quiet -p lancedb` — passed (dead-code warnings). - PR `deny` job — [passed](https://github.com/lancedb/lancedb/actions/runs/40000920646/job/120576455117). |
||
|
KarthikSandXuanwo
|
702389654e |
fix(python): align pylance extra and preview install indexes (#4556)
Fixes #4248. The preview index still contains pylance 1.0.0b6. uv's default first-index resolution can select it, or backtrack to an old LanceDB release when installing the extra. Meanwhile, the extra still permits pylance 5 even though current main bundles Lance 14. Raise the optional dependency to `pylance>=14.0.0b12`, the version already pinned in the test extra, and keep the lockfile metadata consistent. Document both Fury indexes and uv's `unsafe-best-match` strategy in the Python README and the source-owned quickstart (`docs/web/quickstart.mdx`). Explain the index trust requirement and recommend installing the extra to enforce the version floor. Add CI checks that reject older Lance majors and ensure the pylance and tests extras remain compatible. Run these checks when either dependency manifest changes, so a future Lance major bump cannot silently leave the Python minimum behind. ### Validation - Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects `pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and the unexpected `blob_handling` scanner argument using a real blob table. - Resolved the modified `python/pyproject.toml` with the pylance extra against both indexes: selects `pylance==14.0.0b18`, matching the current Rust pin. - Ran both documented pip and uv installations in isolated environments: `lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both package dependency checks pass. - New regression fails on the original dependency bound (27 rejected-version cases incorrectly accepted), then passes with the fix. Full CI-script suite: 3 tests pass. - `uv lock --check` with both indexes passes; focused Ruff checks pass; `git diff --check` passes. - Repository-wide Ruff reports 49 existing errors in untouched `docs/web-tests` files. Unrelated formatting changes were discarded. Runtime smoke checks used released native wheels; source dependency resolution was tested directly from the modified pyproject. No Rust or Python runtime code changes. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
5a79c7d22d |
ci: run aarch64 Windows tests on main only (#4637)
Run the aarch64 Windows build/test cycle only on pushes to main so PRs wait for a single Windows platform (x86_64). Main keeps the same native aarch64 coverage as before. GitHub evaluates job-level `if` before the matrix strategy is applied, so the aarch64 target is split into its own `windows-arm` job instead of being filtered inside the existing matrix. The job keeps the historical check name. |
||
 LuQQiu
|
b762ec221b |
docs: add Enterprise distributed search reference page (#4625)
Adds the Enterprise reference page at `docs/web/enterprise/distributed-search.mdx`, following the same arrangement as Enterprise architecture. The page describes distributed search and the embedded execution model, with links to shared search guides. Enterprise architecture links to it, and the official sidebar lists it immediately after Architecture in the existing Deployment group. Adds stable anchors for the RaBitQ and `approx_mode` sections linked by the Enterprise guide. Companion: https://github.com/lancedb/sophon/pull/8986 supplies the detailed Enterprise overlay at the same path. Merge this reference-page PR first so the overlay has an existing route to replace. Validation: - Combined-site assembly with both PR branches passed (309 files). - Mintlify build validation and link, anchor, and redirect checks passed. |
||
|
Xuanwo
|
a9535684cc |
chore(deps): bump ctutils to 0.4.4 for RUSTSEC-2026-0385 (#4638)
Fixes RUSTSEC-2026-0385: `ctutils` < 0.4.4 generates masks with branches on secret values (`u128` mask generation / comparisons are not constant-time on some targets). - Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0385 - Affected: `ctutils 0.4.2` pinned in `Cargo.lock`, reached transitively via `digest 0.11` (RustCrypto) - Change: `cargo update -p ctutils --precise 0.4.4`; single lockfile-only change (patch bump, no API impact) The `deny` CI job started failing for every PR head after the advisory was published at 06:45Z today; `main`'s last `deny` run (05:51Z) predates it and does not yet cover this advisory. |
||
|
Wyatt Alt
|
45c4b159ca |
fix: cap the remote job poll interval at one second (#4629)
job.wait() polled on a backoff that doubled up to 5 s, so a job's completion was seen 2.5 s late on average, as long as a small refresh itself takes. Cap the interval at 1 s; the first polls still back off from 200 ms, so a quick job is seen quickly and a long one costs one request a second. Linear: OSS-2704 https://linear.app/lancedb/issue/OSS-2704 |