mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-23 12:12:27 +00:00
b205e7c628c327a10f778a8d6db217c63da96e97
3244
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  adenzhou1350andaden
|
b205e7c628 |
fix(node): prevent table listing limits from wrapping (#4492)
## Summary Prevent large JavaScript table-listing limits from wrapping during the native `u32` conversion. With two tables, `limit: 2 ** 32` currently returns no tables and `2 ** 32 + 1` returns only one. Validate nonnegative safe integers using the existing helper and clamp valid limits above `u32::MAX` before calling the native binding. Apply the same normalization to `tableNames` and `listTables`, including namespace overloads. Undefined, zero, small limits and `u32::MAX` keep their existing behavior. ## Validation Built the real Node native extension from main `0aa842b17f13d1f8ffd2851bc57c222bc5a376ba` with the repository's `pnpm build`, using locked dependencies and reusable build caches. No mocked/released native extension or disabled TypeScript diagnostics. With identical regression tests and that native library in fresh Jest processes: - Baseline: 40 passed, 8 failed; candidate: **48 passed**, no failures. - Both arms: `pnpm run tsc` and `pnpm run docs` passed. - Changed-file Biome check passed. Tests cover wrapping limits, invalid numbers, zero/small/u32 limits, defaults and both overload forms. Validation is the focused connection suite, not the complete repository suite. No performance claim. AI-assisted implementation and verification; the patch and exact tested source bytes were reviewed before publication. Signed-off-by: aden <aden@example.com> Co-authored-by: aden <aden@example.com> |
||
|
adenzhou1350
|
e256b3c2cf |
fix(python): honor namespace table storage options (#4483)
Follow-up on current main `7b3ba4c4` (merge commit `0b0475c1`): resolved the test import conflict, preserving both upstream imports and the original `warnings` import. The final PR diff is still limited to the same storage-option helper/call sites and regression class. AST comparisons preserve every upstream test definition and the original regression class/helper; both changed files compile and pass Ruff 0.15.1 check/format. Fresh locked-environment runtime validation was attempted with `uv run --locked --extra tests --extra dev pytest python/tests/test_namespace.py -q`, but dependency setup failed downloading the locked `pylance==14.0.0b12` wheel due a network connection error. No current-main native test result is claimed. Repository-wide Ruff reports 49 findings and 11 formatting differences in unchanged upstream docs/web-tests files; these were left outside this PR. CI approval and a human review are still pending. --- Original implementation and historical validation (before the main merge): Namespace-backed `create_table` accepts `data_storage_version` and `enable_v2_manifest_paths`, but neither sync nor async creation forwards them. The requested format/manifest settings are silently ignored. Normalize these deprecated arguments into the existing `new_table_*` storage options, matching ordinary connections. Explicit arguments take precedence; the caller's dictionary is not mutated. No Rust or native interface changes. Regression tests inspect persisted tables, including sync/async creation, populated/empty tables, stable/legacy formats, both manifest settings and explicit-option precedence. Validation on Linux/Python 3.12, based on current main `d3390a9`: - Rebuilt the exact-main native extension and reused that identical extension in separate baseline/candidate environments; only the candidate Python namespace wrapper differs. Dependencies came from the unchanged `uv.lock`. - Same complete `test_namespace.py`: baseline 16 regression failures and 39 passing controls; candidate **55 passed**, no skips/errors. - Ruff check/format and diff whitespace checks passed. This validates the local directory-namespace path, not a live Cloud service or the complete repository CI suite. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
13d54db594 |
fix(node): preserve caller-owned storage options (#4480)
`getStorageOptions` writes typed creation options back into the caller's
`storageOptions` object. This makes frozen options throw and causes
overrides to leak into later calls that reuse the storage map.
For example, using `{ storageOptions: shared, enableV2ManifestPaths:
true }` changes `shared.newTableEnableV2ManifestPaths` to `"true"`. A
subsequent creation with just `{ storageOptions: shared }` then uses V2
manifest paths even if `shared` originally requested `"false"`.
Merge the typed overrides into a local map instead. Their existing
precedence, key normalization, and undefined/default behavior remain
unchanged. This affects both `createTable` and `createEmptyTable`; no
native/Rust changes.
Adds six native-database regressions covering frozen creation options,
frozen nested storage options with conflicting defaults, and cross-call
reuse for both APIs.
Validation (Linux x64 / Node 24.19.0 / official 0.41.0-beta.0 native
library):
- Before: all six new regressions fail; an existing namespace
storage-options control passes.
- After: the full connection test file passes, 43/43 (current TypeScript
source, transpile-only).
- Same fix on the matching 0.41.0-beta.0 source: TypeScript compilation,
strict ts-jest connection tests (43/43), and TypeDoc with
warnings-as-errors pass.
- Biome on changed files and whitespace checks pass.
A current-main native rebuild was not run. Released native declarations
predate six unrelated lazy-iterator signatures; current-main
typechecking reports the same six errors before and after this fix (only
downstream line coordinates shift).
### Connection entry-point follow-up
The same ownership problem occurs in `connect(uri, options)` and
`connectNamespace`: normalization assigns a new storage map into the
caller's options. Copy the options before normalization instead. The
`connect(options)` overload already copies its input and is retained as
a control.
Nine added native-database cases cover frozen outer options
(with/without storage options) and repeated reuse, including table
creation and row-count checks. Before this entry-point fix, the
URI/namespace cases fail (6), while the object-overload controls pass
(3). Afterward the complete connection file passes 52/52 on current
TypeScript with the official 0.41.0-beta.0 native library. Applying the
entry-point fix to matching-release source also passes strict
TypeScript, strict ts-jest (52/52), and TypeDoc. Biome passes on both
newly changed files.
The same six current-main/released-native lazy-iterator declaration
errors remain unchanged. No Rust/native rebuild or GPU claim is added.
---------
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
|
||
 adenzhou1350andXuanwo
|
036f4ec53f |
fix(node): preserve options when namespace is omitted (#4479)
The namespace overloads allow `undefined` to select the root namespace,
but overload dispatch treats it as the legacy options argument and
discards the trailing options. For example, `createTable(name, data,
undefined, { mode: "overwrite" })` raises an already-exists error
instead of overwriting.
Preserve the trailing options for `createTable`, `createEmptyTable`,
`tableNames`, and `listTables` when the namespace is omitted. Legacy
options-object calls and explicit namespace arrays keep their existing
behavior.
Adds native-database regressions for overwrite, existOk, and both
listing APIs, parameterized over `undefined` and `[]`.
Validation on Windows / Node 24.19.0:
- Before the fix: all five omitted-namespace cases failed; all five
explicit-root controls passed.
- Current TypeScript source with official, integrity-verified
0.41.0-beta.0 native binaries: the entire connection test file passed,
47/47 (transpile-only).
- The same fix replayed onto the matching 0.41.0-beta.0 source:
TypeScript compilation, strict ts-jest connection tests (47/47), and
TypeDoc passed.
- Biome check/format on both changed files and diff whitespace checks
passed.
No Rust changes. A complete current-main native rebuild remains for CI:
released declarations predate the unrelated lazy-iterator APIs, so
current-main typechecking against those declarations reports six
pre-existing iterator signature errors.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
 LeonSGP
|
57a6cce681 |
fix(python): omit excluded model fields from persistence (#4465)
LanceDB serializes Pydantic rows with `model_dump()`, which omits `Field(exclude=True)`, but currently constructs an Arrow schema containing those same fields. Creating a local table therefore fails when an excluded field is non-nullable, including defaulted transient fields and nested models. Filter unconditionally excluded fields when converting model schemas, including nested structs. Skip excluded source/vector fields when deriving embedding configuration so persistent metadata does not refer to absent columns. Keep `field_names()` as model-field reflection. Tests exercise implicit and explicit schemas, synchronous and asynchronous create/add/reopen/readback, nested fields, defaults and default factories, retained fields, and local deterministic embeddings. Excluded values are not persisted; `to_pydantic` restores declared defaults. Conditional exclusion and readback of required excluded fields without defaults are outside this change. Implemented with assistance from gpt-6.1-sol; final diff reviewed locally. Local validation reuses the native extension matching official base |
||
|
LeonSGP
|
694942a1e0 |
fix(python): convert tuple and NumPy datetime update values (#4464)
`Table.update` and `AsyncTable.update` convert Python values into SQL literals. Tuple arrays and `numpy.datetime64` values currently raise `NotImplementedError`, even when the same values can be ingested into the corresponding local Arrow columns. Register tuples with the existing recursive list converter and format NumPy timestamps as complete nanosecond ISO strings with an explicit UTC offset. `NaT` becomes SQL NULL; subnanosecond units raise an explicit error rather than silently losing precision. Regression tests exercise local synchronous and asynchronous create/update/Arrow reads, date and timestamp units, nanosecond precision, NULL, and existing list/ndarray/Python datetime behavior. Nested and empty tuple arrays are compared with equivalent lists on the actual SDK: successful results must match, and the matching d864 native engine's existing list limitations must produce the same stable error without changing either table. This does not change SQL map handling. Implemented with assistance from gpt-6.1-sol; final diff reviewed locally. Local verification uses Python 3.11.14 and 3.13.11 in existing read-only development environments with the native extension matching base |
||
|
LeonSGP
|
554dc01f4d |
fix(python): preserve custom embedding field metadata (#4463)
`SourceField(json_schema_extra={...})` and
`VectorField(json_schema_extra={...})` currently raise `TypeError`
because the helper supplies the same `Field` keyword twice. Explicit
`None` also fails. This prevents users from adding ordinary Pydantic
JSON schema metadata to embedding fields.
Copy and merge the supplied dictionary before attaching the helper's
internal embedding identity. Caller dictionaries remain unchanged, and
reserved embedding keys retain the real function. Other `Field`
arguments keep their existing behavior. This change covers dictionaries
and `None`; callable `json_schema_extra` remains unsupported by the
existing embedding metadata reader and is outside this fix.
The 11 regression cases exercise both helpers, empty/custom/None
metadata, reserved-key collisions, other Field arguments, and a
deterministic local embedding model through JSON schema generation,
table creation, reopening, and search. On the original source, 9 fail
and 2 controls pass; the candidate passes all 11 on Python 3.11 and
3.13.
Validation uses read-only reused environments and the native extension
built for
|
||
|
LeonSGP
|
8552fd2f5b |
fix(python): prevent embedding aliases from replacing registered functions (#4462)
Registering a different embedding class under an existing alias silently replaced the original class. A deterministic real local SDK table reproduction changed the top result for the same query from alpha to beta after reopening the persisted table. Check the resolved key before inserting a class. Preserve repeated registration of the same class under the same nonempty explicit alias as idempotent, as existing SDK tests rely on it. Repeated implicit class-name registration, including an empty alias, still raises KeyError. Failed registration preserves the mapping and class metadata. Existing embedding tests receive isolated registry mappings; test_util.py remains unchanged. Eight regression cases cover different-class alias collisions, alias/class-name collisions, implicit and empty-alias duplicates, equal class names under distinct aliases, explicit-alias idempotence, and real local creation/reopening/search. This is independent of #4356, which fixes no-alias serialization. Validation on |
||
|
LeonSGP
|
e1e87e1cae |
fix(python): preserve logical rows from Hugging Face datasets (#4461)
Hugging Face `select`, `filter`, and `shuffle` can retain a physical
Arrow table while storing logical row selection/order in an indices
table. Ingesting the physical table then silently writes excluded rows
or the wrong order and can lose duplicate selections.
Read indexed datasets through bounded Arrow-format iteration, and keep
the existing direct batch path for datasets without indices. Apply the
same conversion to sync/async table creation and additions, including
DatasetDict splits. The conversion does not mutate the user's formatting
settings.
Regression coverage includes reordered/duplicate selections, filtering,
shuffling, empty selections, projected numpy/pandas/Arrow formatting,
DatasetDicts, rescanning, and bounded batches.
Validation on base
|
||
|
adenzhou1350
|
9c161869e1 |
fix(node): respect transformers tokenizer option (#4457)
## Summary `tokenizer` is a documented option, but initialization always passed the model name to `AutoTokenizer.from_pretrained`. Preserve the resolved tokenizer option, fall back to the model name when it is omitted, and identify the selected tokenizer in loading errors. Add five initialization regressions covering default model/tokenizer selection, an explicit tokenizer, registry variables, and tokenizer loading errors. They run in native Node child processes so the real ESM import works without changing Jest/CI flags. Model loaders are mocked and local/remote model access is disabled; these tests do not download models or test inference. ## Validation - Before: three regressions fail and two default-selection controls pass. After: all five pass. - New tests plus the existing registry and embedding-registry suites: 48 passed. - TypeScript compilation, changed-file Biome checks, and `pnpm run docs` pass; generated tracked docs are unchanged. - Checked with Node 24.14.1 / pnpm 11.1.1 on Linux, both with default Jest flags and the Linux CI ESM flag. The native binding and declarations came from the official `0.40.0-beta.13` packages, with registry integrity verified. Rebuilding Rust from source timed out during Cargo metadata resolution, before compilation; no Rust code is changed. |
||
|
adenzhou1350
|
7a3709b376 |
fix(remote): preserve request IDs in multipart response errors (#4456)
Keep the request ID when a multipart create/complete response has a missing or incorrectly typed `upload_id`/`version`. These errors currently use an empty ID, unlike JSON parse errors, so callers lose the ID needed to correlate the failed request. Add six cases through `Table.add(...).write_parallelism(2).execute()`, capturing the actual outgoing `x-request-id`: four missing/wrong-type cases fail on unchanged main, while two invalid-JSON controls already pass. After the fix all six pass, and all 15 multipart tests pass. Native Linux validation (locked dependencies, `ci` profile): - `cargo test --offline --profile ci --locked -p lancedb --no-default-features --features remote --lib remote::table::tests::test_multipart_write`: 15 passed. - `cargo fmt --all -- --check`: passed. - `cargo clippy --offline --profile ci --locked -p lancedb --no-default-features --features remote --lib --tests -- -D warnings`: passed. The committed source bytes match the tested source. HTTP endpoints are mocked; this does not claim a live Cloud or full-repository test run. |
||
|
adenzhou1350
|
a12aef2a8c |
fix: return a Python error when querying a closed table (#4455)
`AsyncTable.query()` unwraps the native closed-table error, so querying a closed table raises `PanicException` instead of the documented `RuntimeError`. `AsyncTable.head()` reaches the same path. Propagate `inner_ref()`'s error through `PyResult` instead. Add regressions for closed `query()` and `head()`, with `count_rows()`, repeated close, and normal open-table queries as controls. Validation against separately built native bindings: - Before: 2 regression failures (`PanicException`), 2 controls passed. - After: all 4 regressions passed. - Existing query tests plus the basic database test: 102 passed, 1 skipped on each arm (Torch is not installed). - `cargo fmt --all -- --check`, Ruff, and `git diff --check` passed. The native build used the locked Cargo dependencies with `remote,lancedb/metrics-otel,pyo3/extension-module`. This is source-tree native testing, not a complete wheel-packaging or cloud-service test. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
b9ef048abb |
fix: validate Python read consistency intervals (#4454)
Negative `read_consistency_interval` values currently reach Rust's
panicking `Duration::from_secs_f64` conversion. For example:
```python
from datetime import timedelta
import lancedb
lancedb.connect("./db", read_consistency_interval=timedelta(seconds=-1))
```
Validate the interval with `Duration::try_from_secs_f64` at the three
native connection entry points and return `ValueError` before
constructing the connection or async future. `None`, zero and positive
intervals retain their behavior. The raw native binding also rejects
NaN, infinity and overflow; the public Python API still accepts
`timedelta`, not floats.
Tests cover public sync/async connections, native namespace connections,
real Python namespace clients, the three raw binding entry points and
valid-value controls.
Validation on Python 3.12 / Rust 1.97:
- Same 26 regression cases against separately built native baseline and
candidate: baseline 22 failed / 4 passed; candidate 26 passed.
- Existing basic connection and background-loop regression tests pass.
- `cargo fmt --all -- --check`, Ruff check/format and `git diff --check`
pass.
The native builds used the unchanged repository `ci` profile and
Cargo.lock with `--no-default-features --features
remote,lancedb/metrics-otel,pyo3/extension-module`. Python runtime and
focused test dependency versions came from `python/uv.lock`. Source-tree
tests used a minimal distribution-version metadata fixture derived from
the unchanged Cargo/pyproject manifests; both imported module origins
and native versions were checked in fresh processes. This is not a
complete Maturin packaging, cloud backend or full Python suite result.
No GPU was used.
AI-assisted diagnosis, implementation and validation.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
|
||
|
LeonSGP
|
92ada619d2 |
fix(python): preserve metadata during vector schema inference (#4451)
Creating a table from an Arrow `list` or `large_list` vector field infers a fixed-size vector type, but currently drops that field's metadata. Descriptions and tags such as `lancedb:description` and `lancedb:tag:model` disappear even though the input field carries them. Preserve the original field metadata while replacing its inferred type. Tests use public synchronous and asynchronous table creation, reopening, and reads for floating-point and integer inference, checking the complete schema and row values. Existing dtype inference and nullability remain unchanged. Validation on macOS arm64 with an official native extension built from main `d864e719` using Rust 1.97.0: - Python 3.11.14 and 3.13.11: full `pytest -m "not slow and not s3_test" -q -rs python/tests`: **1,533 passed, 37 skipped, 68 deselected** each, wall times 331.464 s and 416.308 s. - Added public SDK cases on unchanged main: **12 failed**; after the fix: **12 passed** on each runtime. Final installed distributions passed **8 public SDK round trips** each, including fixed-size fields and fields without metadata. - Documentation tests on Python 3.11.14: **77 passed, 6 skipped**. Ruff lint/format, Pyright (**0 errors**), applicable pre-commit hooks (**4 passed, 2 without matching files**), compile checks and `git diff --check` passed. - Comparable unchanged-main full baselines from the same day, native extension and dependency versions: **1,521 passed, 37 skipped, 68 deselected** on each runtime. These were reused rather than rerun. The final local wheel is an explicitly verified repack of the official main wheel, replacing only `table.py` and `RECORD`; all production Python files match the tested source and the native extension is unchanged. No candidate Rust rebuild was performed. Slow/S3 integration tests, the remaining OS/runtime and minimum-dependency matrices, and the workflow's Linux/Python 3.13 doctest job were not run locally. The existing Rust migration draft #2970 and issues #3216/#3222 do not fix this metadata loss in the current Python path; a future migration should preserve the same metadata. During final verification main advanced to `81d52a98` through #4373 and the beta.13 version bump. Neither modified the two files here; the local native and full-suite validation above remain explicitly based on `d864e719`, and the updated native changes have not been tested locally. AI assistance: developed and verified with **gpt-6.1-sol**. |
||
|
LeonSGP
|
fcd26c425a |
fix(python): preserve integer Vector and MultiVector values (#4450)
When `Vector` or `MultiVector` declares an integer Arrow value type, Pydantic currently validates every element as a float. This silently rounds integers above `2**53`, and converts `uint64`'s maximum value into an out-of-range float before table insertion. Use Pydantic's integer schema for integer Arrow value types in both validators. Floating-point types retain the existing validation. Tests exercise model serialization and real local table creation, insertion, and reads for signed and unsigned integer boundaries and floating-point controls, plus dimensions and optional/nullability behavior. Fractional floats are rejected for integer declarations; integral floats remain accepted. This does not extend ANN index support. Validation on macOS arm64, with the current main native extension built using Rust 1.97.0: - Python 3.11.14 and 3.13.11, Pydantic 2.13.5: full `pytest -m "not slow and not s3_test" -q python/tests` suite: **1,537 passed, 37 skipped, 68 deselected** on each runtime. Comparable unchanged-main baselines: **1,521 passed, 37 skipped, 68 deselected** each. - Added cases on unchanged main: **8 failed, 8 passed**; complete `test_pydantic.py` after the fix: **40 passed** on each runtime. - Documentation tests with the existing embeddings dependencies in an isolated environment, Python 3.11.14: **77 passed, 6 skipped**, matching the unchanged-main baseline. The workflow's Linux/Python 3.13 doctest matrix was not reproduced locally. - Repository Ruff lint/format, Pyright, all applicable pre-commit hooks, compile checks, and `git diff --check` passed. Eight public SDK table round trips passed on each final installed wheel runtime. Slow, credential-dependent S3 integration tests, the remaining OS/runtime matrix, and the minimum Pydantic/PyArrow dependency matrix were not run locally. The native extension was built from unchanged current main; the final local wheel repack contains the verified Python change with that identical native extension. A separate candidate maturin packaging attempt triggered Rust recompilation and was stopped, so an official candidate distribution build was not completed. The project requires Pydantic `>=2.7.4,<3`; Pydantic v1 support was removed in #3990. AI assistance: developed and verified with **gpt-6.1-sol**. |
||
|
LeonSGP
|
1e313d3c0d |
fix(python): parse decimal exponent literals exactly (#4449)
Decimal literals such as Decimal("1E-7") and Decimal("1E+3") currently
raise ValueError while building an expression. Parse the coefficient and
exponent exactly, and normalize representable negative scales to zero so
both typed filters and predicate.to_sql() queries preserve the value.
Checked arithmetic bounds the Decimal128 coefficient to 38 digits,
preserves existing scale limits, and rejects nonfinite or overflowing
values. This does not add Decimal256 literal support.
Added 21 cases covering actual Arrow-backed table queries, SQL queries,
zero, signed exponents, precision boundaries, and errors. Both Python
3.11 and 3.13 expression suites pass 123 tests; the baseline fails 10 of
the new cases. Python binding Rust tests pass 13; official make doctest
passes 77 with 6 skips; Ruff, Rustfmt, hooks, Pyright and
workspace/tests/all-features Clippy with -D warnings pass. A fresh
environment restored from the original dependency freeze passes the full
required suite: 1528 passed, 37 skipped, 68 deselected (143.51s); while
optional Torch suites have the same baseline remote mock-server ordering
failure.
Tested against f4544628; observed official main
|
||
  Preston ZhaoandPreston Zhao
|
13df747f18 |
fix(python): return the session supplied to LanceDBConnection (#4433)
LanceDBConnection.session read AsyncConnection.session, but AsyncConnection has no session attribute, so access raised AttributeError. Store the session passed to the constructor and return that object. Assisted by Cursor (Grok). Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com> |
||
|
Preston ZhaoandPreston Zhao
|
cc9f01ae26 |
fix(node): accept exist_ok in createTable (#4432)
TypeScript CreateTableOptions.mode omitted exist_ok even though createTable already forwards that mode to the native client. closes: #2900 --------- Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com> |
||
|
adenzhou1350
|
e489d0cd7b |
fix: return errors for unsupported remote alteration types (#4430)
RemoteTable::alter_columns currently unwraps the JsonDataType
conversion. A valid Arrow input that the remote wire format cannot
represent (for example Utf8View, BinaryView, or a struct containing
Utf8View) therefore panics instead of returning an error.
Propagate conversion/serialization failures as Error::InvalidInput,
including the column path. Unsupported inputs are rejected before any
HTTP request; supported alteration payloads are unchanged. This does not
add server support for view types or change server HTTP-error
classification.
Three new native Rust cases reproduce the panic on unchanged production
and return InvalidInput with the fix. The existing supported-type cases
remain passing.
Validation at head
|
||
|
adenzhou1350
|
4bdc8c3c00 |
fix: wait for the submitted remote index job (#4428)
Addresses the client-side waiting portion of #4402. When create_index returns a job ID, wait_timeout now waits for that submission rather than an already-ready index with the same name. It propagates JobFailed / JobCancelled and bounds job polling by the requested timeout. Older servers without job IDs keep name-based waiting, calls without wait_timeout remain fire-and-forget, and the existing two-hour maximum is preserved. The job wrapper is shared with create_index_async; no synchronous server-side config validation is added. Rebased on main |
||
 Mohammad Hijjawi
|
712a8b8699 |
fix(python): write plain Enum fields of a LanceModel as their values (#4357)
Since #3232, `pydantic_to_schema` maps an `Enum` field to the Arrow type of its values, including enums without a `str`/`int` mixin. Writing such models still failed: `model_to_dict()` returns `model.model_dump()`, which keeps the Enum members, and Arrow can only convert members that are also `str`/`int` instances. ```python class Color(Enum): RED = "red" class Item(LanceModel): color: Color tbl = db.create_table("items", schema=Item) tbl.add([Item(color=Color.RED)]) # pyarrow.lib.ArrowTypeError: Expected bytes, got a 'Color' object ``` `create_table(data=[...])` fails the same way, and int-valued enums raise `ArrowInvalid`. `model_to_dict()` now replaces Enum members with their values, including in nested models and lists. Lists of plain scalars (vectors in particular) are returned without being walked, so the per-row cost stays about the same as `model_dump()` alone (21.3µs vs 20.3µs for a 1536-dim model). Reading back is unchanged: Pydantic already validates the stored value into the Enum member. Added `test_plain_enum_values_are_written`, which covers str and int valued enums, `Optional` and `List` fields, and an Enum inside a nested model, and checks that `to_pydantic` round-trips the rows. It fails with the `ArrowTypeError` above before the change. `test_pydantic.py`, `test_embeddings.py`, `test_table.py` and `test_db.py` pass, and `ruff format --check` / `ruff check` are clean. |
||
|
Mohammad Hijjawi
|
5f9a8af27c |
fix(python): keep embedding functions registered without an alias readable (#4356)
`EmbeddingFunctionRegistry.register()` documents that the class name is
used when no alias is given, and it does register the class under that
name. But it stored the alias itself (`None`) as
`__embedding_function_registry_alias__`, which is what
`function_to_metadata()` writes into the table schema. A table using
such a function gets `"name": null` in its `embedding_functions`
metadata, and any later read of the functions fails:
```python
@get_registry().register()
class MyEmbed(TextEmbeddingFunction): ...
f = MyEmbed.create()
class Schema(LanceModel):
text: str = f.SourceField()
vector: Vector(4) = f.VectorField()
tbl = db.create_table("t", schema=Schema)
tbl.add([{"text": "hi"}]) # KeyError: None
tbl.search("hi") # KeyError: None
```
This stores the key the class is registered under instead, so the class
name round-trips. For aliased classes the key is the alias, so their
metadata is unchanged. The TypeScript registry already falls back to
`ctor.name` in the same way.
Added `test_embedding_function_registered_without_alias`, which creates
a table with such a function, reopens it, and adds a row. It fails with
`KeyError` before the change. `test_embeddings.py`, `test_pydantic.py`,
`test_table.py` and `test_db.py` pass, and `ruff format --check` / `ruff
check` are clean.
|
||
|
Mohammad HijjawiandXuanwo
|
a0946282ca |
fix(python): return original scores from hybrid search with normalize="rank" (#4343)
## Summary With `rerank(..., normalize="rank")`, a hybrid query returns ranks (1.0, 2.0, 3.0, ...) in the `_distance` and `_score` columns instead of the real vector distances and BM25 scores. `_combine_hybrid_results` converted both result sets to ranks first, and only then captured `original_distances` / `original_scores`. So the step that restores the original values after reranking put the ranks back. With `normalize="score"`, the real values are returned as expected. Repro (RRFReranker(return_score="all"), vector distances 0 / 25 / 100): | normalize | `_distance` returned | `_score` returned | | --- | --- | --- | | `"score"` | 0.0, 25.0, 100.0 | 0.47, 0.67, None | | `"rank"` (before) | 1.0, 2.0, 3.0 | 1.0, 2.0, None | | `"rank"` (after) | 0.0, 25.0, 100.0 | 0.47, 0.67, None | ## Changes - Capture the original `_distance` / `_score` columns before converting them to ranks, then normalize the ranked column. The reranker input is unchanged; only the values restored afterwards are fixed. `AsyncHybridQuery` uses the same function, so both sync and async are covered. ## Test plan - [x] New `test_hybrid_return_all_restores_original_scores` in `test_rerankers.py`, parametrized over `score` / `rank`: the `rank` case fails before the fix and passes after - [x] `pytest python/tests/test_rerankers.py python/tests/test_hybrid_query.py` - [x] `ruff format --check` / `ruff check` I have no local Rust toolchain, so I ran the tests against the released 0.39.0 native module with the changed Python files overlaid. The changed functions are identical between 0.39.0 and main. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
62d83f2ea2 |
fix(python): expect oidc credential-type header on refreshed tokens (#4632)
## Summary #3971 added `test_zero_lifetime_token_is_refreshed`, but its expected headers omit `x-lancedb-credential-type: oidc`, which `OAuthProvider.get_headers()` has emitted since #2631/#4070. The test fails against current main and breaks the PR CI matrix for every open PR. ## Changes - Align the two zap assertions in `test_zero_lifetime_token_is_refreshed` with the provider contract (see `test_token_refresh` and `test_sync_method`). ## Tests - `pytest python/tests/test_header_provider.py -q` |
||
 Prince Jain
|
7f932b8552 |
fix(nodejs): reject listNamespaces limit above i32::MAX instead of wrapping (#4584)
## Summary Fixes #4583. `Connection::list_namespaces` narrowed `Option<u32>` to `i32` with an unchecked `as` cast, so a JavaScript `limit` between 2^31 and 2^32 - 1 (e.g. `3000000000`) wrapped negative before reaching `ListNamespacesRequest`. The server received a negative page size instead of a clear client-side error. Replaced the cast with a `checked_limit` helper using `i32::try_from`, returning `napi::Error::from_reason` on overflow, matching the existing `try_from` validation used for `limit` elsewhere in the same file. ## Tests Unit tests cover the wrap case (`u32` values above `i32::MAX` reject), the boundary (`i32::MAX` passes), and `None`/zero passthrough. `cargo test -p lancedb-nodejs` passes 8/8 and `pnpm build` compiles the binding and TypeScript definitions cleanly. |
||
 Yingzi Zhang
|
55897d7c3b |
fix(python): preserve Arrow schema when merging reranker scores (#4332)
Fixes #4331 With `return_score="all"`, model rerankers reconstruct merged rows without their Arrow schema. This changes fixed-size float32 vectors into variable-length float64 lists, loses metadata, and can overflow valid uint64 row IDs. Unify the original input schemas before inserting placeholder scores, then use the unified schema when reconstructing the merged table. Row merging and reranking behavior stay the same. Capturing the schema first also preserves float64 scores without conflicting with the float32 null placeholders. Validation: - New regression cases: 5 failures and 5 passing controls before the fix; all 10 pass afterward. - Reranker, hybrid-query and TypeSafe reranker tests: 150 passed, 17 skipped. Optional live model/API tests were not run. - The fake-client tests that use random metadata passed again with seeds 7 and 41. - Repository-wide Ruff lint/format checks, configured Pyright check and applicable pre-commit hooks pass. - Tests used the locally built Rust/Python extension and the repository's locked uv environment. No Rust code changed. AI disclosure: I used Codex for investigation, implementation, tests, local checks, and drafting this description. |
||
 lancedb-gatefixer[bot]
|
624defd109 |
fix: preserve sliced child validity during struct merge (#4261)
## Root cause The pinned Lance Arrow crate rebuilt a sliced struct child with its original Arrow offset but supplied a validity bitmap containing only the visible rows. Arrow requires the bitmap to cover the offset as well, so filtered vector searches involving nullable nested fields could panic with `null_bit_buffer size too small`. ## Fix - Carry an offset-aligned validity bitmap when propagating parent nulls to a sliced child. - Vendor the pinned `lance-arrow` crate until a Lance release includes the fix; include it in Maturin source distributions. - Add unit regressions for sliced children and parent bitmaps, plus a record-batch merge regression. ## Validation - The issue's 8,600-row Python reproducer completes its filtered flat dot search and returns 8,448 rows without a panic. - The targeted Rust merge regression and two focused Python query tests pass. - Rust formatting, Ruff formatting and lint, the vendored crate tests, and the workspace Cargo checks passed. The Maturin source archive includes the patched crate. Fixes #4260 <!-- lance-gatekeeper-fix:v1 agent=e6045f701d9b708ee26528ba36353aba generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
66b5b5bbdc |
fix(python): clarify unavailable CUDA index acceleration (#4249)
## Summary - preflight CUDA availability and initialization before accelerated index training - direct unavailable or misconfigured CUDA environments to the explicit CPU path - cover both legacy and unified Python index APIs ## Root cause LanceDB delegated accelerator requests directly to PyLance. When PyTorch lacked CUDA support or CUDA initialization failed, users received a low-level assertion or runtime error with no indication that omitting `accelerator` or setting `accelerator=None` creates the index on CPU. The repair validates CUDA before training and preserves explicit accelerator semantics without silently changing the requested backend. ## Scope This PR addresses only the separately reproduced CUDA availability and initialization error. It does not close #1766; the reported query failure after successful CUDA index creation remains open pending reproducible evidence. ## Validation - `uv run --extra tests pytest python/tests/test_table.py -k create_index -q` (10 passed) - `ruff format .` - `ruff check .` - CPU-only integration check confirms the actionable fallback message Related to #1766 <!-- lance-gatekeeper-fix:v1 agent=d05c37237b04c2c8f88395a82e00bc67 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 SIDDARTHA REDDY
|
d37a6b76a4 |
fix: gate remote job error construction behind the remote feature (#4233)
## Summary Fixes #4142: `lancedb` failed to compile without the `remote` feature. `TerminalResult::decode` in `rust/lancedb/src/job.rs` constructed the `Error::Http` variant unconditionally, but that variant is declared behind `#[cfg(feature = "remote")]` in `rust/lancedb/src/error.rs`. Since the crate's default feature set is empty, a plain `lancedb = "0.38"` dependency failed with: ``` error[E0599]: no variant named `Http` found for enum `error::Error` --> rust/lancedb/src/job.rs:102:40 ``` The module itself is not remote-only (it carries local job support), so instead of gating `pub mod job`, both `Error::Http` constructions now go through a small `remote_job_error` helper that resolves to `Error::Http` with the `remote` feature and falls back to `Error::Runtime` without it. A `request_id` is only ever `Some` for remote jobs, so the fallback is unreachable in practice without the feature. ## Testing - Reproduced the reported failure on current `main`: `cargo check -p lancedb --no-default-features` failed with the exact E0599 errors from the issue before the change, and passes after it. - `cargo check -p lancedb --features remote` passes with no errors and no new warnings. - Added two regression tests in `rust/lancedb/src/job.rs` (`decode_missing_result_uses_feature_appropriate_error`, `decode_unparsable_result_uses_feature_appropriate_error`) asserting the error variant matches the feature configuration. The test code compiles under the fixed configuration (`cargo check --tests` passes), and the branching logic was executed standalone against both feature cfgs (3/3 green with and without `remote`). The full `cargo test` binary link could not complete on the small build VM available here (the linker stalled), so CI coverage of the new tests is appreciated. - `cargo fmt --all -- --check` is clean. ## Disclosure AI assistance was used while preparing this change (code search, editing, and verification commands were agent-driven); the diagnosis, fix design, and all test results above were verified by actually running the builds. |
||
 Joaquin Hui Gomez
|
82bfe7d38e |
fix: table_names should not list loose .lance files as tables (#4203)
`table_names()` enumerates with `read_dir`, which returns files as well
as directories, filtered only on the `.lance` extension — so a loose
`foo.lance` *file* at the database prefix is reported as a table.
`list_tables()` already applies the correct rule ("Only child
directories can be tables"), and
`test_listing_ignores_non_table_children` documents it, so the two
listing APIs currently disagree.
The practical damage is in `drop_all_tables()`, which is built on
`table_names`: with a loose `.lance` file present it deletes some real
tables, fails on the file with `os error 20 (NotADirectory)`, and then
fails the same way on every retry — the database can never be emptied
through the API, and the partial drop is not recoverable.
`table_names` now lists via `list_with_delimiter` and filters
`common_prefixes` through the same `table_name` helper `list_tables`
uses; the sort/`start_after`/`limit` handling is unchanged.
`drop_tables` additionally attempts every name before returning, still
reporting the first error unchanged (kept rather than an aggregate error
so the mapped exception types in the bindings stay exactly as they are).
Extended the non-table-children test to cover `table_names`, plus tests
for the drop behavior — all three fail without the fix. Full `cargo test
-p lancedb --lib --features remote` 1320 passed on the repo-pinned 1.97
toolchain; clippy/fmt clean.
|
||
 陈志谦
|
9a59091376 |
fix(python): add missing space in the hybrid search error message (#4202)
`LanceHybridQueryBuilder._validate_query`'s ValueError concatenated two adjacent string literals without a separator, rendering "...explicitly for hybrid search.**But** not both.". Added the trailing space. String-only change. |
||
|
陈志谦
|
d4b2290bab |
fix(python): correct the voyageai reranker env var docstring (#4201)
`VoyageAIReranker`'s `api_key` docstring said it falls back to `OPENAI_API_KEY`, but the class actually reads `VOYAGE_API_KEY` (and raises a "VOYAGE_API_KEY not set" error without it) — the docstring was copy-pasted from the OpenAI reranker. Docstring-only fix. |
||
 Mohd Quamar TyagiandXuanwo
|
79e256959f |
fix(python): accept pathlib.Path in get_uri_scheme and get_uri_location (#4132)
This PR converts the uri parameter to string before calling urlparse in get_uri_scheme and get_uri_location, allowing pathlib.Path objects to be passed seamlessly without raising a TypeError. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Madan Kumar
|
0c3a1c92c8 |
fix(python): don't crash inferring schema for an all-null vector column (#4034)
Fixes #4033 `create_table(data=...)` without an explicit schema raised `IndexError: index out of bounds` when a `vector`/`embedding`-named list column was all-null. `_modal_list_size` infers the dimension with `pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips nulls, so an all-null column produces a zero-length result and the `[0]` subscript blows up. The sibling `_infer_vector_dim` already guards the identical `pc.mode(...)` with a `len == 0` check and returns `None`; `_modal_list_size` just didn't. This makes `_modal_list_size` return `Optional[int]` and return `None` when every value is null, and the call site then leaves the column as a variable-length list instead of inventing a fixed size — the dimension is genuinely unknowable when there are no non-null rows. Added a regression test that creates a table from an all-null `list<float32>` vector column and asserts it succeeds, the column stays a `list<float32>`, and the rows are queryable. It fails on `main` with the `IndexError` and passes with this change. |
||
 Vinayak Vathare
|
ae97d43f4b |
fix(python): preserve valid embeddings when an OpenAI batch is rejected (#4005)
Retries individual inputs one at a time when the OpenAI API rejects a batch with BadRequestError, instead of discarding the whole batch. Valid inputs now still get embeddings; only the actually-bad input resolves to None, and the log names the exact failing input. Fixes #1677 |
||
 Zhewen TanandXuanwo
|
6d92307fc3 |
fix(python): accept tuple and Arrow vector queries (#3972)
## Summary Sync `Table.search()` declares tuple, `pyarrow.Array`, and `pyarrow.ChunkedArray` query vectors as supported, and the async query path already accepts them. The sync query builder only classified lists and NumPy arrays as vectors, however, so these inputs were routed through embedding/FTS resolution and then rejected. Arrow and tuple inputs also did not contribute a query dimension when selecting among multiple vector columns. This change: - recognizes the supported tuple and Arrow containers as sync vector queries for both `auto` and explicit `vector` query types; - converts them to the existing internal `float32` NumPy representation at the sync builder boundary; - infers dimensions from tuple, primitive Arrow, and list-valued Arrow queries so schemas with differently sized vector columns select the matching column. The regression test uses two vector columns with dimensions 2 and 3. It verifies tuple, Arrow array, and chunked Arrow queries through both sync query modes and confirms the dimension-2 column is selected. ## Test plan - `pytest python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers -q` (6 passed) - `pytest python/tests/test_util.py python/tests/test_query.py python/tests/test_table.py -q -m "not slow and not s3_test"` (308 passed, 1 skipped, 1 deselected) - `pytest python/tests -q -m "not slow and not s3_test" --ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected) - `ruff format --check python` - `ruff check python` - `cargo fmt --all --check` Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 primorLee
|
548a190321 |
fix(python): refresh zero-lifetime OAuth tokens (#3971)
## Summary - treat an OAuth `expires_in` value of zero as an explicit immediate expiration - refresh zero-lifetime tokens instead of caching them indefinitely - add a regression test covering two consecutive header requests ## Tests - `pytest python/tests/test_header_provider.py -q` (16 passed) - `ruff format --check python/lancedb/remote/header.py python/tests/test_header_provider.py` - `ruff check python/lancedb/remote/header.py python/tests/test_header_provider.py` |
||
 Li Zinuo
|
a1230cf9a4 |
fix: preserve unbounded plain queries in namespace pushdown (#3927)
## What Preserve unbounded plain-query semantics in the Rust Namespace `QueryTable` adapter. Plain queries without an explicit limit are now encoded with `i32::MAX`. Explicit limits remain unchanged, while vector and FTS queries continue to use the default top-k of 10. ## Root cause The Rust Namespace adapter converted a missing limit using: ```rust q.limit.unwrap_or(10) ``` This turned an unbounded plain query into an explicit `k=10`, silently truncating the results returned by Namespace pushdown. ## Tests Added regression coverage for unbounded and explicitly limited plain queries, while verifying that vector and FTS queries retain their default limit. ```shell cargo fmt --all -- --check cargo test --quiet --features remote -p lancedb \ test_convert_to_namespace_query --lib ``` close #3926. |
||
|
lancedb-gatefixer[bot]
|
40af9f350e |
fix(python): avoid deadlock for query-backed iterators (#3820)
## Summary - move synchronous iterator preparation off LanceDB's singleton background event loop - keep the loop available when an input iterator executes a synchronous table query - cover query-backed iterators for both `Table.add` and `create_table` ## Root cause The async implementations prepared input data on the singleton background loop and peeked at the first item of iterators there. If that iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the query on the same loop and blocked waiting for it, so neither operation could progress. ## Validation - `ruff format` on the changed Python files - `ruff check .` - `cd python && uv run --frozen --extra tests pytest python/tests/test_table.py -q` (138 passed) - the broader non-slow suite was attempted but hit a native segmentation fault in the external DataFusion `register_table_provider` path; an isolated control using the pre-change inline preparation behavior reproduced the same unrelated crash Fixes #2107 <!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
102518ebda |
fix(python): preserve nested fields in filtered scans (#3816)
## Summary - add a Python regression for filtered scans over deeply nested list/struct values - verify the public query path returns every nested sibling field without an Arrow schema mismatch ## Root cause Older Lance scanner code split filter projections into partial nested arrays and did not recursively merge List<Struct> children, so a sibling field could be omitted from the returned array while remaining in the declared schema. That merge was fixed upstream in lance-format/lance#4838, and the Lance revision currently pinned by LanceDB contains that fix. This test makes the repair durable at the LanceDB Python API boundary. ## Validation - cd python && uv run --extra tests pytest python/tests/test_query.py::test_filter_preserves_deeply_nested_struct_fields -q - cd python && uv run --extra tests pytest python/tests/test_query.py -q - uv run --project python --extra tests --extra dev ruff check . - uv run --project python --extra tests --extra dev ruff format --check python/python/tests/test_query.py Fixes #2217 <!-- lance-gatekeeper-fix:v1 agent=9ddc0efd1f13888b2a7a896881c6e68a generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
a98667749d |
fix(python): verify Lance-created IVF_RQ compatibility (#3793)
## Summary - add regression coverage for an IVF_RQ index created through the Lance Python API and consumed through LanceDB - verify a fresh LanceDB handle can list and query the externally-created index ## Root cause Lance 4.0.0-beta.5 could serialize `rotate_mat_position` as null, while the Lance 2.0.0 reader used by LanceDB 0.29.2 required a `u32`. Upstream Lance fixed the incompatible IVF_RQ metadata handling and index versioning, and the current LanceDB dependency includes that repair. LanceDB was missing coverage for the reported mixed-API workflow, allowing the compatibility regression to go undetected. ## Validation - `cd python && uv run --extra tests pytest python/tests/test_index.py -q` (24 passed) - `uv run --project python --extra dev ruff format .` - `uv run --project python --extra dev ruff check .` - `uv run --project python --extra dev ruff format --check python/python/tests/test_index.py` - `uv run --project python --extra dev ruff check python/python/tests/test_index.py` Fixes #3092 <!-- lance-gatekeeper-fix:v1 agent=7d8c849bd8421be65d32355c42a60b8a generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
ba7b7f15c2 |
fix(node): reject musl binaries with unresolved AVX-512 symbol (#3791)
## Summary - inspect the assembled x64 musl addon for the unresolved AVX-512 distance-table symbol before npm publication - fail the publish lifecycle with an actionable error if the broken symbol is present - add regression coverage using the dynamic-symbol output from the affected 0.27.x artifact ## Root cause LanceDB 0.27.x could enable the Rust AVX-512 distance-table callsite when the separate C kernel build did not emit its object, leaving `sum_4bit_dist_table_32bytes_batch_avx512` undefined in the musl addon. The current Lance dependency includes the upstream split-kernel cfg fix, and the published 0.33.0 musl artifact no longer has the undefined symbol, but the LanceDB publish path did not verify that property before releasing an addon. This adds a release guard around the corrected implementation so the same invalid binary cannot be published again. ## Validation - `pnpm lint` - `pnpm run build` - `pnpm test --runInBand` (680 passed, 5 skipped) - `pnpm run docs` - checked published artifacts: 0.27.1 is rejected for the unresolved symbol; 0.33.0 is accepted Fixes #3200 <!-- lance-gatekeeper-fix:v1 agent=4094b9b6de4b43665a9ff50fa35322fb generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Lance Release
|
9f0f6400bc | Bump version: 0.41.0-beta.12 → 0.41.0-beta.13 | ||
|
Xuanwo
|
a5079b0fc6 | chore: update lance dependency to v15.0.0-beta.1 (#4628) | ||
|
Xuanwo
|
dd4a527727 |
fix(index): preserve registered scalar plugin type names (#4627)
Native `list_indices()` drops registered scalar plugin types absent from LanceDB's fixed enum, and `index_stats()` reports them as `Unknown`. Preserve their registered display names through `IndexType::Custom(String)`. Resolve these names through Lance's shared `IndexDetails` registry entry point. Registered types such as MinHashLSH become visible; system and unregistered types remain excluded. Existing builtin aliases retain their canonical names. Rust consumers with exhaustive `IndexType` matches need to handle the added variant. Manual validation with [Lance #10224](https://github.com/lance-format/lance/pull/10224) at `fcc070f88a`: a downstream statistics-only plugin appears as `Custom("Sketch")` in native listing and statistics. Its `load_index` deliberately returns an error, and repeated calls do not construct additional plugin instances, verifying metadata-only lookup through the shared registry. |
||
 Jack Ye
|
f9680806c9 |
feat(python): expose materialized view auto-refresh creation options (#4621)
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
|
||
|
Jack Ye
|
1564ec10c8 |
feat: add Bootstrap split strategy for sampling with replacement (#4597)
## Context
`StreamingDataset`'s shipped split strategies (`split_random`,
`split_sequential`, `split_hash`, `split_calculated`) all share one
trait: every row in the base table is assigned to exactly one split,
exactly once. They only differ in *how* that one-to-one assignment is
chosen -- none of them can express "give me a resampled stream of N
rows, allowing repeats."
That's a distinct, common need in ML training: **the bootstrap** -- the
statistical technique (Efron, 1979) of estimating a dataset's properties
by resampling it *with replacement*. It's the same sense scikit-learn's
bagging classifiers mean by `bootstrap=True`. `SplitStrategy::Bootstrap`
adds exactly this to the data loader: sample rows uniformly with
replacement from the (optionally filtered) base table, independently per
split.
## Main use cases
- **Bagging / ensemble training** (e.g. random forests): each split
trains on an independent bootstrap resample of the training set.
- **Oversampling a rare class**: filter to the minority class, then
sample it up to whatever count balances it against the majority, instead
of exhausting its small set once per epoch.
- **Fixed-size training steps independent of dataset size**: want
exactly N batches per epoch regardless of whether the table has 50 rows
or 50 million? Set `num_samples` directly instead of it being implicitly
`len(dataset)`.
- **Weighted sampling without a weight column**: filter to "eligible"
rows (e.g. a windowed-sampling workload filtering to valid window-start
positions), then bootstrap the filtered set -- an item with more
eligible rows is naturally drawn proportionally more often, with no
explicit per-row weight needed.
## Example (an actual run, not a hypothetical one)
A 5-row base table (`row_id` 0..4), sampled via `Bootstrap {
num_samples: 8, sizes: Fixed(2), seed: Some(42) }`:
```
row_id split_id
0 0
2 0
4 0
4 0 <- row 4 repeated within split 0
4 1 <- row 4 also drawn in split 1 (independent draw)
4 1
1 1
2 1
```
8 output rows from a 5-row table -- not 5. Row 4 drawn 4 times total,
row 3 never drawn. That's with-replacement sampling actually behaving
like with-replacement sampling, not an even round-robin dressed up with
a new name.
## What changed
Adds `SplitStrategy::Bootstrap` (uniform sampling with replacement,
optionally filtered, independently seeded per split -- a given split's
content doesn't depend on how many other splits there are, the same
elastic-determinism property documented for the loader's other
strategies). No new `PermutationBuilder` Rust method was needed:
`with_split_strategy` already accepts any `SplitStrategy` value
generically. Python binding:
`PermutationBuilder.bootstrap_sample(num_samples, *, ratios=None,
counts=None, fixed=None, seed=None, split_names=None)`, mirroring
`split_random`'s existing shape.
Also adds a generic `split_strategy` parameter to `StreamingDataset`'s
constructor (a callable taking/returning a `PermutationBuilder`), since
it previously could only reach `split_random`/`split_sequential` even
though `split_hash`/`split_calculated` already shipped. This unblocks
`Bootstrap` and any future strategy without further constructor changes.
|
||
|
Jack Ye
|
a4544c5472 |
feat: configure automatic refresh when declaring computed columns (#4607)
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
|
||
  
|
d97df0c82b |
fix: update trust-store pins for the eMudhra CA compromise (#4624)
Cargo.lock: webpki-root-certs 1.0.10; webpki-roots 1.0.10 and rustls-webpki 0.103.16 are already current. docs/web-tests/rs/Cargo.lock: rustls-webpki 0.103.16. python/uv.lock: certifi 2026.10.19, the only entry that changes. The root crate drops the root CA whose intermediates were compromised (Mozilla bug 2083449), and rustls-webpki 0.103.16 rejects chains that carry the compromised eMudhra intermediate keys. certifi 2026.10.19 is the release the certifi advisory names as fixed: it drops the same root and keeps the issuer's other roots, so HTTPS connections that chain to them keep working. Relative to the previous 2026.6.17 pin it also adds Telia EC and RSA TLS Root CA v3 and SECOM TLS ECC and RSA Root CA 2024 and drops Atos TrustedRoot 2011. The webpki-root-certs entry is written by hand with the crates.io checksum. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Esteban Gutierrez <esteban@lancedb.com> Co-authored-by: Claude Opus 5.1 <noreply@anthropic.com> |
||
 Colin Patrick McCabe
|
0fbc0981dd |
feat: document all OAuthFlowType entries in authentication.mdx (#4565)
Previously, authentication.mdx only documented a subset of the existing OAuthFlowType entries. We should document all of them. |