`retry_with_exponential_backoff` retries every exception except an auth
error (`python/python/lancedb/embeddings/utils.py:298-318`). For Cohere,
Gemini, Jina and VoyageAI a missing API key raises from
`api_key_not_found_help`, and a missing provider package raises
`ImportError` everywhere. Neither can succeed on a retry, yet both sleep
through all seven rounds and then surface wrapped in `Exception("Maximum
number of retries...")`.
Before, on `142d032`, a VoyageAI function with no key set (script below,
`time.sleep` recorded, `random.seed(0)`):
```
$ python lance_retry_deterministic.py python/python
raised: Exception('Maximum number of retries (7) exceeded.', ValueError('Please set the VOYAGEAI_API_KEY environment v
attempts: 8 sleeps: [4, 13, 37, 93, 280, 788, 2811]
slept 4026 s before a missing-API-key error surfaced
```
After:
```
$ python lance_retry_deterministic.py python/python
raised: MissingApiKeyError('Please set the VOYAGEAI_API_KEY environment variable.')
attempts: 1 sleeps: []
```
<details><summary>lance_retry_deterministic.py</summary>
```python
# loads the pure-Python embeddings package from the tree, native extension stubbed
import os, random, sys, time, types
TREE = sys.argv[1]
pkg = types.ModuleType("lancedb"); pkg.__path__ = [TREE + "/lancedb"]; sys.modules["lancedb"] = pkg
native = types.ModuleType("lancedb._lancedb"); native.__getattr__ = lambda name: None
sys.modules["lancedb._lancedb"] = native
import pyarrow as pa
from lancedb.embeddings import get_registry
random.seed(0)
os.environ.pop("VOYAGE_API_KEY", None)
slept = []
time.sleep = slept.append
func = get_registry().get("voyageai").create(name="voyage-3")
try:
func.compute_source_embeddings_with_retry(pa.array(["hello"]))
except Exception as e:
print("raised:", repr(e)[:110])
print("attempts:", len(slept) + 1, "sleeps:", [round(s) for s in slept])
print("slept %.0f s before a missing-API-key error surfaced" % sum(slept))
```
</details>
The message names the wrong variable: `voyageai.py:501` reads
`VOYAGE_API_KEY`. I left that for a separate fix.
`api_key_not_found_help` now raises `MissingApiKeyError`, a `ValueError`
subclass, so existing `except ValueError` callers still catch it. That
error and `ImportError` join the 401 exemption from #2995. A plain
`ValueError` is still retried, because `bedrock.py:171` wraps boto3
throttling in one. No issue exists for this. OpenAI is not covered: with
no key, `openai.OpenAI()` raises `OpenAIError`, which has no status
code, so it still retries. I can follow up on that.
### Verification
```
# lance_stub: an empty stand-in for lancedb._lancedb, loaded from the source tree
$ pytest -p lance_stub --noconftest python/tests/test_embeddings.py -k "retry or 401 or deterministic" -q
5 passed, 23 deselected in 0.59s
# utils.py from 142d032, tests kept:
2 failed, 3 passed, 23 deselected in 0.74s
$ pytest -p lance_stub --noconftest python/tests/test_voyageai_embeddings.py -q
18 passed in 0.18s
# not run: the full suite or pyright; the native extension did not build here
```
Baseline on `142d032` has 2 of these tests. The new ones cover a missing
key, a missing package, and a wrapped throttling error that must still
retry. ruff 0.16.10 check and format are clean.
@wjones127 you merged the 401 change. Is a `ValueError` subclass the way
you want to mark this?
Co-authored-by: Xuanwo <github@xuanwo.io>
Addresses option 2 of #4440.
tokenize() builds the FTS analyzer on the client. For jieba/* and
lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and
when they’re missing you only got lance’s raw Invalid directory path:
... error, even though server-side search works.
Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps
the original cause and adds that tokenize() runs on the client, which
directory the files need to be in, and that for remote tables they must
match the server’s. I also updated the tokenize docs (Rust, Python, TS,
generated JS docs) to say a mismatch can’t be detected, so tokens may
differ from the server’s index.
Behavior when the files exist is unchanged. Detecting a mismatch needs a
server-side tokenize endpoint (option 1), so I left it out. The hint
isn’t limited to remote tables, since Table::as_native is slated for
removal and the wording fits local tables too.
Tests: Rust test with mocked jieba/default index_details and an empty
model dir, one checking other tokenizer errors don’t get the hint, and
Python sync/async tests.
Invalid `wait_timeout` values currently reach
`Duration::extract(...).unwrap()` in both native index-creation
bindings. A normal Python argument error becomes
`pyo3_runtime.PanicException`, which is not caught by `except
Exception`.
Propagate the extraction error with `map(...).transpose()?` before
constructing the index operation. Valid `None` and `timedelta` behavior
is unchanged.
Tests cover both `create_index` and `create_index_async`, three invalid
types, valid `None`/`timedelta`, and no index created after rejection.
Validation against main `0aa842b`, Python 3.12, uv-locked dependencies,
freshly compiled PyO3 extension for each arm:
- Unmodified source: 6 new regressions fail with `PanicException`; 6
controls pass.
- Fixed source: 12/12 pass; broader index-creation regression selection:
18/18 pass.
- `cargo fmt --all --check`, Ruff check/format, and diff check pass.
The broader selection includes the 12 cases above. No cloud, full-suite,
or performance claim.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Follow-up on current main `7b3ba4c4` (merge commit `0b0475c1`): resolved
the test import conflict, preserving both upstream imports and the
original `warnings` import. The final PR diff is still limited to the
same storage-option helper/call sites and regression class. AST
comparisons preserve every upstream test definition and the original
regression class/helper; both changed files compile and pass Ruff 0.15.1
check/format.
Fresh locked-environment runtime validation was attempted with `uv run
--locked --extra tests --extra dev pytest python/tests/test_namespace.py
-q`, but dependency setup failed downloading the locked
`pylance==14.0.0b12` wheel due a network connection error. No
current-main native test result is claimed. Repository-wide Ruff reports
49 findings and 11 formatting differences in unchanged upstream
docs/web-tests files; these were left outside this PR. CI approval and a
human review are still pending.
---
Original implementation and historical validation (before the main
merge):
Namespace-backed `create_table` accepts `data_storage_version` and
`enable_v2_manifest_paths`, but neither sync nor async creation forwards
them.
The requested format/manifest settings are silently ignored.
Normalize these deprecated arguments into the existing `new_table_*`
storage
options, matching ordinary connections. Explicit arguments take
precedence;
the caller's dictionary is not mutated. No Rust or native interface
changes.
Regression tests inspect persisted tables, including sync/async
creation,
populated/empty tables, stable/legacy formats, both manifest settings
and
explicit-option precedence.
Validation on Linux/Python 3.12, based on current main `d3390a9`:
- Rebuilt the exact-main native extension and reused that identical
extension
in separate baseline/candidate environments; only the candidate Python
namespace wrapper differs. Dependencies came from the unchanged
`uv.lock`.
- Same complete `test_namespace.py`: baseline 16 regression failures and
39
passing controls; candidate **55 passed**, no skips/errors.
- Ruff check/format and diff whitespace checks passed.
This validates the local directory-namespace path, not a live Cloud
service or
the complete repository CI suite.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
`Table.update` and `AsyncTable.update` convert Python values into SQL
literals. Tuple arrays and `numpy.datetime64` values currently raise
`NotImplementedError`, even when the same values can be ingested into
the corresponding local Arrow columns. Register tuples with the existing
recursive list converter and format NumPy timestamps as complete
nanosecond ISO strings with an explicit UTC offset. `NaT` becomes SQL
NULL; subnanosecond units raise an explicit error rather than silently
losing precision.
Regression tests exercise local synchronous and asynchronous
create/update/Arrow reads, date and timestamp units, nanosecond
precision, NULL, and existing list/ndarray/Python datetime behavior.
Nested and empty tuple arrays are compared with equivalent lists on the
actual SDK: successful results must match, and the matching d864 native
engine's existing list limitations must produce the same stable error
without changing either table. This does not change SQL map handling.
Implemented with assistance from gpt-6.1-sol; final diff reviewed
locally.
Local verification uses Python 3.11.14 and 3.13.11 in existing read-only
development environments with the native extension matching base
d864e7198f. The current main `util.py` is
byte-identical to that base; current-main native code has not been
rebuilt. Slow tests, S3 tests, remote production services and other
operating systems are not validated here.
Reproducible commands after preparing the repository's documented
development environment (these relative `.venv` examples describe
equivalent commands; the local runs use absolute read-only environment
paths and fresh task-private working directories):
```sh
PYTHONPATH=python/python .venv/bin/python -m pytest python/python/tests/test_util_update_values.py -q -p no:cacheprovider
PYTHONPATH=python/python .venv/bin/python -m pytest python/python/tests -m 'not slow and not s3_test' -q -rs -p no:cacheprovider
PYTHONPATH=python/python .venv/bin/python -m pytest --doctest-modules python/python/lancedb -q --tb=short
(cd python && PYTHONPATH=python ../.venv/bin/pyright --pythonpath ../.venv/bin/python)
.venv/bin/ruff check .
.venv/bin/ruff format --check .
.venv/bin/python -m pre_commit run --files python/python/lancedb/util.py python/python/tests/test_util_update_values.py
git diff --check d864e7198f HEAD
```
Frozen commit `33e74d0ed3d90a20419816a3f1fc15ff038db452` passed all
local gates: Python 3.11.14 full suite 1,590 passed / 36 skipped / 68
deselected; Python 3.13.11 full suite 1,587 passed / 37 skipped / 68
deselected. All 66 added SDK cases passed on both runtimes. Python 3.13
doctests: 71 passed / 6 skipped. Pyright: 0 errors / 0 warnings. Ruff,
formatting, applicable pre-commit hooks, two-file syntax/import checks
and the PR merge-base diff check exited 0. Exact local argv/runtime/wall
metadata and raw logs are retained in the acceptance manifest.
The follow-up addresses the non-UTC timestamp issue reported in review:
an update previously changed the epoch ingested by Arrow (New York +5
hours, Tokyo -9 hours). On the published original head, 4 non-UTC
sync/async SDK regressions failed and 12 naive/UTC/NaT controls passed
(exit 1; wall 3.713 seconds). Formatting with `timezone="UTC"` emits a Z
suffix and preserves the ingested nanosecond instant independently of
the destination timezone. This is consistent with the Arrow-aligned
epoch convention in merged #3775. The original 50 cases also remain
green. Full-suite wall times were 158.619 seconds (3.11) and 163.631
seconds (3.13), both exit 0.
`SourceField(json_schema_extra={...})` and
`VectorField(json_schema_extra={...})` currently raise `TypeError`
because the helper supplies the same `Field` keyword twice. Explicit
`None` also fails. This prevents users from adding ordinary Pydantic
JSON schema metadata to embedding fields.
Copy and merge the supplied dictionary before attaching the helper's
internal embedding identity. Caller dictionaries remain unchanged, and
reserved embedding keys retain the real function. Other `Field`
arguments keep their existing behavior. This change covers dictionaries
and `None`; callable `json_schema_extra` remains unsupported by the
existing embedding metadata reader and is outside this fix.
The 11 regression cases exercise both helpers, empty/custom/None
metadata, reserved-key collisions, other Field arguments, and a
deterministic local embedding model through JSON schema generation,
table creation, reopening, and search. On the original source, 9 fail
and 2 controls pass; the candidate passes all 11 on Python 3.11 and
3.13.
Validation uses read-only reused environments and the native extension
built for d864e719. The production helper is byte-identical to current
main bbc4aebe before this fix; no current-main native rebuild was
performed. The branch contains only this helper fix and its regression
test, independently based on d864e719.
Reproducible checkout commands below assume the documented environment
is installed as `.venv`; the actual local runs used existing
interpreters through absolute paths and isolated generated-data working
directories:
```sh
PYTHONPATH=python/python .venv/bin/python -m pytest python/python/tests -m 'not slow and not s3_test' -q -rs -p no:cacheprovider
PYTHONPATH=python/python .venv/bin/python -m pytest --doctest-modules python/python/lancedb -q --tb=short
(cd python && PYTHONPATH=python ../.venv/bin/pyright --pythonpath ../.venv/bin/python)
git diff --check
.venv/bin/ruff check .
.venv/bin/ruff format --check .
.venv/bin/python -m pre_commit run --files python/python/lancedb/embeddings/base.py python/python/tests/test_embedding_field_metadata.py
```
Doctests require the repository's tests/dev/embeddings dependencies,
including mlx on this platform. All checks below ran against frozen head
3fa6820.
Implemented with assistance from gpt-6.1-sol; the final diff was
reviewed locally.
Frozen-head validation (all exit 0):
- Python 3.11 full: 1535 passed, 36 skipped, 68 deselected; 190.272415
seconds wall time.
- Python 3.13 full: 1532 passed, 37 skipped, 68 deselected; 249.289778
seconds wall time.
- Python 3.13 doctests: 71 passed, 6 skipped; 10.391002 seconds wall
time.
- Pyright: 0 errors, 0 warnings; 4.884641 seconds wall time.
- pre-commit: 4 passed, 2 skipped; 0.747764 seconds wall time.
- Repository Ruff check: passed; 0.078177 seconds wall time.
- Repository Ruff format: 157 files already formatted; 0.034561 seconds
wall time.
Registering a different embedding class under an existing alias silently
replaced the original class. A deterministic real local SDK table
reproduction changed the top result for the same query from alpha to
beta after reopening the persisted table.
Check the resolved key before inserting a class. Preserve repeated
registration of the same class under the same nonempty explicit alias as
idempotent, as existing SDK tests rely on it. Repeated implicit
class-name registration, including an empty alias, still raises
KeyError. Failed registration preserves the mapping and class metadata.
Existing embedding tests receive isolated registry mappings;
test_util.py remains unchanged.
Eight regression cases cover different-class alias collisions,
alias/class-name collisions, implicit and empty-alias duplicates, equal
class names under distinct aliases, explicit-alias idempotence, and real
local creation/reopening/search. This is independent of #4356, which
fixes no-alias serialization.
Validation on 52a1ccee5a:
- Original registry plus eight new cases: 4 failed, 4 passed; Python
3.11.14; exit 1; 2.902604 s. Candidate plus full utility module: 83
passed, 1 skipped; exit 0; 3.144414 s.
- Python 3.11.14 full official non-slow/non-S3 suite: 1532 passed, 36
skipped, 68 deselected; exit 0; 152.631543 s.
- Python 3.13.11 full official non-slow/non-S3 suite: 1529 passed, 37
skipped, 68 deselected; exit 0; 163.970571 s.
- Official Python 3.13 doctest with existing compatible
embeddings/torch/mlx dependencies: 71 passed, 6 skipped; exit 0;
9.635034 s.
- Pyright with matching interpreter/source path: 0 errors, 0 warnings;
exit 0; 4.299743 s.
- Repository Ruff 0.9.9 check and format: exit 0; 157 files already
formatted; 0.047418 s and 0.016815 s. Pre-commit changed files: four
hooks passed, two inapplicable skipped; exit 0; 0.406136 s. git diff
--check passed.
Base is official d864e7198f with its
verified matching native extension reused read-only. All 174
native/config paths are unchanged from that base. Latest main
bbc4aebe55 has byte-identical registry
production code; its complete native build was not run. Verification was
macOS arm64, with optional dependency/service skips above; Linux/Windows
CI remains remote verification.
Exact commands, cwd, identities, raw logs and intermediate failures are
in the local manifest. The earlier strict same-class check caused 13
utility failures and was corrected in production; it is not described as
a baseline failure.
Reproduction commands after preparing the repository's declared
environment (from repository root unless noted):
```sh
PYTHONPATH=python/python .venv/bin/python -m pytest python/python/tests -m 'not slow and not s3_test' -q -rs -p no:cacheprovider
PYTHONPATH=python/python .venv/bin/python -m pytest --doctest-modules python/python/lancedb -q --tb=short
(cd python && PYTHONPATH=python ../.venv/bin/pyright --pythonpath ../.venv/bin/python)
.venv/bin/ruff check .
.venv/bin/ruff format --check .
.venv/bin/python -m pre_commit run --files python/python/lancedb/embeddings/registry.py python/python/tests/test_embedding_registry.py python/python/tests/test_embeddings.py
git diff --check
```
These relative commands describe reproducible source/cwd equivalents.
The recorded local runs used absolute paths to existing Python
3.11.14/3.13.11 environments, a verified matching reused native
extension, and read-only compatible optional-dependency cache paths for
the 3.13 doctest, rather than a newly installed `.venv`. Full tests used
separate task-private working directories and `--basetemp` directories
to avoid cross-run database collisions.
Implemented with assistance from gpt-6.1-sol; the final diff was
reviewed locally by the collaborating agents.
Hugging Face `select`, `filter`, and `shuffle` can retain a physical
Arrow table while storing logical row selection/order in an indices
table. Ingesting the physical table then silently writes excluded rows
or the wrong order and can lose duplicate selections.
Read indexed datasets through bounded Arrow-format iteration, and keep
the existing direct batch path for datasets without indices. Apply the
same conversion to sync/async table creation and additions, including
DatasetDict splits. The conversion does not mutate the user's formatting
settings.
Regression coverage includes reordered/duplicate selections, filtering,
shuffling, empty selections, projected numpy/pandas/Arrow formatting,
DatasetDicts, rescanning, and bounded batches.
Validation on base d864e7198f with its
byte-identical native/config:
- Python 3.11 full CI test selection: 1564 passed, 36 skipped, 68
deselected; 162.51s pytest /171.68s wall.
- Python 3.13 full CI test selection: 1521 passed, 37 skipped, 68
deselected; 160.32s pytest /163.74s wall. Optional HF tests were skipped
in this environment.
- Python 3.13 complete doctest command: 71 passed, 6 skipped; 8.42s
pytest /12.14s wall.
- Python 3.11 HF regression module from a candidate wheel: 43 passed;
the same regression module on the unchanged baseline gives 24 failed /19
passed.
- Ruff check/format, repository pre-commit hooks, pyright with the
source import path, and `git diff --check` passed.
This is a Python-only change. Native binaries were reused from the
verified baseline; no new native build or external-service integration
tests were run. The separate schema-metadata preservation change in
#4451 addresses a different conversion defect.
The target branch currently includes newer native changes than the
verified base. Latest-main native compatibility, Linux CI, and
remote-service tests remain for CI/maintainer validation. Prepared with
AI assistance (gpt-6.1-sol).
`AsyncTable.query()` unwraps the native closed-table error, so querying
a closed table raises `PanicException` instead of the documented
`RuntimeError`. `AsyncTable.head()` reaches the same path.
Propagate `inner_ref()`'s error through `PyResult` instead. Add
regressions for closed `query()` and `head()`, with `count_rows()`,
repeated close, and normal open-table queries as controls.
Validation against separately built native bindings:
- Before: 2 regression failures (`PanicException`), 2 controls passed.
- After: all 4 regressions passed.
- Existing query tests plus the basic database test: 102 passed, 1
skipped on each arm (Torch is not installed).
- `cargo fmt --all -- --check`, Ruff, and `git diff --check` passed.
The native build used the locked Cargo dependencies with
`remote,lancedb/metrics-otel,pyo3/extension-module`. This is source-tree
native testing, not a complete wheel-packaging or cloud-service test.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Negative `read_consistency_interval` values currently reach Rust's
panicking `Duration::from_secs_f64` conversion. For example:
```python
from datetime import timedelta
import lancedb
lancedb.connect("./db", read_consistency_interval=timedelta(seconds=-1))
```
Validate the interval with `Duration::try_from_secs_f64` at the three
native connection entry points and return `ValueError` before
constructing the connection or async future. `None`, zero and positive
intervals retain their behavior. The raw native binding also rejects
NaN, infinity and overflow; the public Python API still accepts
`timedelta`, not floats.
Tests cover public sync/async connections, native namespace connections,
real Python namespace clients, the three raw binding entry points and
valid-value controls.
Validation on Python 3.12 / Rust 1.97:
- Same 26 regression cases against separately built native baseline and
candidate: baseline 22 failed / 4 passed; candidate 26 passed.
- Existing basic connection and background-loop regression tests pass.
- `cargo fmt --all -- --check`, Ruff check/format and `git diff --check`
pass.
The native builds used the unchanged repository `ci` profile and
Cargo.lock with `--no-default-features --features
remote,lancedb/metrics-otel,pyo3/extension-module`. Python runtime and
focused test dependency versions came from `python/uv.lock`. Source-tree
tests used a minimal distribution-version metadata fixture derived from
the unchanged Cargo/pyproject manifests; both imported module origins
and native versions were checked in fresh processes. This is not a
complete Maturin packaging, cloud backend or full Python suite result.
No GPU was used.
AI-assisted diagnosis, implementation and validation.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Creating a table from an Arrow `list` or `large_list` vector field
infers a fixed-size vector type, but currently drops that field's
metadata. Descriptions and tags such as `lancedb:description` and
`lancedb:tag:model` disappear even though the input field carries them.
Preserve the original field metadata while replacing its inferred type.
Tests use public synchronous and asynchronous table creation, reopening,
and reads for floating-point and integer inference, checking the
complete schema and row values. Existing dtype inference and nullability
remain unchanged.
Validation on macOS arm64 with an official native extension built from
main `d864e719` using Rust 1.97.0:
- Python 3.11.14 and 3.13.11: full `pytest -m "not slow and not s3_test"
-q -rs python/tests`: **1,533 passed, 37 skipped, 68 deselected** each,
wall times 331.464 s and 416.308 s.
- Added public SDK cases on unchanged main: **12 failed**; after the
fix: **12 passed** on each runtime. Final installed distributions passed
**8 public SDK round trips** each, including fixed-size fields and
fields without metadata.
- Documentation tests on Python 3.11.14: **77 passed, 6 skipped**. Ruff
lint/format, Pyright (**0 errors**), applicable pre-commit hooks (**4
passed, 2 without matching files**), compile checks and `git diff
--check` passed.
- Comparable unchanged-main full baselines from the same day, native
extension and dependency versions: **1,521 passed, 37 skipped, 68
deselected** on each runtime. These were reused rather than rerun.
The final local wheel is an explicitly verified repack of the official
main wheel, replacing only `table.py` and `RECORD`; all production
Python files match the tested source and the native extension is
unchanged. No candidate Rust rebuild was performed. Slow/S3 integration
tests, the remaining OS/runtime and minimum-dependency matrices, and the
workflow's Linux/Python 3.13 doctest job were not run locally.
The existing Rust migration draft #2970 and issues #3216/#3222 do not
fix this metadata loss in the current Python path; a future migration
should preserve the same metadata. During final verification main
advanced to `81d52a98` through #4373 and the beta.13 version bump.
Neither modified the two files here; the local native and full-suite
validation above remain explicitly based on `d864e719`, and the updated
native changes have not been tested locally.
AI assistance: developed and verified with **gpt-6.1-sol**.
When `Vector` or `MultiVector` declares an integer Arrow value type,
Pydantic currently validates every element as a float. This silently
rounds integers above `2**53`, and converts `uint64`'s maximum value
into an out-of-range float before table insertion.
Use Pydantic's integer schema for integer Arrow value types in both
validators. Floating-point types retain the existing validation. Tests
exercise model serialization and real local table creation, insertion,
and reads for signed and unsigned integer boundaries and floating-point
controls, plus dimensions and optional/nullability behavior. Fractional
floats are rejected for integer declarations; integral floats remain
accepted. This does not extend ANN index support.
Validation on macOS arm64, with the current main native extension built
using Rust 1.97.0:
- Python 3.11.14 and 3.13.11, Pydantic 2.13.5: full `pytest -m "not slow
and not s3_test" -q python/tests` suite: **1,537 passed, 37 skipped, 68
deselected** on each runtime. Comparable unchanged-main baselines:
**1,521 passed, 37 skipped, 68 deselected** each.
- Added cases on unchanged main: **8 failed, 8 passed**; complete
`test_pydantic.py` after the fix: **40 passed** on each runtime.
- Documentation tests with the existing embeddings dependencies in an
isolated environment, Python 3.11.14: **77 passed, 6 skipped**, matching
the unchanged-main baseline. The workflow's Linux/Python 3.13 doctest
matrix was not reproduced locally.
- Repository Ruff lint/format, Pyright, all applicable pre-commit hooks,
compile checks, and `git diff --check` passed. Eight public SDK table
round trips passed on each final installed wheel runtime.
Slow, credential-dependent S3 integration tests, the remaining
OS/runtime matrix, and the minimum Pydantic/PyArrow dependency matrix
were not run locally. The native extension was built from unchanged
current main; the final local wheel repack contains the verified Python
change with that identical native extension. A separate candidate
maturin packaging attempt triggered Rust recompilation and was stopped,
so an official candidate distribution build was not completed. The
project requires Pydantic `>=2.7.4,<3`; Pydantic v1 support was removed
in #3990.
AI assistance: developed and verified with **gpt-6.1-sol**.
Decimal literals such as Decimal("1E-7") and Decimal("1E+3") currently
raise ValueError while building an expression. Parse the coefficient and
exponent exactly, and normalize representable negative scales to zero so
both typed filters and predicate.to_sql() queries preserve the value.
Checked arithmetic bounds the Decimal128 coefficient to 38 digits,
preserves existing scale limits, and rejects nonfinite or overflowing
values. This does not add Decimal256 literal support.
Added 21 cases covering actual Arrow-backed table queries, SQL queries,
zero, signed exponents, precision boundaries, and errors. Both Python
3.11 and 3.13 expression suites pass 123 tests; the baseline fails 10 of
the new cases. Python binding Rust tests pass 13; official make doctest
passes 77 with 6 skips; Ruff, Rustfmt, hooks, Pyright and
workspace/tests/all-features Clippy with -D warnings pass. A fresh
environment restored from the original dependency freeze passes the full
required suite: 1528 passed, 37 skipped, 68 deselected (143.51s); while
optional Torch suites have the same baseline remote mock-server ordering
failure.
Tested against f4544628; observed official main d864e719 contains one
Catalog API commit without expr or Cargo.lock changes. AI assistance:
gpt-6.1-sol assisted implementation and local validation; all reported
checks were actually executed.
LanceDBConnection.session read AsyncConnection.session, but
AsyncConnection has no session attribute, so access raised
AttributeError. Store the session passed to the constructor and return
that object.
Assisted by Cursor (Grok).
Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com>
Since #3232, `pydantic_to_schema` maps an `Enum` field to the Arrow type
of its values, including enums without a `str`/`int` mixin. Writing such
models still failed: `model_to_dict()` returns `model.model_dump()`,
which keeps the Enum members, and Arrow can only convert members that
are also `str`/`int` instances.
```python
class Color(Enum):
RED = "red"
class Item(LanceModel):
color: Color
tbl = db.create_table("items", schema=Item)
tbl.add([Item(color=Color.RED)])
# pyarrow.lib.ArrowTypeError: Expected bytes, got a 'Color' object
```
`create_table(data=[...])` fails the same way, and int-valued enums
raise `ArrowInvalid`.
`model_to_dict()` now replaces Enum members with their values, including
in nested models and lists. Lists of plain scalars (vectors in
particular) are returned without being walked, so the per-row cost stays
about the same as `model_dump()` alone (21.3µs vs 20.3µs for a 1536-dim
model). Reading back is unchanged: Pydantic already validates the stored
value into the Enum member.
Added `test_plain_enum_values_are_written`, which covers str and int
valued enums, `Optional` and `List` fields, and an Enum inside a nested
model, and checks that `to_pydantic` round-trips the rows. It fails with
the `ArrowTypeError` above before the change. `test_pydantic.py`,
`test_embeddings.py`, `test_table.py` and `test_db.py` pass, and `ruff
format --check` / `ruff check` are clean.
`EmbeddingFunctionRegistry.register()` documents that the class name is
used when no alias is given, and it does register the class under that
name. But it stored the alias itself (`None`) as
`__embedding_function_registry_alias__`, which is what
`function_to_metadata()` writes into the table schema. A table using
such a function gets `"name": null` in its `embedding_functions`
metadata, and any later read of the functions fails:
```python
@get_registry().register()
class MyEmbed(TextEmbeddingFunction): ...
f = MyEmbed.create()
class Schema(LanceModel):
text: str = f.SourceField()
vector: Vector(4) = f.VectorField()
tbl = db.create_table("t", schema=Schema)
tbl.add([{"text": "hi"}]) # KeyError: None
tbl.search("hi") # KeyError: None
```
This stores the key the class is registered under instead, so the class
name round-trips. For aliased classes the key is the alias, so their
metadata is unchanged. The TypeScript registry already falls back to
`ctor.name` in the same way.
Added `test_embedding_function_registered_without_alias`, which creates
a table with such a function, reopens it, and adds a row. It fails with
`KeyError` before the change. `test_embeddings.py`, `test_pydantic.py`,
`test_table.py` and `test_db.py` pass, and `ruff format --check` / `ruff
check` are clean.
## Summary
With `rerank(..., normalize="rank")`, a hybrid query returns ranks (1.0,
2.0, 3.0, ...) in the `_distance` and `_score` columns instead of the
real vector distances and BM25 scores.
`_combine_hybrid_results` converted both result sets to ranks first, and
only then captured `original_distances` / `original_scores`. So the step
that restores the original values after reranking put the ranks back.
With `normalize="score"`, the real values are returned as expected.
Repro (RRFReranker(return_score="all"), vector distances 0 / 25 / 100):
| normalize | `_distance` returned | `_score` returned |
| --- | --- | --- |
| `"score"` | 0.0, 25.0, 100.0 | 0.47, 0.67, None |
| `"rank"` (before) | 1.0, 2.0, 3.0 | 1.0, 2.0, None |
| `"rank"` (after) | 0.0, 25.0, 100.0 | 0.47, 0.67, None |
## Changes
- Capture the original `_distance` / `_score` columns before converting
them to ranks, then normalize the ranked column. The reranker input is
unchanged; only the values restored afterwards are fixed.
`AsyncHybridQuery` uses the same function, so both sync and async are
covered.
## Test plan
- [x] New `test_hybrid_return_all_restores_original_scores` in
`test_rerankers.py`, parametrized over `score` / `rank`: the `rank` case
fails before the fix and passes after
- [x] `pytest python/tests/test_rerankers.py
python/tests/test_hybrid_query.py`
- [x] `ruff format --check` / `ruff check`
I have no local Rust toolchain, so I ran the tests against the released
0.39.0 native module with the changed Python files overlaid. The changed
functions are identical between 0.39.0 and main.
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
#3971 added `test_zero_lifetime_token_is_refreshed`, but its expected
headers omit `x-lancedb-credential-type: oidc`, which
`OAuthProvider.get_headers()` has emitted since #2631/#4070. The test
fails against current main and breaks the PR CI matrix for every open
PR.
## Changes
- Align the two zap assertions in
`test_zero_lifetime_token_is_refreshed` with the provider contract (see
`test_token_refresh` and `test_sync_method`).
## Tests
- `pytest python/tests/test_header_provider.py -q`
Fixes#4331
With `return_score="all"`, model rerankers reconstruct merged rows
without their Arrow schema. This changes fixed-size float32 vectors into
variable-length float64 lists, loses metadata, and can overflow valid
uint64 row IDs.
Unify the original input schemas before inserting placeholder scores,
then use the unified schema when reconstructing the merged table. Row
merging and reranking behavior stay the same. Capturing the schema first
also preserves float64 scores without conflicting with the float32 null
placeholders.
Validation:
- New regression cases: 5 failures and 5 passing controls before the
fix; all 10 pass afterward.
- Reranker, hybrid-query and TypeSafe reranker tests: 150 passed, 17
skipped. Optional live model/API tests were not run.
- The fake-client tests that use random metadata passed again with seeds
7 and 41.
- Repository-wide Ruff lint/format checks, configured Pyright check and
applicable pre-commit hooks pass.
- Tests used the locally built Rust/Python extension and the
repository's locked uv environment. No Rust code changed.
AI disclosure: I used Codex for investigation, implementation, tests,
local checks, and drafting this description.
## Root cause
The pinned Lance Arrow crate rebuilt a sliced struct child with its
original Arrow offset but supplied a validity bitmap containing only the
visible rows. Arrow requires the bitmap to cover the offset as well, so
filtered vector searches involving nullable nested fields could panic
with `null_bit_buffer size too small`.
## Fix
- Carry an offset-aligned validity bitmap when propagating parent nulls
to a sliced child.
- Vendor the pinned `lance-arrow` crate until a Lance release includes
the fix; include it in Maturin source distributions.
- Add unit regressions for sliced children and parent bitmaps, plus a
record-batch merge regression.
## Validation
- The issue's 8,600-row Python reproducer completes its filtered flat
dot search and returns 8,448 rows without a panic.
- The targeted Rust merge regression and two focused Python query tests
pass.
- Rust formatting, Ruff formatting and lint, the vendored crate tests,
and the workspace Cargo checks passed. The Maturin source archive
includes the patched crate.
Fixes#4260
<!-- lance-gatekeeper-fix:v1 agent=e6045f701d9b708ee26528ba36353aba
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- preflight CUDA availability and initialization before accelerated
index training
- direct unavailable or misconfigured CUDA environments to the explicit
CPU path
- cover both legacy and unified Python index APIs
## Root cause
LanceDB delegated accelerator requests directly to PyLance. When PyTorch
lacked CUDA support or CUDA initialization failed, users received a
low-level assertion or runtime error with no indication that omitting
`accelerator` or setting `accelerator=None` creates the index on CPU.
The repair validates CUDA before training and preserves explicit
accelerator semantics without silently changing the requested backend.
## Scope
This PR addresses only the separately reproduced CUDA availability and
initialization error. It does not close#1766; the reported query
failure after successful CUDA index creation remains open pending
reproducible evidence.
## Validation
- `uv run --extra tests pytest python/tests/test_table.py -k
create_index -q` (10 passed)
- `ruff format .`
- `ruff check .`
- CPU-only integration check confirms the actionable fallback message
Related to #1766
<!-- lance-gatekeeper-fix:v1 agent=d05c37237b04c2c8f88395a82e00bc67
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`LanceHybridQueryBuilder._validate_query`'s ValueError concatenated two
adjacent string literals without a separator, rendering "...explicitly
for hybrid search.**But** not both.". Added the trailing space.
String-only change.
`VoyageAIReranker`'s `api_key` docstring said it falls back to
`OPENAI_API_KEY`, but the class actually reads `VOYAGE_API_KEY` (and
raises a "VOYAGE_API_KEY not set" error without it) — the docstring was
copy-pasted from the OpenAI reranker. Docstring-only fix.
This PR converts the uri parameter to string before calling urlparse in
get_uri_scheme and get_uri_location, allowing pathlib.Path objects to be
passed seamlessly without raising a TypeError.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4033
`create_table(data=...)` without an explicit schema raised `IndexError:
index out of bounds` when a `vector`/`embedding`-named list column was
all-null. `_modal_list_size` infers the dimension with
`pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips
nulls, so an all-null column produces a zero-length result and the `[0]`
subscript blows up.
The sibling `_infer_vector_dim` already guards the identical
`pc.mode(...)` with a `len == 0` check and returns `None`;
`_modal_list_size` just didn't. This makes `_modal_list_size` return
`Optional[int]` and return `None` when every value is null, and the call
site then leaves the column as a variable-length list instead of
inventing a fixed size — the dimension is genuinely unknowable when
there are no non-null rows.
Added a regression test that creates a table from an all-null
`list<float32>` vector column and asserts it succeeds, the column stays
a `list<float32>`, and the rows are queryable. It fails on `main` with
the `IndexError` and passes with this change.
Retries individual inputs one at a time when the OpenAI API rejects a
batch with BadRequestError, instead of discarding the whole batch.
Valid inputs now still get embeddings; only the actually-bad input
resolves to None, and the log names the exact failing input.
Fixes#1677
## Summary
Sync `Table.search()` declares tuple, `pyarrow.Array`, and
`pyarrow.ChunkedArray` query vectors as supported, and the async query
path already accepts them. The sync query builder only classified lists
and NumPy arrays as vectors, however, so these inputs were routed
through embedding/FTS resolution and then rejected. Arrow and tuple
inputs also did not contribute a query dimension when selecting among
multiple vector columns.
This change:
- recognizes the supported tuple and Arrow containers as sync vector
queries for both `auto` and explicit `vector` query types;
- converts them to the existing internal `float32` NumPy representation
at the sync builder boundary;
- infers dimensions from tuple, primitive Arrow, and list-valued Arrow
queries so schemas with differently sized vector columns select the
matching column.
The regression test uses two vector columns with dimensions 2 and 3. It
verifies tuple, Arrow array, and chunked Arrow queries through both sync
query modes and confirms the dimension-2 column is selected.
## Test plan
- `pytest
python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers
-q` (6 passed)
- `pytest python/tests/test_util.py python/tests/test_query.py
python/tests/test_table.py -q -m "not slow and not s3_test"` (308
passed, 1 skipped, 1 deselected)
- `pytest python/tests -q -m "not slow and not s3_test"
--ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected)
- `ruff format --check python`
- `ruff check python`
- `cargo fmt --all --check`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- treat an OAuth `expires_in` value of zero as an explicit immediate
expiration
- refresh zero-lifetime tokens instead of caching them indefinitely
- add a regression test covering two consecutive header requests
## Tests
- `pytest python/tests/test_header_provider.py -q` (16 passed)
- `ruff format --check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
- `ruff check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
## Summary
- move synchronous iterator preparation off LanceDB's singleton
background event loop
- keep the loop available when an input iterator executes a synchronous
table query
- cover query-backed iterators for both `Table.add` and `create_table`
## Root cause
The async implementations prepared input data on the singleton
background loop and peeked at the first item of iterators there. If that
iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the
query on the same loop and blocked waiting for it, so neither operation
could progress.
## Validation
- `ruff format` on the changed Python files
- `ruff check .`
- `cd python && uv run --frozen --extra tests pytest
python/tests/test_table.py -q` (138 passed)
- the broader non-slow suite was attempted but hit a native segmentation
fault in the external DataFusion `register_table_provider` path; an
isolated control using the pre-change inline preparation behavior
reproduced the same unrelated crash
Fixes#2107
<!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression for filtered scans over deeply nested
list/struct values
- verify the public query path returns every nested sibling field
without an Arrow schema mismatch
## Root cause
Older Lance scanner code split filter projections into partial nested
arrays and did not recursively merge List<Struct> children, so a sibling
field could be omitted from the returned array while remaining in the
declared schema. That merge was fixed upstream in
lance-format/lance#4838, and the Lance revision currently pinned by
LanceDB contains that fix. This test makes the repair durable at the
LanceDB Python API boundary.
## Validation
- cd python && uv run --extra tests pytest
python/tests/test_query.py::test_filter_preserves_deeply_nested_struct_fields
-q
- cd python && uv run --extra tests pytest python/tests/test_query.py -q
- uv run --project python --extra tests --extra dev ruff check .
- uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_query.py
Fixes#2217
<!-- lance-gatekeeper-fix:v1 agent=9ddc0efd1f13888b2a7a896881c6e68a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for an IVF_RQ index created through the Lance
Python API and consumed through LanceDB
- verify a fresh LanceDB handle can list and query the
externally-created index
## Root cause
Lance 4.0.0-beta.5 could serialize `rotate_mat_position` as null, while
the Lance 2.0.0 reader used by LanceDB 0.29.2 required a `u32`. Upstream
Lance fixed the incompatible IVF_RQ metadata handling and index
versioning, and the current LanceDB dependency includes that repair.
LanceDB was missing coverage for the reported mixed-API workflow,
allowing the compatibility regression to go undetected.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_index.py
-q` (24 passed)
- `uv run --project python --extra dev ruff format .`
- `uv run --project python --extra dev ruff check .`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_index.py`
- `uv run --project python --extra dev ruff check
python/python/tests/test_index.py`
Fixes#3092
<!-- lance-gatekeeper-fix:v1 agent=7d8c849bd8421be65d32355c42a60b8a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
## Context
`StreamingDataset`'s shipped split strategies (`split_random`,
`split_sequential`, `split_hash`, `split_calculated`) all share one
trait: every row in the base table is assigned to exactly one split,
exactly once. They only differ in *how* that one-to-one assignment is
chosen -- none of them can express "give me a resampled stream of N
rows, allowing repeats."
That's a distinct, common need in ML training: **the bootstrap** -- the
statistical technique (Efron, 1979) of estimating a dataset's properties
by resampling it *with replacement*. It's the same sense scikit-learn's
bagging classifiers mean by `bootstrap=True`. `SplitStrategy::Bootstrap`
adds exactly this to the data loader: sample rows uniformly with
replacement from the (optionally filtered) base table, independently per
split.
## Main use cases
- **Bagging / ensemble training** (e.g. random forests): each split
trains on an independent bootstrap resample of the training set.
- **Oversampling a rare class**: filter to the minority class, then
sample it up to whatever count balances it against the majority, instead
of exhausting its small set once per epoch.
- **Fixed-size training steps independent of dataset size**: want
exactly N batches per epoch regardless of whether the table has 50 rows
or 50 million? Set `num_samples` directly instead of it being implicitly
`len(dataset)`.
- **Weighted sampling without a weight column**: filter to "eligible"
rows (e.g. a windowed-sampling workload filtering to valid window-start
positions), then bootstrap the filtered set -- an item with more
eligible rows is naturally drawn proportionally more often, with no
explicit per-row weight needed.
## Example (an actual run, not a hypothetical one)
A 5-row base table (`row_id` 0..4), sampled via `Bootstrap {
num_samples: 8, sizes: Fixed(2), seed: Some(42) }`:
```
row_id split_id
0 0
2 0
4 0
4 0 <- row 4 repeated within split 0
4 1 <- row 4 also drawn in split 1 (independent draw)
4 1
1 1
2 1
```
8 output rows from a 5-row table -- not 5. Row 4 drawn 4 times total,
row 3 never drawn. That's with-replacement sampling actually behaving
like with-replacement sampling, not an even round-robin dressed up with
a new name.
## What changed
Adds `SplitStrategy::Bootstrap` (uniform sampling with replacement,
optionally filtered, independently seeded per split -- a given split's
content doesn't depend on how many other splits there are, the same
elastic-determinism property documented for the loader's other
strategies). No new `PermutationBuilder` Rust method was needed:
`with_split_strategy` already accepts any `SplitStrategy` value
generically. Python binding:
`PermutationBuilder.bootstrap_sample(num_samples, *, ratios=None,
counts=None, fixed=None, seed=None, split_names=None)`, mirroring
`split_random`'s existing shape.
Also adds a generic `split_strategy` parameter to `StreamingDataset`'s
constructor (a callable taking/returning a `PermutationBuilder`), since
it previously could only reach `split_random`/`split_sequential` even
though `split_hash`/`split_calculated` already shipped. This unblocks
`Bootstrap` and any future strategy without further constructor changes.
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
Cargo.lock: webpki-root-certs 1.0.10; webpki-roots 1.0.10 and
rustls-webpki 0.103.16 are already current.
docs/web-tests/rs/Cargo.lock: rustls-webpki 0.103.16. python/uv.lock:
certifi 2026.10.19, the only entry that changes.
The root crate drops the root CA whose intermediates were compromised
(Mozilla bug 2083449), and rustls-webpki 0.103.16 rejects chains that
carry the compromised eMudhra intermediate keys. certifi 2026.10.19 is
the release the certifi advisory names as fixed: it drops the same root
and keeps the issuer's other roots, so HTTPS connections that chain to
them keep working. Relative to the previous 2026.6.17 pin it also adds
Telia EC and RSA TLS Root CA v3 and SECOM TLS ECC and RSA Root CA 2024
and drops Atos TrustedRoot 2011.
The webpki-root-certs entry is written by hand with the crates.io
checksum.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
Co-authored-by: Claude Opus 5.1 <noreply@anthropic.com>
Adds a user-facing realtime engine (RTE) page at
`docs/web/tables/realtime-engine.mdx`, listed under **Management** after
Branches. It is adapted from the internal WAL field guide
(lancedb/sophon#6213), with the deployment and operations material left
out.
The page covers:
- what RTE is and how it works (memtable → SSTables → compaction; reads
merge the base table with the fresh tier)
- getting started: `set_unenforced_primary_key`, indexes, `enable_rte` /
`ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial
updates, deletes, and `checkpoint_rte` / `disable_rte`
- performance: write throughput and shard scaling, and read latency with
and without RTE
- a note on operations not yet supported (time travel, branching,
`count_rows`, row IDs, `TABLESAMPLE`)
Also aligns related docs with current RTE naming and behavior:
- `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` /
`UNSET RTE`
- `maintenance/background-jobs.mdx`: job names are `memwal_compaction`
and `memwal_catchup_<type>`
- `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine
- Python `AsyncTable.enable_rte` /
`RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes
created after enabling RTE are maintained too
The examples are written directly in the page rather than generated from
`docs/web-tests`, because RTE needs an Enterprise server to run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: XY Zhan <zhaner08@hotmail.com>
## Problem
`permutation_builder(table).execute()` and `StreamingDataset(table,
...)` (which builds a permutation at construction) panic with
`RustPanic` on any table whose Arrow schema has schema-level metadata:
```python
schema = pa.schema([pa.field("x", pa.int64())], metadata={b"huggingface": b"{}"})
t = lancedb.connect(tmp).create_table("t", pa.table({"x": list(range(100))}, schema=schema))
permutation_builder(t).split_random(fixed=2, seed=1).execute() # RustPanic
```
Every Hugging Face Lance dataset triggers this, e.g.
`hf://datasets/lance-format/lerobot-xvla-soft-fold/data` table `frames`
carries `source-column-name-map`.
## Root cause
`PermutationBuilder::add_config_metadata` replaced the stream schema's
metadata with the permutation config and then called
`batch.with_schema(schema).unwrap()`. The base-table scan's batches
carry the base schema metadata, and `RecordBatch::with_schema` requires
the target to be a superset of the current schema, metadata included. It
returned `SchemaError("target schema is not superset of current schema
...")`, and the `unwrap` turned that into a panic on a tokio worker.
## Fix
- The permutation table deliberately does **not** inherit base-table
schema metadata. That metadata describes columns the permutation table
(`row_id`, `split_id`) does not have, and its keys can collide with
config keys such as `split_names`. A merge would let a base table's
`split_names` leak into, or override, the permutation's split names.
Batches are rebuilt with `RecordBatch::try_new(schema, columns)`.
- Errors are propagated (`map_ok` → `and_then`) instead of `unwrap`.
- The same panic-on-worker pattern in the splitter's streaming closures
(`split.rs`: `try_new`/`filter(...).unwrap()`) now propagates errors
too. Nothing else in `dataloader/` used `with_schema(...).unwrap()`.
## Tests
- Rust: `test_permutation_over_table_with_schema_metadata` in
`builder.rs` builds a random split + shuffled permutation over a table
with `huggingface` and a colliding `split_names` key. It panics with the
original `SchemaError` on the old code and passes now.
- Python: `test_permutation_over_table_with_schema_metadata` in
`test_permutation.py` covers the repro and reads both splits through
`Permutations`/`Permutation.from_tables`.
`test_streaming_dataset_over_table_with_schema_metadata` in
`test_elastic_dataloader.py` covers `StreamingDataset` construction and
iteration.
- `cargo fmt`, `cargo clippy --features remote --tests --examples`, the
dataloader Rust tests (42), `test_permutation.py` (72), and
`test_elastic_dataloader.py` all pass.
`test_torch.py::test_remote_permutation_dataloader_multiprocessing`
fails locally with `take query result did not include ordering column
'__lancedb_take_row_offset'`. It uses `Permutation.identity`, which does
not go through `PermutationBuilder`, so this change does not cause it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Closes#4246.
lancedb publishes no source distribution, so a platform without a
prebuilt wheel (#4104, #4239) has nothing to install. This adds an sdist
to the Python release.
## The sdist job
The release workflow builds an sdist with the maturin version of the
wheel jobs. On a tag, the publish job uploads it with the wheels: to
PyPI, and to Fury for beta tags. The job also runs on the pull request
dry run, which now starts on a change to `python/pyproject.toml`.
## Checks
- The job builds a wheel from the sdist with `--locked` and smoke-tests
it in a fresh venv.
- `ci/check_sdist.py` fails if a tracked file of the Python package is
missing from the tarball.
- `twine check --strict` checks that the metadata renders. `pydistcheck`
enforces PyPI's 100 MiB size limit.
- The unit tests of both check scripts run in the CI scripts workflow.
## Cargo.lock
maturin ships the repository lockfile, which pins crates that only the
dropped `nodejs` member uses, so `cargo --locked` fails
(PyO3/maturin#2609). The job first runs `cargo metadata --locked` on the
repository workspace, so a lockfile that has drifted from `Cargo.toml`
fails there. It then removes the nodejs-only pins and repacks the sdist.
`ci/check_cargo_lock.py` fails if the prune changes any package it
keeps, or any dependency of one. The published file is the repacked one.
## Behavior change
After the first release with an sdist, pip on a platform with no current
wheel builds the newest version from source, which needs a Rust
toolchain and `protoc`. Before, pip installed the last version with a
wheel for that platform.
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.
## Summary
Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.
The table can no longer grant itself trust:
```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```
Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.
Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.
## Evidence
`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:
```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
- runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```
## Merge Risk
**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- `StreamingDataset`'s Stage 1 (I/O) queue (`raw_batches`) had no cap:
`_fill_io` only limited *in-flight* fetch futures (`io_pending`) to
`io_queue_depth`, but every completed fetch was unconditionally drained
into `raw_batches` and the freed slot immediately refilled, so I/O could
race arbitrarily far ahead of a slow transform stage with no bound on
buffered memory.
- `transform_queue_depth`'s admission check (`_try_submit_tx`) couldn't
compensate either, since it only counts `cooked` rows and in-flight
transform futures — it never looks at `raw_batches`.
- Fix: `_fill_io` now counts both `io_pending` and `raw_batches` against
the `io_queue_depth` budget, matching the parameter's documented
"batches to keep in flight per split" contract.
## Test plan
- [x] Added
`test_raw_queue_depth_bounded_when_transform_is_the_bottleneck`
(deterministic, via synchronous I/O) asserting `raw_queue_depth <=
io_queue_depth * read_batch_size` even when the transform stage is the
bottleneck.
- [x] Full `python/python/tests/test_elastic_dataloader.py` suite passes
(190/190), confirming no change to sample ordering, counts, or
checkpoint/resumability behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Christopher KrahandXuanwo
Aviral Saxena
adenzhou1350
LeonSGP
Preston ZhaoandPreston Zhao
Mohammad Hijjawi
Yingzi Zhang
lancedb-gatefixer[bot]
陈志谦
Mohd Quamar TyagiandXuanwo
Madan Kumar
Vinayak Vathare
Zhewen TanandXuanwo
primorLee
Lance Release
Jack Ye
Lei XuandClaude Opus 5.5
Tobias
Will JonesandClaude Opus 5.5
Weston PaceandClaude Sonnet 5