Downstream workspaces adopting Lance 15.0.0-beta.11 currently also
resolve beta.10 because the SDK pins every Lance crate exactly. This
produces incompatible Lance types across the SDK and the consumer.
Update the workspace pins and lockfile to the published
[v15.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.11).
This release includes payload-free list length reads and preserves those
read modes across serialized plans (lance-format/lance#10759,
lance-format/lance#10766). Existing feature flags and unrelated locked
packages are unchanged.
The release also includes the metric/k-dependent search-effort
calibration from lance-format/lance#10695. Update the Node and Python
execution tests to its exact probe budgets (for example, effort 0.25
with L2, k=8 and Auto=6 now probes 4 partitions), and cover the 0.1
tier. Recall, caller bounds and probe-precedence assertions remain in
place.
Validation:
- `cargo fmt --all --check`
- `cargo clippy --quiet --features remote --tests --examples`
- Downstream metadata fixture with SDK source and the released Lance
beta.11 path overrides resolves exactly one Lance version.
- `cargo test --quiet --features remote --tests`: 2,271 passed, 0
failed, 79 existing ignored tests.
- Node native/TypeScript build, lint and API documentation generation
passed; 41 query tests passed.
- Python native binding build and package Ruff checks passed; 208 query
tests passed, 1 slow test deselected, 5 existing deprecation warnings.
- Rust results remain applicable: the follow-up changes only binding
tests, with product source and dependency graph unchanged from the
tested dependency update.
Repository-root Ruff currently reports 59 lint findings and 6 formatting
failures in untouched documentation snippets. Their blobs match the PR
base; the changed Python test and the entire Python package pass
lint/format checks.
Remote Python `drop_function` currently discards the dependent-view
warnings returned by SQL DROP. Both the synchronous and coroutine APIs
now emit those warnings through `warnings.warn` while preserving their
boolean result and executing DROP once. The server support is already
merged in lancedb/sophon#9820.
The Rust decoder reads warnings across every result batch, including an
empty first batch. Missing warning columns remain compatible with older
servers; malformed or null warning lists are rejected. The hidden
binding bridge carries the warnings to Python. The separate
`drop_function_async` API and its `(bool, Job)` contract are unchanged.
Python warning filters can raise after the catalog mutation has
completed.
Validation: a compiled PyO3/public-server reproduction failed before the
fix and passed afterward; the public async path also returned `true`
with the expected dependent-view warning. Final source passes two
focused Rust tests, two Python wrapper tests, strict targeted Clippy,
formatting, and Ruff. Three decoder tests cover batch/null/type and
older-server behavior. The public runs preceded a two-file
simplification of the completed Job projection; focused final-source
checks cover that delta. Current-main integration is conflict-free, and
its overlapping changes concern independent retry-policy APIs/tests.
Documentation now covers warning filters, the post-removal
warnings-as-errors behavior, and silent compatibility with older
servers. The follow-up changes only prose and a Python docstring; the
runtime validation above remains applicable.
Refs ENT-3659. Whole-issue independent acceptance remains pending.
## Summary
Bumps dependencies that Aikido flags with critical advisories, where a
patched version is available without code changes.
| Package | Where | From → To | Ships to users |
|---|---|---|---|
| jackson (BOM) | `java/pom.xml` | 2.18.11 → 2.18.12 | Yes, via the
published pom |
| zerovec / zerovec-derive | `Cargo.lock` | 0.11.6 → 0.11.8 / 0.11.3 →
0.11.7 | Yes, linked into the wheels and Node binaries |
| pillow | `python/uv.lock` | 12.3.0 → 13.0.0 | No, dev lockfile |
| pyarrow | `python/uv.lock` | 24.0.0 → 26.0.1 | No, dev lockfile |
| pyjwt, urllib3, anyio, cryptography | `python/uv.lock` | → 2.15.1,
2.8.0, 4.16.0, 50.0.2 | No, dev lockfile |
| pillow | `docs/web-tests/blob-v2/py/requirements.txt` | 12.1.1 →
13.0.0 | No |
pyarrow was held at 24.0.0 by a `pyarrow<25` pin in the `tests` extra.
It came in with #3675 without a stated reason, so this PR removes it.
pillow 13 and pyarrow 26 require Python 3.11+, so the lockfile keeps
pillow 12.3.0 and pyarrow 25.0.1 on Python 3.10, and those remain
flagged.
The sharp findings in the Node examples and docs are not addressed here.
They come from the `@huggingface/transformers@3.0.2` optional
dependency, which needs a major upgrade to 4.3+.
## Evidence
With the pyarrow pin removed and pyarrow 26.0.1 installed, the Python
suite (`-m "not slow and not s3_test"`) passed locally on Python 3.12:
3319 passed, 164 skipped. That run was on main as of #4879; this branch
is based on a slightly newer main.
## Merge Risk
**Door:** Two-way. Patch and minor version bumps that a revert undoes.
**Blast Radius:** Java consumers (jackson patch release), native
binaries (zerovec patch release), Python test environments.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Functions that import a module unavailable on the client currently
package silently even when they declare no pip/conda dependencies. Emit
an advisory warning for these unresolved imports so callers can declare
the required package or confirm that the base image supplies it.
Any pip/conda declaration suppresses this new advisory because
distribution names need not match import names. The warning does not
claim that the module is absent remotely and follows normal Python
warning filters, including warnings-as-errors.
Validation includes the original `import scipy.ndimage` with `pip=()`
through the public decorator/packager path, plus declared-package and
existing local-source controls. This does not establish remote image
availability or deployment acceptance.
Refs ENT-3448.
Expose `FunctionRetryPolicy` on Function declarations and refresh
overrides, together with `error_policy` and recorded attempt counts.
Python uses the Rust validation rules before sending a request; local
refreshes reject these Enterprise options.
The policy bounds calls and backoff, supports
class/message/provider-code matching, and can disable retries with
`max_attempts=1`. Batch retry and skip operate on the whole failed call.
Requires Enterprise server support for the new policy.
Validation: rebuilt the Python extension; 47 targeted Python tests and
10 Rust tests passed. Python `make check`, Rust formatting, and `cargo
clippy --features remote --tests --examples -- -D warnings` passed. The
additional repository-root Ruff sweep reports existing errors under
`docs/web-tests`; the Python workspace checks pass.
Draft user guidance covers the supporting server requirement, safe
repetition of partial work and whole-batch exhaustion. The guide
sections stay hidden until release. Vale 3.24.0 passed for the added
draft prose; no paired deployed SDK/server retry acceptance was run.
Companion server: https://github.com/lancedb/sophon/pull/9812.
BREAKING CHANGE: Rust callers constructing `FunctionRegistrationRequest`
or `FunctionErrorRecord` with struct literals must supply the new
`retry_policy` or `attempts` field, respectively.
Bare Rust test runners do not install the `lancedb` Python package, so
the typed-error tests fail with `ModuleNotFoundError`. Load the
checked-in exception modules once for the embedded interpreter,
preserving exception-class identity across parallel tests and the
existing type/cause assertions.
Rust CI now watches the binding sources and the Python exception modules
embedded by these tests, so those changes exercise the macOS Rust test
jobs.
Catalog lookup failures lose their structured resource identity at the
Rust/Python SDK boundaries, leaving callers with generic errors or empty
correlation fields. Decode the existing Flight SQL error metadata and
preserve resource kind/name, namespace, error code, and request/job IDs
in the existing error types.
Direct `get_function` calls unwrap named Function or namespace failures.
Generic SQL job failures remain `JobFailedError` with the typed resource
error as their cause; native deferred commands correctly leave durable
job IDs null. Existing unstructured SQL classifications, including
`FailedPrecondition`, remain supported.
Validation: focused Rust/Python regressions and a native-extension run
against a real Sophon process passed all five Function/Secret
resource-error cases. The final one-commit candidate passes `cargo check
-p lancedb-python` and formatting; earlier runtime evidence applies
after removal of an unused tuple member. Independent deployed acceptance
remains pending for ENT-3373.
Server companion: https://github.com/lancedb/sophon/pull/9804.
Integration validation: all 14 DocLinks pass on the merge candidate with
#4869. The subsequent main documentation routing change was inspected;
it does not alter the typed-error paths. The inherited pin-utils
cargo-deny finding is tracked in a separate dependency repair.
`take_row_ids` used an `IN` filter, so the backend could reorder rows
and collapse duplicate IDs. That can silently misalign training rows
with labels or sample weights.
Restore the requested order and duplicate occurrences in the shared Rust
take path for native and remote tables, and retain that information
through Python query serialization. Missing or deleted IDs are omitted.
Explicit Rust `order_by` overrides request order, and pagination applies
after restoring occurrences. Empty projections and `_rowid` projection
aliases retain their output schema.
RTE tables reject unsupported takes with the existing base-table
guidance before checking the internal row-ID projection used for
ordering. An explicit `use_rte(False)` reads the base table with the
same requested order and occurrences.
A pre-existing fixture defect also blocked CI:
`test_table_add_in_threadpool` responded without reading uploads,
allowing a connection reset during response decoding. This PR includes a
fix confined to that test's handler: consume each request body before
responding and verify one create row plus ten inserted rows. This was
not a regression in the take implementation. Production write retries,
concurrency and HTTP client code are unchanged.
The selective sampling guide scopes the row-ID guarantees to SDK
releases containing this fix, preserves the distinct offset-take
semantics, and explains that ordered takes collect and combine matching
rows before output batching. It recommends smaller ID requests and
narrow projections because `max_batch_length` does not cap working
memory. This uses the existing REST query protocol and requires a new
SDK release; no deployed service was changed.
Validation:
- Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX
compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed
lines. Full-site Mint validation reports four existing snippet-import
warnings on the reranking/vector-search pages, identical with the
original page restored in a baseline comparison.
- The applied threadpool test passed locally after the fixture change.
The identical candidate previously passed 10 trials with the ordinary
reader and 10 with an unbuffered reader; each trial verifies all
uploaded rows. Those checks used macOS arm64/Python 3.14/default
features, not the full failing CI environment. The normal new-head macOS
Python 3.13/experimental-functions job remains the required platform
check.
- Python package Ruff format/check passed. Repository-root Ruff still
reports 59 existing errors and six formatting differences in unchanged
`docs/web-tests` files; none are in changed files.
- At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed;
5 targeted cases and strict workspace Clippy passed. The fixture-only
follow-up changes no production Rust/Python code, so that evidence is
reused.
- Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests
and 24 Python take/getitems tests passed; the focused 49-test suite also
covered offset dataloaders. A local 9,216-row, four-fragment
reproduction returned 200 seeded random batches of 64 rows with zero
ordering or label-alignment mismatches. The ordering and projection
implementation is unchanged by the follow-ups.
- No live Enterprise run was performed.
Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469).
The client registers Functions, but the decorator that packages one was
called `udf` and produced a `UdfDefinition`. Rename them to `function`
and `FunctionDefinition`, and say "function" instead of "UDF" in the
docstrings, error messages and docs around them.
`udf` and `UdfDefinition` stay as deprecated aliases that emit a
`DeprecationWarning`. The source packager strips either spelling of the
decorator, so `@udf` and `@function` produce the same artifact and
digest, and the wire contract is unchanged. DataFusion and Lance UDF
types, and Geneva's `@udf` in the integration docs, are unrelated and
untouched.
Clarify `FunctionVersion` forward compatibility: unknown top-level
fields are accepted during decoding and omitted from canonical JSON,
while all defined fields and the open `image.descriptor` content retain
their values. Extend the existing Rust and Python canonical round-trip
tests to assert the complete projected object, including future
discriminator values.
This documents and tests the existing behavior; it does not change
runtime serialization or promise lossless retention of unknown top-level
fields.
Related: ENT-2176.
Function refresh failures and skipped-row diagnostics lose the traceback
when decoded by the SDK. Expose the server-provided optional `traceback`
on job failure details and Function error records in Rust, Python, and
Node.js, while keeping existing compact error messages.
Responses from older servers remain supported when the field is absent.
Tracebacks may be truncated by the server. The Rust `Error::JobFailed`
variant now stores `Box<JobFailure>` so the added diagnostics do not
enlarge every `Result` error payload.
Function catalog responses include `batch_size`, `row_timeout_seconds`,
and `result_stability`, but the Rust and Python `FunctionVersion` models
silently discarded them. Preserve these optional execution controls
through decoding, typed access, and canonical serialization; versions
without the fields retain their existing representation.
The regression exercises raw SQL transport responses and public Python
sync/async get/list paths. Live Sophon catalog acceptance remains
pending under ENT-2073 and the related FunctionVersion round-trip
requirement in ENT-2176.
Update the Rust workspace Lance dependencies to
[v15.0.0-beta.10](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.10),
which contains the released MinHash signature scanning API needed by
downstream consumers. Pin Python tests to the published `15.0.0b10`
wheels, including macOS, and refresh both lockfiles. Java remains on
beta.8.
No SDK source compatibility changes were needed. All 25 Lance Git
packages resolve to release commit
`f56a08d40b5630ce1124535c4c3b1406d5ceedf1`; third-party package versions
are unchanged.
Validation on macOS ARM64 with pinned Rust 1.97.0: strict workspace
Clippy with default and all features, Rust formatting, 32 CI-script
tests, three count tests, one WAL/vector regression, and the Python
lockfile check passed. The native test binary emitted a macOS
compact-unwind size warning; both test commands completed successfully
without changing linker settings. Downstream release and integrated
Sophon acceptance remain separate.
Function error records expose physical row locations, which can change
when a table is compacted. Expose the server-provided optional stable
`row_id` through Rust and Python so callers can locate the same row
after compaction. Document stable-ID retries and retain physical
addresses for tables without stable row IDs. Older server responses
continue to deserialize with `row_id=None`.
The Python remote response test and Rust remote deserialization test
passed on verification SDK `4e1725fa352035e2079dfd235b8271cd05adf433`,
including decoding a legacy response without the new field. This branch
applies the same SDK patch to current main; intervening main changes
leave the affected error types and assertions unchanged. Formatting and
lint checks passed after correcting the retry examples to use the
`table.name` property. In a separate Linux server experiment, a real
skipped Function row retained `_rowid=506` after compaction changed its
physical address. The SDK field requires a server that emits stable row
IDs; this PR does not change server dependencies.
Remote SQL failures currently become generic runtime errors, and HTTP
table lookup can discard server error metadata. Preserve transport
status, current request ID, SQL job ID and worker request ID across Rust
and Python. Select typed column, table, reference and schema exceptions
only when an explicit server code matches the transport status.
Remote `HttpError` and `RuntimeError` handlers remain compatible. Failed
SQL jobs retain `JobFailedError` with a typed SQL cause when available.
Missing metadata stays absent; message text never determines the
exception class. Local missing-column errors retain their type and
qualified-field suggestions, and `JobFailure.source` documents
structured remote Flight SQL failures.
This branch integrates main at `a9ad4997`, including its released Lance
15.0.0-beta.9 dependency update and remote optimize support. The only
textual conflict was two independent additions to `test_remote_db.py`;
both test groups and all original assertions are retained. The feature
has no dependency or lockfile delta against current main.
SDK runtime validation on `03ee633a` (runtime source unchanged in
`f3f2d0b8`), macOS ARM64, pinned Rust 1.97.0 and the frozen uv Python
3.14 environment:
- Rebuilt the native extension from the integrated source. All 22
targeted Python tests passed: native local errors, HTTP/Flight metadata,
legacy sync/async table-not-found behavior, and the three main
remote-optimize regressions.
- All 52 Rust SQL, metadata/source and nested-field tests passed,
including the new main optimize cases.
- Root and Python-binding Rust Clippy with `-D warnings`, Rust
formatting, Python package Ruff and conflict-file formatting passed.
- Repository-wide Ruff reports 59 existing documentation findings, and
its format check identifies six documentation files. Every affected file
is byte-identical to current main; these checks are not reported as
passing.
- Direct integration review and AST comparison confirm both conflicting
test groups were preserved. Earlier exception-constructor/pickling
evidence remains applicable to the unchanged Python error modules.
The unlocked build exhausted disk again on a runner with the same label
but only 9.4 GiB free on a 72 GiB root filesystem. Keep the existing
runner and dependency caches, serialize Cargo compilation, and remove
only completed test/bench executables reported by Cargo. This job never
runs those binaries; normal binaries, libraries and build-script outputs
remain available. Fresh resolution, the ci profile, all features, and
test/bench compilation and linking are retained.
Three probes using Cargo and target-path validation pass with `-D
warnings`: all fixture targets link while ordinary outputs remain
usable; an undefined symbol still produces Cargo exit 101 and its linker
diagnostic; files outside the target directory are never removed. The
[current-head build-no-lock
job](https://github.com/lancedb/lancedb/actions/runs/41911613851/job/126860364065)
passed on `f3f2d0b8`: all compilation/linking completed, 20 completed
test executables were removed, and `target` retained about 7.3 GiB. This
successful run had a 145 GiB root with about 78 GiB free afterward; it
was not an identical rerun of the failed 72 GiB image. The driver tests
also passed in this job. All previous-head jobs ended naturally before
the push; new-head CI and review gates still apply.
Typed remote input exceptions require the server metadata contract.
Older servers retain generic transport errors and existing missing-table
compatibility. The optimize tests use mock servers and do not establish
downstream server acceptance.
BREAKING CHANGE: Rust callers with exhaustive matches on
`lancedb::Error` must handle the new `FieldNotFound` and `Sql` variants.
Local `Error::ColumnNotFound` failures now raise
`lancedb.exceptions.ColumnNotFoundError`, a `ValueError` subclass.
Python callers that previously caught only `RuntimeError` for this local
failure must catch `ColumnNotFoundError` or `ValueError`.
`Table::optimize()` previously returned `NotSupported` for remote
tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE
TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL.
Only `OptimizeAction::Compact` without a custom `remap_options` or
non-default `CompactionOptions` is supported remotely.
`OptimizeAction::All`, `::Prune`, and `::Index` all return a clear
`NotSupported` error instead of silently doing the wrong thing. A
branch-scoped table handle submits a `WITH (branch = '<branch>')`
clause; the statement doesn't validate the branch exists itself, so an
unknown branch surfaces as an async job failure rather than an error
from this call.
The server's SQL job adapter doesn't expose compaction metrics, so
`OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a
placeholder rather than real numbers — it must still be `Some`, since
callers (including the Python/Node bindings) unwrap it unconditionally
on success. This mirrors how `refresh_materialized_view_async` already
handles the same limitation for its own metrics.
The Python and Node `optimize()` convenience methods call a new shared
`Table::optimize_full`, which runs `Compact` and then, on a remote
table, skips `Prune` and `Index` instead of attempting and failing on
them — returning zeroed prune stats rather than submitting a compaction
job and then raising. Python's `RemoteTable.optimize()` also now
forwards `compaction_options` to the Rust layer, so a non-default value
is rejected before submission instead of being silently dropped.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Update Rust Lance dependencies to
[v15.0.0-beta.9](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.9),
including the scoped stable-row-ID conversion fix. Python tests use
published beta.9 wheels on Linux/Windows while macOS retains beta.4.
Java stays on beta.8, matching the original base, so this update does
not depend on beta.9 Maven publication or macOS Java packaging.
The dependency updater preserves workspace inheritance, the WAL test
wrapper uses the new StoredBatch module, and Linux/remote Rust CI uses
its installed mold linker. The Python lockfile also records the
already-declared OCI extra. CLI CI now caches Cargo downloads under a
separate key instead of restoring compiled targets: the previous runner
exhausted its disk during cache restoration, before compilation. Its
test matrix and commands are unchanged.
This branch includes #4820's row-freshness changes: successful rows
become current while skipped rows retain their values and remain
eligible for retry. Repair identities, replay handling and Restore
fencing protect publication completion.
BREAKING CHANGE: `StalenessPlan::row_dirty` is a new public field. Rust
callers using exhaustive struct literals must initialize it.
Validation:
- All reported non-Java checks passed on
`f5fa66fcbcf31a8eb188935f728735560c77c873`, including Rust
lint/Linux/remote, Python, NodeJS and CLI checks. Current head
`c660bc0ac9fd5920ad4b64be9c19f732ef4c56a5` preserves those product
sources, manifests, lockfiles and compiler/test commands; only the Java
pin restoration and CLI cache configuration differ. Current-head CI is
tracked separately.
- The complete Java directory is identical to the original PR base. The
scope correction reuses [Xuanwo's prepared and validated
commit](https://github.com/lancedb/lancedb/pull/4814#issuecomment-6626088466);
no new Java build was run.
- CLI workflow YAML parses successfully, its inputs match the pinned
rust-cache action schema, and the existing jobs, matrix and commands are
unchanged. Hosted execution must confirm the cache-restoration repair.
- Existing validation includes the updater regressions and 32 CI-script
tests, dependency-policy checks, count/WAL coverage, and #4820's 42
freshness tests, strict core Clippy and Rust doctest. Root Ruff
previously reported unchanged documentation-example issues;
Python-package Ruff passed.
This PR adopts the existing released Lance beta.9. A subsequent official
SDK release and normal Sophon dependency adoption remain separate
delivery steps.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Materialized-view creation and refresh can outlive the submitting
connection, but their SQL query IDs cannot be reopened through the Job
Registry. On servers that negotiate early handles, return the durable
registry job ID and use the existing job API to describe, reopen, wait
for, or cancel it. Older servers retain the connection-scoped query
handle. Registry refresh handles forward running and terminal progress
observations and establish the committed-version read-freshness fence
before wait returns.
Preserve a nested SQL failure ID separately from the outer job ID
through Rust and the Python native bindings, so Function failures remain
traceable. Rust callers constructing `JobFailure` directly must supply
the new optional `sql_job_id` field or use `..Default::default()`.
Validation: seven SDK regression cases and the docs-link test pass,
including plain wait and running-to-terminal progress observations; the
observer regression fails before the fix. Formatting and strict Clippy
pass. Earlier accepted coverage includes a real Flight test service
exercising materialized-view creation followed by registry
describe/reopen/cancel, and two Python tests after rebuilding the native
extension. The published product tree matches the validated candidate.
Refs ENT-3562. The matching Sophon server change is being validated
separately before publication.
Python Function declarations can now pin Hugging Face snapshots with
`models=[HuggingFaceModel(repo=..., revision=...)]`. The native Rust
registration path carries those declarations into `CREATE FUNCTION`, and
create/get/list results retain `image.models`. Functions without
declarations preserve their existing payload shape. The API reference
includes a model-loading example that initializes once per Function
instance.
Validation:
- The real SQL mock-service regression fails on the previous native
implementation and passes after the fix. It covers immediate/deferred
execution, requests with/without models, and create/get/list responses.
- Rust Function wire tests, strict Clippy, the public-type doctest, and
formatting.
- Rebuilt Python extension and targeted registration/golden tests.
The service-side snapshot preparation and cache are delivered separately
for ENT-3483.
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot
integration guide with `Table.fetch_blob_files`, document how to open
`.lance` datasets directly with `lancedb` alongside a
`pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in
installation docs that the optional `pylance` extra is only needed for
engine/connector authors or format-level fragment APIs.
`pydantic_to_schema` rejected three common field types with `TypeError: Converting Pydantic type to Arrow Type: unsupported type ...`. This maps them to native Arrow types:
| Python | Arrow |
| --- | --- |
| `datetime.time` | `time64("us")` |
| `datetime.timedelta` | `duration("us")` |
| `Literal[...]` | the type of its values, e.g. `Literal["a", "b"]` -> `utf8`, `Literal[1, 2]` -> `int64` |
`Optional[Literal[...]]` and `List[Literal[...]]` use the existing optional/list handling. A `Literal` whose values have different types (`Literal["a", 1]`) raises `TypeError` with a message stating that all values must share one type.
`datetime.time` is stored as timezone-naive wall-clock time. Pydantic model writes reject a `time` with non-null `tzinfo` before Arrow conversion, including direct, list, nested, and Enum values, with a `ValueError` explaining that the offset would be lost.
Tests cover schema mapping including optional and list forms, mixed-type `Literal` rejection, naive time/timedelta round trips, and timezone-aware rejection at the write path.
On remote tables, `create_index` returns as soon as the build is scheduled. Until it finishes, a vector index is missing from `list_indices`, a scalar or FTS index shows 0 indexed rows, and queries don't use the index. The docs didn't say any of this, so it was easy to assume the index was ready.
This adds that to the `create_index` and `list_indices` docs in Python (`Table` and `AsyncTable`) and TypeScript, and points to `wait_timeout` / `waitTimeoutSeconds`, `wait_for_index`, and `create_index_async` for blocking until the index is ready. It also explains that completion tracking is tied to the submitted build only when the server returns a job ID, while legacy name-based waits can return early when replacing an existing index.
Closes#4405
Expand the Linux Python test run without pandas and pylance from two files to
the suite while retaining the regular dependency run for optional-package
modules.
Move optional imports into tests that need them and use Arrow assertions where
pandas is not required, preserving the existing behavioral assertions.
Validation:
- Physical no-pandas/no-pylance broad run: 2884 passed, 316 skipped, 8 existing
marker cases deselected.
- Regular affected suite: 1687 passed, 39 skipped.
- Current-head hosted integration: all 15 checks and 4 workflows passed.
Co-authored-by: importcpp <57519936+importcpp@users.noreply.github.com>
Job callers currently lose the server's structured progress when
describing, listing, or waiting for a job. Expose Rust and Python
progress snapshots and optional wait observers using the existing
backend poller. Python supports callbacks, a caller-owned tqdm-like bar,
or `progress=True`; `status()` still returns a string and the default
wait behavior is unchanged.
Preserve unknown progress fields and phases from newer servers, and
report omitted counters as unknown. Each wait calibrates a caller-owned
bar from its first complete snapshot, then uses a high-water mark within
the same execution epoch; a new epoch or smaller plan resets the bar.
ETA is best-effort and omitted for missing, stale, paused, or
incompatible samples.
Newly submitted index and refresh handles forward observers through
their freshness wrapper while retaining successful-wait read
consistency. The Jobs guide includes draft Python/Rust snapshot and
observer examples for release; TypeScript progress support remains
tracked under ENT-3500.
Validation on the development EC2 host: the initial 11 Python progress
tests, REST fixture caller, and two Rust progress regressions passed.
After fixing freshness-wrapper forwarding, rebuilt the Python extension
and ran two submitted-handle Python tests plus the new Rust refresh and
index regressions (one test each); all passed. Strict Clippy, rustfmt,
and targeted Ruff passed. The final commit only wraps the new guide
section in an MDX draft and clarifies language support. Repository-wide
Ruff reports 48 existing findings in six unchanged docs/web-tests files.
This publishes the SDK portion of ENT-3500. Server-side live-split and
terminal-progress changes and the combined server/SDK end-to-end
scenario remain in progress. Refs ENT-3500.
BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress:
None`, `phase: None`, and `paused: false`. Literals for `JobDescription`
must add `progress: None`, `phase: None`, `queue_reason: None`,
`waiting_for_job_id: None`, and `paused_at_millis: None` when that
metadata is unavailable. Existing Python calls remain compatible.
Python UDF registration could not set `batch_size` or
`row_timeout_seconds`, and the Rust registration model discarded these
fields before submitting SQL. Expose both optional controls and preserve
them through registration, with invalid values rejected and unchanged
canonical bytes when the controls are omitted.
The timeout applies to each Python call: one row for scalar Functions
and one batch for vectorized Functions. Initialization keeps its
existing service deadline.
Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust
registration-contract tests, 2 outbound-SQL tests, and all 103 Python
slice2 tests passed after rebuilding the native extension. Rust
formatting and the repository Python CI checks (Ruff 0.9.9, python/
scope) passed. A real REST + Flight SQL server integration also passed
using the rebuilt SDK: `batch_size=2` limits observed calls in a direct
SELECT over 8 rows, initial materialized-view creation, and
append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime
passthrough; it does not exercise GPU initialization.
Refs ENT-3246. The original GPU initialization and video-decoding
scenario still needs verification on a current deployment; this SDK
change does not close that investigation.
The additional repository-root style checks report existing
docs/web-tests findings (nine format files and 48 lint diagnostics); all
affected files are unchanged from main. These are outside the Python CI
job scope.
Rust compatibility: callers constructing `FunctionRegistrationRequest`
must initialize the two new optional fields (`None` preserves defaults);
the request no longer implements `Eq`.
The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds
creation-guide documentation and integrates main’s namespace
repr/optional OCI embedding changes. The Function registration
implementation is unchanged. All 103 Python slice2 tests passed again on
EC2 using the unchanged native extension; Rust and server-bridge
evidence remains applicable.
Adds `OCIGenAIEmbeddings`, registered as `"oci-genai"`, embedding text with the Cohere models hosted on Oracle Cloud Infrastructure Generative AI (default `cohere.embed-v4.0`) via the `oci` SDK, alongside the existing Bedrock, Gemini and watsonx functions.
- Auth delegated to the OCI SDK: `API_KEY` / `SECURITY_TOKEN` profiles from the OCI config file, or `INSTANCE_PRINCIPAL` / `RESOURCE_PRINCIPAL`. No credentials are stored on the function or in table metadata; `compartment_id` can come from `OCI_COMPARTMENT_ID` so the OCID need not be baked into metadata either.
- Endpoint derived from `region` (default `us-chicago-1`) or overridden with `service_endpoint`; dedicated AI cluster endpoint OCIDs use `DedicatedServingMode`.
- Asymmetric Cohere input types (`SEARCH_DOCUMENT` for the source column, `SEARCH_QUERY` for queries); `truncate` and `output_dimensions` (embed-v4.0: 256/512/1024/1536) forwarded.
- Batched at the service limit of 96 inputs; empty inputs are not sent and come back as `None`.
- `ndims()` uses `output_dimensions` when set, otherwise a single cached probe request, so there is no model→dims table to keep in sync with the catalog.
- SDK retries disabled in favour of LanceDB's `max_retries` backoff; 401/403 service errors carry `status_code` so they surface without retrying, with an actionable message (expired session token, IAM policy, region, model).
- `oci>=2.165.1` added to the `embeddings` extra (the floor where `EmbedTextDetails.output_dimensions` exists).
Tests: 22 mocked unit tests (auth paths, batching, input types, config validation, retry behaviour, pickling, table-metadata round trip) and a live test skipped unless `OCI_COMPARTMENT_ID` is set, which passes against us-chicago-1 for embed-v4.0 at 1536 and 512 dims and embed-multilingual-v3.0 at 1024. Verified end to end with a real LanceDB table (`LanceModel` with `SourceField`/`VectorField`, 6 rows, two semantic queries ranked correctly). `ruff format --check`, `ruff check`, pyright and typos are clean.
Docs: a page under `docs/src/embeddings/available_embedding_models/text_embedding_functions/` in the shape of the remaining `voyageai_embedding.md`; the function is exported from `lancedb.embeddings` so the API reference picks it up. The hosted docs' available-models table lives outside this repo and would need a separate update.
A synchronous remote table opened in a namespace is displayed as if it
were in the database root. For example, `db.open_table("test",
namespace_path=["analytics", "features"])` shows
`RemoteTable(dev.test)`.
Include the cached namespace path using the existing `$`-delimited table
identifier: `RemoteTable(dev.analytics$features$test)`. Root output and
the leaf `table.name` stay the same. Using cached identity also keeps
repr available after pickle without reopening the native handle or
making a request.
Addresses the repr portion of ENT-3559. The native formatter already
carries the qualified identifier used by Python async and TypeScript;
this is a Python sync wrapper fix with no new API or naming rules.
Validation:
- Before the fix, three namespaced sync cases failed while root and
async controls passed. All six cases now pass through the real SDK and
mock HTTP endpoint.
- Covers default/explicit root, nested paths, valid
dotted/mixed-case/dashed names, and repr after pickle with the server
stopped. Requests remain limited to the original table open.
- Rebuilt current editable bindings in the repository's uv environment
with Python 3.12 and pinned Rust 1.97.0 through mbx.
- Ruff check and format check pass for the Python package (178 files).
- Required repository-wide Ruff check reports 48 pre-existing
diagnostics in six `docs/web-tests` files; the repository-wide format
check reports nine unchanged documentation files. Their source matches
the base commit. No lint suppression or documentation cleanup is
included.
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.
- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.
Linear: OSS-2772
Running a Function body in a local test or another engine currently
requires discovering undocumented APIs. Document direct calls to
decorated Functions and classes, including initialization fields,
instance reuse, and cleanup, and explain how they differ from registered
`FunctionVersion` applications.
Add examples of `BlobReader(read_range, size)` backed by an in-memory
byte-range callback, including cursor-independent reads. The local
examples and rendered API documentation were validated on EC2 at
`df6076e1ae605f425a380d282c4f25b845a3165f`; integration with current
main leaves these documented call paths unchanged.
Fixes ENT-3561.
Packages passed through `@udf(code=[...])` silently omit non-Python
resources, so registration can appear successful before a worker fails
to open a data file. Warn during function preparation with the omitted
paths and document the source-only boundary next to `code=`. The
diagnostic explains how to install a distribution that includes the data
and remove the package from `code=` to avoid shadowing that
installation.
Python cache files do not produce warnings. Excluded directories are
reported at their boundary, preserving the existing traversal and
Python-source artifact format. This implements the
warning-and-documentation alternative in ENT-3492; it does not add
resource uploads.
On EC2, the regression fails on baseline `f9389f113` because no warning
is emitted; all 60 packaging tests pass at `e23445b96`, using the
matching native extension built from unchanged Rust sources.
CI-scoped Python lint with Ruff 0.9.9 passes. Whole-site Mintlify
validation reports the same four snippet-import warnings on the baseline
and this head; repository-root Ruff likewise reports the same 48
findings under `docs/web-tests`. These checks are not clean passes, but
their failures do not involve this change.
Fixes ENT-3492
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.
Co-authored-by: Leo <leo@insight.co>
Add a Python regression test for merging metadata-free data into a table whose target schema has metadata. The test calls the native `execute_merge_insert` binding directly so it exercises the native partial-schema comparison without the public merge path normalizing source metadata.
The test verifies that merge insert succeeds with the expected rows and preserves the target schema metadata.
Fixes#2058
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.
Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.
SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.
Fixes#1959.
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
Fixes#4761
`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.
The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.
This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.
Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
Renaming or dropping an embedding source or vector column leaves schema metadata pointing at missing fields, causing later `add()` calls to fail.
Reject these operations in local schema evolution before committing any changes. The guard reads Python/Node embedding column metadata and native Rust `lancedb::column_definitions` without loading embedding models. It supports explicit and default Rust vector-column names and respects quoted column paths. Unrelated column changes and nullability updates remain supported.
The restriction is documented in the Python, Rust, and TypeScript APIs. Regression coverage includes both SDK metadata formats, native Rust definitions with explicit and default destinations, atomic rejection of batched changes, and continued embedding generation after reopening.
Fixes#4763
Validation at the exact PR head and GitHub test-merge:
- Hosted macOS Python wheel suite: 3061 passed, 162 skipped, 70 deselected, including all 8 sync and 4 async embedding schema-evolution cases.
- Hosted Windows Rust suite: 1900 library tests passed, 8 embedding integration tests passed, and 107 doctests passed, including the native explicit/default destination and binding guard cases.
- Hosted Linux Python 3.10 rerun: 3084 passed, 159 skipped, 50 deselected; the previously observed remote-fork timeout passed on the replacement job.
- All 10 applicable exact-head workflow runs completed successfully. The existing main-only Windows ARM job was skipped by its workflow condition.
Fixes#4773.
Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.
Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
A mixed-value Pydantic Enum previously produced a UTF-8 Arrow schema even when some member values were non-string, causing writes to fail after schema construction. `pydantic_to_schema` now rejects actual member values with more than one type using `TypeError`, including direct, optional, list, and nested fields. Homogeneous string and integer enums and empty enums retain their mappings, and existing plain-Enum writes remain supported. This intentionally changes mixed-value enums from failing at write time to failing at schema construction.
Validation on the exact head: the Python suite passed with 3034 passed, 126 skipped, and 70 deselected, including 12 mixed-value container cases, the empty-enum case, homogeneous schema coverage, and real plain-Enum writes. All applicable exact-head checks and workflows completed successfully.
Reject empty multivector result lists with `ValueError` before accessing the first result, matching the existing RRF and MRR behavior. Non-empty lists, zero-row tables, and both deduplication modes retain their existing behavior.
Regression coverage exercises the shared base reranker through both deduplication settings. The exact-head Python suite passed with 3021 passed, 126 skipped, and 70 deselected.
When `LinearCombinationReranker` uses `return_score="all"` and either the
vector or full-text side has no rows, the unavailable score column is now a
typed Arrow `float32` null column instead of a column of NaNs. Existing scores
from the non-empty side and their descending `_relevance_score` ordering are
preserved.
Regression coverage exercises both missing sides through the hybrid query path.
It checks the missing column's `float32` type, null count and row count, exact
preservation of the original present-side values, and descending relevance
ordering. The existing empty-result, score-column override, and `return_all`
cases remain covered.
The exact head `db33a068263cd69ee578f95a375f0060dcbb4e21` passed all 15
applicable check runs and all 4 applicable workflow runs, with no commit
statuses. The hosted macOS Python run completed with 3019 passed, 126 skipped,
and 70 deselected; the two new cases and nine adjacent regression cases passed.
The retained CI collector reached the all-success snapshot but exited 1 while
writing its final metadata because of a harness `TypeError`; it did not rerun
or invalidate the product or CI results.
BREAKING CHANGE: For `return_score="all"` with an empty vector or full-text
side, the missing side's score column now contains typed nulls instead of NaN.
Consumers that distinguish these values will observe different results.
The `Reranker` docstring says the default `return_score="relevance"` "will return only the relevance score" (`rerankers/base.py`), but a hybrid search whose filter matches nothing also returns `_distance` and `_score`. The early return for two empty result sets in `_combine_hybrid_results` (from #3030) skips the reranker's `_keep_relevance_score`, which every built-in reranker applies to a non-empty result.
Before, on `45dbf5e`:
```
$ python repro.py
match: ['id', 'text', 'vector', '_relevance_score']
no match: ['id', 'text', 'vector', '_distance', '_score', '_relevance_score']
```
After, `no match` also prints `['id', 'text', 'vector', '_relevance_score']`.
<details><summary>repro.py</summary>
```python
import tempfile, lancedb, pyarrow as pa
db = lancedb.connect(tempfile.mkdtemp())
t = db.create_table("t", pa.table({"id": [1, 2], "text": ["red fox", "blue fox"],
"vector": pa.array([[1.0, 2.0], [3.0, 4.0]], pa.list_(pa.float32(), 2))}))
t.create_fts_index("text")
q = lambda where: t.search(query_type="hybrid").vector([1.0, 2.0]).text("fox").where(where, prefilter=True).limit(5)
print("match: ", q("id > 0").to_arrow().column_names)
print("no match:", q("id > 100").to_arrow().column_names)
```
</details>
No issue exists for this. The fix calls `reranker._keep_relevance_score(empty)` before returning. Sync and async hybrid queries both go through it. With `return_score="all"` both columns stay, as before. Code reading `_distance` from an empty result now gets a `KeyError`, which it already got from every non-empty one. The new test asserts, for both settings, that a filter matching nothing gives the same columns as one matching every row.
### Verification
```
# Python 3.12.6; lancedb 0.41.0b15 wheel from pypi.fury.io, its _lancedb.abi3.so in the tree; no pylance (none >= 15 resolves)
# the wheel predates #4688, so a 3-line -p plugin sets _lancedb._experimental_functions_enabled = False for tests/conftest.py
$ PYTHONPATH=python python -m pytest -q -p no:cacheprovider python/tests/test_rerankers.py python/tests/test_hybrid_query.py python/tests/test_query.py
45dbf5e: 5 failed, 197 passed, 57 skipped
this branch: 5 failed, 199 passed, 57 skipped
query.py reverted, test kept: 6 failed, 198 passed, 57 skipped
FAILED test_rerankers.py::test_empty_hybrid_result_keeps_reranker_columns[relevance]
At index 2 diff: '_distance' != '_relevance_score'
$ ruff format --check <both files> && ruff check <both files> # ruff 0.9.9, as pre-commit pins it
2 files already formatted
All checks passed!
# the 5 failures fail identically on 45dbf5e: 4 test_query_serialization_async_search_options (wheel older than #4659) and
# test_hybrid_query_with_stale_fixed_size_binary_prefilter (needs pylance)
# not run: pyright, the Rust build, the remote client
```
@Xuanwo, a custom reranker that keeps extra columns still gets the generic empty table. I'd keep it that way, since #3030 returns early so that no reranker runs on empty input. If you'd rather call `rerank_hybrid` on the empty tables instead, I'll change it in this PR.
Part of #4364 (Python side; the TypeScript `FtsOptions` change will follow in a separate PR).
Lance already supports the code analyzer, but `FTS` and `lancedb.tokenize()` had no way to set it, so `FTS({'analyzer': 'code'})` and `FTS(split_identifiers=True)` raised `TypeError`.
## Changes
- `FTS` and `lancedb.tokenize()` gain `analyzer` (`"text"` or `"code"`), `split_identifiers`, `split_on_numerics`, `preserve_original` and `index_operators`. They default to `None`, which keeps the tokenizer's own defaults, so existing indexes and calls are unchanged.
- Validation follows pylance: `analyzer="code"` with a `base_tokenizer` other than `"code"` or the default `"simple"` is rejected, `analyzer="text"` with `base_tokenizer="code"` is rejected, and turning on a code flag without the code analyzer is rejected. Passing a flag as `False` is always allowed.
- `analyzer="code"` with the default `base_tokenizer="simple"` selects the code tokenizer. `FTS` can't tell an explicit `"simple"` from the default, so that case is accepted too.
- The deprecated `create_fts_index` is left alone; it points users at `create_index(config=FTS(...))`.
- Rust `FtsIndexBuilder` is Lance's `InvertedIndexParams`, so the Rust SDK already had these options. The remote client serializes the same struct, so the options reach LanceDB Distributed in the `create_index` request body. A new case in the Rust `test_create_index` checks that body.
## Tests
- `python/tests/test_fts.py`: tokenizer output for each flag, validation errors for `tokenize` and `FTS`, `False` flags on a text index, and local sync and async indexes that match `http`, `request` and `parseHTTPRequest` with splitting on and not match subwords with it off.
- `cargo test --features remote remote::table::tests::test_create_index`