Fixes#4773.
Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.
Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
## Problem
`RUSTSEC-2026-0429` affects `zip 7.4.0`, which reaches the optional
`sentence-transformers` feature through `candle-core 0.9.2`. The `deny` check on
`main` fails on this advisory.
## Change
Update `candle-core`, `candle-nn`, and `candle-transformers` to `0.11.1`. The
focused lockfile resolution replaces `zip 7.4.0` with `zip 9.2.2`, and no other
zip version remains in the all-features graph. This also pulls candle's required
newer transitive `tokenizers` and `safetensors` releases; the directly declared
`tokenizers 0.19.1` and the embedding implementation remain unchanged.
The optional `sentence-transformers` Rust API exposes this dependency version:
`SentenceTransformersEmbeddingsBuilder::device` accepts
`D: Into<candle_core::Device>`, and the `Device` type is now from candle-core
0.11. The default CPU path constructs that device internally and is unchanged.
`deny.toml` is unchanged. Its obsolete zip ignores now produce
`advisory-not-detected` warnings, which do not fail the check.
## Validation
- The `sentence-transformers` feature builds with tests and examples.
- The existing CPU example loads the default model and retrieves
`The human body has 206 bones.` for the matching query.
- A local contract check verified 384-dimensional, finite, non-zero Float32
output and higher similarity for the matching fact than for an unrelated fact.
It was not committed because it would add an 88 MB model download to
all-features CI.
- `cargo deny` `0.20.2`, the version used by the Rust workflow, passes
`advisories`, `bans`, `licenses`, and `sources`.
- The locally installed `cargo-deny 0.18.9` is not green: it treats
informational unsound advisories differently from the workflow's `0.20.2`.
The result above uses the exact CI version.
- Targeted clippy, formatting, lockfile review, and the reverse `zip` dependency
tree pass.
- Hosted CI completed successfully for exact head `bffbcda6490fa1ed51838b3801e01ae2f82faed6`.
CI mitigation: Reclaim completed lancedb CI-profile artifacts before the Blob v2 example after disk sampling showed a minimum of 24,764,416 available bytes. This may rebuild lancedb for the example; the exact OS cause of the SIGBUS remains unproven.
BREAKING CHANGE: The optional `sentence-transformers` Rust API now takes
`candle_core::Device` from candle-core 0.11. Downstream crates that explicitly
construct and pass a candle-core 0.9 `Device` to
`SentenceTransformersEmbeddingsBuilder::device` must use a `Device` from
candle-core 0.11.x aligned with lancedb. This normally means updating a direct
candle-core dependency; if the value comes through a re-export, update the
dependency that supplies it. Callers that use the default CPU path without
passing a `Device` do not need to migrate.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
The vector-search guides do not clearly explain when to use Auto, search
effort or a fixed probe count. They also describe `minimum_nprobes` as
the exact starting count, although Auto can search more partitions from
the start.
Explain how to choose public query controls for recall and latency, with
short executable examples. Clarify that probe limits apply per segment,
calibration targets are not recall guarantees, and more probes cannot
remove quantization error. Keep necessary client/server version guidance
in a short note.
Validation:
- The existing example test passes against published `lancedb==0.41.0`,
covering all four snippets, result counts, filtering, finite distances
and recall against an exact cosine query. The test body takes 0.20 s.
Generated snippets match their source.
- Vale reports no findings on added prose. Changed-function Python
formatting and `git diff --check` pass. Repository-wide Ruff has 137
findings identical to the previous scan, including seven pre-existing
findings in the example file.
- Mint 4.2.888 strict validation reports the same four
`RsExampleBody`/`RsExampleTail` import warnings as pristine base
`0b20965d`, with no additional warnings.
A mixed-value Pydantic Enum previously produced a UTF-8 Arrow schema even when some member values were non-string, causing writes to fail after schema construction. `pydantic_to_schema` now rejects actual member values with more than one type using `TypeError`, including direct, optional, list, and nested fields. Homogeneous string and integer enums and empty enums retain their mappings, and existing plain-Enum writes remain supported. This intentionally changes mixed-value enums from failing at write time to failing at schema construction.
Validation on the exact head: the Python suite passed with 3034 passed, 126 skipped, and 70 deselected, including 12 mixed-value container cases, the empty-enum case, homogeneous schema coverage, and real plain-Enum writes. All applicable exact-head checks and workflows completed successfully.
Reject empty multivector result lists with `ValueError` before accessing the first result, matching the existing RRF and MRR behavior. Non-empty lists, zero-row tables, and both deduplication modes retain their existing behavior.
Regression coverage exercises the shared base reranker through both deduplication settings. The exact-head Python suite passed with 3021 passed, 126 skipped, and 70 deselected.
When `LinearCombinationReranker` uses `return_score="all"` and either the
vector or full-text side has no rows, the unavailable score column is now a
typed Arrow `float32` null column instead of a column of NaNs. Existing scores
from the non-empty side and their descending `_relevance_score` ordering are
preserved.
Regression coverage exercises both missing sides through the hybrid query path.
It checks the missing column's `float32` type, null count and row count, exact
preservation of the original present-side values, and descending relevance
ordering. The existing empty-result, score-column override, and `return_all`
cases remain covered.
The exact head `db33a068263cd69ee578f95a375f0060dcbb4e21` passed all 15
applicable check runs and all 4 applicable workflow runs, with no commit
statuses. The hosted macOS Python run completed with 3019 passed, 126 skipped,
and 70 deselected; the two new cases and nine adjacent regression cases passed.
The retained CI collector reached the all-success snapshot but exited 1 while
writing its final metadata because of a harness `TypeError`; it did not rerun
or invalidate the product or CI results.
BREAKING CHANGE: For `return_score="all"` with an empty vector or full-text
side, the missing side's score column now contains typed nulls instead of NaN.
Consumers that distinguish these values will observe different results.
The `Reranker` docstring says the default `return_score="relevance"` "will return only the relevance score" (`rerankers/base.py`), but a hybrid search whose filter matches nothing also returns `_distance` and `_score`. The early return for two empty result sets in `_combine_hybrid_results` (from #3030) skips the reranker's `_keep_relevance_score`, which every built-in reranker applies to a non-empty result.
Before, on `45dbf5e`:
```
$ python repro.py
match: ['id', 'text', 'vector', '_relevance_score']
no match: ['id', 'text', 'vector', '_distance', '_score', '_relevance_score']
```
After, `no match` also prints `['id', 'text', 'vector', '_relevance_score']`.
<details><summary>repro.py</summary>
```python
import tempfile, lancedb, pyarrow as pa
db = lancedb.connect(tempfile.mkdtemp())
t = db.create_table("t", pa.table({"id": [1, 2], "text": ["red fox", "blue fox"],
"vector": pa.array([[1.0, 2.0], [3.0, 4.0]], pa.list_(pa.float32(), 2))}))
t.create_fts_index("text")
q = lambda where: t.search(query_type="hybrid").vector([1.0, 2.0]).text("fox").where(where, prefilter=True).limit(5)
print("match: ", q("id > 0").to_arrow().column_names)
print("no match:", q("id > 100").to_arrow().column_names)
```
</details>
No issue exists for this. The fix calls `reranker._keep_relevance_score(empty)` before returning. Sync and async hybrid queries both go through it. With `return_score="all"` both columns stay, as before. Code reading `_distance` from an empty result now gets a `KeyError`, which it already got from every non-empty one. The new test asserts, for both settings, that a filter matching nothing gives the same columns as one matching every row.
### Verification
```
# Python 3.12.6; lancedb 0.41.0b15 wheel from pypi.fury.io, its _lancedb.abi3.so in the tree; no pylance (none >= 15 resolves)
# the wheel predates #4688, so a 3-line -p plugin sets _lancedb._experimental_functions_enabled = False for tests/conftest.py
$ PYTHONPATH=python python -m pytest -q -p no:cacheprovider python/tests/test_rerankers.py python/tests/test_hybrid_query.py python/tests/test_query.py
45dbf5e: 5 failed, 197 passed, 57 skipped
this branch: 5 failed, 199 passed, 57 skipped
query.py reverted, test kept: 6 failed, 198 passed, 57 skipped
FAILED test_rerankers.py::test_empty_hybrid_result_keeps_reranker_columns[relevance]
At index 2 diff: '_distance' != '_relevance_score'
$ ruff format --check <both files> && ruff check <both files> # ruff 0.9.9, as pre-commit pins it
2 files already formatted
All checks passed!
# the 5 failures fail identically on 45dbf5e: 4 test_query_serialization_async_search_options (wheel older than #4659) and
# test_hybrid_query_with_stale_fixed_size_binary_prefilter (needs pylance)
# not run: pyright, the Rust build, the remote client
```
@Xuanwo, a custom reranker that keeps extra columns still gets the generic empty table. I'd keep it that way, since #3030 returns early so that no reranker runs on empty input. If you'd rather call `rerank_hybrid` on the empty tables instead, I'll change it in this PR.
Part of #4364 (Python side; the TypeScript `FtsOptions` change will follow in a separate PR).
Lance already supports the code analyzer, but `FTS` and `lancedb.tokenize()` had no way to set it, so `FTS({'analyzer': 'code'})` and `FTS(split_identifiers=True)` raised `TypeError`.
## Changes
- `FTS` and `lancedb.tokenize()` gain `analyzer` (`"text"` or `"code"`), `split_identifiers`, `split_on_numerics`, `preserve_original` and `index_operators`. They default to `None`, which keeps the tokenizer's own defaults, so existing indexes and calls are unchanged.
- Validation follows pylance: `analyzer="code"` with a `base_tokenizer` other than `"code"` or the default `"simple"` is rejected, `analyzer="text"` with `base_tokenizer="code"` is rejected, and turning on a code flag without the code analyzer is rejected. Passing a flag as `False` is always allowed.
- `analyzer="code"` with the default `base_tokenizer="simple"` selects the code tokenizer. `FTS` can't tell an explicit `"simple"` from the default, so that case is accepted too.
- The deprecated `create_fts_index` is left alone; it points users at `create_index(config=FTS(...))`.
- Rust `FtsIndexBuilder` is Lance's `InvertedIndexParams`, so the Rust SDK already had these options. The remote client serializes the same struct, so the options reach LanceDB Distributed in the `create_index` request body. A new case in the Rust `test_create_index` checks that body.
## Tests
- `python/tests/test_fts.py`: tokenizer output for each flag, validation errors for `tokenize` and `FTS`, `False` flags on a text index, and local sync and async indexes that match `http`, `request` and `parseHTTPRequest` with splitting on and not match subwords with it off.
- `cargo test --features remote remote::table::tests::test_create_index`
Update the Rust workspace and Java SDK dependencies to [Lance
v15.0.0-beta.8](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.8),
preserve the optional `manifest_size` field in remote version responses,
and verify directory-namespace renames preserve registration and data
both within and across namespaces. Update the Rust and TypeScript API
documentation to reflect directory-namespace rename support.
Rust callers constructing `Version` literals must now provide
`manifest_size` (`None` when unavailable). Java is intentionally pinned
to beta.8 even while Maven artifacts are still publishing;
dependency-resolution CI failures caused solely by that publishing delay
are accepted for this upgrade. Python package pins remain unchanged.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
## Summary
- Add sync/async `DBConnection.list_gpu_classes()` returning typed
`GpuClass` records for `GET /v1/gpu/classes`. Records include the class
name, Kubernetes resource, GPU model and memory, per-worker GPU maximum,
and worker CPU/memory/ephemeral-storage limits.
- Add `udf(gpu_class=...)` for immutable Function-level GPU placement.
It requires `gpu=True`; Function versions expose their registered
`gpu_class`.
- Add `gpu_class` and `gpu_count` to `refresh_column` /
`refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as
top-level `backfill_column` fields and validated before any request.
`gpu_count` is the aggregate GPU budget across Function workers.
- Validate GPU class names with the server lowercase 63-character
DNS-label rule.
- Document the discovery-to-registration flow, configured hardware
versus free capacity, System Usage permission, and per-refresh GPU
placement in the API and web guides.
Older servers without `/v1/gpu/classes` produce a clear
`NotImplementedError` instead of a bare 404.
Part of ENT-3470. The client follows the in-flight Sophon contract on
`xuanwo/ent-3470-fix` for GPU class discovery and backfill placement
options.
## Breaking change
`PythonRuntimeSpec::PythonV2` adds `gpu_class`, and
`RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers
constructing either value with all fields explicitly must supply the new
fields.
## Validation
At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python
extension build, pyright, ruff, and Python Function/runtime, discovery,
refresh placement, and refresh regressions (147 passed, 6 skipped).
At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright,
Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed).
The final commit only updates documentation and Rust doc comments.
## Summary
- Make `AsyncJob.wait(timeout)` and `Job.wait(timeout)` raise a
descriptive `TimeoutError` containing the job ID and elapsed timeout.
Registry jobs explain that work continues on the server and include the
`db.open_job(id).wait()` reattach call. SQL query handles instead say to
keep the original handle because query IDs cannot be reopened through
the registry jobs API.
- Render `JobFailureInfo` and `FunctionErrorRecord` with Python values
instead of Rust `Option` wrappers.
- Decode and expose the server's `row_address` on `FunctionErrorRecord`,
and document how to retry recorded rows with `REFRESH COLUMN ... WHERE
_rowaddr IN (...)` before compaction or rewrite.
- Warn when an import inside a packaged Function body resolves to
undeclared local source. The `udf` documentation explains why arbitrary
base-image coverage and distribution-to-import-name mapping cannot be
validated reliably by the SDK.
`row_address` is optional, so records from older servers still decode
with `None`. Rust code that constructs `FunctionErrorRecord` literals
must now supply this new field.
Part of ENT-3448 and ENT-3442. This consumes the server-side diagnostics
from lancedb/sophon#9509 and logical row addresses from
lancedb/sophon#9512.
## Validation
- Rust `function_errors` remote decoding and
legacy-missing-`row_address` tests.
- Rust/PyO3 clippy for `lancedb` and `lancedb-python`.
- Python extension build, pyright, and ruff.
- Targeted Python job, Function packaging, Function error, and job
detail tests: 62 passed, plus timeout-handle tests: 3 passed.
## Summary
- Parse structured `error_code` and `resource` fields from remote HTTP
error bodies and carry them through Rust/PyO3.
- Add `SecretNotFoundError`, `FunctionNotFoundError`,
`NamespaceNotFoundError`, and `AlreadyExistsError` as `HttpError`
subclasses.
- Expose `error_code`, `resource_kind`, `resource_name`, and
`namespace_path` on HTTP exceptions while preserving `status_code`,
`request_id`, and existing `except HttpError` handlers.
- Do not retry structured domain errors that would otherwise become
`RetryError` (notably `already_exists` on 409).
Older servers that do not provide the structured fields continue to
raise a plain `HttpError`.
## Compatibility
This consumes the stable resource error contract added in
lancedb/sophon#9516 (merge commit `35858561b3`). The existing
`TableNotFoundError` behavior is unchanged.
Part of ENT-3373.
`list_functions()` promises every published immutable revision. Request
`SHOW FUNCTIONS HISTORY` so the SQL server's latest-only default does
not hide older revisions, and preserve the server's numeric revision
ordering when constructing page cursors (`1`, `2`, `10`).
```python
[(fn.name, fn.version) for fn in db.list_functions(page_limit=2)]
# [("embed", "1"), ("embed", "2"), ("embed", "10")]
```
```sql
SHOW FUNCTIONS; -- latest visible revision per name
SHOW FUNCTIONS HISTORY WITH (limit = 3, include_definition = true);
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '2', limit = 3);
```
Each listing page uses the existing SQL query and polling transport; the
continuation token stores the last returned name/revision and remains
bound to its database and namespace. Function mutations keep their
query-backed job handles: `Job.id` is the query's ID and `job_type` is
`query`, so waiting observes that same SQL execution. No registry job or
second query ID is added.
Update the SQL and Python documentation for exact-revision drops:
sibling revisions and grants survive, and only the last visible
revision's removal releases the name and schedules object cleanup.
Extend the existing Flight SQL regression to paginate revisions `1`,
`2`, and `10` and verify the explicit history statement. Formatting and
static Python checks pass; local builds and tests were intentionally not
run.
View APIs now read the native `name`, `query`, and `schema` columns
returned by `CREATE VIEW` and `DESCRIBE VIEW`. The schema JSON carries
the defining database and namespace; the requested namespace identifies
where the view lives. The unreleased JSON-envelope response is removed
without a compatibility fallback.
```python
view = db.create_view("adults", "SELECT name FROM people WHERE age >= 18")
view = db.describe_view("adults")
```
```sql
CREATE VIEW "adults" AS SELECT name FROM people WHERE age >= 18;
DESCRIBE VIEW "adults";
```
Both Python calls execute SQL through the query-backed job adapter and
wait on the resulting query. A job handle uses that query's `id` and
type `query`; reading metadata consumes its completed result rather than
resubmitting the DDL. Creation returns the definition from the creating
execution, with no extra describe request.
Secret APIs now submit the native SQL statements with the credential
bound as an Arrow string parameter:
```sql
CREATE SECRET "api-key" TYPE = GENERIC_STRING SECRET_STRING = $1;
ALTER SECRET "api-key" SET SECRET_STRING = $1;
DESCRIBE SECRET "api-key";
```
`create_secret` and `alter_secret` execute SQL through the existing
query-backed job adapter and wait for that query to complete. The job's
`id` is the query ID and its type is `query`; credentials stay out of
SQL text. `describe_secret` reads the native `name`,
`created_at_millis`, and `updated_at_millis` columns. This replaces the
unreleased `AS $1` syntax and JSON-envelope response without a
compatibility fallback.
Python methods and namespace handling remain the same:
```python
db.create_secret("api-key", "credential", namespace_path=["team"])
db.alter_secret("api-key", "rotated", namespace_path=["team"])
info = db.describe_secret("api-key", namespace_path=["team"])
```
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
Fixes#4749
Python's `__version__` forwards Cargo's SemVer spelling, so prereleases
report values such as `0.41.0-beta.17` while the installed package
metadata reports `0.41.0b17`.
Normalize the native version with the existing
`packaging.version.Version` dependency before exposing it in Python.
Keep native version sourcing so imports still work with `lancedb-compat`
and do not require distribution metadata. Regression tests cover
metadata equality and stable, alpha, beta, release-candidate, and
development versions with metadata lookups disabled.
Validation:
- Rebuilt the native extension through the uv-managed development
environment.
- Confirmed the metadata regression fails on the original code with the
reported string mismatch.
- Import tests, package metadata tests, the remote client user-agent
test, and CI Python dependency tests: 14 passed, 2 skipped (Windows-only
wheel check and uninstalled `lancedb-compat`).
- macOS ARM64/Python 3.13 dependency resolution and `uv lock --check`
pass with the platform-specific test dependency pins from current
`main`.
- Python-tree Ruff lint and formatting checks, focused Pyright check,
Rust formatting, and PR diff whitespace checks pass.
- Repository-wide Ruff lint reports existing findings in unchanged
`docs/web-tests` files.
<!-- lance-gatekeeper-fix:v1 agent=3e23555743cc116ea53ca8a7e3c4ebb1
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4750
Remote missing-table errors retained HTTP metadata in Rust but the Python binding converted them to a plain `ValueError`, discarding the request ID and status. This raises a `TableNotFoundError` that preserves the original message, request ID, status code, and chained HTTP cause while remaining compatible with existing `ValueError` and `lance_namespace.errors.TableNotFoundError` handlers. Local and non-HTTP errors retain their previous behavior.
Validation covers sync and async opens across cached/uncached and root/nested paths, exception pickling, local compatibility, the remote and error test suites, Clippy, formatting, Ruff, and all applicable PR checks including CodeQL.
`LanceModel` instances with validation aliases write field-name keys to Arrow, while `to_pydantic` previously reconstructed models by alias. This normalizes field-name rows to the applicable single-component validation alias for both sync and async reads, while preserving existing alias keys, alias precedence, name-only validation, and collision behavior.
Adds sync and async regression coverage for string aliases, `AliasChoices`, single-component `AliasPath`, `populate_by_name`, and explicit `validate_by_name` precedence.
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model
names outside the three built-in OpenAI names, so schemas can be built for
compatible models.
Preserve existing model defaults and request behavior. Add regression coverage
for the resulting Arrow schema dimensions and for all known OpenAI model
dimensions.
Allows Python namespace implementations to read request dictionary entries as attributes while preserving mapping and `model_dump()` behavior. Missing dunder attributes continue to raise `AttributeError`.
Adds a real namespace connection regression covering list/open operations and omitted optional fields.
Fixes#4701
The invalid-table-name diagnostic now includes the missing space before
"periods are allowed".
The regression test covers the corrected message returned for `my/table`.
Add a regression test for the public optimize-all path on a table with an IVF_HNSW_SQ index. The test creates 512 rows, runs the full optimize action, and verifies that the single retained index still covers all 512 rows with zero unindexed rows.
Test-only change; no production behavior changes.
References #3211.
The core `col()` camelCase integration coverage requested by #3211 is already on `main` from #4493. This PR adds the adjacent raw-SQL table-surface regressions that were previously uncovered:
- unquoted `where("camelCase = 1")`
- `update` and `delete` with unquoted camelCase predicates
- `mergeInsert` with a camelCase join key, including matched-update and not-matched-insert input
- `addColumns` `valueSql` referencing a camelCase column
- `createIndex` on a camelCase column
These are characterization tests for the existing local-backend behavior. They do not add `col()` coverage or require a live remote service.
## Summary
- add a regression test for object-form `createTable` on two independent local Node.js connections
- verify both connections use the same table name while preserving each database's rows
## Scope
This test covers the public local `connect(tmpDir)` path. It does not exercise LanceDB Cloud or remote create requests, and it does not establish a remote root cause or a complete fix for #952.
References #952
Fix the dead README ecosystem links by pointing readers to the current official LanceDB documentation for LangChain and LlamaIndex.
Original contribution by @sarankumar1325.
Python column refresh methods rejected `batch_size`, and the Rust remote
backend never included it in the backfill request. Callers could not
reduce the rows passed to a slow Function to stay within the executor's
per-call deadline.
Add optional, keyword-only `batch_size` arguments throughout the
synchronous and asynchronous Python APIs and bindings. Add compatible
Rust `RefreshColumnOptions` APIs while preserving existing Rust method
signatures. Reject non-positive sizes before submitting a job, send
positive sizes in `backfill_column`, and omit the field when unset while
retaining branch scoping.
Batch size controls remote Function calls. Embedded expression refreshes
accept and validate the option without changing execution, and blocking
remote refreshes retain their existing unsupported behavior; remote
callers use `refresh_column_async` and wait on the returned job.
Validation:
- Reproduced the original Python `TypeError` from the committed method.
- Five focused Rust tests passed, covering request contents, omission,
branch scoping, and rejection before any request.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo clippy --quiet --features remote --tests --examples -- -D
warnings` passed.
- Nineteen targeted Python tests passed against the rebuilt extension,
including synchronous and asynchronous remote requests, invalid sizes,
and default-build behavior. Eight embedded refresh cases require the
experimental-functions build and were skipped.
- Rust formatting and Python formatting/lint checks passed, including
CI's pinned Ruff version. Repository-wide Ruff reports 48 existing
findings in untouched documentation files; none are in changed files.
Fixes#4776
<!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
A second flattened Function declaration fails when the remote describe
schema omits an existing binding's internal assignment field. The client
incorrectly requires that server-owned field and compares it as a
physical user output.
Exclude assignment fields from binding/schema validation and the
output-schema comparison. Filter declared assignment fields from remote
schemas, including seeded caches and snapshot handles, while retaining
schema metadata.
The regression tests declare two all-nullable flattened bindings
consecutively with no assignment declaration, a hidden assignment field,
or a returned assignment field. They also cover cached and snapshot
schemas and server-owned assignment types and metadata. All four
regression cases failed before the fix.
Validation:
- `cargo fmt --all`
- `cargo test --quiet --profile ci --features remote -p lancedb --lib --
table::computed_columns::tests remote::table::tests` — 379 passed, 30
ignored.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`
Fixes#4778
<!-- lance-gatekeeper-fix:v1 agent=c55eff66e22d736a6b954a9eeec88898
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Adds `HUGGINGFACE` and an optional token Secret reference to the
existing storage integration methods. The same SQL transport handles
create, describe, list and drop. Only a Secret reference is stored in
integration metadata; mixed-provider credential options are rejected.
```python
from lancedb.storage_integration import SecretReference
info = db.create_storage_integration(
"private_hub", "hf://datasets/acme/private/raw/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference(
"hf_token", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```
Omit the token Secret for anonymous access at Hugging Face's default
anonymous
rate limits. A tokenless integration does not use a cluster's Hugging
Face token:
```python
info = db.create_storage_integration(
"public_hub", "hf://datasets/acme/public/raw/",
storage_provider="HUGGINGFACE",
)
```
```sql
CREATE STORAGE INTEGRATION public_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```
```sql
CREATE STORAGE INTEGRATION private_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
HUGGINGFACE_TOKEN_SECRET = shared.credentials.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing SQL transport]
SQL --> Metadata[Integration and token Secret reference]
```
Adds Rust request construction, SQL rendering, typed metadata decoding,
sync/async Flight SQL regressions and public documentation. Local
builds/tests are excluded by request; CI provides execution validation.
Adds GCP inputs to the existing database-owned storage integration
methods. GCP takes one Secret reference; the Secret stores the complete
application credentials JSON as an opaque string. The same methods and
SQL transport handle create, describe, list and drop.
```python
info = db.create_storage_integration(
"gcs", "gs://customer-data/raw/", storage_provider="GCP",
gcp_credentials_json_secret=SecretReference(
"gcp_json", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("gcs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("gcs")
```
```sql
CREATE STORAGE INTEGRATION gcs
STORAGE_PROVIDER = 'GCP'
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = shared.credentials.gcp_json;
DESCRIBE STORAGE INTEGRATION gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION gcs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing Flight SQL transport]
SQL --> Config[Configuration and one Secret reference]
```
Adds Rust request construction, SQL rendering and typed metadata
decoding, plus sync/async Flight SQL regressions across the three Arrow
string encodings. Rejects missing or mixed-cloud credential options.
Local builds/tests are excluded by request; validation runs in CI.
Register Azure Blob integrations through Flight SQL using an explicit
storage account and three Secret references for tenant ID, client ID and
client secret. Extends the database-owned integration APIs with
provider-specific inputs while keeping the same create, describe, list
and drop operations.
```python
info = db.create_storage_integration(
"customer_blobs", "az://customer-data/raw/",
storage_provider="AZURE", azure_storage_account="customeraccount",
azure_tenant_id_secret="tenant", azure_client_id_secret="client",
azure_client_secret_secret="credential",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("customer_blobs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("customer_blobs")
```
Sync and async connections accept the same options. Use
`SecretReference` for cross-database or namespace qualification. Values
remain in Secrets; returned metadata includes only references and setup
guidance.
```sql
CREATE STORAGE INTEGRATION customer_blobs
STORAGE_PROVIDER = 'AZURE'
STORAGE_PREFIX = 'az://customer-data/raw/'
STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
STORAGE_AZURE_TENANT_ID_SECRET = public.tenant
STORAGE_AZURE_CLIENT_ID_SECRET = public.client
STORAGE_AZURE_CLIENT_SECRET_SECRET = public.credential;
DESCRIBE STORAGE INTEGRATION customer_blobs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_blobs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Flight SQL]
SQL --> Metadata[Configuration and Secret references]
```
Azure supports service principal client credentials only. Adds SQL
rendering, provider validation and Python Flight SQL regressions across
Utf8, LargeUtf8 and Utf8View results. Local builds and tests are
excluded as requested; validation runs in CI.
Add `on_transform_error="fill"` to `StreamingDataset`, replacing failed
transform rows at their original positions with a deep-copied fill value or
the result of `repair(exc, row_batch)`.
Fill mode preserves split length, elastic step layout, worker accounting, and
checkpoint progress. Repair failures remain fatal, and `rows_filled` is shared
across workers.
Validated with focused fill and adjacent dataloader tests (10 and 6 passed)
and the complete applicable CI run for
6b51b33e56: four workflows and fifteen checks
all succeeded.
Closes#3916
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.
Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.
The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.
Validated with focused Rust and Python override tests and the complete applicable
CI run for 43bccbadc5.
Resolve retry settings from the six documented `LANCE_CLIENT_RETRY_*`
environment variables through the shared retry configuration. Explicit
configuration takes precedence, invalid values return `InvalidInput`, and
status lists tolerate whitespace and trailing commas while a blank list
disables status-based retries.
Saturate calculated backoff at one hour instead of panicking when the
exponential exceeds the representable `Duration` range, including retry
counts configured from the environment.
Includes coverage for defaults, all six environment variables, explicit
configuration precedence, status parsing, invalid environment and direct
configuration values, real process-environment wiring, and saturating
backoff.
Closes#2350
Originally authored by Joshua Snider (@jsnider3).
Replaces #4322, keeping only its property graph defined over tables.
A property graph maps node labels and relationship types to existing tables of its namespace, in the shape of SQL/PGQ's `CREATE PROPERTY GRAPH`. It stores only that definition: a query reads the tables as nodes and edges through SQL `GRAPH_TABLE`, as they are when it runs.
- **Rust:** `Connection::create_property_graph` takes a `PropertyGraphDefinition` of `NodeTable`s and `EdgeTable`s. `describe_property_graph`, `drop_property_graph` and `list_property_graphs` manage it.
- **Python:** `lancedb.graph` has `NodeTable`, `EdgeTable` and `Endpoint`. Sync and async connections gain `create_property_graph`, `open_property_graph`, `drop_property_graph` and `list_property_graphs`, and a `PropertyGraph` handle describes its graph.
- **Support:** remote connections only. Local connections raise `NotImplementedError` in Python and return `NotSupported` in Rust.
```python
import lancedb
from lancedb.graph import EdgeTable, Endpoint, NodeTable
db = lancedb.connect("db://demo", api_key="...")
db.create_property_graph(
"social",
nodes=[NodeTable("person", key="person_id", label="Person")],
edges=[
EdgeTable(
"knows",
label="KNOWS",
source=Endpoint("src_id", references=("person", "person_id")),
destination=Endpoint("dst_id", references=("person", "person_id")),
)
],
)
db.execute_query("""
SELECT * FROM GRAPH_TABLE (social
MATCH (p:Person)-[:KNOWS]->(f:Person) WHERE p.age > 30
COLUMNS (p.name AS person, f.name AS friend))
""").read_all()
```
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
Fixes#4403
Implement remote drop_all_tables() support by listing tables and dropping them individually, including tables within namespaces. Handle paginated table listings and propagate errors appropriately. Add tests covering root and namespace tables, pagination, empty databases, and listing failures to ensure reliable remote database cleanup.
Rename the SQL-facing table functions:
- `vector_dedup` becomes `dedupe`.
- `vector_duplicate_pairs` becomes `duplicate_pairs`.
This is an intentional breaking rename without compatibility aliases. It updates the native table-function diagnostics and SQL coverage; the coordinated Sophon PR will update distributed binding and deployment integration.
BREAKING CHANGE: `vector_dedup` is now `dedupe`, and `vector_duplicate_pairs` is now `duplicate_pairs`; no compatibility aliases are provided.
The affected jobs were cancelled before runner assignment with GitHub's scheduled macOS 14 brownout annotation. macOS 14 runner images retire on November 2, 2026: https://github.com/actions/runner-images/issues/13518.
This moves the Python and CLI workflows to `macos-15` and keeps two distinct Rust macOS legs with `macos-15` and `macos-26`. All selected labels are public ARM64 runners, preserving the architecture of the previous plain `macos-14` jobs.
Validation: `git diff --check` and Ruby YAML parsing passed for all three changed workflows. A targeted `actionlint` check also passed; existing repository-wide custom-runner, reusable-workflow input, and shellcheck findings are outside this diff.
Package the standalone lancedb CLI with every stable and beta GitHub
release. Publish archives for Linux x86_64, Linux ARM64, and Apple
Silicon macOS, along with SHA-256 checksums, licenses, and third-party
notices.
Synchronize the CLI version with the SDKs and validate that release tags
match both manifests. Add PR and manual packaging dry runs, runtime
compatibility checks, and smoke tests of extracted executables. GitHub
release creation now waits for all three archives to pass validation;
SDK registry publishing remains independent.
Update installation instructions and the release checklist. Linux
binaries target glibc 2.28, retaining existing x86_64 CPU requirements;
macOS binaries target macOS 14 or newer.
Adds `image` to `@udf`, setting `runtime.environment.image` so a
Function can be built on a digest-pinned registry image instead of the
default base, for dependencies pip, conda and apt cannot express. The
client refuses references that are not pinned by digest; the server
validates the rest. The field is omitted when unset, so existing
Function identities are unchanged.
Stacked on #4675.
Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.
`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
Stacked on #4674.
Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens.
The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
Removes the Codex Fix CI workflow. It ran an AI agent that read CI logs,
which outside contributors can write to, while the agent had the
repository's write token and no sandbox. Hardening it would take ongoing
work, and we don't use it enough to justify that, so we're removing it.
## Merge Risk
**Door:** two-way. The workflow can be restored from history.
**Blast Radius:** ci. Only affects manual dispatches of this workflow.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates.
Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`.
Document vectorized `pyarrow.Array` UDFs for Blob v2 inputs and outputs.
Clarify annotation-selected array execution and the adapter-v1 client/executor
boundary. Add public regression coverage for registering, hydrating, executing,
publishing, and reading back Blob arrays, including large, empty, and null values.
Use `ThreadingHTTPServer` for the shared remote-test mock so concurrent `table.add` requests are not handled serially by the test server.
Validation:
- Reproduced the connection-reset/response-decoding failure against base `13749a3f138fde2647c8bdb8a1c05545026aee57`: 27 targeted runs passed, then the 28th matched the original CI failure.
- The patched target passed 100/100 runs; `test_remote_db.py` passed 261 tests with 2 skipped.
- Ruff 0.9.9 format and lint checks passed.
- Current-head GitHub CI: 19/19 check runs and 5/5 workflow runs succeeded.
## Summary
- Add a downstream IVF-HNSW-SQ merge regression using descending row IDs.
- Detect any reintroduction of merge-time row sorting without allocating the original multi-gigabyte fixture.
## Root cause
Lance 4 sorted each partition by row ID with `RecordBatch::take`. Arrow represented those take indices as `UInt32`, and `take_fixed_size_list` overflowed when `num_rows * vector_dimension` exceeded `u32::MAX`. The Lance version pinned by current `main` already includes lance-format/lance#6332, which removes that unnecessary sort. This test locks the corrected behavior at the LanceDB dependency boundary: the old implementation reorders the sentinel row IDs, while the fixed implementation preserves them.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet --features remote -p lancedb --lib table::create_index::tests::test_ivf_hnsw_sq_merge_preserves_shuffled_row_order -- --exact`
- Full current-head PR CI: 40 successful checks and one expected Windows ARM check skipped because that job runs only on `main`.
Fixes#3126
lancedb-gatefixer[bot]
Xuanwo
TaylorandBrendan Clement
Yang Cen
Minh Vu
SHAIK SHIVAJI
Christopher Krah
Lance Release
LanceDB RobotandJack Ye
BHAWESHBHASKAR
Hien Hoang
alexcastgonz
Yui Numazawa
Prince Jain
sarankumar
importcpp
Bruno Ramirez
Joshua Snider
ryan-haynesandRyan Haynes
somil jain
Colin Patrick McCabe
Wyatt Alt
Will JonesandClaude Opus 5.5