Commit Graph
3437 Commits
Author SHA1 Message Date
Yang Cen b49b603ebc fix(rerank): use one-based ranks for reciprocal rank fusion (#4801)
Rust RRF counted ranks from zero, while Python and the [RRF
paper](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) count
from one. At the default `k=60`, this reverses the order of two rows
ranked `(9, 6)` and `(2, 14)` by the two searches.

Start ranks at 1 in the Rust reranker, which is also used by Node, and
document the formula. This corrects relevance scores and can change the
order of close results.

Validation:

- Both RRF regression tests fail before the fix and pass afterward,
covering `k=1` and the default `k=60` rank reversal.
- All 8 hybrid tests pass, including actual query scores, an empty FTS
side, empty results, and remote query paths.
- `cargo fmt --all` and `cargo clippy --profile ci --features remote
--tests --examples -- -D warnings` pass.
2026-11-03 17:45:47 +08:00
Xuanwo ad70597435 feat(python): expose Function batch size and row timeout controls (#4807)
Python UDF registration could not set `batch_size` or
`row_timeout_seconds`, and the Rust registration model discarded these
fields before submitting SQL. Expose both optional controls and preserve
them through registration, with invalid values rejected and unchanged
canonical bytes when the controls are omitted.

The timeout applies to each Python call: one row for scalar Functions
and one batch for vectorized Functions. Initialization keeps its
existing service deadline.

Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust
registration-contract tests, 2 outbound-SQL tests, and all 103 Python
slice2 tests passed after rebuilding the native extension. Rust
formatting and the repository Python CI checks (Ruff 0.9.9, python/
scope) passed. A real REST + Flight SQL server integration also passed
using the rebuilt SDK: `batch_size=2` limits observed calls in a direct
SELECT over 8 rows, initial materialized-view creation, and
append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime
passthrough; it does not exercise GPU initialization.

Refs ENT-3246. The original GPU initialization and video-decoding
scenario still needs verification on a current deployment; this SDK
change does not close that investigation.

The additional repository-root style checks report existing
docs/web-tests findings (nine format files and 48 lint diagnostics); all
affected files are unchanged from main. These are outside the Python CI
job scope.

Rust compatibility: callers constructing `FunctionRegistrationRequest`
must initialize the two new optional fields (`None` preserves defaults);
the request no longer implements `Eq`.

The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds
creation-guide documentation and integrates main’s namespace
repr/optional OCI embedding changes. The Function registration
implementation is unchanged. All 103 Python slice2 tests passed again on
EC2 using the unchanged native extension; Rust and server-bridge
evidence remains applicable.
2026-11-03 16:38:22 +08:00
Fede Kamelhar 8be1f8e00f feat(python): add OCI Generative AI text embedding function
Adds `OCIGenAIEmbeddings`, registered as `"oci-genai"`, embedding text with the Cohere models hosted on Oracle Cloud Infrastructure Generative AI (default `cohere.embed-v4.0`) via the `oci` SDK, alongside the existing Bedrock, Gemini and watsonx functions.

- Auth delegated to the OCI SDK: `API_KEY` / `SECURITY_TOKEN` profiles from the OCI config file, or `INSTANCE_PRINCIPAL` / `RESOURCE_PRINCIPAL`. No credentials are stored on the function or in table metadata; `compartment_id` can come from `OCI_COMPARTMENT_ID` so the OCID need not be baked into metadata either.
- Endpoint derived from `region` (default `us-chicago-1`) or overridden with `service_endpoint`; dedicated AI cluster endpoint OCIDs use `DedicatedServingMode`.
- Asymmetric Cohere input types (`SEARCH_DOCUMENT` for the source column, `SEARCH_QUERY` for queries); `truncate` and `output_dimensions` (embed-v4.0: 256/512/1024/1536) forwarded.
- Batched at the service limit of 96 inputs; empty inputs are not sent and come back as `None`.
- `ndims()` uses `output_dimensions` when set, otherwise a single cached probe request, so there is no model→dims table to keep in sync with the catalog.
- SDK retries disabled in favour of LanceDB's `max_retries` backoff; 401/403 service errors carry `status_code` so they surface without retrying, with an actionable message (expired session token, IAM policy, region, model).
- `oci>=2.165.1` added to the `embeddings` extra (the floor where `EmbedTextDetails.output_dimensions` exists).

Tests: 22 mocked unit tests (auth paths, batching, input types, config validation, retry behaviour, pickling, table-metadata round trip) and a live test skipped unless `OCI_COMPARTMENT_ID` is set, which passes against us-chicago-1 for embed-v4.0 at 1536 and 512 dims and embed-multilingual-v3.0 at 1024. Verified end to end with a real LanceDB table (`LanceModel` with `SourceField`/`VectorField`, 6 rows, two semantic queries ranked correctly). `ruff format --check`, `ruff check`, pyright and typos are clean.

Docs: a page under `docs/src/embeddings/available_embedding_models/text_embedding_functions/` in the shape of the remaining `voyageai_embedding.md`; the function is exported from `lancedb.embeddings` so the API reference picks it up. The hosted docs' available-models table lives outside this repo and would need a separate update.
2026-11-03 16:08:15 +08:00
Yang Cen c16216e1c4 fix(python): include namespaces in remote table repr (#4799)
A synchronous remote table opened in a namespace is displayed as if it
were in the database root. For example, `db.open_table("test",
namespace_path=["analytics", "features"])` shows
`RemoteTable(dev.test)`.

Include the cached namespace path using the existing `$`-delimited table
identifier: `RemoteTable(dev.analytics$features$test)`. Root output and
the leaf `table.name` stay the same. Using cached identity also keeps
repr available after pickle without reopening the native handle or
making a request.

Addresses the repr portion of ENT-3559. The native formatter already
carries the qualified identifier used by Python async and TypeScript;
this is a Python sync wrapper fix with no new API or naming rules.

Validation:

- Before the fix, three namespaced sync cases failed while root and
async controls passed. All six cases now pass through the real SDK and
mock HTTP endpoint.
- Covers default/explicit root, nested paths, valid
dotted/mixed-case/dashed names, and repr after pickle with the server
stopped. Requests remain limited to the original table open.
- Rebuilt current editable bindings in the repository's uv environment
with Python 3.12 and pinned Rust 1.97.0 through mbx.
- Ruff check and format check pass for the Python package (178 files).
- Required repository-wide Ruff check reports 48 pre-existing
diagnostics in six `docs/web-tests` files; the repository-wide format
check reports nine unchanged documentation files. Their source matches
the base commit. No lint suppression or documentation cleanup is
included.
2026-11-03 16:02:49 +08:00
Xuanwo 2d70bec53a feat: add UTM-tagged docs links to SDK warnings and enterprise-only errors (#4797)
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.

- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.

Linear: OSS-2772
2026-11-03 14:12:15 +08:00
Wyatt Alt f9e25960e3 test(mv): cover vector index retention across a rebuild
The only coverage of the fragment-swap commit's index retention used a
BTree index. Vector is the type a view is usually built for and the one a
rebuild strains hardest, since every fragment the index was trained over
is replaced in the same commit.

The case also pins down what retention does and does not mean: the index
definition survives, but it covers none of the swapped-in rows, so the
view falls back to a flat scan until maintenance rebuilds it. Asserting
`num_unindexed_rows` keeps that from being mistaken for a regression
later.
2026-11-03 14:11:38 +08:00
Xuanwo c0aa5d9d28 docs: show how to run Functions and BlobReader locally (#4803)
Running a Function body in a local test or another engine currently
requires discovering undocumented APIs. Document direct calls to
decorated Functions and classes, including initialization fields,
instance reuse, and cleanup, and explain how they differ from registered
`FunctionVersion` applications.

Add examples of `BlobReader(read_range, size)` backed by an in-memory
byte-range callback, including cursor-independent reads. The local
examples and rendered API documentation were validated on EC2 at
`df6076e1ae605f425a380d282c4f25b845a3165f`; integration with current
main leaves these documented call paths unchanged.

Fixes ENT-3561.
2026-11-03 13:21:30 +08:00
Xuanwo 77af3bd746 fix(python): warn when Functions code packages omit resource files (#4804)
Packages passed through `@udf(code=[...])` silently omit non-Python
resources, so registration can appear successful before a worker fails
to open a data file. Warn during function preparation with the omitted
paths and document the source-only boundary next to `code=`. The
diagnostic explains how to install a distribution that includes the data
and remove the package from `code=` to avoid shadowing that
installation.

Python cache files do not produce warnings. Excluded directories are
reported at their boundary, preserving the existing traversal and
Python-source artifact format. This implements the
warning-and-documentation alternative in ENT-3492; it does not add
resource uploads.

On EC2, the regression fails on baseline `f9389f113` because no warning
is emitted; all 60 packaging tests pass at `e23445b96`, using the
matching native extension built from unchanged Rust sources.

CI-scoped Python lint with Ruff 0.9.9 passes. Whole-site Mintlify
validation reports the same four snippet-import warnings on the baseline
and this head; repository-root Ruff likewise reports the same 48
findings under `docs/web-tests`. These checks are not clean passes, but
their failures do not involve this change.

Fixes ENT-3492
2026-11-03 13:20:29 +08:00
LeoandLeo f2bcb156ad feat: expose approx_mode on vector queries
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.

Co-authored-by: Leo <leo@insight.co>
2026-11-03 12:45:52 +08:00
lancedb-gatefixer[bot] f9389f113e test(python): cover merge insert with schema metadata
Add a Python regression test for merging metadata-free data into a table whose target schema has metadata. The test calls the native `execute_merge_insert` binding directly so it exercises the native partial-schema comparison without the public merge path normalizing source metadata.

The test verifies that merge insert succeeds with the expected rows and preserves the target schema metadata.

Fixes #2058
2026-11-03 10:48:51 +08:00
Srijan Keshri 90b2b264d9 feat(python): accept DataFrames and readers in add_columns
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.

Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.

SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.

Fixes #1959.
2026-11-03 09:45:41 +08:00
Wyatt Alt 8ee7698545 feat(python): allow Blob v2 under a list in Function outputs
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.

Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
2026-11-03 08:14:30 +08:00
lancedb-gatefixer[bot] 7466dc9d1d fix(python): reject embedding source updates without vectors
Fixes #4761

`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.

The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.

This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.

Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
2026-11-03 07:07:35 +08:00
lancedb-gatefixer[bot] c18aa7b033 fix: reject renaming or dropping embedding columns
Renaming or dropping an embedding source or vector column leaves schema metadata pointing at missing fields, causing later `add()` calls to fail.

Reject these operations in local schema evolution before committing any changes. The guard reads Python/Node embedding column metadata and native Rust `lancedb::column_definitions` without loading embedding models. It supports explicit and default Rust vector-column names and respects quoted column paths. Unrelated column changes and nullability updates remain supported.

The restriction is documented in the Python, Rust, and TypeScript APIs. Regression coverage includes both SDK metadata formats, native Rust definitions with explicit and default destinations, atomic rejection of batched changes, and continued embedding generation after reopening.

Fixes #4763

Validation at the exact PR head and GitHub test-merge:

- Hosted macOS Python wheel suite: 3061 passed, 162 skipped, 70 deselected, including all 8 sync and 4 async embedding schema-evolution cases.
- Hosted Windows Rust suite: 1900 library tests passed, 8 embedding integration tests passed, and 107 doctests passed, including the native explicit/default destination and binding guard cases.
- Hosted Linux Python 3.10 rerun: 3084 passed, 159 skipped, 50 deselected; the previously observed remote-fork timeout passed on the replacement job.
- All 10 applicable exact-head workflow runs completed successfully. The existing main-only Windows ARM job was skipped by its workflow condition.
2026-11-03 06:08:33 +08:00
lancedb-gatefixer[bot] ea9194a7f3 fix(python): retry transient job polling errors
Fixes #4773.

Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.

Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
2026-11-03 05:05:30 +08:00
Xuanwo 5ca8d836e7 fix(deps)!: upgrade candle to remove vulnerable zip
## Problem

`RUSTSEC-2026-0429` affects `zip 7.4.0`, which reaches the optional
`sentence-transformers` feature through `candle-core 0.9.2`. The `deny` check on
`main` fails on this advisory.

## Change

Update `candle-core`, `candle-nn`, and `candle-transformers` to `0.11.1`. The
focused lockfile resolution replaces `zip 7.4.0` with `zip 9.2.2`, and no other
zip version remains in the all-features graph. This also pulls candle's required
newer transitive `tokenizers` and `safetensors` releases; the directly declared
`tokenizers 0.19.1` and the embedding implementation remain unchanged.

The optional `sentence-transformers` Rust API exposes this dependency version:
`SentenceTransformersEmbeddingsBuilder::device` accepts
`D: Into<candle_core::Device>`, and the `Device` type is now from candle-core
0.11. The default CPU path constructs that device internally and is unchanged.

`deny.toml` is unchanged. Its obsolete zip ignores now produce
`advisory-not-detected` warnings, which do not fail the check.

## Validation

- The `sentence-transformers` feature builds with tests and examples.
- The existing CPU example loads the default model and retrieves
  `The human body has 206 bones.` for the matching query.
- A local contract check verified 384-dimensional, finite, non-zero Float32
  output and higher similarity for the matching fact than for an unrelated fact.
  It was not committed because it would add an 88 MB model download to
  all-features CI.
- `cargo deny` `0.20.2`, the version used by the Rust workflow, passes
  `advisories`, `bans`, `licenses`, and `sources`.
- The locally installed `cargo-deny 0.18.9` is not green: it treats
  informational unsound advisories differently from the workflow's `0.20.2`.
  The result above uses the exact CI version.
- Targeted clippy, formatting, lockfile review, and the reverse `zip` dependency
  tree pass.
- Hosted CI completed successfully for exact head `bffbcda6490fa1ed51838b3801e01ae2f82faed6`.

CI mitigation: Reclaim completed lancedb CI-profile artifacts before the Blob v2 example after disk sampling showed a minimum of 24,764,416 available bytes. This may rebuild lancedb for the example; the exact OS cause of the SIGBUS remains unproven.

BREAKING CHANGE: The optional `sentence-transformers` Rust API now takes
`candle_core::Device` from candle-core 0.11. Downstream crates that explicitly
construct and pass a candle-core 0.9 `Device` to
`SentenceTransformersEmbeddingsBuilder::device` must use a `Device` from
candle-core 0.11.x aligned with lancedb. This normally means updating a direct
candle-core dependency; if the value comes through a re-export, update the
dependency that supplies it. Callers that use the default CPU path without
passing a `Device` do not need to migrate.
2026-11-03 03:16:42 +08:00
TaylorandBrendan Clement 9e65bd2c27 docs: use Enterprise terminology across website and SDK documentation (#4787)
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.

- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.

Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.

Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).

---------

Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
2026-11-02 10:25:14 -08:00
Yang Cen 2d86532de9 docs(search): clarify auto probing and query tuning (#4796)
The vector-search guides do not clearly explain when to use Auto, search
effort or a fixed probe count. They also describe `minimum_nprobes` as
the exact starting count, although Auto can search more partitions from
the start.

Explain how to choose public query controls for recall and latency, with
short executable examples. Clarify that probe limits apply per segment,
calibration targets are not recall guarantees, and more probes cannot
remove quantization error. Keep necessary client/server version guidance
in a short note.

Validation:
- The existing example test passes against published `lancedb==0.41.0`,
covering all four snippets, result counts, filtering, finite distances
and recall against an exact cosine query. The test body takes 0.20 s.
Generated snippets match their source.
- Vale reports no findings on added prose. Changed-function Python
formatting and `git diff --check` pass. Repository-wide Ruff has 137
findings identical to the previous scan, including seven pre-existing
findings in the example file.
- Mint 4.2.888 strict validation reports the same four
`RsExampleBody`/`RsExampleTail` import warnings as pristine base
`0b20965d`, with no additional warnings.
2026-11-02 23:13:47 +08:00
Minh Vu 0b20965d09 fix(python)!: reject mixed-value enums in schemas
A mixed-value Pydantic Enum previously produced a UTF-8 Arrow schema even when some member values were non-string, causing writes to fail after schema construction. `pydantic_to_schema` now rejects actual member values with more than one type using `TypeError`, including direct, optional, list, and nested fields. Homogeneous string and integer enums and empty enums retain their mappings, and existing plain-Enum writes remain supported. This intentionally changes mixed-value enums from failing at write time to failing at schema construction.

Validation on the exact head: the Python suite passed with 3034 passed, 126 skipped, and 70 deselected, including 12 mixed-value container cases, the empty-enum case, homogeneous schema coverage, and real plain-Enum writes. All applicable exact-head checks and workflows completed successfully.
2026-11-02 20:04:45 +08:00
Minh Vu b5d3561066 fix(python): reject empty multivector reranker input
Reject empty multivector result lists with `ValueError` before accessing the first result, matching the existing RRF and MRR behavior. Non-empty lists, zero-row tables, and both deduplication modes retain their existing behavior.

Regression coverage exercises the shared base reranker through both deduplication settings. The exact-head Python suite passed with 3021 passed, 126 skipped, and 70 deselected.
2026-11-02 19:19:13 +08:00
SHAIK SHIVAJI 1a037c0315 fix(python)!: use null for missing linear combination scores
When `LinearCombinationReranker` uses `return_score="all"` and either the
vector or full-text side has no rows, the unavailable score column is now a
typed Arrow `float32` null column instead of a column of NaNs. Existing scores
from the non-empty side and their descending `_relevance_score` ordering are
preserved.

Regression coverage exercises both missing sides through the hybrid query path.
It checks the missing column's `float32` type, null count and row count, exact
preservation of the original present-side values, and descending relevance
ordering. The existing empty-result, score-column override, and `return_all`
cases remain covered.

The exact head `db33a068263cd69ee578f95a375f0060dcbb4e21` passed all 15
applicable check runs and all 4 applicable workflow runs, with no commit
statuses. The hosted macOS Python run completed with 3019 passed, 126 skipped,
and 70 deselected; the two new cases and nine adjacent regression cases passed.
The retained CI collector reached the all-success snapshot but exited 1 while
writing its final metadata because of a harness `TypeError`; it did not rerun
or invalidate the product or CI results.

BREAKING CHANGE: For `return_score="all"` with an empty vector or full-text
side, the missing side's score column now contains typed nulls instead of NaN.
Consumers that distinguish these values will observe different results.
2026-11-02 18:30:42 +08:00
Christopher Krah 9af3bac6e4 fix(python)!: drop vector and FTS scores from an empty hybrid result
The `Reranker` docstring says the default `return_score="relevance"` "will return only the relevance score" (`rerankers/base.py`), but a hybrid search whose filter matches nothing also returns `_distance` and `_score`. The early return for two empty result sets in `_combine_hybrid_results` (from #3030) skips the reranker's `_keep_relevance_score`, which every built-in reranker applies to a non-empty result.

Before, on `45dbf5e`:

```
$ python repro.py
match:    ['id', 'text', 'vector', '_relevance_score']
no match: ['id', 'text', 'vector', '_distance', '_score', '_relevance_score']
```

After, `no match` also prints `['id', 'text', 'vector', '_relevance_score']`.

<details><summary>repro.py</summary>

```python
import tempfile, lancedb, pyarrow as pa
db = lancedb.connect(tempfile.mkdtemp())
t = db.create_table("t", pa.table({"id": [1, 2], "text": ["red fox", "blue fox"],
    "vector": pa.array([[1.0, 2.0], [3.0, 4.0]], pa.list_(pa.float32(), 2))}))
t.create_fts_index("text")
q = lambda where: t.search(query_type="hybrid").vector([1.0, 2.0]).text("fox").where(where, prefilter=True).limit(5)
print("match:   ", q("id > 0").to_arrow().column_names)
print("no match:", q("id > 100").to_arrow().column_names)
```

</details>

No issue exists for this. The fix calls `reranker._keep_relevance_score(empty)` before returning. Sync and async hybrid queries both go through it. With `return_score="all"` both columns stay, as before. Code reading `_distance` from an empty result now gets a `KeyError`, which it already got from every non-empty one. The new test asserts, for both settings, that a filter matching nothing gives the same columns as one matching every row.

### Verification

```
# Python 3.12.6; lancedb 0.41.0b15 wheel from pypi.fury.io, its _lancedb.abi3.so in the tree; no pylance (none >= 15 resolves)
# the wheel predates #4688, so a 3-line -p plugin sets _lancedb._experimental_functions_enabled = False for tests/conftest.py
$ PYTHONPATH=python python -m pytest -q -p no:cacheprovider python/tests/test_rerankers.py python/tests/test_hybrid_query.py python/tests/test_query.py
45dbf5e:                     5 failed, 197 passed, 57 skipped
this branch:                  5 failed, 199 passed, 57 skipped
query.py reverted, test kept: 6 failed, 198 passed, 57 skipped
  FAILED test_rerankers.py::test_empty_hybrid_result_keeps_reranker_columns[relevance]
    At index 2 diff: '_distance' != '_relevance_score'
$ ruff format --check <both files> && ruff check <both files>      # ruff 0.9.9, as pre-commit pins it
2 files already formatted
All checks passed!
# the 5 failures fail identically on 45dbf5e: 4 test_query_serialization_async_search_options (wheel older than #4659) and
# test_hybrid_query_with_stale_fixed_size_binary_prefilter (needs pylance)
# not run: pyright, the Rust build, the remote client
```

@Xuanwo, a custom reranker that keeps extra columns still gets the generic empty table. I'd keep it that way, since #3030 returns early so that no reranker runs on empty input. If you'd rather call `rerank_hybrid` on the empty tables instead, I'll change it in this PR.
2026-11-02 17:24:20 +08:00
SHAIK SHIVAJI 45edaa350e feat(python): expose code analyzer options on FTS and tokenize
Part of #4364 (Python side; the TypeScript `FtsOptions` change will follow in a separate PR).

Lance already supports the code analyzer, but `FTS` and `lancedb.tokenize()` had no way to set it, so `FTS({'analyzer': 'code'})` and `FTS(split_identifiers=True)` raised `TypeError`.

## Changes

- `FTS` and `lancedb.tokenize()` gain `analyzer` (`"text"` or `"code"`), `split_identifiers`, `split_on_numerics`, `preserve_original` and `index_operators`. They default to `None`, which keeps the tokenizer's own defaults, so existing indexes and calls are unchanged.
- Validation follows pylance: `analyzer="code"` with a `base_tokenizer` other than `"code"` or the default `"simple"` is rejected, `analyzer="text"` with `base_tokenizer="code"` is rejected, and turning on a code flag without the code analyzer is rejected. Passing a flag as `False` is always allowed.
- `analyzer="code"` with the default `base_tokenizer="simple"` selects the code tokenizer. `FTS` can't tell an explicit `"simple"` from the default, so that case is accepted too.
- The deprecated `create_fts_index` is left alone; it points users at `create_index(config=FTS(...))`.
- Rust `FtsIndexBuilder` is Lance's `InvertedIndexParams`, so the Rust SDK already had these options. The remote client serializes the same struct, so the options reach LanceDB Distributed in the `create_index` request body. A new case in the Rust `test_create_index` checks that body.

## Tests

- `python/tests/test_fts.py`: tokenizer output for each flag, validation errors for `tokenize` and `FTS`, `False` flags on a text index, and local sync and async indexes that match `http`, `request` and `parseHTTPRequest` with splitting on and not match subwords with it off.
- `cargo test --features remote remote::table::tests::test_create_index`
2026-11-02 16:49:14 +08:00
Lance Release 558f64c45f Bump version: 0.41.0-beta.17 → 0.42.0-beta.0 2026-11-02 06:41:10 +00:00
LanceDB RobotandJack Ye 8893443a68 chore: update lance dependency to v15.0.0-beta.8 (#4792)
Update the Rust workspace and Java SDK dependencies to [Lance
v15.0.0-beta.8](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.8),
preserve the optional `manifest_size` field in remote version responses,
and verify directory-namespace renames preserve registration and data
both within and across namespaces. Update the Rust and TypeScript API
documentation to reflect directory-namespace rename support.

Rust callers constructing `Version` literals must now provide
`manifest_size` (`None` when unavailable). Java is intentionally pinned
to beta.8 even while Maven artifacts are still publishing;
dependency-resolution CI failures caused solely by that publishing delay
are accepted for this upgrade. Python package pins remain unchanged.

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
2026-11-01 22:35:30 -08:00
Xuanwo 63aa06ef7d feat(python): expose Function GPU classes and placement (#4791)
## Summary

- Add sync/async `DBConnection.list_gpu_classes()` returning typed
`GpuClass` records for `GET /v1/gpu/classes`. Records include the class
name, Kubernetes resource, GPU model and memory, per-worker GPU maximum,
and worker CPU/memory/ephemeral-storage limits.
- Add `udf(gpu_class=...)` for immutable Function-level GPU placement.
It requires `gpu=True`; Function versions expose their registered
`gpu_class`.
- Add `gpu_class` and `gpu_count` to `refresh_column` /
`refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as
top-level `backfill_column` fields and validated before any request.
`gpu_count` is the aggregate GPU budget across Function workers.
- Validate GPU class names with the server lowercase 63-character
DNS-label rule.
- Document the discovery-to-registration flow, configured hardware
versus free capacity, System Usage permission, and per-refresh GPU
placement in the API and web guides.

Older servers without `/v1/gpu/classes` produce a clear
`NotImplementedError` instead of a bare 404.

Part of ENT-3470. The client follows the in-flight Sophon contract on
`xuanwo/ent-3470-fix` for GPU class discovery and backfill placement
options.

## Breaking change

`PythonRuntimeSpec::PythonV2` adds `gpu_class`, and
`RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers
constructing either value with all fields explicitly must supply the new
fields.

## Validation

At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python
extension build, pyright, ruff, and Python Function/runtime, discovery,
refresh placement, and refresh regressions (147 passed, 6 skipped).

At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright,
Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed).

The final commit only updates documentation and Rust doc comments.
2026-11-02 08:50:36 +08:00
Xuanwo 6a255926d7 fix(python): improve function failure diagnostics (#4790)
## Summary

- Make `AsyncJob.wait(timeout)` and `Job.wait(timeout)` raise a
descriptive `TimeoutError` containing the job ID and elapsed timeout.
Registry jobs explain that work continues on the server and include the
`db.open_job(id).wait()` reattach call. SQL query handles instead say to
keep the original handle because query IDs cannot be reopened through
the registry jobs API.
- Render `JobFailureInfo` and `FunctionErrorRecord` with Python values
instead of Rust `Option` wrappers.
- Decode and expose the server's `row_address` on `FunctionErrorRecord`,
and document how to retry recorded rows with `REFRESH COLUMN ... WHERE
_rowaddr IN (...)` before compaction or rewrite.
- Warn when an import inside a packaged Function body resolves to
undeclared local source. The `udf` documentation explains why arbitrary
base-image coverage and distribution-to-import-name mapping cannot be
validated reliably by the SDK.

`row_address` is optional, so records from older servers still decode
with `None`. Rust code that constructs `FunctionErrorRecord` literals
must now supply this new field.

Part of ENT-3448 and ENT-3442. This consumes the server-side diagnostics
from lancedb/sophon#9509 and logical row addresses from
lancedb/sophon#9512.

## Validation

- Rust `function_errors` remote decoding and
legacy-missing-`row_address` tests.
- Rust/PyO3 clippy for `lancedb` and `lancedb-python`.
- Python extension build, pyright, and ruff.
- Targeted Python job, Function packaging, Function error, and job
detail tests: 62 passed, plus timeout-handle tests: 3 passed.
2026-11-02 06:44:48 +08:00
Xuanwo 9786f2b63b fix(python): surface typed remote resource errors (#4789)
## Summary

- Parse structured `error_code` and `resource` fields from remote HTTP
error bodies and carry them through Rust/PyO3.
- Add `SecretNotFoundError`, `FunctionNotFoundError`,
`NamespaceNotFoundError`, and `AlreadyExistsError` as `HttpError`
subclasses.
- Expose `error_code`, `resource_kind`, `resource_name`, and
`namespace_path` on HTTP exceptions while preserving `status_code`,
`request_id`, and existing `except HttpError` handlers.
- Do not retry structured domain errors that would otherwise become
`RetryError` (notably `already_exists` on 409).

Older servers that do not provide the structured fields continue to
raise a plain `HttpError`.

## Compatibility

This consumes the stable resource error contract added in
lancedb/sophon#9516 (merge commit `35858561b3`). The existing
`TableNotFoundError` behavior is unchanged.

Part of ENT-3373.
2026-11-02 05:11:38 +08:00
Jack Ye eeb244fa57 fix: preserve all function revisions through SQL history (#4786)
`list_functions()` promises every published immutable revision. Request
`SHOW FUNCTIONS HISTORY` so the SQL server's latest-only default does
not hide older revisions, and preserve the server's numeric revision
ordering when constructing page cursors (`1`, `2`, `10`).

```python
[(fn.name, fn.version) for fn in db.list_functions(page_limit=2)]
# [("embed", "1"), ("embed", "2"), ("embed", "10")]
```

```sql
SHOW FUNCTIONS; -- latest visible revision per name
SHOW FUNCTIONS HISTORY WITH (limit = 3, include_definition = true);
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '2', limit = 3);
```

Each listing page uses the existing SQL query and polling transport; the
continuation token stores the last returned name/revision and remains
bound to its database and namespace. Function mutations keep their
query-backed job handles: `Job.id` is the query's ID and `job_type` is
`query`, so waiting observes that same SQL execution. No registry job or
second query ID is added.

Update the SQL and Python documentation for exact-revision drops:
sibling revisions and grants survive, and only the last visible
revision's removal releases the name and schedules object cleanup.
Extend the existing Flight SQL regression to paginate revisions `1`,
`2`, and `10` and verify the explicit history statement. Formatting and
static Python checks pass; local builds and tests were intentionally not
run.
2026-11-02 01:15:39 +08:00
Jack Ye cbc24fa84f fix: decode native SQL view metadata (#4785)
View APIs now read the native `name`, `query`, and `schema` columns
returned by `CREATE VIEW` and `DESCRIBE VIEW`. The schema JSON carries
the defining database and namespace; the requested namespace identifies
where the view lives. The unreleased JSON-envelope response is removed
without a compatibility fallback.

```python
view = db.create_view("adults", "SELECT name FROM people WHERE age >= 18")
view = db.describe_view("adults")
```

```sql
CREATE VIEW "adults" AS SELECT name FROM people WHERE age >= 18;
DESCRIBE VIEW "adults";
```

Both Python calls execute SQL through the query-backed job adapter and
wait on the resulting query. A job handle uses that query's `id` and
type `query`; reading metadata consumes its completed result rather than
resubmitting the DDL. Creation returns the definition from the creating
execution, with no extra describe request.
2026-11-02 01:15:36 +08:00
Jack Ye 42e8e7dfca fix: align Secret APIs with native SQL syntax (#4784)
Secret APIs now submit the native SQL statements with the credential
bound as an Arrow string parameter:

```sql
CREATE SECRET "api-key" TYPE = GENERIC_STRING SECRET_STRING = $1;
ALTER SECRET "api-key" SET SECRET_STRING = $1;
DESCRIBE SECRET "api-key";
```

`create_secret` and `alter_secret` execute SQL through the existing
query-backed job adapter and wait for that query to complete. The job's
`id` is the query ID and its type is `query`; credentials stay out of
SQL text. `describe_secret` reads the native `name`,
`created_at_millis`, and `updated_at_millis` columns. This replaces the
unreleased `AS $1` syntax and JSON-envelope response without a
compatibility fallback.

Python methods and namespace handling remain the same:

```python
db.create_secret("api-key", "credential", namespace_path=["team"])
db.alter_secret("api-key", "rotated", namespace_path=["team"])
info = db.describe_secret("api-key", namespace_path=["team"])
```
2026-11-02 01:15:32 +08:00
BHAWESHBHASKAR a9146d536a feat(python): expose the NGram scalar index
## What kind of change does this PR introduce?

Feature. Exposes the NGram scalar index in the Python SDK.

## What is the current behavior?

The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.

## What is the new behavior?

Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:

```python
from lancedb.index import NGram

await table.create_index("text", config=NGram())
```

The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.

Changes:

- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.

This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
2026-11-01 11:54:27 +08:00
lancedb-gatefixer[bot] e5d4e9bf6e fix(python): normalize __version__ to PEP 440 (#4752)
Fixes #4749

Python's `__version__` forwards Cargo's SemVer spelling, so prereleases
report values such as `0.41.0-beta.17` while the installed package
metadata reports `0.41.0b17`.

Normalize the native version with the existing
`packaging.version.Version` dependency before exposing it in Python.
Keep native version sourcing so imports still work with `lancedb-compat`
and do not require distribution metadata. Regression tests cover
metadata equality and stable, alpha, beta, release-candidate, and
development versions with metadata lookups disabled.

Validation:

- Rebuilt the native extension through the uv-managed development
environment.
- Confirmed the metadata regression fails on the original code with the
reported string mismatch.
- Import tests, package metadata tests, the remote client user-agent
test, and CI Python dependency tests: 14 passed, 2 skipped (Windows-only
wheel check and uninstalled `lancedb-compat`).
- macOS ARM64/Python 3.13 dependency resolution and `uv lock --check`
pass with the platform-specific test dependency pins from current
`main`.
- Python-tree Ruff lint and formatting checks, focused Pyright check,
Rust formatting, and PR diff whitespace checks pass.
- Repository-wide Ruff lint reports existing findings in unchanged
`docs/web-tests` files.

<!-- lance-gatekeeper-fix:v1 agent=3e23555743cc116ea53ca8a7e3c4ebb1
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-31 20:41:22 -07:00
lancedb-gatefixer[bot] 71b23ed9fb fix(python): preserve request metadata for missing remote tables
Fixes #4750

Remote missing-table errors retained HTTP metadata in Rust but the Python binding converted them to a plain `ValueError`, discarding the request ID and status. This raises a `TableNotFoundError` that preserves the original message, request ID, status code, and chained HTTP cause while remaining compatible with existing `ValueError` and `lance_namespace.errors.TableNotFoundError` handlers. Local and non-HTTP errors retain their previous behavior.

Validation covers sync and async opens across cached/uncached and root/nested paths, exception pickling, local compatibility, the remote and error test suites, Clippy, formatting, Ruff, and all applicable PR checks including CodeQL.
2026-11-01 11:10:44 +08:00
Hien Hoang fdf886d42a fix(python): let to_pydantic read tables written by an aliased LanceModel
`LanceModel` instances with validation aliases write field-name keys to Arrow, while `to_pydantic` previously reconstructed models by alias. This normalizes field-name rows to the applicable single-component validation alias for both sync and async reads, while preserving existing alias keys, alias precedence, name-only validation, and collision behavior.

Adds sync and async regression coverage for string aliases, `AliasChoices`, single-component `AliasPath`, `populate_by_name`, and explicit `validate_by_name` precedence.
2026-11-01 10:27:13 +08:00
alexcastgonz e9b2f7e6e1 fix(python): use dim as the embedding size for OpenAI-compatible models
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model
names outside the three built-in OpenAI names, so schemas can be built for
compatible models.

Preserve existing model defaults and request behavior. Add regression coverage
for the resulting Arrow schema dimensions and for all known OpenAI model
dimensions.
2026-11-01 08:49:49 +08:00
Yui Numazawa faa8953d81 fix(python): allow attribute access on namespace request dicts
Allows Python namespace implementations to read request dictionary entries as attributes while preserving mapping and `model_dump()` behavior. Missing dunder attributes continue to raise `AttributeError`.

Adds a real namespace connection regression covering list/open operations and omitted optional fields.
2026-11-01 08:01:05 +08:00
SHAIK SHIVAJI 6da2c68a80 fix: add missing space in invalid table name error message
Fixes #4701

The invalid-table-name diagnostic now includes the missing space before
"periods are allowed".

The regression test covers the corrected message returned for `my/table`.
2026-11-01 06:51:57 +08:00
lancedb-gatefixer[bot] f28391f8ba test: cover HNSW index optimization
Add a regression test for the public optimize-all path on a table with an IVF_HNSW_SQ index. The test creates 512 rows, runs the full optimize action, and verifies that the single retained index still covers all 512 rows with zero unindexed rows.

Test-only change; no production behavior changes.
2026-11-01 05:39:28 +08:00
Prince Jain 8bc11bfcfc test(nodejs): cover camelCase column names across table surfaces
References #3211.

The core `col()` camelCase integration coverage requested by #3211 is already on `main` from #4493. This PR adds the adjacent raw-SQL table-surface regressions that were previously uncovered:

- unquoted `where("camelCase = 1")`
- `update` and `delete` with unquoted camelCase predicates
- `mergeInsert` with a camelCase join key, including matched-update and not-matched-insert input
- `addColumns` `valueSql` referencing a camelCase column
- `createIndex` on a camelCase column

These are characterization tests for the existing local-backend behavior. They do not add `col()` coverage or require a live remote service.
2026-11-01 04:24:28 +08:00
lancedb-gatefixer[bot] 156c12b017 test(nodejs): cover local object-form table creation across connections
## Summary

- add a regression test for object-form `createTable` on two independent local Node.js connections
- verify both connections use the same table name while preserving each database's rows

## Scope

This test covers the public local `connect(tmpDir)` path. It does not exercise LanceDB Cloud or remote create requests, and it does not establish a remote root cause or a complete fix for #952.

References #952
2026-11-01 02:48:39 +08:00
sarankumar 0ecb4c8033 docs: fix dead LangChain and LlamaIndex links in README
Fix the dead README ecosystem links by pointing readers to the current official LanceDB documentation for LangChain and LlamaIndex.

Original contribution by @sarankumar1325.
2026-11-01 02:11:42 +08:00
lancedb-gatefixer[bot] 9804f26bc3 fix: support batch size options for column refreshes (#4777)
Python column refresh methods rejected `batch_size`, and the Rust remote
backend never included it in the backfill request. Callers could not
reduce the rows passed to a slow Function to stay within the executor's
per-call deadline.

Add optional, keyword-only `batch_size` arguments throughout the
synchronous and asynchronous Python APIs and bindings. Add compatible
Rust `RefreshColumnOptions` APIs while preserving existing Rust method
signatures. Reject non-positive sizes before submitting a job, send
positive sizes in `backfill_column`, and omit the field when unset while
retaining branch scoping.

Batch size controls remote Function calls. Embedded expression refreshes
accept and validate the option without changing execution, and blocking
remote refreshes retain their existing unsupported behavior; remote
callers use `refresh_column_async` and wait on the returned job.

Validation:

- Reproduced the original Python `TypeError` from the committed method.
- Five focused Rust tests passed, covering request contents, omission,
branch scoping, and rejection before any request.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo clippy --quiet --features remote --tests --examples -- -D
warnings` passed.
- Nineteen targeted Python tests passed against the rebuilt extension,
including synchronous and asynchronous remote requests, invalid sizes,
and default-build behavior. Eight embedded refresh cases require the
experimental-functions build and were skipped.
- Rust formatting and Python formatting/lint checks passed, including
CI's pinned Ruff version. Repository-wide Ruff reports 48 existing
findings in untouched documentation files; none are in changed files.

Fixes #4776

<!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-31 10:00:10 -07:00
lancedb-gatefixer[bot] 569952d150 fix: allow hidden function assignment columns (#4779)
A second flattened Function declaration fails when the remote describe
schema omits an existing binding's internal assignment field. The client
incorrectly requires that server-owned field and compares it as a
physical user output.

Exclude assignment fields from binding/schema validation and the
output-schema comparison. Filter declared assignment fields from remote
schemas, including seeded caches and snapshot handles, while retaining
schema metadata.

The regression tests declare two all-nullable flattened bindings
consecutively with no assignment declaration, a hidden assignment field,
or a returned assignment field. They also cover cached and snapshot
schemas and server-owned assignment types and metadata. All four
regression cases failed before the fix.

Validation:

- `cargo fmt --all`
- `cargo test --quiet --profile ci --features remote -p lancedb --lib --
table::computed_columns::tests remote::table::tests` — 379 passed, 30
ignored.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`

Fixes #4778

<!-- lance-gatekeeper-fix:v1 agent=c55eff66e22d736a6b954a9eeec88898
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-31 09:07:09 -07:00
Jack Ye 784d649380 feat: support Hugging Face storage integrations with optional token Secrets (#4775)
Adds `HUGGINGFACE` and an optional token Secret reference to the
existing storage integration methods. The same SQL transport handles
create, describe, list and drop. Only a Secret reference is stored in
integration metadata; mixed-provider credential options are rejected.

```python
from lancedb.storage_integration import SecretReference

info = db.create_storage_integration(
    "private_hub", "hf://datasets/acme/private/raw/",
    storage_provider="HUGGINGFACE",
    huggingface_token_secret=SecretReference(
        "hf_token", database="shared", namespace_path=("credentials",),
    ),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```

Omit the token Secret for anonymous access at Hugging Face's default
anonymous
rate limits. A tokenless integration does not use a cluster's Hugging
Face token:

```python
info = db.create_storage_integration(
    "public_hub", "hf://datasets/acme/public/raw/",
    storage_provider="HUGGINGFACE",
)
```

```sql
CREATE STORAGE INTEGRATION public_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```


```sql
CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
    HUGGINGFACE_TOKEN_SECRET = shared.credentials.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```

```mermaid
flowchart LR
    Python[Sync or async Python] --> SQL[Existing SQL transport]
    SQL --> Metadata[Integration and token Secret reference]
```

Adds Rust request construction, SQL rendering, typed metadata decoding,
sync/async Flight SQL regressions and public documentation. Local
builds/tests are excluded by request; CI provides execution validation.
2026-10-31 01:56:42 -07:00
Jack Ye 1fb2dd1551 feat: accept GCP credentials Secret in storage integration clients (#4771)
Adds GCP inputs to the existing database-owned storage integration
methods. GCP takes one Secret reference; the Secret stores the complete
application credentials JSON as an opaque string. The same methods and
SQL transport handle create, describe, list and drop.

```python
info = db.create_storage_integration(
    "gcs", "gs://customer-data/raw/", storage_provider="GCP",
    gcp_credentials_json_secret=SecretReference(
        "gcp_json", database="shared", namespace_path=("credentials",),
    ),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("gcs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("gcs")
```

```sql
CREATE STORAGE INTEGRATION gcs
    STORAGE_PROVIDER = 'GCP'
    STORAGE_PREFIX = 'gs://customer-data/raw/'
    GCP_CREDENTIALS_JSON_SECRET = shared.credentials.gcp_json;
DESCRIBE STORAGE INTEGRATION gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION gcs;
```

```mermaid
flowchart LR
    Python[Sync or async Python] --> SQL[Existing Flight SQL transport]
    SQL --> Config[Configuration and one Secret reference]
```

Adds Rust request construction, SQL rendering and typed metadata
decoding, plus sync/async Flight SQL regressions across the three Arrow
string encodings. Rejects missing or mixed-cloud credential options.
Local builds/tests are excluded by request; validation runs in CI.
2026-10-31 01:11:25 -07:00
Jack Ye b15fa9d6e4 feat: accept Azure inputs in storage integration clients (#4759)
Register Azure Blob integrations through Flight SQL using an explicit
storage account and three Secret references for tenant ID, client ID and
client secret. Extends the database-owned integration APIs with
provider-specific inputs while keeping the same create, describe, list
and drop operations.

```python
info = db.create_storage_integration(
    "customer_blobs", "az://customer-data/raw/",
    storage_provider="AZURE", azure_storage_account="customeraccount",
    azure_tenant_id_secret="tenant", azure_client_id_secret="client",
    azure_client_secret_secret="credential",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("customer_blobs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("customer_blobs")
```

Sync and async connections accept the same options. Use
`SecretReference` for cross-database or namespace qualification. Values
remain in Secrets; returned metadata includes only references and setup
guidance.

```sql
CREATE STORAGE INTEGRATION customer_blobs
    STORAGE_PROVIDER = 'AZURE'
    STORAGE_PREFIX = 'az://customer-data/raw/'
    STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
    STORAGE_AZURE_TENANT_ID_SECRET = public.tenant
    STORAGE_AZURE_CLIENT_ID_SECRET = public.client
    STORAGE_AZURE_CLIENT_SECRET_SECRET = public.credential;
DESCRIBE STORAGE INTEGRATION customer_blobs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_blobs;
```

```mermaid
flowchart LR
    Python[Sync or async Python] --> SQL[Flight SQL]
    SQL --> Metadata[Configuration and Secret references]
```

Azure supports service principal client credentials only. Adds SQL
rendering, provider validation and Python Flight SQL regressions across
Utf8, LargeUtf8 and Utf8View results. Local builds and tests are
excluded as requested; validation runs in CI.
2026-10-31 14:39:46 +08:00
importcpp e56150568c feat(python): fill failed streaming transform rows
Add `on_transform_error="fill"` to `StreamingDataset`, replacing failed
transform rows at their original positions with a deep-copied fill value or
the result of `repair(exc, row_batch)`.

Fill mode preserves split length, elastic step layout, worker accounting, and
checkpoint progress. Repair failures remain fatal, and `rows_filled` is shared
across workers.

Validated with focused fill and adjacent dataloader tests (10 and 6 passed)
and the complete applicable CI run for
6b51b33e56: four workflows and fifteen checks
all succeeded.

Closes #3916
2026-10-31 12:32:16 +08:00
Bruno Ramirez 5f7ec4e0a2 feat(sdk): expose table overrides for remote tables
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.

Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.

The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.

Validated with focused Rust and Python override tests and the complete applicable
CI run for 43bccbadc5.
2026-10-31 10:44:57 +08:00
Joshua Snider 1022e91434 feat: resolve retry config from environment variables
Resolve retry settings from the six documented `LANCE_CLIENT_RETRY_*`
environment variables through the shared retry configuration. Explicit
configuration takes precedence, invalid values return `InvalidInput`, and
status lists tolerate whitespace and trailing commas while a blank list
disables status-based retries.

Saturate calculated backoff at one hour instead of panicking when the
exponential exceeds the representable `Duration` range, including retry
counts configured from the environment.

Includes coverage for defaults, all six environment variables, explicit
configuration precedence, status parsing, invalid environment and direct
configuration values, real process-environment wiring, and saturating
backoff.

Closes #2350

Originally authored by Joshua Snider (@jsnider3).
2026-10-31 09:38:32 +08:00