Commit Graph
1505 Commits
Author SHA1 Message Date
ryan-haynesandRyan Haynes d316cbbed4 fix!: support opt-in PKCE for device-code login (#4747)
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.

Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.

OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.

The supported-flows guide documents the Rust-only device PKCE option and
its default.

## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.

## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.

---------

Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
2026-10-30 17:26:05 -06:00
somil jain bd6e907c73 feat(remote): support dropping all tables in remote databases
Fixes #4403

Implement remote drop_all_tables() support by listing tables and dropping them individually, including tables within namespaces. Handle paginated table listings and propagate errors appropriately. Add tests covering root and namespace tables, pagination, empty databases, and listing failures to ensure reliable remote database cleanup.
2026-10-31 07:17:07 +08:00
Wyatt Alt 9b543c3848 feat(python): build a Function on a user-specified image (#4768)
Adds `image` to `@udf`, setting `runtime.environment.image` so a
Function can be built on a digest-pinned registry image instead of the
default base, for dependencies pip, conda and apt cannot express. The
client refuses references that are not pinned by digest; the server
validates the rest. The field is omitted when unset, so existing
Function identities are unchanged.
2026-10-30 11:21:46 -07:00
Jack Ye ac7930fa77 feat: use SQL for remote view lifecycle
Stacked on #4675.

Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.

`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
2026-10-31 02:09:54 +08:00
Jack YeandXuanwo 45e00d4de7 feat: use parameterized SQL for remote secrets
Stacked on #4674.

Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens.

The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values.


---------

Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-31 00:15:53 +08:00
Jack Ye 564474f51a feat: use SQL for remote function lifecycle
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates.

Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`.
2026-10-30 22:35:35 +08:00
Jack Ye 10c9973479 docs: explain vectorized blob UDFs
Document vectorized `pyarrow.Array` UDFs for Blob v2 inputs and outputs.

Clarify annotation-selected array execution and the adapter-v1 client/executor
boundary. Add public regression coverage for registering, hydrating, executing,
publishing, and reading back Blob arrays, including large, empty, and null values.
2026-10-30 20:50:29 +08:00
Xuanwo ca5281a2d3 test(python): use a threaded mock server for remote tests
Use `ThreadingHTTPServer` for the shared remote-test mock so concurrent `table.add` requests are not handled serially by the test server.

Validation:

- Reproduced the connection-reset/response-decoding failure against base `13749a3f138fde2647c8bdb8a1c05545026aee57`: 27 targeted runs passed, then the 28th matched the original CI failure.
- The patched target passed 100/100 runs; `test_remote_db.py` passed 261 tests with 2 skipped.
- Ruff 0.9.9 format and lint checks passed.
- Current-head GitHub CI: 19/19 check runs and 5/5 workflow runs succeeded.
2026-10-30 20:11:44 +08:00
Jack Ye 6d126a3601 feat: add database storage integration APIs (#4618)
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.

### Python experience

```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
    "raw", "s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
    aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```

For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.

### SQL experience

```sql
CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
    STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;

CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
    STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```

CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.

Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
2026-10-30 00:26:42 -07:00
Xuanwo a39a31807f fix(python): pin an available pylance wheel on macOS
## Problem

The Python test extra pins `pylance==15.0.0b7` for every platform. On macOS, pip cannot resolve that pin: the configured `lance-format` index publishes b7 wheels for Linux and Windows but not macOS, while public PyPI returns HTTP 404 for the version and the `lancedb` index contains no 15.x release. The macOS Python dependency installation consequently cannot resolve the pin.

## Change

- Keep the exact `pylance==15.0.0b7` test pin on non-macOS platforms.
- Use `pylance==15.0.0b4` on macOS. This is the newest release in this range with a macOS wheel (`cp310-abi3-macosx_11_0_arm64`).
- Refresh `python/uv.lock` for the platform-specific resolutions.

The production Rust/Lance inputs are unchanged: `Cargo.lock` is byte-identical to `main` and remains on Lance `15.0.0-beta.7`. No tests, assertions, coverage, or CI checks are removed or weakened.

## Validation

- Confirmed the wheel identity from the configured index and matched its published SHA-256 to `uv.lock`; wheel metadata reports `Name: pylance`, `Version: 15.0.0b4`.
- Verified the b4 APIs used by the Python tests (`blob_field`, `blob_array`, `BlobType`, `Blob`, `dataset`, `write_dataset`, `FFILanceTableProvider`, and `DirectoryNamespace`). The Python test suite does not reference the b7-only `mark_sstables_as_compacted`, `mem_wal_dir_id`, `mem_wal_index_details`, `clustering_id`, `LanceFragment`, `IndexSegment`, `UpdateConfig`, or `max_source_fragments` changes.
- Verified scalar read/write interoperability in both directions between b4 and the beta.7 LanceDB native binding, plus beta.7 reads of a b4-written blob dataset with byte and null fidelity.
- `uv lock --check` with both configured Fury indexes and `unsafe-best-match`.
- CI dependency tests.
- Ruff 0.9.9 format and lint checks.
- Targeted Python tests: 911 passed, 22 skipped.
- Full Python suite (`-m "not slow and not s3_test"`): 2858 passed, 157 skipped, 70 deselected.
2026-10-30 14:00:16 +08:00
Brendan ClementandClaude Opus 5.5 ca343a5c06 docs: fix table, accuracy and structure problems found in the pre-launch review (#4745)
## Summary

Fixes the problems a page-by-page review of the docs found before
launch, after checking each one against the current source and, where
noted, the code. The matching fixes for the dataset and Geneva pages,
which live in `lancedb/docs`, are in lancedb/docs#402.

**Tables**
- `integrations/embedding/cohere`: each input-type description was split
across two table rows and wrapped in literal quotes
(`"`search_document`"`). Now one row per type.
- `integrations/embedding/gemini`: removes the same stray quotes from
the task-type table.
- `integrations/ai/langchain`: the MMR search table's `kwargs` row was
missing a cell, so its description sat under Type.
- Six reranker pages (answerdotai, cohere, colbert, jina, openai,
voyageai): closes the backtick before `_relevance_score`.
- `indexing/vector-index`: finishes a cell cut off at "with scalar".
- `sql/dql/dedupe`: labels the empty header cell in the resolution
table.

**Corrections**
- `curation/sampling`: sampling works in LanceDB Embedded through the
Python SDK, so the page is available in both, with the SQL sections
marked Enterprise.
- `feature-engineering/index`, `add-computed-column`: automatic refresh
is on by default for each computed column and takes effect once a
deployment enables its scheduler. `refresh-column` already said so;
these two implied it was off.
- `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates
`array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also
merges scalar-index's two "nested fields" sections into one.
- `integrations/embedding/ibm`: lists the four current models from
`CURRENT_MODELS` (it listed a deprecated one) and explains that the
default stays a legacy ID so existing tables reload with their model.
- `integrations/reranking/openai`: the class is `OpenaiReranker`, not
`OpenAI()`.
- `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit`
throughout, each step runs one command, each step passes the file it
writes to the next, and `save-as` uses `-f`/`-o`.
- `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the
pipeline reads, and the run command runs the file the page creates.
- `search/vector-search`: the multivector walkthrough connected to
Enterprise in step 1 and a local path in step 2. It now runs locally
throughout, with a note on connecting to Enterprise, and is renamed so
it doesn't read as a duplicate of the short section above.

**Structure and wording**
- `integrations/embedding/nvidia`: adds headings, and drops test-suite
details aimed at contributors.
- `integrations/ai/genkit`: makes the retrieval test a real heading and
describes the query step.
- `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a
`<Note>`.
- `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading,
not bold text.
- `curation/windowing`: sidebar title matches the curation overview
("Rank and Deduplicate").
- `deployment/storage`, `basics/agentic-skills`: fixes a garbled
sentence and a stale reference to the quickstart.


**Checked:** no new Vale alerts on the changed pages. Assembled together
with lancedb/docs#402, the site passes `mint validate` and `mint
broken-links --check-anchors --check-redirects`. Every changed page was
reviewed in a local preview.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 21:26:37 -04:00
3306e5fbf2 chore: update lance dependency to v15.0.0-beta.7 (#4746)
Updates Rust workspace and Java Lance dependencies to
[v15.0.0-beta.7](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.7),
and pins Python tests to the published `pylance==15.0.0b7` release with
refreshed Cargo and uv lockfiles. No compatibility fixes were required.

Validation: workspace Clippy with all features and warnings denied,
Cargo formatting, uv lock consistency, and all 18 CI-script tests pass;
repository-wide Ruff diagnostics are pre-existing.

---------

Co-authored-by: Lu Qiu <lu@lancedb.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-29 16:49:26 -07:00
d5013459eb chore: update lance dependency to v15.0.0-beta.6 (#4742)
Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.

Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.

Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.

---------

Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:14:09 -05:00
alexcastgonz 03c7d1f199 fix(python): add the missing f prefix to the permutation split error (#4709) 2026-10-29 13:03:32 -07:00
lancedb-gatefixer[bot] a4f52cd205 fix(python): add a Blob V2 array construction helper (#4734)
Fixes #4733

PyArrow expects Blob V2 storage structs when constructing a record batch
and cannot infer the extension array from raw bytes. LanceDB already
converts binary values during ingestion, but batch writers had no
supported array constructor.

Add `lancedb.schema.blob_array(payloads)` for iterables of inline bytes
and nulls, using the same extension type resolver as `blob()`. The
helper preserves empty blobs and null blobs and works with both
pylance's type and LanceDB's fallback. Document the descriptor fields,
exclusive inline/external representations, batch construction, and
existing binary conversion in synchronous and asynchronous
`create_table` / `add` APIs.

Validation:
- Reproduced the reported `ArrowTypeError` on the original code; both
new regression cases fail against the original schema module.
- 256 focused blob tests and schema doctests passed; one optional PyAV
test skipped.
- Both new regression cases also passed with PyArrow 25, with and
without pylance imports.
- Python package Ruff checks and formatting passed.
- MkDocs build passed, with an existing unrelated `SecretReference`
cross-reference warning. Repository-wide Ruff reports 48 existing errors
in `docs/web-tests`.

<!-- lance-gatekeeper-fix:v1 agent=20d1a296d76ff5eb914c0848428a747a
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-29 11:28:54 -07:00
Taylor a065b4b9c6 feat(python): add NVIDIA Nemotron embedding provider
Adds a Python `nvidia` embedding provider for Nemotron 3.5 Embed 1B and 8B. It works with NVIDIA's hosted API and self-hosted NIM `/v1` endpoints, and uses the existing `openai` client. The OpenAI provider can't send NVIDIA's passage/query input types or pick Nemotron's dimensions.

- Documents are embedded as `passage`, searches as `query`.
- Default dimensions are 2048 (1B) and 4096 (8B), and `dim` is checked against each model's supported range. Self-hosted model IDs need an explicit `dim`.
- Response count, order, dimensions and values are checked. Null and empty text get a null vector without a request.
- Each request is retried on its own with LanceDB's retry policy. Invalid input or responses fail immediately.
- `NVIDIA_API_KEY` is only sent to the hosted API. Custom endpoints need an explicit `api_key` (a `$var:` reference), and endpoints restored from table metadata must be allowed with `allow_endpoint`.
- Request dimensions are sent through `extra_body`, preserving the NVIDIA wire contract while supporting the declared minimum `openai==1.6.1`.

Python and text only. Docs are in `integrations/embedding/nvidia`.

Originally authored by Taylor (@tsmithv11).

### Testing

- `test_nvidia_embeddings.py`: 45 passed, 2 deselected with both the locked `openai==2.44.0` and minimum `openai==1.6.1`. These use the real OpenAI client with mock HTTP transport and real local tables.
- Embedding, registry, field-metadata and trust suites: 163 passed, 4 skipped, 10 deselected.
- Historical live experiments: hosted 8B passed a five-request smoke test; hosted 1B returned HTTP 500. No live request was made in this stage, so current hosted 1B availability was not verified. Enterprise coverage uses the repository mock HTTP server, not a live deployment.
2026-10-29 18:10:50 +08:00
BHAWESHBHASKAR 891774343e feat(python): add CombinedFieldsQuery for BM25F full text search
Add a typed Python `CombinedFieldsQuery` for BM25F search across multiple columns.

The query mirrors Lance's combined-fields semantics with ordered per-column boosts and AND/OR term matching. Convert it in both directions at the PyO3 boundary, including nested Boolean queries, and document the Python typed convenience entry.

Closes #4652.
2026-10-29 17:29:22 +08:00
Jay Pokale 280ca41061 feat(python): support time and timedelta values in table.update
Support `time` and `timedelta` values in `table.update` by generating explicit Arrow casts.

- Preserve Arrow duration/time units and pandas nanosecond precision.
- Reject timezone-aware `time` values before data can be changed.
- Cover time64 us/ns, duration s/us/ns, negative timedeltas, filters, and nanosecond round trips.
2026-10-29 16:08:20 +08:00
Wyatt Alt 41c1a2a8b1 feat(python): declare Debian system packages on a Function
Adds `apt` to `@udf` and `system_packages` to the Rust `PythonEnvironmentSpec`. The server adds the packages to the Function's image before installing its Python environment, for shared libraries that pip and conda cannot provide, such as FFmpeg's for torchcodec. The field is omitted when empty, so existing Function identities are unchanged.
2026-10-29 15:28:36 +08:00
Ishita Singh 079711d4ad feat(python): list registered embedding functions
Closes #1605.

Adds `EmbeddingFunctionRegistry.list_functions()` to expose registered lookup names in registration order without accessing `_functions`. The method returns an independent snapshot, and every returned name can be passed to `get()`.

Tests cover empty registries, aliases and class names, registration order, snapshot isolation, and reset.
2026-10-29 13:56:27 +08:00
Lei Xu 03bd909dab test(python): resolve aliased projections in the torch remote mock server
Map dynamic projections by output name to their source columns in the remote
torch test mock so permutation take queries preserve the aliased ordering
column. Run the regression in the embeddings-installed Python test job and
require torch to import so a missing dependency cannot skip coverage.
2026-10-29 12:19:44 +08:00
max 9ffd3d33e1 test(python): remove fixed per-test delays from mock HTTP servers
The Python test jobs spend much of their pytest time waiting on the in-process mock HTTP servers.

1. Each server ran `serve_forever()` with the default 0.5 s poll interval, so every `server.shutdown()` blocked for up to 0.5 s. Around 300 tests use such a server (test_remote_db, test_authz/test_catalog, test_listing, the permutation helpers), about 150 s per test job on every platform. A shared `start_server_thread()` helper in `tests/utils.py` serves with a 0.01 s poll interval, so shutdown returns in about 10 ms.
2. The servers bind IPv4 only, but clients used `http://localhost:<port>`. The client tries ::1 first and falls back after the happy-eyeballs delay, about 0.3 s per connection on Windows. The mock handlers close the connection after each response, so one listing test took 9.5 s on Windows CI versus 0.6 s on Linux. Clients now use 127.0.0.1, as test_catalog, test_listing and test_namespace already do.

Measured on Windows with the 0.41.0b6 wheel and `-m "not slow and not s3_test"`: full suite 546.7 s to 212.3 s; test_remote_db.py, test_listing.py and test_catalog.py together 284 s to 14 s on current main. Collected test ids are identical (1898) and pass/fail/skip results match. No assertion, test or marker changed.
2026-10-29 11:31:23 +08:00
lancedb-gatefixer[bot]andXuanwo 913ee160b0 test(python): cover vector index prewarming by name (#3859)
## Summary

- cover vector-index prewarming through the Python API using the logical
index name returned by list_indices
- prevent regression to interpreting a vector index name as a schema
column

## Root cause

The Lance version used by the reported release opened a vector index by
treating the prewarm argument as a column before resolving the index
metadata. Current LanceDB dependencies contain the upstream
metadata-based resolution, but LanceDB only exercised prewarming for FTS
indexes, leaving the vector behavior unguarded.

## Validation

- uv run --extra tests pytest
python/tests/test_index.py::test_create_vector_index -q
- python/.venv/bin/ruff format --check python/python/tests/test_index.py
- python/.venv/bin/ruff check .

Fixes #2897

<!-- lance-gatekeeper-fix:v1 agent=c2f8273221061d8e2b6f7393066e68c6
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-29 10:45:34 +08:00
Bilal Atique 65954bf9cc docs(python): document connection_factory on StreamingDataset
Closes #3693.

`columns` is documented on `StreamingDataset` now, but `connection_factory` still wasn't. This adds it to the parameter list, in signature order.

The text follows `__getstate__` / `__setstate__`: with a factory, pickling the dataset replaces the table's reopen state with its name, and each DataLoader worker calls the factory to reopen the table. The factory is pickled along with the rest of the dataset, so it has to be picklable, and credentials stay out of the pickle only when the factory loads them in the worker instead of capturing them (the same guidance as `Permutation.with_connection_factory`). Without a factory, the table's own reopen state is pickled.

This picks up #3697, which documented the same parameter and was closed by its author without a merge.
2026-10-29 09:52:15 +08:00
Wyatt Alt 3506b43c03 feat(python): accept input_schema or output_schema alone in @udf (#4731)
Binding a nullable column needed an explicit input schema, and that
forced spelling out the output schema too. Either schema can now be
given alone; the other side is inferred from annotations.

The udf docstring now says a bare annotation is non-nullable and
`Optional[T]` binds a nullable column, and that a `bytes` input counts
against the per-row input limit, with Blob V2 and `BlobReader` for large
values.
2026-10-29 09:16:32 +08:00
Wyatt Alt 5fbf9fd9f0 docs(python): note that packaged Functions import lance_blob.BlobReader (#4730)
Function authors annotate inputs with lancedb.BlobReader, but the
packaged source binds that annotation to the worker's
lance_blob.BlobReader, so grepping the artifact for the client class
finds nothing. Say so on the class.
2026-10-29 08:01:12 +08:00
lancedb-gatefixer[bot] 04e509278e fix(python): clarify schemas for batch UDF annotations (#4727)
`@udf` documented batch annotations without explaining that `pa.Array`
and `pa.RecordBatch` do not specify concrete Arrow element types or
field schemas. Document the requirement for both explicit schemas
(including class `__call__` methods), add an array example, and make
inference errors point callers to `input_schema` and `output_schema`.

Fixes #4721

Validation:
- Five new regression cases pass; all four diagnostic cases fail with
the original implementation.
- Function packaging, first-class Function slice 1/2 tests, and UDF
doctests: 168 passed, 3 failed. This includes the upstream BlobReader
annotation coverage. The three existing packaging tests fail while
importing temporary modules, before UDF registration; all three failures
also reproduce with the unchanged source and tests at `057a39a1`.
- Rebuilt the native extension in the uv-managed tests/dev environment.
- Python lint and formatting checks pass (`ruff check python`, `ruff
format --check python`); `cargo fmt --all` also passes without changes.
- Repository-wide `ruff check .` reports 48 existing errors under
`docs/web-tests`.

<!-- lance-gatekeeper-fix:v1 agent=2c6567ee3f128d3a0c3b9c68e36eca38
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-28 16:37:57 -07:00
lancedb-gatefixer[bot] 741cb9ad4e fix(python): implement remote namespace and table existence checks (#4725)
Remote Python connections inherited the unsupported-operation stubs for
`namespace_exists` and `table_exists`, so both raised
`NotImplementedError` even when namespace and table operations worked.

Add the missing remote overrides, async methods, and PyO3 bindings using
the existing namespace describe and table open paths. Only an explicit
not-found result returns `False`; permission, transport, and server
errors propagate. This preserves authentication, dynamic headers,
retries, and nested namespace handling, and rechecks cached tables
before reporting that they exist.

The async namespace helper also recognizes native `NamespaceNotFound`
errors so missing local namespaces return `False`. The table and
namespace guides document Python remote support and distinguish it from
the unsupported `exists` routes used by direct REST namespace clients.

Validation:
- All 40 focused regression cases pass, covering successful and missing
lookups, local and remote namespaces, dynamic headers, permission/server
errors, invalid table identifiers, and cached tables that disappear.
- The original inherited stubs reproduce the reported
`NotImplementedError`; the local namespace regression reproduced the
native missing-namespace `RuntimeError` before its fix.
- Full remote connection suite: 255 passed. Local namespace tests: 14
passed.
- `cargo clippy --quiet -p lancedb-python --features remote --lib`,
`cargo fmt --all`, Python Ruff lint, and Python formatting checks pass.
- Root `ruff check .` reports 48 existing errors in documentation
examples under `docs/web-tests`; the Python subtree passes and those
unrelated files are unchanged.

Fixes #4722

<!-- lance-gatekeeper-fix:v1 agent=4d369a0681756fdb5a8b35853d7f5607
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-28 16:08:43 -07:00
lancedb-gatefixer[bot] fc5170cbed fix(python): expose function source digests (#4729)
Function descriptions include `source_digest`, but both the Rust
response model and Python `FunctionVersion` discarded it. Clients
therefore could not compare a catalog entry with
`udf_def.registration_request.artifact.digest`.

Preserve the digest through Rust serialization and expose it on Python
`FunctionVersion`, with a documented Rust accessor. Missing or null
digests decode as `None`, keeping older responses compatible. Regression
tests cover serialization, immutability, registration jobs, get, and
synchronous/asynchronous list responses while keeping the image digest
independent.

Validation:
- Confirmed that the original Python model silently discarded a
server-provided source digest.
- Rebuilt the Python extension in the uv-managed environment; both
focused Function test files passed (106 tests).
- Focused Rust Function model tests passed (10 tests).
- Full workspace `cargo check --quiet --features remote --tests
--examples` and `cargo clippy --quiet --features remote --tests
--examples` passed.
- Rust formatting and Ruff checks/formatting for changed Python files
passed.
- Repository-wide Ruff reports 138 existing violations in unchanged
docs/CI files.

Fixes #4723

<!-- lance-gatekeeper-fix:v1 agent=c145f25136f7b908217e04bde8a2548b
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-28 16:04:04 -07:00
lancedb-gatefixer[bot] 3035636bed fix(python): restore blob extensions in table schemas (#4728)
Reading a Blob v2 table in a fresh process could return its storage
struct if neither `blob()` nor pylance had registered the extension.
Both local and remote schema reads were affected, so `field.type` could
depend on import order.

Restore metadata-only blob fields on every table schema read through the
existing BlobType resolver and Arrow's schema deserializer. Preserve
nested fields, nullability, custom metadata, and both supported storage
layouts. The fallback deserializer now honors the stored shape too.

Fresh-process regressions cover local and remote tables, sync/async
reads, and operation with and without pylance. The four native cases
failed before removing the native-table guard. Existing merge-insert
tests now require the restored extension type while continuing to verify
updated and inserted blob contents.

Validation:

- All 12 fresh-process local/remote schema regression cases pass.
- `uv run --extra tests pytest python/tests/test_table.py
python/tests/test_blob.py python/tests/test_namespace.py
python/tests/test_util.py -q -m 'not slow and not s3_test'`: 835 passed,
11 skipped.
- Python package Ruff lint and formatting checks pass.
- Repository-wide `ruff check .` reports 48 pre-existing diagnostics in
unchanged `docs/web-tests` files.

Fixes #4719

<!-- lance-gatekeeper-fix:v1 agent=67c656bb5889e698efce5f2fd9d58ea7
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-28 16:02:41 -07:00
lancedb-gatefixer[bot] ca60b402aa fix(python): reject invalid punctuation in UDF names (#4724)
Function names containing periods or hyphens passed `@udf` validation
and failed only when registered with the server. Restrict the name
pattern to ASCII letters, digits, and underscores, preserving the
existing first-character rule so invalid names fail when the decorator
runs.

Document the accepted format and add regression coverage for dotted and
hyphenated names, including trailing punctuation, alongside valid name
overrides.

Fixes #4720

Validation:

- The four new invalid-name cases failed before the validator change
because no `ValueError` was raised.
- `cd python && uv run --extra tests pytest
python/tests/test_first_class_function_slice2.py -q`: 86 passed.
- Ruff 0.9.9: `ruff check python` and `ruff format --check python`
passed.
- Repository-root `ruff check .` reports 48 pre-existing errors in
`docs/web-tests/`.

<!-- lance-gatekeeper-fix:v1 agent=5e2441d3d264b685a6cee3c3c21ab048
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-28 15:57:48 -07:00
lancedb-gatefixer[bot] 5f6f7b591d fix(python): bind packaged BlobReader annotations to the worker type (#4726)
Packaged UDFs omit globals used only in postponed annotations, so a
`BlobReader` input has a correct wire signature but
`typing.get_type_hints()` on the loaded function raises `NameError`.

Use the resolved input signature to bind reader annotations to an
imported `lance_blob.BlobReader` alias. Preserve signature nullability
and avoid collisions with existing globals. Regression tests cover
functions and callable classes, eager and postponed annotations,
aliases, forward references, qualified names, and body globals.

Validation:

- Confirmed the new regression fails with `NameError: name 'BlobReader'
is not defined` against the original packager at
`057a39a1c70e15c78461bb7e3db8cef4c067cd3e`.
- The packaging, first-class Function slice 1 and 2, and BlobReader
suites pass all 159 tests with Python file-import directory caches
refreshed before lookups. The standard run passes 156 tests; three
temporary-module import failures reproduce unchanged on the original
packager because directory timestamps do not advance between file writes
in this environment. All 37 new regression cases pass in the standard
run.
- Python Ruff lint and formatting checks pass. Repository-wide Ruff lint
reports 48 existing errors in documentation examples.

Fixes #4718

<!-- lance-gatekeeper-fix:v1 agent=a32bb6efa79217b3c1c5c9d7a7ff9928
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
2026-10-28 15:27:22 -07:00
Will JonesandClaude Opus 5.5 4d7a64712f docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary

Adopts the product terminology guide across the docs site pages, SDK
docstrings and error messages. "LanceDB OSS" and local mode become
**LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB
Distributed**, which LanceDB Enterprise provides. Where free vs paid is
the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB
Enterprise". This is a first trial of the Embedded name, so reviewers
can judge it in context.

- **Embedded vs Distributed** for where execution runs (`optimize()`,
consistency, branching, maintenance). Main pages define it once:
"LanceDB Enterprise runs LanceDB Distributed: …"
- **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://`
region, the brute-force KNN limit) and marked legacy. "Cloud and
Enterprise" becomes Enterprise or Distributed.
- **Plan executor** becomes "executor" or the fleet name.
- **Error messages** change, e.g. `optimize is not supported for remote
tables.` → `... in LanceDB Distributed.` Tests that match on them are
updated.
- **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`,
`DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the
terms in prose.

Unchanged: identifiers (`availability.oss`, the `remote` module and
feature, `RemoteTable`, `db://`), heading anchors, internal code
comments, and Geneva.

## Evidence

- Rust lib tests for the changed messages and the Python tests in
`test_remote_db.py`, `test_materialized_views.py` and `test_table.py`
pass locally.
- Not run locally:
- The Node tests: `remote.test.ts` and `table.test.ts` have updated
assertions.
  - Vale.
  - The `experimental_functions`-marked materialized-view tests.
- `docs/src` API reference was hand-edited to match the docstrings
rather than regenerated.

## Merge Risk

**Door:** two-way; prose and message text only.
**Blast Radius:** `docs`, `error-messages`: callers matching on old
error strings would break.

Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon
(Enterprise pages).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 14:04:41 -07:00
Joshua Snider edb79d38e3 docs: correct the documented default retry statuses (#3964)
#2349 changed the default retryable status codes to `[409, 429, 500, 502, 503, 504]`, but only the Rust docstring was updated. The Python and TypeScript bindings — and the generated TypeScript API reference — still document the old `[429, 500, 502, 503]`.

The actual default is set in `ResolvedRetryConfig::try_from` in `rust/lancedb/src/remote/retry.rs`, and `rust/lancedb/src/remote/client.rs` already documents it correctly. This brings the other three in line.

Docs only — no behavior change.

Split out from #3963 per the "one problem at a time" guidance in `CONTRIBUTING.md`.
2026-10-29 03:05:40 +08:00
Lance Release ece54d8917 Bump version: 0.41.0-beta.16 → 0.41.0-beta.17 2026-10-28 17:31:48 +00:00
659ea11ac4 fix(embeddings): default Cohere to embed-multilingual-v3.0, the v2.0 models are retired (#4698)
Cohere retired `embed-english-v2.0`, `embed-english-light-v2.0` and
`embed-multilingual-v2.0` on April 4, 2026
([deprecations](https://docs.cohere.com/docs/deprecations), which lists
`embed-english-v3.0`, `embed-multilingual-v3.0` and `embed-v4.0` as the
embedding alternatives). `CohereEmbeddingFunction` still defaulted to
`embed-multilingual-v2.0`, so every call without an explicit `name`
fails.

The default is now `embed-multilingual-v3.0`, the multilingual v3 model,
and the retired models are removed from the docstring and the Cohere
integration page (the page also listed `embed-english-v2.0` as the
default, which did not match the code). The docs example, its generated
snippet, the docs test and the slow Cohere test use the new name.

This changes the default vector size: `ndims()` returns 1024 for
`embed-multilingual-v3.0`, where the old default gave 768. Tables
created from a schema that uses `cohere.ndims()` get 1024-dimension
vectors from now on. A table that already holds 768-dimension vectors
from the retired model needs a re-embed with a current model, since the
old model can no longer be called. The `ndims()` entries for the v2
names are left as they are.

Test: `test_cohere_default_model_is_not_a_retired_v2_model` in
`python/python/tests/test_embeddings.py` checks the default name and
that `ndims()` is 1024. The repository's pytest session could not be run
for this change (its `conftest.py` needs a newer native module than the
released wheel); the body of that test was run against the released
`lancedb` 0.40.0 with this version of `cohere.py` in place, where it
passes, and it returns `embed-multilingual-v2.0` / 768 without the
change. ruff check and format are clean on the touched Python files. No
Cohere request was made.

This change was made with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-29 00:16:28 +08:00
Lance Release 45f987a29b Bump version: 0.41.0-beta.15 → 0.41.0-beta.16 2026-10-28 05:01:52 +00:00
Wyatt Alt bb6aa1a147 feat: add a cascade option to drop_database (#4697)
The remote catalog could only drop an empty database: it always asked
the server for a restricted drop.

`DropDatabaseRequest::cascade`, `drop_database(..., cascade=True)` in
Python and `dropDatabase(name, { cascade: true })` in TypeScript ask the
server to drop everything in the database first. The name stays taken
until the server finishes cleaning up the dropped objects' storage. A
restricted drop remains the default.
2026-10-27 21:51:10 -07:00
Jack YeandXuanwo 45dbf5eff5 fix: use SQL for all remote materialized view APIs (#4673)
Remote materialized-view APIs now use SQL for the full lifecycle. Create
and refresh already submit SQL; this change moves describe, paginated
listing, and drop onto `DESCRIBE MATERIALIZED VIEW`, `SHOW MATERIALIZED
VIEWS`, and `DROP MATERIALIZED VIEW`.

Async create, refresh, and drop return the existing Job handle with
`job_type = "query"`. The handle has one `id`: the connection-scoped SQL
query ID, also accepted by `describe_query`. Submission starts the
query, the job drains its result once, and repeated waits reuse the
completed result. Synchronous drop waits for catalog removal; physical
cleanup proceeds in the background.

Listing uses namespace-scoped keyset pagination with opaque resume
tokens. Streamed and deferred SQL responses share the same path. Adds
transport coverage and updates Python/TypeScript documentation for
query-backed drop jobs.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-28 04:37:00 +08:00
Jay PokaleandXuanwo f6a348ac5b fix(python): keep FTS scores in RRF and MRR rerankers with return_score="all" (#4677)
`RRFReranker.rerank_hybrid` and `MRRReranker.rerank_hybrid` merge the
vector and FTS results with the base `merge_results`, which keeps only
the first copy of a duplicated row. With `return_score="all"`, a row
that both searches found therefore comes back with `_score = None`, even
though the FTS search scored it. `LinearCombinationReranker` already
copies the FTS score onto such rows.

Reproduction against `lancedb` 0.40.0:

```python
import pyarrow as pa
from lancedb.rerankers import RRFReranker

vector = pa.table({"_rowid": pa.array([1, 2], pa.uint64()), "text": ["a", "b"],
                   "_distance": pa.array([0.1, 0.2], pa.float32())})
fts = pa.table({"_rowid": pa.array([2, 3], pa.uint64()), "text": ["b", "c"],
                "_score": pa.array([5.0, 4.0], pa.float32())})
RRFReranker(return_score="all").rerank_hybrid("q", vector, fts).to_pylist()
# row 2: _distance=0.2, _score=None   (expected _score=5.0)
```

`table.search(query_type="hybrid")` is not affected, because
`_combine_hybrid_results` restores the original `_distance` and `_score`
by `_rowid` after reranking. The wrong value shows up when
`rerank_hybrid` is called directly, for example from a custom reranker
that wraps RRF or MRR.

The fix uses `_merge_and_keep_scores` when `return_score="all"`, the
same pattern the model-based rerankers (`CohereReranker`,
`CrossEncoderReranker`, `JinaReranker`, `OpenaiReranker`,
`VoyageAIReranker`, ...) already use. The relevance-only path is
unchanged.

Tests: added `test_rank_fusion_return_score_all_keeps_both_scores` for
both rerankers. It fails without the fix and passes with it. I ran it
against the released 0.40.0 package with this branch's
`rerankers/{base,rrf,mrr}.py` dropped in; `rrf.py` and `mrr.py` on
`main` are identical to 0.40.0. The other reranker tests give the same
results with and without the change. `ruff format` and `ruff check` are
clean.

I used an AI assistant (Claude) while working on this; I reviewed and
tested the change.

Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-28 03:33:36 +08:00
lancedb-gatefixer[bot]andXuanwo ac99ab173b fix(python): support native blob conversion in pandas and batches (#4647)
Plain local blob v2 scans tried to open a pylance dataset before
reaching LanceDB's byte-fetch path, so `to_pandas(blob_mode="bytes")`
failed when the optional dependency was absent. Batch readers also had
no `blob_mode` option.

Route supported blob v2 pandas conversions through the native Arrow
path. Add `blob_mode` to sync and async batch readers, convert payloads
during iteration, and expose the converted schema without fetching the
first batch. Apply conversion after reranking and retain the default
descriptor behavior. Clean up scanner error punctuation and remove the
misleading plain-scan suggestion.

Bytes mode retains the existing `to_arrow` restrictions for legacy/list
blobs, remote tables, and namespace query pushdown. Lazy pandas handles
and legacy pandas scans still require pylance; sync FTS/hybrid batching
remains unsupported.

Validation:
- UV-managed editable Python development build completed successfully.
- The absent-pylance regression tests fail against the original query
implementation and pass with this change.
- Blob and query suites: 403 passed, 1 skipped, 1 deselected.
- Remote query suite: 190 passed.
- Python-tree Ruff lint/format checks and the async reader Pyright check
pass. Repository-wide Ruff reports existing errors outside the Python
tree.

Fixes #4645

<!-- lance-gatekeeper-fix:v1 agent=fd85998f14bb0b3c92b2d050df93a615
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-28 02:59:02 +08:00
Dan Tasse 6e4f523e14 docs: remove logical-column from docs (#4693)
We've removed this feature from LanceDB Enterprise and no longer
privilege the "lancedb:logical-column" key. So, removing that from these
docs.
2026-10-27 13:59:51 -04:00
Xuanwo b44a0d2362 chore: update Lance to v15.0.0-beta.3 (#4692)
Upgrade Lance to 15.0.0-beta.3 so `Dataset::concat_data_file_parts`
returns a `Send` future and downstream workers can run it on Tokio. The
upstream fix is https://github.com/lance-format/lance/pull/10370.

Keep the vendored Arrow validity fix; upstream Arrow sources are
unchanged between beta.2 and beta.3.
2026-10-28 01:49:32 +08:00
Ozawa SongandXuanwo 4b373b2070 fix(python)!: support compat-only imports (#3973)
Read `__version__` from the native `_lancedb` module so `import lancedb`
works when only `lancedb-compat` is installed. Add the type declaration
and a subprocess regression test.

Ruff and Linux-targeted Pyright pass. Import tests: 1 passed and 1
Linux-only test skipped on Windows. Runtime tests used the official
same-version Windows native module; a fresh source build still needs CI.

BREAKING CHANGE: `lancedb.__version__` now uses the native version
string.
Prereleases may report `0.41.0-beta.13` instead of `0.41.0b13`.

Fixes #3950

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-28 00:29:22 +08:00
Jay PokaleandXuanwo dcac834233 fix(python)!: reject tuple fields with mixed element types in pydantic models (#4679)
`pydantic_to_schema` maps a `Tuple[...]` field to an Arrow list of its
**first** element type. A tuple with different element types gets the
wrong schema, and the error only surfaces when a row is written:

```python
class Pair(LanceModel):
    pair: Tuple[int, str]

pydantic_to_schema(Pair)        # pair: list<item: int64>
db.create_table("t", schema=Pair).add([Pair(pair=(1, "a"))])
# pyarrow.lib.ArrowInvalid: Could not convert 'a' with type str: tried to convert to int64
```

An Arrow list has a single element type, so such a tuple cannot be
stored as one. This raises a `TypeError` when the schema is built,
saying the tuple elements must share one type and pointing to a nested
`LanceModel` for a fixed set of differently typed fields. Homogeneous
tuples are unchanged: `Tuple[int, int]` and `Tuple[float, ...]` still
map to lists.

**Breaking change:** a model with a mixed-type tuple field, which used
to get a (wrong) `list<...>` schema from `pydantic_to_schema`, now
raises `TypeError` when the schema is built. Such a schema could not
hold the model's values: writing a row failed with `ArrowInvalid`. The
requirement and the nested-`LanceModel` alternative are documented in
the `pydantic_to_schema` docstring.

Tests (in `test_pydantic.py`): `TypeError` for `Tuple[int, str]` (fails
on `main`), and the list mapping for `Tuple[int, int]` and `Tuple[float,
...]`. The whole file passes; `ruff format` and `ruff check` are clean.

I used an AI assistant (Claude) while working on this; I reviewed and
tested the change.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-27 23:35:04 +08:00
陈志谦andXuanwo 3575f972ac fix(python): accept tuples and pyarrow arrays as hybrid search vectors (#4662)
The synchronous hybrid query builder rejected vector containers that the
rest of the Python API accepts:

```python
table.search(query_type="hybrid").vector((1.0, 2.0)).text("hello").to_pandas()   # ValueError
table.search(query_type="hybrid").vector(pa.array([1.0, 2.0], pa.float32()))...
# ValueError: Vector query must be either a string or a vector
```

`lancedb.common.VEC` documents `list[float] | tuple[float, ...] |
np.ndarray | pa.Array`, and the same two containers work for
`table.search(vector)` and for the async hybrid API
(`AsyncQuery.nearest_to(...).nearest_to_text(...)`).
`LanceHybridQueryBuilder._validate_query` only listed `str | list |
np.ndarray`, so a tuple or a pyarrow array was refused before reaching
the query planner.

This normalises the vector exactly the way `LanceQueryBuilder.create`
already does — list/tuple to a `np.float32` array, `np.ndarray` through
`astype(float32)`, `pa.Array`/`pa.ChunkedArray` through `np.array(...,
dtype=float32)`, strings passed through — and keeps the existing
`ValueError` for anything else.


`python/python/tests/test_hybrid_query.py::test_hybrid_query_accepts_every_vector_container`
is parametrised over tuple, pyarrow array and pyarrow chunked array, and
asserts the returned `_rowid`s match the `list` form: 3 failed on `main`
(`ValueError` from `query.py:2452`), 3 passed with the change.
Neighbouring suites (`test_hybrid_query.py`, `test_query.py`,
`test_rerankers.py`) are 220 passed / 9 skipped, and `test_table.py`,
`test_remote_db.py`, `test_fts.py`, `test_index.py`,
`test_permutation.py` are 724 passed / 5 skipped. `ruff` at the version
pinned in the repository's pre-commit config (0.9.9) reports no findings
and `ruff format --check` is clean.

One source file (`python/python/lancedb/query.py`), no behaviour change
for containers that already worked.

Prepared with AI assistance; every line was verified by running the
cases above.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-27 22:49:31 +08:00
lancedb-gatefixer[bot]andXuanwo 2cd635d41c fix: correct remote materialized view refresh results and options (#4672)
Remote materialized-view refreshes drained the SQL result and then
fabricated `mode=rebuild`, `rows_written=<total view rows>`, and
`source_version=0`. This made incremental refreshes report incorrect
statistics.

Make these three result fields optional across Rust, Python, and
TypeScript, and leave them unset for remote SQL refreshes. Remove the
post-refresh row count request. Local refreshes continue to populate all
metrics, and `version` remains the view version observed after
completion. Remote per-operation statistics remain unavailable until the
SQL job adapter can retrieve them.

Expose the forced-rebuild option through the Rust builder,
synchronous/asynchronous Python refresh and job methods, and TypeScript
refresh. Remote `full=true` sends `WITH (full = true)`; the default lets
the server choose its refresh mode. Local source-version pinning and
remote job cancellation are preserved.

Local refresh regression tests respect the `experimental-functions`
build gate, while remote refresh coverage also runs in default builds.

Validation:

- Rust: 20 materialized-view tests and 21 remote SQL tests passed with
`remote,experimental-functions`; `cargo fmt --all` passed.
- Python: rebuilt the extension in the uv-managed environment. The
default build passed 23 tests with 27 feature-dependent skips. The
opt-in build passed 61 tests with 3 expected default-build-only skips,
covering the materialized-view suite and 14 async query-serialization
cases from current main. Python Ruff lint and formatting checks passed.
- TypeScript: default and opt-in native builds and TypeScript
compilation passed. The default build passed 5 tests with 10 expected
feature-dependent skips; the opt-in build passed all 14
materialized-view tests with the default-build-only test skipped.
Lint/format checks and documentation generation passed.
- Repository-wide `ruff check .` reports 48 pre-existing violations in
unchanged `docs/web-tests` files; `ruff check python` passes.

Fixes #4670

<!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-27 21:34:01 +08:00
lancedb-gatefixer[bot]andXuanwo c167b99fa0 fix: read legacy blob columns through blob APIs (#4649)
Legacy blob v1 columns were rejected by LanceDB's blob validation and
omitted from `blob_columns()`, even though the pinned Lance reader
already supports them. Accept both blob versions in read validation and
discovery so `fetch_blobs`, `fetch_blob_files`, and `fetch_blob_ranges`
can read legacy payloads through the existing native reader.

Keep list-nested blobs excluded from row-based reads and document how to
fetch legacy blobs using explicit row IDs. Regression coverage checks
sync and async APIs, independently written Lance datasets, ordering and
duplicates across fragments, null and empty blobs, lazy seeking, ranges,
and deleted rows.

Fixes #4644

Validation:

- Both new Python regressions failed on the original implementation with
the reported legacy-column rejection and empty discovery result, then
passed with the fix.
- Python blob suite: 202 passed, 1 skipped.
- Remote blob tests: 29 passed.
- Focused Rust blob unit tests with `remote`: 13 passed.
- Rust blob integration suite with `remote`: 59 passed, including legacy
storage 2.0 and 2.1.
- Rust compile and Clippy checks with `remote`, tests, and examples
passed.
- Python package Ruff lint and format checks passed; Rust formatting
passed.
- Repository-wide `ruff check .` reports 49 existing errors in unchanged
`docs/web-tests` examples; the Python package is clean.

<!-- lance-gatekeeper-fix:v1 agent=02b7edcbfa8ff0e7fad4615159249b3e
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-27 19:56:50 +08:00
lancedb-gatefixer[bot]andXuanwo 83b7275137 fix(remote): make HTTP timeout errors actionable (#4650)
Remote `create_table` reader uploads can exceed the read deadline while
waiting for response headers, but reqwest's main error message only says
"error sending request". Make HTTP and retry errors explicitly identify
timeouts and name the relevant `client_config.timeout_config` settings
and environment variables, preserving typed sources, request metadata,
and retry classification.

Document that the read timeout covers waiting for headers during upload.
This implements the issue's diagnostic option: creation still uses one
streaming request, so large or slow readers may require a longer
`client_config.timeout_config.read_timeout`.

Validation:
- 48 focused Rust client, error, and retry tests passed. Regression
coverage holds an upload open until its read timeout and also exercises
TLS connection timeouts, with and without retries.
- 8 Python create-table and HTTP/retry error tests passed in the
uv-managed environment after rebuilding the extension, including reader
creation with read and overall timeouts.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo clippy --quiet --features remote --tests --examples` passed.
- Rust formatting and Python package Ruff lint/format checks passed.
- Repository-wide `ruff check .` reports 49 existing findings under
`docs/web-tests`; those files are unchanged.

Fixes #4646

<!-- lance-gatekeeper-fix:v1 agent=ce1c040fa581f1748166ecb686d9c9c0
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
2026-10-27 18:51:06 +08:00
Xuanwo ad1212fb32 chore: update Lance to v15.0.0-beta.2 (#4690)
**Blocked on the corrected Lance release:**
https://github.com/lance-format/lance/pull/10370 restores the `Send`
contract required by downstream Function workers. Refresh this upgrade
with that release before merging.

Upgrade the Rust, Java, and Python Lance dependencies to
https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.2. This
brings in bounded vector-index consolidation without retraining
(https://github.com/lance-format/lance/pull/10335) and deletion
filtering for index entries whose fragment coverage was pruned
(https://github.com/lance-format/lance/pull/10325).

Synchronize the vendored `lance-arrow` with the release and retain its
offset-safe validity patch. The existing `nullable_struct_merge`
regression still panics with the unpatched release, so removing that
patch would regress nullable nested queries. Refresh both lockfiles,
including the `zstd-sys` version required by the new Lance encoding
dependency.

Update the custom memtable bitmap test adapter to train on row
addresses, matching the new bitmap contract; remove its obsolete row-ID
conversion.

Preserve the Blob APIs' existing missing-row error messages when newer
Lance rejects a missing fragment earlier in the read path. Retain
upstream attribution in vendored sources while satisfying the repository
license-header check.

Validation of the compatibility fixes: all 58 Rust Blob integration
tests and 219 Python Blob tests pass (one Python test skipped). Strict
workspace Clippy, formatting, and diff checks pass.
2026-10-27 17:35:43 +08:00