rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-13 14:16:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,450 Commits 451 Branches 581 Tags
b09735529b7720ba610ecc4f51a8f2f32cbd86d2
Commit Graph
290 Commits
Author SHA1 Message Date
Xuanwo b09735529b feat(sql): return durable materialized view job handles (#4824) 
Materialized-view creation and refresh can outlive the submitting
connection, but their SQL query IDs cannot be reopened through the Job
Registry. On servers that negotiate early handles, return the durable
registry job ID and use the existing job API to describe, reopen, wait
for, or cancel it. Older servers retain the connection-scoped query
handle. Registry refresh handles forward running and terminal progress
observations and establish the committed-version read-freshness fence
before wait returns.

Preserve a nested SQL failure ID separately from the outer job ID
through Rust and the Python native bindings, so Function failures remain
traceable. Rust callers constructing `JobFailure` directly must supply
the new optional `sql_job_id` field or use `..Default::default()`.

Validation: seven SDK regression cases and the docs-link test pass,
including plain wait and running-to-terminal progress observations; the
observer regression fails before the fix. Formatting and strict Clippy
pass. Earlier accepted coverage includes a real Flight test service
exercising materialized-view creation followed by registry
describe/reopen/cancel, and two Python tests after rebuilding the native
extension. The published product tree matches the validated candidate.

Refs ENT-3562. The matching Sophon server change is being validated
separately before publication.
 2026-11-04 12:04:02 +08:00
Xuanwo 1544ad65a8 feat: expose structured job progress to SDK callers (#4808) 
Job callers currently lose the server's structured progress when
describing, listing, or waiting for a job. Expose Rust and Python
progress snapshots and optional wait observers using the existing
backend poller. Python supports callbacks, a caller-owned tqdm-like bar,
or `progress=True`; `status()` still returns a string and the default
wait behavior is unchanged.

Preserve unknown progress fields and phases from newer servers, and
report omitted counters as unknown. Each wait calibrates a caller-owned
bar from its first complete snapshot, then uses a high-water mark within
the same execution epoch; a new epoch or smaller plan resets the bar.
ETA is best-effort and omitted for missing, stale, paused, or
incompatible samples.

Newly submitted index and refresh handles forward observers through
their freshness wrapper while retaining successful-wait read
consistency. The Jobs guide includes draft Python/Rust snapshot and
observer examples for release; TypeScript progress support remains
tracked under ENT-3500.

Validation on the development EC2 host: the initial 11 Python progress
tests, REST fixture caller, and two Rust progress regressions passed.
After fixing freshness-wrapper forwarding, rebuilt the Python extension
and ran two submitted-handle Python tests plus the new Rust refresh and
index regressions (one test each); all passed. Strict Clippy, rustfmt,
and targeted Ruff passed. The final commit only wraps the new guide
section in an MDX draft and clarifies language support. Repository-wide
Ruff reports 48 existing findings in six unchanged docs/web-tests files.

This publishes the SDK portion of ENT-3500. Server-side live-split and
terminal-progress changes and the combined server/SDK end-to-end
scenario remain in progress. Refs ENT-3500.

BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress:
None`, `phase: None`, and `paused: false`. Literals for `JobDescription`
must add `progress: None`, `phase: None`, `queue_reason: None`,
`waiting_for_job_id: None`, and `paused_at_millis: None` when that
metadata is unavailable. Existing Python calls remain compatible.
 2026-11-03 18:17:19 +08:00
Xuanwo 2d70bec53a feat: add UTM-tagged docs links to SDK warnings and enterprise-only errors (#4797) 
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.

- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.

Linear: OSS-2772
 2026-11-03 14:12:15 +08:00
LeoandLeo f2bcb156ad feat: expose approx_mode on vector queries 
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.

Co-authored-by: Leo <leo@insight.co>
 2026-11-03 12:45:52 +08:00
Srijan Keshri 90b2b264d9 feat(python): accept DataFrames and readers in add_columns 
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.

Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.

SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.

Fixes #1959.
 2026-11-03 09:45:41 +08:00
lancedb-gatefixer[bot] 7466dc9d1d fix(python): reject embedding source updates without vectors 
Fixes #4761

`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.

The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.

This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.

Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
 2026-11-03 07:07:35 +08:00
lancedb-gatefixer[bot] ea9194a7f3 fix(python): retry transient job polling errors 
Fixes #4773.

Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.

Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
 2026-11-03 05:05:30 +08:00
TaylorandBrendan Clement 9e65bd2c27 docs: use Enterprise terminology across website and SDK documentation (#4787) 
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.

- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.

Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.

Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).

---------

Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
 2026-11-02 10:25:14 -08:00
SHAIK SHIVAJI 45edaa350e feat(python): expose code analyzer options on FTS and tokenize 
Part of #4364 (Python side; the TypeScript `FtsOptions` change will follow in a separate PR).

Lance already supports the code analyzer, but `FTS` and `lancedb.tokenize()` had no way to set it, so `FTS({'analyzer': 'code'})` and `FTS(split_identifiers=True)` raised `TypeError`.

## Changes

- `FTS` and `lancedb.tokenize()` gain `analyzer` (`"text"` or `"code"`), `split_identifiers`, `split_on_numerics`, `preserve_original` and `index_operators`. They default to `None`, which keeps the tokenizer's own defaults, so existing indexes and calls are unchanged.
- Validation follows pylance: `analyzer="code"` with a `base_tokenizer` other than `"code"` or the default `"simple"` is rejected, `analyzer="text"` with `base_tokenizer="code"` is rejected, and turning on a code flag without the code analyzer is rejected. Passing a flag as `False` is always allowed.
- `analyzer="code"` with the default `base_tokenizer="simple"` selects the code tokenizer. `FTS` can't tell an explicit `"simple"` from the default, so that case is accepted too.
- The deprecated `create_fts_index` is left alone; it points users at `create_index(config=FTS(...))`.
- Rust `FtsIndexBuilder` is Lance's `InvertedIndexParams`, so the Rust SDK already had these options. The remote client serializes the same struct, so the options reach LanceDB Distributed in the `create_index` request body. A new case in the Rust `test_create_index` checks that body.

## Tests

- `python/tests/test_fts.py`: tokenizer output for each flag, validation errors for `tokenize` and `FTS`, `False` flags on a text index, and local sync and async indexes that match `http`, `request` and `parseHTTPRequest` with splitting on and not match subwords with it off.
- `cargo test --features remote remote::table::tests::test_create_index`
 2026-11-02 16:49:14 +08:00
Xuanwo 63aa06ef7d feat(python): expose Function GPU classes and placement (#4791) 
## Summary

- Add sync/async `DBConnection.list_gpu_classes()` returning typed
`GpuClass` records for `GET /v1/gpu/classes`. Records include the class
name, Kubernetes resource, GPU model and memory, per-worker GPU maximum,
and worker CPU/memory/ephemeral-storage limits.
- Add `udf(gpu_class=...)` for immutable Function-level GPU placement.
It requires `gpu=True`; Function versions expose their registered
`gpu_class`.
- Add `gpu_class` and `gpu_count` to `refresh_column` /
`refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as
top-level `backfill_column` fields and validated before any request.
`gpu_count` is the aggregate GPU budget across Function workers.
- Validate GPU class names with the server lowercase 63-character
DNS-label rule.
- Document the discovery-to-registration flow, configured hardware
versus free capacity, System Usage permission, and per-refresh GPU
placement in the API and web guides.

Older servers without `/v1/gpu/classes` produce a clear
`NotImplementedError` instead of a bare 404.

Part of ENT-3470. The client follows the in-flight Sophon contract on
`xuanwo/ent-3470-fix` for GPU class discovery and backfill placement
options.

## Breaking change

`PythonRuntimeSpec::PythonV2` adds `gpu_class`, and
`RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers
constructing either value with all fields explicitly must supply the new
fields.

## Validation

At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python
extension build, pyright, ruff, and Python Function/runtime, discovery,
refresh placement, and refresh regressions (147 passed, 6 skipped).

At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright,
Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed).

The final commit only updates documentation and Rust doc comments.
 2026-11-02 08:50:36 +08:00
Xuanwo 6a255926d7 fix(python): improve function failure diagnostics (#4790) 
## Summary

- Make `AsyncJob.wait(timeout)` and `Job.wait(timeout)` raise a
descriptive `TimeoutError` containing the job ID and elapsed timeout.
Registry jobs explain that work continues on the server and include the
`db.open_job(id).wait()` reattach call. SQL query handles instead say to
keep the original handle because query IDs cannot be reopened through
the registry jobs API.
- Render `JobFailureInfo` and `FunctionErrorRecord` with Python values
instead of Rust `Option` wrappers.
- Decode and expose the server's `row_address` on `FunctionErrorRecord`,
and document how to retry recorded rows with `REFRESH COLUMN ... WHERE
_rowaddr IN (...)` before compaction or rewrite.
- Warn when an import inside a packaged Function body resolves to
undeclared local source. The `udf` documentation explains why arbitrary
base-image coverage and distribution-to-import-name mapping cannot be
validated reliably by the SDK.

`row_address` is optional, so records from older servers still decode
with `None`. Rust code that constructs `FunctionErrorRecord` literals
must now supply this new field.

Part of ENT-3448 and ENT-3442. This consumes the server-side diagnostics
from lancedb/sophon#9509 and logical row addresses from
lancedb/sophon#9512.

## Validation

- Rust `function_errors` remote decoding and
legacy-missing-`row_address` tests.
- Rust/PyO3 clippy for `lancedb` and `lancedb-python`.
- Python extension build, pyright, and ruff.
- Targeted Python job, Function packaging, Function error, and job
detail tests: 62 passed, plus timeout-handle tests: 3 passed.
 2026-11-02 06:44:48 +08:00
Xuanwo 9786f2b63b fix(python): surface typed remote resource errors (#4789) 
## Summary

- Parse structured `error_code` and `resource` fields from remote HTTP
error bodies and carry them through Rust/PyO3.
- Add `SecretNotFoundError`, `FunctionNotFoundError`,
`NamespaceNotFoundError`, and `AlreadyExistsError` as `HttpError`
subclasses.
- Expose `error_code`, `resource_kind`, `resource_name`, and
`namespace_path` on HTTP exceptions while preserving `status_code`,
`request_id`, and existing `except HttpError` handlers.
- Do not retry structured domain errors that would otherwise become
`RetryError` (notably `already_exists` on 409).

Older servers that do not provide the structured fields continue to
raise a plain `HttpError`.

## Compatibility

This consumes the stable resource error contract added in
lancedb/sophon#9516 (merge commit `35858561b3`). The existing
`TableNotFoundError` behavior is unchanged.

Part of ENT-3373.
 2026-11-02 05:11:38 +08:00
BHAWESHBHASKAR a9146d536a feat(python): expose the NGram scalar index 
## What kind of change does this PR introduce?

Feature. Exposes the NGram scalar index in the Python SDK.

## What is the current behavior?

The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.

## What is the new behavior?

Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:

```python
from lancedb.index import NGram

await table.create_index("text", config=NGram())
```

The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.

Changes:

- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.

This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
 2026-11-01 11:54:27 +08:00
lancedb-gatefixer[bot] 71b23ed9fb fix(python): preserve request metadata for missing remote tables 
Fixes #4750

Remote missing-table errors retained HTTP metadata in Rust but the Python binding converted them to a plain `ValueError`, discarding the request ID and status. This raises a `TableNotFoundError` that preserves the original message, request ID, status code, and chained HTTP cause while remaining compatible with existing `ValueError` and `lance_namespace.errors.TableNotFoundError` handlers. Local and non-HTTP errors retain their previous behavior.

Validation covers sync and async opens across cached/uncached and root/nested paths, exception pickling, local compatibility, the remote and error test suites, Clippy, formatting, Ruff, and all applicable PR checks including CodeQL.
 2026-11-01 11:10:44 +08:00
Yui Numazawa faa8953d81 fix(python): allow attribute access on namespace request dicts 
Allows Python namespace implementations to read request dictionary entries as attributes while preserving mapping and `model_dump()` behavior. Missing dunder attributes continue to raise `AttributeError`.

Adds a real namespace connection regression covering list/open operations and omitted optional fields.
 2026-11-01 08:01:05 +08:00
lancedb-gatefixer[bot] 9804f26bc3 fix: support batch size options for column refreshes (#4777) 
Python column refresh methods rejected `batch_size`, and the Rust remote
backend never included it in the backfill request. Callers could not
reduce the rows passed to a slow Function to stay within the executor's
per-call deadline.

Add optional, keyword-only `batch_size` arguments throughout the
synchronous and asynchronous Python APIs and bindings. Add compatible
Rust `RefreshColumnOptions` APIs while preserving existing Rust method
signatures. Reject non-positive sizes before submitting a job, send
positive sizes in `backfill_column`, and omit the field when unset while
retaining branch scoping.

Batch size controls remote Function calls. Embedded expression refreshes
accept and validate the option without changing execution, and blocking
remote refreshes retain their existing unsupported behavior; remote
callers use `refresh_column_async` and wait on the returned job.

Validation:

- Reproduced the original Python `TypeError` from the committed method.
- Five focused Rust tests passed, covering request contents, omission,
branch scoping, and rejection before any request.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo clippy --quiet --features remote --tests --examples -- -D
warnings` passed.
- Nineteen targeted Python tests passed against the rebuilt extension,
including synchronous and asynchronous remote requests, invalid sizes,
and default-build behavior. Eight embedded refresh cases require the
experimental-functions build and were skipped.
- Rust formatting and Python formatting/lint checks passed, including
CI's pinned Ruff version. Repository-wide Ruff reports 48 existing
findings in untouched documentation files; none are in changed files.

Fixes #4776

<!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-31 10:00:10 -07:00
Bruno Ramirez 5f7ec4e0a2 feat(sdk): expose table overrides for remote tables 
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.

Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.

The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.

Validated with focused Rust and Python override tests and the complete applicable
CI run for 43bccbadc5.
 2026-10-31 10:44:57 +08:00
Jack Ye 29691719fa feat: manage property graphs defined over tables 
Replaces #4322, keeping only its property graph defined over tables.

A property graph maps node labels and relationship types to existing tables of its namespace, in the shape of SQL/PGQ's `CREATE PROPERTY GRAPH`. It stores only that definition: a query reads the tables as nodes and edges through SQL `GRAPH_TABLE`, as they are when it runs.

- **Rust:** `Connection::create_property_graph` takes a `PropertyGraphDefinition` of `NodeTable`s and `EdgeTable`s. `describe_property_graph`, `drop_property_graph` and `list_property_graphs` manage it.
- **Python:** `lancedb.graph` has `NodeTable`, `EdgeTable` and `Endpoint`. Sync and async connections gain `create_property_graph`, `open_property_graph`, `drop_property_graph` and `list_property_graphs`, and a `PropertyGraph` handle describes its graph.
- **Support:** remote connections only. Local connections raise `NotImplementedError` in Python and return `NotSupported` in Rust.

```python
import lancedb
from lancedb.graph import EdgeTable, Endpoint, NodeTable

db = lancedb.connect("db://demo", api_key="...")
db.create_property_graph(
    "social",
    nodes=[NodeTable("person", key="person_id", label="Person")],
    edges=[
        EdgeTable(
            "knows",
            label="KNOWS",
            source=Endpoint("src_id", references=("person", "person_id")),
            destination=Endpoint("dst_id", references=("person", "person_id")),
        )
    ],
)
db.execute_query("""
    SELECT * FROM GRAPH_TABLE (social
        MATCH (p:Person)-[:KNOWS]->(f:Person) WHERE p.age > 30
        COLUMNS (p.name AS person, f.name AS friend))
""").read_all()
```
 2026-10-31 08:25:06 +08:00
ryan-haynesandRyan Haynes d316cbbed4 fix!: support opt-in PKCE for device-code login (#4747) 
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.

Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.

OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.

The supported-flows guide documents the Rust-only device PKCE option and
its default.

## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.

## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.

---------

Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
 2026-10-30 17:26:05 -06:00
Jack Ye 6d126a3601 feat: add database storage integration APIs (#4618) 
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.

### Python experience

```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
    "raw", "s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
    aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```

For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.

### SQL experience

```sql
CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
    STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;

CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
    STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```

CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.

Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
 2026-10-30 00:26:42 -07:00
BHAWESHBHASKAR 891774343e feat(python): add CombinedFieldsQuery for BM25F full text search 
Add a typed Python `CombinedFieldsQuery` for BM25F search across multiple columns.

The query mirrors Lance's combined-fields semantics with ordered per-column boosts and AND/OR term matching. Convert it in both directions at the PyO3 boundary, including nested Boolean queries, and document the Python typed convenience entry.

Closes #4652.
 2026-10-29 17:29:22 +08:00
lancedb-gatefixer[bot] 741cb9ad4e fix(python): implement remote namespace and table existence checks (#4725) 
Remote Python connections inherited the unsupported-operation stubs for
`namespace_exists` and `table_exists`, so both raised
`NotImplementedError` even when namespace and table operations worked.

Add the missing remote overrides, async methods, and PyO3 bindings using
the existing namespace describe and table open paths. Only an explicit
not-found result returns `False`; permission, transport, and server
errors propagate. This preserves authentication, dynamic headers,
retries, and nested namespace handling, and rechecks cached tables
before reporting that they exist.

The async namespace helper also recognizes native `NamespaceNotFound`
errors so missing local namespaces return `False`. The table and
namespace guides document Python remote support and distinguish it from
the unsupported `exists` routes used by direct REST namespace clients.

Validation:
- All 40 focused regression cases pass, covering successful and missing
lookups, local and remote namespaces, dynamic headers, permission/server
errors, invalid table identifiers, and cached tables that disappear.
- The original inherited stubs reproduce the reported
`NotImplementedError`; the local namespace regression reproduced the
native missing-namespace `RuntimeError` before its fix.
- Full remote connection suite: 255 passed. Local namespace tests: 14
passed.
- `cargo clippy --quiet -p lancedb-python --features remote --lib`,
`cargo fmt --all`, Python Ruff lint, and Python formatting checks pass.
- Root `ruff check .` reports 48 existing errors in documentation
examples under `docs/web-tests`; the Python subtree passes and those
unrelated files are unchanged.

Fixes #4722

<!-- lance-gatekeeper-fix:v1 agent=4d369a0681756fdb5a8b35853d7f5607
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-28 16:08:43 -07:00
Will JonesandClaude Opus 5.5 4d7a64712f docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713) 
## Summary

Adopts the product terminology guide across the docs site pages, SDK
docstrings and error messages. "LanceDB OSS" and local mode become
**LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB
Distributed**, which LanceDB Enterprise provides. Where free vs paid is
the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB
Enterprise". This is a first trial of the Embedded name, so reviewers
can judge it in context.

- **Embedded vs Distributed** for where execution runs (`optimize()`,
consistency, branching, maintenance). Main pages define it once:
"LanceDB Enterprise runs LanceDB Distributed: …"
- **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://`
region, the brute-force KNN limit) and marked legacy. "Cloud and
Enterprise" becomes Enterprise or Distributed.
- **Plan executor** becomes "executor" or the fleet name.
- **Error messages** change, e.g. `optimize is not supported for remote
tables.` → `... in LanceDB Distributed.` Tests that match on them are
updated.
- **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`,
`DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the
terms in prose.

Unchanged: identifiers (`availability.oss`, the `remote` module and
feature, `RemoteTable`, `db://`), heading anchors, internal code
comments, and Geneva.

## Evidence

- Rust lib tests for the changed messages and the Python tests in
`test_remote_db.py`, `test_materialized_views.py` and `test_table.py`
pass locally.
- Not run locally:
- The Node tests: `remote.test.ts` and `table.test.ts` have updated
assertions.
  - Vale.
  - The `experimental_functions`-marked materialized-view tests.
- `docs/src` API reference was hand-edited to match the docstrings
rather than regenerated.

## Merge Risk

**Door:** two-way; prose and message text only.
**Blast Radius:** `docs`, `error-messages`: callers matching on old
error strings would break.

Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon
(Enterprise pages).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-28 14:04:41 -07:00
Wyatt Alt bb6aa1a147 feat: add a cascade option to drop_database (#4697) 
The remote catalog could only drop an empty database: it always asked
the server for a restricted drop.

`DropDatabaseRequest::cascade`, `drop_database(..., cascade=True)` in
Python and `dropDatabase(name, { cascade: true })` in TypeScript ask the
server to drop everything in the database first. The name stays taken
until the server finishes cleaning up the dropped objects' storage. A
restricted drop remains the default.
 2026-10-27 21:51:10 -07:00
lancedb-gatefixer[bot]andXuanwo 2cd635d41c fix: correct remote materialized view refresh results and options (#4672) 
Remote materialized-view refreshes drained the SQL result and then
fabricated `mode=rebuild`, `rows_written=<total view rows>`, and
`source_version=0`. This made incremental refreshes report incorrect
statistics.

Make these three result fields optional across Rust, Python, and
TypeScript, and leave them unset for remote SQL refreshes. Remove the
post-refresh row count request. Local refreshes continue to populate all
metrics, and `version` remains the view version observed after
completion. Remote per-operation statistics remain unavailable until the
SQL job adapter can retrieve them.

Expose the forced-rebuild option through the Rust builder,
synchronous/asynchronous Python refresh and job methods, and TypeScript
refresh. Remote `full=true` sends `WITH (full = true)`; the default lets
the server choose its refresh mode. Local source-version pinning and
remote job cancellation are preserved.

Local refresh regression tests respect the `experimental-functions`
build gate, while remote refresh coverage also runs in default builds.

Validation:

- Rust: 20 materialized-view tests and 21 remote SQL tests passed with
`remote,experimental-functions`; `cargo fmt --all` passed.
- Python: rebuilt the extension in the uv-managed environment. The
default build passed 23 tests with 27 feature-dependent skips. The
opt-in build passed 61 tests with 3 expected default-build-only skips,
covering the materialized-view suite and 14 async query-serialization
cases from current main. Python Ruff lint and formatting checks passed.
- TypeScript: default and opt-in native builds and TypeScript
compilation passed. The default build passed 5 tests with 10 expected
feature-dependent skips; the opt-in build passed all 14
materialized-view tests with the default-build-only test skipped.
Lint/format checks and documentation generation passed.
- Repository-wide `ruff check .` reports 48 pre-existing violations in
unchanged `docs/web-tests` files; `ruff check python` passes.

Fixes #4670

<!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-10-27 21:34:01 +08:00
lancedb-gatefixer[bot]andXuanwo 7fca4d666a fix(python): preserve async query search options during serialization (#4659) 
Async `to_query_object()` dropped enabled `fast_search` on every query
type and `postfilter` on plain and full-text queries. Re-executing a
serialized fast search could therefore return unindexed rows.

Copy enabled `fast_search` into the Python `Query` and serialize enabled
plain/FTS postfilter from the Rust request's prefilter setting. Preserve
the existing unset defaults: `None` for these options, with the vector
query's existing `False` postfilter default.

Validation:

- Added 12 serialization cases across plain, FTS, and vector queries,
covering both enabled and unset options, plus two FTS execution
round-trip cases. Before the fix, nine cases failed, including returning
an unindexed row after serialization.
- Rebuilt the Python extension using the uv-managed environment.
- Query tests and existing FTS fast-search tests: 192 passed, 1 slow
test deselected.
- Rust formatting and Python package lint/format checks pass.
Repository-wide `ruff check .` reports 49 pre-existing errors in
unchanged `docs/web-tests` files.

Fixes #4658

<!-- lance-gatekeeper-fix:v1 agent=f828b24871a2dc928668be2de4f85078
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-10-27 16:32:32 +08:00
lancedb-gatefixer[bot]andXuanwo 1edfa6077b fix(python): allow phrase queries without a column (#4653) 
Quoted async full-text queries without `columns` carry an unset phrase
column, but converting them to Python unwrapped it and raised
`PanicException`. Python also required `PhraseQuery.column`, preventing
callers from constructing the equivalent query object.

Make the Python column optional and preserve `Option<String>` in both
Rust conversion directions, leaving column selection to query planning.
Regression tests cover quoted strings and structured queries with
omitted, `None`, and explicit columns, query reuse, JSON serialization,
slop, and document granularity.

Validation:

- Reproduced both missing-column failures before the fix;
explicit-column controls passed.
- Rebuilt the extension with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`.
- `cd python && uv run --no-sync --extra tests pytest
python/tests/test_fts.py
python/tests/test_query.py::test_query_serialization_async -q`: 85
passed.
- `cargo fmt --all -- --check` and `ruff check python` pass; touched
Python files pass formatting checks.
- `cargo clippy --quiet -p lancedb-python --lib --features remote`
passes.
- Root-wide `ruff check .` reports 49 existing findings in unchanged
documentation examples.

Fixes #4651

<!-- lance-gatekeeper-fix:v1 agent=62d36fad70c8717236283b8e97dd6b30
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-10-27 15:53:14 +08:00
Xuanwo 8a4effa06a fix: gate experimental local functions behind an opt-in feature (#4688) 
Published SDK builds currently expose local SQL computed columns and
materialized-view execution that are not ready for release. Add the
opt-in `experimental-functions` Cargo feature to the Rust core and
forward it through the Python and Node bindings; published packages
leave it disabled.

Local declarations and refreshes now return `NotSupported` before
mutating tables or spawning jobs, including prepared materialized-view
creation. Remote Cloud/Enterprise client operations, Python UDF
packaging, ordinary eager `add_columns`, and access to existing data
remain available. Stored metadata and write protections stay active, and
shared server commit helpers remain usable.

CI covers both feature configurations. Existing local execution tests
run with the feature enabled; default-build regressions cover early
rejection, unchanged table versions, existing metadata protections, and
normal eager transforms. API and source-build documentation explain the
opt-in.

Validation: Rust library tests pass with the feature disabled (1,869
tests) and enabled (1,947 tests), plus three default-build integration
regressions. Python and TypeScript bindings were rebuilt and tested in
both configurations, including mocked remote requests; Rust formatting,
Python package lint, TypeScript lint/build, and generated docs were
checked. Repository-wide Ruff still reports 48 pre-existing errors in
unchanged `docs/web-tests` files.
 2026-10-27 14:32:09 +08:00
lancedb-gatefixer[bot]andXuanwo 3352bf49df fix(python): expose optimize compaction options (#3870) 
## Summary
- expose typed compaction options on synchronous and asynchronous table
optimization
- translate validated Python options into native Lance compaction
settings
- document the row-based steady-state limit and first-pass/source-budget
caveats
- carry the Lance beta.15 fragment-exclusion option through the typed
Python API

## Root cause
The native Python optimize binding always constructed default compaction
settings, and the public AsyncTable API offered no way to lower the
fragment row target. Wide-row fragments therefore remained compaction
candidates on every later optimize call, repeatedly rewriting the
accumulated table.

## Limitations
This is an explicit row-based mitigation, not byte-native task sizing or
a hard memory cap. Callers must derive target_rows_per_fragment from
expected row size. Existing oversized fragments are rewritten whole on
the first pass; max_bytes_per_file limits output file size rather than
compaction memory; and source budgets admit whole planned tasks, so a
budget below the first task performs no compaction. Splitting tasks by a
byte budget or returning a planner error requires a Lance-side change.

## Validation
- cargo check --release -p lancedb-python
- cargo clippy --quiet -p lancedb-python
- cargo fmt --all --check
- ruff format --check python tests
- ruff check python tests
- uv run --no-sync pytest python/tests/test_table.py -k
optimize_compaction -q (16 passed)

Fixes #2325

<!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-10-27 03:59:52 +08:00
lancedb-gatefixer[bot]andXuanwo 66357275bb fix: inherit Lance defaults for IVF RQ quantization (#4585) 
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding
Lance's five-bit default. The shared native index creation path also
substituted one bit when the Rust or Node builder left the value unset.

Make Python's bit count optional and forward only explicit values. Use
`RQBuildParams::default()` for an unset native bit count, and omit unset
values when dispatching to pylance for accelerator builds. Update the
Python, Rust, Node, and vector-index documentation.

New indexes with an unset bit count now inherit Lance's default,
currently five bits, across Python, Rust, and Node. Five-bit vector
codes use roughly five times the storage of one-bit codes, plus other
index overhead. This also changes reader compatibility: indexes created
with `num_bits >= 2` use a newer on-disk layout that some older LanceDB
versions cannot read. Upgrade readers to versions that support multi-bit
IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)`
in Python) when sharing indexes with readers limited to one-bit IVF_RQ.

Validation:

- Reproduced six regression failures before the fix: one-bit default
metadata, rejected `None`, and incorrect remote/accelerator forwarding.
- After merging `main` through `b762ec22`, rebuilt both native bindings
against Lance 15.0.0-beta.1. Python validation passed with 106 tests and
15 optional Ollama skips. Coverage includes the full index suite,
omitted and explicit bit counts, reading Lance-created IVF_RQ indexes,
accelerator forwarding, CUDA-unavailability handling, timeout
validation, remote requests, and table/namespace listing behavior.
- Rust formatting and Python package Ruff lint/format checks pass. Node
`pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run
docs`, the full connection suite, and the existing IVF_RQ regression
pass. Generated documentation adds no further changes.

The merge preserves upstream's explicit RaBitQ heading anchor alongside
the `IvfRq()` example and five-bit default/reader-compatibility
guidance.

Repository-wide `ruff check .` still reports 49 existing errors in
unchanged documentation examples under `docs/web-tests`; root formatting
also finds 11 unchanged documentation example files. The Python package
checks pass.

Fixes #4582

<!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-10-26 18:57:25 +08:00
Jack Ye dcfef7c5ab feat(python): open lazy blob files from SQL descriptors (#4668) 
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.

### Python experience

```python
rows = db.execute_query("""
    SELECT f.timestamp, blob_descriptor(v.video) AS video
    FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()

videos = db.open_table("videos")
for row in rows.to_pylist():
    if row["video"] is not None:
        with videos.open_blob_file(row["video"]) as video:
            frame = extract_frame(video, row["timestamp"])
```

The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.

The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:

```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
    if video is not None:
        with video:
            frame = extract_frame(video, row["timestamp"])
```

Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.

For an async table:

```python
async for video in async_videos.open_blob_files(rows["video"]):
    if video is not None:
        with video:
            payload = await video.aread()
```

### Execution

The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.

Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.

Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.

Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
 2026-10-25 02:31:21 -07:00
Jack Ye 9af7757053 fix: execute remote materialized view creation through SQL (#4665) 
Remote materialized-view creation currently expects a REST response
containing a registry job ID, while refresh already submits SQL. Route
both through the SQL query backend and preserve Python's synchronous and
asynchronous return types.

### Query to Job mapping

`SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through
the same backend as `execute_query_async()`. Once submission succeeds,
it retains the returned `Query` and starts one client-side task that
consumes its result reader to completion. `Job::from_query_task()` wraps
that task, its cached outcome, a single ID and a cancellation callback.
Python exposes this as `Job[T]` or `AsyncJob[T]`.

The existing `job_type` property maps SQL-backed handles to the new
`"query"` kind. The shared `id` contains that query's UUID string, so
Python callers can inspect it with `db.describe_query(job.id)` on the
submitting connection. No separate `query_id`, backend enum or `type()`
API is added.

| Handle | `job_type` | `id` | Available lookup |
| --- | --- | --- | --- |
| SQL-backed | `"query"`, known at submission | Connection-scoped query
UUID string | `describe_query(job.id)` |
| Registry-backed | Existing server-reported kind, after `status()` or
`refresh()` | Registry job ID string | Registry job APIs, descriptions
and events |
| Local | `None` | `None` | The current in-process handle |

| Job concept | SQL-backed behavior |
| --- | --- |
| Submission | Await initial SQL submission before returning the handle.
Submission errors propagate immediately. |
| `status()` | Tracks the adapter task: running during result
consumption/finalization, finished after both succeed, failed if either
fails, or cancelled after cancellation. |
| `wait()` | Returns the operation's typed terminal result. One task
consumes the query reader once; repeated waits reuse its cached outcome
without resubmitting SQL or reopening the reader. |
| `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported
cancellation leaves the task running; an accepted request stops the
task. Cancelling a finished operation is a no-op. |
| Registry details | Query-backed handles expose no registry
description, event history, pause/resume or reattachment after a client
restart. `refresh()` updates cached lifecycle state. |

The Job tracks command completion rather than exposing SQL batches. A
wait timeout does not cancel the query. Query IDs remain scoped to the
submitting connection and cannot be used with registry APIs or as
durable resume tokens. Python `describe_query()` accepts both UUID
objects and the string `id` exposed by query-backed jobs. Rust, Python
and TypeScript expose the same classification through their existing
job-type accessors.

### Materialized-view operations

- **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace
and optional auto-refresh settings. After command completion, resolve
the job to `None`. Synchronous creation waits on the job, then opens and
returns the materialized view.
- **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same
adapter. After command completion, read the refreshed table's version
and row count to construct the existing typed refresh result.
Synchronous refresh waits on the same path.

Remote creation requires the SQL service. Execution placement remains a
server decision. REST endpoints are unchanged. The job guide documents
the ID/job-type contract and query-backed limitations.

Regression coverage exercises SQL submission, namespaces, refresh
options, job-type/ID preservation through typed result mapping, repeated
waits, failures and cancellation rejection. No local builds or tests
were run.
 2026-10-25 01:20:13 -07:00
Colin Patrick McCabe 631ea41a63 feat!: return lazy iterators for table and namespace listings (#4599) 
Make table and namespace listings consistent with other resource
listings across Rust, Python, and TypeScript. Callers can iterate over
individual names across all pages, with configurable page sizes and
continuation tokens.

In order to avoid breaking existing Python users, this change adds some
compatibility methods to the returned Python iterators. Code such as:
```
for t in db.list_tables().tables:
    print(t)
```
will continue to work as expected because the iterator has a `tables`
field and a `page_token` field, which function as expected to reveal the
first page of results. In other words, it is never necessary to use the
iterator __as an iterator__. It may be used as if it were an old-style
response object. (As the iterator progresses, the `tables` list may
switch to the second, third, etc. page of results.) Similar
compatibility code allows the async python API to keep working as well.
 2026-10-22 15:41:30 -07:00
adenzhou1350 3d883af674 fix(python): propagate invalid index wait_timeout errors (#4495) 
Invalid `wait_timeout` values currently reach
`Duration::extract(...).unwrap()` in both native index-creation
bindings. A normal Python argument error becomes
`pyo3_runtime.PanicException`, which is not caught by `except
Exception`.

Propagate the extraction error with `map(...).transpose()?` before
constructing the index operation. Valid `None` and `timedelta` behavior
is unchanged.

Tests cover both `create_index` and `create_index_async`, three invalid
types, valid `None`/`timedelta`, and no index created after rejection.

Validation against main `0aa842b`, Python 3.12, uv-locked dependencies,
freshly compiled PyO3 extension for each arm:
- Unmodified source: 6 new regressions fail with `PanicException`; 6
controls pass.
- Fixed source: 12/12 pass; broader index-creation regression selection:
18/18 pass.
- `cargo fmt --all --check`, Ruff check/format, and diff check pass.

The broader selection includes the 12 cases above. No cloud, full-suite,
or performance claim.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
 2026-10-23 03:14:31 +08:00
adenzhou1350 a12aef2a8c fix: return a Python error when querying a closed table (#4455) 
`AsyncTable.query()` unwraps the native closed-table error, so querying
a closed table raises `PanicException` instead of the documented
`RuntimeError`. `AsyncTable.head()` reaches the same path.

Propagate `inner_ref()`'s error through `PyResult` instead. Add
regressions for closed `query()` and `head()`, with `count_rows()`,
repeated close, and normal open-table queries as controls.

Validation against separately built native bindings:

- Before: 2 regression failures (`PanicException`), 2 controls passed.
- After: all 4 regressions passed.
- Existing query tests plus the basic database test: 102 passed, 1
skipped on each arm (Torch is not installed).
- `cargo fmt --all -- --check`, Ruff, and `git diff --check` passed.

The native build used the locked Cargo dependencies with
`remote,lancedb/metrics-otel,pyo3/extension-module`. This is source-tree
native testing, not a complete wheel-packaging or cloud-service test.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
 2026-10-23 02:44:48 +08:00
adenzhou1350 b9ef048abb fix: validate Python read consistency intervals (#4454) 
Negative `read_consistency_interval` values currently reach Rust's
panicking `Duration::from_secs_f64` conversion. For example:

```python
from datetime import timedelta
import lancedb

lancedb.connect("./db", read_consistency_interval=timedelta(seconds=-1))
```

Validate the interval with `Duration::try_from_secs_f64` at the three
native connection entry points and return `ValueError` before
constructing the connection or async future. `None`, zero and positive
intervals retain their behavior. The raw native binding also rejects
NaN, infinity and overflow; the public Python API still accepts
`timedelta`, not floats.

Tests cover public sync/async connections, native namespace connections,
real Python namespace clients, the three raw binding entry points and
valid-value controls.

Validation on Python 3.12 / Rust 1.97:

- Same 26 regression cases against separately built native baseline and
candidate: baseline 22 failed / 4 passed; candidate 26 passed.
- Existing basic connection and background-loop regression tests pass.
- `cargo fmt --all -- --check`, Ruff check/format and `git diff --check`
pass.

The native builds used the unchanged repository `ci` profile and
Cargo.lock with `--no-default-features --features
remote,lancedb/metrics-otel,pyo3/extension-module`. Python runtime and
focused test dependency versions came from `python/uv.lock`. Source-tree
tests used a minimal distribution-version metadata fixture derived from
the unchanged Cargo/pyproject manifests; both imported module origins
and native versions were checked in fresh processes. This is not a
complete Maturin packaging, cloud backend or full Python suite result.
No GPU was used.

AI-assisted diagnosis, implementation and validation.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
 2026-10-23 02:44:34 +08:00
LeonSGP 1e313d3c0d fix(python): parse decimal exponent literals exactly (#4449) 
Decimal literals such as Decimal("1E-7") and Decimal("1E+3") currently
raise ValueError while building an expression. Parse the coefficient and
exponent exactly, and normalize representable negative scales to zero so
both typed filters and predicate.to_sql() queries preserve the value.
Checked arithmetic bounds the Decimal128 coefficient to 38 digits,
preserves existing scale limits, and rejects nonfinite or overflowing
values. This does not add Decimal256 literal support.

Added 21 cases covering actual Arrow-backed table queries, SQL queries,
zero, signed exponents, precision boundaries, and errors. Both Python
3.11 and 3.13 expression suites pass 123 tests; the baseline fails 10 of
the new cases. Python binding Rust tests pass 13; official make doctest
passes 77 with 6 skips; Ruff, Rustfmt, hooks, Pyright and
workspace/tests/all-features Clippy with -D warnings pass. A fresh
environment restored from the original dependency freeze passes the full
required suite: 1528 passed, 37 skipped, 68 deselected (143.51s); while
optional Torch suites have the same baseline remote mock-server ordering
failure.

Tested against f4544628; observed official main d864e719 contains one
Catalog API commit without expr or Cargo.lock changes. AI assistance:
gpt-6.1-sol assisted implementation and local validation; all reported
checks were actually executed.
 2026-10-23 02:43:50 +08:00
Jack Ye f9680806c9 feat(python): expose materialized view auto-refresh creation options (#4621) 
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.

```python
view = db.create_materialized_view(
    "recent_orders", "orders",
    auto_refresh_enabled=True,
    auto_refresh_interval_seconds=300,
)
```

## Architecture and workflow

- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.

Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.

Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
 2026-10-21 23:47:09 -07:00
Jack Ye 1564ec10c8 feat: add Bootstrap split strategy for sampling with replacement (#4597) 
## Context

`StreamingDataset`'s shipped split strategies (`split_random`,
`split_sequential`, `split_hash`, `split_calculated`) all share one
trait: every row in the base table is assigned to exactly one split,
exactly once. They only differ in *how* that one-to-one assignment is
chosen -- none of them can express "give me a resampled stream of N
rows, allowing repeats."

That's a distinct, common need in ML training: **the bootstrap** -- the
statistical technique (Efron, 1979) of estimating a dataset's properties
by resampling it *with replacement*. It's the same sense scikit-learn's
bagging classifiers mean by `bootstrap=True`. `SplitStrategy::Bootstrap`
adds exactly this to the data loader: sample rows uniformly with
replacement from the (optionally filtered) base table, independently per
split.

## Main use cases

- **Bagging / ensemble training** (e.g. random forests): each split
trains on an independent bootstrap resample of the training set.
- **Oversampling a rare class**: filter to the minority class, then
sample it up to whatever count balances it against the majority, instead
of exhausting its small set once per epoch.
- **Fixed-size training steps independent of dataset size**: want
exactly N batches per epoch regardless of whether the table has 50 rows
or 50 million? Set `num_samples` directly instead of it being implicitly
`len(dataset)`.
- **Weighted sampling without a weight column**: filter to "eligible"
rows (e.g. a windowed-sampling workload filtering to valid window-start
positions), then bootstrap the filtered set -- an item with more
eligible rows is naturally drawn proportionally more often, with no
explicit per-row weight needed.

## Example (an actual run, not a hypothetical one)

A 5-row base table (`row_id` 0..4), sampled via `Bootstrap {
num_samples: 8, sizes: Fixed(2), seed: Some(42) }`:

```
row_id  split_id
     0         0
     2         0
     4         0
     4         0      <- row 4 repeated within split 0
     4         1      <- row 4 also drawn in split 1 (independent draw)
     4         1
     1         1
     2         1
```

8 output rows from a 5-row table -- not 5. Row 4 drawn 4 times total,
row 3 never drawn. That's with-replacement sampling actually behaving
like with-replacement sampling, not an even round-robin dressed up with
a new name.

## What changed

Adds `SplitStrategy::Bootstrap` (uniform sampling with replacement,
optionally filtered, independently seeded per split -- a given split's
content doesn't depend on how many other splits there are, the same
elastic-determinism property documented for the loader's other
strategies). No new `PermutationBuilder` Rust method was needed:
`with_split_strategy` already accepts any `SplitStrategy` value
generically. Python binding:
`PermutationBuilder.bootstrap_sample(num_samples, *, ratios=None,
counts=None, fixed=None, seed=None, split_names=None)`, mirroring
`split_random`'s existing shape.

Also adds a generic `split_strategy` parameter to `StreamingDataset`'s
constructor (a callable taking/returning a `PermutationBuilder`), since
it previously could only reach `split_random`/`split_sequential` even
though `split_hash`/`split_calculated` already shipped. This unblocks
`Bootstrap` and any future strategy without further constructor changes.
 2026-10-21 23:18:30 -07:00
Jack Ye a4544c5472 feat: configure automatic refresh when declaring computed columns (#4607) 
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.

Python users can configure refresh when declaring a column:

```python
table.add_columns(
    computed={"total": "price * quantity"},
    auto_refresh_enabled=True,
    auto_refresh_interval_seconds=300,
)
```

Omit the interval to let the server determine when to refresh:

```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```

The async API accepts the same options:

```python
await async_table.add_columns(
    computed={"total": "price * quantity"},
    auto_refresh_enabled=True,
    auto_refresh_interval_seconds=300,
)
```

For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:

```python
table.add_columns(
    {"features": function_application},
    auto_refresh_enabled=False,
    auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```

Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.

Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
 2026-10-21 20:27:11 -07:00
Dan RammerandClaude Opus 5.5 92825f6f13 refactor!: rename MemWAL APIs to RTE (realtime engine) (#4598) 
## What

Renames the user-facing MemWAL surface to **RTE** (realtime engine)
across Rust, Python, TypeScript, and Java, including the REST wire
format. This follows the LSM → MemWAL rename in #4075.

| Before | After |
|---|---|
| `MemWalWriteSpec` / `MemWalStats` | `RTEWriteSpec` / `RTEStats` |
| `enable_memwal` / `disable_memwal` / `get_memwal_spec` | `enable_rte`
/ `disable_rte` / `get_rte_spec` |
| `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | `flush_rte`
/ `compact_rte` / `checkpoint_rte` |
| `get_memwal_stats` / `close_memwal_writers` | `get_rte_stats` /
`close_rte_writers` |
| `use_memwal` (query, merge insert) | `use_rte` |
| TS/Java `enableMemWal`, `useMemWal`, ... | `enableRTE`, `useRTE`, ...
(napi `js_name` set explicitly) |
| Java `LanceDbTableMemWal` | `LanceDbTableRTE` |
| `table/memwal_stats.rs` | `table/rte_stats.rs` |

Docs and comments now say "realtime engine (RTE)". Text about Lance's
underlying mechanism keeps "MemWAL" ("Lance's MemWAL", "MemWAL index").

**Left alone:** Lance-owned names (`MemWalIndexDetails`,
`DatasetMemWalExt`, `InitializeMemWal*`, `mem_wal` paths), internal
`Lsm*` names and modules, and the `does not support with_row_id` refusal
text that the hybrid fallback matches on.

## Compatibility

Breaking, for an experimental API. The wire format changes, so this must
roll out together with the matching sophon change (lancedb/sophon#8870):

- Routes: `/enable_memwal/` → `/enable_rte/`, `/disable_memwal/` →
`/disable_rte/`, `/get_memwal_spec/` → `/get_rte_spec/`,
`/get_memwal_stats/` → `/get_rte_stats/`, `/flush_memwal/` →
`/flush_rte/`, `/compact_memwal/` → `/compact_rte/`
- Response keys: `memwal_spec` → `rte_spec`, `memwal_stats` →
`rte_stats`
- Request field: `use_memwal` → `use_rte` on query and merge-insert
bodies

The SQL keyword becomes `ALTER TABLE ... SET RTE` server-side. The
guides under `docs/web` will be updated separately.

## Validation

- Rust: `cargo clippy --features remote --tests --examples` clean;
`cargo test --features remote -p lancedb --lib`: 1533 passed
- Java: `mvn test`: 37 passed (Spotless applied with JDK 11)
- Python: `ruff format` / `ruff check` clean; targeted tests
(`test_rte_spec`, `test_merge_insert_lsm`, `test_remote_db`,
`test_wal_hybrid`, `test_query`, `test_permutation`, `test_table`): 708
passed
- TypeScript: `pnpm build`; `table`, `query` tests: 606 passed; `biome
check` clean; `docs/src/js` regenerated with `pnpm run docs`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-20 11:58:57 -05:00
Dan RammerandClaude Opus 5 fe34eb3549 refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075) 
## What

Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.

### 1. MemWAL APIs drop "LSM"

The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.

| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |

TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.

Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.

### 2. Stats use SSTable and table shard

Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.

**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.

**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.

| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |

Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.

`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.

## Compatibility

Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:

- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies

## Validation

- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-10-16 13:33:57 -05:00
lancedb-gatefixer[bot] a1e4dc9f91 fix(python): align BlobFile seek and read behavior (#4290) 2026-10-14 10:41:54 -07:00
lancedb-gatefixer[bot] 3336bdfbec fix(remote): bound concurrent blob reads across handles (#4311) 2026-10-14 10:27:25 -07:00
Colin Patrick McCabe 6cc4919c3c feat: add an authorization API (#4363) 
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
 2026-10-13 19:59:25 -07:00
lancedb-gatefixer[bot] 0212cfd3d6 fix(python): normalize blob row ID inputs and errors (#4293) 2026-10-13 14:49:51 -07:00
Yang Cen 0bceaaf3f0 feat(search): expose search effort (#4520) 
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.

The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.

Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.

BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.

Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.

Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):

- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.

All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.

Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
 2026-10-12 19:32:38 +08:00
Jack Ye f305a05fd4 fix: preserve adaptive IVF probe defaults (#4168) 
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.

The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
 2026-10-09 12:45:02 -07:00
lancedb-gatefixer[bot] 0acb61471c fix(python): route async remote blob reads through server (#4306) 
## Cause

Async remote pandas conversion could call `AsyncTable.to_lance()`, which
opened the server's table storage URI in the client process. This
bypassed server query and authorization APIs. The first repair routed
blob reads through the server, but a query and subsequent blob fetch
could resolve different table snapshots after replacement or compaction.

## Fix

- Reject `AsyncTable.to_lance()` for remote tables before requesting a
table URI.
- Convert remote nonblob and blob description results through the server
Arrow path.
- Reject automatic remote byte and lazy blob materialization until the
server can bind the query and fetches to one verified table identity and
version. The default lazy mode now raises `NotImplementedError` when a
blob column is projected; callers can request
`blob_mode="descriptions"`.

## Validation

- Mocked remote tests cover direct storage rejection, description
results, byte and lazy rejection before any blob fetch, and nonblob
projections.
- Focused local and remote blob tests passed (19 tests). `ruff check .`,
`ruff format .`, and pyright passed.

Fixes #4272

<!-- lance-gatekeeper-fix:v1 agent=35937e6e1d968f62eea59adcc56c84e9
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-08 19:24:29 -07:00
lancedb-gatefixer[bot] bcf5ee263f fix(python): preserve code tokenizer filter defaults (#4370) 
Python's FTS bindings always forwarded `stem=True` and
`remove_stop_words=True`, overriding Lance's code tokenizer defaults. As
a result, code indexes stemmed identifiers and discarded stop words, and
`lancedb.tokenize` reproduced the same behavior.

Use `None` for omitted filter options and only apply explicit booleans
in the Rust bindings. This preserves tokenizer defaults across local and
remote index creation, the deprecated `create_fts_index` helpers, and
standalone tokenization. Update the signatures and documentation to
describe the inherited defaults.

Regression tests cover stored local settings, token output, stop-word
searches, remote request payloads, explicit overrides, and the existing
simple tokenizer behavior. The regression suite produced 16 failures
before the fix and passes all 28 cases after it.

Validation:

- Rebuilt the Python extension using the uv-managed development
environment.
- Full Python FTS and index test files plus relevant remote index tests:
105 passed.
- `cargo clippy --quiet --features remote --tests --examples -p
lancedb-python`
- `cargo fmt --all`, Ruff format/lint with CI's pinned version 0.9.9,
and Pyright (zero errors).

Fixes #4365

<!-- lance-gatekeeper-fix:v1 agent=9c1db2d3fc25c49c5b340f8debf43a0f
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-08 18:53:16 -07:00