mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-21 20:45:57 +00:00
86835da5dbd980b9f70ce435f5cdc559b0cbea5d
2998
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
86835da5db | Bump version: 0.39.0-beta.10 → 0.40.0-beta.0 | ||
|
|
a4f66afc69 |
feat(rust): add zonemap index builder (#4199)
Lance supports ZoneMap scalar indexes, but the LanceDB Rust API did not
expose a first-class way to request one through `Table::create_index`.
Users had builders for the other scalar index families, while ZoneMap
was missing from the public `Index` model and remote create-index
serialization. This PR adds ZoneMap as a supported scalar index option
in LanceDB.
This was accomplished with the following changes:
- Added `ZoneMapIndexBuilder` in `rust/lancedb/src/index/scalar.rs`.
- Added `Index::ZoneMap` and `IndexType::ZoneMap`, including
display/from-string aliases for `ZONEMAP` and `ZONE_MAP`.
- Mapped local index creation to
`ScalarIndexParams::for_builtin(BuiltinIndexType::ZoneMap)` and Lance
`IndexType::ZoneMap` in `rust/lancedb/src/table/create_index.rs`.
- Serialized remote create-index requests as `index_type: "ZONEMAP"` in
`rust/lancedb/src/remote/table.rs`.
- Added coverage for both local ZoneMap index creation and remote
request serialization.
Example:
```rust
table
.create_index(&["my_column"], Index::ZoneMap(Default::default()))
.execute()
.await?;
```
### Testing
Added `test_create_zonemap_index` for local index creation and extended
the remote request body test matrix for `ZONEMAP`.
|
||
|
|
6a07f88980 |
feat: add remote catalogs and Python and TypeScript bindings (#4195)
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
|
99ed25f753 |
fix: return InvalidTableName instead of panicking in open_table/create_table (#4192)
Passing an invalid table name to `open_table` or `create_table` panics
instead of returning an error:
thread '...' panicked at rust/lancedb/src/database/listing.rs:1155:62:
called `Result::unwrap()` on an `Err` value: InvalidTableName { name:
"my table", ... }
Both call sites build the table URI with
`request.location.clone().unwrap_or_else(||
self.table_uri(&request.name).unwrap())`, and `table_uri` is the
function that validates the name — so every rejected name (empty,
spaces, slashes, non-ASCII) hits the inner `unwrap`.
`Error::InvalidTableName` clearly is the intended contract here: the
variant exists for exactly this, and the Python binding maps it to
`ValueError`.
Replaced the closure with a `match` that propagates the validation
error; behavior with an explicit `location` is unchanged (the name is
not validated on that path, as before). Added tests asserting
`InvalidTableName` for `create_table` and `open_table` over a set of
rejected names — both panic without the fix. Full `cargo test -p lancedb
--lib --features remote`: 1214 passed; clippy/fmt clean; `cargo check
--workspace --all-targets` clean.
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
|
f3ef21b8ca |
feat: use oauth2 crate for OAuth with configurable client auth (#4181)
Stacked on #4173 (`jack/restore-oidc-flows`, base branch mirrored to this repo so the diff shows only this change); context from review: https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100. Rebase to `main` once #4173 and #4179 merge. ## What moved to the `oauth2` crate (5.0, no default features) - Authorization URL generation and CSRF state (`authorize_url`, `CsrfToken`) - PKCE S256 challenge/verifier generation and code exchange - Client-credentials, authorization-code, refresh-token, and device-code grant request construction - Device authorization request and the device token polling loop (`authorization_pending`, `slow_down` +5s, expiry deadline, denial, network backoff capped at 10s) - Standard success/error response parsing (`RequestTokenError`) - Token-endpoint client authentication and standards-compliant parameter encoding (RFC 6749 2.3.1 Basic encoding) LanceDB keeps ownership of OIDC discovery (compared `openidconnect`: no measurable win for our 3-field metadata + strict validation, at real dependency cost), HTTPS-or-loopback endpoint enforcement, the loopback callback server, browser/stderr prompts, token caching and refresh orchestration, and the dedicated hardened Azure IMDS source, which is unchanged. ## Client authentication methods New `ClientAuthMethod` enum (`none` | `client_secret_basic` | `client_secret_post`), exposed in Rust, Python, and Node. Unset resolves to `client_secret_basic` when a secret is present (RFC 6749 2.3.1 recommendation and the normal Okta confidential-app default, so a default Okta app works without weakening its configuration) and to `none` for public clients (PKCE/device). Explicit `none` with a secret, or basic/post without one, is rejected. The method applies to client credentials, code exchange, refresh, and device requests. Deliberate behavior change: confidential clients previously always sent the secret in the POST body; they now default to Basic (Keycloak accepts both). No `audience`/`resource` parameters were added: the supported target is an Okta custom authorization server with the API audience configured server-side, so client-provided audience parameters are unnecessary; `add_extra_param` support exists if a concrete provider contract ever needs them. ## Device polling behavior changes (deliberate, tested) - The first token poll now happens immediately rather than after one interval (RFC 8628 allows both). - Transient failures (HTTP 429, 5xx, `temporarily_unavailable`, network errors) now retry with exponential backoff capped at 10s instead of retrying at the fixed interval; polling never spins faster than once per second even if a server reports a zero interval. ## Security and compatibility - Issuer and discovered endpoints (and device verification URIs) still require HTTPS except explicit loopback HTTP, enforced before any crate URL type is built - Token HTTP client keeps the hardened redirect policy that refuses insecure redirect targets; regression test added - Errors never embed raw response bodies (avoids leaking tokens through parse failures); all credential types stay redacted in Debug - Transient conditions (429/5xx/`temporarily_unavailable`) remain retryable in device polling and hard errors elsewhere; refresh keeps rotation and reauthentication semantics - Existing public APIs stay source-compatible except the added `OAuthConfig.client_auth_method` field ## Tests Rust: client-auth methods across code exchange/refresh/client-credentials/device (none/basic/post), auth-method resolution and validation, transient device retries, denial/expiry, redirect rejection, malformed-response leak check, PKCE URL assertions, redaction. Python and Node: enum values, conversion, unknown-method errors, config defaults. Manual Okta validation recipe (no automated Okta credentials): create a custom authorization server with an API audience, one confidential web app (Basic) for authorization-code, one native app (PKCE, no secret), one native device app; point `issuer_url` at the custom server, set `client_auth_method` only for the POST-required case; verify token acquisition, refresh after expiry, and `x-lancedb-credential-type: oidc` against a LanceDB deployment. Never commit tenant URLs or secrets. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
ed100ccc31 | Bump version: 0.39.0-beta.9 → 0.39.0-beta.10 | ||
|
|
161f81276d |
chore: update lance dependency to v13.0.0-beta.3 (#4197)
Update the Rust workspace and Java Lance dependencies to [v13.0.0-beta.3](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.3), which includes the nullable-list encoding fix in lance-format/lance#9268. Also resolve existing strict Clippy diagnostics in crate-private OAuth modules and the Python token-cache conversion, without changing effective visibility. Validated the rebuilt Python SDK with nullable and nested lists: Match/Phrase searches preserve document coordinates before and after appending data and optimizing the table. |
||
|
|
7991e27f35 |
fix(node): prevent OAuth tests from launching browsers (#4196)
OAuth tests set `LANCEDB_OAUTH_BROWSER` inside Jest's sandbox, which does not update the process environment read by Rust. As a result, the native login flow launches a real browser; the [failing macOS main job](https://github.com/lancedb/lancedb/actions/runs/35067074667/job/104699667840) ends by terminating an orphaned Safari process. Set the no-op browser helper while loading Jest configuration, before creating sandboxes and workers, so both local and CI tests inherit it. Check the inherited process environment before OAuth login to catch regressions without opening a browser. Windows uses a no-op command fixture. Test deadlines and worker counts are unchanged. A negative control restoring the sandbox-only assignment fails in the new pre-login check. The full macOS test suite passes with the standard test launcher; the Windows helper has not been executed locally. The [first hosted macOS run](https://github.com/lancedb/lancedb/actions/runs/35071525843/job/104713928139) passes all 843 tests (5 skipped), with no Safari process in the job log. Further normal runs are needed to establish sustained stability. |
||
|
|
34ab278a5b |
fix: expose public FTS paths in remote index listings (#4194)
Remote `list_indices()` exposes physical FTS paths such as `docs.item.content`, while index creation, queries, and native table listings use `docs.content`. Normalize FTS columns to the public path after parsing the server response, using the same field-ID-based conversion as native tables. Keep the physical paths on the wire: existing clients need them to resolve the Arrow schema. Cover both legacy responses that fetch index statistics and enriched responses that already include the index type. |
||
|
|
3b37ea2c7a |
feat: represent registered Functions by OCI image identity (#4176)
Function versions identify independent Function objects and their numeric revisions. Rust and Python expose the object ID, location, canonical decimal version, metadata, and availability separately from the OCI image digest. Computed-column applications carry the complete object reference, so existing bindings retain their identity after a name is removed and reused. Source authoring keeps the existing create_function/create_function_async, job wait, and column-binding APIs. The server coordinates baking followed by registration; users do not have to manage manifest digests to create a Function. The previous stored Function representation is intentionally unsupported. Existing contract tests and shared wire fixtures are migrated to the new model. This SDK change accompanies the final integration layer https://github.com/lancedb/sophon/pull/7887 in the Sophon stack: https://github.com/lancedb/sophon/pull/7885 → https://github.com/lancedb/sophon/pull/7886 → https://github.com/lancedb/sophon/pull/7887. A metadata-only revision can retain the same executable image; Function version numbers must not be used as image cache keys. --------- Co-authored-by: lancedb automation <robot@lancedb.com> Co-authored-by: Yang Cen <bubble-cal@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
f7579d2aae |
fix(python): correct pylance test extra pin (#4051)
Closes #4045 ## What changed - replace the unpublished pylance 9.0.0rc1 test-extra pin with the published 9.0.0 release - restore dependency resolution for editable installs using the tests extra ## Validation - downloaded pylance==9.0.0 from PyPI with pip --no-deps - parsed python/pyproject.toml with tomllib and verified the tests extra - git diff --check --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
be3215a6ae |
feat(nodejs): add JSON field helper (#4082)
## Issue Fixes #4063 ## Background The Node.js SDK currently requires callers to know the Arrow extension metadata needed to represent JSON fields. This makes a common LanceDB schema type unnecessarily verbose and easy to get wrong. ## Changes - Add `makeJsonField(name, nullable = true)` to create a UTF-8 Arrow field with the `arrow.json` extension metadata. - Re-export the helper from the public Node.js entry point. - Add coverage for the default nullable behavior, explicit non-nullable fields, and the extension metadata. - Add the generated TypeDoc function page and public globals entry, including a usage example. ## Implementation The helper uses the existing Apache Arrow `Field` type and sets `ARROW:extension:name` to `arrow.json`, matching the metadata convention already used by LanceDB. ## Compatibility This is an additive Node.js API. Existing schema construction and Arrow behavior are unchanged. ## Verification - `pnpm test -- arrow.test.ts --runInBand` — 236 tests passed. - `pnpm exec biome ci lancedb/arrow.ts lancedb/index.ts __test__/arrow.test.ts` — passed. - `git diff --check` — passed. ## Not run / known limitations - `pnpm build` and `pnpm run docs` were attempted after expanding the checkout. Both are blocked locally by the native binding build/type declarations: Cargo did not complete, and TypeDoc reported the missing generated `nodejs/lancedb/native` module. The docs files were generated from the updated TypeScript comments; full build and docs validation are left to CI. |
||
|
|
3a1d3be256 |
feat(oidc): support resource and audience (#4193)
Support configuring resource and audience for OAuth authorization, token exchange, and refresh requests. |
||
|
|
ba693ae43d |
fix: do not allow . and .. as table names (#4191)
Do not allow . and .. as table names. They are incompatible with the local filesystem, and confusing in cases where they are supported. |
||
|
|
1d2a5d084b |
fix: accept all-null batches and plain JSON strings for json columns (#4067)
Two ways of writing to a `json` column failed or silently corrupted data. **All-null batches were rejected.** `add()` refused a batch whose values for a `json` column were all null, while every plain Arrow type accepted the same batch. This bites row-at-a-time inserts hardest: a one-row batch with no value for an optional column is trivially all-null, so most such writes failed. pyarrow infers `null` as the column's type, and the write path had no handling for it — casting to the table's type dropped the field metadata that identifies the column as `lance.json`, so lance rejected the batch (`` `val` should have type json but type was large_binary ``). A null-typed input column now becomes typed nulls matching the table's field exactly, metadata included. **Unlabelled JSON text was stored raw.** JSON supplied as plain strings (what pyarrow infers for a column of `str`) was cast to the column's `LargeBinary` storage type and relabelled `lance.json`, putting unparsed text where JSONB was expected. Reads returned the text unnormalized and `json_extract` failed with `InvalidJsonb`. Lance-core does the JSONB encoding, but only for input labelled `arrow.json`, so string input is now labelled rather than cast — at the top level and inside structs. Both fixes are in the shared Rust write path, so they apply to any binding, including hand-built Arrow tables that never pass through Python's list-of-dicts type inference. `_align_field` gets the same JSON-string fix for the legacy Python `_sanitize_data` path, which `on_bad_vectors` and embedding functions still route through. The blob v2 half of the issue landed separately in #4065, which added a `DataType::Null` arm to blob coercion. This PR keeps that implementation and adds end-to-end add-path coverage for it. The tests from #4066 are included here and pass, so that PR's Python-layer inference changes are no longer needed to close the issue. Fixes #3759 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
95055c4c54 | Bump version: 0.39.0-beta.8 → 0.39.0-beta.9 | ||
|
|
3fca33fcb5 |
fix: shut down the shared Tokio runtime on interpreter exit (#4175)
Short-lived Python processes using this client can occasionally crash with SIGABRT during interpreter shutdown, even after every operation they ran completed successfully. The cause is the shared Tokio runtime backing every async call: it's never told to shut down at normal process exit, only reset (and deliberately leaked) on `fork()`. Its worker threads keep running, uncoordinated with the interpreter, until the process actually ends, and if one is mid-task exactly as `Py_Finalize` starts tearing down interpreter state, it can panic on state that's already gone. That panic happens on a background thread with no PyO3-wrapped call frame to catch it, so Rust aborts the whole process instead of just failing that one call. This PR gives the runtime a coordinated, bounded shutdown by registering a Python `atexit` callback that runs while the interpreter is still fully valid. Getting the exit lifecycle right took a few rounds of review. Earlier versions freed the runtime as soon as `Arc::strong_count` looked low, but that's the wrong signal — it reflects who currently holds a reference, not who's logically still in flight. That mistake showed up three ways: a caller could dereference memory already freed out from under it; an install already in progress could finish invisibly after `shutdown()` had already decided there was nothing to do; and a spawned task could end up as the final owner of the `Runtime`, so completing it dropped the runtime from inside one of its own worker threads, which Tokio itself forbids and panics on (this reproduced unprompted in this branch's own test suite). Fixing all three meant replacing reference-count-based tracking with an explicit counter of in-flight top-level calls that `shutdown()` waits on directly. This was accomplished with the following changes: - The runtime lives in an `ArcSwapOption<Tagged>`, where `Tagged` pairs the `Runtime` with the fork generation it was built in. - An `OUTSTANDING` counter, incremented before a top-level `spawn`/`spawn_blocking`/`block_on` call does anything else and decremented only once it has truly finished (via an `OutstandingGuard` token that carries no reference to the runtime), is what `shutdown()` waits on — not `Arc::strong_count` or whether the slot looks empty. This closes the install-race and makes it impossible for a task's own completion to be the runtime's final drop. - Once `shutdown()`'s bound elapses, it stops waiting and attempts retirement anyway, rather than returning with the runtime and its workers left fully alive. - `spawn`/`spawn_blocking` use `Handle::try_current()` to pin any nested spawn (`future_into_py` spawns a task that itself spawns a second one for the real work) to whichever runtime is already executing it, so a reclaim landing between the two calls can't split one logical operation across two different runtime instances. - The fork-child handler now only bumps a bare `GENERATION` counter — no `ArcSwapOption` call of any kind from that context, since `swap`/`compare_and_swap` do real reader-reconciliation work (thread-local state, potentially an allocation) that isn't safe in a forked child. `get_runtime()` compares its installed runtime's generation against the live counter from ordinary context and rebuilds on a mismatch. - Registered `shutdown_runtime` as a Python `atexit` callback in the `_lancedb` module init, running with the GIL released (`Python::detach`) since the bounded wait could otherwise deadlock against any in-flight task that itself needs the GIL. ### Testing - Unit tests in `runtime.rs` cover: shutdown with no runtime created, shutdown after use and lazy rebuild afterward, calling shutdown twice in a row, a concurrent stress test racing many threads against shutdown, a nested-spawn test reproducing `future_into_py`'s own spawn-within-a-spawn shape under concurrent shutdown, a test confirming a top-level task in flight survives a concurrent shutdown reclaim, and a test forcing the install-vs-shutdown race directly. - Built the wheel and ran a concurrent reproducer (many threads hammering the client while `atexit` fires) over 100 times with no hangs or crashes, plus a 30-second-join variant and repeated runs of a short-lived process confirming clean exits with no added latency. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
|
|
2d4622491e |
fix: keep computed-column freshness across branches and clones (#4188)
A branch or shallow clone recomputed every computed column on its first refresh: the input signature carried each data file's raw base id, which the clone commit rewrites, and the signature sidecar was looked up under the branch's own tree, where nothing was ever written. A file is now identified by where its store resolves it, so the source's own root and a clone's registered reference to that root sign the same while different bases stay distinct; compaction products are matched the same way. Sidecars live in one `_computed/` at the table root shared by main and its branches; a shallow clone reads through the base it was cloned from and keeps a copy. Pruning collects references across every branch, and the staleness walk treats the versions a branch does not hold as unknown. Stamps written before this change no longer match, so the first refresh after upgrading recomputes once; a deep clone, which copies no sidecar, still starts over. |
||
|
|
2f88b71c21 |
feat: add persistent OAuth token cache and session APIs (#4182)
Stacked on #4173 (diff includes it until that merges; will rebase after). Addresses the token-cache part of [Colin's review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100). Adds an explicit, opt-in persistent OAuth token cache shared by Rust, Python, and Node clients, plus `login` / `status` / `logout` session APIs, so short-lived processes (CLIs, scripts, notebooks) reuse one session instead of restarting a browser or device flow on every start. - **Opt-in and minimal**: existing callers stay memory-only and lazy. Only refresh tokens are persisted (never access tokens, never client secrets), so there are no local token-expiry decisions to get wrong when clocks move. Each process start performs one silent refresh grant. - **Hardened file backend**: private directory (`0700`), per-record files (`0600`), owner validation, symlink rejection, and atomic `rename` replacement. Corrupt, truncated, unknown-version, or permission-invalid records fail with actionable errors naming the file. Native keyring backends were evaluated (keyring crate routes Linux through D-Bus/zbus: heavy deps, headless/CI flakiness) and are deferred; the file store is the explicit opt-in, not a downgrade from a keyring. - **Cache key**: SHA-256 of the canonical identity (issuer, client ID, sorted/de-duplicated scopes, flow, public/confidential), so no secret appears in a filename and distinct identities never collide. Versioned record schema (`version: 1`). One record per identity: last login wins, documented. - **Cross-process rotation locking**: per-key `fs4` file lock (`flock` / `LockFileEx`) around the refresh critical section — acquire, reread the durable record, refresh exactly once, atomically store the rotated refresh token, release. The OS releases locks on process death, so crashes cannot strand stale locks. Only confirmed `invalid_grant`/`invalid_token` deletes a record and reauthenticates; transport, 5xx, 429, and parse failures retain it. - **Session APIs**: `OAuthSession::login/status/logout` in Rust, `lancedb.remote.OAuthSession` (async) in Python, `OAuthSession` class in Node. `status` returns non-secret metadata only. `logout` removes only the local credential — provider revocation (RFC 7009) is a deliberate follow-up, and local logout never terminates browser SSO. Azure managed identity is rejected for persistence (machine identity stays in memory); client credentials have nothing refreshable to persist and stay memory-only. - No CLI binary exists in this repo, so this ships library APIs plus doc examples in all three languages. Tests: Rust unit + mock-IdP integration (cache-key canonicalization/separation, record versioning/corruption/truncation/symlink/owner/perms, lock serialization + release, two concurrent providers proving no `invalid_grant` and correct rotation, transient-failure retention, `invalid_grant` delete + reauthenticate, login/status/logout lifecycle, client-credentials no-op, IMDS rejection, secret redaction); Python lifecycle + a true two-subprocess cross-process reuse test (second process refreshes once, never hits the device endpoint); Node lifecycle + device-flow login test. Local builds were skipped in development; CI validates all bindings. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
575286922b |
fix: redact authentication headers from debug logs (#4179)
Prevent bearer tokens, API keys, cookies, and proxy credentials from appearing in request debug output through a reusable `redact_sensitive_headers` utility. The utility is applied after default/configured header construction and dynamic header merging, so it also covers retry requests. Invalid dynamic-header values are no longer echoed, and regression coverage verifies that credentials are redacted while ordinary diagnostic headers remain visible. Extracted from the OAuth discussion in #4173, following [Colin's review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100). |
||
|
|
09e5418943 |
build(deps): bump the rust-minor-patch group across 1 directory with 5 updates (#4178)
Bumps the rust-minor-patch group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [uuid](https://github.com/uuid-rs/uuid) | `1.26.0` | `1.26.1` | | [serde_with](https://github.com/jonasbb/serde_with) | `3.22.0` | `3.23.0` | | [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.4.8` | `1.6.3` | | [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.3` | `3.6.5` | | [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.1` | `2.4.2` | Updates `uuid` from 1.26.0 to 1.26.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/uuid-rs/uuid/releases">uuid's releases</a>.</em></p> <blockquote> <h2>v1.26.1</h2> <h2>What's Changed</h2> <ul> <li>Seat the v7 counter below the version nibble by <a href="https://github.com/lenamonj"><code>@lenamonj</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/907">uuid-rs/uuid#907</a></li> <li>Don't panic in overflowing Timestamp to SystemTime conversion by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/909">uuid-rs/uuid#909</a></li> <li>Prepare for 1.26.1 release by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/910">uuid-rs/uuid#910</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/lenamonj"><code>@lenamonj</code></a> made their first contribution in <a href="https://redirect.github.com/uuid-rs/uuid/pull/907">uuid-rs/uuid#907</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1">https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/uuid-rs/uuid/commit/9f927126c89892ddfed6cd2f92df16852f3f9aa6"><code>9f92712</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/910">#910</a> from uuid-rs/cargo/v1.26.1</li> <li><a href="https://github.com/uuid-rs/uuid/commit/d4df8f0cd9f461b4ef493254420052ffa5ce6277"><code>d4df8f0</code></a> prepare for 1.26.1 release</li> <li><a href="https://github.com/uuid-rs/uuid/commit/5613f2357c1fc06afc5fffd98e96da2ccf25a608"><code>5613f23</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/909">#909</a> from uuid-rs/fix/ts-conversion-overflow</li> <li><a href="https://github.com/uuid-rs/uuid/commit/fda00eba938383d242bad33143c8af73227f2a2c"><code>fda00eb</code></a> don't panic in overflowing Timestamp to SystemTime conversion</li> <li><a href="https://github.com/uuid-rs/uuid/commit/c82e88ca184e4ab83ce6b4ac0be33d32a0b9c3c4"><code>c82e88c</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/907">#907</a> from lenamonj/v7-counter-placement</li> <li><a href="https://github.com/uuid-rs/uuid/commit/ac065a6c17389a67dc6e98ef5f9a2e1d7ad4a670"><code>ac065a6</code></a> Align the counter diagram</li> <li><a href="https://github.com/uuid-rs/uuid/commit/34ec10208d813672928c299146dfe4c18cedcec7"><code>34ec102</code></a> Seat the v7 counter below the version nibble</li> <li>See full diff in <a href="https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1">compare view</a></li> </ul> </details> <br /> Updates `serde_with` from 3.22.0 to 3.23.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/jonasbb/serde_with/releases">serde_with's releases</a>.</em></p> <blockquote> <h2>serde_with v3.23.0</h2> <h3>Changed</h3> <ul> <li>Update <code>syn</code> and <code>darling</code> dependencies to use <code>syn</code> v3 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/992">#992</a>)</li> <li>Update dev-dependencies to newer versions (<a href="https://redirect.github.com/jonasbb/serde_with/issues/993">#993</a>)</li> <li>Update <code>base64</code> to a newer version. This should not have any API change, but some error messages might change. (<a href="https://redirect.github.com/jonasbb/serde_with/issues/993">#993</a>)</li> <li><code>serde_as</code> can now parse <code>cfg_attr(true, ...)</code> and <code>cfg_attr(false, ...)</code> (<a href="https://redirect.github.com/jonasbb/serde_with/issues/995">#995</a>) <code>true</code>/<code>false</code> are new literals as of Rust 1.88 but need to be parsed explicitly with the <code>syn</code> types. This is used when emitting <code>schemars</code> annotations.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/jonasbb/serde_with/commit/ea5dfdc6fd1b4732188519e871e2fd2a8fe49f88"><code>ea5dfdc</code></a> Bump version to v3.23.0 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1005">#1005</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/e52e85b1db55e4dc6305b6b19c2710d6a8b2d433"><code>e52e85b</code></a> Bump version to v3.23.0</li> <li><a href="https://github.com/jonasbb/serde_with/commit/39955b6954796d963d02d2f28e0a22087c48fcd4"><code>39955b6</code></a> Bump rmp dev-dependency (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1004">#1004</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/b51e5fb877c642c47da36c296c222e1d172d75da"><code>b51e5fb</code></a> Bump rmp dev-dependency</li> <li><a href="https://github.com/jonasbb/serde_with/commit/ef598c6683d591851e09cac3a4bd767c93bacb9e"><code>ef598c6</code></a> Use setup-rust-toolchain v2 instead of v1 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1003">#1003</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/9ea2658f45e763369f18078d80ea3b109d491de8"><code>9ea2658</code></a> Fix cargo lint about workspace lints being inherited in the test crate</li> <li><a href="https://github.com/jonasbb/serde_with/commit/e5ad81af7721bb770772e8e28de32437f3ff217f"><code>e5ad81a</code></a> Use setup-rust-toolchain v2 instead of v1</li> <li><a href="https://github.com/jonasbb/serde_with/commit/81001414f153d18fc1aca6c2f52990c557f90471"><code>8100141</code></a> Bump the github-actions group across 1 directory with 2 updates (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1002">#1002</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/940f4a62a418784e23026953db8a8fed577bff6d"><code>940f4a6</code></a> Bump jsonschema from 0.49.8 to 0.52.0 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1001">#1001</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/6f42b94d7bb71b004d0e591a26fbf8d900878c91"><code>6f42b94</code></a> Bump the github-actions group across 1 directory with 2 updates</li> <li>Additional commits viewable in <a href="https://github.com/jonasbb/serde_with/compare/v3.22.0...v3.23.0">compare view</a></li> </ul> </details> <br /> Updates `aws-smithy-types` from 1.4.8 to 1.6.3 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Updates `napi-derive` from 3.6.3 to 3.6.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-derive's releases</a>.</em></p> <blockquote> <h2>napi-derive-v3.6.5</h2> <h3>Other</h3> <ul> <li>update Cargo.toml dependencies</li> </ul> <h2>napi-derive-v3.6.4</h2> <h3>Fixed</h3> <ul> <li><em>(deps)</em> update rust crate convert_case to 0.12 (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3469">#3469</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/1492b220d5ad01807b2dcbd250e8383f9d738311"><code>1492b22</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3502">#3502</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/828983aab9b93f516275625f89905fffca459780"><code>828983a</code></a> fix(napi): return errors from the serde deserializer for unexpected JS value ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/606b14238b8d82578705161c0b0d6b6f4b7c2556"><code>606b142</code></a> fix(napi): validate wrapped payload provenance in Object::unwrap/remove_wrapp...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/a75d89f85c0a9a0ffd02b8bb11a2c0897b72a2ae"><code>a75d89f</code></a> fix(napi): point from_external slices at the engine-owned copy after finalize...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/e414c8c21924a4c47bf1c8c5d6ad244dac479578"><code>e414c8c</code></a> fix(deps): update dependency obug to v3 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3499">#3499</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/a5fedde2597dc3ac17c7d422f158adce5fec59bf"><code>a5fedde</code></a> chore(deps): update release-plz/action action to v0.5.136 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3505">#3505</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/31c27a1676a7c4b317f4e144e0a9cb94e8354143"><code>31c27a1</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3470">#3470</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/7e3f293e2d6a3032eabfe51ff38bcaa82d342a2f"><code>7e3f293</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/f772ee0aabf4dbb27250a8df47a73463e8f78cf4"><code>f772ee0</code></a> fix(cli): align generated file formats (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3501">#3501</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1cf5ec573637d96dddb77b1a0ae6aaf316739bd0"><code>1cf5ec5</code></a> fix(cli): use accessible WASI preopen root on Android (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3485">#3485</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.3...napi-derive-v3.6.5">compare view</a></li> </ul> </details> <br /> Updates `napi-build` from 2.4.1 to 2.4.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-build's releases</a>.</em></p> <blockquote> <h2>napi-build-v2.4.2</h2> <h3>Fixed</h3> <ul> <li><em>(cli,build)</em> make wasm32-wasip1-threads link with wasi-sdk 34 and Rust nightly (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3492">#3492</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/31c27a1676a7c4b317f4e144e0a9cb94e8354143"><code>31c27a1</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3470">#3470</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/7e3f293e2d6a3032eabfe51ff38bcaa82d342a2f"><code>7e3f293</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/f772ee0aabf4dbb27250a8df47a73463e8f78cf4"><code>f772ee0</code></a> fix(cli): align generated file formats (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3501">#3501</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1cf5ec573637d96dddb77b1a0ae6aaf316739bd0"><code>1cf5ec5</code></a> fix(cli): use accessible WASI preopen root on Android (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3485">#3485</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/cf9245d0b367a8fc2b3c1209269832b9f076de32"><code>cf9245d</code></a> chore(deps): update vitest monorepo to v5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3481">#3481</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/78fa3ad79d38b52e7c49f9a86ba2e57f46805d1b"><code>78fa3ad</code></a> fix(macro): recognize fully-qualified napi::Env as the special Env parameter ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/37283c75a1f1f20d166a8f9f641ada43d04d24a5"><code>37283c7</code></a> chore(deps): lock file maintenance (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3474">#3474</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1932d2f42198cb9f79704aba66302a0956590c48"><code>1932d2f</code></a> chore(deps): update release-plz/action action to v0.5.135 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3500">#3500</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/0238e8b1b8fcafa2261d4d1b96e7568dd0fb1fc3"><code>0238e8b</code></a> fix(deps): update dependency js-yaml to v5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3344">#3344</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/9ed9d3509e1c2cab2cd89dcff04963adaad4cb7d"><code>9ed9d35</code></a> chore(deps): update dependency electron to v44 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3468">#3468</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.1...napi-build-v2.4.2">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
21ecafe9ae |
chore: update lance dependency to v13.0.0-beta.1 (#4183)
Update the Rust workspace Lance dependencies and Java lance-core from v12.0.0-beta.18 to [v13.0.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.1), and refresh Cargo.lock; no compatibility fixes were required. Validation passed: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. Also fixes the flaky Node test `when optimizing a dataset › cleanups old versions` that failed the NPM Publish Linux test jobs on this PR. The test captured `new Date()` (millisecond precision) in the same millisecond as the last commit, while Lance compares version timestamps at nanosecond precision, so that version was not pruned. The test now waits for the clock to tick to the next millisecond before taking the cutoff. This is a pre-existing flake on `main` since #4160, unrelated to the Lance upgrade. --------- Co-authored-by: Yang Cen <bubble-cal@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
56bbd19ff4 |
test(python): cover namespace operations without pylance (#4174)
The "Test without pylance or pandas" CI job only ran `test_table.py`, so the regression fixed in #3606 had no guard: sync namespace operations used to route through the Python `lance_namespace` client, whose `dir` implementation ships in the optional `pylance` extra, so `lancedb.connect(path).list_namespaces()` failed with `No module named 'lance'` while the async API worked. Adds `python/python/tests/test_namespace_no_pylance.py` and runs it in that job. Its `without_pylance` fixture blocks `lance` imports, so the guard also fires in environments that do have `pylance` installed. Coverage: the original reproducer, nested namespace lifecycle, namespaced and root table lifecycle, the async path, and the one API that legitimately still needs `pylance` (`namespace_client()`). Verified the guard actually catches the regression: against `lancedb==0.33.0`, three of these tests fail with the original error; against a fixed build all pass. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
ffe94a65a1 |
fix(node): preserve optimize cleanup timestamp (#4160)
<!-- lance-gatekeeper-fix:v1 agent=278cb095b2e5d69051442bc254d803a6 generation=1 --> ## Summary - pass the TypeScript `cleanupOlderThan` date to the native binding as an unchanged epoch timestamp - prune with Lance's absolute `before_timestamp` policy so dispatch and compaction time cannot move the cutoff - retain versions created after the supplied cutoff and document that behavior - add boundary and end-to-end regression coverage ## Root cause The TypeScript layer converted the absolute date into an elapsed duration before calling native optimize. Lance converted that duration back into a timestamp only after compaction, which silently advanced the requested cutoff and made the cleanup count depend on a millisecond timing boundary. ## Validation - `cargo fmt --all` - `cargo clippy --quiet --features remote --tests --examples -p lancedb -p lancedb-nodejs` - `pnpm build` - `pnpm lint` - `pnpm run docs` - `pnpm test __test__/table.test.ts --runInBand` (309 passed) Fixes #4159 --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
37771fd4fc |
fix(deps): update rustls and cap aws-smithy-types to unbreak CI (#4177)
Two upstream dependency releases broke CI on `main`. Both fixes are dependency constraints, so they ride together. ## `deny` — RUSTSEC-2026-0285 rustls 0.23.40 accepts TLS 1.3 handshake messages sent at the wrong encryption level ([advisory](https://rustsec.org/advisories/RUSTSEC-2026-0285)), patched in 0.23.45. rustls 0.23.45 requires `aws-lc-rs >= 1.18`, which the nodejs crate pinned to `=1.16.3`, so this also bumps that pin and its `aws-lc-sys` companion to `=1.18.1` / `=0.45.0`. The pin comment already calls for periodic updates on security patches. The workspace's other rustls (0.21.12) is below the advisory's affected range (`unaffected = ["< 0.23.13"]`). ## `build-no-lock` — aws-smithy-types 1.7.0 `aws-smithy-types` 1.7.0 and `aws-smithy-json` 0.64.0 both released 2026-09-14. 1.7.0 made `Document` `non_exhaustive`, which `aws-smithy-json` 0.63 does not compile against: ``` error[E0004]: non-exhaustive patterns: `&_` not covered --> aws-smithy-json-0.63.0/src/serialize.rs:36:15 note: `aws_smithy_types::Document` defined here --> aws-smithy-types-1.7.0/src/document/mod.rs:91:1 ``` Every `aws-sdk-*` crate moved to `aws-smithy-json ^0.64`, but `aws-config` 1.12.0 still requires `^0.63`, so a lockfile-free resolve pairs json 0.63.0 with types 1.7.0 and fails. This caps `aws-smithy-types` below 1.7 as a constraint-only dev-dependency, matching the existing `aws-smithy-runtime` entry. Revert once `aws-config` moves to `aws-smithy-json` 0.64. Note this break is not specific to this PR — `build-no-lock` fails the same way on unrelated branches (e.g. `jon/secrets-client-api` run 34903587364), which passed it hours earlier. ## Verification Resolution only, no local build: - Locked resolve unchanged: `aws-smithy-types` stays 1.4.8; the only `Cargo.lock` delta from the cap is the new dev-dep edge. - Fresh resolve (`rm Cargo.lock`): `aws-smithy-json` 0.63.0 with `aws-smithy-types` 1.6.3, `aws-sdk-*` one release back, `rustls` 0.23.45 retained. |
||
|
|
c44b192334 | Bump version: 0.39.0-beta.7 → 0.39.0-beta.8 | ||
|
|
7575c2597a |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
|
|
0665575a76 |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
|
|
0113cee489 |
docs(embeddings): correct the documented max_retries default (#4145)
`lancedb/embeddings/utils.py` documents `max_retries` with "(default is 10)" — the signature default is `7`. Docs-only; conventional title per the contribution guide. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
6bb64c3edb |
fix: reject repeated job pagination tokens (#4139)
Fixes #4138 `RemoteDatabase::list_jobs` followed every returned pagination token without remembering previously seen values. A server-side token cycle therefore caused repeated requests and duplicate accumulation until the 100-page safeguard returned partial results as a success. This change tracks non-empty job-list page tokens and returns an HTTP-context error as soon as a token repeats, matching the existing `list_functions` behavior. A mock-handler regression test verifies a repeated `loop` token is rejected after two requests. Validation: - `cargo test --quiet --features remote -p lancedb test_list_jobs` - `cargo fmt --all` - `cargo check --quiet --features remote --tests --examples` <!-- lance-gatekeeper-fix:v1 agent=ddd56389737a405d32cf4f43c7696032 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
255da8a854 |
fix(python): resolve native job types in API docs (#4170)
Native job metadata types report `builtins` as their module, so Griffe cannot resolve the public `lancedb.job` re-exports and the Python API reference build fails. Set the PyO3 module metadata for `JobInfo`, `JobDescription`, and `JobFailureInfo`, and cover import resolution in the existing package metadata tests. Reproduced the failure and validated the fix with the docs CI toolchain (`griffe==0.49.0`, `mkdocstrings==0.25.2`, and `mkdocstrings-python==1.10.9`). After rebuilding the native extension, the full `PYTHONPATH=. mkdocs build` succeeds and all three classes and their public members appear in the generated reference. |
||
|
|
ec410e015a | Bump version: 0.39.0-beta.6 → 0.39.0-beta.7 | ||
|
|
9fe10c7362 |
fix(remote): align Function CRUD routes (#4166)
Align the experimental Function HTTP transport with the equivalent Table
CRUD API shape. This is an intentional breaking change to the
experimental Function routes; public Rust and Python APIs remain
unchanged.
## Route comparison
| Operation | Function before | Function after | Equivalent Table API |
| --- | --- | --- | --- |
| Create | `POST /v1/functions/create` | `POST /v1/function/{id}/create`
| `POST /v1/table/{id}/create` |
| Describe | `POST /v1/functions/describe` | `POST
/v1/function/{id}/describe` | `POST /v1/table/{id}/describe` |
| List | `POST /v1/functions/list` | `GET
/v1/namespace/{id}/function/list` | `GET /v1/namespace/{id}/table/list`
|
| Drop | `POST /v1/functions/drop` | `POST /v1/function/{id}/drop` |
`POST /v1/table/{id}/drop` |
## Contract details
- Create, describe, and drop use a singular resource path. Their `{id}`
path parameter is the URL-encoded Function name, and the duplicate
Function identifier is removed from each request body.
- Create continues to accept `202 Accepted`.
- List changes from a POST with a JSON body to a namespace-scoped GET.
Its `{id}` path parameter is the namespace identifier rather than a
Function name.
- Functions do not support nested namespaces yet, so the client lists
against the root namespace identifier (`$` with the default delimiter).
A non-root namespace is rejected.
- The optional list filter is named `name`. `limit`, `page_token`, and
`include_definition` remain available as query parameters.
- The paginated list response shape is unchanged.
|
||
|
|
b8f0048b5a |
chore: update lance dependency to v12.0.0-beta.18 (#4164)
Update the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core to [v12.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.18). Fix redundant visibility declarations in the Node.js Rust blob helpers required by the workspace Clippy check. Validated with workspace Clippy (all features and tests, warnings denied), cargo fmt, pnpm build, and 13 targeted Node.js blob tests. |
||
|
|
6702e3fec1 |
feat(node): add blob v2 fetch and field helpers (#4155)
this PR blob v2 field helpers and reads to the Node SDK.
`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.
Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.
Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.
```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
"videos",
[{ id: 1n, video }],
{
schema: new Schema([
new Field("id", new Int64()),
blob("video"),
]),
},
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);
```
### Testing
- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
|
||
|
|
e0bd4b5fa1 |
chore: update lance dependency to v12.0.0-beta.17 (#4162)
Update the Rust workspace Lance dependencies and Java lance-core to [v12.0.0-beta.17](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.17). Align object_store to 0.14.1 for compatibility with Lance and refresh the Cargo lockfile, including the required reqsign updates. Validation passed: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings` and `cargo fmt --all --quiet`. |
||
|
|
13f9dd630b |
build(deps): bump prost from 0.14.3 to 0.14.4 in the rust-minor-patch group (#4135)
Bumps the rust-minor-patch group with 1 update: [prost](https://github.com/tokio-rs/prost). Updates `prost` from 0.14.3 to 0.14.4 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md">prost's changelog</a>.</em></p> <blockquote> <h1>Prost version 0.14.4</h1> <p><em>PROST!</em> is a <a href="https://protobuf.dev/">Protocol Buffers</a> implementation for the <a href="https://www.rust-lang.org/">Rust Language</a>. <code>prost</code> generates simple, idiomatic Rust code from <code>proto2</code> and <code>proto3</code> files.</p> <h3>🚀 Features</h3> <ul> <li><em>(prost-derive)</em> Make is_valid a constant function (<a href="https://redirect.github.com/tokio-rs/prost/issues/1401">#1401</a>)</li> <li>Increase MSRV to 1.85 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1428">#1428</a>)</li> </ul> <h3>🐛 Bug Fixes</h3> <ul> <li>Use Display instead of Debug for generated enumeration attributes (<a href="https://redirect.github.com/tokio-rs/prost/issues/1419">#1419</a>)</li> <li><em>(prost-derive)</em> Return error for invalid enumeration default identifiers (<a href="https://redirect.github.com/tokio-rs/prost/issues/1426">#1426</a>)</li> <li><em>(build)</em> Grab binary path from cargo (<a href="https://redirect.github.com/tokio-rs/prost/issues/1429">#1429</a>)</li> <li><em>(build)</em> Fix C++ build on GCC 15 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1395">#1395</a>)</li> </ul> <h3>📚 Documentation</h3> <ul> <li>Add example for <code>decode_length_delimiter</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1311">#1311</a>)</li> <li>Update protobuf-src example to avoid unsafe set_var</li> </ul> <h3>🧪 Testing</h3> <ul> <li>Test derive Eq behavior (<a href="https://redirect.github.com/tokio-rs/prost/issues/1422">#1422</a>)</li> <li><em>(groups)</em> Actually construct <code>NestedGroup</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1363">#1363</a>)</li> </ul> <h3>💼 Dependencies</h3> <ul> <li><em>(deps)</em> Update criterion requirement from 0.7 to 0.8 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1374">#1374</a>)</li> <li><em>(deps)</em> Remove <code>getrandom@0.4.1</code> from build-dependencies (<a href="https://redirect.github.com/tokio-rs/prost/issues/1400">#1400</a>)</li> <li><em>(deps)</em> Update rand requirement from 0.9 to 0.10 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1397">#1397</a>)</li> <li><em>(deps)</em> Bump actions/upload-artifact from 6 to 7 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1409">#1409</a>)</li> <li><em>(deps)</em> Update <code>cargo clippy</code> to 1.89 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1433">#1433</a>)</li> <li><em>(deps)</em> Update <code>cargo clippy</code> to 1.91 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1435">#1435</a>)</li> <li><em>(deps)</em> Update and improve nix devshell (<a href="https://redirect.github.com/tokio-rs/prost/issues/1393">#1393</a>)</li> </ul> <h3>🎨 Styling</h3> <ul> <li>Prevent needless borrow (<a href="https://redirect.github.com/tokio-rs/prost/issues/1404">#1404</a>)</li> <li>Use <code>std::hint::black_box()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1403">#1403</a>)</li> <li>Use variables directly in <code>format!()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1432">#1432</a>)</li> <li>Remove explicit <code>.into_iter()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1434">#1434</a>)</li> <li>Run clippy on benches (<a href="https://redirect.github.com/tokio-rs/prost/issues/1405">#1405</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/tokio-rs/prost/commit/13646cde7eab75c81b3047767aa0a86e7dbecf12"><code>13646cd</code></a> chore: Release version 0.14.4 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1437">#1437</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/dad79d5c8e3549d93ebe6f6c723bb42928d805d8"><code>dad79d5</code></a> fix(prost-derive): return error for invalid enumeration default identifiers (...</li> <li><a href="https://github.com/tokio-rs/prost/commit/b0b6c93e3aac89df28690a4967a8bbe93ec95391"><code>b0b6c93</code></a> ci: Update <code>cargo clippy</code> to 1.91 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1435">#1435</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/32cfffbc494f2faf461cab85e04a42412484c0e4"><code>32cfffb</code></a> style: remove explicit <code>.into_iter()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1434">#1434</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/2710efdb9978d9c75fb19b0b092a369a2d385b55"><code>2710efd</code></a> ci: Update <code>cargo clippy</code> to 1.89 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1433">#1433</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/18ea4e42bbc307d33d65e05ad47b3c45623c0500"><code>18ea4e4</code></a> style: use variables directly in <code>format!()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1432">#1432</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/2821bd1d8c20137c83ead4db39f8e1da00b4e854"><code>2821bd1</code></a> build(deps): bump actions/upload-artifact from 6 to 7 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1409">#1409</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/3ce3b39f9206b5e3bbe34c6e1aa69fe3c53f0924"><code>3ce3b39</code></a> test(groups): Actually construct <code>NestedGroup</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1363">#1363</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/8776405574b3ba0a0fe96ada8799ac8bc61ceb3e"><code>8776405</code></a> docs: Update changelog for version 0.14.3 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1431">#1431</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/33d3ef18c008da13e862d7e7674d751ab2776360"><code>33d3ef1</code></a> build: Grab binary path from cargo (<a href="https://redirect.github.com/tokio-rs/prost/issues/1429">#1429</a>)</li> <li>Additional commits viewable in <a href="https://github.com/tokio-rs/prost/compare/v0.14.3...v0.14.4">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
bc4497b21a |
chore: update lance dependency to v12.0.0-beta.16 (#4156)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from v12.0.0-beta.15 to [v12.0.0-beta.16](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.16). No compatibility fixes were required; `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed. |
||
|
|
1da5876870 |
ci: add spell checking (#4148)
Adds [typos](https://github.com/crate-ci/typos) as a CI check and pre-commit hook, the same way Lance does it, so misspellings like the ones fixed in #4146 get caught automatically going forward. This also fixes the misspellings `typos` found across the repo (Rust, Python, TypeScript source, comments, and generated docs), and adds a small `.typos.toml` with `extend-words` entries for terms that are correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ` (a real quantization algorithm name), `mmaped` (the actual name of a `candle-core` API we call), and `Writeable` (from Python's `_typeshed.WriteableBuffer`). Third-party license files are excluded. Fixes #4147 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
577fb48376 |
fix(python): apply offset when combining async hybrid results (#4028)
Fixes #4027 ## Summary `AsyncHybridQuery` (`table.query().nearest_to(...).nearest_to_text(...)`) paginates incorrectly when `.offset()` is used: the second page repeats rows from the first page and silently drops others. `offset()` on a hybrid query pushes the offset down into *both* sub-queries (`HybridQuery::offset` in `python/src/query.rs` forwards to `inner_vec` and `inner_fts`), so each sub-query independently skips its own first `offset` rows before the results are fused. `AsyncHybridQuery.to_batches` then called `_combine_hybrid_results(..., limit=self._inner.get_limit())` without an `offset`, so the reranked table was sliced starting at position 0 and the sub-query limits were never raised to cover the skipped prefix. On the 4-row fixture in `test_hybrid_query.py`, with `_rowid` ordering `[3, 0, 2, 1]`: | query | before | after | | --- | --- | --- | | `.limit(2)` | `[0, 3]` | `[0, 3]` | | `.offset(2).limit(2)` | `[3, 1]` | `[2, 1]` | Row `3` was returned on both pages and row `2` was never returned at all. This is the async counterpart of #3769 (`Fixes #3765`), which fixed the same bug in the synchronous `LanceHybridQueryBuilder`. #3765 explicitly deferred the async path; this PR closes that gap and reuses the `offset` parameter that #3769 already added to `_combine_hybrid_results`. The synchronous path is unaffected — it was fixed in #3769. ## Changes `python/python/lancedb/query.py`, `AsyncHybridQuery.to_batches`: - Each sub-query now fetches `limit + offset` rows and its own offset is reset to 0, so the fused result contains the full prefix the window is sliced out of. - The combined, reranked table is sliced with `offset=` instead of always starting at 0. Both halves are needed: raising the sub-query limits without the final slice still returns page 1, and slicing without raising the limits still misses rows. `nodejs` has no equivalent hybrid combine path, so there is no SDK parity gap here. ## Test plan - [x] New regression test `test_async_hybrid_query_offset` in `python/python/tests/test_hybrid_query.py`, mirroring the sync `test_hybrid_query_offset`. It asserts the offset window is a suffix of the un-offset result *and* that page 1 + page 2 together cover every row exactly once (a row-count-only assertion would pass even with duplicates). - [x] `pytest python/tests/test_hybrid_query.py` — 16 passed - [x] `pytest python/tests/test_rerankers.py` — 9 passed, 11 skipped - [x] `pytest python/tests/test_query.py` — 86 passed - [x] `pytest --doctest-modules python/lancedb/query.py` — 13 passed - [x] `ruff format --check` / `ruff check` — clean --- ## Scope, after review @lancedb-gatekeeper raised three points. Two were mine and are fixed in `04d07c2`; the third is deliberately left alone and I'd like a maintainer's call on it. **Fixed — effective limit was read from the FTS child only.** `HybridQuery::get_limit()` (`python/src/query.rs:1159`) returns `self.inner_fts.inner.current_request().limit`, so an FTS-first hybrid with no explicit `.limit()` yielded `None`, skipped the widening branch and passed `limit=None` to the combiner — returning the union of both candidate lists instead of the documented default of 10. The limit is now derived from both children with a `DEFAULT_HYBRID_LIMIT = 10` fallback, so construction order no longer matters. **Fixed — `explain_plan()` / `analyze_plan()` described a different query than the one that ran.** Both built their children straight from `self._inner`, bypassing the limit/offset rewrite in `to_batches`, and reported `skip=2, fetch=2` while execution used `skip=0, fetch=4`. Child preparation now lives in one `_create_child_queries()` helper used by all three. > **Visible change to `explain_plan()` output:** because the plan is now built from the real execution children, which carry `with_row_id()`, the two `ProjectionExec` lines gain a `_rowid` column. The doctest is updated to match. This is the diagnostic becoming truthful rather than the assertion being weakened — it is still an exact-match comparison. **Not fixed here — RRF candidate-pool invariance.** Widening each sub-query to `limit + offset` does change the candidate pool between page requests, so the fused ranking can shift and pagination can still repeat rows. That's a real problem, but it is exactly what the merged sync path does today: ```python # LanceHybridQueryBuilder (sync), merged in #3769 sub_query_limit = self._limit + (self._offset or 0) ``` Making the pool invariant means choosing a contract — a fixed candidate pool, or an explicit cursor — and that ought to apply to sync and async together rather than letting the two paths diverge. I've asked in the review thread which way you'd prefer, and I'm happy to do it here or in a follow-up covering both paths. So, to be precise about what this PR delivers: it makes `.offset()` take effect on the async hybrid path and makes the diagnostics honest. It does not make hybrid pagination stable across pages under reranking — that needs the contract decision above. |
||
|
|
c7b051aff7 |
docs: fix spelling typos across python package docstrings (#4146)
Six files carried spelling typos in user-visible docstrings: - `table.py` (×3) + `remote/table.py`: "The **targetted** vector to search for" → "targeted" - `query.py`: "pa.Array **wouln't** be allowed" → "wouldn't" - `embeddings/gte.py`: "mlx package **insalled**" → "installed" - `rerankers/base.py`: "This is **inteded**" → "intended" - `index.py`: "dimension **divded** by 8" → "divided" Docstrings only. |
||
|
|
2e205ac9bb | Bump version: 0.39.0-beta.5 → 0.39.0-beta.6 | ||
|
|
3e3878b223 |
chore: update lance dependency to v12.0.0-beta.15 (#4143)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from v12.0.0-beta.14 to [v12.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.15). No compatibility fixes were required; `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed. --------- Co-authored-by: Jack Ye <yezhaoqin@gmail.com> |
||
|
|
19fb665c76 | Bump version: 0.39.0-beta.4 → 0.39.0-beta.5 | ||
|
|
1f95398c34 |
feat: function columns on materialized views (#4119)
A view could not carry a column it does not compute: the definition planned every output as a SQL expression, and the refresh engine treated any commit it did not make as drift and rebuilt. Servers fill such columns on tables with a separate job, as computed columns bound to a registered function, and want the same column on a view. This lets a declaration add computed columns, placed at their positions in the select list and validated by the existing computed-column contract, with the view created in one commit. Refresh writes those columns NULL on every path and never reads them, so a rewritten row comes back unfilled, and a commit that rewrites only computed columns is recognised as a fill rather than drift, so the next refresh carries on incrementally. A source column a computed column reads without the view projecting it is held as an internal projection, so the select list stays the view's column list. Nothing in the stored definition changes; an older reader fails closed on the schema check. Two smaller changes ride along because the feature needs them: an identity projection keeps its source column's nullability, with the schema check accepting a nullable physical field for a non-null planned one so existing views keep refreshing; and `prepare_declaration` takes `Option` projections, so an empty list declares no projection rather than `SELECT *`. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
0111a72dc3 | Bump version: 0.39.0-beta.3 → 0.39.0-beta.4 | ||
|
|
a487d4033e |
chore: update lance dependency to v12.0.0-beta.14 (#4141)
Update the Rust workspace Lance dependencies and Java lance-core from v12.0.0-beta.11 to [v12.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.14), refreshing Cargo.lock. Resolve two Clippy diagnostics by making an internal Node.js helper private and using a byte string literal in a remote-table test fixture. Validation: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, `git diff --check`, and `pnpm build` in nodejs. --------- Co-authored-by: Jack Ye <yezhaoqin@gmail.com> |
||
|
|
02ea0dda9f |
build(deps-dev): bump the nodejs-deps group across 1 directory with 2 updates (#4134)
Bumps the nodejs-deps group with 2 updates in the /nodejs directory: [@opentelemetry/sdk-metrics](https://github.com/open-telemetry/opentelemetry-js) and [ts-jest](https://github.com/kulshekhar/ts-jest). Updates `@opentelemetry/sdk-metrics` from 2.10.0 to 2.11.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/open-telemetry/opentelemetry-js/releases">@opentelemetry/sdk-metrics's releases</a>.</em></p> <blockquote> <h2>v2.11.0</h2> <h2>2.11.0</h2> <h3>🚀 Features</h3> <ul> <li>feat(context-async-hooks): implement <code>attach()</code> on <code>AsyncLocalStorageContextManager</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6845">#6845</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a> <ul> <li>On Node.js 25.9+, delegates to <code>AsyncLocalStorage.withScope()</code> returning a native <code>RunScope</code>. On older Node.js, falls back to <code>enterWith()</code> with a manual disposable wrapper.</li> </ul> </li> <li>feat(sdk-trace): allow configuring the force flush timeout per call <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6929">#6929</a> <a href="https://github.com/LarryHu0217"><code>@LarryHu0217</code></a></li> </ul> <h3>🐛 Bug Fixes</h3> <ul> <li>fix(sdk-metrics): ignore <code>Infinity</code> in exponential histograms <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7015">#7015</a> <a href="https://github.com/mwear"><code>@mwear</code></a></li> </ul> <h3>🏠 Internal</h3> <ul> <li>perf(sdk-metrics): reuse a single DataView for exponential histogram bit reads <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6998">#6998</a> <a href="https://github.com/mwear"><code>@mwear</code></a></li> <li>chore(ci): run documentation tests on a weekly schedule <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6920">#6920</a> <a href="https://github.com/LarryHu0217"><code>@LarryHu0217</code></a></li> <li>feat(ci): support pre-releases and major version bumps in the release workflow <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6768">#6768</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a></li> <li>chore(resources): Ensure that multiple uses of serviceInstanceIdDetector.detect() return the <em>same</em> value for <code>service.instance.id</code></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md">@opentelemetry/sdk-metrics's changelog</a>.</em></p> <blockquote> <h2>2.11.0</h2> <h3>🚀 Features</h3> <ul> <li>feat(context-async-hooks): implement <code>attach()</code> on <code>AsyncLocalStorageContextManager</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6845">#6845</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a> <ul> <li>On Node.js 25.9+, delegates to <code>AsyncLocalStorage.withScope()</code> returning a native <code>RunScope</code>. On older Node.js, falls back to <code>enterWith()</code> with a manual disposable wrapper.</li> </ul> </li> <li>feat(sdk-trace): allow configuring the force flush timeout per call <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6929">#6929</a> <a href="https://github.com/LarryHu0217"><code>@LarryHu0217</code></a></li> </ul> <h3>🐛 Bug Fixes</h3> <ul> <li>fix(sdk-trace-base): avoid a Webpack self-reference error in CommonJS output <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6981">#6981</a> <a href="https://github.com/sansynx"><code>@sansynx</code></a></li> <li>fix(sdk-metrics): ignore <code>Infinity</code> in exponential histograms <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7015">#7015</a> <a href="https://github.com/mwear"><code>@mwear</code></a></li> </ul> <h3>🏠 Internal</h3> <ul> <li>perf(sdk-metrics): reuse a single DataView for exponential histogram bit reads <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6998">#6998</a> <a href="https://github.com/mwear"><code>@mwear</code></a></li> <li>chore(ci): run documentation tests on a weekly schedule <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6920">#6920</a> <a href="https://github.com/LarryHu0217"><code>@LarryHu0217</code></a></li> <li>feat(ci): support pre-releases and major version bumps in the release workflow <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6768">#6768</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a></li> <li>chore(resources): Ensure that multiple uses of serviceInstanceIdDetector.detect() return the <em>same</em> value for <code>service.instance.id</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/0b72a81636fa476e8f1f1afd2ae0c90a1362194c"><code>0b72a81</code></a> chore: prepare next release (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7044">#7044</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/a9c5338a9f485f1433df9308f24bd7397a4a0321"><code>a9c5338</code></a> ci: roll prerelease changelog into one final release changelog (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7045">#7045</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/f41805e769ba10fb6dae72a4b7a5a3dc67cca82e"><code>f41805e</code></a> chore: prepare next release (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7042">#7042</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/b85eb28343ff6234e2bb4d46b7b4a3d360e5ea2f"><code>b85eb28</code></a> chore(instrumentation-http): fix lint errors (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7039">#7039</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/3f9253009be2ae48419432e79a597ead7be8be6a"><code>3f92530</code></a> ci: support pre-releases and major version bumps in release workflow (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7035">#7035</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/82a58316a63b6fde62afd268e145b82222d328cf"><code>82a5831</code></a> docs(otlp-exporter-base): document HTTP exporter options (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6735">#6735</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/e086dec7f9304107ef6d50b5877be88895c06aa7"><code>e086dec</code></a> Merge commit from fork</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/59dac70d00d46fa56b2b921cf721fd922730f23d"><code>59dac70</code></a> chore(deps): update jamesives/github-pages-deploy-action action to v4.9.0 (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7">#7</a>...</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/d0ce7532b058631ec9eec111c04fefe7fd873e1f"><code>d0ce753</code></a> chore: add <a href="https://github.com/maryliag"><code>@maryliag</code></a> to maintainers (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7024">#7024</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/03469a129f97265c8eec93d566e9e00d4f741db3"><code>03469a1</code></a> chore(deps): update open-telemetry/shared-workflows action to v0.10.0 (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7032">#7032</a>)</li> <li>Additional commits viewable in <a href="https://github.com/open-telemetry/opentelemetry-js/compare/v2.10.0...v2.11.0">compare view</a></li> </ul> </details> <br /> Updates `ts-jest` from 29.4.9 to 29.4.12 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/kulshekhar/ts-jest/releases">ts-jest's releases</a>.</em></p> <blockquote> <h2>v29.4.12</h2> <p>Please refer to <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v29.4.11</h2> <p>Please refer to <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v29.4.10</h2> <p>Please refer to <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a> for details.</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">ts-jest's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.11...v29.4.12">29.4.12</a> (2026-07-22)</h2> <h3>Features</h3> <ul> <li><strong>compiler:</strong> support TypeScript 7 projects through compatibility aliases (<a href="https://redirect.github.com/kulshekhar/ts-jest/pull/5386">#5386</a>)</li> </ul> <h2><a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.10...v29.4.11">29.4.11</a> (2026-05-21)</h2> <h3>Bug Fixes</h3> <ul> <li>preserve Bundler on the CJS path under TypeScript >= 6 (<a href="https://github.com/kulshekhar/ts-jest/commit/39418187515f11b6584d35a4e3ddf50231f74936">3941818</a>), closes <a href="https://redirect.github.com/kulshekhar/ts-jest/issues/4198">#4198</a></li> </ul> <h2><a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.9...v29.4.10">29.4.10</a> (2026-05-18)</h2> <h3>Bug Fixes</h3> <ul> <li>pass <code>resolutionMode</code> to <code>ts.resolveModuleName</code> for hybrid module support (<a href="https://github.com/kulshekhar/ts-jest/commit/b557a85f85c3fd34523ec3a15293afbdc9dea83c">b557a85</a>)</li> <li>rebuild <code>Program</code> when consecutive compiles need different module kinds (<a href="https://github.com/kulshekhar/ts-jest/commit/a82a2b32c4987a5249fd5284283117dd2fa3be47">a82a2b3</a>), closes <a href="https://redirect.github.com/kulshekhar/ts-jest/issues/4774">#4774</a></li> <li>respect tsconfig <code>moduleResolution</code> instead of forcing <code>Node10</code> (<a href="https://github.com/kulshekhar/ts-jest/commit/1bffffc667557c173ae0c1f93dd436920775dac4">1bffffc</a>)</li> <li><strong>transformer:</strong> transpile <code>mjs</code> files from <code>node_modules</code> for CJS mode (<a href="https://github.com/kulshekhar/ts-jest/commit/96d025dd912ea2bceb18b67d2d509ada7a756d9d">96d025d</a>)</li> <li><strong>transformer:</strong> use a consistent comparator in hoist-jest sortStatements (<a href="https://github.com/kulshekhar/ts-jest/commit/8a8fd2fb8446655bba18367db9306a1089490e62">8a8fd2f</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/kulshekhar/ts-jest/commit/3f05625da10da954fdf0a10394385008275ddbb3"><code>3f05625</code></a> chore(release): 29.4.12</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/df28b27f2e60edf275866763a3cdf745360d3eae"><code>df28b27</code></a> docs: clarify TypeScript version prerequisites</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/c8a614af419b1dcc90f3d1a7a48238ac1b637e6b"><code>c8a614a</code></a> docs: mention TypeScript 7 setup in README</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/06c79d4cebfa785749b2f96ef2dbeffc12798c47"><code>06c79d4</code></a> fix: address TypeScript 7 review feedback</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/f10746008e512203ad7d8c581e2c58cc7dcd43c8"><code>f107460</code></a> docs: explain TypeScript 7 compatibility setup</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/33882274c1e4fc3a06cece3b21f4873182c8fee7"><code>3388227</code></a> test(e2e): add TypeScript compatibility matrix</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/891dc731ff9f31785ad9698e4d7cfa6078991fe6"><code>891dc73</code></a> fix(compiler): support TypeScript 7 compatibility aliases</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/eb135ebe819991b1e10c998915cc6db2057c4de1"><code>eb135eb</code></a> build(deps-dev): bump shell-quote from 1.8.4 to 1.10.0 in /examples</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/d5d80a34bc575be130db2b9f5cd0958173982cda"><code>d5d80a3</code></a> ci: pin google osv scan action at v2.3.5</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/6bf293f0a4ddf468735d81fcdf04f923278e030c"><code>6bf293f</code></a> build(deps): bump shell-quote from 1.8.4 to 1.10.0 in /website</li> <li>Additional commits viewable in <a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.9...v29.4.12">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
c7980dbc40 | Bump version: 0.39.0-beta.2 → 0.39.0-beta.3 |