mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-21 20:45:57 +00:00
54efd4949c344d6f8fcbda09019f6c5435d18959
3014
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 LanceDB Robot
|
54efd4949c |
chore: update lance dependency to v13.0.0-beta.7 (#4234)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from v13.0.0-beta.6 to [v13.0.0-beta.7](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.7); no compatibility fixes are required. Validation: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed. |
||
|
LanceDB Robot
|
df5709efd8 |
chore: update lance dependency to v13.0.0-beta.6 (#4224)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from 13.0.0-beta.4 to [v13.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.6). Fixes the redundant visibility qualifier on the internal identifier delimiter constant reported by Clippy. Validation: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. |
||
 Wyatt Alt
|
01ee01dbc8 |
feat: define a materialized view by its query, with Functions in FROM position (#4190)
A view definition was a structured record under a `kind` tag, one kind per query shape, and a Function returning `list<struct>` was about to add a third. That names shapes instead of describing a relation. A materialized view is now a relation defined by a query, stored as one canonical SQL string under a format number: ```sql SELECT columns FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias] [WHERE predicate] [LIMIT n] ``` Any other clause is refused at parse time. Older readers report the view as unrefreshable, the pre-format layouts still read, and a legacy view is rewritten on its next refresh that commits, rebuilt only where its raw text meant something else under lance's parser. A Function in FROM position yields one row per element it returns, as a table function does in any dialect. The server stages its list output in a hidden table and records that binding beside the query, which stays as the user wrote it; refresh scans the staging and unnests the column, the same operator as UNNEST over a list column the table already holds. Row ids repeat per element, so eviction and incremental append are unchanged. A local database refuses a Function in FROM position, since it has no executor. |
||
  Hongzhu YiandXuanwo
|
7955c50929 |
fix(python): decode file URIs in OpenCLIP (#4131)
## What Decode the path component of `file://` image URIs before passing it to Pillow. ## Why `Path.as_uri()` percent-encodes characters such as spaces. Passing `parsed.path` directly to Pillow therefore tries to open a literal `%20` path and fails. ## Testing - Added a regression test that opens an image whose local filename contains a space. - Verified the focused URI conversion behavior against the changed method. - Ruff check, formatting check, and `compileall` on both changed files. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Lance Release
|
63cd121225 | Bump version: 0.40.0-beta.2 → 0.40.0-beta.3 | ||
 Jack Ye
|
97de3ccd98 |
fix: accept synchronous remote materialized-view drops (#4212)
Remote materialized-view deletion currently rejects HTTP 200 even when the server has completed cleanup synchronously. Treat 200 as an already-finished Job with no ID, matching table deletion; retain the cleanup Job for 202, require its ID, and invalidate the table cache in both cases. Add regressions for synchronous completion and malformed or unexpected responses, alongside the existing asynchronous Job coverage. No local builds or tests were run. |
||
 Vivek
|
a0c5612aee |
feat(rust): make MetadataEraserExec public (#4213)
Export `MetadataEraserExec` and its constructor from `lancedb::table::datafusion`. Engines that serialise a physical plan containing a LanceDB scan have to rebuild the operator outside this crate, which a private type makes impossible. |
||
|
Lance Release
|
89ad06c782 | Bump version: 0.40.0-beta.1 → 0.40.0-beta.2 | ||
|
Wyatt Alt
|
a91b29efc1 |
fix: scope the Function binding guard on schema evolution to the bound columns (#4204)
A table with one registered Function binding refused every add_columns, alter_columns, drop_columns and field-metadata update, whatever column they named. The hazard is narrower: a binding stores the exact Arrow fields of its inputs, outputs and assignment column, so editing one of those strands it and the table stops accepting rows. Any other column was never at risk. The guard now compares the columns a request names, a rename's target included, against the set every binding depends on, and refuses only on an intersection. Local and remote tables apply the same rule. The blanket guard stays on update and merge insert, which cannot say what they touch. |
||
|
Wyatt Alt
|
5231d37f8d |
feat: list a table's per-row Function errors from the client (#4208)
A refresh running under a skip policy records each row it skipped, with the failing input and the error, but the client could not read that store: the server exposes it over SQL and, since recently, a REST route. A user who hit per-row failures still had to open a SQL session. `Table::function_errors` calls the route. The listing is table-addressed with optional job and column filters, the same addressing the SQL surface uses, so the two cannot disagree about what a table's errors are. The two non-record signals come back as their own fields rather than as rows: capped-fragment summaries, and whether the listing stopped at its limit. Local tables refuse rather than answer with an empty list. Python and Node expose the same call, with the same optional filters. |
||
|
Xuanwo
|
c72931b30f |
feat(python): add TypeSafe reranker (#4209)
Adds `TypeSafeReranker`, which reranks vector, FTS, and hybrid results with the [TypeSafe System One API](https://docs.typesafe.ai/introduction). Each result is scored independently: TypeSafe reads `{"query", "document"}` and answers a yes/no (noul) question, and the probability of yes becomes `_relevance_score`. Unlike listwise LLM rerankers, the score is an absolute probability, so it is comparable across queries and can be thresholded. The question's `instructions` and `true`/`false` `criteria` are configurable, since domain-specific criteria are what make this kind of scoring work well ([TypeSafe's re-ranking cookbook](https://docs.typesafe.ai/cookbooks/rerank_typesafe)). The API takes one state per request, so the reranker sends one request per result on a thread pool bounded by `max_concurrency`. It deliberately does not use the background event loop: rerankers are called synchronously from inside the async query APIs, where `LOOP.run` would deadlock. TypeSafe scores for the same pair vary slightly between calls, so results with close scores can swap places when a search is repeated. The shared reranker test helper now takes `deterministic=False` for this case: it still checks result sizes and descending scores, but not that two identical searches return the same order. The SDK is imported only when the client is created and questions are sent as plain dicts, so the new tests run in CI with a fake client and without `typesafe-sdk` installed. The live-API test is skipped without `TYPESAFE_API_KEY`. Ranking quality has not been compared with other API rerankers. |
||
  Jonathan HsiehandClaude Opus 5
|
60a1b4c219 |
feat(secrets): named Secrets, bindings, and namespace addressing (#4150)
Adds the client half of database-scoped named Secrets: a Secret is a
name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads. Secrets are addressed by
a
namespace path plus a name.
The UDF body is unchanged and stays portable — it reads `OPENAI_API_KEY`
the
way it always did, and the binding is what puts a value there:
```python
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
function = db.create_function(
analyze_caption,
secrets=[
EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
],
)
function.secret_bindings # the Secret's name, never its value
```
- `create_secret` / `alter_secret` / `list_secrets` / `describe_secret`
/
`drop_secret` on sync, async and remote connections, with the pyo3
binding
and the Rust client behind them. Each takes `namespace_path`
keyword-only,
defaulting to the root.
- **There is no read API, by construction rather than by policy** — no
code
path returns a stored credential, and `describe_secret` answers with
metadata only.
- `EnvVarSecret` is a pure local constructor: it contacts no server, so
it
cannot fail on a Secret that does not exist. It exists so that a bare
string
in that position — which would be a credential — is a `TypeError` rather
than a plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`: a list of `SecretBinding` tagged by `kind`, so a
later
delivery mode is a variant rather than a sibling field. The value never
travels — it is resolved by the service when the Function runs, which is
what
lets a rotation reach columns already pinned to an older
FunctionVersion.
- A binding names its Secret as a `SecretReference` of `{name,
namespace_path}`
rather than one joined string, so no delimiter has to be excluded from
every
name and segment forever, and `ClientConfig.id_delimiter` cannot
contradict
an identity built on a fixed separator.
- A root namespace is omitted from the request body rather than sent
empty, so
a root request is byte-identical to one from a client that predates
namespaces. Tests pin it.
This is the client surface the design's §4 describes; the service side
lives in
sophon.
**Previously split across two PRs.** Namespace addressing was #4151,
stacked on
this one; it is folded in here so the Secret identity contract — name,
namespace path, and the binding that carries both — is reviewable as one
piece
rather than as a shape introduced and then replaced.
## Identifier safety, merged from #4189
**#4189 is merged into this branch**, so the client half of Secrets and
the
guards on the identity it puts in the URL are one PR. What it added:
- Components are checked where the identifier is built, before a request
is
constructed. `create_secret("../jobs", value)` no longer resolves to
`/v1/jobs/create` and delivers a credential-bearing body to a route with
none
of this one's body suppression.
- Each component is percent-encoded and joined by the delimiter, so
nothing
inside a component can end the path segment or add one.
- A component may not be empty, a relative segment (`.`, `..`, and their
`%2e`
spellings), or the delimiter itself — the three ways a component erases
a
boundary the split has to recover. `["prod", ""]` joined to `prod$`,
which
reads back as `["prod"]`.
- `$` is the only accepted `id_delimiter`, refused at client
construction.
`ClientConfig.id_delimiter` remains, since the identifier grammar comes
from
the Lance REST catalog standard, but a value that would produce
identifiers no
service splits the caller's way is now an error where it was written.
- One `build_object_identifier` and one character set serve tables,
namespaces,
Secrets, Functions and materialized views.
Components are checked for *addressability*, not a character set: the
name's
own grammar stays each object's own, so a catalog database keeps the `/`
that
`RemoteCatalog::validate_name` allows.
## Known shortcoming
`secret_bindings` is omitted from a registration body when empty, so a
client
that binds nothing sends what a client without bindings sends. When a
client
does bind a Secret and the service does not know the field, the field is
ignored: registration succeeds, the returned version carries no
bindings, and
the Function fails at execution with the variable unset, far from the
call that
asked for it.
`ServerVersion` is how this codebase refuses a feature the service is
too old
for, and it gates five features already. It does not gate this one: it
is held
per table, and registering a Function is a database-level call. Noted at
the
field in `remote/db.rs`; wiring the gate is follow-up work.
**Tests:** lancedb lib 1340 passed, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, plus Python tests across both slices.
Rebased onto `main` after #4176 (OCI Function identity), #4191 (`.`/`..`
table
names) and #4195 (remote catalogs).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
LanceDB Robot
|
f8d73b3447 |
chore: update lance dependency to v13.0.0-beta.4 (#4207)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from v13.0.0-beta.3 to [v13.0.0-beta.4](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.4); no compatibility fixes were required. Validation passed: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. |
||
 Colin Patrick McCabe
|
04150b3b82 |
feat: index builders for NGram, BloomFilter, RTree (#4205)
Add index builders in the lancedb library for NGram, BloomFilter, and RTree indexes. |
||
|
Lance Release
|
3309c71a6e | Bump version: 0.40.0-beta.0 → 0.40.0-beta.1 | ||
 Bruno Ramirez
|
32871e97a7 |
fix: widen zonemap index type support (#4206)
ZoneMap indexes were added to the Rust API in #4199, but LanceDB reused the BTree type validation when creating them. That made the public builder reject some types that Lance ZoneMap can support, including `LargeUtf8`, `Binary`, and `LargeBinary`. This PR gives ZoneMap its own validation helper so it can accept the broader scalar set while keeping the rest of the create-index path unchanged. |
||
|
Lance Release
|
86835da5db | Bump version: 0.39.0-beta.10 → 0.40.0-beta.0 | ||
|
Bruno Ramirez
|
a4f66afc69 |
feat(rust): add zonemap index builder (#4199)
Lance supports ZoneMap scalar indexes, but the LanceDB Rust API did not
expose a first-class way to request one through `Table::create_index`.
Users had builders for the other scalar index families, while ZoneMap
was missing from the public `Index` model and remote create-index
serialization. This PR adds ZoneMap as a supported scalar index option
in LanceDB.
This was accomplished with the following changes:
- Added `ZoneMapIndexBuilder` in `rust/lancedb/src/index/scalar.rs`.
- Added `Index::ZoneMap` and `IndexType::ZoneMap`, including
display/from-string aliases for `ZONEMAP` and `ZONE_MAP`.
- Mapped local index creation to
`ScalarIndexParams::for_builtin(BuiltinIndexType::ZoneMap)` and Lance
`IndexType::ZoneMap` in `rust/lancedb/src/table/create_index.rs`.
- Serialized remote create-index requests as `index_type: "ZONEMAP"` in
`rust/lancedb/src/remote/table.rs`.
- Added coverage for both local ZoneMap index creation and remote
request serialization.
Example:
```rust
table
.create_index(&["my_column"], Index::ZoneMap(Default::default()))
.execute()
.await?;
```
### Testing
Added `test_create_zonemap_index` for local index creation and extended
the remote request body test matrix for `ZONEMAP`.
|
||
|
Jack YeandXuanwo
|
6a07f88980 |
feat: add remote catalogs and Python and TypeScript bindings (#4195)
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
 Joaquin HuiandXuanwo
|
99ed25f753 |
fix: return InvalidTableName instead of panicking in open_table/create_table (#4192)
Passing an invalid table name to `open_table` or `create_table` panics
instead of returning an error:
thread '...' panicked at rust/lancedb/src/database/listing.rs:1155:62:
called `Result::unwrap()` on an `Err` value: InvalidTableName { name:
"my table", ... }
Both call sites build the table URI with
`request.location.clone().unwrap_or_else(||
self.table_uri(&request.name).unwrap())`, and `table_uri` is the
function that validates the name — so every rejected name (empty,
spaces, slashes, non-ASCII) hits the inner `unwrap`.
`Error::InvalidTableName` clearly is the intended contract here: the
variant exists for exactly this, and the Python binding maps it to
`ValueError`.
Replaced the closure with a `match` that propagates the validation
error; behavior with an explicit `location` is unchanged (the name is
not validated on that path, as before). Added tests asserting
`InvalidTableName` for `create_table` and `open_table` over a set of
rejected names — both panic without the fix. Full `cargo test -p lancedb
--lib --features remote`: 1214 passed; clippy/fmt clean; `cargo check
--workspace --all-targets` clean.
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
Jack YeandXuanwo
|
f3ef21b8ca |
feat: use oauth2 crate for OAuth with configurable client auth (#4181)
Stacked on #4173 (`jack/restore-oidc-flows`, base branch mirrored to this repo so the diff shows only this change); context from review: https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100. Rebase to `main` once #4173 and #4179 merge. ## What moved to the `oauth2` crate (5.0, no default features) - Authorization URL generation and CSRF state (`authorize_url`, `CsrfToken`) - PKCE S256 challenge/verifier generation and code exchange - Client-credentials, authorization-code, refresh-token, and device-code grant request construction - Device authorization request and the device token polling loop (`authorization_pending`, `slow_down` +5s, expiry deadline, denial, network backoff capped at 10s) - Standard success/error response parsing (`RequestTokenError`) - Token-endpoint client authentication and standards-compliant parameter encoding (RFC 6749 2.3.1 Basic encoding) LanceDB keeps ownership of OIDC discovery (compared `openidconnect`: no measurable win for our 3-field metadata + strict validation, at real dependency cost), HTTPS-or-loopback endpoint enforcement, the loopback callback server, browser/stderr prompts, token caching and refresh orchestration, and the dedicated hardened Azure IMDS source, which is unchanged. ## Client authentication methods New `ClientAuthMethod` enum (`none` | `client_secret_basic` | `client_secret_post`), exposed in Rust, Python, and Node. Unset resolves to `client_secret_basic` when a secret is present (RFC 6749 2.3.1 recommendation and the normal Okta confidential-app default, so a default Okta app works without weakening its configuration) and to `none` for public clients (PKCE/device). Explicit `none` with a secret, or basic/post without one, is rejected. The method applies to client credentials, code exchange, refresh, and device requests. Deliberate behavior change: confidential clients previously always sent the secret in the POST body; they now default to Basic (Keycloak accepts both). No `audience`/`resource` parameters were added: the supported target is an Okta custom authorization server with the API audience configured server-side, so client-provided audience parameters are unnecessary; `add_extra_param` support exists if a concrete provider contract ever needs them. ## Device polling behavior changes (deliberate, tested) - The first token poll now happens immediately rather than after one interval (RFC 8628 allows both). - Transient failures (HTTP 429, 5xx, `temporarily_unavailable`, network errors) now retry with exponential backoff capped at 10s instead of retrying at the fixed interval; polling never spins faster than once per second even if a server reports a zero interval. ## Security and compatibility - Issuer and discovered endpoints (and device verification URIs) still require HTTPS except explicit loopback HTTP, enforced before any crate URL type is built - Token HTTP client keeps the hardened redirect policy that refuses insecure redirect targets; regression test added - Errors never embed raw response bodies (avoids leaking tokens through parse failures); all credential types stay redacted in Debug - Transient conditions (429/5xx/`temporarily_unavailable`) remain retryable in device polling and hard errors elsewhere; refresh keeps rotation and reauthentication semantics - Existing public APIs stay source-compatible except the added `OAuthConfig.client_auth_method` field ## Tests Rust: client-auth methods across code exchange/refresh/client-credentials/device (none/basic/post), auth-method resolution and validation, transient device retries, denial/expiry, redirect rejection, malformed-response leak check, PKCE URL assertions, redaction. Python and Node: enum values, conversion, unknown-method errors, config defaults. Manual Okta validation recipe (no automated Okta credentials): create a custom authorization server with an API audience, one confidential web app (Basic) for authorization-code, one native app (PKCE, no secret), one native device app; point `issuer_url` at the custom server, set `client_auth_method` only for the POST-required case; verify token acquisition, refresh after expiry, and `x-lancedb-credential-type: oidc` against a LanceDB deployment. Never commit tenant URLs or secrets. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Lance Release
|
ed100ccc31 | Bump version: 0.39.0-beta.9 → 0.39.0-beta.10 | ||
|
Xuanwo
|
161f81276d |
chore: update lance dependency to v13.0.0-beta.3 (#4197)
Update the Rust workspace and Java Lance dependencies to [v13.0.0-beta.3](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.3), which includes the nullable-list encoding fix in lance-format/lance#9268. Also resolve existing strict Clippy diagnostics in crate-private OAuth modules and the Python token-cache conversion, without changing effective visibility. Validated the rebuilt Python SDK with nullable and nested lists: Match/Phrase searches preserve document coordinates before and after appending data and optimizing the table. |
||
|
Xuanwo
|
7991e27f35 |
fix(node): prevent OAuth tests from launching browsers (#4196)
OAuth tests set `LANCEDB_OAUTH_BROWSER` inside Jest's sandbox, which does not update the process environment read by Rust. As a result, the native login flow launches a real browser; the [failing macOS main job](https://github.com/lancedb/lancedb/actions/runs/35067074667/job/104699667840) ends by terminating an orphaned Safari process. Set the no-op browser helper while loading Jest configuration, before creating sandboxes and workers, so both local and CI tests inherit it. Check the inherited process environment before OAuth login to catch regressions without opening a browser. Windows uses a no-op command fixture. Test deadlines and worker counts are unchanged. A negative control restoring the sandbox-only assignment fails in the new pre-login check. The full macOS test suite passes with the standard test launcher; the Windows helper has not been executed locally. The [first hosted macOS run](https://github.com/lancedb/lancedb/actions/runs/35071525843/job/104713928139) passes all 843 tests (5 skipped), with no Safari process in the job log. Further normal runs are needed to establish sustained stability. |
||
|
Xuanwo
|
34ab278a5b |
fix: expose public FTS paths in remote index listings (#4194)
Remote `list_indices()` exposes physical FTS paths such as `docs.item.content`, while index creation, queries, and native table listings use `docs.content`. Normalize FTS columns to the public path after parsing the server response, using the same field-ID-based conversion as native tables. Keep the physical paths on the wire: existing clients need them to resolve the Arrow schema. Cover both legacy responses that fetch index statistics and enriched responses that already include the index type. |
||
|
3b37ea2c7a |
feat: represent registered Functions by OCI image identity (#4176)
Function versions identify independent Function objects and their numeric revisions. Rust and Python expose the object ID, location, canonical decimal version, metadata, and availability separately from the OCI image digest. Computed-column applications carry the complete object reference, so existing bindings retain their identity after a name is removed and reused. Source authoring keeps the existing create_function/create_function_async, job wait, and column-binding APIs. The server coordinates baking followed by registration; users do not have to manage manifest digests to create a Function. The previous stored Function representation is intentionally unsupported. Existing contract tests and shared wire fixtures are migrated to the new model. This SDK change accompanies the final integration layer https://github.com/lancedb/sophon/pull/7887 in the Sophon stack: https://github.com/lancedb/sophon/pull/7885 → https://github.com/lancedb/sophon/pull/7886 → https://github.com/lancedb/sophon/pull/7887. A metadata-only revision can retain the same executable image; Function version numbers must not be used as image cache keys. --------- Co-authored-by: lancedb automation <robot@lancedb.com> Co-authored-by: Yang Cen <bubble-cal@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
 Shiduo LiandXuanwo
|
f7579d2aae |
fix(python): correct pylance test extra pin (#4051)
Closes #4045 ## What changed - replace the unpublished pylance 9.0.0rc1 test-extra pin with the published 9.0.0 release - restore dependency resolution for editable installs using the tests extra ## Validation - downloaded pylance==9.0.0 from PyPI with pip --no-deps - parsed python/pyproject.toml with tomllib and verified the tests extra - git diff --check --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 mikemikimike
|
be3215a6ae |
feat(nodejs): add JSON field helper (#4082)
## Issue Fixes #4063 ## Background The Node.js SDK currently requires callers to know the Arrow extension metadata needed to represent JSON fields. This makes a common LanceDB schema type unnecessarily verbose and easy to get wrong. ## Changes - Add `makeJsonField(name, nullable = true)` to create a UTF-8 Arrow field with the `arrow.json` extension metadata. - Re-export the helper from the public Node.js entry point. - Add coverage for the default nullable behavior, explicit non-nullable fields, and the extension metadata. - Add the generated TypeDoc function page and public globals entry, including a usage example. ## Implementation The helper uses the existing Apache Arrow `Field` type and sets `ARROW:extension:name` to `arrow.json`, matching the metadata convention already used by LanceDB. ## Compatibility This is an additive Node.js API. Existing schema construction and Arrow behavior are unchanged. ## Verification - `pnpm test -- arrow.test.ts --runInBand` — 236 tests passed. - `pnpm exec biome ci lancedb/arrow.ts lancedb/index.ts __test__/arrow.test.ts` — passed. - `git diff --check` — passed. ## Not run / known limitations - `pnpm build` and `pnpm run docs` were attempted after expanding the checkout. Both are blocked locally by the native binding build/type declarations: Cargo did not complete, and TypeDoc reported the missing generated `nodejs/lancedb/native` module. The docs files were generated from the updated TypeScript comments; full build and docs validation are left to CI. |
||
|
Colin Patrick McCabe
|
3a1d3be256 |
feat(oidc): support resource and audience (#4193)
Support configuring resource and audience for OAuth authorization, token exchange, and refresh requests. |
||
|
Colin Patrick McCabe
|
ba693ae43d |
fix: do not allow . and .. as table names (#4191)
Do not allow . and .. as table names. They are incompatible with the local filesystem, and confusing in cases where they are supported. |
||
 Will JonesandClaude Opus 5
|
1d2a5d084b |
fix: accept all-null batches and plain JSON strings for json columns (#4067)
Two ways of writing to a `json` column failed or silently corrupted data. **All-null batches were rejected.** `add()` refused a batch whose values for a `json` column were all null, while every plain Arrow type accepted the same batch. This bites row-at-a-time inserts hardest: a one-row batch with no value for an optional column is trivially all-null, so most such writes failed. pyarrow infers `null` as the column's type, and the write path had no handling for it — casting to the table's type dropped the field metadata that identifies the column as `lance.json`, so lance rejected the batch (`` `val` should have type json but type was large_binary ``). A null-typed input column now becomes typed nulls matching the table's field exactly, metadata included. **Unlabelled JSON text was stored raw.** JSON supplied as plain strings (what pyarrow infers for a column of `str`) was cast to the column's `LargeBinary` storage type and relabelled `lance.json`, putting unparsed text where JSONB was expected. Reads returned the text unnormalized and `json_extract` failed with `InvalidJsonb`. Lance-core does the JSONB encoding, but only for input labelled `arrow.json`, so string input is now labelled rather than cast — at the top level and inside structs. Both fixes are in the shared Rust write path, so they apply to any binding, including hand-built Arrow tables that never pass through Python's list-of-dicts type inference. `_align_field` gets the same JSON-string fix for the legacy Python `_sanitize_data` path, which `on_bad_vectors` and embedding functions still route through. The blob v2 half of the issue landed separately in #4065, which added a `DataType::Null` arm to blob coercion. This PR keeps that implementation and adds end-to-end add-path coverage for it. The tests from #4066 are included here and pass, so that PR's Python-layer inference changes are no longer needed to close the issue. Fixes #3759 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Lance Release
|
95055c4c54 | Bump version: 0.39.0-beta.8 → 0.39.0-beta.9 | ||
|
Bruno RamirezandClaude Sonnet 5
|
3fca33fcb5 |
fix: shut down the shared Tokio runtime on interpreter exit (#4175)
Short-lived Python processes using this client can occasionally crash with SIGABRT during interpreter shutdown, even after every operation they ran completed successfully. The cause is the shared Tokio runtime backing every async call: it's never told to shut down at normal process exit, only reset (and deliberately leaked) on `fork()`. Its worker threads keep running, uncoordinated with the interpreter, until the process actually ends, and if one is mid-task exactly as `Py_Finalize` starts tearing down interpreter state, it can panic on state that's already gone. That panic happens on a background thread with no PyO3-wrapped call frame to catch it, so Rust aborts the whole process instead of just failing that one call. This PR gives the runtime a coordinated, bounded shutdown by registering a Python `atexit` callback that runs while the interpreter is still fully valid. Getting the exit lifecycle right took a few rounds of review. Earlier versions freed the runtime as soon as `Arc::strong_count` looked low, but that's the wrong signal — it reflects who currently holds a reference, not who's logically still in flight. That mistake showed up three ways: a caller could dereference memory already freed out from under it; an install already in progress could finish invisibly after `shutdown()` had already decided there was nothing to do; and a spawned task could end up as the final owner of the `Runtime`, so completing it dropped the runtime from inside one of its own worker threads, which Tokio itself forbids and panics on (this reproduced unprompted in this branch's own test suite). Fixing all three meant replacing reference-count-based tracking with an explicit counter of in-flight top-level calls that `shutdown()` waits on directly. This was accomplished with the following changes: - The runtime lives in an `ArcSwapOption<Tagged>`, where `Tagged` pairs the `Runtime` with the fork generation it was built in. - An `OUTSTANDING` counter, incremented before a top-level `spawn`/`spawn_blocking`/`block_on` call does anything else and decremented only once it has truly finished (via an `OutstandingGuard` token that carries no reference to the runtime), is what `shutdown()` waits on — not `Arc::strong_count` or whether the slot looks empty. This closes the install-race and makes it impossible for a task's own completion to be the runtime's final drop. - Once `shutdown()`'s bound elapses, it stops waiting and attempts retirement anyway, rather than returning with the runtime and its workers left fully alive. - `spawn`/`spawn_blocking` use `Handle::try_current()` to pin any nested spawn (`future_into_py` spawns a task that itself spawns a second one for the real work) to whichever runtime is already executing it, so a reclaim landing between the two calls can't split one logical operation across two different runtime instances. - The fork-child handler now only bumps a bare `GENERATION` counter — no `ArcSwapOption` call of any kind from that context, since `swap`/`compare_and_swap` do real reader-reconciliation work (thread-local state, potentially an allocation) that isn't safe in a forked child. `get_runtime()` compares its installed runtime's generation against the live counter from ordinary context and rebuilds on a mismatch. - Registered `shutdown_runtime` as a Python `atexit` callback in the `_lancedb` module init, running with the GIL released (`Python::detach`) since the bounded wait could otherwise deadlock against any in-flight task that itself needs the GIL. ### Testing - Unit tests in `runtime.rs` cover: shutdown with no runtime created, shutdown after use and lazy rebuild afterward, calling shutdown twice in a row, a concurrent stress test racing many threads against shutdown, a nested-spawn test reproducing `future_into_py`'s own spawn-within-a-spawn shape under concurrent shutdown, a test confirming a top-level task in flight survives a concurrent shutdown reclaim, and a test forcing the install-vs-shutdown race directly. - Built the wheel and ran a concurrent reproducer (many threads hammering the client while `atexit` fires) over 100 times with no hangs or crashes, plus a 30-second-join variant and repeated runs of a short-lived process confirming clean exits with no added latency. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack Ye
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
|
Wyatt Alt
|
2d4622491e |
fix: keep computed-column freshness across branches and clones (#4188)
A branch or shallow clone recomputed every computed column on its first refresh: the input signature carried each data file's raw base id, which the clone commit rewrites, and the signature sidecar was looked up under the branch's own tree, where nothing was ever written. A file is now identified by where its store resolves it, so the source's own root and a clone's registered reference to that root sign the same while different bases stay distinct; compaction products are matched the same way. Sidecars live in one `_computed/` at the table root shared by main and its branches; a shallow clone reads through the base it was cloned from and keeps a copy. Pruning collects references across every branch, and the staleness walk treats the versions a branch does not hold as unknown. Stamps written before this change no longer match, so the first refresh after upgrading recomputes once; a deep clone, which copies no sidecar, still starts over. |
||
|
Jack YeandXuanwo
|
2f88b71c21 |
feat: add persistent OAuth token cache and session APIs (#4182)
Stacked on #4173 (diff includes it until that merges; will rebase after). Addresses the token-cache part of [Colin's review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100). Adds an explicit, opt-in persistent OAuth token cache shared by Rust, Python, and Node clients, plus `login` / `status` / `logout` session APIs, so short-lived processes (CLIs, scripts, notebooks) reuse one session instead of restarting a browser or device flow on every start. - **Opt-in and minimal**: existing callers stay memory-only and lazy. Only refresh tokens are persisted (never access tokens, never client secrets), so there are no local token-expiry decisions to get wrong when clocks move. Each process start performs one silent refresh grant. - **Hardened file backend**: private directory (`0700`), per-record files (`0600`), owner validation, symlink rejection, and atomic `rename` replacement. Corrupt, truncated, unknown-version, or permission-invalid records fail with actionable errors naming the file. Native keyring backends were evaluated (keyring crate routes Linux through D-Bus/zbus: heavy deps, headless/CI flakiness) and are deferred; the file store is the explicit opt-in, not a downgrade from a keyring. - **Cache key**: SHA-256 of the canonical identity (issuer, client ID, sorted/de-duplicated scopes, flow, public/confidential), so no secret appears in a filename and distinct identities never collide. Versioned record schema (`version: 1`). One record per identity: last login wins, documented. - **Cross-process rotation locking**: per-key `fs4` file lock (`flock` / `LockFileEx`) around the refresh critical section — acquire, reread the durable record, refresh exactly once, atomically store the rotated refresh token, release. The OS releases locks on process death, so crashes cannot strand stale locks. Only confirmed `invalid_grant`/`invalid_token` deletes a record and reauthenticates; transport, 5xx, 429, and parse failures retain it. - **Session APIs**: `OAuthSession::login/status/logout` in Rust, `lancedb.remote.OAuthSession` (async) in Python, `OAuthSession` class in Node. `status` returns non-secret metadata only. `logout` removes only the local credential — provider revocation (RFC 7009) is a deliberate follow-up, and local logout never terminates browser SSO. Azure managed identity is rejected for persistence (machine identity stays in memory); client credentials have nothing refreshable to persist and stay memory-only. - No CLI binary exists in this repo, so this ships library APIs plus doc examples in all three languages. Tests: Rust unit + mock-IdP integration (cache-key canonicalization/separation, record versioning/corruption/truncation/symlink/owner/perms, lock serialization + release, two concurrent providers proving no `invalid_grant` and correct rotation, transient-failure retention, `invalid_grant` delete + reauthenticate, login/status/logout lifecycle, client-credentials no-op, IMDS rejection, secret redaction); Python lifecycle + a true two-subprocess cross-process reuse test (second process refreshes once, never hits the device endpoint); Node lifecycle + device-flow login test. Local builds were skipped in development; CI validates all bindings. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jack Ye
|
575286922b |
fix: redact authentication headers from debug logs (#4179)
Prevent bearer tokens, API keys, cookies, and proxy credentials from appearing in request debug output through a reusable `redact_sensitive_headers` utility. The utility is applied after default/configured header construction and dynamic header merging, so it also covers retry requests. Invalid dynamic-header values are no longer echoed, and regression coverage verifies that credentials are redacted while ordinary diagnostic headers remain visible. Extracted from the OAuth discussion in #4173, following [Colin's review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100). |
||
 dependabot[bot]andXuanwo
|
09e5418943 |
build(deps): bump the rust-minor-patch group across 1 directory with 5 updates (#4178)
Bumps the rust-minor-patch group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [uuid](https://github.com/uuid-rs/uuid) | `1.26.0` | `1.26.1` | | [serde_with](https://github.com/jonasbb/serde_with) | `3.22.0` | `3.23.0` | | [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.4.8` | `1.6.3` | | [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.3` | `3.6.5` | | [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.1` | `2.4.2` | Updates `uuid` from 1.26.0 to 1.26.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/uuid-rs/uuid/releases">uuid's releases</a>.</em></p> <blockquote> <h2>v1.26.1</h2> <h2>What's Changed</h2> <ul> <li>Seat the v7 counter below the version nibble by <a href="https://github.com/lenamonj"><code>@lenamonj</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/907">uuid-rs/uuid#907</a></li> <li>Don't panic in overflowing Timestamp to SystemTime conversion by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/909">uuid-rs/uuid#909</a></li> <li>Prepare for 1.26.1 release by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/910">uuid-rs/uuid#910</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/lenamonj"><code>@lenamonj</code></a> made their first contribution in <a href="https://redirect.github.com/uuid-rs/uuid/pull/907">uuid-rs/uuid#907</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1">https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/uuid-rs/uuid/commit/9f927126c89892ddfed6cd2f92df16852f3f9aa6"><code>9f92712</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/910">#910</a> from uuid-rs/cargo/v1.26.1</li> <li><a href="https://github.com/uuid-rs/uuid/commit/d4df8f0cd9f461b4ef493254420052ffa5ce6277"><code>d4df8f0</code></a> prepare for 1.26.1 release</li> <li><a href="https://github.com/uuid-rs/uuid/commit/5613f2357c1fc06afc5fffd98e96da2ccf25a608"><code>5613f23</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/909">#909</a> from uuid-rs/fix/ts-conversion-overflow</li> <li><a href="https://github.com/uuid-rs/uuid/commit/fda00eba938383d242bad33143c8af73227f2a2c"><code>fda00eb</code></a> don't panic in overflowing Timestamp to SystemTime conversion</li> <li><a href="https://github.com/uuid-rs/uuid/commit/c82e88ca184e4ab83ce6b4ac0be33d32a0b9c3c4"><code>c82e88c</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/907">#907</a> from lenamonj/v7-counter-placement</li> <li><a href="https://github.com/uuid-rs/uuid/commit/ac065a6c17389a67dc6e98ef5f9a2e1d7ad4a670"><code>ac065a6</code></a> Align the counter diagram</li> <li><a href="https://github.com/uuid-rs/uuid/commit/34ec10208d813672928c299146dfe4c18cedcec7"><code>34ec102</code></a> Seat the v7 counter below the version nibble</li> <li>See full diff in <a href="https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1">compare view</a></li> </ul> </details> <br /> Updates `serde_with` from 3.22.0 to 3.23.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/jonasbb/serde_with/releases">serde_with's releases</a>.</em></p> <blockquote> <h2>serde_with v3.23.0</h2> <h3>Changed</h3> <ul> <li>Update <code>syn</code> and <code>darling</code> dependencies to use <code>syn</code> v3 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/992">#992</a>)</li> <li>Update dev-dependencies to newer versions (<a href="https://redirect.github.com/jonasbb/serde_with/issues/993">#993</a>)</li> <li>Update <code>base64</code> to a newer version. This should not have any API change, but some error messages might change. (<a href="https://redirect.github.com/jonasbb/serde_with/issues/993">#993</a>)</li> <li><code>serde_as</code> can now parse <code>cfg_attr(true, ...)</code> and <code>cfg_attr(false, ...)</code> (<a href="https://redirect.github.com/jonasbb/serde_with/issues/995">#995</a>) <code>true</code>/<code>false</code> are new literals as of Rust 1.88 but need to be parsed explicitly with the <code>syn</code> types. This is used when emitting <code>schemars</code> annotations.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/jonasbb/serde_with/commit/ea5dfdc6fd1b4732188519e871e2fd2a8fe49f88"><code>ea5dfdc</code></a> Bump version to v3.23.0 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1005">#1005</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/e52e85b1db55e4dc6305b6b19c2710d6a8b2d433"><code>e52e85b</code></a> Bump version to v3.23.0</li> <li><a href="https://github.com/jonasbb/serde_with/commit/39955b6954796d963d02d2f28e0a22087c48fcd4"><code>39955b6</code></a> Bump rmp dev-dependency (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1004">#1004</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/b51e5fb877c642c47da36c296c222e1d172d75da"><code>b51e5fb</code></a> Bump rmp dev-dependency</li> <li><a href="https://github.com/jonasbb/serde_with/commit/ef598c6683d591851e09cac3a4bd767c93bacb9e"><code>ef598c6</code></a> Use setup-rust-toolchain v2 instead of v1 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1003">#1003</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/9ea2658f45e763369f18078d80ea3b109d491de8"><code>9ea2658</code></a> Fix cargo lint about workspace lints being inherited in the test crate</li> <li><a href="https://github.com/jonasbb/serde_with/commit/e5ad81af7721bb770772e8e28de32437f3ff217f"><code>e5ad81a</code></a> Use setup-rust-toolchain v2 instead of v1</li> <li><a href="https://github.com/jonasbb/serde_with/commit/81001414f153d18fc1aca6c2f52990c557f90471"><code>8100141</code></a> Bump the github-actions group across 1 directory with 2 updates (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1002">#1002</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/940f4a62a418784e23026953db8a8fed577bff6d"><code>940f4a6</code></a> Bump jsonschema from 0.49.8 to 0.52.0 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1001">#1001</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/6f42b94d7bb71b004d0e591a26fbf8d900878c91"><code>6f42b94</code></a> Bump the github-actions group across 1 directory with 2 updates</li> <li>Additional commits viewable in <a href="https://github.com/jonasbb/serde_with/compare/v3.22.0...v3.23.0">compare view</a></li> </ul> </details> <br /> Updates `aws-smithy-types` from 1.4.8 to 1.6.3 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Updates `napi-derive` from 3.6.3 to 3.6.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-derive's releases</a>.</em></p> <blockquote> <h2>napi-derive-v3.6.5</h2> <h3>Other</h3> <ul> <li>update Cargo.toml dependencies</li> </ul> <h2>napi-derive-v3.6.4</h2> <h3>Fixed</h3> <ul> <li><em>(deps)</em> update rust crate convert_case to 0.12 (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3469">#3469</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/1492b220d5ad01807b2dcbd250e8383f9d738311"><code>1492b22</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3502">#3502</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/828983aab9b93f516275625f89905fffca459780"><code>828983a</code></a> fix(napi): return errors from the serde deserializer for unexpected JS value ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/606b14238b8d82578705161c0b0d6b6f4b7c2556"><code>606b142</code></a> fix(napi): validate wrapped payload provenance in Object::unwrap/remove_wrapp...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/a75d89f85c0a9a0ffd02b8bb11a2c0897b72a2ae"><code>a75d89f</code></a> fix(napi): point from_external slices at the engine-owned copy after finalize...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/e414c8c21924a4c47bf1c8c5d6ad244dac479578"><code>e414c8c</code></a> fix(deps): update dependency obug to v3 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3499">#3499</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/a5fedde2597dc3ac17c7d422f158adce5fec59bf"><code>a5fedde</code></a> chore(deps): update release-plz/action action to v0.5.136 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3505">#3505</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/31c27a1676a7c4b317f4e144e0a9cb94e8354143"><code>31c27a1</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3470">#3470</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/7e3f293e2d6a3032eabfe51ff38bcaa82d342a2f"><code>7e3f293</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/f772ee0aabf4dbb27250a8df47a73463e8f78cf4"><code>f772ee0</code></a> fix(cli): align generated file formats (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3501">#3501</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1cf5ec573637d96dddb77b1a0ae6aaf316739bd0"><code>1cf5ec5</code></a> fix(cli): use accessible WASI preopen root on Android (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3485">#3485</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.3...napi-derive-v3.6.5">compare view</a></li> </ul> </details> <br /> Updates `napi-build` from 2.4.1 to 2.4.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-build's releases</a>.</em></p> <blockquote> <h2>napi-build-v2.4.2</h2> <h3>Fixed</h3> <ul> <li><em>(cli,build)</em> make wasm32-wasip1-threads link with wasi-sdk 34 and Rust nightly (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3492">#3492</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/31c27a1676a7c4b317f4e144e0a9cb94e8354143"><code>31c27a1</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3470">#3470</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/7e3f293e2d6a3032eabfe51ff38bcaa82d342a2f"><code>7e3f293</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/f772ee0aabf4dbb27250a8df47a73463e8f78cf4"><code>f772ee0</code></a> fix(cli): align generated file formats (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3501">#3501</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1cf5ec573637d96dddb77b1a0ae6aaf316739bd0"><code>1cf5ec5</code></a> fix(cli): use accessible WASI preopen root on Android (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3485">#3485</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/cf9245d0b367a8fc2b3c1209269832b9f076de32"><code>cf9245d</code></a> chore(deps): update vitest monorepo to v5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3481">#3481</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/78fa3ad79d38b52e7c49f9a86ba2e57f46805d1b"><code>78fa3ad</code></a> fix(macro): recognize fully-qualified napi::Env as the special Env parameter ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/37283c75a1f1f20d166a8f9f641ada43d04d24a5"><code>37283c7</code></a> chore(deps): lock file maintenance (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3474">#3474</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1932d2f42198cb9f79704aba66302a0956590c48"><code>1932d2f</code></a> chore(deps): update release-plz/action action to v0.5.135 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3500">#3500</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/0238e8b1b8fcafa2261d4d1b96e7568dd0fb1fc3"><code>0238e8b</code></a> fix(deps): update dependency js-yaml to v5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3344">#3344</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/9ed9d3509e1c2cab2cd89dcff04963adaad4cb7d"><code>9ed9d35</code></a> chore(deps): update dependency electron to v44 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3468">#3468</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.1...napi-build-v2.4.2">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
21ecafe9ae |
chore: update lance dependency to v13.0.0-beta.1 (#4183)
Update the Rust workspace Lance dependencies and Java lance-core from v12.0.0-beta.18 to [v13.0.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.1), and refresh Cargo.lock; no compatibility fixes were required. Validation passed: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. Also fixes the flaky Node test `when optimizing a dataset › cleanups old versions` that failed the NPM Publish Linux test jobs on this PR. The test captured `new Date()` (millisecond precision) in the same millisecond as the last commit, while Lance compares version timestamps at nanosecond precision, so that version was not pruned. The test now waits for the clock to tick to the next millisecond before taking the cutoff. This is a pre-existing flake on `main` since #4160, unrelated to the Lance upgrade. --------- Co-authored-by: Yang Cen <bubble-cal@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Jack YeandXuanwo
|
56bbd19ff4 |
test(python): cover namespace operations without pylance (#4174)
The "Test without pylance or pandas" CI job only ran `test_table.py`, so the regression fixed in #3606 had no guard: sync namespace operations used to route through the Python `lance_namespace` client, whose `dir` implementation ships in the optional `pylance` extra, so `lancedb.connect(path).list_namespaces()` failed with `No module named 'lance'` while the async API worked. Adds `python/python/tests/test_namespace_no_pylance.py` and runs it in that job. Its `without_pylance` fixture blocks `lance` imports, so the guard also fires in environments that do have `pylance` installed. Coverage: the original reproducer, nested namespace lifecycle, namespaced and root table lifecycle, the async path, and the one API that legitimately still needs `pylance` (`namespace_client()`). Verified the guard actually catches the regression: against `lancedb==0.33.0`, three of these tests fail with the original error; against a fixed build all pass. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 lancedb-gatefixer[bot]andXuanwo
|
ffe94a65a1 |
fix(node): preserve optimize cleanup timestamp (#4160)
<!-- lance-gatekeeper-fix:v1 agent=278cb095b2e5d69051442bc254d803a6 generation=1 --> ## Summary - pass the TypeScript `cleanupOlderThan` date to the native binding as an unchanged epoch timestamp - prune with Lance's absolute `before_timestamp` policy so dispatch and compaction time cannot move the cutoff - retain versions created after the supplied cutoff and document that behavior - add boundary and end-to-end regression coverage ## Root cause The TypeScript layer converted the absolute date into an elapsed duration before calling native optimize. Lance converted that duration back into a timestamp only after compaction, which silently advanced the requested cutoff and made the cleanup count depend on a millisecond timing boundary. ## Validation - `cargo fmt --all` - `cargo clippy --quiet --features remote --tests --examples -p lancedb -p lancedb-nodejs` - `pnpm build` - `pnpm lint` - `pnpm run docs` - `pnpm test __test__/table.test.ts --runInBand` (309 passed) Fixes #4159 --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jack Ye
|
37771fd4fc |
fix(deps): update rustls and cap aws-smithy-types to unbreak CI (#4177)
Two upstream dependency releases broke CI on `main`. Both fixes are dependency constraints, so they ride together. ## `deny` — RUSTSEC-2026-0285 rustls 0.23.40 accepts TLS 1.3 handshake messages sent at the wrong encryption level ([advisory](https://rustsec.org/advisories/RUSTSEC-2026-0285)), patched in 0.23.45. rustls 0.23.45 requires `aws-lc-rs >= 1.18`, which the nodejs crate pinned to `=1.16.3`, so this also bumps that pin and its `aws-lc-sys` companion to `=1.18.1` / `=0.45.0`. The pin comment already calls for periodic updates on security patches. The workspace's other rustls (0.21.12) is below the advisory's affected range (`unaffected = ["< 0.23.13"]`). ## `build-no-lock` — aws-smithy-types 1.7.0 `aws-smithy-types` 1.7.0 and `aws-smithy-json` 0.64.0 both released 2026-09-14. 1.7.0 made `Document` `non_exhaustive`, which `aws-smithy-json` 0.63 does not compile against: ``` error[E0004]: non-exhaustive patterns: `&_` not covered --> aws-smithy-json-0.63.0/src/serialize.rs:36:15 note: `aws_smithy_types::Document` defined here --> aws-smithy-types-1.7.0/src/document/mod.rs:91:1 ``` Every `aws-sdk-*` crate moved to `aws-smithy-json ^0.64`, but `aws-config` 1.12.0 still requires `^0.63`, so a lockfile-free resolve pairs json 0.63.0 with types 1.7.0 and fails. This caps `aws-smithy-types` below 1.7 as a constraint-only dev-dependency, matching the existing `aws-smithy-runtime` entry. Revert once `aws-config` moves to `aws-smithy-json` 0.64. Note this break is not specific to this PR — `build-no-lock` fails the same way on unrelated branches (e.g. `jon/secrets-client-api` run 34903587364), which passed it hours earlier. ## Verification Resolution only, no local build: - Locked resolve unchanged: `aws-smithy-types` stays 1.4.8; the only `Cargo.lock` delta from the cap is the new dev-dep edge. - Fresh resolve (`rm Cargo.lock`): `aws-smithy-json` 0.63.0 with `aws-smithy-types` 1.6.3, `aws-sdk-*` one release back, `rustls` 0.23.45 retained. |
||
|
Lance Release
|
c44b192334 | Bump version: 0.39.0-beta.7 → 0.39.0-beta.8 | ||
|
Wyatt Alt
|
7575c2597a |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
|
Wyatt Alt
|
0665575a76 |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
 陈志谦andXuanwo
|
0113cee489 |
docs(embeddings): correct the documented max_retries default (#4145)
`lancedb/embeddings/utils.py` documents `max_retries` with "(default is 10)" — the signature default is `7`. Docs-only; conventional title per the contribution guide. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
6bb64c3edb |
fix: reject repeated job pagination tokens (#4139)
Fixes #4138 `RemoteDatabase::list_jobs` followed every returned pagination token without remembering previously seen values. A server-side token cycle therefore caused repeated requests and duplicate accumulation until the 100-page safeguard returned partial results as a success. This change tracks non-empty job-list page tokens and returns an HTTP-context error as soon as a token repeats, matching the existing `list_functions` behavior. A mock-handler regression test verifies a repeated `loop` token is rejected after two requests. Validation: - `cargo test --quiet --features remote -p lancedb test_list_jobs` - `cargo fmt --all` - `cargo check --quiet --features remote --tests --examples` <!-- lance-gatekeeper-fix:v1 agent=ddd56389737a405d32cf4f43c7696032 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
255da8a854 |
fix(python): resolve native job types in API docs (#4170)
Native job metadata types report `builtins` as their module, so Griffe cannot resolve the public `lancedb.job` re-exports and the Python API reference build fails. Set the PyO3 module metadata for `JobInfo`, `JobDescription`, and `JobFailureInfo`, and cover import resolution in the existing package metadata tests. Reproduced the failure and validated the fix with the docs CI toolchain (`griffe==0.49.0`, `mkdocstrings==0.25.2`, and `mkdocstrings-python==1.10.9`). After rebuilding the native extension, the full `PYTHONPATH=. mkdocs build` succeeds and all three classes and their public members appear in the generated reference. |
||
|
Lance Release
|
ec410e015a | Bump version: 0.39.0-beta.6 → 0.39.0-beta.7 | ||
|
Jack Ye
|
9fe10c7362 |
fix(remote): align Function CRUD routes (#4166)
Align the experimental Function HTTP transport with the equivalent Table
CRUD API shape. This is an intentional breaking change to the
experimental Function routes; public Rust and Python APIs remain
unchanged.
## Route comparison
| Operation | Function before | Function after | Equivalent Table API |
| --- | --- | --- | --- |
| Create | `POST /v1/functions/create` | `POST /v1/function/{id}/create`
| `POST /v1/table/{id}/create` |
| Describe | `POST /v1/functions/describe` | `POST
/v1/function/{id}/describe` | `POST /v1/table/{id}/describe` |
| List | `POST /v1/functions/list` | `GET
/v1/namespace/{id}/function/list` | `GET /v1/namespace/{id}/table/list`
|
| Drop | `POST /v1/functions/drop` | `POST /v1/function/{id}/drop` |
`POST /v1/table/{id}/drop` |
## Contract details
- Create, describe, and drop use a singular resource path. Their `{id}`
path parameter is the URL-encoded Function name, and the duplicate
Function identifier is removed from each request body.
- Create continues to accept `202 Accepted`.
- List changes from a POST with a JSON body to a namespace-scoped GET.
Its `{id}` path parameter is the namespace identifier rather than a
Function name.
- Functions do not support nested namespaces yet, so the client lists
against the root namespace identifier (`$` with the default delimiter).
A non-root namespace is rejected.
- The optional list filter is named `name`. `limit`, `page_token`, and
`include_definition` remain available as query parameters.
- The paginated list response shape is unchanged.
|