rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-09-22 13:05:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,012 Commits 333 Branches 540 Tags
01ee01dbc856f75bba4a1d1e63835f408b04535e
Commit Graph
175 Commits
Author SHA1 Message Date
Wyatt Alt 01ee01dbc8 feat: define a materialized view by its query, with Functions in FROM position (#4190) 
A view definition was a structured record under a `kind` tag, one kind
per query shape, and a Function returning `list<struct>` was about to
add a third. That names shapes instead of describing a relation.

A materialized view is now a relation defined by a query, stored as one
canonical SQL string under a format number:

```sql
SELECT columns FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias]
[WHERE predicate] [LIMIT n]
```

Any other clause is refused at parse time. Older readers report the view
as unrefreshable, the pre-format layouts still read, and a legacy view
is rewritten on its next refresh that commits, rebuilt only where its
raw text meant something else under lance's parser.

A Function in FROM position yields one row per element it returns, as a
table function does in any dialect. The server stages its list output in
a hidden table and records that binding beside the query, which stays as
the user wrote it; refresh scans the staging and unnests the column, the
same operator as UNNEST over a list column the table already holds. Row
ids repeat per element, so eviction and incremental append are
unchanged. A local database refuses a Function in FROM position, since
it has no executor.
 2026-09-18 12:16:25 -07:00
Wyatt Alt 5231d37f8d feat: list a table's per-row Function errors from the client (#4208) 
A refresh running under a skip policy records each row it skipped, with
the failing input and the error, but the client could not read that
store: the server exposes it over SQL and, since recently, a REST route.
A user who hit per-row failures still had to open a SQL session.

`Table::function_errors` calls the route. The listing is table-addressed
with optional job and column filters, the same addressing the SQL
surface uses, so the two cannot disagree about what a table's errors
are. The two non-record signals come back as their own fields rather
than as rows: capped-fragment summaries, and whether the listing stopped
at its limit. Local tables refuse rather than answer with an empty list.

Python and Node expose the same call, with the same optional filters.
 2026-09-17 05:12:50 -07:00
Jonathan HsiehandClaude Opus 5 60a1b4c219 feat(secrets): named Secrets, bindings, and namespace addressing (#4150) 
Adds the client half of database-scoped named Secrets: a Secret is a
name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads. Secrets are addressed by
a
namespace path plus a name.

The UDF body is unchanged and stays portable — it reads `OPENAI_API_KEY`
the
way it always did, and the binding is what puts a value there:

```python
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
function = db.create_function(
    analyze_caption,
    secrets=[
        EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
    ],
)
function.secret_bindings   # the Secret's name, never its value
```

- `create_secret` / `alter_secret` / `list_secrets` / `describe_secret`
/
`drop_secret` on sync, async and remote connections, with the pyo3
binding
and the Rust client behind them. Each takes `namespace_path`
keyword-only,
  defaulting to the root.
- **There is no read API, by construction rather than by policy** — no
code
  path returns a stored credential, and `describe_secret` answers with
  metadata only.
- `EnvVarSecret` is a pure local constructor: it contacts no server, so
it
cannot fail on a Secret that does not exist. It exists so that a bare
string
in that position — which would be a credential — is a `TypeError` rather
  than a plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`: a list of `SecretBinding` tagged by `kind`, so a
later
delivery mode is a variant rather than a sibling field. The value never
travels — it is resolved by the service when the Function runs, which is
what
lets a rotation reach columns already pinned to an older
FunctionVersion.
- A binding names its Secret as a `SecretReference` of `{name,
namespace_path}`
rather than one joined string, so no delimiter has to be excluded from
every
name and segment forever, and `ClientConfig.id_delimiter` cannot
contradict
  an identity built on a fixed separator.
- A root namespace is omitted from the request body rather than sent
empty, so
  a root request is byte-identical to one from a client that predates
  namespaces. Tests pin it.

This is the client surface the design's §4 describes; the service side
lives in
sophon.

**Previously split across two PRs.** Namespace addressing was #4151,
stacked on
this one; it is folded in here so the Secret identity contract — name,
namespace path, and the binding that carries both — is reviewable as one
piece
rather than as a shape introduced and then replaced.

## Identifier safety, merged from #4189

**#4189 is merged into this branch**, so the client half of Secrets and
the
guards on the identity it puts in the URL are one PR. What it added:

- Components are checked where the identifier is built, before a request
is
  constructed. `create_secret("../jobs", value)` no longer resolves to
`/v1/jobs/create` and delivers a credential-bearing body to a route with
none
  of this one's body suppression.
- Each component is percent-encoded and joined by the delimiter, so
nothing
  inside a component can end the path segment or add one.
- A component may not be empty, a relative segment (`.`, `..`, and their
`%2e`
spellings), or the delimiter itself — the three ways a component erases
a
boundary the split has to recover. `["prod", ""]` joined to `prod$`,
which
  reads back as `["prod"]`.
- `$` is the only accepted `id_delimiter`, refused at client
construction.
`ClientConfig.id_delimiter` remains, since the identifier grammar comes
from
the Lance REST catalog standard, but a value that would produce
identifiers no
  service splits the caller's way is now an error where it was written.
- One `build_object_identifier` and one character set serve tables,
namespaces,
  Secrets, Functions and materialized views.

Components are checked for *addressability*, not a character set: the
name's
own grammar stays each object's own, so a catalog database keeps the `/`
that
`RemoteCatalog::validate_name` allows.

## Known shortcoming

`secret_bindings` is omitted from a registration body when empty, so a
client
that binds nothing sends what a client without bindings sends. When a
client
does bind a Secret and the service does not know the field, the field is
ignored: registration succeeds, the returned version carries no
bindings, and
the Function fails at execution with the variable unset, far from the
call that
asked for it.

`ServerVersion` is how this codebase refuses a feature the service is
too old
for, and it gates five features already. It does not gate this one: it
is held
per table, and registering a Function is a database-level call. Noted at
the
field in `remote/db.rs`; wiring the gate is follow-up work.

**Tests:** lancedb lib 1340 passed, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, plus Python tests across both slices.
Rebased onto `main` after #4176 (OCI Function identity), #4191 (`.`/`..`
table
names) and #4195 (remote catalogs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 17:00:23 -07:00
Jack YeandXuanwo 6a07f88980 feat: add remote catalogs and Python and TypeScript bindings (#4195) 
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.

## Rust API

`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.

```rust
use lancedb::catalog::{
    CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};

let catalog = lancedb::connect_catalog("https://my-server.example")
    .api_key("my-api-key")
    .execute()
    .await?;

let db = catalog.create_database(
    CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
    ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
    DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```

Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.

## Python API

```python
import lancedb

catalog = lancedb.connect_catalog(
    "https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
    next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```

`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.

## TypeScript API

```typescript
import { connectCatalog } from "@lancedb/lancedb";

const catalog = await connectCatalog("https://my-server.example", {
  apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
  const nextPage = await catalog.listDatabases({
    limit: 20, pageToken: page.pageToken,
  });
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```

Create/connect return the existing `Connection` API. All four methods
are asynchronous.

## REST mapping

All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.

| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |

For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.

Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.

For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.

Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.

Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.

## Scope

This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.

The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-17 01:51:10 +08:00
Jack YeandXuanwo f3ef21b8ca feat: use oauth2 crate for OAuth with configurable client auth (#4181) 
Stacked on #4173 (`jack/restore-oidc-flows`, base branch mirrored to
this repo so the diff shows only this change); context from review:
https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100.
Rebase to `main` once #4173 and #4179 merge.

## What moved to the `oauth2` crate (5.0, no default features)

- Authorization URL generation and CSRF state (`authorize_url`,
`CsrfToken`)
- PKCE S256 challenge/verifier generation and code exchange
- Client-credentials, authorization-code, refresh-token, and device-code
grant request construction
- Device authorization request and the device token polling loop
(`authorization_pending`, `slow_down` +5s, expiry deadline, denial,
network backoff capped at 10s)
- Standard success/error response parsing (`RequestTokenError`)
- Token-endpoint client authentication and standards-compliant parameter
encoding (RFC 6749 2.3.1 Basic encoding)

LanceDB keeps ownership of OIDC discovery (compared `openidconnect`: no
measurable win for our 3-field metadata + strict validation, at real
dependency cost), HTTPS-or-loopback endpoint enforcement, the loopback
callback server, browser/stderr prompts, token caching and refresh
orchestration, and the dedicated hardened Azure IMDS source, which is
unchanged.

## Client authentication methods

New `ClientAuthMethod` enum (`none` | `client_secret_basic` |
`client_secret_post`), exposed in Rust, Python, and Node. Unset resolves
to `client_secret_basic` when a secret is present (RFC 6749 2.3.1
recommendation and the normal Okta confidential-app default, so a
default Okta app works without weakening its configuration) and to
`none` for public clients (PKCE/device). Explicit `none` with a secret,
or basic/post without one, is rejected. The method applies to client
credentials, code exchange, refresh, and device requests. Deliberate
behavior change: confidential clients previously always sent the secret
in the POST body; they now default to Basic (Keycloak accepts both).

No `audience`/`resource` parameters were added: the supported target is
an Okta custom authorization server with the API audience configured
server-side, so client-provided audience parameters are unnecessary;
`add_extra_param` support exists if a concrete provider contract ever
needs them.

## Device polling behavior changes (deliberate, tested)

- The first token poll now happens immediately rather than after one
interval (RFC 8628 allows both).
- Transient failures (HTTP 429, 5xx, `temporarily_unavailable`, network
errors) now retry with exponential backoff capped at 10s instead of
retrying at the fixed interval; polling never spins faster than once per
second even if a server reports a zero interval.

## Security and compatibility

- Issuer and discovered endpoints (and device verification URIs) still
require HTTPS except explicit loopback HTTP, enforced before any crate
URL type is built
- Token HTTP client keeps the hardened redirect policy that refuses
insecure redirect targets; regression test added
- Errors never embed raw response bodies (avoids leaking tokens through
parse failures); all credential types stay redacted in Debug
- Transient conditions (429/5xx/`temporarily_unavailable`) remain
retryable in device polling and hard errors elsewhere; refresh keeps
rotation and reauthentication semantics
- Existing public APIs stay source-compatible except the added
`OAuthConfig.client_auth_method` field

## Tests

Rust: client-auth methods across code
exchange/refresh/client-credentials/device (none/basic/post),
auth-method resolution and validation, transient device retries,
denial/expiry, redirect rejection, malformed-response leak check, PKCE
URL assertions, redaction. Python and Node: enum values, conversion,
unknown-method errors, config defaults.

Manual Okta validation recipe (no automated Okta credentials): create a
custom authorization server with an API audience, one confidential web
app (Basic) for authorization-code, one native app (PKCE, no secret),
one native device app; point `issuer_url` at the custom server, set
`client_auth_method` only for the POST-required case; verify token
acquisition, refresh after expiry, and `x-lancedb-credential-type: oidc`
against a LanceDB deployment. Never commit tenant URLs or secrets.

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-16 20:11:08 +08:00
Colin Patrick McCabe 3a1d3be256 feat(oidc): support resource and audience (#4193) 
Support configuring resource and audience for OAuth authorization, token
exchange, and refresh requests.
 2026-09-15 16:35:37 -07:00
Jack Ye f12996557f feat(remote): support materialized view APIs (#4180) 
## Summary

Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.

Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.

## Route comparison

| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |

Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.

## Python API changes

| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |

The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
 2026-09-15 12:13:39 -07:00
Jack YeandXuanwo 2f88b71c21 feat: add persistent OAuth token cache and session APIs (#4182) 
Stacked on #4173 (diff includes it until that merges; will rebase
after). Addresses the token-cache part of [Colin's
review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100).

Adds an explicit, opt-in persistent OAuth token cache shared by Rust,
Python, and Node clients, plus `login` / `status` / `logout` session
APIs, so short-lived processes (CLIs, scripts, notebooks) reuse one
session instead of restarting a browser or device flow on every start.

- **Opt-in and minimal**: existing callers stay memory-only and lazy.
Only refresh tokens are persisted (never access tokens, never client
secrets), so there are no local token-expiry decisions to get wrong when
clocks move. Each process start performs one silent refresh grant.
- **Hardened file backend**: private directory (`0700`), per-record
files (`0600`), owner validation, symlink rejection, and atomic `rename`
replacement. Corrupt, truncated, unknown-version, or permission-invalid
records fail with actionable errors naming the file. Native keyring
backends were evaluated (keyring crate routes Linux through D-Bus/zbus:
heavy deps, headless/CI flakiness) and are deferred; the file store is
the explicit opt-in, not a downgrade from a keyring.
- **Cache key**: SHA-256 of the canonical identity (issuer, client ID,
sorted/de-duplicated scopes, flow, public/confidential), so no secret
appears in a filename and distinct identities never collide. Versioned
record schema (`version: 1`). One record per identity: last login wins,
documented.
- **Cross-process rotation locking**: per-key `fs4` file lock (`flock` /
`LockFileEx`) around the refresh critical section — acquire, reread the
durable record, refresh exactly once, atomically store the rotated
refresh token, release. The OS releases locks on process death, so
crashes cannot strand stale locks. Only confirmed
`invalid_grant`/`invalid_token` deletes a record and reauthenticates;
transport, 5xx, 429, and parse failures retain it.
- **Session APIs**: `OAuthSession::login/status/logout` in Rust,
`lancedb.remote.OAuthSession` (async) in Python, `OAuthSession` class in
Node. `status` returns non-secret metadata only. `logout` removes only
the local credential — provider revocation (RFC 7009) is a deliberate
follow-up, and local logout never terminates browser SSO. Azure managed
identity is rejected for persistence (machine identity stays in memory);
client credentials have nothing refreshable to persist and stay
memory-only.
- No CLI binary exists in this repo, so this ships library APIs plus doc
examples in all three languages.

Tests: Rust unit + mock-IdP integration (cache-key
canonicalization/separation, record
versioning/corruption/truncation/symlink/owner/perms, lock serialization
+ release, two concurrent providers proving no `invalid_grant` and
correct rotation, transient-failure retention, `invalid_grant` delete +
reauthenticate, login/status/logout lifecycle, client-credentials no-op,
IMDS rejection, secret redaction); Python lifecycle + a true
two-subprocess cross-process reuse test (second process refreshes once,
never hits the device endpoint); Node lifecycle + device-flow login
test. Local builds were skipped in development; CI validates all
bindings.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-16 02:01:10 +08:00
lancedb-gatefixer[bot]andXuanwo ffe94a65a1 fix(node): preserve optimize cleanup timestamp (#4160) 
<!-- lance-gatekeeper-fix:v1 agent=278cb095b2e5d69051442bc254d803a6
generation=1 -->

## Summary

- pass the TypeScript `cleanupOlderThan` date to the native binding as
an unchanged epoch timestamp
- prune with Lance's absolute `before_timestamp` policy so dispatch and
compaction time cannot move the cutoff
- retain versions created after the supplied cutoff and document that
behavior
- add boundary and end-to-end regression coverage

## Root cause

The TypeScript layer converted the absolute date into an elapsed
duration before calling native optimize. Lance converted that duration
back into a timestamp only after compaction, which silently advanced the
requested cutoff and made the cleanup count depend on a millisecond
timing boundary.

## Validation

- `cargo fmt --all`
- `cargo clippy --quiet --features remote --tests --examples -p lancedb
-p lancedb-nodejs`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test __test__/table.test.ts --runInBand` (309 passed)

Fixes #4159

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-15 14:10:48 +08:00
LanceDB Robot b8f0048b5a chore: update lance dependency to v12.0.0-beta.18 (#4164) 
Update the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v12.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.18).
Fix redundant visibility declarations in the Node.js Rust blob helpers
required by the workspace Clippy check.
Validated with workspace Clippy (all features and tests, warnings
denied), cargo fmt, pnpm build, and 13 targeted Node.js blob tests.
 2026-09-12 22:25:12 -07:00
Drew 6702e3fec1 feat(node): add blob v2 fetch and field helpers (#4155) 
this PR blob v2 field helpers and reads to the Node SDK.

`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.

Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.

Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.

```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
  "videos",
  [{ id: 1n, video }],
  {
    schema: new Schema([
      new Field("id", new Int64()),
      blob("video"),
    ]),
  },
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);

```

### Testing

- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
 2026-09-11 14:46:27 -07:00
LanceDB RobotandJack Ye a487d4033e chore: update lance dependency to v12.0.0-beta.14 (#4141) 
Update the Rust workspace Lance dependencies and Java lance-core from
v12.0.0-beta.11 to
[v12.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.14),
refreshing Cargo.lock. Resolve two Clippy diagnostics by making an
internal Node.js helper private and using a byte string literal in a
remote-table test fixture.

Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `git diff --check`, and `pnpm
build` in nodejs.

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
 2026-09-06 13:43:35 -07:00
Jack Ye 21f11b4463 feat!: replace get_job/job_history with describe_job/query_job_events (#4130) 
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.

## Before

Four ways to ask about a job, none of which told you much.

```python
job = table.refresh_column_async("embedding")
job.status()               # "running". That was the entire debug surface.
db.get_job(job_id)         # state, and a spec. No result, no progress.
db.job_history(job_id)     # raw record batches, no limit, no filter
db.job(job_id)             # a handle that knew nothing
```

## After

Open a job the way you open a table; the handle answers everything.

```python
job = db.open_job(job_id)      # raises JobNotFoundError if there is no such job
```

```python
>>> print(job)
Job(
    id='job-1',
    state='failed',
    job_type='refresh_column',
    creation_ms=1757000000000,
    spec={
        "column": "embedding",
        "num_workers": 4
    },
    failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```

Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.

Per-fragment progress *while it is still running*:

```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist()      # [5000, 5000, ...]
```

The handle an async action returns is the same object, one `refresh()`
away:

```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```

TypeScript is the same experience, down to `console.log`:

```ts
const job = await db.openJob(jobId);   // rejects if there is no such job
console.log(job);                      // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```

## Why each piece matters

- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.

`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.

## Breaking

All shipped in 0.38.0. No deprecated aliases.

| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |

Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
 2026-09-05 16:24:23 -07:00
dependabot[bot]Will JonesClaude Sonnet 5
 		e773d1e093 build(deps): bump the rust-minor-patch group across 1 directory with 9 updates (#4084) 
Bumps the rust-minor-patch group with 9 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-trait](https://github.com/dtolnay/async-trait) | `0.1.91` |
`0.1.92` |
| [log](https://github.com/rust-lang/log) | `0.4.33` | `0.4.34` |
| [moka](https://github.com/moka-rs/moka) | `0.12.15` | `0.12.16` |
| [uuid](https://github.com/uuid-rs/uuid) | `1.24.0` | `1.26.0` |
| [serde_with](https://github.com/jonasbb/serde_with) | `3.21.0` |
`3.22.0` |
| [roaring](https://github.com/RoaringBitmap/roaring-rs) | `0.11.4` |
`0.11.5` |
| [napi](https://github.com/napi-rs/napi-rs) | `3.11.0` | `3.12.0` |
| [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.1` | `3.6.3`
|
| [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.0` | `2.4.1` |


Updates `async-trait` from 0.1.91 to 0.1.92
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/async-trait/releases">async-trait's
releases</a>.</em></p>
<blockquote>
<h2>0.1.92</h2>
<ul>
<li>Resolve double_must_use clippy lint in generated code (<a
href="https://redirect.github.com/dtolnay/async-trait/issues/303">#303</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/async-trait/commit/82e7e9edd60f622294373a23c0ce9c0077ad0263"><code>82e7e9e</code></a>
Release 0.1.92</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/9a35cb87f9366cd992bbc00d430e1b5fe1aa0cdd"><code>9a35cb8</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/async-trait/issues/303">#303</a>
from dtolnay/mustuse</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/875ceecb100bab2cf369178633b4791336d92b75"><code>875ceec</code></a>
Resolve double_must_use clippy lint</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/62993a57bc6a8d5bd3de23fbae48cede333cb925"><code>62993a5</code></a>
Raise minimum tested compiler to rust 1.88</li>
<li>See full diff in <a
href="https://github.com/dtolnay/async-trait/compare/0.1.91...0.1.92">compare
view</a></li>
</ul>
</details>
<br />

Updates `log` from 0.4.33 to 0.4.34
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/log/releases">log's
releases</a>.</em></p>
<blockquote>
<h2>0.4.34</h2>
<h2>What's Changed</h2>
<ul>
<li>doc: Add context-logger utility to README by <a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li>Add alloc support for boxed loggers by <a
href="https://github.com/malezjaa"><code>@​malezjaa</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li><a href="https://github.com/malezjaa"><code>@​malezjaa</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">https://github.com/rust-lang/log/compare/0.4.33...0.4.34</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/log/blob/master/CHANGELOG.md">log's
changelog</a>.</em></p>
<blockquote>
<h2>[0.4.34] - 2026-08-22</h2>
<h2>What's Changed</h2>
<ul>
<li>doc: Add context-logger utility to README by <a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li>Add alloc support for boxed loggers by <a
href="https://github.com/malezjaa"><code>@​malezjaa</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li><a href="https://github.com/malezjaa"><code>@​malezjaa</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">https://github.com/rust-lang/log/compare/0.4.33...0.4.34</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rust-lang/log/commit/8034743dd9d7f7583bd9a670271483d176130911"><code>8034743</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/738">#738</a>
from rust-lang/cargo/0.4.34</li>
<li><a
href="https://github.com/rust-lang/log/commit/7d1e24e3506d4ffa1badf6c9ea357779877adaf0"><code>7d1e24e</code></a>
prepare for 0.4.34 release</li>
<li><a
href="https://github.com/rust-lang/log/commit/3b939b6714616dc32193c12019861c7c518c5edb"><code>3b939b6</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/737">#737</a>
from malezjaa/master</li>
<li><a
href="https://github.com/rust-lang/log/commit/b88266cfed8b287f8c35b2015808b09b056f61af"><code>b88266c</code></a>
Add alloc support for boxed loggers</li>
<li><a
href="https://github.com/rust-lang/log/commit/037d7a58f6ad184abb3afc4db81d37c43a5696ec"><code>037d7a5</code></a>
doc: Add context-logger utility to README</li>
<li>See full diff in <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">compare
view</a></li>
</ul>
</details>
<br />

Updates `moka` from 0.12.15 to 0.12.16
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/moka-rs/moka/releases">moka's
releases</a>.</em></p>
<blockquote>
<h2>v0.12.16</h2>
<h2>Version 0.12.16</h2>
<h3>Fixed</h3>
<ul>
<li>Fixed a bug where cache eviction could stall permanently when the
cache was configured with the <strong>non-default</strong> LRU eviction
policy (<code>EvictionPolicy::lru()</code>) by a race between insert and
remove operations on the same key (<a
href="https://redirect.github.com/moka-rs/moka/issues/592">#592</a><a
href="https://redirect.github.com/moka-rs/moka/pull/592/">gh-pull-0592</a>
by <a
href="https://github.com/kim-jhyeon"><code>@​kim-jhyeon</code></a>,
reported in <a
href="https://redirect.github.com/moka-rs/moka/issues/590">#590</a><a
href="https://redirect.github.com/moka-rs/moka/issues/590/">gh-issue-0590</a>):
<ul>
<li>This bug was introduced in v0.12.0 and affected
<code>sync::Cache</code>, <code>sync::SegmentedCache</code> and
<code>future::Cache</code>.</li>
<li>A race between applying a write recording for an entry and
concurrently removing that entry from the internal concurrent hash table
could leave an orphaned node at the front of the LRU queue. Once
present, no entry was ever evicted again and the cache grew unboundedly
past <code>max_capacity</code>.</li>
<li>The same race also affected the default TinyLFU eviction policy, but
with a milder symptom: each occurrence permanently leaked one phantom
entry slot, causing <code>entry_count</code> and
<code>weighted_size</code> to over-report and the usable capacity to
shrink by one entry per occurrence. Fixed by the same change.</li>
</ul>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Worked around a ThreadSanitizer false positive (<a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a><a
href="https://redirect.github.com/moka-rs/moka/pull/602/">gh-pull-0602</a>):
<ul>
<li>Replaced the standalone <code>fence(Acquire)</code> in the internal
<code>MiniArc</code>'s drop path with an <code>Acquire</code> load of
the reference count, so that downstream projects can now run
ThreadSanitizer on code using Moka without hitting this false
positive.</li>
<li><code>std::sync::Arc</code> has a similar workaround.</li>
</ul>
</li>
<li>Raised the minimum version of the <code>crossbeam-epoch</code> crate
from <code>v0.9.18</code> to <code>v0.9.20</code> to avoid the following
advisory (<a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a><a
href="https://redirect.github.com/moka-rs/moka/pull/603/">gh-pull-0603</a>):
<ul>
<li>[RUSTSEC-2026-0204] crossbeam-epoch: invalid pointer dereference in
<code>fmt::Pointer</code> for <code>Atomic</code> and
<code>Shared</code></li>
<li>Moka is <em>not</em> affected by this advisory because it never
formats these pointer types. However, raising the minimum version
prevents downstream lockfiles from resolving to an affected
<code>crossbeam-epoch</code> version via Moka.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/moka-rs/moka/blob/main/CHANGELOG.md">moka's
changelog</a>.</em></p>
<blockquote>
<h2>Version 0.12.16</h2>
<h3>Fixed</h3>
<ul>
<li>Fixed a bug where cache eviction could stall permanently when the
cache was
configured with the <strong>non-default</strong> LRU eviction policy
(<code>EvictionPolicy::lru()</code>)
by a race between insert and remove operations on the same key
(<a
href="https://redirect.github.com/moka-rs/moka/issues/592">#592</a>[gh-pull-0592]
by [<a
href="https://github.com/kim-jhyeon"><code>@​kim-jhyeon</code></a>][gh-kim-jhyeon],
reported in
<a
href="https://redirect.github.com/moka-rs/moka/issues/590">#590</a>[gh-issue-0590]):
<ul>
<li>This bug was introduced in v0.12.0 and affected
<code>sync::Cache</code>,
<code>sync::SegmentedCache</code> and <code>future::Cache</code>.</li>
<li>A race between applying a write recording for an entry and
concurrently
removing that entry from the internal concurrent hash table could leave
an
orphaned node at the front of the LRU queue. Once present, no entry was
ever
evicted again and the cache grew unboundedly past
<code>max_capacity</code>.</li>
<li>The same race also affected the default TinyLFU eviction policy, but
with
a milder symptom: each occurrence permanently leaked one phantom entry
slot, causing <code>entry_count</code> and <code>weighted_size</code> to
over-report and the
usable capacity to shrink by one entry per occurrence. Fixed by the same
change.</li>
</ul>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Worked around a ThreadSanitizer false positive (<a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a>[gh-pull-0602]):
<ul>
<li>Replaced the standalone <code>fence(Acquire)</code> in the internal
<code>MiniArc</code>'s drop
path with an <code>Acquire</code> load of the reference count, so that
downstream
projects can now run ThreadSanitizer on code using Moka without hitting
this false positive.</li>
<li><code>std::sync::Arc</code> has a similar workaround.</li>
</ul>
</li>
<li>Raised the minimum version of the <code>crossbeam-epoch</code> crate
from <code>v0.9.18</code> to
<code>v0.9.20</code> to avoid the following advisory (<a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a>[gh-pull-0603]):
<ul>
<li>[RUSTSEC-2026-0204] crossbeam-epoch: invalid pointer dereference in
<code>fmt::Pointer</code> for <code>Atomic</code> and
<code>Shared</code></li>
<li>Moka is <em>not</em> affected by this advisory because it never
formats these
pointer types. However, raising the minimum version prevents downstream
lockfiles from resolving to an affected <code>crossbeam-epoch</code>
version via
Moka.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/moka-rs/moka/commit/a616ec19e8d4ed938caf8b2c88090331d778d5da"><code>a616ec1</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/604">#604</a> from
moka-rs/chore/bump-v0.12.16</li>
<li><a
href="https://github.com/moka-rs/moka/commit/3b140a627e9faa4ec6a8e682224c7f81efc2b6e4"><code>3b140a6</code></a>
Bump the version to v0.12.16</li>
<li><a
href="https://github.com/moka-rs/moka/commit/51b802dc5cfc9e0da21de04177d79028bfbe5d47"><code>51b802d</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a> from
moka-rs/bump-crossbeam-epoch-floor</li>
<li><a
href="https://github.com/moka-rs/moka/commit/4f9071684161d59212c32a0e89762c3a5d6385a4"><code>4f90716</code></a>
Raise the minimum crossbeam-epoch version to 0.9.20</li>
<li><a
href="https://github.com/moka-rs/moka/commit/08d0e0458bd95af7f9435ff3ffbba6d1e91647c1"><code>08d0e04</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a> from
moka-rs/gh600-tsan-workaround</li>
<li><a
href="https://github.com/moka-rs/moka/commit/14447a7cbe441639e2aa3570e411fe493c71c9ac"><code>14447a7</code></a>
Restructure the v0.12.16 TSan workaround CHANGELOG entry</li>
<li><a
href="https://github.com/moka-rs/moka/commit/7b14c37b009a25c9a2ec27e2669dc5f8db7ce254"><code>7b14c37</code></a>
Avoid a TSan false positive by replacing the fence in MiniArc::drop</li>
<li><a
href="https://github.com/moka-rs/moka/commit/05b37c63098473034e7e961c1010284163ad8634"><code>05b37c6</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/599">#599</a> from
moka-rs/gh590-deterministic-tests</li>
<li><a
href="https://github.com/moka-rs/moka/commit/fc318584d25c0ea01109872da37d395754647e04"><code>fc31858</code></a>
Replace private doc references in gh590 test comments</li>
<li><a
href="https://github.com/moka-rs/moka/commit/57435922036ff4ab9f1b5bd0c3bffe6c8acd9921"><code>5743592</code></a>
Improve the v0.12.16 CHANGELOG entry</li>
<li>Additional commits viewable in <a
href="https://github.com/moka-rs/moka/compare/v0.12.15...v0.12.16">compare
view</a></li>
</ul>
</details>
<br />

Updates `uuid` from 1.24.0 to 1.26.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/uuid-rs/uuid/releases">uuid's
releases</a>.</em></p>
<blockquote>
<h2>v1.26.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Add ContextV7::with_additional_precision_bits by <a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/904">uuid-rs/uuid#904</a></li>
<li>Prepare for 1.26.0 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/905">uuid-rs/uuid#905</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/1.25.0...v1.26.0">https://github.com/uuid-rs/uuid/compare/1.25.0...v1.26.0</a></p>
<h2>1.25.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Add a serde::bytes module that encodes a Uuid as a byte string by <a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/902">uuid-rs/uuid#902</a></li>
<li>Prepare for 1.25.0 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/903">uuid-rs/uuid#903</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> made
their first contribution in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/902">uuid-rs/uuid#902</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.1...1.25.0">https://github.com/uuid-rs/uuid/compare/v1.24.1...1.25.0</a></p>
<h2>v1.24.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Fix non-ASCII character handling in parse diagnostics by <a
href="https://github.com/questfever"><code>@​questfever</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/899">uuid-rs/uuid#899</a></li>
<li>Prepare for 1.24.1 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/900">uuid-rs/uuid#900</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/questfever"><code>@​questfever</code></a> made
their first contribution in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/899">uuid-rs/uuid#899</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.24.1">https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.24.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/uuid-rs/uuid/commit/cdc96a87bddc38d0eb8f894c764e151d2299b4b3"><code>cdc96a8</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/905">#905</a> from
uuid-rs/cargo/v1.26.0</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/34e4f49c0d50c12f1b3021baf98b8fb91f6407bb"><code>34e4f49</code></a>
don't test macros under miri</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/d9e7242b37755d844d19fa74559a88e1c46c5206"><code>d9e7242</code></a>
update nightly used for miri</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/ec16819865b89aa3c52456c8afd0ce9a90f0fcdb"><code>ec16819</code></a>
prepare for 1.26.0 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/162cd208a4521138f1d8ce05b63342ba7ba5c4e6"><code>162cd20</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/904">#904</a> from
ChrisJr404/v7-additional-precision-bits</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/97eceffa708f87969792af604291d3e4984dfc90"><code>97eceff</code></a>
Add ContextV7::with_additional_precision_bits for microsecond
clocks</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/302e0bf6dc5abf949c06973a37f1f3a093cc2699"><code>302e0bf</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/903">#903</a> from
uuid-rs/cargo/1.25.0</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/b7ccde885d770d013f413a2685ebe7f38932e1d0"><code>b7ccde8</code></a>
prepare for 1.25.0 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/c62dffbc038034ff045f3009f2536362e313bf34"><code>c62dffb</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/902">#902</a> from
ChrisJr404/serde-bytes-module</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/8c198b24b1aa55948c0fa4b3433c1954be19c8c8"><code>8c198b2</code></a>
Add a serde::bytes module that encodes as a byte string</li>
<li>Additional commits viewable in <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.26.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `serde_with` from 3.21.0 to 3.22.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/jonasbb/serde_with/releases">serde_with's
releases</a>.</em></p>
<blockquote>
<h2>serde_with v3.22.0</h2>
<h3>Added</h3>
<ul>
<li>Add support for <code>jiff</code> v0.2 behind the new
<code>jiff_0_2</code> feature flag (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/936">#936</a>)
<code>jiff::SignedDuration</code> works with
<code>DurationSeconds</code> and its variants.
<code>jiff::Timestamp</code>, <code>jiff::Zoned</code>, and
<code>jiff::civil::DateTime</code> work with
<code>TimestampSeconds</code> and its variants.
Deserializing a <code>jiff::Zoned</code> uses the system time zone, like
<code>chrono::DateTime&lt;Local&gt;</code>.</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>Extend the <a
href="https://github.com/jonasbb/serde_with/security/advisories/GHSA-7gcf-g7xr-8hxj">GHSA-7gcf-g7xr-8hxj</a>
fix to the duplicate-key-prevention collections.
The <code>rust::sets_duplicate_value_is_error</code>,
<code>rust::maps_duplicate_key_is_error</code>,
<code>rust::sets_last_value_wins</code>, and
<code>rust::maps_first_key_wins</code> adapters created their backing
sets/maps with <code>with_capacity_and_hasher</code> using the raw
deserializer <code>size_hint</code>, bypassing the
<code>size_hint_cautious</code> cap added in <a
href="https://redirect.github.com/jonasbb/serde_with/issues/966">#966</a>
(the <code>clippy.toml</code> <code>disallowed_methods</code> lint only
covers <code>Vec::with_capacity</code>, not
<code>with_capacity_and_hasher</code>, so these sites were not flagged).
Attacker-controlled input claiming a huge length could panic with
<code>Hash table capacity overflow</code> before a single element was
read. All such constructions now route through
<code>size_hint_cautious</code>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/jonasbb/serde_with/commit/88f576a17c5cd45cea6a30252ef10653dde69fa8"><code>88f576a</code></a>
Bump version to 3.22.0 (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/991">#991</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/931e664445c2139446b84e76b339924f136a1565"><code>931e664</code></a>
Bump version to 3.22.0</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/e26930e0b7a6c1e6463086a2447e7cc8fc6f0a24"><code>e26930e</code></a>
Bump github/codeql-action from 4.37.3 to 4.37.4 in the github-actions
group (...</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/92cd5a0bd5c7a80fc7eae90bb99b873c40429aa3"><code>92cd5a0</code></a>
Bump github/codeql-action in the github-actions group</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/32be66fecc5c1fe4c90ac0230c0af04d1977df53"><code>32be66f</code></a>
Guard with_capacity_and_hasher against untrusted size_hint (DoS) (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/971">#971</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/33871cd4dd1ecef2c3af0ead9528c8b407c16f04"><code>33871cd</code></a>
Merge branch 'master' into
fix/duplicate-key-impls-capacity-overflow</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/bb1e06484261595c8cec7fd7f4ed33ecdfb951c0"><code>bb1e064</code></a>
Change function position within impl (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/968">#968</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/202d3dd617d7b5a9db5f490fa752d6ccb48454e8"><code>202d3dd</code></a>
Improve the time unit macros to remove unnecessary repetition and make
the co...</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/b347efb536caf83c850d4808f90503835fd78755"><code>b347efb</code></a>
Move the <code>use_duration_signed_ser</code>/<code>*_de</code> macros
utils</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/65905455527c0abf51f2f906bc08724426b2b922"><code>6590545</code></a>
chrono_0_4: Implement the same time unit macro cleanup as jiff_0_2</li>
<li>Additional commits viewable in <a
href="https://github.com/jonasbb/serde_with/compare/v3.21.0...v3.22.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `roaring` from 0.11.4 to 0.11.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/RoaringBitmap/roaring-rs/releases">roaring's
releases</a>.</em></p>
<blockquote>
<h2>v0.11.5</h2>
<h2>What's Changed</h2>
<ul>
<li>Implement std Error for IntegerTooSmall by <a
href="https://github.com/Kerollmops"><code>@​Kerollmops</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/362">RoaringBitmap/roaring-rs#362</a></li>
<li>fix: invalid treemap iter advance by <a
href="https://github.com/silver-ymz"><code>@​silver-ymz</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/360">RoaringBitmap/roaring-rs#360</a></li>
<li>Fix off-by-one that corrupts a bitmap in
remove_smallest/remove_biggest (<a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/359">#359</a>)
by <a href="https://github.com/youdie006"><code>@​youdie006</code></a>
in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/363">RoaringBitmap/roaring-rs#363</a></li>
<li>Upgrade dependencies bump version by <a
href="https://github.com/Kerollmops"><code>@​Kerollmops</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/364">RoaringBitmap/roaring-rs#364</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/silver-ymz"><code>@​silver-ymz</code></a> made
their first contribution in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/360">RoaringBitmap/roaring-rs#360</a></li>
<li><a href="https://github.com/youdie006"><code>@​youdie006</code></a>
made their first contribution in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/363">RoaringBitmap/roaring-rs#363</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5">https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/0ce3fc8b55b193ce220253bfbc0c3e09bd171375"><code>0ce3fc8</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/364">#364</a>
from RoaringBitmap/upgrade-dependencies-bump-version</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/a961a042db8d325515e6b5273a2e9369fe5c931d"><code>a961a04</code></a>
Remove the once_cell dependency</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/5e8445b2d6914e8e56d85de340f9156825f4e91b"><code>5e8445b</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/363">#363</a>
from youdie006/fix/359-interval-remove-boundary</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/bf2961d99fb4da2c540a55eb699228a7bb00a732"><code>bf2961d</code></a>
Bump version to v0.11.5</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/048a8b05fcae08636354607f0c00d6e95f262107"><code>048a8b0</code></a>
Fix off-by-one that corrupts a bitmap in
remove_smallest/remove_biggest</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/27d84f567dd85243194d2b87683262ef43a5dd97"><code>27d84f5</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/360">#360</a>
from silver-ymz/fix/treemap-iter-advance-across-bitmaps</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/aac2de82a7f9e44fd73d8364de840169447d580b"><code>aac2de8</code></a>
Make clippy happy</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/a3d1d54be985fe22c01e882f85c3ee7055ad9c8b"><code>a3d1d54</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/362">#362</a>
from RoaringBitmap/std-error-for-integer-too-small</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/9a3c33e42c0b14bdd3f296313ee367526092aa81"><code>9a3c33e</code></a>
Implement std Error for IntegerTooSmall</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/f46c0ffe90b6d5d52a93106253bb6fa51a08c137"><code>f46c0ff</code></a>
fix: invalid treemap iter advance</li>
<li>See full diff in <a
href="https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi` from 3.11.0 to 3.12.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi's
releases</a>.</em></p>
<blockquote>
<h2>napi-v3.12.0</h2>
<h3>Added</h3>
<ul>
<li><em>(cli)</em> support non-threaded WASI targets (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3353">#3353</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/58bd87fa524a837a7c962ab4103e5588557ccd81"><code>58bd87f</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3414">#3414</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/9da87236dbc4fef99f066b7a130f4d0377308d44"><code>9da8723</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/8d22196aa98a1e6e70584561f5446d117d9c802c"><code>8d22196</code></a>
chore(deps): update dependency oxc-parser to ^0.142.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3422">#3422</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/abc30fbafc2e3967d499cef970c68b3edfefd850"><code>abc30fb</code></a>
build(deps): bump postcss from 8.5.17 to 8.5.23 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3421">#3421</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/55421392cbaa24d4df69419e4c6d4958fbcb6a12"><code>5542139</code></a>
build(deps): bump fast-xml-parser from 5.9.3 to 5.10.1 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3418">#3418</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/dc4ee8c89cc27ce30e239482199b3b3d786bf8b6"><code>dc4ee8c</code></a>
build(deps): bump fast-uri from 3.1.3 to 3.1.4 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3419">#3419</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/050d985196174b4be830cdb813d09e2705258455"><code>050d985</code></a>
feat(async-runtime): drain-linger surface + lock-free scheduler
internals (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3">#3</a>...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/e0b87086eefe0e7efeea6d269e9403c4be4ba9aa"><code>e0b8708</code></a>
chore(deps): update dependency oxc-parser to ^0.141.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3417">#3417</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/fc8494010697d078a93a528c3180271f6f187504"><code>fc84940</code></a>
chore(deps): update actions/setup-node action to v7 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3413">#3413</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ee598db45985ef11e18c7340801c28bb2452b688"><code>ee598db</code></a>
build(deps): bump protobufjs from 7.6.4 to 7.6.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3410">#3410</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-v3.11.0...napi-v3.12.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-derive` from 3.6.1 to 3.6.3
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-derive's
releases</a>.</em></p>
<blockquote>
<h2>napi-derive-v3.6.3</h2>
<h3>Other</h3>
<ul>
<li>updated the following local packages: napi-derive-backend</li>
</ul>
<h2>napi-derive-v3.6.2</h2>
<h3>Other</h3>
<ul>
<li>updated the following local packages: napi-derive-backend</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/956e4525fea6a676ea3680b711382f167b899af9"><code>956e452</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3448">#3448</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/73048f5a7fdbd42cdc2f46f2d5ac60ef27417bfa"><code>73048f5</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61fae8a1440ad8b7249f3cd7838fc2bafe00a906"><code>61fae8a</code></a>
fix(napi): stop unloading addons with live native code, preserve
non-Error re...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/93e86ce167095e84f2be2ae1c66a6c0bb96fec49"><code>93e86ce</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2c905991899b9f12a0df4072c4bff6d62ef70d26"><code>2c90599</code></a>
fix(cli): support npm 12 pack output (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3449">#3449</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/360b1ec99ab0d001147d29e11c416c8338d3d1c9"><code>360b1ec</code></a>
fix(wasi): avoid randomness during module registration (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3447">#3447</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b648c4090e7518ced18c9eca6059d27af3ab511b"><code>b648c40</code></a>
build(deps): bump nanoid from 3.3.16 to 3.3.18 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3446">#3446</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ffda4efff4bc4ebb6bef1f629dd0a6f09dc8f210"><code>ffda4ef</code></a>
chore(deps): update dependency js-yaml to v4.3.1 [security] (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3445">#3445</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/387b0dc7986018e44a4a0b466b030dc414170411"><code>387b0dc</code></a>
feat(cli): size WASI browser worker pools from
navigator.hardwareConcurrency ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61e4346ce3d9a9c13e5c5dd6fb3b7d5e1b1d6e0d"><code>61e4346</code></a>
build(deps): bump fast-uri from 3.1.4 to 3.1.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3440">#3440</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.1...napi-derive-v3.6.3">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-build` from 2.4.0 to 2.4.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-build's
releases</a>.</em></p>
<blockquote>
<h2>napi-build-v2.4.1</h2>
<h3>Fixed</h3>
<ul>
<li><em>(napi)</em> stop unloading addons with live native code,
preserve non-Error rejections, and add the wasm teardown barrier (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3423">#3423</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/956e4525fea6a676ea3680b711382f167b899af9"><code>956e452</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3448">#3448</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/73048f5a7fdbd42cdc2f46f2d5ac60ef27417bfa"><code>73048f5</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61fae8a1440ad8b7249f3cd7838fc2bafe00a906"><code>61fae8a</code></a>
fix(napi): stop unloading addons with live native code, preserve
non-Error re...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/93e86ce167095e84f2be2ae1c66a6c0bb96fec49"><code>93e86ce</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2c905991899b9f12a0df4072c4bff6d62ef70d26"><code>2c90599</code></a>
fix(cli): support npm 12 pack output (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3449">#3449</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/360b1ec99ab0d001147d29e11c416c8338d3d1c9"><code>360b1ec</code></a>
fix(wasi): avoid randomness during module registration (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3447">#3447</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b648c4090e7518ced18c9eca6059d27af3ab511b"><code>b648c40</code></a>
build(deps): bump nanoid from 3.3.16 to 3.3.18 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3446">#3446</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ffda4efff4bc4ebb6bef1f629dd0a6f09dc8f210"><code>ffda4ef</code></a>
chore(deps): update dependency js-yaml to v4.3.1 [security] (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3445">#3445</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/387b0dc7986018e44a4a0b466b030dc414170411"><code>387b0dc</code></a>
feat(cli): size WASI browser worker pools from
navigator.hardwareConcurrency ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61e4346ce3d9a9c13e5c5dd6fb3b7d5e1b1d6e0d"><code>61e4346</code></a>
build(deps): bump fast-uri from 3.1.4 to 3.1.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3440">#3440</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.0...napi-build-v2.4.1">compare
view</a></li>
</ul>
</details>
<br />

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
 2026-08-31 10:40:11 -07:00
lancedb-gatefixer[bot]andXuanwo 9b825c5f29 fix(node): route auto search using table embeddings (#3832) 
## Summary

- Resolve automatic string-search routing from the active table schema
whenever the query executes.
- Defer embedding-provider construction while leaving explicit vector
and FTS routes unchanged.
- Cover unrelated global registrations and metadata transitions across
repeated executions of one query builder.

## Root cause

LocalTable.search used the number of globally registered embedding
providers to choose between vector and full-text search. A provider
registered for any other table therefore sent a plain FTS table down the
vector path. A wrapper-lifetime metadata snapshot avoided that
contamination but became stale after time travel or read-consistency
refreshes. The query now records fluent builder operations and creates
the appropriate native vector or FTS query from the active schema on
each execution.

## Validation

- pnpm build
- pnpm tsc
- pnpm lint
- pnpm run docs
- pnpm test --runInBand (681 passed, 5 skipped)

Fixes #1557

<!-- lance-gatekeeper-fix:v1 agent=b6183df8296db4aabdc5d19a2256b029
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-08-26 10:08:48 +08:00
lancedb-gatefixer[bot]andXuanwo 35b5d015ac fix(node): preserve embedding registration in server bundles (#3806) 
## Summary

- lazily initialize built-in OpenAI and Hugging Face providers when
consumers call the public embedding registry API
- choose automatic vector versus FTS search from embedding metadata on a
fresh pinned table revision for every execution
- expose automatic string searches as an `AutoQuery` with only
operations common to both native query families
- keep the registry shared and built-in registration safe across
duplicated module graphs

## Root cause

Nitro treats dependency modules as side-effect-free and removes the bare
OpenAI provider import from its generated route. Registration therefore
never runs, so `getRegistry().get("openai")` remains undefined even when
the registry itself is shared globally. Bundlers may also duplicate the
provider and registry module graphs.

The public embedding entry point now initializes built-in providers only
when `getRegistry()` is explicitly called, keeping initialization on a
live path that Nitro retains. Each terminal automatic-search execution
pins the exact table revision visible at dispatch, reads embedding
metadata and computes an embedding from that snapshot, replays the
builder operations, and constructs and executes the selected native
query against the same snapshot. Pinned native snapshots execute locally
when namespace pushdown cannot carry their revision, while remote
snapshots are seeded directly from one version-and-schema response. The
public `AutoQuery` builder exposes only the operations shared by FTS and
vector search, so runtime class narrowing cannot expose invalid
vector-only methods. Repeated built-in registration replaces stale
constructors from duplicated module graphs while public `register()`
retains its duplicate-alias error.

## Validation

- `cargo fmt --all`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test --runInBand` (783 passed, 5 skipped)
- serial examples suite with a local OpenAI mock (11 passed), including
`sentence-transformers.test.ts`
- packaged Nitro 2.13.4 server route using the reported imports returned
`{"registered":true}`
- fresh-process FTS fixture initialized both public built-ins and
confirmed automatic string search still returned the indexed row
- schema-consistency regressions cover read-consistency refresh,
checkout, checkoutLatest, restore, runtime class narrowing, concurrent
overwrite during embedding computation, and reused automatic-search
builders
- focused regressions confirm pinned native snapshots bypass unversioned
namespace pushdown and remote snapshots use one describe request

Fixes #2429

<!-- lance-gatekeeper-fix:v1 agent=2adf0f21b8bfb634606ed8897a849e30
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-08-26 04:17:32 +08:00
Will JonesandClaude Opus 5 fce45ba9fc feat(nodejs): add listTables, deprecate tableNames (#4041) 
`table_names` is being replaced by `list_tables` across the SDKs, but
TypeScript only had `tableNames`. This PR adds `listTables`, which
returns a page of table names together with the token that resumes after
it, and marks `tableNames` and `TableNamesOptions` deprecated in favor
of it.

It binds the `Connection::list_tables` that already exists, so nothing
in the Rust API changes and nothing existing breaks. `pageToken` is
documented as opaque rather than as a table name, since what resumes a
listing is the database's to decide — that keeps callers off a detail
that is going to change.

Stacked on #4040, which fixes a table being dropped at every page
boundary. The page-walking test here needs that fix to pass. Review the
last commit only until #4040 lands.

## Example

```ts
const names = [];
let pageToken = undefined;
do {
  const page = await conn.listTables({ pageToken, limit: 100 });
  names.push(...page.tables);
  pageToken = page.pageToken;
} while (pageToken);
```

A namespace can be listed by passing its path first, mirroring
`tableNames`:

```ts
const page = await conn.listTables(["analytics"], { limit: 100 });
```

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-24 16:25:14 -07:00
Xuanwo b0dae5eb0b feat: return typed refresh job results (#4013) 
## Problem

`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.

## Behavior

`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.

The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.

## Ownership and integration boundary

LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
 2026-08-25 00:01:32 +08:00
Wyatt Alt 68749ecfa3 feat(nodejs): materialized view bindings (#3935) 
Exposes materialized views to TypeScript: createMaterializedView,
openMaterializedView and listMaterializedViews on Connection, and a
MaterializedView handle carrying the parsed definition and
refresh({full, sourceVersion}), which returns the typed refresh result.
select accepts column names, [alias, expression] pairs, or a record of
the
same; the definition reads back off the stored schema, so a reopened
handle
needs no side channel. Remote connections surface the core's
not-supported
error up front.

The napi crate needed the same recursion-limit raise as the core crate:
the
refresh future's type graph overflows the default trait-recursion depth.


<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-21 23:48:43 -07:00
Drew Gallardo e98d8ac685 feat!: rename branch merge to cherry_pick (#3986) 
This PR is a **breaking** rename of #3686.

merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.

This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.

Cherry-pick is terminology that aligns more with that.

```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")

exp.add_columns({"tag": "cast('draft' as string)"})

diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")

if result["status"] == "cherryPicked":
    print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
    print(result["diff"]["errors"])
```

### Behavior

- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.

### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
 2026-08-21 23:37:12 -07:00
Dan RammerandClaude Opus 5 f1c4967eeb feat: bring the MemWAL LSM surface to parity across the SDKs (#3962) 
## Why

Four of the eight LSM methods are **remote-only in the core**. `impl
BaseTable for NativeTable` implements only
`set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`,
`compact_lsm` and `get_lsm_stats` fall through to trait defaults
returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and
`checkpoint_lsm` is built on all three.

That explains the state of the bindings: Node had bound the four that
work against a local table and stopped, so a Cloud user could install an
LSM write spec but had no way to observe fresh-tier state or drive a
checkpoint. Java had none of it at all.

| SDK | set/unset/get spec | closeWriters | flush | compact | getStats |
checkpoint |
|---|---|---|---|---|---|---|
| Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Node *(before)* | ✅ | ✅ | — | — | — | — |
| **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** |
| Java *(before)* | — | — | — | — | — | — |
| **Java (after)** | **new** | n/a | **new** | **new** | **new** |
**new** |

Go and C are separate repos and are out of scope here. `closeLsmWriters`
drains cached in-process shard writers, so it has no meaning for Java,
which is a pure REST client.

## Node

Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and
`getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats`
/ `MemtableStats` objects — typed rather than a JSON blob, matching the
existing `LsmWriteSpec` object in the same file, with `u64` cast to
`i64` per that file's convention.

Because these four are remote-only, the new tests assert each binding
reaches the core and surfaces `NotSupported` against a local table. That
covers the wiring; behavior against a real endpoint stays covered by the
mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`.

## Python

No new methods. All eight are on `LanceTable`, `AsyncTable` and
`RemoteTable` — the last four landed on the sync `RemoteTable` in #3961,
which is merged into this branch.

What was missing here was reachability. `LsmWriteSpec` was importable
only from the private `lancedb._lancedb`, appearing in `table.py` solely
under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no
mention of it, which per the repo's docs guidance means it rendered
nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in
`__all__`, and documented.

## Java

Java reaches LanceDB purely over REST through the generated Lance
Namespace client, and these routes are not in that spec, so they are
issued through a small dedicated client rather than added to the spec.
That call is revisitable — LSM is one of four unspecified route families
alongside `multipart_write`, `page_cache/prewarm` and
`branches/diff|merge`. If those are ever regularized into the spec as a
group, `LanceDbTableLsm` is one file that gets deleted.

`LsmWriteSpec` here is deliberately **not**
`org.lance.memwal.InitializeMemWalParams`. That type defaults to
maintaining *no* indexes where a spec here defaults to maintaining
*every* index, and it cannot express the `null` that asks the server to
resolve the set:

| Value | On the wire | Meaning |
|---|---|---|
| unset (null) | `null` | Server resolves **every** maintainable index |
| `Collections.emptyList()` | `[]` | Maintain **none** |
| `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those |

A dedicated test pins null and `[]` as distinct on the wire, since
collapsing them is the failure mode that motivated a LanceDB-owned type.

`checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs`
with its constants and status semantics intact: 429/503 retried in place
against an 8-budget, 421 restarting from flush against a 3-budget, 5s
poll, and a target watermark fixed after the seal so it terminates under
write load.

`getLsmStats` returns typed `LsmStats` / `BucketStats` /
`GenerationStats` / `MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes.
Decoding is strict — see below.

## Review feedback

Both gatekeeper findings were real. Each was reproduced against the
scripted test server first, and each fix ships with the reproducer as a
regression test.

**The transport was doubling every checkpoint retry budget.**
`HttpClients.createDefault()` installs Apache's default response retry
strategy, whose retryable-status list is exactly 429 and 503 — the two
statuses `isRetryable` owns. A 429 held against `flush_lsm` issued
**18** wire requests where the loop intends 9, and `compact_lsm` was
retried in place despite the loop being built to fall through to a fresh
stats poll instead. Timing confirmed the mechanism: that run took 25.4s
≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry
interval.

Automatic retries are now disabled, so the checkpoint loop is the sole
owner of the 421/429/503 transitions. A side effect worth noting:
`testCheckpointRetriesRetryableStatusInPlace` was passing on a
transport-absorbed 429 and never reaching `issue()`'s retry branch at
all. It now exercises the real path.

**Stats decoding failed open.** `getLsmStats` read the response with
Jackson's `path()`, which yields a missing node that iterates as an
empty array — making "malformed" indistinguishable from "no buckets",
which is indistinguishable from "drained". Four separate payloads made
`checkpointLsm()` report convergence for a checkpoint that never ran:

| Response | Before | Now |
|---|---|---|
| `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ |
| `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` |
| empty response body | **reported success** | `IllegalStateException` |
| bucket missing required fields | **reported success** |
`IllegalStateException` |

The empty-body row is the one to weight: a proxy 200 with no body is a
realistic production event, and it silently reported a checkpoint that
never happened.

Decoding is now strict and fails closed, matching the serde contract on
the Rust side exactly. One deliberate deviation from the review comment,
which asked that *only* explicit JSON `null` count as disabled: Rust has
`#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to
`None` there too. Java now matches that. It is an absent-or-malformed
**`buckets`** that fails closed, which is the case the comment was
actually protecting.

## Testing

- Java: **33 passing** (8 existing + 25 LSM) against a scripted
`com.sun.net.httpserver.HttpServer` — no new test dependency. Wire
assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`;
checkpoint tests cover convergence, not piling onto a latched bucket,
421 restart-from-flush, 429 retry-in-place, terminal-status propagation,
reissue exhaustion, the exact wire-request count against the retry
budget, and five malformed stats payloads.
- Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm
run tsc`, `npm run lint`, `npm run docs` all clean.
- Python: `ruff format --check` and `ruff check` clean.
- Java formatting: `./mvnw -pl lancedb-core spotless:apply` and
`spotless:check` both clean under a JDK 11 toolchain.

## Note: spotless needs a pre-16 JDK

`./mvnw spotless:apply` fails on JDK 16+ with
`JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7,
pinned at `java/pom.xml:34`, predates JDK 16's compiler API change.
**This is pre-existing** and reproduces on a pristine `main` checkout.

It is not a blocker, just a toolchain requirement. Spotless was run
against these sources under JDK 11 and both `spotless:apply` and
`spotless:check` pass on the whole module:

```shell
JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply
```

Bumping the plugin so it works on modern JDKs is still worth doing, but
separately from this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-19 11:44:46 -05:00
Wyatt Alt c429863122 feat: refresh_column_async returns a job handle (#3939) 
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-14 16:05:04 -07:00
Wyatt Alt fc0d917d32 feat: refresh computed columns (#3938) 
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-14 14:43:41 -07:00
Wyatt Alt def869bb78 feat: declare computed columns by SQL expression (#3937) 
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.

The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-14 14:17:41 -07:00
Jack Ye ffd35c1a8f feat: add asynchronous drop table API (#3936) 
## Summary

- add `drop_table_async` and return a job handle while preserving
`drop_table`
- consume remote 202 responses with cleanup job IDs and retain
older-server compatibility
- expose the API through Python and TypeScript connection wrappers
 2026-08-13 18:05:44 -07:00
Dan TasseandWill Jones 77a93fee76 fix: get table size from metadata, not files (#3790) 
Some issues:
- file_size_bytes is optional in the manifest, so if it's not there (old
writer I guess) it'll under-report the table size.
- it changes results a little bit from the old way by including per-file
footers and metadata (probably not a big difference at real scale)

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
 2026-08-07 17:41:41 -04:00
Dan RammerandClaude Opus 5 706a9c327f feat: infer maintained indexes when an LsmWriteSpec omits them (#3748) 
## What

`LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`:

| value | meaning |
|---|---|
| `None` (new default) | every index the MemWAL supports, resolved when
the spec is installed |
| `Some([])` | maintain nothing — a scan/filter-only WAL table |
| `Some([..])` | exactly these, taken verbatim |

`with_maintained_indexes` keeps its signature;
`with_no_maintained_indexes()` is new. Surfaced through the remote path
(null on the wire), Python, and Node.

## Why

Callers had to state the maintained set by hand every time, which is
both tedious and easy to get wrong — the common case is "maintain what I
already built."

Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating
to lance's `is_maintainable_index_type`. This is load-bearing rather
than cosmetic: lance does **not** skip an index type its memtable cannot
build, it errors when the shard writer opens, so sweeping up a bitmap
index would fail every memtable claim and leave the table unwritable.
The inferred set excludes those, and an explicit list naming one is now
rejected at spec time instead of at claim time.

## Behavior change

A freshly constructed spec used to maintain **nothing**; it now
maintains **everything supported**. This flipped because napi collapses
`undefined` and `null` to `None`, so TypeScript cannot express "absent
means nothing, null means all" — any other choice makes the bindings
disagree with the wire. The error direction also favors it: an unwanted
maintained index costs memory, while a silently unmaintained one
degrades FTS to an unscored scan.

Three existing tests encoded the old default and are updated rather than
worked around.

## Caveat

The resolved set is a snapshot, not a subscription. An index created
after the spec is installed is not maintained until the spec is unset
and set again. `get_lsm_write_spec` therefore always reports a concrete
list — `None` never round-trips.

## Dependency

Needs a lance release carrying `is_maintainable_index_type`
(lance-format/lance#8095) before this builds against the pinned tag.
Draft until then.

## Testing

38 Rust LSM tests and 10 Python tests pass against a local lance build,
including new coverage that a bitmap index is excluded from inference
and rejected when named, and that `[]` stays distinguishable from null
on the wire.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-07 14:50:22 -05:00
Wyatt Alt 8e24dd3828 feat(rust)!: make add_columns a builder (#3778) 
Table::add_columns now takes no arguments and returns AddColumnsBuilder,
so calls become .add_columns().transform(t).execute().

read_columns was the second positional argument but reaches only one of
the five transform variants. In lance's add_columns_to_fragments only
BatchUDF receives the caller's value: SqlExpressions replaces it with
the columns its expressions reference, Stream and Reader pass None, and
AllNulls reads nothing. So it was mandatory on every call -- all
eighteen call sites here passed None -- and silently discarded four
times out of five. As a builder method it is optional, and setting it
where lance would discard it is now an error, which does reject a call
that previously succeeded while ignoring the argument.

Matches the builders add, update, and merge_insert already use.
 2026-08-04 11:18:22 -07:00
Wyatt AltandClaude Fable 5 e3b472c212 feat: connection-level job operations (#3755) 
Adds job operations to the connection surface, building on the Job
handle from #3742: job(id), list_jobs, get_job, cancel_job, and
job_history, plus a non-blocking Job.status(). Implemented on the
Database trait (defaulting to NotSupported), the remote backend
(/v1/jobs), and the Python and Node bindings; job_history returns Arrow
batches.

errors() and progress() are not included.

Tested with mocked endpoints in all three languages.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-31 12:51:43 -07:00
Wyatt AltandClaude Fable 5 a6418b6cb9 feat: create_index returns a Job handle (#3742) 
IndexBuilder::execute now returns a Job with wait and cancel methods.
Local tables build the index synchronously and return an already-done
job. Remote tables read the job id the server returns from create_index
and track it through the /v1/jobs API: wait polls describe until the job
reaches a terminal state and cancel posts a cancellation. Servers that
return no job id yield a done job, so behavior against older servers is
unchanged. The job id is not exposed on the handle.

The Python and TypeScript bindings keep their current signatures and
discard the handle; exposing Job there is left to follow-ups.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-31 07:32:28 -07:00
Yang CenandYang Cen f7feed48c3 feat(fts): support custom stop-word lists (#3734) 
## What

Expose custom FTS stop-word lists in the Python and TypeScript public
APIs, including their standalone tokenize helpers and remote index
creation.

This PR supports concrete string lists only. It does not add file or
LanceDB-table stop-word sources.

## Why

Rust already exposes Lance's custom stop-word list option. The Python
and TypeScript APIs did not pass it through, and local index details did
not retain the full tokenizer parameters needed by index-backed
tokenization after reopening a table.

## How

- Add `custom_stop_words` / `customStopWords` to the Python and
TypeScript FTS and tokenize options.
- Preserve `None` / `undefined`, empty lists, and list contents without
normalization.
- Load the persisted FTS segment parameters when returning local index
details.
- Serialize the concrete list in remote create-index requests.
- Keep Python and TypeScript tests thin; behavior, persistence, query
tokenization, and remote JSON coverage live primarily in Rust.

## Validation

- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
- Python extension rebuild with `uv` and `maturin`
- Targeted Python tests: 4 passed
- Python `ruff format --check` and `ruff check`
- TypeScript build, typecheck, Biome lint, generated docs, and targeted
tests

---------

Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
 2026-07-29 17:40:12 +08:00
Heng Ge f655f62e09 feat(query): add use_lsm to read MemWAL LSM data (#3489) 
## What

MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).

## How

- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
    - **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.

Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
 2026-07-25 23:45:27 -07:00
Yang CenandYang Cen 9dc5ec03aa feat(fts): add block size configuration (#3691) 
## What changed

- add `block_size` to Python FTS configuration and the deprecated
local/remote helpers
- add `blockSize` to the TypeScript FTS options and propagate it through
the NAPI binding
- serialize the value as `block_size` for remote index creation
- document the existing Rust builder API and generate the TypeScript API
reference
- add local, remote, metadata, search, and invalid-value regression
coverage

## Why

Lance supports configuring the number of documents per compressed FTS
posting block, but LanceDB's Python and TypeScript APIs did not expose
the setting. This made the experimental FTS V3 layout unavailable
through those clients and allowed the value to be dropped before index
creation.

## How it works

The default remains `128`. Supported values are `128` and `256`;
selecting `256` uses the experimental FTS V3 format. Invalid values are
rejected by the Lance builder and surfaced as Python or JavaScript
errors.

## Validation

- `cargo check --quiet --features remote --tests --examples`
- `cargo +1.94.0 clippy --quiet --features remote --tests --examples --
-D warnings`
- targeted Rust local and remote index tests
- Rust doctests: 34 passed
- Python Ruff checks, doctest, and targeted local/remote tests: 5 passed
- TypeScript build, Biome lint, generated docs, and targeted Jest tests:
9 passed
- `git diff --check`

## Limitations

The Java client remains unchanged because its external remote REST model
does not currently expose `block_size`.

Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
 2026-07-24 15:02:38 -07:00
Drew GallardoandCursor 65cd142c7e feat: add remote branch diff and merge client APIs (#3686) 
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.


This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`

Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.

Merge here means promoting a branch's added columns onto `main`.

### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.

### Example
```python

table = db.open_table("images")

table.branches.create("exp")
exp = table.branches.checkout("exp")

exp.add_columns({"tag": "cast('draft' as string)"})

diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)

if result["status"] == "merged":
    print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
    print(result["diff"]["mergeBlockers"])
```

### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-18 12:38:05 -07:00
Will JonesandClaude Opus 4.8 5d0a1ef66c fix(rust): bound remote insert request size to avoid ingestion timeouts (#3630) 
## Problem

On the remote (LanceDB Cloud) write path, each write partition is
uploaded as a **single** `/insert?upload_id=...` request that stays open
until the whole partition has been streamed and the server has written
it to object storage. For large bulk ingests a partition can be many GB,
so a single request can run longer than the client read timeout (default
300s), surfacing as:

```
lancedb.remote.errors.HttpError: operation timed out
```

The server already supports staging **multiple** parts under one
`upload_id` (each `/insert` writes a separate transaction that
`complete` merges atomically), but the client never used that — it sent
one part per partition.

## Change

Split each partition into multiple parts of at most
`max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded
as its own `/insert?upload_id=...&upload_part_id=...` request. This
bounds how long any single request stays open, independent of total data
size or write parallelism.

Key properties:
- **Still streamed, not buffered.** Each part's body is driven through a
bounded channel while the request is in flight (`futures::join!` of a
producer + the send), so peak memory stays at a couple of batches per
partition regardless of the part size. Backpressure from a
slow/throttled server still propagates upstream.
- **Correct part accounting.** An empty partition still sends exactly
one (schema-only) part so `complete` has a transaction to commit; a size
cut landing exactly on the end of input does not emit a trailing empty
part.
- **Multipart only.** The single-request (non-multipart) path is
unchanged.

## Config

New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable
via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable.
**Default 1 GiB** (`Some(0)` disables splitting → one request per
partition). Python users pick up the default/env automatically through
the remote client.

## Tests

- `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each
batch in its own part → N requests, each carrying the shared `upload_id`
and a distinct `upload_part_id`.
- `test_multipart_single_part_when_under_budget`: a large budget keeps
the partition in a single request.
- Verified end-to-end against a live remote table: a forced-chunked
multipart add (many parts) assembles to the correct row count.

Related to ENT-1883.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-17 14:37:41 -07:00
Jack Ye 37032151d3 feat: support distributed analyze plan metrics in clients (#3675) 
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
 2026-07-15 21:21:40 -07:00
Jack Ye 06b53c97d6 feat: add table FTS query tokenization (#3659) 
## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage
 2026-07-14 10:59:33 -07:00
Will JonesandClaude Opus 4.8 285add40dd feat: expose Lance metrics via OpenTelemetry in Python and Node (#3609) 
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.

## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.

## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.

## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.

Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.

## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-09 15:36:03 -07:00
Dan RammerandClaude Opus 4.8 6c066530e5 feat: add get_lsm_write_spec to read the installed LSM write spec (#3631) 
## Summary

Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the
read counterpart to the existing `set_lsm_write_spec` /
`unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is
not enabled; otherwise reconstructs the spec (mode, shard column,
`num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly
as installed.

## Changes

- **Rust core (`NativeTable`)** — reconstructs the spec from
`mem_wal_index_details()`, resolving the shard column from its Lance
field id via the dataset schema. This is a raw metadata read, so it is
unaffected by `describe_indices` system-index filtering.
- **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index
through `index/list` with `include_system: true` (so the curated
`list_indices` surface stays unchanged), then parses the index `details`
JSON. It matches the index by name and ignores `index_type`, so no
client `IndexType` variant is needed. It uses the **server-resolved
`column` name** from the details (Lance field ids do not travel to the
remote client).
- **Python + TypeScript bindings** — sync and async, mirroring
`set`/`unset`, with round-trip tests (bucket / identity / unsharded,
plus `None` when unset).

## Tests

- Rust: native round-trip unit test + remote mock-endpoint tests
(present + absent). All green (`cargo test --features remote -p
lancedb`).
- Python/TS: round-trip tests added; binding-runtime execution runs in
CI.

## Dependencies for the remote path

The remote path is complete on the client side but depends on two
out-of-repo pieces to work end-to-end:
1. **lance** — emit the server-resolved shard **`column`** name in the
MemWAL index `details` JSON (field ids can't reach the client). See
lance-format/lance#7667.
2. **server** — honor `include_system` on `index/list` so the
`__lance_mem_wal` entry is returned for this read.

Against an older server (no `include_system`), the remote getter
degrades gracefully to `Ok(None)` rather than erroring.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-08 14:05:41 -05:00
Weston PaceandClaude Sonnet 4.6 c6db80dd0b feat: add an elastic dataloader as an iterable dataset (#3509) 
# Elastic Streaming Dataloader

## Motivation

Training large models on LanceDB tables today requires loading the
entire dataset
into memory or writing bespoke batching logic. This PR introduces
`StreamingDataset`, a PyTorch `IterableDataset` that streams directly
from a
LanceDB table with two hard guarantees that are difficult to achieve
together:
**elastic determinism** and **resumability**.

## Goals

### Elastic determinism

The dataset partitions the table into a fixed number of *splits*
(controlled by
`num_splits`, `shuffle_seed`, and `epoch`). Samples are yielded by
round-robining
over splits one sample per split per cycle. Because the split structure
is fixed,
the set of samples that makes up each global training step is identical
regardless
of `world_size` or `num_workers`. You can scale your cluster up or down
between
runs and the model sees the same data in the same order — no
re-sharding, no
gradient variance from topology changes.

### Resumability

`state_dict()` / `load_state_dict()` capture how many samples each split
has
consumed. Because all splits are the same size and the round-robin
design keeps
them in lockstep, the state reduces to a single scalar
(`samples_consumed_per_split`)
that is topology-independent. A checkpoint saved with 8 GPUs can resume
correctly
on 4 GPUs or 16 GPUs without any adjustment.

### PyTorch `IterableDataset` / streaming

`StreamingDataset` implements the standard PyTorch `IterableDataset`
interface, so
it drops into any existing `DataLoader` pipeline without modification.
Data is
fetched lazily from Lance in chunks — only the rows needed for the
current batch are
ever in memory.

Compared to the map dataset this takes more work from pytorch and puts
it into the dataset itself (e.g. shuffling, filtering, etc.). We do this
because we cannot achieve things like elastic determinism or
prefiltering otherwise.

### Multi-worker support

DataLoader workers are automatically assigned contiguous sub-blocks of
splits (the
rank's splits are divided evenly across workers). Each worker is
independent:
no shared state, no inter-process coordination. The only constraint is
that
`num_splits` must be divisible by `world_size * num_workers`.

That being said, multi-worker is highly discouraged as it relies on
multiprocessing which is inefficient. Still, we want to support it.

### Filters as prefilters

Filters are applied at *permutation-build time* via
`PermutationBuilder.filter()`,
not re-evaluated on every fetch. The filtered row IDs are stored in the
permutation
table so that subsequent reads see only the matching rows. This allows
us to avoid loading rows that don't match the filter (which is the
default pytorch behavior)

### Prefetching

Two parameters control the I/O pipeline:

- `read_batch_size` (default 64) — number of rows fetched per
`take_offsets` call.
Larger values amortise per-request overhead, which is critical on object
storage
  where a single round-trip can cost ~100 ms.
- `prefetch_batches` (default 4) — number of batches prefetched in
parallel per
split via a `ThreadPoolExecutor`. While the model processes the current
batch,
the next several batches are already in flight, hiding storage latency
behind
  compute.

If set correctly then you can get good performance even with
num_workers=0 (unless you are bottlenecked on transform).

### Transform parallelism

The underlying `Permutation` API supports a `with_transform()` callback
for
decoding, augmentation, and format conversion. Unfortunately, this is
not parallelized. Pytorch typically parallelizes this with num_workers
which is multiprocessing which is highly inefficient. For simple
transforms we should be able to utilize multithreading and Rust based
UDFs. For complex python UDFs we could have a dedicated multiprocessing
pipeline for just the transform. Or we could just utilize
multithreading. In both cases we would exclude the I/O stage from the
multiprocessing because that ends up being very memory hungry and
inefficient.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-07-06 05:50:45 -07:00
Weston Pace f6c9d31f98 feat: add polars dataframe integration (#3584) 
This PR is part cleanup, part feature, part example.

It removes `IntoArrow` and `IntoArrowStream`. There was only one
redundant call site between the two. Once we moved everything to
`Scannable` these traits no longer serve any purpose.

It adds a `Scannable` impl for a polars DataFrame. We used to have this
at one point for `IntoArrow` so this is more like a regression fix than
anything.

It adds an example (and unit test) which ensures we can ingest from a
Polars DataFrame and export to one. LazyFrame support would be a
follow-up (though a pretty straightforward one) but we've never had
proper LazyFrame support before.
 2026-06-30 08:28:41 -07:00
Jack Ye 10fecdf051 feat(node): expose OAuth connection config (#3587) 
Expose the merged Rust OAuth header provider through the Node/TypeScript
connection path.

Includes:
- Native OAuthConfig conversion for napi-rs
- ConnectionOptions.oauthConfig plumbing
- Public TypeScript OAuthConfig and OAuthFlowType exports
- Generated TypeScript API docs for the new config surface
- input-validation and debug-redaction coverage in the Rust binding
layer

Local validation: cargo fmt --all; git diff --check.
 2026-06-29 16:55:45 -07:00
Brendan Clement f76b075d13 feat: add table branch support to remote tables and Python/TS bindings (#3540) 
### Description
Adding branch support for RemoteTable by threading a branch selector
onto every operation the data plane accepts it on. Exposes the
currentBranch to nodejs and python through the bindings.

Matching the server handlers, the branch rides as:
- a `?branch=` query parameter for Arrow-body and query-only ops
(insert, merge_insert, multipart_*, version/list, drop_index)
- a `branch` field in the JSON body for everything else (count_rows,
query, update, delete, create_index, column ops, index list/stats,
stats, restore, describe, tags create/update)

A main-branch handle (`branch == None`) produces byte-identical requests
to before: no `branch` field and no `?branch=`

- Handle-per-branch: `create_branch` / `checkout_branch` return a new
handle with fresh caches and reset version/freshness state, mirroring
`NativeTable`.
- `create_branch` maps 409 to already-exists, 400 to invalid, and 404 to
not-found with source context, and sends without retry so the 409 stays
observable.
- `Ref` translation covers version, version-number (relative to the
handle's branch), and tag (resolved via the tags endpoint); `"main"` and
empty normalize to the main branch.
- Python branch handles persist their branch (and pinned version) across
pickle/fork, so a forked or pickled handle reopens on its branch rather
than silently reverting to main.

### Tests

- Rust mock tests per op category (query-param and body mechanisms,
branch CRUD, error paths, backward-compat).
- Python sync branch CRUD, `open_table(branch=)`, and a pickle
round-trip regression test.
 2026-06-15 18:07:40 -04:00
Will JonesandClaude Sonnet 4.6 f8caef3aca feat(bindings): expose new IndexConfig fields in Python and Node.js (#3534) 
## Summary

Surfaces the rich per-index metadata added in #3497 to the Python and
Node.js language bindings. Closes #3495.

New optional fields exposed on `IndexConfig` in both bindings:

- `index_uuid` / `indexUuid` — UUID of the first index segment
- `type_url` / `typeUrl` — protobuf type URL for the index
- `created_at` / `createdAt` — creation timestamp (milliseconds since
Unix epoch)
- `num_indexed_rows` / `numIndexedRows` — rows covered by the index
- `num_unindexed_rows` / `numUnindexedRows` — rows not yet indexed
- `size_bytes` / `sizeBytes` — total index file size in bytes
- `num_segments` / `numSegments` — number of index segments
- `index_version` / `indexVersion` — on-disk format version
- `index_details` / `indexDetails` — type-specific JSON details string

All fields are `None`/`undefined` for remote tables (which don't yet
surface this metadata through the server response).

## Changes

- `python/src/index.rs`: extend `IndexConfig` pyclass; update `From`
impl; update `__getitem__`
- `python/python/lancedb/_lancedb.pyi`: add type hints for new fields
- `python/python/tests/test_table.py`: new `test_index_config_fields`
test
- `nodejs/src/table.rs`: extend `IndexConfig` napi struct; update `From`
impl
- `nodejs/__test__/table.test.ts`: new test; update existing `toEqual`
assertions to `expect.objectContaining` to accommodate new fields

## Test plan

- [x] Python: `uv run --extra tests pytest
python/tests/test_table.py::test_index_config_fields`
- [x] Node.js: `pnpm test __test__/table.test.ts`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-11 13:37:39 -07:00
Jack Ye 8373318e89 feat: support FM-Index scalar index for substring search (#3532) 
Adds an FM-Index — a scalar index over string and binary columns that
accelerates substring search (`contains(col, 'needle')`), distinct from
the tokenized `FTS` index — across the Rust core and the Python and
TypeScript bindings.

## Rust

- `Index::Fm(FmIndexBuilder)` and `IndexType::Fm`.
- `make_index_params` maps `Index::Fm` to Lance's
`ScalarIndexParams::for_builtin(BuiltinIndexType::Fm)`.
- `supported_fm_data_type` validates
`Utf8`/`LargeUtf8`/`Binary`/`LargeBinary` columns.
- `list_indices` round-trips the type (`"Fm"` → `IndexType::Fm`); the
remote wire type is `"FM"`.

## Python

Adds `lancedb.index.Fm`, accepted by `create_index`:

```python
from lancedb.index import Fm

await tbl.create_index("text", config=Fm())
```

## TypeScript

Adds the `Index.fm()` factory:

```ts
await tbl.createIndex("text", { config: Index.fm() });
```
 2026-06-10 12:28:20 -07:00
Brendan Clement d9018067b3 feat: support checking out a version on a branch (#3504) 
### Description

Stacked on #3490. Adds an optional version to branch checkout across the
Rust core and the Python and TypeScript SDKs, so you can open a specific
version on a branch ("version V of branch B"), not just the branch's
latest version

Rust

```rust
// Open version 3 of branch "exp" (a read-only view): check out from an
// existing table, or open it directly from the connection.
let exp_v3 = table.checkout_branch("exp", Some(3)).await?;
let exp_v3 = db.open_table("items").branch("exp").version(3).execute().await?;
// checkout_latest re-attaches to the branch's writable HEAD.
exp_v3.checkout_latest().await?;

// With no branch, a version opens main at that version.
let main_v3 = db.open_table("items").version(3).execute().await?;
```

Python

```python
# Open version 3 of branch "exp" (a read-only view): check out from an
# existing table, or open it directly from the connection.
branch_v3 = await table.branches.checkout("exp", version=3)
branch_v3 = await db.open_table("items", branch="exp", version=3)
# checkout_latest re-attaches to the branch's writable HEAD.
await branch_v3.checkout_latest()

# With no branch, a version opens main at that version.
main_v3 = await db.open_table("items", version=3)
```

TypeScript

```typescript
// Open version 3 of branch "exp" (a read-only view): check out from an
// existing table, or open it directly from the connection.
const branchV3 = await (await table.branches()).checkout("exp", 3);
const opened = await db.openTable("items", undefined, { branch: "exp", version: 3 });
// checkoutLatest re-attaches to the branch's writable HEAD.
await branchV3.checkoutLatest();

// With no branch, a version opens main at that version.
const mainV3 = await db.openTable("items", undefined, { version: 3 });
```

### Testing
- Added unit tests (Rust, Python sync + async, TypeScript):
branch-scoped resolution at a version number shared with `main` and with
another branch, read-only enforcement on a pinned handle,
`checkout_latest` recovery to the branch's HEAD, fork-point reads, and
the nonexistent-version/branch error paths.
- Ran smoke tests against the Python and TypeScript SDKs on local
machine.
 2026-06-08 17:36:38 -07:00
Brendan Clement 53517b3aaa feat: add table branch support (#3490) 
### Description

Adds first-class support for table branches across the Rust core and the
Python and TypeScript SDKs.

Rust

```rust
use lance::dataset::refs::Ref;

// Create a branch from main and write to it — main is untouched.
let exp = table.create_branch("exp", Ref::Version(None, None)).await?;
exp.add(batches).await?;

// Reopen the branch later: check out from a table, or open it directly.
let exp = table.checkout_branch("exp").await?;
let exp = db.open_table("items").branch("exp").execute().await?;

let branches = table.list_branches().await?;
table.delete_branch("exp").await?;
```

Python

```python
# Create a branch from main and write to it
branch = await table.branches.create("exp", from_ref="main")
await branch.add(data)

# Reopen the branch later: check out from a table, or open it directly.
branch = await table.branches.checkout("exp")
branch = await db.open_table("items", branch="exp")

await table.branches.list()
await table.branches.delete("exp")
```

TypeScript

```typescript
const branches = await table.branches();

// Create a branch from main and write to it
const branch = await branches.create("exp");
await branch.add(data);

// Reopen the branch later: check out from a table, or open it directly.
const checkedOut = await branches.checkout("exp");
const opened = await db.openTable("items", undefined, { branch: "exp" });

await branches.list();
await branches.delete("exp");
```

### Testing
- Added unit tests
- ran smoke tests against python and typescript sdks on local machine


### Next steps
- Add RemoteTable support
- Add Branch Comparison support
- Merge Branching support
 2026-06-08 16:26:46 -07:00
Will JonesandClaude Opus 4.8 09b1bbc12a refactor!: drop unused loss field from IndexStatistics (#3496) 
BREAKING CHANGE: direct Rust users lose the `IndexStatistics::loss`
field. Python and Node.js consumers are unaffected in practice for
remote tables (the value was always `None`/absent), but the attribute is
gone for local tables too.

`IndexStatistics::loss` was local-only — LanceDB Cloud never returned
it, so
`RemoteTable::index_stats` always set `loss: None`. It's vestigial; this
removes it.

- Remove `loss` from `IndexStatistics` and the internal `IndexMetadata`
in `rust/lancedb/src/index.rs`, plus the summing logic in
`NativeTable::index_stats`.
- Drop `loss` from the Python and Node.js bindings (and their
tests/docs).

Fixes #3493

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-05 07:52:40 -07:00
Brendan Clement d065be0474 feat: add update_field_metadata to edit per-field metadata (#3482) 
### Summary
Adds update_field_metadata to the client SDK (Rust core, Python, and
TypeScript) so clients can edit per-field (column) Arrow metadata
(schema.fields[].metadata)

### Testing
- added unit tests
- ran E2E against a local server on both local and remote tables (set →
merge → delete), across Python sync/async and TypeScript

### Next steps
- deprecate replace_field_metadata in the python lancedb favor of this
(typescript didn't have replace_field_metadata method). This matches
Lance's API direction (Lance already deprecated replace_field_metadata
for update_field_metadata)
 2026-06-02 07:00:00 -07:00
Heng GeandJack Ye 048f52c2aa feat(table): route merge_insert through the MemWAL LSM write path (#3354) 
## Summary

When an `LsmWriteSpec` is installed on a table (#3396), `merge_insert`
upsert
calls are dispatched through Lance's MemWAL `ShardWriter` (LSM-style
append)
instead of the standard merge path.

- **`use_lsm_write`** — a `merge_insert` builder option, default `true`;
set it
  `false` to use the standard path for a call even when a spec is set.
- **`assume_pre_sharded`** — a `merge_insert` builder option, default
`false`;
  skips the per-row shard check and routes by the first row only.
- **`close_lsm_writers`** — drains and closes the table's cached MemWAL
shard
  writers.
- The `merge_insert` **`on`** columns default to, and are validated
against,
  the table's unenforced primary key.
- Shard writers are cached alongside the dataset (in
  `DatasetConsistencyWrapper`) and reused for the session.
- `MergeResult` gains **`num_rows`** — on the LSM path the insert/update
  breakdown is unknown until compaction, so only the total is reported.

Routing covers all three sharding strategies — bucket (murmur3,
Iceberg-compatible), identity, and unsharded. Each `merge_insert` call
targets
a single shard; the whole input is collected and validated before a
single
atomic `ShardWriter::put`, so a validation failure leaves the MemWAL
untouched.

Bindings: Python (`merge_insert(...).use_lsm_write(...)` /
`.assume_pre_sharded(...)`, `Table.close_lsm_writers`) and TypeScript
(`mergeInsert(...).useLsmWrite(...)` / `.assumePreSharded(...)`,
`Table.closeLsmWriters`).

## Context

Reconstructed from the original #3354 branch onto current `main`: the
branch
predated the #3394 (unenforced primary key) / #3396 (`LsmWriteSpec`)
split and
has been rebuilt on that merged foundation. Depends on Lance
`v7.0.0-beta.13`.

The MemWAL read path (reading un-flushed shard data back into queries)
and
remote (LanceDB Cloud) LSM support are follow-ups.

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
 2026-05-29 08:48:11 -07:00