rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-13 15:22:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,118 Commits 375 Branches 556 Tags
0acb61471c848277097e56bb525e04e71b970bbc
Commit Graph
190 Commits
Author SHA1 Message Date
lancedb-gatefixer[bot] 1e4662ac23 fix(remote): unify unsupported operation errors and warnings (#4414) 
Remote SDKs report Cloud-specific errors against self-hosted servers,
and some Python wrappers raise different exception types for the same
unsupported operation. The sync index wrapper also warns that `replace`
is ignored even though Rust forwards it to the server.

Use backend-neutral table/server diagnostics and the existing
`NotSupported` error conversions for unsupported manifest operations,
blob ranges, primary keys, optimization, and HNSW-PQ indexes. Sync
Python delegates to the async paths, so manifest operations and
`optimize()` consistently raise `NotImplementedError`. Legacy and
config-based HNSW-PQ creation share the same error and alternative-index
advice.

Remove the false replacement warning and simplify the ignored
GPU-accelerator warning. Keep the deprecated cleanup/compaction no-ops,
with backend-neutral warnings that do not assume automatic server
maintenance.

Embedding configuration forwarding and stale warning removal are now
supplied by #4408 in `main`. Retain the regression that verifies full
embedding metadata and warning-free remote creation alongside the
diagnostics tests.

Validation against the merged `main` revision `bbc4aebe` and Lance
14.0.0-beta.1:

- Rebuilt the Python extension with `uv run --extra tests --extra dev
maturin develop --extras tests,dev`; all 122 remote Python tests passed.
Root Ruff checks and formatting passed using the uv-managed environment.
- `pnpm build`, `pnpm lint`, formatting, and `pnpm run docs` passed; all
47 remote Node tests and the focused local vector-index regression
passed.
- All 583 Rust remote tests and 54 query unit tests passed, along with
`cargo check --quiet --features remote --tests --examples`, `cargo
clippy --quiet --features remote --tests --examples`, and `cargo fmt
--all`.

Fixes #4393

<!-- lance-gatekeeper-fix:v1 agent=10bafd337999f4910855c3a9b71e7dcf
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-08 18:42:42 -07:00
Colin Patrick McCabe de005f1d61 refactor!: implement lazy iterators for several APIs (#4459) 
Implement lazy iterators for:
- secrets
- views
- jobs
- functions
- materialized views
 2026-10-07 10:24:20 -07:00
Yuval LifshitzandClaude Opus 5.0 069b2f7d31 feat: add delete rows to table stats (#3783) 
add to table stats the number of rows that has
been marked for deletion, but not yet deleted.
note that when a table fragment is removed due to
deletion of all of its rows, these rows are not counted. since the main
use of this feature is to determine when table compaction is needed, the
fact that these rows are not reported should not impact that decision
(deleted table fragments don't need compaction).

Fixes: https://github.com/lancedb/lancedb/issues/3761

Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com>
 2026-10-07 08:38:31 -07:00
Jack Ye 7e8f0d4982 refactor: always fully refresh materialized views (#4373) 
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.

## Current layout

Local materialized views currently maintain incremental-refresh state
alongside the materialized values:

- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.

Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.

## New layout

An MV is an ordinary Lance backing table containing only:

- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.

There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.

Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.

## New remote create workflow

```text
Connection::create_materialized_view(...)
        -> builder renders the defining SELECT
        -> remote Database receives a CreateMaterializedViewRequest
        -> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
        -> retain the SQL Query handle while the Job is running
        -> consume the durable SQL-job result
        -> open the created table and read mv.definition
        -> return MaterializedView
```

Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.

## API changes

- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.

The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
 2026-10-05 21:33:34 -07:00
Colin Patrick McCabe d864e7198f feat!: list_databases returns an iterator (#4442) 
CatalogConnection.list_databases should return an iterator so that users
don't have to manually re-implement pagination. For those who do want
more control over pagination, add num_page_results and page_token to the
Iterator object, and also allow the iterator object to be created at a
specific page token.
 2026-10-05 19:41:53 -07:00
lancedb-gatefixer[bot] 3e0a1a8780 fix: reject zero vector search probes across SDKs (#4418) 
`nprobes(0)` bypassed the validation used by the minimum and maximum
probe setters, allowing zero into query requests and producing
inconsistent local and remote results. Sync Python could additionally
interpret the zero maximum as an unbounded search.

Reject zero in the Rust vector-query builder and propagate its input
error through the Python vector/hybrid and Node bindings. Validate
nonpositive counts in the sync Python vector and hybrid builders so
every SDK and backend reports `nprobes must be greater than 0` before
execution. Update the Rust examples and document the positive-count
requirement.

Regression tests cover local and remote vector/hybrid builders,
rejection before remote query requests, promised TypeScript query
vectors, and positive counts replacing both probe bounds.

Validation:

- Rust probe regression tests: 2 passed.
- Focused Python query, remote, hybrid, and serialization tests: 22
passed. The four sync regression cases fail with the original setters.
- Focused TypeScript indexed local and mocked remote tests: 2 passed.
- Workspace `cargo check --features remote --tests --examples` and
Clippy passed.
- Rust formatting, Python formatting/lint, and TypeScript
build/lint/docs passed. Repository-wide `ruff check .` reports 29
existing errors in unchanged CI scripts; `ruff check python` passes.

Fixes #4388

BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>`
instead of `Self`; callers must handle the result, as with the existing
minimum/maximum probe setters.

<!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 21:29:47 -07:00
lancedb-gatefixer[bot] 7e979c1416 fix(node): preserve schemas for empty query results (#4423) 
Empty local TypeScript queries returned a zero-row Arrow table with no
columns because `toArrow()` built the table solely from record batches.
Local streams can yield no batches, while remote streams often supply an
empty batch carrying the schema.

Expose the executed stream's schema through the native iterator and
decode it into an empty Arrow table when no batches arrive. This
preserves projected and search columns, types, nullability, and metadata
without executing another query. Document the empty-result behavior.

Validation:

- Eleven regression cases failed with the original `toArrow()`
implementation; all twelve now pass, including remote schema-only and
empty-batch responses.
- All 1,075 tests in the query, remote, Arrow, and table suites pass.
- Native build, TypeScript compilation, Rust formatting and Clippy (`-D
warnings`), TypeScript lint/formatting, and documentation generation
pass.

Fixes #4386

<!-- lance-gatekeeper-fix:v1 agent=7d589da1f7db5c7383f9600490e111a9
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
 2026-10-03 05:56:50 -05:00
LanceDB RobotDaniel RammerClaude Opus 5.5
 		857e455361 chore: update lance dependency to v13.1.0-beta.1 (#4358) 
Updates the Rust workspace and Java Lance dependencies to
[v13.1.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.1.0-beta.1)
and refreshes Cargo.lock. Removes a redundant `pub(crate)` declaration
in the private Node.js blob module to satisfy the required Clippy check.

Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `pnpm build`, and `pnpm test
__test__/blob.test.ts --runInBand` (26 tests passed) in `nodejs`.

---------

Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-01 09:13:08 -05:00
Bilal Atique 0a1c1a04c7 feat(node): accept an AbortSignal to cancel blob reads (#4350) 
Closes #4288.

Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange`
and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an
optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When
the signal aborts, the call rejects with `signal.reason` (an
`AbortError` unless the caller gave a reason) and the native future is
dropped, so the read stops instead of running to completion. A signal
that is already aborted rejects before any read starts.

How it works:

- On the Rust side, these five methods are now sync `#[napi]` methods
that return `env.spawn_future(...)`. The future is wrapped in
`futures::future::abortable`, and the abort handle is registered with
napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so
it can't be an argument of an `async fn`, which is why the methods
changed shape. The TypeScript signatures are unchanged apart from the
new optional argument.
- napi-rs installs its own `onabort` on the signal it receives, which
would overwrite a handler the caller set. So the TypeScript wrapper
passes a private `AbortController` signal to the native call for each
request, forwards the caller's abort to it with `addEventListener(..., {
once: true })`, and removes the listener when the call settles.

Tests in `table.test.ts`:

- an already aborted signal rejects on all five methods, and the handle
still reads afterwards
- a custom abort reason is what the call rejects with
- a read with a signal that never aborts returns normally and leaves no
`abort` listener on the signal
- aborting a 32 MiB `readRange` right after it starts rejects with
`AbortError`, and the same handle reads again afterwards

The Node docs under `docs/src/js` are regenerated.

I didn't add a test for where the `read()` cursor ends up after a
mid-flight abort on a local table. The remote `BlobFile` documents that
a cancelled read leaves the cursor unchanged, but I haven't confirmed
the same for the native Lance handle, so the docs don't claim it.
 2026-09-29 14:01:17 -05:00
Bilal Atique 4e3ec7cc27 feat(node): expose seek, tell, bounded read, readRanges and close on BlobFile (#4348) 
Closes #4300.

The TypeScript `BlobFile` only had `size()`, `read()` and `readRange()`,
while the Rust `BlobFile` behind it also has `seek`, `tell`,
`read_up_to`, `read_ranges` and `close`. This exposes those on the Node
binding:

- `read(maxBytes?)` reads at most `maxBytes` from the cursor (Rust
`read_up_to`). Without the argument it reads to the end, as before.
- `readRanges([{ start, end }, ...])` reads several half-open ranges and
returns one buffer per range, in order. Like `readRange`, it does not
move the cursor.
- `seek(position)` and `tell()` move and report the cursor.
- `close()` releases the handle, and `isClosed()` reports it. Every
read, `seek` and `tell` after `close()` fails with "blob file is already
closed". A second `close()` does nothing. The binding keeps its own
`closed` flag, the same way the Python `BlobFile` does.

Offsets and lengths are `bigint`, like `readRange` and `size()`.
`BlobRange` is exported as a type.

The new tests in `table.test.ts` cover chunked reads with
`read(maxBytes)`, `seek` and `tell`, a negative seek position,
`readRanges` order and cursor, a `readRanges` end past the blob size,
and reads after `close()`. The Node docs under `docs/src/js` are
regenerated with `pnpm run docs`.

I left the `ReadableStream`/async iterator idea from the issue out of
this PR. The AbortSignal work in #4288 touches the same methods, so it
can follow on top of this.
 2026-09-29 11:32:30 -05:00
Xuanwo ea223c4ec4 chore: update lance dependency to v13.0.0-beta.18 (#4339) 
Updates the Rust workspace dependencies, Cargo lockfile, and Java
`lance-core` to [Lance
v13.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.18).
It includes lance-format/lance#9574, which bounds the memory an IVF
partition split uses when it re-reads raw vectors during
`optimize_indices`, and lance-format/lance#9582, which keeps the mTLS
REST namespace client on rustls.

Lance's new combined-fields FTS query variant (introduced by
v13.0.0-beta.17) is handled the same way as in #4336: the remote
capability check and Rust/Node bindings accept it, with a regression
test for the remote request payload, and the Python conversion reports
that this query type is not yet supported. This PR supersedes #4336.
 2026-09-29 23:19:59 +08:00
ningsh7 ef11e44adc feat(node): expose allowExternalBlobOutsideBases in Table.add (#4317) 
## Summary

- Add `allowExternalBlobOutsideBases` to the Node `AddDataOptions` and
forward it through N-API to the Rust `AddDataBuilder`.
- Document its default and local-only external blob behavior.
- Cover URI strings and URI objects, option scoping, progress callbacks,
overwrite mode, and remote rejection.

## Validation

- `pnpm test --runInBand` (21 suites passed, 1 skipped; 908 tests
passed, 6 skipped)
- `pnpm lint-ci`
- `pnpm tsc`
- `pnpm run docs`
- `cargo fmt --all -- --check`
- `CC=clang cargo clippy --profile ci -p lancedb-nodejs --all-features
-- -D warnings`

Fixes #4267
 2026-09-28 20:19:36 -05:00
Wyatt Alt 604b1b583b feat: pause and resume jobs from the Python SDK (#4112) 
Cancellation is the only job control the SDK exposes, and it is
terminal, so a long-running server-side job cannot be parked and picked
up again. This adds `pause_job` and `resume_job` to the Rust core
connection and the Python bindings (sync and async), posting to the
server's `/v1/jobs/pause` and `/v1/jobs/resume` endpoints.

A pause parks the job until it is resumed: its workers drain and stop.
The outcome strings mirror the server's answers — a job finalizing its
results reports `"committing"` and cannot be parked, and a resume before
the drain is confirmed reports `"still_pausing"`; both are retried
rather than failed. Resuming re-queues the job and its workers pick
their work back up from checkpoints. Local connections report the
operations as unsupported, like the rest of the jobs API.
 2026-09-24 23:47:00 +08:00
Jack Ye e4d0f9e2da feat: return a cleanup job from drop_view (#4262) 
Dropping a view unbinds its name and leaves the definition dataset to a
server-side cleanup job, so the two are separate events a caller may
want to wait on.

`drop_view_async` returns that job — the same shape
`drop_materialized_view_async` and `drop_function_async` already use: a
`202` carries the job id, a `200` (nothing was bound to the name) yields
an already-finished job with no id, and any other success status is an
error rather than a silent no-op.

## `drop_view` waits

`drop_view` now awaits the job before returning, so a caller who does
not want to think about cleanup gets the stronger guarantee: when it
returns, the definition really is deleted.

That is deliberately **different** from `drop_materialized_view` and
`drop_function`, which return as soon as the name is unbound and
document that content may still be deleting. The view API is the newer
one, and waiting is the semantic worth having; the other two are left
alone rather than changing behaviour already released.

## Surfaces

`Database` trait, the remote client, `Connection`, and the Python and
Node bindings — matching where `drop_materialized_view_async` is already
exposed.

Four client tests cover the accepted case reporting its job id, the
nothing-bound case reporting a finished job, a `202` without a usable
`job_id`, and an unexpected success status.
 2026-09-24 00:27:20 -07:00
Jack Ye 8541d6d9df feat: add view CRUD APIs (#4236) 
A view is a named query a database stores and plans on every read. It
holds no rows, which is the whole difference from a materialized view.

## API

| Verb | Route |
| --- | --- |
| `create_view(name, query, namespace_path)` | `POST
/v1/view/{id}/create` |
| `describe_view(name, namespace_path)` | `POST /v1/view/{id}/describe`
|
| `drop_view(name, namespace_path)` | `POST /v1/view/{id}/drop` |
| `list_views(namespace_path)` | `GET /v1/namespace/{id}/view/list` |

On `Connection` and the `Database` trait, with the remote client, Python
(sync and async) and Node bindings. Local databases return
`NotSupported`: the server side is Sophon's, where a view is an object
of the database manifest.

`ViewDescription` carries the defining query, the database *and
namespace path* unqualified names in it resolve against, and the schema
the query resolved to. `create_view` returns one, so a caller has the
schema without a second call.

Both defaults travel with the view because it outlives the session that
declared it: the server re-plans the stored query on every read, so a
reader resolving an unqualified name against its own defaults would read
a different table. `default_namespace_path` crosses the wire as
`default_namespace`, a path like `namespace`, absent for the root.

There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create, each authorized against what it
actually touches.

Querying a view stays SQL's job. There are no rows behind a view, so
there is no `open_view` returning a `Table`.
 2026-09-23 15:09:40 +08:00
Wyatt Alt 01ee01dbc8 feat: define a materialized view by its query, with Functions in FROM position (#4190) 
A view definition was a structured record under a `kind` tag, one kind
per query shape, and a Function returning `list<struct>` was about to
add a third. That names shapes instead of describing a relation.

A materialized view is now a relation defined by a query, stored as one
canonical SQL string under a format number:

```sql
SELECT columns FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias]
[WHERE predicate] [LIMIT n]
```

Any other clause is refused at parse time. Older readers report the view
as unrefreshable, the pre-format layouts still read, and a legacy view
is rewritten on its next refresh that commits, rebuilt only where its
raw text meant something else under lance's parser.

A Function in FROM position yields one row per element it returns, as a
table function does in any dialect. The server stages its list output in
a hidden table and records that binding beside the query, which stays as
the user wrote it; refresh scans the staging and unnests the column, the
same operator as UNNEST over a list column the table already holds. Row
ids repeat per element, so eviction and incremental append are
unchanged. A local database refuses a Function in FROM position, since
it has no executor.
 2026-09-18 12:16:25 -07:00
Wyatt Alt 5231d37f8d feat: list a table's per-row Function errors from the client (#4208) 
A refresh running under a skip policy records each row it skipped, with
the failing input and the error, but the client could not read that
store: the server exposes it over SQL and, since recently, a REST route.
A user who hit per-row failures still had to open a SQL session.

`Table::function_errors` calls the route. The listing is table-addressed
with optional job and column filters, the same addressing the SQL
surface uses, so the two cannot disagree about what a table's errors
are. The two non-record signals come back as their own fields rather
than as rows: capped-fragment summaries, and whether the listing stopped
at its limit. Local tables refuse rather than answer with an empty list.

Python and Node expose the same call, with the same optional filters.
 2026-09-17 05:12:50 -07:00
Jonathan HsiehandClaude Opus 5 60a1b4c219 feat(secrets): named Secrets, bindings, and namespace addressing (#4150) 
Adds the client half of database-scoped named Secrets: a Secret is a
name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads. Secrets are addressed by
a
namespace path plus a name.

The UDF body is unchanged and stays portable — it reads `OPENAI_API_KEY`
the
way it always did, and the binding is what puts a value there:

```python
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
function = db.create_function(
    analyze_caption,
    secrets=[
        EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
    ],
)
function.secret_bindings   # the Secret's name, never its value
```

- `create_secret` / `alter_secret` / `list_secrets` / `describe_secret`
/
`drop_secret` on sync, async and remote connections, with the pyo3
binding
and the Rust client behind them. Each takes `namespace_path`
keyword-only,
  defaulting to the root.
- **There is no read API, by construction rather than by policy** — no
code
  path returns a stored credential, and `describe_secret` answers with
  metadata only.
- `EnvVarSecret` is a pure local constructor: it contacts no server, so
it
cannot fail on a Secret that does not exist. It exists so that a bare
string
in that position — which would be a credential — is a `TypeError` rather
  than a plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`: a list of `SecretBinding` tagged by `kind`, so a
later
delivery mode is a variant rather than a sibling field. The value never
travels — it is resolved by the service when the Function runs, which is
what
lets a rotation reach columns already pinned to an older
FunctionVersion.
- A binding names its Secret as a `SecretReference` of `{name,
namespace_path}`
rather than one joined string, so no delimiter has to be excluded from
every
name and segment forever, and `ClientConfig.id_delimiter` cannot
contradict
  an identity built on a fixed separator.
- A root namespace is omitted from the request body rather than sent
empty, so
  a root request is byte-identical to one from a client that predates
  namespaces. Tests pin it.

This is the client surface the design's §4 describes; the service side
lives in
sophon.

**Previously split across two PRs.** Namespace addressing was #4151,
stacked on
this one; it is folded in here so the Secret identity contract — name,
namespace path, and the binding that carries both — is reviewable as one
piece
rather than as a shape introduced and then replaced.

## Identifier safety, merged from #4189

**#4189 is merged into this branch**, so the client half of Secrets and
the
guards on the identity it puts in the URL are one PR. What it added:

- Components are checked where the identifier is built, before a request
is
  constructed. `create_secret("../jobs", value)` no longer resolves to
`/v1/jobs/create` and delivers a credential-bearing body to a route with
none
  of this one's body suppression.
- Each component is percent-encoded and joined by the delimiter, so
nothing
  inside a component can end the path segment or add one.
- A component may not be empty, a relative segment (`.`, `..`, and their
`%2e`
spellings), or the delimiter itself — the three ways a component erases
a
boundary the split has to recover. `["prod", ""]` joined to `prod$`,
which
  reads back as `["prod"]`.
- `$` is the only accepted `id_delimiter`, refused at client
construction.
`ClientConfig.id_delimiter` remains, since the identifier grammar comes
from
the Lance REST catalog standard, but a value that would produce
identifiers no
  service splits the caller's way is now an error where it was written.
- One `build_object_identifier` and one character set serve tables,
namespaces,
  Secrets, Functions and materialized views.

Components are checked for *addressability*, not a character set: the
name's
own grammar stays each object's own, so a catalog database keeps the `/`
that
`RemoteCatalog::validate_name` allows.

## Known shortcoming

`secret_bindings` is omitted from a registration body when empty, so a
client
that binds nothing sends what a client without bindings sends. When a
client
does bind a Secret and the service does not know the field, the field is
ignored: registration succeeds, the returned version carries no
bindings, and
the Function fails at execution with the variable unset, far from the
call that
asked for it.

`ServerVersion` is how this codebase refuses a feature the service is
too old
for, and it gates five features already. It does not gate this one: it
is held
per table, and registering a Function is a database-level call. Noted at
the
field in `remote/db.rs`; wiring the gate is follow-up work.

**Tests:** lancedb lib 1340 passed, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, plus Python tests across both slices.
Rebased onto `main` after #4176 (OCI Function identity), #4191 (`.`/`..`
table
names) and #4195 (remote catalogs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 17:00:23 -07:00
Jack YeandXuanwo 6a07f88980 feat: add remote catalogs and Python and TypeScript bindings (#4195) 
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.

## Rust API

`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.

```rust
use lancedb::catalog::{
    CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};

let catalog = lancedb::connect_catalog("https://my-server.example")
    .api_key("my-api-key")
    .execute()
    .await?;

let db = catalog.create_database(
    CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
    ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
    DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```

Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.

## Python API

```python
import lancedb

catalog = lancedb.connect_catalog(
    "https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
    next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```

`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.

## TypeScript API

```typescript
import { connectCatalog } from "@lancedb/lancedb";

const catalog = await connectCatalog("https://my-server.example", {
  apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
  const nextPage = await catalog.listDatabases({
    limit: 20, pageToken: page.pageToken,
  });
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```

Create/connect return the existing `Connection` API. All four methods
are asynchronous.

## REST mapping

All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.

| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |

For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.

Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.

For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.

Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.

Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.

## Scope

This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.

The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-17 01:51:10 +08:00
Jack YeandXuanwo f3ef21b8ca feat: use oauth2 crate for OAuth with configurable client auth (#4181) 
Stacked on #4173 (`jack/restore-oidc-flows`, base branch mirrored to
this repo so the diff shows only this change); context from review:
https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100.
Rebase to `main` once #4173 and #4179 merge.

## What moved to the `oauth2` crate (5.0, no default features)

- Authorization URL generation and CSRF state (`authorize_url`,
`CsrfToken`)
- PKCE S256 challenge/verifier generation and code exchange
- Client-credentials, authorization-code, refresh-token, and device-code
grant request construction
- Device authorization request and the device token polling loop
(`authorization_pending`, `slow_down` +5s, expiry deadline, denial,
network backoff capped at 10s)
- Standard success/error response parsing (`RequestTokenError`)
- Token-endpoint client authentication and standards-compliant parameter
encoding (RFC 6749 2.3.1 Basic encoding)

LanceDB keeps ownership of OIDC discovery (compared `openidconnect`: no
measurable win for our 3-field metadata + strict validation, at real
dependency cost), HTTPS-or-loopback endpoint enforcement, the loopback
callback server, browser/stderr prompts, token caching and refresh
orchestration, and the dedicated hardened Azure IMDS source, which is
unchanged.

## Client authentication methods

New `ClientAuthMethod` enum (`none` | `client_secret_basic` |
`client_secret_post`), exposed in Rust, Python, and Node. Unset resolves
to `client_secret_basic` when a secret is present (RFC 6749 2.3.1
recommendation and the normal Okta confidential-app default, so a
default Okta app works without weakening its configuration) and to
`none` for public clients (PKCE/device). Explicit `none` with a secret,
or basic/post without one, is rejected. The method applies to client
credentials, code exchange, refresh, and device requests. Deliberate
behavior change: confidential clients previously always sent the secret
in the POST body; they now default to Basic (Keycloak accepts both).

No `audience`/`resource` parameters were added: the supported target is
an Okta custom authorization server with the API audience configured
server-side, so client-provided audience parameters are unnecessary;
`add_extra_param` support exists if a concrete provider contract ever
needs them.

## Device polling behavior changes (deliberate, tested)

- The first token poll now happens immediately rather than after one
interval (RFC 8628 allows both).
- Transient failures (HTTP 429, 5xx, `temporarily_unavailable`, network
errors) now retry with exponential backoff capped at 10s instead of
retrying at the fixed interval; polling never spins faster than once per
second even if a server reports a zero interval.

## Security and compatibility

- Issuer and discovered endpoints (and device verification URIs) still
require HTTPS except explicit loopback HTTP, enforced before any crate
URL type is built
- Token HTTP client keeps the hardened redirect policy that refuses
insecure redirect targets; regression test added
- Errors never embed raw response bodies (avoids leaking tokens through
parse failures); all credential types stay redacted in Debug
- Transient conditions (429/5xx/`temporarily_unavailable`) remain
retryable in device polling and hard errors elsewhere; refresh keeps
rotation and reauthentication semantics
- Existing public APIs stay source-compatible except the added
`OAuthConfig.client_auth_method` field

## Tests

Rust: client-auth methods across code
exchange/refresh/client-credentials/device (none/basic/post),
auth-method resolution and validation, transient device retries,
denial/expiry, redirect rejection, malformed-response leak check, PKCE
URL assertions, redaction. Python and Node: enum values, conversion,
unknown-method errors, config defaults.

Manual Okta validation recipe (no automated Okta credentials): create a
custom authorization server with an API audience, one confidential web
app (Basic) for authorization-code, one native app (PKCE, no secret),
one native device app; point `issuer_url` at the custom server, set
`client_auth_method` only for the POST-required case; verify token
acquisition, refresh after expiry, and `x-lancedb-credential-type: oidc`
against a LanceDB deployment. Never commit tenant URLs or secrets.

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-16 20:11:08 +08:00
Colin Patrick McCabe 3a1d3be256 feat(oidc): support resource and audience (#4193) 
Support configuring resource and audience for OAuth authorization, token
exchange, and refresh requests.
 2026-09-15 16:35:37 -07:00
Jack Ye f12996557f feat(remote): support materialized view APIs (#4180) 
## Summary

Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.

Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.

## Route comparison

| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |

Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.

## Python API changes

| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |

The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
 2026-09-15 12:13:39 -07:00
Jack YeandXuanwo 2f88b71c21 feat: add persistent OAuth token cache and session APIs (#4182) 
Stacked on #4173 (diff includes it until that merges; will rebase
after). Addresses the token-cache part of [Colin's
review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100).

Adds an explicit, opt-in persistent OAuth token cache shared by Rust,
Python, and Node clients, plus `login` / `status` / `logout` session
APIs, so short-lived processes (CLIs, scripts, notebooks) reuse one
session instead of restarting a browser or device flow on every start.

- **Opt-in and minimal**: existing callers stay memory-only and lazy.
Only refresh tokens are persisted (never access tokens, never client
secrets), so there are no local token-expiry decisions to get wrong when
clocks move. Each process start performs one silent refresh grant.
- **Hardened file backend**: private directory (`0700`), per-record
files (`0600`), owner validation, symlink rejection, and atomic `rename`
replacement. Corrupt, truncated, unknown-version, or permission-invalid
records fail with actionable errors naming the file. Native keyring
backends were evaluated (keyring crate routes Linux through D-Bus/zbus:
heavy deps, headless/CI flakiness) and are deferred; the file store is
the explicit opt-in, not a downgrade from a keyring.
- **Cache key**: SHA-256 of the canonical identity (issuer, client ID,
sorted/de-duplicated scopes, flow, public/confidential), so no secret
appears in a filename and distinct identities never collide. Versioned
record schema (`version: 1`). One record per identity: last login wins,
documented.
- **Cross-process rotation locking**: per-key `fs4` file lock (`flock` /
`LockFileEx`) around the refresh critical section — acquire, reread the
durable record, refresh exactly once, atomically store the rotated
refresh token, release. The OS releases locks on process death, so
crashes cannot strand stale locks. Only confirmed
`invalid_grant`/`invalid_token` deletes a record and reauthenticates;
transport, 5xx, 429, and parse failures retain it.
- **Session APIs**: `OAuthSession::login/status/logout` in Rust,
`lancedb.remote.OAuthSession` (async) in Python, `OAuthSession` class in
Node. `status` returns non-secret metadata only. `logout` removes only
the local credential — provider revocation (RFC 7009) is a deliberate
follow-up, and local logout never terminates browser SSO. Azure managed
identity is rejected for persistence (machine identity stays in memory);
client credentials have nothing refreshable to persist and stay
memory-only.
- No CLI binary exists in this repo, so this ships library APIs plus doc
examples in all three languages.

Tests: Rust unit + mock-IdP integration (cache-key
canonicalization/separation, record
versioning/corruption/truncation/symlink/owner/perms, lock serialization
+ release, two concurrent providers proving no `invalid_grant` and
correct rotation, transient-failure retention, `invalid_grant` delete +
reauthenticate, login/status/logout lifecycle, client-credentials no-op,
IMDS rejection, secret redaction); Python lifecycle + a true
two-subprocess cross-process reuse test (second process refreshes once,
never hits the device endpoint); Node lifecycle + device-flow login
test. Local builds were skipped in development; CI validates all
bindings.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-16 02:01:10 +08:00
lancedb-gatefixer[bot]andXuanwo ffe94a65a1 fix(node): preserve optimize cleanup timestamp (#4160) 
<!-- lance-gatekeeper-fix:v1 agent=278cb095b2e5d69051442bc254d803a6
generation=1 -->

## Summary

- pass the TypeScript `cleanupOlderThan` date to the native binding as
an unchanged epoch timestamp
- prune with Lance's absolute `before_timestamp` policy so dispatch and
compaction time cannot move the cutoff
- retain versions created after the supplied cutoff and document that
behavior
- add boundary and end-to-end regression coverage

## Root cause

The TypeScript layer converted the absolute date into an elapsed
duration before calling native optimize. Lance converted that duration
back into a timestamp only after compaction, which silently advanced the
requested cutoff and made the cleanup count depend on a millisecond
timing boundary.

## Validation

- `cargo fmt --all`
- `cargo clippy --quiet --features remote --tests --examples -p lancedb
-p lancedb-nodejs`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test __test__/table.test.ts --runInBand` (309 passed)

Fixes #4159

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-15 14:10:48 +08:00
LanceDB Robot b8f0048b5a chore: update lance dependency to v12.0.0-beta.18 (#4164) 
Update the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v12.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.18).
Fix redundant visibility declarations in the Node.js Rust blob helpers
required by the workspace Clippy check.
Validated with workspace Clippy (all features and tests, warnings
denied), cargo fmt, pnpm build, and 13 targeted Node.js blob tests.
 2026-09-12 22:25:12 -07:00
Drew 6702e3fec1 feat(node): add blob v2 fetch and field helpers (#4155) 
this PR blob v2 field helpers and reads to the Node SDK.

`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.

Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.

Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.

```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
  "videos",
  [{ id: 1n, video }],
  {
    schema: new Schema([
      new Field("id", new Int64()),
      blob("video"),
    ]),
  },
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);

```

### Testing

- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
 2026-09-11 14:46:27 -07:00
LanceDB RobotandJack Ye a487d4033e chore: update lance dependency to v12.0.0-beta.14 (#4141) 
Update the Rust workspace Lance dependencies and Java lance-core from
v12.0.0-beta.11 to
[v12.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.14),
refreshing Cargo.lock. Resolve two Clippy diagnostics by making an
internal Node.js helper private and using a byte string literal in a
remote-table test fixture.

Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `git diff --check`, and `pnpm
build` in nodejs.

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
 2026-09-06 13:43:35 -07:00
Jack Ye 21f11b4463 feat!: replace get_job/job_history with describe_job/query_job_events (#4130) 
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.

## Before

Four ways to ask about a job, none of which told you much.

```python
job = table.refresh_column_async("embedding")
job.status()               # "running". That was the entire debug surface.
db.get_job(job_id)         # state, and a spec. No result, no progress.
db.job_history(job_id)     # raw record batches, no limit, no filter
db.job(job_id)             # a handle that knew nothing
```

## After

Open a job the way you open a table; the handle answers everything.

```python
job = db.open_job(job_id)      # raises JobNotFoundError if there is no such job
```

```python
>>> print(job)
Job(
    id='job-1',
    state='failed',
    job_type='refresh_column',
    creation_ms=1757000000000,
    spec={
        "column": "embedding",
        "num_workers": 4
    },
    failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```

Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.

Per-fragment progress *while it is still running*:

```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist()      # [5000, 5000, ...]
```

The handle an async action returns is the same object, one `refresh()`
away:

```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```

TypeScript is the same experience, down to `console.log`:

```ts
const job = await db.openJob(jobId);   // rejects if there is no such job
console.log(job);                      // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```

## Why each piece matters

- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.

`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.

## Breaking

All shipped in 0.38.0. No deprecated aliases.

| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |

Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
 2026-09-05 16:24:23 -07:00
dependabot[bot]Will JonesClaude Sonnet 5
 		e773d1e093 build(deps): bump the rust-minor-patch group across 1 directory with 9 updates (#4084) 
Bumps the rust-minor-patch group with 9 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-trait](https://github.com/dtolnay/async-trait) | `0.1.91` |
`0.1.92` |
| [log](https://github.com/rust-lang/log) | `0.4.33` | `0.4.34` |
| [moka](https://github.com/moka-rs/moka) | `0.12.15` | `0.12.16` |
| [uuid](https://github.com/uuid-rs/uuid) | `1.24.0` | `1.26.0` |
| [serde_with](https://github.com/jonasbb/serde_with) | `3.21.0` |
`3.22.0` |
| [roaring](https://github.com/RoaringBitmap/roaring-rs) | `0.11.4` |
`0.11.5` |
| [napi](https://github.com/napi-rs/napi-rs) | `3.11.0` | `3.12.0` |
| [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.1` | `3.6.3`
|
| [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.0` | `2.4.1` |


Updates `async-trait` from 0.1.91 to 0.1.92
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/async-trait/releases">async-trait's
releases</a>.</em></p>
<blockquote>
<h2>0.1.92</h2>
<ul>
<li>Resolve double_must_use clippy lint in generated code (<a
href="https://redirect.github.com/dtolnay/async-trait/issues/303">#303</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/async-trait/commit/82e7e9edd60f622294373a23c0ce9c0077ad0263"><code>82e7e9e</code></a>
Release 0.1.92</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/9a35cb87f9366cd992bbc00d430e1b5fe1aa0cdd"><code>9a35cb8</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/async-trait/issues/303">#303</a>
from dtolnay/mustuse</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/875ceecb100bab2cf369178633b4791336d92b75"><code>875ceec</code></a>
Resolve double_must_use clippy lint</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/62993a57bc6a8d5bd3de23fbae48cede333cb925"><code>62993a5</code></a>
Raise minimum tested compiler to rust 1.88</li>
<li>See full diff in <a
href="https://github.com/dtolnay/async-trait/compare/0.1.91...0.1.92">compare
view</a></li>
</ul>
</details>
<br />

Updates `log` from 0.4.33 to 0.4.34
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/log/releases">log's
releases</a>.</em></p>
<blockquote>
<h2>0.4.34</h2>
<h2>What's Changed</h2>
<ul>
<li>doc: Add context-logger utility to README by <a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li>Add alloc support for boxed loggers by <a
href="https://github.com/malezjaa"><code>@​malezjaa</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li><a href="https://github.com/malezjaa"><code>@​malezjaa</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">https://github.com/rust-lang/log/compare/0.4.33...0.4.34</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/log/blob/master/CHANGELOG.md">log's
changelog</a>.</em></p>
<blockquote>
<h2>[0.4.34] - 2026-08-22</h2>
<h2>What's Changed</h2>
<ul>
<li>doc: Add context-logger utility to README by <a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li>Add alloc support for boxed loggers by <a
href="https://github.com/malezjaa"><code>@​malezjaa</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li><a href="https://github.com/malezjaa"><code>@​malezjaa</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">https://github.com/rust-lang/log/compare/0.4.33...0.4.34</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rust-lang/log/commit/8034743dd9d7f7583bd9a670271483d176130911"><code>8034743</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/738">#738</a>
from rust-lang/cargo/0.4.34</li>
<li><a
href="https://github.com/rust-lang/log/commit/7d1e24e3506d4ffa1badf6c9ea357779877adaf0"><code>7d1e24e</code></a>
prepare for 0.4.34 release</li>
<li><a
href="https://github.com/rust-lang/log/commit/3b939b6714616dc32193c12019861c7c518c5edb"><code>3b939b6</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/737">#737</a>
from malezjaa/master</li>
<li><a
href="https://github.com/rust-lang/log/commit/b88266cfed8b287f8c35b2015808b09b056f61af"><code>b88266c</code></a>
Add alloc support for boxed loggers</li>
<li><a
href="https://github.com/rust-lang/log/commit/037d7a58f6ad184abb3afc4db81d37c43a5696ec"><code>037d7a5</code></a>
doc: Add context-logger utility to README</li>
<li>See full diff in <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">compare
view</a></li>
</ul>
</details>
<br />

Updates `moka` from 0.12.15 to 0.12.16
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/moka-rs/moka/releases">moka's
releases</a>.</em></p>
<blockquote>
<h2>v0.12.16</h2>
<h2>Version 0.12.16</h2>
<h3>Fixed</h3>
<ul>
<li>Fixed a bug where cache eviction could stall permanently when the
cache was configured with the <strong>non-default</strong> LRU eviction
policy (<code>EvictionPolicy::lru()</code>) by a race between insert and
remove operations on the same key (<a
href="https://redirect.github.com/moka-rs/moka/issues/592">#592</a><a
href="https://redirect.github.com/moka-rs/moka/pull/592/">gh-pull-0592</a>
by <a
href="https://github.com/kim-jhyeon"><code>@​kim-jhyeon</code></a>,
reported in <a
href="https://redirect.github.com/moka-rs/moka/issues/590">#590</a><a
href="https://redirect.github.com/moka-rs/moka/issues/590/">gh-issue-0590</a>):
<ul>
<li>This bug was introduced in v0.12.0 and affected
<code>sync::Cache</code>, <code>sync::SegmentedCache</code> and
<code>future::Cache</code>.</li>
<li>A race between applying a write recording for an entry and
concurrently removing that entry from the internal concurrent hash table
could leave an orphaned node at the front of the LRU queue. Once
present, no entry was ever evicted again and the cache grew unboundedly
past <code>max_capacity</code>.</li>
<li>The same race also affected the default TinyLFU eviction policy, but
with a milder symptom: each occurrence permanently leaked one phantom
entry slot, causing <code>entry_count</code> and
<code>weighted_size</code> to over-report and the usable capacity to
shrink by one entry per occurrence. Fixed by the same change.</li>
</ul>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Worked around a ThreadSanitizer false positive (<a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a><a
href="https://redirect.github.com/moka-rs/moka/pull/602/">gh-pull-0602</a>):
<ul>
<li>Replaced the standalone <code>fence(Acquire)</code> in the internal
<code>MiniArc</code>'s drop path with an <code>Acquire</code> load of
the reference count, so that downstream projects can now run
ThreadSanitizer on code using Moka without hitting this false
positive.</li>
<li><code>std::sync::Arc</code> has a similar workaround.</li>
</ul>
</li>
<li>Raised the minimum version of the <code>crossbeam-epoch</code> crate
from <code>v0.9.18</code> to <code>v0.9.20</code> to avoid the following
advisory (<a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a><a
href="https://redirect.github.com/moka-rs/moka/pull/603/">gh-pull-0603</a>):
<ul>
<li>[RUSTSEC-2026-0204] crossbeam-epoch: invalid pointer dereference in
<code>fmt::Pointer</code> for <code>Atomic</code> and
<code>Shared</code></li>
<li>Moka is <em>not</em> affected by this advisory because it never
formats these pointer types. However, raising the minimum version
prevents downstream lockfiles from resolving to an affected
<code>crossbeam-epoch</code> version via Moka.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/moka-rs/moka/blob/main/CHANGELOG.md">moka's
changelog</a>.</em></p>
<blockquote>
<h2>Version 0.12.16</h2>
<h3>Fixed</h3>
<ul>
<li>Fixed a bug where cache eviction could stall permanently when the
cache was
configured with the <strong>non-default</strong> LRU eviction policy
(<code>EvictionPolicy::lru()</code>)
by a race between insert and remove operations on the same key
(<a
href="https://redirect.github.com/moka-rs/moka/issues/592">#592</a>[gh-pull-0592]
by [<a
href="https://github.com/kim-jhyeon"><code>@​kim-jhyeon</code></a>][gh-kim-jhyeon],
reported in
<a
href="https://redirect.github.com/moka-rs/moka/issues/590">#590</a>[gh-issue-0590]):
<ul>
<li>This bug was introduced in v0.12.0 and affected
<code>sync::Cache</code>,
<code>sync::SegmentedCache</code> and <code>future::Cache</code>.</li>
<li>A race between applying a write recording for an entry and
concurrently
removing that entry from the internal concurrent hash table could leave
an
orphaned node at the front of the LRU queue. Once present, no entry was
ever
evicted again and the cache grew unboundedly past
<code>max_capacity</code>.</li>
<li>The same race also affected the default TinyLFU eviction policy, but
with
a milder symptom: each occurrence permanently leaked one phantom entry
slot, causing <code>entry_count</code> and <code>weighted_size</code> to
over-report and the
usable capacity to shrink by one entry per occurrence. Fixed by the same
change.</li>
</ul>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Worked around a ThreadSanitizer false positive (<a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a>[gh-pull-0602]):
<ul>
<li>Replaced the standalone <code>fence(Acquire)</code> in the internal
<code>MiniArc</code>'s drop
path with an <code>Acquire</code> load of the reference count, so that
downstream
projects can now run ThreadSanitizer on code using Moka without hitting
this false positive.</li>
<li><code>std::sync::Arc</code> has a similar workaround.</li>
</ul>
</li>
<li>Raised the minimum version of the <code>crossbeam-epoch</code> crate
from <code>v0.9.18</code> to
<code>v0.9.20</code> to avoid the following advisory (<a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a>[gh-pull-0603]):
<ul>
<li>[RUSTSEC-2026-0204] crossbeam-epoch: invalid pointer dereference in
<code>fmt::Pointer</code> for <code>Atomic</code> and
<code>Shared</code></li>
<li>Moka is <em>not</em> affected by this advisory because it never
formats these
pointer types. However, raising the minimum version prevents downstream
lockfiles from resolving to an affected <code>crossbeam-epoch</code>
version via
Moka.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/moka-rs/moka/commit/a616ec19e8d4ed938caf8b2c88090331d778d5da"><code>a616ec1</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/604">#604</a> from
moka-rs/chore/bump-v0.12.16</li>
<li><a
href="https://github.com/moka-rs/moka/commit/3b140a627e9faa4ec6a8e682224c7f81efc2b6e4"><code>3b140a6</code></a>
Bump the version to v0.12.16</li>
<li><a
href="https://github.com/moka-rs/moka/commit/51b802dc5cfc9e0da21de04177d79028bfbe5d47"><code>51b802d</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a> from
moka-rs/bump-crossbeam-epoch-floor</li>
<li><a
href="https://github.com/moka-rs/moka/commit/4f9071684161d59212c32a0e89762c3a5d6385a4"><code>4f90716</code></a>
Raise the minimum crossbeam-epoch version to 0.9.20</li>
<li><a
href="https://github.com/moka-rs/moka/commit/08d0e0458bd95af7f9435ff3ffbba6d1e91647c1"><code>08d0e04</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a> from
moka-rs/gh600-tsan-workaround</li>
<li><a
href="https://github.com/moka-rs/moka/commit/14447a7cbe441639e2aa3570e411fe493c71c9ac"><code>14447a7</code></a>
Restructure the v0.12.16 TSan workaround CHANGELOG entry</li>
<li><a
href="https://github.com/moka-rs/moka/commit/7b14c37b009a25c9a2ec27e2669dc5f8db7ce254"><code>7b14c37</code></a>
Avoid a TSan false positive by replacing the fence in MiniArc::drop</li>
<li><a
href="https://github.com/moka-rs/moka/commit/05b37c63098473034e7e961c1010284163ad8634"><code>05b37c6</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/599">#599</a> from
moka-rs/gh590-deterministic-tests</li>
<li><a
href="https://github.com/moka-rs/moka/commit/fc318584d25c0ea01109872da37d395754647e04"><code>fc31858</code></a>
Replace private doc references in gh590 test comments</li>
<li><a
href="https://github.com/moka-rs/moka/commit/57435922036ff4ab9f1b5bd0c3bffe6c8acd9921"><code>5743592</code></a>
Improve the v0.12.16 CHANGELOG entry</li>
<li>Additional commits viewable in <a
href="https://github.com/moka-rs/moka/compare/v0.12.15...v0.12.16">compare
view</a></li>
</ul>
</details>
<br />

Updates `uuid` from 1.24.0 to 1.26.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/uuid-rs/uuid/releases">uuid's
releases</a>.</em></p>
<blockquote>
<h2>v1.26.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Add ContextV7::with_additional_precision_bits by <a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/904">uuid-rs/uuid#904</a></li>
<li>Prepare for 1.26.0 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/905">uuid-rs/uuid#905</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/1.25.0...v1.26.0">https://github.com/uuid-rs/uuid/compare/1.25.0...v1.26.0</a></p>
<h2>1.25.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Add a serde::bytes module that encodes a Uuid as a byte string by <a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/902">uuid-rs/uuid#902</a></li>
<li>Prepare for 1.25.0 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/903">uuid-rs/uuid#903</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> made
their first contribution in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/902">uuid-rs/uuid#902</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.1...1.25.0">https://github.com/uuid-rs/uuid/compare/v1.24.1...1.25.0</a></p>
<h2>v1.24.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Fix non-ASCII character handling in parse diagnostics by <a
href="https://github.com/questfever"><code>@​questfever</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/899">uuid-rs/uuid#899</a></li>
<li>Prepare for 1.24.1 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/900">uuid-rs/uuid#900</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/questfever"><code>@​questfever</code></a> made
their first contribution in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/899">uuid-rs/uuid#899</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.24.1">https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.24.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/uuid-rs/uuid/commit/cdc96a87bddc38d0eb8f894c764e151d2299b4b3"><code>cdc96a8</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/905">#905</a> from
uuid-rs/cargo/v1.26.0</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/34e4f49c0d50c12f1b3021baf98b8fb91f6407bb"><code>34e4f49</code></a>
don't test macros under miri</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/d9e7242b37755d844d19fa74559a88e1c46c5206"><code>d9e7242</code></a>
update nightly used for miri</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/ec16819865b89aa3c52456c8afd0ce9a90f0fcdb"><code>ec16819</code></a>
prepare for 1.26.0 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/162cd208a4521138f1d8ce05b63342ba7ba5c4e6"><code>162cd20</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/904">#904</a> from
ChrisJr404/v7-additional-precision-bits</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/97eceffa708f87969792af604291d3e4984dfc90"><code>97eceff</code></a>
Add ContextV7::with_additional_precision_bits for microsecond
clocks</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/302e0bf6dc5abf949c06973a37f1f3a093cc2699"><code>302e0bf</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/903">#903</a> from
uuid-rs/cargo/1.25.0</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/b7ccde885d770d013f413a2685ebe7f38932e1d0"><code>b7ccde8</code></a>
prepare for 1.25.0 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/c62dffbc038034ff045f3009f2536362e313bf34"><code>c62dffb</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/902">#902</a> from
ChrisJr404/serde-bytes-module</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/8c198b24b1aa55948c0fa4b3433c1954be19c8c8"><code>8c198b2</code></a>
Add a serde::bytes module that encodes as a byte string</li>
<li>Additional commits viewable in <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.26.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `serde_with` from 3.21.0 to 3.22.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/jonasbb/serde_with/releases">serde_with's
releases</a>.</em></p>
<blockquote>
<h2>serde_with v3.22.0</h2>
<h3>Added</h3>
<ul>
<li>Add support for <code>jiff</code> v0.2 behind the new
<code>jiff_0_2</code> feature flag (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/936">#936</a>)
<code>jiff::SignedDuration</code> works with
<code>DurationSeconds</code> and its variants.
<code>jiff::Timestamp</code>, <code>jiff::Zoned</code>, and
<code>jiff::civil::DateTime</code> work with
<code>TimestampSeconds</code> and its variants.
Deserializing a <code>jiff::Zoned</code> uses the system time zone, like
<code>chrono::DateTime&lt;Local&gt;</code>.</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>Extend the <a
href="https://github.com/jonasbb/serde_with/security/advisories/GHSA-7gcf-g7xr-8hxj">GHSA-7gcf-g7xr-8hxj</a>
fix to the duplicate-key-prevention collections.
The <code>rust::sets_duplicate_value_is_error</code>,
<code>rust::maps_duplicate_key_is_error</code>,
<code>rust::sets_last_value_wins</code>, and
<code>rust::maps_first_key_wins</code> adapters created their backing
sets/maps with <code>with_capacity_and_hasher</code> using the raw
deserializer <code>size_hint</code>, bypassing the
<code>size_hint_cautious</code> cap added in <a
href="https://redirect.github.com/jonasbb/serde_with/issues/966">#966</a>
(the <code>clippy.toml</code> <code>disallowed_methods</code> lint only
covers <code>Vec::with_capacity</code>, not
<code>with_capacity_and_hasher</code>, so these sites were not flagged).
Attacker-controlled input claiming a huge length could panic with
<code>Hash table capacity overflow</code> before a single element was
read. All such constructions now route through
<code>size_hint_cautious</code>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/jonasbb/serde_with/commit/88f576a17c5cd45cea6a30252ef10653dde69fa8"><code>88f576a</code></a>
Bump version to 3.22.0 (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/991">#991</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/931e664445c2139446b84e76b339924f136a1565"><code>931e664</code></a>
Bump version to 3.22.0</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/e26930e0b7a6c1e6463086a2447e7cc8fc6f0a24"><code>e26930e</code></a>
Bump github/codeql-action from 4.37.3 to 4.37.4 in the github-actions
group (...</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/92cd5a0bd5c7a80fc7eae90bb99b873c40429aa3"><code>92cd5a0</code></a>
Bump github/codeql-action in the github-actions group</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/32be66fecc5c1fe4c90ac0230c0af04d1977df53"><code>32be66f</code></a>
Guard with_capacity_and_hasher against untrusted size_hint (DoS) (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/971">#971</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/33871cd4dd1ecef2c3af0ead9528c8b407c16f04"><code>33871cd</code></a>
Merge branch 'master' into
fix/duplicate-key-impls-capacity-overflow</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/bb1e06484261595c8cec7fd7f4ed33ecdfb951c0"><code>bb1e064</code></a>
Change function position within impl (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/968">#968</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/202d3dd617d7b5a9db5f490fa752d6ccb48454e8"><code>202d3dd</code></a>
Improve the time unit macros to remove unnecessary repetition and make
the co...</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/b347efb536caf83c850d4808f90503835fd78755"><code>b347efb</code></a>
Move the <code>use_duration_signed_ser</code>/<code>*_de</code> macros
utils</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/65905455527c0abf51f2f906bc08724426b2b922"><code>6590545</code></a>
chrono_0_4: Implement the same time unit macro cleanup as jiff_0_2</li>
<li>Additional commits viewable in <a
href="https://github.com/jonasbb/serde_with/compare/v3.21.0...v3.22.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `roaring` from 0.11.4 to 0.11.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/RoaringBitmap/roaring-rs/releases">roaring's
releases</a>.</em></p>
<blockquote>
<h2>v0.11.5</h2>
<h2>What's Changed</h2>
<ul>
<li>Implement std Error for IntegerTooSmall by <a
href="https://github.com/Kerollmops"><code>@​Kerollmops</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/362">RoaringBitmap/roaring-rs#362</a></li>
<li>fix: invalid treemap iter advance by <a
href="https://github.com/silver-ymz"><code>@​silver-ymz</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/360">RoaringBitmap/roaring-rs#360</a></li>
<li>Fix off-by-one that corrupts a bitmap in
remove_smallest/remove_biggest (<a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/359">#359</a>)
by <a href="https://github.com/youdie006"><code>@​youdie006</code></a>
in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/363">RoaringBitmap/roaring-rs#363</a></li>
<li>Upgrade dependencies bump version by <a
href="https://github.com/Kerollmops"><code>@​Kerollmops</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/364">RoaringBitmap/roaring-rs#364</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/silver-ymz"><code>@​silver-ymz</code></a> made
their first contribution in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/360">RoaringBitmap/roaring-rs#360</a></li>
<li><a href="https://github.com/youdie006"><code>@​youdie006</code></a>
made their first contribution in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/363">RoaringBitmap/roaring-rs#363</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5">https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/0ce3fc8b55b193ce220253bfbc0c3e09bd171375"><code>0ce3fc8</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/364">#364</a>
from RoaringBitmap/upgrade-dependencies-bump-version</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/a961a042db8d325515e6b5273a2e9369fe5c931d"><code>a961a04</code></a>
Remove the once_cell dependency</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/5e8445b2d6914e8e56d85de340f9156825f4e91b"><code>5e8445b</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/363">#363</a>
from youdie006/fix/359-interval-remove-boundary</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/bf2961d99fb4da2c540a55eb699228a7bb00a732"><code>bf2961d</code></a>
Bump version to v0.11.5</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/048a8b05fcae08636354607f0c00d6e95f262107"><code>048a8b0</code></a>
Fix off-by-one that corrupts a bitmap in
remove_smallest/remove_biggest</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/27d84f567dd85243194d2b87683262ef43a5dd97"><code>27d84f5</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/360">#360</a>
from silver-ymz/fix/treemap-iter-advance-across-bitmaps</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/aac2de82a7f9e44fd73d8364de840169447d580b"><code>aac2de8</code></a>
Make clippy happy</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/a3d1d54be985fe22c01e882f85c3ee7055ad9c8b"><code>a3d1d54</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/362">#362</a>
from RoaringBitmap/std-error-for-integer-too-small</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/9a3c33e42c0b14bdd3f296313ee367526092aa81"><code>9a3c33e</code></a>
Implement std Error for IntegerTooSmall</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/f46c0ffe90b6d5d52a93106253bb6fa51a08c137"><code>f46c0ff</code></a>
fix: invalid treemap iter advance</li>
<li>See full diff in <a
href="https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi` from 3.11.0 to 3.12.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi's
releases</a>.</em></p>
<blockquote>
<h2>napi-v3.12.0</h2>
<h3>Added</h3>
<ul>
<li><em>(cli)</em> support non-threaded WASI targets (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3353">#3353</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/58bd87fa524a837a7c962ab4103e5588557ccd81"><code>58bd87f</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3414">#3414</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/9da87236dbc4fef99f066b7a130f4d0377308d44"><code>9da8723</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/8d22196aa98a1e6e70584561f5446d117d9c802c"><code>8d22196</code></a>
chore(deps): update dependency oxc-parser to ^0.142.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3422">#3422</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/abc30fbafc2e3967d499cef970c68b3edfefd850"><code>abc30fb</code></a>
build(deps): bump postcss from 8.5.17 to 8.5.23 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3421">#3421</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/55421392cbaa24d4df69419e4c6d4958fbcb6a12"><code>5542139</code></a>
build(deps): bump fast-xml-parser from 5.9.3 to 5.10.1 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3418">#3418</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/dc4ee8c89cc27ce30e239482199b3b3d786bf8b6"><code>dc4ee8c</code></a>
build(deps): bump fast-uri from 3.1.3 to 3.1.4 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3419">#3419</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/050d985196174b4be830cdb813d09e2705258455"><code>050d985</code></a>
feat(async-runtime): drain-linger surface + lock-free scheduler
internals (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3">#3</a>...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/e0b87086eefe0e7efeea6d269e9403c4be4ba9aa"><code>e0b8708</code></a>
chore(deps): update dependency oxc-parser to ^0.141.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3417">#3417</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/fc8494010697d078a93a528c3180271f6f187504"><code>fc84940</code></a>
chore(deps): update actions/setup-node action to v7 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3413">#3413</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ee598db45985ef11e18c7340801c28bb2452b688"><code>ee598db</code></a>
build(deps): bump protobufjs from 7.6.4 to 7.6.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3410">#3410</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-v3.11.0...napi-v3.12.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-derive` from 3.6.1 to 3.6.3
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-derive's
releases</a>.</em></p>
<blockquote>
<h2>napi-derive-v3.6.3</h2>
<h3>Other</h3>
<ul>
<li>updated the following local packages: napi-derive-backend</li>
</ul>
<h2>napi-derive-v3.6.2</h2>
<h3>Other</h3>
<ul>
<li>updated the following local packages: napi-derive-backend</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/956e4525fea6a676ea3680b711382f167b899af9"><code>956e452</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3448">#3448</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/73048f5a7fdbd42cdc2f46f2d5ac60ef27417bfa"><code>73048f5</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61fae8a1440ad8b7249f3cd7838fc2bafe00a906"><code>61fae8a</code></a>
fix(napi): stop unloading addons with live native code, preserve
non-Error re...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/93e86ce167095e84f2be2ae1c66a6c0bb96fec49"><code>93e86ce</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2c905991899b9f12a0df4072c4bff6d62ef70d26"><code>2c90599</code></a>
fix(cli): support npm 12 pack output (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3449">#3449</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/360b1ec99ab0d001147d29e11c416c8338d3d1c9"><code>360b1ec</code></a>
fix(wasi): avoid randomness during module registration (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3447">#3447</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b648c4090e7518ced18c9eca6059d27af3ab511b"><code>b648c40</code></a>
build(deps): bump nanoid from 3.3.16 to 3.3.18 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3446">#3446</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ffda4efff4bc4ebb6bef1f629dd0a6f09dc8f210"><code>ffda4ef</code></a>
chore(deps): update dependency js-yaml to v4.3.1 [security] (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3445">#3445</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/387b0dc7986018e44a4a0b466b030dc414170411"><code>387b0dc</code></a>
feat(cli): size WASI browser worker pools from
navigator.hardwareConcurrency ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61e4346ce3d9a9c13e5c5dd6fb3b7d5e1b1d6e0d"><code>61e4346</code></a>
build(deps): bump fast-uri from 3.1.4 to 3.1.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3440">#3440</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.1...napi-derive-v3.6.3">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-build` from 2.4.0 to 2.4.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-build's
releases</a>.</em></p>
<blockquote>
<h2>napi-build-v2.4.1</h2>
<h3>Fixed</h3>
<ul>
<li><em>(napi)</em> stop unloading addons with live native code,
preserve non-Error rejections, and add the wasm teardown barrier (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3423">#3423</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/956e4525fea6a676ea3680b711382f167b899af9"><code>956e452</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3448">#3448</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/73048f5a7fdbd42cdc2f46f2d5ac60ef27417bfa"><code>73048f5</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61fae8a1440ad8b7249f3cd7838fc2bafe00a906"><code>61fae8a</code></a>
fix(napi): stop unloading addons with live native code, preserve
non-Error re...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/93e86ce167095e84f2be2ae1c66a6c0bb96fec49"><code>93e86ce</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2c905991899b9f12a0df4072c4bff6d62ef70d26"><code>2c90599</code></a>
fix(cli): support npm 12 pack output (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3449">#3449</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/360b1ec99ab0d001147d29e11c416c8338d3d1c9"><code>360b1ec</code></a>
fix(wasi): avoid randomness during module registration (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3447">#3447</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b648c4090e7518ced18c9eca6059d27af3ab511b"><code>b648c40</code></a>
build(deps): bump nanoid from 3.3.16 to 3.3.18 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3446">#3446</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ffda4efff4bc4ebb6bef1f629dd0a6f09dc8f210"><code>ffda4ef</code></a>
chore(deps): update dependency js-yaml to v4.3.1 [security] (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3445">#3445</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/387b0dc7986018e44a4a0b466b030dc414170411"><code>387b0dc</code></a>
feat(cli): size WASI browser worker pools from
navigator.hardwareConcurrency ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61e4346ce3d9a9c13e5c5dd6fb3b7d5e1b1d6e0d"><code>61e4346</code></a>
build(deps): bump fast-uri from 3.1.4 to 3.1.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3440">#3440</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.0...napi-build-v2.4.1">compare
view</a></li>
</ul>
</details>
<br />

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
 2026-08-31 10:40:11 -07:00
lancedb-gatefixer[bot]andXuanwo 9b825c5f29 fix(node): route auto search using table embeddings (#3832) 
## Summary

- Resolve automatic string-search routing from the active table schema
whenever the query executes.
- Defer embedding-provider construction while leaving explicit vector
and FTS routes unchanged.
- Cover unrelated global registrations and metadata transitions across
repeated executions of one query builder.

## Root cause

LocalTable.search used the number of globally registered embedding
providers to choose between vector and full-text search. A provider
registered for any other table therefore sent a plain FTS table down the
vector path. A wrapper-lifetime metadata snapshot avoided that
contamination but became stale after time travel or read-consistency
refreshes. The query now records fluent builder operations and creates
the appropriate native vector or FTS query from the active schema on
each execution.

## Validation

- pnpm build
- pnpm tsc
- pnpm lint
- pnpm run docs
- pnpm test --runInBand (681 passed, 5 skipped)

Fixes #1557

<!-- lance-gatekeeper-fix:v1 agent=b6183df8296db4aabdc5d19a2256b029
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-08-26 10:08:48 +08:00
lancedb-gatefixer[bot]andXuanwo 35b5d015ac fix(node): preserve embedding registration in server bundles (#3806) 
## Summary

- lazily initialize built-in OpenAI and Hugging Face providers when
consumers call the public embedding registry API
- choose automatic vector versus FTS search from embedding metadata on a
fresh pinned table revision for every execution
- expose automatic string searches as an `AutoQuery` with only
operations common to both native query families
- keep the registry shared and built-in registration safe across
duplicated module graphs

## Root cause

Nitro treats dependency modules as side-effect-free and removes the bare
OpenAI provider import from its generated route. Registration therefore
never runs, so `getRegistry().get("openai")` remains undefined even when
the registry itself is shared globally. Bundlers may also duplicate the
provider and registry module graphs.

The public embedding entry point now initializes built-in providers only
when `getRegistry()` is explicitly called, keeping initialization on a
live path that Nitro retains. Each terminal automatic-search execution
pins the exact table revision visible at dispatch, reads embedding
metadata and computes an embedding from that snapshot, replays the
builder operations, and constructs and executes the selected native
query against the same snapshot. Pinned native snapshots execute locally
when namespace pushdown cannot carry their revision, while remote
snapshots are seeded directly from one version-and-schema response. The
public `AutoQuery` builder exposes only the operations shared by FTS and
vector search, so runtime class narrowing cannot expose invalid
vector-only methods. Repeated built-in registration replaces stale
constructors from duplicated module graphs while public `register()`
retains its duplicate-alias error.

## Validation

- `cargo fmt --all`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test --runInBand` (783 passed, 5 skipped)
- serial examples suite with a local OpenAI mock (11 passed), including
`sentence-transformers.test.ts`
- packaged Nitro 2.13.4 server route using the reported imports returned
`{"registered":true}`
- fresh-process FTS fixture initialized both public built-ins and
confirmed automatic string search still returned the indexed row
- schema-consistency regressions cover read-consistency refresh,
checkout, checkoutLatest, restore, runtime class narrowing, concurrent
overwrite during embedding computation, and reused automatic-search
builders
- focused regressions confirm pinned native snapshots bypass unversioned
namespace pushdown and remote snapshots use one describe request

Fixes #2429

<!-- lance-gatekeeper-fix:v1 agent=2adf0f21b8bfb634606ed8897a849e30
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-08-26 04:17:32 +08:00
Will JonesandClaude Opus 5 fce45ba9fc feat(nodejs): add listTables, deprecate tableNames (#4041) 
`table_names` is being replaced by `list_tables` across the SDKs, but
TypeScript only had `tableNames`. This PR adds `listTables`, which
returns a page of table names together with the token that resumes after
it, and marks `tableNames` and `TableNamesOptions` deprecated in favor
of it.

It binds the `Connection::list_tables` that already exists, so nothing
in the Rust API changes and nothing existing breaks. `pageToken` is
documented as opaque rather than as a table name, since what resumes a
listing is the database's to decide — that keeps callers off a detail
that is going to change.

Stacked on #4040, which fixes a table being dropped at every page
boundary. The page-walking test here needs that fix to pass. Review the
last commit only until #4040 lands.

## Example

```ts
const names = [];
let pageToken = undefined;
do {
  const page = await conn.listTables({ pageToken, limit: 100 });
  names.push(...page.tables);
  pageToken = page.pageToken;
} while (pageToken);
```

A namespace can be listed by passing its path first, mirroring
`tableNames`:

```ts
const page = await conn.listTables(["analytics"], { limit: 100 });
```

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-24 16:25:14 -07:00
Xuanwo b0dae5eb0b feat: return typed refresh job results (#4013) 
## Problem

`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.

## Behavior

`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.

The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.

## Ownership and integration boundary

LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
 2026-08-25 00:01:32 +08:00
Wyatt Alt 68749ecfa3 feat(nodejs): materialized view bindings (#3935) 
Exposes materialized views to TypeScript: createMaterializedView,
openMaterializedView and listMaterializedViews on Connection, and a
MaterializedView handle carrying the parsed definition and
refresh({full, sourceVersion}), which returns the typed refresh result.
select accepts column names, [alias, expression] pairs, or a record of
the
same; the definition reads back off the stored schema, so a reopened
handle
needs no side channel. Remote connections surface the core's
not-supported
error up front.

The napi crate needed the same recursion-limit raise as the core crate:
the
refresh future's type graph overflows the default trait-recursion depth.


<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-21 23:48:43 -07:00
Drew Gallardo e98d8ac685 feat!: rename branch merge to cherry_pick (#3986) 
This PR is a **breaking** rename of #3686.

merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.

This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.

Cherry-pick is terminology that aligns more with that.

```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")

exp.add_columns({"tag": "cast('draft' as string)"})

diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")

if result["status"] == "cherryPicked":
    print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
    print(result["diff"]["errors"])
```

### Behavior

- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.

### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
 2026-08-21 23:37:12 -07:00
Dan RammerandClaude Opus 5 f1c4967eeb feat: bring the MemWAL LSM surface to parity across the SDKs (#3962) 
## Why

Four of the eight LSM methods are **remote-only in the core**. `impl
BaseTable for NativeTable` implements only
`set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`,
`compact_lsm` and `get_lsm_stats` fall through to trait defaults
returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and
`checkpoint_lsm` is built on all three.

That explains the state of the bindings: Node had bound the four that
work against a local table and stopped, so a Cloud user could install an
LSM write spec but had no way to observe fresh-tier state or drive a
checkpoint. Java had none of it at all.

| SDK | set/unset/get spec | closeWriters | flush | compact | getStats |
checkpoint |
|---|---|---|---|---|---|---|
| Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Node *(before)* | ✅ | ✅ | — | — | — | — |
| **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** |
| Java *(before)* | — | — | — | — | — | — |
| **Java (after)** | **new** | n/a | **new** | **new** | **new** |
**new** |

Go and C are separate repos and are out of scope here. `closeLsmWriters`
drains cached in-process shard writers, so it has no meaning for Java,
which is a pure REST client.

## Node

Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and
`getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats`
/ `MemtableStats` objects — typed rather than a JSON blob, matching the
existing `LsmWriteSpec` object in the same file, with `u64` cast to
`i64` per that file's convention.

Because these four are remote-only, the new tests assert each binding
reaches the core and surfaces `NotSupported` against a local table. That
covers the wiring; behavior against a real endpoint stays covered by the
mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`.

## Python

No new methods. All eight are on `LanceTable`, `AsyncTable` and
`RemoteTable` — the last four landed on the sync `RemoteTable` in #3961,
which is merged into this branch.

What was missing here was reachability. `LsmWriteSpec` was importable
only from the private `lancedb._lancedb`, appearing in `table.py` solely
under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no
mention of it, which per the repo's docs guidance means it rendered
nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in
`__all__`, and documented.

## Java

Java reaches LanceDB purely over REST through the generated Lance
Namespace client, and these routes are not in that spec, so they are
issued through a small dedicated client rather than added to the spec.
That call is revisitable — LSM is one of four unspecified route families
alongside `multipart_write`, `page_cache/prewarm` and
`branches/diff|merge`. If those are ever regularized into the spec as a
group, `LanceDbTableLsm` is one file that gets deleted.

`LsmWriteSpec` here is deliberately **not**
`org.lance.memwal.InitializeMemWalParams`. That type defaults to
maintaining *no* indexes where a spec here defaults to maintaining
*every* index, and it cannot express the `null` that asks the server to
resolve the set:

| Value | On the wire | Meaning |
|---|---|---|
| unset (null) | `null` | Server resolves **every** maintainable index |
| `Collections.emptyList()` | `[]` | Maintain **none** |
| `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those |

A dedicated test pins null and `[]` as distinct on the wire, since
collapsing them is the failure mode that motivated a LanceDB-owned type.

`checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs`
with its constants and status semantics intact: 429/503 retried in place
against an 8-budget, 421 restarting from flush against a 3-budget, 5s
poll, and a target watermark fixed after the seal so it terminates under
write load.

`getLsmStats` returns typed `LsmStats` / `BucketStats` /
`GenerationStats` / `MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes.
Decoding is strict — see below.

## Review feedback

Both gatekeeper findings were real. Each was reproduced against the
scripted test server first, and each fix ships with the reproducer as a
regression test.

**The transport was doubling every checkpoint retry budget.**
`HttpClients.createDefault()` installs Apache's default response retry
strategy, whose retryable-status list is exactly 429 and 503 — the two
statuses `isRetryable` owns. A 429 held against `flush_lsm` issued
**18** wire requests where the loop intends 9, and `compact_lsm` was
retried in place despite the loop being built to fall through to a fresh
stats poll instead. Timing confirmed the mechanism: that run took 25.4s
≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry
interval.

Automatic retries are now disabled, so the checkpoint loop is the sole
owner of the 421/429/503 transitions. A side effect worth noting:
`testCheckpointRetriesRetryableStatusInPlace` was passing on a
transport-absorbed 429 and never reaching `issue()`'s retry branch at
all. It now exercises the real path.

**Stats decoding failed open.** `getLsmStats` read the response with
Jackson's `path()`, which yields a missing node that iterates as an
empty array — making "malformed" indistinguishable from "no buckets",
which is indistinguishable from "drained". Four separate payloads made
`checkpointLsm()` report convergence for a checkpoint that never ran:

| Response | Before | Now |
|---|---|---|
| `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ |
| `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` |
| empty response body | **reported success** | `IllegalStateException` |
| bucket missing required fields | **reported success** |
`IllegalStateException` |

The empty-body row is the one to weight: a proxy 200 with no body is a
realistic production event, and it silently reported a checkpoint that
never happened.

Decoding is now strict and fails closed, matching the serde contract on
the Rust side exactly. One deliberate deviation from the review comment,
which asked that *only* explicit JSON `null` count as disabled: Rust has
`#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to
`None` there too. Java now matches that. It is an absent-or-malformed
**`buckets`** that fails closed, which is the case the comment was
actually protecting.

## Testing

- Java: **33 passing** (8 existing + 25 LSM) against a scripted
`com.sun.net.httpserver.HttpServer` — no new test dependency. Wire
assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`;
checkpoint tests cover convergence, not piling onto a latched bucket,
421 restart-from-flush, 429 retry-in-place, terminal-status propagation,
reissue exhaustion, the exact wire-request count against the retry
budget, and five malformed stats payloads.
- Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm
run tsc`, `npm run lint`, `npm run docs` all clean.
- Python: `ruff format --check` and `ruff check` clean.
- Java formatting: `./mvnw -pl lancedb-core spotless:apply` and
`spotless:check` both clean under a JDK 11 toolchain.

## Note: spotless needs a pre-16 JDK

`./mvnw spotless:apply` fails on JDK 16+ with
`JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7,
pinned at `java/pom.xml:34`, predates JDK 16's compiler API change.
**This is pre-existing** and reproduces on a pristine `main` checkout.

It is not a blocker, just a toolchain requirement. Spotless was run
against these sources under JDK 11 and both `spotless:apply` and
`spotless:check` pass on the whole module:

```shell
JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply
```

Bumping the plugin so it works on modern JDKs is still worth doing, but
separately from this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-19 11:44:46 -05:00
Wyatt Alt c429863122 feat: refresh_column_async returns a job handle (#3939) 
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-14 16:05:04 -07:00
Wyatt Alt fc0d917d32 feat: refresh computed columns (#3938) 
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-14 14:43:41 -07:00
Wyatt Alt def869bb78 feat: declare computed columns by SQL expression (#3937) 
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.

The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-14 14:17:41 -07:00
Jack Ye ffd35c1a8f feat: add asynchronous drop table API (#3936) 
## Summary

- add `drop_table_async` and return a job handle while preserving
`drop_table`
- consume remote 202 responses with cleanup job IDs and retain
older-server compatibility
- expose the API through Python and TypeScript connection wrappers
 2026-08-13 18:05:44 -07:00
Dan TasseandWill Jones 77a93fee76 fix: get table size from metadata, not files (#3790) 
Some issues:
- file_size_bytes is optional in the manifest, so if it's not there (old
writer I guess) it'll under-report the table size.
- it changes results a little bit from the old way by including per-file
footers and metadata (probably not a big difference at real scale)

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
 2026-08-07 17:41:41 -04:00
Dan RammerandClaude Opus 5 706a9c327f feat: infer maintained indexes when an LsmWriteSpec omits them (#3748) 
## What

`LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`:

| value | meaning |
|---|---|
| `None` (new default) | every index the MemWAL supports, resolved when
the spec is installed |
| `Some([])` | maintain nothing — a scan/filter-only WAL table |
| `Some([..])` | exactly these, taken verbatim |

`with_maintained_indexes` keeps its signature;
`with_no_maintained_indexes()` is new. Surfaced through the remote path
(null on the wire), Python, and Node.

## Why

Callers had to state the maintained set by hand every time, which is
both tedious and easy to get wrong — the common case is "maintain what I
already built."

Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating
to lance's `is_maintainable_index_type`. This is load-bearing rather
than cosmetic: lance does **not** skip an index type its memtable cannot
build, it errors when the shard writer opens, so sweeping up a bitmap
index would fail every memtable claim and leave the table unwritable.
The inferred set excludes those, and an explicit list naming one is now
rejected at spec time instead of at claim time.

## Behavior change

A freshly constructed spec used to maintain **nothing**; it now
maintains **everything supported**. This flipped because napi collapses
`undefined` and `null` to `None`, so TypeScript cannot express "absent
means nothing, null means all" — any other choice makes the bindings
disagree with the wire. The error direction also favors it: an unwanted
maintained index costs memory, while a silently unmaintained one
degrades FTS to an unscored scan.

Three existing tests encoded the old default and are updated rather than
worked around.

## Caveat

The resolved set is a snapshot, not a subscription. An index created
after the spec is installed is not maintained until the spec is unset
and set again. `get_lsm_write_spec` therefore always reports a concrete
list — `None` never round-trips.

## Dependency

Needs a lance release carrying `is_maintainable_index_type`
(lance-format/lance#8095) before this builds against the pinned tag.
Draft until then.

## Testing

38 Rust LSM tests and 10 Python tests pass against a local lance build,
including new coverage that a bitmap index is excluded from inference
and rejected when named, and that `[]` stays distinguishable from null
on the wire.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-07 14:50:22 -05:00
Wyatt Alt 8e24dd3828 feat(rust)!: make add_columns a builder (#3778) 
Table::add_columns now takes no arguments and returns AddColumnsBuilder,
so calls become .add_columns().transform(t).execute().

read_columns was the second positional argument but reaches only one of
the five transform variants. In lance's add_columns_to_fragments only
BatchUDF receives the caller's value: SqlExpressions replaces it with
the columns its expressions reference, Stream and Reader pass None, and
AllNulls reads nothing. So it was mandatory on every call -- all
eighteen call sites here passed None -- and silently discarded four
times out of five. As a builder method it is optional, and setting it
where lance would discard it is now an error, which does reject a call
that previously succeeded while ignoring the argument.

Matches the builders add, update, and merge_insert already use.
 2026-08-04 11:18:22 -07:00
Wyatt AltandClaude Fable 5 e3b472c212 feat: connection-level job operations (#3755) 
Adds job operations to the connection surface, building on the Job
handle from #3742: job(id), list_jobs, get_job, cancel_job, and
job_history, plus a non-blocking Job.status(). Implemented on the
Database trait (defaulting to NotSupported), the remote backend
(/v1/jobs), and the Python and Node bindings; job_history returns Arrow
batches.

errors() and progress() are not included.

Tested with mocked endpoints in all three languages.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-31 12:51:43 -07:00
Wyatt AltandClaude Fable 5 a6418b6cb9 feat: create_index returns a Job handle (#3742) 
IndexBuilder::execute now returns a Job with wait and cancel methods.
Local tables build the index synchronously and return an already-done
job. Remote tables read the job id the server returns from create_index
and track it through the /v1/jobs API: wait polls describe until the job
reaches a terminal state and cancel posts a cancellation. Servers that
return no job id yield a done job, so behavior against older servers is
unchanged. The job id is not exposed on the handle.

The Python and TypeScript bindings keep their current signatures and
discard the handle; exposing Job there is left to follow-ups.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-31 07:32:28 -07:00
Yang CenandYang Cen f7feed48c3 feat(fts): support custom stop-word lists (#3734) 
## What

Expose custom FTS stop-word lists in the Python and TypeScript public
APIs, including their standalone tokenize helpers and remote index
creation.

This PR supports concrete string lists only. It does not add file or
LanceDB-table stop-word sources.

## Why

Rust already exposes Lance's custom stop-word list option. The Python
and TypeScript APIs did not pass it through, and local index details did
not retain the full tokenizer parameters needed by index-backed
tokenization after reopening a table.

## How

- Add `custom_stop_words` / `customStopWords` to the Python and
TypeScript FTS and tokenize options.
- Preserve `None` / `undefined`, empty lists, and list contents without
normalization.
- Load the persisted FTS segment parameters when returning local index
details.
- Serialize the concrete list in remote create-index requests.
- Keep Python and TypeScript tests thin; behavior, persistence, query
tokenization, and remote JSON coverage live primarily in Rust.

## Validation

- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
- Python extension rebuild with `uv` and `maturin`
- Targeted Python tests: 4 passed
- Python `ruff format --check` and `ruff check`
- TypeScript build, typecheck, Biome lint, generated docs, and targeted
tests

---------

Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
 2026-07-29 17:40:12 +08:00
Heng Ge f655f62e09 feat(query): add use_lsm to read MemWAL LSM data (#3489) 
## What

MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).

## How

- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
    - **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.

Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
 2026-07-25 23:45:27 -07:00
Yang CenandYang Cen 9dc5ec03aa feat(fts): add block size configuration (#3691) 
## What changed

- add `block_size` to Python FTS configuration and the deprecated
local/remote helpers
- add `blockSize` to the TypeScript FTS options and propagate it through
the NAPI binding
- serialize the value as `block_size` for remote index creation
- document the existing Rust builder API and generate the TypeScript API
reference
- add local, remote, metadata, search, and invalid-value regression
coverage

## Why

Lance supports configuring the number of documents per compressed FTS
posting block, but LanceDB's Python and TypeScript APIs did not expose
the setting. This made the experimental FTS V3 layout unavailable
through those clients and allowed the value to be dropped before index
creation.

## How it works

The default remains `128`. Supported values are `128` and `256`;
selecting `256` uses the experimental FTS V3 format. Invalid values are
rejected by the Lance builder and surfaced as Python or JavaScript
errors.

## Validation

- `cargo check --quiet --features remote --tests --examples`
- `cargo +1.94.0 clippy --quiet --features remote --tests --examples --
-D warnings`
- targeted Rust local and remote index tests
- Rust doctests: 34 passed
- Python Ruff checks, doctest, and targeted local/remote tests: 5 passed
- TypeScript build, Biome lint, generated docs, and targeted Jest tests:
9 passed
- `git diff --check`

## Limitations

The Java client remains unchanged because its external remote REST model
does not currently expose `block_size`.

Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
 2026-07-24 15:02:38 -07:00
Drew GallardoandCursor 65cd142c7e feat: add remote branch diff and merge client APIs (#3686) 
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.


This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`

Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.

Merge here means promoting a branch's added columns onto `main`.

### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.

### Example
```python

table = db.open_table("images")

table.branches.create("exp")
exp = table.branches.checkout("exp")

exp.add_columns({"tag": "cast('draft' as string)"})

diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)

if result["status"] == "merged":
    print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
    print(result["diff"]["mergeBlockers"])
```

### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-18 12:38:05 -07:00
Will JonesandClaude Opus 4.8 5d0a1ef66c fix(rust): bound remote insert request size to avoid ingestion timeouts (#3630) 
## Problem

On the remote (LanceDB Cloud) write path, each write partition is
uploaded as a **single** `/insert?upload_id=...` request that stays open
until the whole partition has been streamed and the server has written
it to object storage. For large bulk ingests a partition can be many GB,
so a single request can run longer than the client read timeout (default
300s), surfacing as:

```
lancedb.remote.errors.HttpError: operation timed out
```

The server already supports staging **multiple** parts under one
`upload_id` (each `/insert` writes a separate transaction that
`complete` merges atomically), but the client never used that — it sent
one part per partition.

## Change

Split each partition into multiple parts of at most
`max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded
as its own `/insert?upload_id=...&upload_part_id=...` request. This
bounds how long any single request stays open, independent of total data
size or write parallelism.

Key properties:
- **Still streamed, not buffered.** Each part's body is driven through a
bounded channel while the request is in flight (`futures::join!` of a
producer + the send), so peak memory stays at a couple of batches per
partition regardless of the part size. Backpressure from a
slow/throttled server still propagates upstream.
- **Correct part accounting.** An empty partition still sends exactly
one (schema-only) part so `complete` has a transaction to commit; a size
cut landing exactly on the end of input does not emit a trailing empty
part.
- **Multipart only.** The single-request (non-multipart) path is
unchanged.

## Config

New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable
via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable.
**Default 1 GiB** (`Some(0)` disables splitting → one request per
partition). Python users pick up the default/env automatically through
the remote client.

## Tests

- `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each
batch in its own part → N requests, each carrying the shared `upload_id`
and a distinct `upload_part_id`.
- `test_multipart_single_part_when_under_budget`: a large budget keeps
the partition in a single request.
- Verified end-to-end against a live remote table: a forced-chunked
multipart add (many parts) assembles to the correct row count.

Related to ENT-1883.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-17 14:37:41 -07:00