rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-24 20:52:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,070 Commits 362 Branches 563 Tags
003b87bfa089997d7bed95123e010ddbcc1c376e
Commit Graph
55 Commits
Author SHA1 Message Date
Dan TasseVincent KocClaude Fable 5.1
 		003b87bfa0 fix(node): support Apache Arrow 19 through 21 and run the library on each 
Builds on #4359 (which added the Arrow 19-21 test aliases) and makes the
package genuinely work across the whole peer range:

* Resolve the library's own `apache-arrow` to 21.2.0 (devDependency) so
  the main CI jobs build, type-check and test on the newest release, and
  widen the peer range to `>=15.0.0 <22.0.0` instead of an exact pin.
* Add a CI job that re-installs with `apache-arrow` resolved to 19.0.1 and
  20.0.0 and runs the suite, covering what a consumer on those versions
  gets.
* Arrow 21 overrides `Symbol.hasInstance` on Schema/Field/Table/RecordBatch/
  Data so `instanceof` matches objects from another copy of the library,
  but not on DataType. The sanitizer's short-circuits therefore returned
  foreign schemas unchanged and Arrow's builders then failed with
  "Unrecognized type 'NONE'". Check the real prototype chain instead.
* Sanitize the types added in Arrow 20/21 (LargeList, Utf8View, BinaryView,
  IntervalMonthDayNano), looked up dynamically so the library still compiles
  and runs on older Arrow releases, with a clear error when the installed
  Arrow lacks the type. Carry `variadicBuffers` through `Data`.
* Loosen `FieldLike` so a branded DataType from another Arrow copy still
  type-checks; export the new `DataTypeLike`.
* Share the Arrow version list between test files, exercise embedding
  function types per version in the registry tests, and add sanitize and
  end-to-end coverage for the new column types.
* Enable `skipLibCheck`: flatbuffers 25 (via Arrow 21) ships declarations
  that need TypeScript 5.7+.
* Record the new dev dependencies in the third-party license inventory.

Co-Authored-By: Vincent Koc <vincentkoc@ieee.org>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 11:21:59 -05:00
Bilal Atique 0a1c1a04c7 feat(node): accept an AbortSignal to cancel blob reads (#4350) 
Closes #4288.

Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange`
and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an
optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When
the signal aborts, the call rejects with `signal.reason` (an
`AbortError` unless the caller gave a reason) and the native future is
dropped, so the read stops instead of running to completion. A signal
that is already aborted rejects before any read starts.

How it works:

- On the Rust side, these five methods are now sync `#[napi]` methods
that return `env.spawn_future(...)`. The future is wrapped in
`futures::future::abortable`, and the abort handle is registered with
napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so
it can't be an argument of an `async fn`, which is why the methods
changed shape. The TypeScript signatures are unchanged apart from the
new optional argument.
- napi-rs installs its own `onabort` on the signal it receives, which
would overwrite a handler the caller set. So the TypeScript wrapper
passes a private `AbortController` signal to the native call for each
request, forwards the caller's abort to it with `addEventListener(..., {
once: true })`, and removes the listener when the call settles.

Tests in `table.test.ts`:

- an already aborted signal rejects on all five methods, and the handle
still reads afterwards
- a custom abort reason is what the call rejects with
- a read with a signal that never aborts returns normally and leaves no
`abort` listener on the signal
- aborting a 32 MiB `readRange` right after it starts rejects with
`AbortError`, and the same handle reads again afterwards

The Node docs under `docs/src/js` are regenerated.

I didn't add a test for where the `read()` cursor ends up after a
mid-flight abort on a local table. The remote `BlobFile` documents that
a cancelled read leaves the cursor unchanged, but I haven't confirmed
the same for the native Lance handle, so the docs don't claim it.
 2026-09-29 14:01:17 -05:00
Bilal Atique 4e3ec7cc27 feat(node): expose seek, tell, bounded read, readRanges and close on BlobFile (#4348) 
Closes #4300.

The TypeScript `BlobFile` only had `size()`, `read()` and `readRange()`,
while the Rust `BlobFile` behind it also has `seek`, `tell`,
`read_up_to`, `read_ranges` and `close`. This exposes those on the Node
binding:

- `read(maxBytes?)` reads at most `maxBytes` from the cursor (Rust
`read_up_to`). Without the argument it reads to the end, as before.
- `readRanges([{ start, end }, ...])` reads several half-open ranges and
returns one buffer per range, in order. Like `readRange`, it does not
move the cursor.
- `seek(position)` and `tell()` move and report the cursor.
- `close()` releases the handle, and `isClosed()` reports it. Every
read, `seek` and `tell` after `close()` fails with "blob file is already
closed". A second `close()` does nothing. The binding keeps its own
`closed` flag, the same way the Python `BlobFile` does.

Offsets and lengths are `bigint`, like `readRange` and `size()`.
`BlobRange` is exported as a type.

The new tests in `table.test.ts` cover chunked reads with
`read(maxBytes)`, `seek` and `tell`, a negative seek position,
`readRanges` order and cursor, a `readRanges` end past the blob size,
and reads after `close()`. The Node docs under `docs/src/js` are
regenerated with `pnpm run docs`.

I left the `ReadableStream`/async iterator idea from the issue out of
this PR. The AbortSignal work in #4288 touches the same methods, so it
can follow on top of this.
 2026-09-29 11:32:30 -05:00
Jack Ye 8541d6d9df feat: add view CRUD APIs (#4236) 
A view is a named query a database stores and plans on every read. It
holds no rows, which is the whole difference from a materialized view.

## API

| Verb | Route |
| --- | --- |
| `create_view(name, query, namespace_path)` | `POST
/v1/view/{id}/create` |
| `describe_view(name, namespace_path)` | `POST /v1/view/{id}/describe`
|
| `drop_view(name, namespace_path)` | `POST /v1/view/{id}/drop` |
| `list_views(namespace_path)` | `GET /v1/namespace/{id}/view/list` |

On `Connection` and the `Database` trait, with the remote client, Python
(sync and async) and Node bindings. Local databases return
`NotSupported`: the server side is Sophon's, where a view is an object
of the database manifest.

`ViewDescription` carries the defining query, the database *and
namespace path* unqualified names in it resolve against, and the schema
the query resolved to. `create_view` returns one, so a caller has the
schema without a second call.

Both defaults travel with the view because it outlives the session that
declared it: the server re-plans the stored query on every read, so a
reader resolving an unqualified name against its own defaults would read
a different table. `default_namespace_path` crosses the wire as
`default_namespace`, a path like `namespace`, absent for the root.

There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create, each authorized against what it
actually touches.

Querying a view stays SQL's job. There are no rows behind a view, so
there is no `open_view` returning a `Table`.
 2026-09-23 15:09:40 +08:00
Wyatt Alt 5231d37f8d feat: list a table's per-row Function errors from the client (#4208) 
A refresh running under a skip policy records each row it skipped, with
the failing input and the error, but the client could not read that
store: the server exposes it over SQL and, since recently, a REST route.
A user who hit per-row failures still had to open a SQL session.

`Table::function_errors` calls the route. The listing is table-addressed
with optional job and column filters, the same addressing the SQL
surface uses, so the two cannot disagree about what a table's errors
are. The two non-record signals come back as their own fields rather
than as rows: capped-fragment summaries, and whether the listing stopped
at its limit. Local tables refuse rather than answer with an empty list.

Python and Node expose the same call, with the same optional filters.
 2026-09-17 05:12:50 -07:00
Jack YeandXuanwo 6a07f88980 feat: add remote catalogs and Python and TypeScript bindings (#4195) 
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.

## Rust API

`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.

```rust
use lancedb::catalog::{
    CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};

let catalog = lancedb::connect_catalog("https://my-server.example")
    .api_key("my-api-key")
    .execute()
    .await?;

let db = catalog.create_database(
    CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
    ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
    DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```

Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.

## Python API

```python
import lancedb

catalog = lancedb.connect_catalog(
    "https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
    next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```

`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.

## TypeScript API

```typescript
import { connectCatalog } from "@lancedb/lancedb";

const catalog = await connectCatalog("https://my-server.example", {
  apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
  const nextPage = await catalog.listDatabases({
    limit: 20, pageToken: page.pageToken,
  });
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```

Create/connect return the existing `Connection` API. All four methods
are asynchronous.

## REST mapping

All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.

| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |

For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.

Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.

For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.

Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.

Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.

## Scope

This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.

The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-17 01:51:10 +08:00
Jack YeandXuanwo f3ef21b8ca feat: use oauth2 crate for OAuth with configurable client auth (#4181) 
Stacked on #4173 (`jack/restore-oidc-flows`, base branch mirrored to
this repo so the diff shows only this change); context from review:
https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100.
Rebase to `main` once #4173 and #4179 merge.

## What moved to the `oauth2` crate (5.0, no default features)

- Authorization URL generation and CSRF state (`authorize_url`,
`CsrfToken`)
- PKCE S256 challenge/verifier generation and code exchange
- Client-credentials, authorization-code, refresh-token, and device-code
grant request construction
- Device authorization request and the device token polling loop
(`authorization_pending`, `slow_down` +5s, expiry deadline, denial,
network backoff capped at 10s)
- Standard success/error response parsing (`RequestTokenError`)
- Token-endpoint client authentication and standards-compliant parameter
encoding (RFC 6749 2.3.1 Basic encoding)

LanceDB keeps ownership of OIDC discovery (compared `openidconnect`: no
measurable win for our 3-field metadata + strict validation, at real
dependency cost), HTTPS-or-loopback endpoint enforcement, the loopback
callback server, browser/stderr prompts, token caching and refresh
orchestration, and the dedicated hardened Azure IMDS source, which is
unchanged.

## Client authentication methods

New `ClientAuthMethod` enum (`none` | `client_secret_basic` |
`client_secret_post`), exposed in Rust, Python, and Node. Unset resolves
to `client_secret_basic` when a secret is present (RFC 6749 2.3.1
recommendation and the normal Okta confidential-app default, so a
default Okta app works without weakening its configuration) and to
`none` for public clients (PKCE/device). Explicit `none` with a secret,
or basic/post without one, is rejected. The method applies to client
credentials, code exchange, refresh, and device requests. Deliberate
behavior change: confidential clients previously always sent the secret
in the POST body; they now default to Basic (Keycloak accepts both).

No `audience`/`resource` parameters were added: the supported target is
an Okta custom authorization server with the API audience configured
server-side, so client-provided audience parameters are unnecessary;
`add_extra_param` support exists if a concrete provider contract ever
needs them.

## Device polling behavior changes (deliberate, tested)

- The first token poll now happens immediately rather than after one
interval (RFC 8628 allows both).
- Transient failures (HTTP 429, 5xx, `temporarily_unavailable`, network
errors) now retry with exponential backoff capped at 10s instead of
retrying at the fixed interval; polling never spins faster than once per
second even if a server reports a zero interval.

## Security and compatibility

- Issuer and discovered endpoints (and device verification URIs) still
require HTTPS except explicit loopback HTTP, enforced before any crate
URL type is built
- Token HTTP client keeps the hardened redirect policy that refuses
insecure redirect targets; regression test added
- Errors never embed raw response bodies (avoids leaking tokens through
parse failures); all credential types stay redacted in Debug
- Transient conditions (429/5xx/`temporarily_unavailable`) remain
retryable in device polling and hard errors elsewhere; refresh keeps
rotation and reauthentication semantics
- Existing public APIs stay source-compatible except the added
`OAuthConfig.client_auth_method` field

## Tests

Rust: client-auth methods across code
exchange/refresh/client-credentials/device (none/basic/post),
auth-method resolution and validation, transient device retries,
denial/expiry, redirect rejection, malformed-response leak check, PKCE
URL assertions, redaction. Python and Node: enum values, conversion,
unknown-method errors, config defaults.

Manual Okta validation recipe (no automated Okta credentials): create a
custom authorization server with an API audience, one confidential web
app (Basic) for authorization-code, one native app (PKCE, no secret),
one native device app; point `issuer_url` at the custom server, set
`client_auth_method` only for the POST-required case; verify token
acquisition, refresh after expiry, and `x-lancedb-credential-type: oidc`
against a LanceDB deployment. Never commit tenant URLs or secrets.

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-16 20:11:08 +08:00
mikemikimike be3215a6ae feat(nodejs): add JSON field helper (#4082) 
## Issue

Fixes #4063

## Background

The Node.js SDK currently requires callers to know the Arrow extension
metadata needed to represent JSON fields. This makes a common LanceDB
schema type unnecessarily verbose and easy to get wrong.

## Changes

- Add `makeJsonField(name, nullable = true)` to create a UTF-8 Arrow
field with the `arrow.json` extension metadata.
- Re-export the helper from the public Node.js entry point.
- Add coverage for the default nullable behavior, explicit non-nullable
fields, and the extension metadata.
- Add the generated TypeDoc function page and public globals entry,
including a usage example.

## Implementation

The helper uses the existing Apache Arrow `Field` type and sets
`ARROW:extension:name` to `arrow.json`, matching the metadata convention
already used by LanceDB.

## Compatibility

This is an additive Node.js API. Existing schema construction and Arrow
behavior are unchanged.

## Verification

- `pnpm test -- arrow.test.ts --runInBand` — 236 tests passed.
- `pnpm exec biome ci lancedb/arrow.ts lancedb/index.ts
__test__/arrow.test.ts` — passed.
- `git diff --check` — passed.

## Not run / known limitations

- `pnpm build` and `pnpm run docs` were attempted after expanding the
checkout. Both are blocked locally by the native binding build/type
declarations: Cargo did not complete, and TypeDoc reported the missing
generated `nodejs/lancedb/native` module. The docs files were generated
from the updated TypeScript comments; full build and docs validation are
left to CI.
 2026-09-15 18:01:42 -07:00
Jack YeandXuanwo 2f88b71c21 feat: add persistent OAuth token cache and session APIs (#4182) 
Stacked on #4173 (diff includes it until that merges; will rebase
after). Addresses the token-cache part of [Colin's
review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100).

Adds an explicit, opt-in persistent OAuth token cache shared by Rust,
Python, and Node clients, plus `login` / `status` / `logout` session
APIs, so short-lived processes (CLIs, scripts, notebooks) reuse one
session instead of restarting a browser or device flow on every start.

- **Opt-in and minimal**: existing callers stay memory-only and lazy.
Only refresh tokens are persisted (never access tokens, never client
secrets), so there are no local token-expiry decisions to get wrong when
clocks move. Each process start performs one silent refresh grant.
- **Hardened file backend**: private directory (`0700`), per-record
files (`0600`), owner validation, symlink rejection, and atomic `rename`
replacement. Corrupt, truncated, unknown-version, or permission-invalid
records fail with actionable errors naming the file. Native keyring
backends were evaluated (keyring crate routes Linux through D-Bus/zbus:
heavy deps, headless/CI flakiness) and are deferred; the file store is
the explicit opt-in, not a downgrade from a keyring.
- **Cache key**: SHA-256 of the canonical identity (issuer, client ID,
sorted/de-duplicated scopes, flow, public/confidential), so no secret
appears in a filename and distinct identities never collide. Versioned
record schema (`version: 1`). One record per identity: last login wins,
documented.
- **Cross-process rotation locking**: per-key `fs4` file lock (`flock` /
`LockFileEx`) around the refresh critical section — acquire, reread the
durable record, refresh exactly once, atomically store the rotated
refresh token, release. The OS releases locks on process death, so
crashes cannot strand stale locks. Only confirmed
`invalid_grant`/`invalid_token` deletes a record and reauthenticates;
transport, 5xx, 429, and parse failures retain it.
- **Session APIs**: `OAuthSession::login/status/logout` in Rust,
`lancedb.remote.OAuthSession` (async) in Python, `OAuthSession` class in
Node. `status` returns non-secret metadata only. `logout` removes only
the local credential — provider revocation (RFC 7009) is a deliberate
follow-up, and local logout never terminates browser SSO. Azure managed
identity is rejected for persistence (machine identity stays in memory);
client credentials have nothing refreshable to persist and stay
memory-only.
- No CLI binary exists in this repo, so this ships library APIs plus doc
examples in all three languages.

Tests: Rust unit + mock-IdP integration (cache-key
canonicalization/separation, record
versioning/corruption/truncation/symlink/owner/perms, lock serialization
+ release, two concurrent providers proving no `invalid_grant` and
correct rotation, transient-failure retention, `invalid_grant` delete +
reauthenticate, login/status/logout lifecycle, client-credentials no-op,
IMDS rejection, secret redaction); Python lifecycle + a true
two-subprocess cross-process reuse test (second process refreshes once,
never hits the device endpoint); Node lifecycle + device-flow login
test. Local builds were skipped in development; CI validates all
bindings.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-09-16 02:01:10 +08:00
Drew 6702e3fec1 feat(node): add blob v2 fetch and field helpers (#4155) 
this PR blob v2 field helpers and reads to the Node SDK.

`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.

Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.

Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.

```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
  "videos",
  [{ id: 1n, video }],
  {
    schema: new Schema([
      new Field("id", new Int64()),
      blob("video"),
    ]),
  },
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);

```

### Testing

- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
 2026-09-11 14:46:27 -07:00
Jack Ye 21f11b4463 feat!: replace get_job/job_history with describe_job/query_job_events (#4130) 
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.

## Before

Four ways to ask about a job, none of which told you much.

```python
job = table.refresh_column_async("embedding")
job.status()               # "running". That was the entire debug surface.
db.get_job(job_id)         # state, and a spec. No result, no progress.
db.job_history(job_id)     # raw record batches, no limit, no filter
db.job(job_id)             # a handle that knew nothing
```

## After

Open a job the way you open a table; the handle answers everything.

```python
job = db.open_job(job_id)      # raises JobNotFoundError if there is no such job
```

```python
>>> print(job)
Job(
    id='job-1',
    state='failed',
    job_type='refresh_column',
    creation_ms=1757000000000,
    spec={
        "column": "embedding",
        "num_workers": 4
    },
    failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```

Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.

Per-fragment progress *while it is still running*:

```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist()      # [5000, 5000, ...]
```

The handle an async action returns is the same object, one `refresh()`
away:

```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```

TypeScript is the same experience, down to `console.log`:

```ts
const job = await db.openJob(jobId);   // rejects if there is no such job
console.log(job);                      // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```

## Why each piece matters

- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.

`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.

## Breaking

All shipped in 0.38.0. No deprecated aliases.

| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |

Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
 2026-09-05 16:24:23 -07:00
lancedb-gatefixer[bot]andXuanwo 35b5d015ac fix(node): preserve embedding registration in server bundles (#3806) 
## Summary

- lazily initialize built-in OpenAI and Hugging Face providers when
consumers call the public embedding registry API
- choose automatic vector versus FTS search from embedding metadata on a
fresh pinned table revision for every execution
- expose automatic string searches as an `AutoQuery` with only
operations common to both native query families
- keep the registry shared and built-in registration safe across
duplicated module graphs

## Root cause

Nitro treats dependency modules as side-effect-free and removes the bare
OpenAI provider import from its generated route. Registration therefore
never runs, so `getRegistry().get("openai")` remains undefined even when
the registry itself is shared globally. Bundlers may also duplicate the
provider and registry module graphs.

The public embedding entry point now initializes built-in providers only
when `getRegistry()` is explicitly called, keeping initialization on a
live path that Nitro retains. Each terminal automatic-search execution
pins the exact table revision visible at dispatch, reads embedding
metadata and computes an embedding from that snapshot, replays the
builder operations, and constructs and executes the selected native
query against the same snapshot. Pinned native snapshots execute locally
when namespace pushdown cannot carry their revision, while remote
snapshots are seeded directly from one version-and-schema response. The
public `AutoQuery` builder exposes only the operations shared by FTS and
vector search, so runtime class narrowing cannot expose invalid
vector-only methods. Repeated built-in registration replaces stale
constructors from duplicated module graphs while public `register()`
retains its duplicate-alias error.

## Validation

- `cargo fmt --all`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test --runInBand` (783 passed, 5 skipped)
- serial examples suite with a local OpenAI mock (11 passed), including
`sentence-transformers.test.ts`
- packaged Nitro 2.13.4 server route using the reported imports returned
`{"registered":true}`
- fresh-process FTS fixture initialized both public built-ins and
confirmed automatic string search still returned the indexed row
- schema-consistency regressions cover read-consistency refresh,
checkout, checkoutLatest, restore, runtime class narrowing, concurrent
overwrite during embedding computation, and reused automatic-search
builders
- focused regressions confirm pinned native snapshots bypass unversioned
namespace pushdown and remote snapshots use one describe request

Fixes #2429

<!-- lance-gatekeeper-fix:v1 agent=2adf0f21b8bfb634606ed8897a849e30
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2026-08-26 04:17:32 +08:00
Will JonesandClaude Opus 5 fce45ba9fc feat(nodejs): add listTables, deprecate tableNames (#4041) 
`table_names` is being replaced by `list_tables` across the SDKs, but
TypeScript only had `tableNames`. This PR adds `listTables`, which
returns a page of table names together with the token that resumes after
it, and marks `tableNames` and `TableNamesOptions` deprecated in favor
of it.

It binds the `Connection::list_tables` that already exists, so nothing
in the Rust API changes and nothing existing breaks. `pageToken` is
documented as opaque rather than as a table name, since what resumes a
listing is the database's to decide — that keeps callers off a detail
that is going to change.

Stacked on #4040, which fixes a table being dropped at every page
boundary. The page-walking test here needs that fix to pass. Review the
last commit only until #4040 lands.

## Example

```ts
const names = [];
let pageToken = undefined;
do {
  const page = await conn.listTables({ pageToken, limit: 100 });
  names.push(...page.tables);
  pageToken = page.pageToken;
} while (pageToken);
```

A namespace can be listed by passing its path first, mirroring
`tableNames`:

```ts
const page = await conn.listTables(["analytics"], { limit: 100 });
```

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-24 16:25:14 -07:00
Wyatt Alt 68749ecfa3 feat(nodejs): materialized view bindings (#3935) 
Exposes materialized views to TypeScript: createMaterializedView,
openMaterializedView and listMaterializedViews on Connection, and a
MaterializedView handle carrying the parsed definition and
refresh({full, sourceVersion}), which returns the typed refresh result.
select accepts column names, [alias, expression] pairs, or a record of
the
same; the definition reads back off the stored schema, so a reopened
handle
needs no side channel. Remote connections surface the core's
not-supported
error up front.

The napi crate needed the same recursion-limit raise as the core crate:
the
refresh future's type graph overflows the default trait-recursion depth.


<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-21 23:48:43 -07:00
Drew Gallardo e98d8ac685 feat!: rename branch merge to cherry_pick (#3986) 
This PR is a **breaking** rename of #3686.

merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.

This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.

Cherry-pick is terminology that aligns more with that.

```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")

exp.add_columns({"tag": "cast('draft' as string)"})

diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")

if result["status"] == "cherryPicked":
    print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
    print(result["diff"]["errors"])
```

### Behavior

- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.

### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
 2026-08-21 23:37:12 -07:00
Dan RammerandClaude Opus 5 f1c4967eeb feat: bring the MemWAL LSM surface to parity across the SDKs (#3962) 
## Why

Four of the eight LSM methods are **remote-only in the core**. `impl
BaseTable for NativeTable` implements only
`set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`,
`compact_lsm` and `get_lsm_stats` fall through to trait defaults
returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and
`checkpoint_lsm` is built on all three.

That explains the state of the bindings: Node had bound the four that
work against a local table and stopped, so a Cloud user could install an
LSM write spec but had no way to observe fresh-tier state or drive a
checkpoint. Java had none of it at all.

| SDK | set/unset/get spec | closeWriters | flush | compact | getStats |
checkpoint |
|---|---|---|---|---|---|---|
| Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Node *(before)* | ✅ | ✅ | — | — | — | — |
| **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** |
| Java *(before)* | — | — | — | — | — | — |
| **Java (after)** | **new** | n/a | **new** | **new** | **new** |
**new** |

Go and C are separate repos and are out of scope here. `closeLsmWriters`
drains cached in-process shard writers, so it has no meaning for Java,
which is a pure REST client.

## Node

Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and
`getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats`
/ `MemtableStats` objects — typed rather than a JSON blob, matching the
existing `LsmWriteSpec` object in the same file, with `u64` cast to
`i64` per that file's convention.

Because these four are remote-only, the new tests assert each binding
reaches the core and surfaces `NotSupported` against a local table. That
covers the wiring; behavior against a real endpoint stays covered by the
mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`.

## Python

No new methods. All eight are on `LanceTable`, `AsyncTable` and
`RemoteTable` — the last four landed on the sync `RemoteTable` in #3961,
which is merged into this branch.

What was missing here was reachability. `LsmWriteSpec` was importable
only from the private `lancedb._lancedb`, appearing in `table.py` solely
under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no
mention of it, which per the repo's docs guidance means it rendered
nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in
`__all__`, and documented.

## Java

Java reaches LanceDB purely over REST through the generated Lance
Namespace client, and these routes are not in that spec, so they are
issued through a small dedicated client rather than added to the spec.
That call is revisitable — LSM is one of four unspecified route families
alongside `multipart_write`, `page_cache/prewarm` and
`branches/diff|merge`. If those are ever regularized into the spec as a
group, `LanceDbTableLsm` is one file that gets deleted.

`LsmWriteSpec` here is deliberately **not**
`org.lance.memwal.InitializeMemWalParams`. That type defaults to
maintaining *no* indexes where a spec here defaults to maintaining
*every* index, and it cannot express the `null` that asks the server to
resolve the set:

| Value | On the wire | Meaning |
|---|---|---|
| unset (null) | `null` | Server resolves **every** maintainable index |
| `Collections.emptyList()` | `[]` | Maintain **none** |
| `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those |

A dedicated test pins null and `[]` as distinct on the wire, since
collapsing them is the failure mode that motivated a LanceDB-owned type.

`checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs`
with its constants and status semantics intact: 429/503 retried in place
against an 8-budget, 421 restarting from flush against a 3-budget, 5s
poll, and a target watermark fixed after the seal so it terminates under
write load.

`getLsmStats` returns typed `LsmStats` / `BucketStats` /
`GenerationStats` / `MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes.
Decoding is strict — see below.

## Review feedback

Both gatekeeper findings were real. Each was reproduced against the
scripted test server first, and each fix ships with the reproducer as a
regression test.

**The transport was doubling every checkpoint retry budget.**
`HttpClients.createDefault()` installs Apache's default response retry
strategy, whose retryable-status list is exactly 429 and 503 — the two
statuses `isRetryable` owns. A 429 held against `flush_lsm` issued
**18** wire requests where the loop intends 9, and `compact_lsm` was
retried in place despite the loop being built to fall through to a fresh
stats poll instead. Timing confirmed the mechanism: that run took 25.4s
≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry
interval.

Automatic retries are now disabled, so the checkpoint loop is the sole
owner of the 421/429/503 transitions. A side effect worth noting:
`testCheckpointRetriesRetryableStatusInPlace` was passing on a
transport-absorbed 429 and never reaching `issue()`'s retry branch at
all. It now exercises the real path.

**Stats decoding failed open.** `getLsmStats` read the response with
Jackson's `path()`, which yields a missing node that iterates as an
empty array — making "malformed" indistinguishable from "no buckets",
which is indistinguishable from "drained". Four separate payloads made
`checkpointLsm()` report convergence for a checkpoint that never ran:

| Response | Before | Now |
|---|---|---|
| `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ |
| `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` |
| empty response body | **reported success** | `IllegalStateException` |
| bucket missing required fields | **reported success** |
`IllegalStateException` |

The empty-body row is the one to weight: a proxy 200 with no body is a
realistic production event, and it silently reported a checkpoint that
never happened.

Decoding is now strict and fails closed, matching the serde contract on
the Rust side exactly. One deliberate deviation from the review comment,
which asked that *only* explicit JSON `null` count as disabled: Rust has
`#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to
`None` there too. Java now matches that. It is an absent-or-malformed
**`buckets`** that fails closed, which is the case the comment was
actually protecting.

## Testing

- Java: **33 passing** (8 existing + 25 LSM) against a scripted
`com.sun.net.httpserver.HttpServer` — no new test dependency. Wire
assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`;
checkpoint tests cover convergence, not piling onto a latched bucket,
421 restart-from-flush, 429 retry-in-place, terminal-status propagation,
reissue exhaustion, the exact wire-request count against the retry
budget, and five malformed stats payloads.
- Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm
run tsc`, `npm run lint`, `npm run docs` all clean.
- Python: `ruff format --check` and `ruff check` clean.
- Java formatting: `./mvnw -pl lancedb-core spotless:apply` and
`spotless:check` both clean under a JDK 11 toolchain.

## Note: spotless needs a pre-16 JDK

`./mvnw spotless:apply` fails on JDK 16+ with
`JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7,
pinned at `java/pom.xml:34`, predates JDK 16's compiler API change.
**This is pre-existing** and reproduces on a pristine `main` checkout.

It is not a blocker, just a toolchain requirement. Spotless was run
against these sources under JDK 11 and both `spotless:apply` and
`spotless:check` pass on the whole module:

```shell
JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply
```

Bumping the plugin so it works on modern JDKs is still worth doing, but
separately from this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-19 11:44:46 -05:00
Wyatt Alt fc0d917d32 feat: refresh computed columns (#3938) 
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
 2026-08-14 14:43:41 -07:00
Wyatt AltandClaude Fable 5 e3b472c212 feat: connection-level job operations (#3755) 
Adds job operations to the connection surface, building on the Job
handle from #3742: job(id), list_jobs, get_job, cancel_job, and
job_history, plus a non-blocking Job.status(). Implemented on the
Database trait (defaulting to NotSupported), the remote backend
(/v1/jobs), and the Python and Node bindings; job_history returns Arrow
batches.

errors() and progress() are not included.

Tested with mocked endpoints in all three languages.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-31 12:51:43 -07:00
Wyatt AltandClaude Fable 5 a6418b6cb9 feat: create_index returns a Job handle (#3742) 
IndexBuilder::execute now returns a Job with wait and cancel methods.
Local tables build the index synchronously and return an already-done
job. Remote tables read the job id the server returns from create_index
and track it through the /v1/jobs API: wait polls describe until the job
reaches a terminal state and cancel posts a cancellation. Servers that
return no job id yield a done job, so behavior against older servers is
unchanged. The job id is not exposed on the handle.

The Python and TypeScript bindings keep their current signatures and
discard the handle; exposing Job there is left to follow-ups.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-31 07:32:28 -07:00
Drew GallardoandCursor 65cd142c7e feat: add remote branch diff and merge client APIs (#3686) 
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.


This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`

Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.

Merge here means promoting a branch's added columns onto `main`.

### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.

### Example
```python

table = db.open_table("images")

table.branches.create("exp")
exp = table.branches.checkout("exp")

exp.add_columns({"tag": "cast('draft' as string)"})

diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)

if result["status"] == "merged":
    print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
    print(result["diff"]["mergeBlockers"])
```

### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-18 12:38:05 -07:00
Jack Ye 37032151d3 feat: support distributed analyze plan metrics in clients (#3675) 
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
 2026-07-15 21:21:40 -07:00
Jack Ye 06b53c97d6 feat: add table FTS query tokenization (#3659) 
## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage
 2026-07-14 10:59:33 -07:00
Will JonesandClaude Opus 4.8 285add40dd feat: expose Lance metrics via OpenTelemetry in Python and Node (#3609) 
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.

## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.

## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.

## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.

Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.

## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-09 15:36:03 -07:00
Jack Ye 10fecdf051 feat(node): expose OAuth connection config (#3587) 
Expose the merged Rust OAuth header provider through the Node/TypeScript
connection path.

Includes:
- Native OAuthConfig conversion for napi-rs
- ConnectionOptions.oauthConfig plumbing
- Public TypeScript OAuthConfig and OAuthFlowType exports
- Generated TypeScript API docs for the new config surface
- input-validation and debug-redaction coverage in the Rust binding
layer

Local validation: cargo fmt --all; git diff --check.
 2026-06-29 16:55:45 -07:00
Brendan Clement 53517b3aaa feat: add table branch support (#3490) 
### Description

Adds first-class support for table branches across the Rust core and the
Python and TypeScript SDKs.

Rust

```rust
use lance::dataset::refs::Ref;

// Create a branch from main and write to it — main is untouched.
let exp = table.create_branch("exp", Ref::Version(None, None)).await?;
exp.add(batches).await?;

// Reopen the branch later: check out from a table, or open it directly.
let exp = table.checkout_branch("exp").await?;
let exp = db.open_table("items").branch("exp").execute().await?;

let branches = table.list_branches().await?;
table.delete_branch("exp").await?;
```

Python

```python
# Create a branch from main and write to it
branch = await table.branches.create("exp", from_ref="main")
await branch.add(data)

# Reopen the branch later: check out from a table, or open it directly.
branch = await table.branches.checkout("exp")
branch = await db.open_table("items", branch="exp")

await table.branches.list()
await table.branches.delete("exp")
```

TypeScript

```typescript
const branches = await table.branches();

// Create a branch from main and write to it
const branch = await branches.create("exp");
await branch.add(data);

// Reopen the branch later: check out from a table, or open it directly.
const checkedOut = await branches.checkout("exp");
const opened = await db.openTable("items", undefined, { branch: "exp" });

await branches.list();
await branches.delete("exp");
```

### Testing
- Added unit tests
- ran smoke tests against python and typescript sdks on local machine


### Next steps
- Add RemoteTable support
- Add Branch Comparison support
- Merge Branching support
 2026-06-08 16:26:46 -07:00
Brendan Clement d065be0474 feat: add update_field_metadata to edit per-field metadata (#3482) 
### Summary
Adds update_field_metadata to the client SDK (Rust core, Python, and
TypeScript) so clients can edit per-field (column) Arrow metadata
(schema.fields[].metadata)

### Testing
- added unit tests
- ran E2E against a local server on both local and remote tables (set →
merge → delete), across Python sync/async and TypeScript

### Next steps
- deprecate replace_field_metadata in the python lancedb favor of this
(typescript didn't have replace_field_metadata method). This matches
Lance's API direction (Lance already deprecated replace_field_metadata
for update_field_metadata)
 2026-06-02 07:00:00 -07:00
Brendan Clement 4cb9147bbf feat(nodejs): add renameTable on Connection (#3386) 
Adds `Connection.renameTable` to the Node SDK. Closes #3381.
 2026-05-20 09:05:48 -07:00
Brendan Clement 049b0c8f09 feat(nodejs): add progress to Table.add (#3398) 
### Summary

- Add an optional `progress` callback to `Table.add(data, { progress
})`. Callback fires once per batch written and once more with `done:
true` when the write completes.
- Errors thrown from the user's callback are logged with `console.warn`
and swallowed

### Testing
- npm test 
- ran smoke test script to verify functionality
 2026-05-19 18:35:07 -07:00
Heng Ge 0d30b31998 feat: support setting LSM write spec for a table (#3396) 
## Summary

Split out from #3354

Adds `LsmWriteSpec` and `Table::set_lsm_write_spec` /
`unset_lsm_write_spec` to
install and clear the spec that selects Lance's MemWAL LSM-style write
path for
`merge_insert`.

`LsmWriteSpec` offers three sharding strategies, all built on Lance's
`InitializeMemWalBuilder`:

- `LsmWriteSpec::bucket(column, num_buckets)` — hash-bucket sharding by
the
  single-column unenforced primary key.
- `LsmWriteSpec::identity(column)` — identity sharding by the raw value
of a
  scalar column.
- `LsmWriteSpec::unsharded()` — a single MemWAL shard.

Each can be refined with `with_maintained_indexes(...)` (indexes the
MemWAL
keeps up to date as rows are appended) and
`with_writer_config_defaults(...)`
(default `ShardWriter` configuration recorded in the MemWAL index, so
every
writer starts from the same defaults). All variants require the table to
have
an unenforced primary key.

- `set_lsm_write_spec` installs the spec by initializing the MemWAL
index;
`unset_lsm_write_spec` removes it (dropping the MemWAL index), reverting
to
  the standard `merge_insert` path. `unset` is idempotent.
- Bindings: Python (`LsmWriteSpec.bucket` / `.identity` / `.unsharded`,
  `set_lsm_write_spec` / `unset_lsm_write_spec`) and TypeScript
  (`setLsmWriteSpec` with `specType` `"bucket"` / `"identity"` /
  `"unsharded"`). `RemoteTable` returns `NotSupported`.

The actual `merge_insert` LSM dispatch and `ShardWriter` write path are
a
follow-up — this PR only installs and clears the spec.
 2026-05-18 00:11:33 -07:00
Xin Sun ab2c5adf5e feat(nodejs): add order_by method to Query (#3123) 2026-05-16 22:49:08 -07:00
Tanay df4ad9f851 feat(nodejs): add Scannable primitive for streaming ingestion (#3271) 
## **Summary**

This PR adds a **Scannable primitive** to the Node.js bindings, bringing
parity with Python's `PyScannable`.

A `Scannable` wraps a schema, an optional row count hint, a rescannable
flag, and a batch producing callback. On the Rust side it implements
`lancedb::data::scannable::Scannable`. The goal is to give consumers
such as `Table.add`, `createTable`, and `mergeInsert` a way to stream
data without materializing the full dataset in JS memory.

This PR introduces only the primitive. Migrating existing consumers to
use it will come in follow up work.

---

## **Design**

### **Transport**

The transport uses the **Arrow IPC Stream format, one batch at a time**.

The JS side encodes each `RecordBatch` into a self contained IPC Stream
message containing schema, batch, and end of stream. The message is
returned as a `Buffer` through a napi `ThreadsafeFunction`. The Rust
side decodes it using `arrow_ipc::reader::StreamReader`.

Only one batch is active at a time, so JS memory stays bounded by the
batch size. The Node `Buffer` size limit of about 4 GiB therefore does
not constrain the stream as a whole.

I initially evaluated the Arrow C Data Interface, which is the approach
used in Python. I dropped that path after confirming that the
`apache-arrow` npm package does not expose a C Data Interface export in
any supported version from 15 to 18. JavaScript is not listed in Arrow's
C Data Interface implementation table, and the upstream tracking issue
remains open with no scheduled work.

Third party FFI shims would introduce additional dependency risk without
solving the core maintenance problem. Using IPC adds one encode and
decode step per batch, but the cost is predictable and typically
dominated by Lance's write path.

---

### **API**

```ts
class Scannable {
  readonly schema: Schema
  readonly numRows: number | null
  readonly rescannable: boolean

  static fromFactory(schema, factory, opts?)
  static fromTable(table, opts?)
  static fromIterable(schema, iter, opts?)
  static fromRecordBatchReader(reader, opts?)
}
```

The FFI boundary consists of a single callback:

`getNextBatch(isStart: boolean): Promise<Buffer | null>`

`isStart` is `true` on the first call of each new scan and `false` for
every call after it. The JS side uses it to drop any cached iterator and
re-invoke the factory at scan boundaries. This is what makes a
rescannable source restart at batch 0 on every `scan_as_stream` call,
even when a previous scan ended mid stream, for example a retried write
after a network error. Without this signal a retry would resume a stale
iterator and silently skip already emitted batches.

In addition, a schema only IPC buffer is transferred once during
construction.

---

## **Changes**

* `nodejs/src/scannable.rs`
Adds `NapiScannable` and the `LanceScannable` implementation. Implements
`schema()`, `num_rows()`, `rescannable()`, and `scan_as_stream()`.
Includes per batch schema validation against the declared schema, one
shot enforcement for non rescannable sources, and a scan boundary reset
signal (`isStart`) so rescannable sources restart from batch 0 on every
`scan_as_stream` call rather than resuming a stale iterator.

* `nodejs/src/lib.rs`
  Module registration.

* `nodejs/lancedb/scannable.ts`
Defines the `Scannable` class and the four constructors listed above.
Each constructor rejects option combinations it cannot honor, for
example a `rescannable: true` request on a one shot iterable or reader,
and a `numRows` that disagrees with an in memory table's row count.

* `nodejs/lancedb/index.ts`
  Exports the new primitive.

* `nodejs/__test__/scannable.test.ts`
  Test suite for the primitive.

---

## **Validation**

Before implementing the bridge, I ran an end to end harness with a JS
producer feeding a standalone Rust consumer built against the same
`arrow-ipc` version used in the bridge.

The harness covered the following scenarios:

* happy path
* empty stream
* 1,000 small batches
* 10 large batches
* mixed primitive types with nullables
* nested `List<Struct<>>`
* truncated stream error handling
* declared schema mismatch validation
* a 6 GB stress test through the pipe

All scenarios completed with bounded memory usage. The goal of this
harness was to confirm that the IPC Stream transport works correctly end
to end and that Node's `Buffer` size limit does not constrain the
overall stream.

Separately, the rescannable restart contract was verified with a focused
harness. A rescannable source is consumed partially and the scan is
dropped mid stream, then re-scanned. The re-scan replays from batch 0
rather than resuming the stale iterator. The same harness was run with
the `isStart` reset path disabled and the mid stream restart case failed
as expected, confirming the test exercises the real regression.

These harnesses are not meant to replace the full test suite, which is
described below.

---

## **Tests**

`__test__/scannable.test.ts` covers construction, metadata reflection,
per constructor defaults and overrides, construction time validation,
the native handle surface, and schema variety across empty tables,
nested types, `FixedSizeList`, and wide schemas.

Runtime scan behavior including `scan_as_stream`, one shot enforcement
on non rescannable sources, schema mismatch detection, IPC decode
failures, and rescannable restart semantics is not exercised here. There
is no in tree JS consumer of `NapiScannable` yet. This mirrors Python's
`PyScannable`, which has no dedicated test file and is covered
transitively through the consumers that accept a Scannable.

Runtime coverage will follow in the consumer migration work.

---

## **Status**

Ready for review.

Closes #3223

---
 2026-05-14 15:07:41 -07:00
Brendan Clement 9330a9b851 feat(nodejs): expose connectNamespace for namespace-backed connections (#3383) 
### Summary 

Adds a `connectNamespace(implName, properties, options?)` to the NodeJS
SDK`. Closes #3380.

### Testing
- pnpm test
- Ran smoke test

```
import { connectNamespace } from "lancedb"
import { tmpdir } from "os";
import { mkdtempSync } from "fs";
import { join } from "path";

const dir = mkdtempSync(join(tmpdir(), "lancedb-connect-namespace-smoke-"));
console.log(`Using temp dir: ${dir}\n`);

// 1. Happy path: connect via the "dir" namespace impl, create + list a table.
console.log('Connecting via connectNamespace("dir", { root })...');
const db = await connectNamespace("dir", { root: dir });
console.log("  ✓ connected:", db.display());

console.log("Creating a table and listing it...");
await db.createTable("users", [
  { id: 1, name: "alice" },
  { id: 2, name: "bob" },
]);
console.log("  ✓ tableNames ->", await db.tableNames());

const table = await db.openTable("users");
console.log("  ✓ users.countRows ->", await table.countRows());

// 2. Storage options pass-through.
console.log("\nReconnecting with storageOptions (plumbing check)...");
const dbWithOpts = await connectNamespace(
  "dir",
  { root: dir },
  { storageOptions: { newTableDataStorageVersion: "stable" } },
);
console.log("  ✓ connected with storageOptions:", dbWithOpts.display());
await dbWithOpts.close();

// 3. Empty implName -> clear error.
console.log("\nCalling connectNamespace('', {}) (expect error)...");
try {
  await connectNamespace("", {});
  console.error("  UNEXPECTED: empty implName did not throw");
} catch (err) {
  console.log(`  ✓ Got expected error: ${err.message.split("\n")[0]}`);
}

// 4. Unknown impl -> error.
console.log("\nCalling connectNamespace('not-a-real-impl', {}) (expect error)...");
try {
  await connectNamespace("not-a-real-impl", {});
  console.error("  UNEXPECTED: unknown impl did not throw");
} catch (err) {
  console.log(`  ✓ Got expected error: ${err.message.split("\n")[0]}`);
}

// 5. Create a table inside a child namespace, then reconnect with a fresh
//    connectNamespace call and confirm the table is reachable via that
//    namespace path. (The dir+manifest impl keeps the namespace hierarchy in
//    a root manifest, so "scoping" happens via namespacePath args, not by
//    pointing root at a subdir.)
console.log("\nCreating a table inside a child namespace...");
const dir2 = mkdtempSync(join(tmpdir(), "lancedb-connect-namespace-smoke-"));
const writer = await connectNamespace("dir", {
  root: dir2,
  manifest_enabled: "true",
});
await writer.createNamespace(["analytics"]);
await writer.createTable(
  "orders",
  [
    { id: 1, total: 10 },
    { id: 2, total: 20 },
  ],
  ["analytics"],
);
console.log(
  "  ✓ writer sees tables under [analytics] ->",
  await writer.tableNames(["analytics"]),
);
await writer.close();

console.log("Reconnecting and reading the table via its namespace path...");
const reader = await connectNamespace("dir", {
  root: dir2,
  manifest_enabled: "true",
});
console.log(
  "  ✓ reader tableNames(['analytics']) ->",
  await reader.tableNames(["analytics"]),
);
const orders = await reader.openTable("orders", ["analytics"]);
console.log("  ✓ orders.countRows via reader ->", await orders.countRows());
await reader.close();

await db.close();
console.log("\nAll checks passed.");
```

```
Using temp dir: /var/folders/bj/hn6jv9c50y301d1nx0y8xmn00000gn/T/lancedb-connect-namespace-smoke-WByF1P

Connecting via connectNamespace("dir", { root })...
  ✓ connected: LanceNamespaceDatabase
Creating a table and listing it...
  ✓ tableNames -> [ 'users' ]
  ✓ users.countRows -> 2

Reconnecting with storageOptions (plumbing check)...
  ✓ connected with storageOptions: LanceNamespaceDatabase

Calling connectNamespace('', {}) (expect error)...
  ✓ Got expected error: implName must be a non-empty string

Calling connectNamespace('not-a-real-impl', {}) (expect error)...
  ✓ Got expected error: Invalid input, Failed to connect to namespace: Namespace { source: Unsupported { message: "Implementation 'not-a-real-impl' is not available. Supported: dir, rest" }, location: Location { file: "/Users/brendan/.cargo/git/checkouts/lance-8ddea23c38163eda/f693245/rust/lance-namespace-impls/src/connect.rs", line: 216, column: 14 } }

Creating a table inside a child namespace...
  ✓ writer sees tables under [analytics] -> [ 'orders' ]
Reconnecting and reading the table via its namespace path...
  ✓ reader tableNames(['analytics']) -> [ 'orders' ]
  ✓ orders.countRows via reader -> 2

All checks passed.
```

### Docs
- regenerated docs
 2026-05-13 16:16:56 -07:00
Brendan Clement 02de07576e feat(nodejs): add namespace management methods on Connection (#3371) 
### Summary

Closes #3363 

Adds the four namespace management methods to the NodeJS `Connection`,
bringing parity with the Rust core and Python bindings:

- `listNamespaces(parent?, options?)`
- `createNamespace(namespacePath, options?)`
- `dropNamespace(namespacePath, options?)`
- `describeNamespace(namespacePath)`

### Test plan
- npm test
- Ran a smoke test script

```typescript
import { connect } from '<lancePath>'
import { tmpdir } from "os";
import { mkdtempSync } from "fs";
import { join } from "path";

const dir = mkdtempSync(join(tmpdir(), "lancedb-smoke-"));
console.log(`Using temp dir: ${dir}\n`);

const db = await connect(dir, {
  namespaceClientProperties: { manifest_enabled: "true" },
});

console.log("Creating namespaces...");
await db.createNamespace(["analytics"]);
await db.createNamespace(["analytics", "sales"], {
  properties: { owner: "brendan", purpose: "smoke-test" },
});
await db.createNamespace(["marketing"]);

const root = await db.listNamespaces();
console.log("Root namespaces:", root.namespaces);

const children = await db.listNamespaces(["analytics"]);
console.log("Children of 'analytics':", children.namespaces);

const descWithProps = await db.describeNamespace(["analytics", "sales"]);
console.log("Describe analytics/sales (with properties):", descWithProps);

const descNoProps = await db.describeNamespace(["analytics"]);
console.log("Describe analytics (no properties):", descNoProps);

console.log("Describing a non-existent namespace (expect error)...");
try {
  await db.describeNamespace(["does-not-exist"]);
  console.error("  UNEXPECTED: describe succeeded for non-existent namespace");
} catch (err) {
  console.log(`  ✓ Got expected error: ${err.message.split("\n")[0]}`);
}

await db.dropNamespace(["marketing"]);
const afterDrop = await db.listNamespaces();
console.log("Root after dropping marketing:", afterDrop.namespaces);

await db.close();
console.log("\nAll operations completed successfully.");
```

```
Using temp dir: /var/folders/bj/hn6jv9c50y301d1nx0y8xmn00000gn/T/lancedb-smoke-MUC5NI

Creating namespaces...
Root namespaces: [ 'analytics', 'marketing' ]
Children of 'analytics': [ 'sales' ]
Describe analytics/sales (with properties): { properties: { purpose: 'smoke-test', owner: 'brendan' } }
Describe analytics (no properties): {}
Describing a non-existent namespace (expect error)...
  ✓ Got expected error: lance error: Namespace error: Namespace not found: does-not-exist, rust/lance-namespace-impls/src/dir/manifest.rs:2495:14  Caused by: Namespace error: Namespace not found: does-not-exist, rust/lance-namespace-impls/src/dir/manifest.rs:2495:14    Caused by: Namespace not found: does-not-exist
Root after dropping marketing: [ 'analytics' ]

All operations completed successfully.
```

### Documentation
- regenerated docs
 2026-05-13 11:49:27 -07:00
Weston Pace aeac9c7644 feat: add python Permutation class to mimic hugging face dataset and provide pytorch dataloader (#2725) 2025-11-06 16:15:33 -08:00
S.A.N 20bec61ecb refactor(node): async generator for RecordBatchIterator (#2744) 
JS native Async Generator, more efficient asynchronous iteration, fewer
synthetic promises, and the ability to handle `catch` or `break` of
parent loop in `finally` block
 2025-10-30 14:36:24 -07:00
Weston Pace 5a19cf15a6 feat: a utility for creating "permutation views" (#2552) 
I'm working on a lancedb version of pytorch data loading (and hopefully
addressing https://github.com/lancedb/lance/issues/3727).

However, rather than rely on pytorch for everything I'm moving some of
the things that pytorch does into rust. This gives us more control over
data loading (e.g. using shards or a hash-based split) and it allows
permutations to be persistent. In particular I hope to be able to:

* Create a persistent permutation
* This permutation can handle splits, filtering, shuffling, and sharding
* Create a rust data loader that can read a permutation (one or more
splits), or a subset of a permutation (for DDP)
* Create a python data loader that delegates to the rust data loader

Eventually create integrations for other data loading libraries,
including rust & node
 2025-10-09 18:07:31 -07:00
BubbleCal b59d1007d3 feat(index): add IVF_RQ index type (#2687) 
this expose IVF_RQ (RabitQ quantization) index type to lancedb

---------

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
 2025-10-09 15:46:18 +08:00
Will Jones 48e5caabda ci(nodejs): lint for unused imports (#2673) 2025-09-23 18:49:42 -07:00
Jack YeandClaude 8da74dcb37 feat: support per-request header override (#2631) 
## Summary

This PR introduces a `HeaderProvider` which is called for all remote
HTTP calls to get the latest headers to inject. This is useful for
features like adding the latest auth tokens where the header provider
can auto-refresh tokens internally and each request always set the
refreshed token.

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2025-09-10 13:44:00 -07:00
Weston Pace ed640a76d9 feat: add take_offsets and take_row_ids (#2584) 
These operations have existed in lance for a long while and many users
need to drop down to lance for this capability. This PR adds the API and
implements it using filters (e.g. `_rowid IN (...)`) so that in doesn't
currently add any load to `BaseTable`. I'm not sure that is sustainable
as base table implementations may want to specialize how they handle
this method. However, I figure it is a good starting point.

In addition, unlike Lance, this API does not currently guarantee
anything about the order of the take results. This is necessary for the
fallback filter approach to work (SQL filters cannot guarantee result
order)
 2025-08-15 06:48:24 -07:00
Will JonesandClaude 3d1f102087 feat: allow Python and Typescript users to create Sessions (#2530) 
## Summary
- Exposes `Session` in Python and Typescript so users can set the
`index_cache_size_bytes` and `metadata_cache_size_bytes`
* The `Session` is attached to the `Connection`, and thus shared across
all tables in that connection.
- Adds deprecation warnings for table-level cache configuration


🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2025-07-24 12:06:29 -07:00
BubbleCal 96c66fd087 feat: support multivector for JS SDK (#2527) 
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
 2025-07-22 21:19:34 +08:00
BubbleCal fec8d58f06 feat: support a bunch or FTS features in JS SDK (#2431) 
- operator for match query
- slop for phrase query
- boolean query

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced support for boolean full-text search queries with AND/OR
logic and occurrence conditions.
- Added operator options for match and multi-match queries to control
term combination logic.
- Enabled phrase queries to specify proximity (slop) for flexible phrase
matching.
- Added new enumerations (`Operator`, `Occur`) and the `BooleanQuery`
class for enhanced query expressiveness.

- **Bug Fixes**
- Improved validation and error handling for invalid operator and
occurrence inputs in full-text queries.

- **Tests**
- Expanded test coverage with new cases for boolean queries and
operator-based full-text searches.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
 2025-06-12 17:04:19 +08:00
Will Jones 272e4103b2 feat: provide timeout parameter for merge_insert (#2378) 
Provides the ability to set a timeout for merge insert. The default
underlying timeout is however long the first attempt takes, or if there
are multiple attempts, 30 seconds. This has two use cases:

1. Make the timeout shorter, when you want to fail if it takes too long.
2. Allow taking more time to do retries.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added support for specifying a timeout when performing merge insert
operations in Python, Node.js, and Rust APIs.
- Introduced a new option to control the maximum allowed execution time
for merge inserts, including retry timeout handling.

- **Documentation**
- Updated and added documentation to describe the new timeout option and
its usage in APIs.

- **Tests**
- Added and updated tests to verify correct timeout behavior during
merge insert operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-05-08 13:07:05 -07:00
LuQQiu ed594b0f76 feat: return version for all write operations (#2368) 
return version info for all write operations (add, update, merge_insert
and column modification operations)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Table modification operations (add, update, delete, merge,
add/alter/drop columns) now return detailed result objects including
version numbers and operation statistics.
- Result objects provide clearer feedback such as rows affected and new
table version after each operation.

- **Documentation**
- Updated documentation to describe new result objects and their fields
for all relevant table operations.
- Added documentation for new result interfaces and updated method
return types in Node.js and Python APIs.

- **Tests**
- Enhanced test coverage to assert correctness of returned versioning
and operation metadata after table modifications.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-05-05 14:25:34 -07:00
Alex PilonandWill Jones f315f9665a feat: implement bindings to return merge stats (#2367) 
Based on this comment:
https://github.com/lancedb/lancedb/issues/2228#issuecomment-2730463075
and https://github.com/lancedb/lance/pull/2357

Here is my attempt at implementing bindings for returning merge stats
from a `merge_insert.execute` call for lancedb.

Note: I have almost no idea what I am doing in Rust but tried to follow
existing code patterns and pay attention to compiler hints.
- The change in nodejs binding appeared to be necessary to get
compilation to work, presumably this could actual work properly by
returning some kind of NAPI JS object of the stats data?
- I am unsure of what to do with the remote/table.rs changes -
necessarily for compilation to work; I assume this is related to LanceDB
cloud, but unsure the best way to handle that at this point.

Proof of function:

```python
import pandas as pd
import lancedb


db = lancedb.connect("/tmp/test.db")

test_data = pd.DataFrame(
    {
        "title": ["Hello", "Test Document", "Example", "Data Sample", "Last One"],
        "id": [1, 2, 3, 4, 5],
        "content": [
            "World",
            "This is a test",
            "Another example",
            "More test data",
            "Final entry",
        ],
    }
)

table = db.create_table("documents", data=test_data, exist_ok=True, mode="overwrite")

update_data = pd.DataFrame(
    {
        "title": [
            "Hello, World",
            "Test Document, it's good",
            "Example",
            "Data Sample",
            "Last One",
            "New One",
        ],
        "id": [1, 2, 3, 4, 5, 6],
        "content": [
            "World",
            "This is a test",
            "Another example",
            "More test data",
            "Final entry",
            "New content",
        ],
    }
)

stats = (
    table.merge_insert(on="id")
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute(update_data)
)

print(stats)
```

returns

```
{'num_inserted_rows': 1, 'num_updated_rows': 5, 'num_deleted_rows': 0}
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Summary by CodeRabbit

- **New Features**
- Merge-insert operations now return detailed statistics, including
counts of inserted, updated, and deleted rows.
- **Bug Fixes**
- Tests updated to validate returned merge-insert statistics for
accuracy.
- **Documentation**
- Method documentation improved to reflect new return values and clarify
merge operation results.
- Added documentation for the new `MergeStats` interface detailing
operation statistics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
 2025-05-01 10:00:20 -07:00
Ryan Green af54e0ce06 feat: add table stats API (#2363) 
* Add a new "table stats" API to expose basic table and fragment
statistics with local and remote table implementations

### Questions
* This is using `calculate_data_stats` to determine total bytes in the
table. This seems like a potentially expensive operation - are there any
concerns about performance for large datasets?

### Notes
* bytes_on_disk seems to be stored at the column level but there does
not seem to be a way to easily calculate total bytes per fragment. This
may need to be added in lance before we can support fragment size
(bytes) statistics.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added a method to retrieve comprehensive table statistics, including
total rows, index counts, storage size, and detailed fragment size
metrics such as minimum, maximum, mean, and percentiles.
- Enabled fetching of table statistics from remote sources through
asynchronous requests.
- Extended table interfaces across Python, Rust, and Node.js to support
synchronous and asynchronous retrieval of table statistics.
- **Tests**
- Introduced tests to verify the accuracy of the new table statistics
feature for both populated and empty tables.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-04-29 15:19:08 -02:30
LuQQiu a9311c4dc0 feat: add list/create/delete/update/checkout tag API (#2353) 
add the tag related API to list existing tags, attach tag to a version,
update the tag version, delete tag, get the version of the tag, and
checkout the version that the tag bounded to.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced table version tagging, allowing users to create, update,
delete, and list human-readable tags for specific table versions.
  - Enabled checking out a table by either version number or tag name.
- Added new interfaces for tag management in both Python and Node.js
APIs, supporting synchronous and asynchronous workflows.

- **Bug Fixes**
  - None.

- **Documentation**
- Updated documentation to describe the new tagging features, including
usage examples.

- **Tests**
- Added comprehensive tests for tag creation, updating, deletion,
listing, and version checkout by tag in both Python and Node.js
environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2025-04-28 10:04:46 -07:00
Weston PaceandBubbleCal 625bab3f21 feat: update to lance 0.25.3b1 (#2294) 
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Chores**
- Updated dependency versions for improved performance and
compatibility.

- **New Features**
- Added support for structured full-text search with expanded query
types (e.g., match, phrase, boost, multi-match) and flexible input
formats.
- Introduced a new method to check server support for structural
full-text search features.
- Enhanced the query system with new classes and interfaces for handling
various full-text queries.
- Expanded the functionality of existing methods to accept more complex
query structures, including updates to method signatures.

- **Bug Fixes**
  - Improved error handling and reporting for full-text search queries.

- **Refactor**
- Enhanced query processing with streamlined input handling and improved
error reporting, ensuring more robust and consistent search results
across platforms.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Co-authored-by: BubbleCal <bubble-cal@outlook.com>
 2025-04-01 06:36:42 -07:00
BubbleCal bdb6c09c3b feat: support binary vector and IVF_FLAT in TypeScript (#2221) 
resolve #2218

---------

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
 2025-03-21 10:57:08 -07:00