mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
d7e921dbec9fd8b7735d97eb10a56cf98ac8dfdb
97
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Daniel RammerandClaude Opus 5.5
|
d7e921dbec |
refactor!: rename the LSM table APIs to MemWAL
The write path these APIs drive is Lance's MemWAL, and "LSM" described its shape rather than naming it. The surface now says MemWAL, and turning it on and off reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. The flush/compact/checkpoint methods keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` and `optimize`, which compact the base table. ## Wire The remote routes and JSON envelopes move with the SDKs and must ship together with the matching server change: - `/set_lsm_write_spec/` -> `/enable_memwal/`, `/unset_lsm_write_spec/` -> `/disable_memwal/`, `/get_lsm_write_spec/` -> `/get_memwal_spec/`, `/flush_lsm/`, `/compact_lsm/`, `/get_lsm_stats/` -> `*_memwal/` - response keys `lsm_write_spec` -> `memwal_spec`, `lsm_stats` -> `memwal_stats`; query and merge-insert body field `use_lsm` -> `use_memwal` ## Also fixed The previous rename turned the Java `bucket` sharding mode into `tableShard` in the javadoc example and tests, and the test fixtures sent `tableShards` where the parser reads `table_shards`, so the Java tests did not compile. Both are restored. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` and `query::lsm` modules) are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Daniel RammerandClaude Opus 5.5
|
d5544cd5d9 |
Merge branch 'main' into refactor/wal-naming
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
 Yang Cen
|
0bceaaf3f0 |
feat(search): expose search effort (#4520)
Lance supports `search_effort`, but LanceDB cannot pass it through its query APIs. Expose the parameter in Rust, Python sync/async and TypeScript, including remote requests and hybrid query conversion, using the [Lance v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13) dependency already on main. The default is `None` (Auto, internally 0.5). Any explicit effort, including 0.5, overrides fixed `nprobes` in either setter order; explicit minimum/maximum bounds remain effective. Invalid values are rejected in the Rust core. Hybrid conversion now retains the vector probe parameters and handles an inferred FTS column without panicking. Namespace queries with explicit effort execute locally because the namespace wire model cannot carry it. MemWAL LSM queries reject it explicitly because Lance's LSM scanner does not yet expose this option. Maturin is included in the dev extra so the documented build command works in the project environment. BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now includes `search_effort: Option<f64>`. Downstream code that initializes every field explicitly must add `search_effort: None` or use `..Default::default()`. Exhaustive destructuring patterns must include the new field or `..`. Query-builder calls and struct literals that already use `..Default::default()` require no source changes. The new effort setters in Python and TypeScript are additive. This PR does not change the persisted file format. Using explicit effort with remote tables requires server support for `search_effort` and its precedence over `nprobes`; the client wire tests do not establish that a deployed server supports it. Server-side support is tracked in lancedb/sophon#8607. Validation on AWS after merging current main (Lance beta.13 / SDK beta.5): - 14 Rust regressions and the API doctest pass. - 60 parameter regressions, the full 47-test namespace suite, and three API doctests pass. The parameter selection includes some namespace cases. - 40 TypeScript query tests and 54 remote tests pass; build, lint and generated docs pass. - Rust Clippy, Python package lint/format, Pyright and lockfile checks pass. Repository-wide Ruff reports 49 lint errors and 11 formatting failures in unchanged `docs/web-tests` files; those files are byte-identical to main and outside this change. All 1,103 Git-tracked files match the AWS validation checkout. No benchmark scripts or result artifacts are included, and no performance improvement is claimed. Uses lance-format/lance#9798. Server-side support is in lancedb/sophon#8607. |
||
 Jack Ye
|
f305a05fd4 |
fix: preserve adaptive IVF probe defaults (#4168)
Keep IVF probe fields unset by default so Lance supplies adaptive behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes` independently through Rust, Python, namespace, and remote request construction, with compatibility fields serialized only when explicitly configured. The final Rust namespace and LSM propagation update depends on the Lance release containing lance-format/lance#9184. |
||
 Colin Patrick McCabe
|
de005f1d61 |
refactor!: implement lazy iterators for several APIs (#4459)
Implement lazy iterators for: - secrets - views - jobs - functions - materialized views |
||
 XY Zhan
|
2bf70758a7 |
fix: read back the maintained-index selection a table was given (#4447)
`maintained_indexes` on a MemWAL table has three values: a named list, `[]` for none, and `null` for every index the table has, including ones created later. This PR makes every client read all three back as they were set. - **Rust remote client** (also used by the Python and Node SDKs) reads `null` as `None`. - **Java client** reads `null` as `null`, not as an empty list. - **Docs** for the setter, the field and the getter describe the three values the same way in Rust, Python, Node and Java. The generated JS docs are regenerated to match. ## Testing - `rust/lancedb`: 73 LSM tests pass, including one case per value. With the decoder typed as a list, the `null` case fails. - Java: two cases for `null` and `[]` on the read path, run by the Java CI job. - `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean. |
||
|
Jack Ye
|
7e8f0d4982 |
refactor: always fully refresh materialized views (#4373)
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.
## Current layout
Local materialized views currently maintain incremental-refresh state
alongside the materialized values:
- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.
Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.
## New layout
An MV is an ordinary Lance backing table containing only:
- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.
There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.
Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.
## New remote create workflow
```text
Connection::create_materialized_view(...)
-> builder renders the defining SELECT
-> remote Database receives a CreateMaterializedViewRequest
-> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
-> retain the SQL Query handle while the Job is running
-> consume the durable SQL-job result
-> open the created table and read mv.definition
-> return MaterializedView
```
Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.
## API changes
- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.
The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
|
||
|
Colin Patrick McCabe
|
d864e7198f |
feat!: list_databases returns an iterator (#4442)
CatalogConnection.list_databases should return an iterator so that users don't have to manually re-implement pagination. For those who do want more control over pagination, add num_page_results and page_token to the Iterator object, and also allow the iterator object to be created at a specific page token. |
||
 lancedb-gatefixer[bot]
|
3e0a1a8780 |
fix: reject zero vector search probes across SDKs (#4418)
`nprobes(0)` bypassed the validation used by the minimum and maximum probe setters, allowing zero into query requests and producing inconsistent local and remote results. Sync Python could additionally interpret the zero maximum as an unbounded search. Reject zero in the Rust vector-query builder and propagate its input error through the Python vector/hybrid and Node bindings. Validate nonpositive counts in the sync Python vector and hybrid builders so every SDK and backend reports `nprobes must be greater than 0` before execution. Update the Rust examples and document the positive-count requirement. Regression tests cover local and remote vector/hybrid builders, rejection before remote query requests, promised TypeScript query vectors, and positive counts replacing both probe bounds. Validation: - Rust probe regression tests: 2 passed. - Focused Python query, remote, hybrid, and serialization tests: 22 passed. The four sync regression cases fail with the original setters. - Focused TypeScript indexed local and mocked remote tests: 2 passed. - Workspace `cargo check --features remote --tests --examples` and Clippy passed. - Rust formatting, Python formatting/lint, and TypeScript build/lint/docs passed. Repository-wide `ruff check .` reports 29 existing errors in unchanged CI scripts; `ruff check python` passes. Fixes #4388 BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>` instead of `Self`; callers must handle the result, as with the existing minimum/maximum probe setters. <!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
7e979c1416 |
fix(node): preserve schemas for empty query results (#4423)
Empty local TypeScript queries returned a zero-row Arrow table with no columns because `toArrow()` built the table solely from record batches. Local streams can yield no batches, while remote streams often supply an empty batch carrying the schema. Expose the executed stream's schema through the native iterator and decode it into an empty Arrow table when no batches arrive. This preserves projected and search columns, types, nullability, and metadata without executing another query. Document the empty-result behavior. Validation: - Eleven regression cases failed with the original `toArrow()` implementation; all twelve now pass, including remote schema-only and empty-batch responses. - All 1,075 tests in the query, remote, Arrow, and table suites pass. - Native build, TypeScript compilation, Rust formatting and Clippy (`-D warnings`), TypeScript lint/formatting, and documentation generation pass. Fixes #4386 <!-- lance-gatekeeper-fix:v1 agent=7d589da1f7db5c7383f9600490e111a9 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 
|
2734e10767 |
fix(node): update arrow to 21 (#4362)
Thanks @vincentkoc for starting this! I was just looking into this myself. I started with your PR and added: - make arrow 21 the default - widen the peer range to <22 so if they release 21.3 that'll be included - have to do skipLibCheck in tsconfig.json, because of flatbuffers 25 which needs TypeScript 5.7 - fix override in Symbol.hasInstance - support new types in Arrow 20-21 (LargeList, Utf8View, BinaryView, IntervalMonthDayNano) - arrow-compat job in nodejs.yml that runs tests with arrow 19 and 20 (adds 2 sec) - put the list of arrow types in one place - update NODEJS_THIRD_PARTY_LICENSES.md - regenerated typescript docs (sorry about reorderings, one time cost) - reformat arrow.test.ts (sorry, almost all whitespace, thanks to biome. git diff -w shows 110 lines edited) - fixed Sliced Utf8View / BinaryView serialization error, by just converting them to Utf8 and Binary (as our rust code would anyway) --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
 Bilal Atique
|
0a1c1a04c7 |
feat(node): accept an AbortSignal to cancel blob reads (#4350)
Closes #4288. Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange` and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When the signal aborts, the call rejects with `signal.reason` (an `AbortError` unless the caller gave a reason) and the native future is dropped, so the read stops instead of running to completion. A signal that is already aborted rejects before any read starts. How it works: - On the Rust side, these five methods are now sync `#[napi]` methods that return `env.spawn_future(...)`. The future is wrapped in `futures::future::abortable`, and the abort handle is registered with napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so it can't be an argument of an `async fn`, which is why the methods changed shape. The TypeScript signatures are unchanged apart from the new optional argument. - napi-rs installs its own `onabort` on the signal it receives, which would overwrite a handler the caller set. So the TypeScript wrapper passes a private `AbortController` signal to the native call for each request, forwards the caller's abort to it with `addEventListener(..., { once: true })`, and removes the listener when the call settles. Tests in `table.test.ts`: - an already aborted signal rejects on all five methods, and the handle still reads afterwards - a custom abort reason is what the call rejects with - a read with a signal that never aborts returns normally and leaves no `abort` listener on the signal - aborting a 32 MiB `readRange` right after it starts rejects with `AbortError`, and the same handle reads again afterwards The Node docs under `docs/src/js` are regenerated. I didn't add a test for where the `read()` cursor ends up after a mid-flight abort on a local table. The remote `BlobFile` documents that a cancelled read leaves the cursor unchanged, but I haven't confirmed the same for the native Lance handle, so the docs don't claim it. |
||
|
Bilal Atique
|
4e3ec7cc27 |
feat(node): expose seek, tell, bounded read, readRanges and close on BlobFile (#4348)
Closes #4300. The TypeScript `BlobFile` only had `size()`, `read()` and `readRange()`, while the Rust `BlobFile` behind it also has `seek`, `tell`, `read_up_to`, `read_ranges` and `close`. This exposes those on the Node binding: - `read(maxBytes?)` reads at most `maxBytes` from the cursor (Rust `read_up_to`). Without the argument it reads to the end, as before. - `readRanges([{ start, end }, ...])` reads several half-open ranges and returns one buffer per range, in order. Like `readRange`, it does not move the cursor. - `seek(position)` and `tell()` move and report the cursor. - `close()` releases the handle, and `isClosed()` reports it. Every read, `seek` and `tell` after `close()` fails with "blob file is already closed". A second `close()` does nothing. The binding keeps its own `closed` flag, the same way the Python `BlobFile` does. Offsets and lengths are `bigint`, like `readRange` and `size()`. `BlobRange` is exported as a type. The new tests in `table.test.ts` cover chunked reads with `read(maxBytes)`, `seek` and `tell`, a negative seek position, `readRanges` order and cursor, a `readRanges` end past the blob size, and reads after `close()`. The Node docs under `docs/src/js` are regenerated with `pnpm run docs`. I left the `ReadableStream`/async iterator idea from the issue out of this PR. The AbortSignal work in #4288 touches the same methods, so it can follow on top of this. |
||
|
lancedb-gatefixer[bot]
|
8e3606fbe3 |
fix: handle remote blob fetch request caps (#4309)
## Cause Remote `fetch_blobs` sent every row ID in one Cloud request, exceeding the server's 1024-row or 64 MiB byte cap. A single blob over the byte cap could not be read through this method. ## Fix - Send at most 1024 row IDs per request and split batches further when the server reports the blob-byte cap, preserving input order and nulls. - Resolve one exact dataset version before multi-request reads. If an initial single request hits the byte cap, resolve the version before any successful split or Range read; use that version for every resulting request. - Read an individual oversized blob through the existing Range route. - Document the per-request caps and transparent handling in the Rust, Python, and TypeScript APIs and generated JS docs. ## Validation - Focused Rust `fetch_blobs` tests: 26 passed, including row-count, byte-cap, Range fallback, and exact-version assertions. - Rust clippy and formatting passed. - Focused Python blob tests: 13 passed; Ruff lint and formatting passed. - Node build, lint, docs generation, and focused `fetchBlobs` test passed. Fixes #4275 <!-- lance-gatekeeper-fix:v1 agent=6ae94d49bf1d4cae057bba72584e7b90 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Wyatt Alt
|
604b1b583b |
feat: pause and resume jobs from the Python SDK (#4112)
Cancellation is the only job control the SDK exposes, and it is terminal, so a long-running server-side job cannot be parked and picked up again. This adds `pause_job` and `resume_job` to the Rust core connection and the Python bindings (sync and async), posting to the server's `/v1/jobs/pause` and `/v1/jobs/resume` endpoints. A pause parks the job until it is resumed: its workers drain and stop. The outcome strings mirror the server's answers — a job finalizing its results reports `"committing"` and cannot be parked, and a resume before the drain is confirmed reports `"still_pausing"`; both are retried rather than failed. Resuming re-queues the job and its workers pick their work back up from checkpoints. Local connections report the operations as unsupported, like the rest of the jobs API. |
||
|
Jack Ye
|
e4d0f9e2da |
feat: return a cleanup job from drop_view (#4262)
Dropping a view unbinds its name and leaves the definition dataset to a server-side cleanup job, so the two are separate events a caller may want to wait on. `drop_view_async` returns that job — the same shape `drop_materialized_view_async` and `drop_function_async` already use: a `202` carries the job id, a `200` (nothing was bound to the name) yields an already-finished job with no id, and any other success status is an error rather than a silent no-op. ## `drop_view` waits `drop_view` now awaits the job before returning, so a caller who does not want to think about cleanup gets the stronger guarantee: when it returns, the definition really is deleted. That is deliberately **different** from `drop_materialized_view` and `drop_function`, which return as soon as the name is unbound and document that content may still be deleting. The view API is the newer one, and waiting is the semantic worth having; the other two are left alone rather than changing behaviour already released. ## Surfaces `Database` trait, the remote client, `Connection`, and the Python and Node bindings — matching where `drop_materialized_view_async` is already exposed. Four client tests cover the accepted case reporting its job id, the nothing-bound case reporting a finished job, a `202` without a usable `job_id`, and an unexpected success status. |
||
|
Jack Ye
|
8541d6d9df |
feat: add view CRUD APIs (#4236)
A view is a named query a database stores and plans on every read. It
holds no rows, which is the whole difference from a materialized view.
## API
| Verb | Route |
| --- | --- |
| `create_view(name, query, namespace_path)` | `POST
/v1/view/{id}/create` |
| `describe_view(name, namespace_path)` | `POST /v1/view/{id}/describe`
|
| `drop_view(name, namespace_path)` | `POST /v1/view/{id}/drop` |
| `list_views(namespace_path)` | `GET /v1/namespace/{id}/view/list` |
On `Connection` and the `Database` trait, with the remote client, Python
(sync and async) and Node bindings. Local databases return
`NotSupported`: the server side is Sophon's, where a view is an object
of the database manifest.
`ViewDescription` carries the defining query, the database *and
namespace path* unqualified names in it resolve against, and the schema
the query resolved to. `create_view` returns one, so a caller has the
schema without a second call.
Both defaults travel with the view because it outlives the session that
declared it: the server re-plans the stored query on every read, so a
reader resolving an unqualified name against its own defaults would read
a different table. `default_namespace_path` crosses the wire as
`default_namespace`, a path like `namespace`, absent for the root.
There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create, each authorized against what it
actually touches.
Querying a view stays SQL's job. There are no rows behind a view, so
there is no `open_view` returning a `Table`.
|
||
|
Wyatt Alt
|
5231d37f8d |
feat: list a table's per-row Function errors from the client (#4208)
A refresh running under a skip policy records each row it skipped, with the failing input and the error, but the client could not read that store: the server exposes it over SQL and, since recently, a REST route. A user who hit per-row failures still had to open a SQL session. `Table::function_errors` calls the route. The listing is table-addressed with optional job and column filters, the same addressing the SQL surface uses, so the two cannot disagree about what a table's errors are. The two non-record signals come back as their own fields rather than as rows: capped-fragment summaries, and whether the listing stopped at its limit. Local tables refuse rather than answer with an empty list. Python and Node expose the same call, with the same optional filters. |
||
 Jack YeandXuanwo
|
6a07f88980 |
feat: add remote catalogs and Python and TypeScript bindings (#4195)
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
Jack Ye
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
|
Jack YeandXuanwo
|
2f88b71c21 |
feat: add persistent OAuth token cache and session APIs (#4182)
Stacked on #4173 (diff includes it until that merges; will rebase after). Addresses the token-cache part of [Colin's review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100). Adds an explicit, opt-in persistent OAuth token cache shared by Rust, Python, and Node clients, plus `login` / `status` / `logout` session APIs, so short-lived processes (CLIs, scripts, notebooks) reuse one session instead of restarting a browser or device flow on every start. - **Opt-in and minimal**: existing callers stay memory-only and lazy. Only refresh tokens are persisted (never access tokens, never client secrets), so there are no local token-expiry decisions to get wrong when clocks move. Each process start performs one silent refresh grant. - **Hardened file backend**: private directory (`0700`), per-record files (`0600`), owner validation, symlink rejection, and atomic `rename` replacement. Corrupt, truncated, unknown-version, or permission-invalid records fail with actionable errors naming the file. Native keyring backends were evaluated (keyring crate routes Linux through D-Bus/zbus: heavy deps, headless/CI flakiness) and are deferred; the file store is the explicit opt-in, not a downgrade from a keyring. - **Cache key**: SHA-256 of the canonical identity (issuer, client ID, sorted/de-duplicated scopes, flow, public/confidential), so no secret appears in a filename and distinct identities never collide. Versioned record schema (`version: 1`). One record per identity: last login wins, documented. - **Cross-process rotation locking**: per-key `fs4` file lock (`flock` / `LockFileEx`) around the refresh critical section — acquire, reread the durable record, refresh exactly once, atomically store the rotated refresh token, release. The OS releases locks on process death, so crashes cannot strand stale locks. Only confirmed `invalid_grant`/`invalid_token` deletes a record and reauthenticates; transport, 5xx, 429, and parse failures retain it. - **Session APIs**: `OAuthSession::login/status/logout` in Rust, `lancedb.remote.OAuthSession` (async) in Python, `OAuthSession` class in Node. `status` returns non-secret metadata only. `logout` removes only the local credential — provider revocation (RFC 7009) is a deliberate follow-up, and local logout never terminates browser SSO. Azure managed identity is rejected for persistence (machine identity stays in memory); client credentials have nothing refreshable to persist and stay memory-only. - No CLI binary exists in this repo, so this ships library APIs plus doc examples in all three languages. Tests: Rust unit + mock-IdP integration (cache-key canonicalization/separation, record versioning/corruption/truncation/symlink/owner/perms, lock serialization + release, two concurrent providers proving no `invalid_grant` and correct rotation, transient-failure retention, `invalid_grant` delete + reauthenticate, login/status/logout lifecycle, client-credentials no-op, IMDS rejection, secret redaction); Python lifecycle + a true two-subprocess cross-process reuse test (second process refreshes once, never hits the device endpoint); Node lifecycle + device-flow login test. Local builds were skipped in development; CI validates all bindings. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Wyatt Alt
|
0665575a76 |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
 Drew
|
6702e3fec1 |
feat(node): add blob v2 fetch and field helpers (#4155)
this PR blob v2 field helpers and reads to the Node SDK.
`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.
Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.
Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.
```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
"videos",
[{ id: 1n, video }],
{
schema: new Schema([
new Field("id", new Int64()),
blob("video"),
]),
},
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);
```
### Testing
- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
|
||
 Will JonesandClaude Sonnet 5
|
1da5876870 |
ci: add spell checking (#4148)
Adds [typos](https://github.com/crate-ci/typos) as a CI check and pre-commit hook, the same way Lance does it, so misspellings like the ones fixed in #4146 get caught automatically going forward. This also fixes the misspellings `typos` found across the repo (Rust, Python, TypeScript source, comments, and generated docs), and adds a small `.typos.toml` with `extend-words` entries for terms that are correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ` (a real quantization algorithm name), `mmaped` (the actual name of a `candle-core` API we call), and `Writeable` (from Python's `_typeshed.WriteableBuffer`). Third-party license files are excluded. Fixes #4147 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack Ye
|
21f11b4463 |
feat!: replace get_job/job_history with describe_job/query_job_events (#4130)
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.
## Before
Four ways to ask about a job, none of which told you much.
```python
job = table.refresh_column_async("embedding")
job.status() # "running". That was the entire debug surface.
db.get_job(job_id) # state, and a spec. No result, no progress.
db.job_history(job_id) # raw record batches, no limit, no filter
db.job(job_id) # a handle that knew nothing
```
## After
Open a job the way you open a table; the handle answers everything.
```python
job = db.open_job(job_id) # raises JobNotFoundError if there is no such job
```
```python
>>> print(job)
Job(
id='job-1',
state='failed',
job_type='refresh_column',
creation_ms=1757000000000,
spec={
"column": "embedding",
"num_workers": 4
},
failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```
Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.
Per-fragment progress *while it is still running*:
```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist() # [5000, 5000, ...]
```
The handle an async action returns is the same object, one `refresh()`
away:
```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```
TypeScript is the same experience, down to `console.log`:
```ts
const job = await db.openJob(jobId); // rejects if there is no such job
console.log(job); // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```
## Why each piece matters
- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.
`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.
## Breaking
All shipped in 0.38.0. No deprecated aliases.
| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |
Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
|
||
|
Will JonesandClaude Opus 5
|
d2ca0ce0ab |
feat: accept multiple on columns for merge insert on remote tables (#4102)
Merge insert has always taken a list of columns to match on, and local tables have always joined on all of them. Remote tables did not: any list longer than one was rejected with `MergeInsertBuilder only supports a single 'on' column`, so a composite-key upsert was impossible against LanceDB Cloud and Enterprise from Rust, Python or TypeScript. The remote request now carries `on` as a list and sends it as one repeated query parameter per column — `?on=shard_key&on=id`. That is how the lance-namespace spec encodes an array-valued `on`, so the server receives a composite key in the shape it expects. A single column still serializes to `?on=id`, exactly what clients sent before, so existing callers are unaffected. A column repeated within `on` is now rejected client-side rather than sent for the server to reject with a 400. No binding changes were needed: `Table.merge_insert` in Python and `Table.mergeInsert` in TypeScript already accepted a list, it just could not reach a remote table. Both gain a test for composite keys, and the doc comments now say what passing several columns means. Part of [ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the). ## Example ```python table.merge_insert(["shard_key", "id"]) \ .when_matched_update_all() \ .when_not_matched_insert_all() \ .execute(new_data) ``` A row whose `id` matches an existing row but whose `shard_key` differs is an insert, not an update. ## Not included Java. Java callers reach merge insert through `org.lance.namespace.LanceNamespace`, whose `MergeInsertIntoTableRequest.on` is a single string until lance-namespace 0.12 ([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363), [lance#8915](https://github.com/lance-format/lance/pull/8915)). There is nothing in this repo's Java SDK to change until the `lance-core` pin can move. Sending more than one column requires a server that accepts the repeated parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571)); an older server returns a 400 rather than silently merging on one column. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Daniel RammerandClaude Opus 5
|
ab74aa620c |
refactor(lsm): rename LSM stats to SSTable and table shard
Aligns the LSM stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an SSTable. Two terms in this API predate that pass. **Generation -> SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket -> table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding transform, so it was wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `LsmStats.buckets` | `LsmStats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Applied across Rust, Python, TypeScript, and Java, including the `get_lsm_stats` JSON field names. `LsmWriteSpec::Bucket` and `num_buckets` are unchanged — those name the sharding transform, not the shard. ## Compatibility Breaking for the MemWAL LSM stats API, which is experimental and paired with a server that renames the same fields. The JSON keys `table_shards` and `sstables` must roll out together with the WAL server change. ## Validation - `cargo check -p lancedb --all-features`, `cargo fmt --all` - `ruff format --check` and `ruff check` on the touched Python - `biome check` on the touched TypeScript The typedoc markdown under `docs/src/js` was updated by hand, not regenerated: `npm run docs` needs the napi-built `./native` types. Worth running `npm run docs` on this branch to confirm the generator agrees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgVf5C2yRbwK6pe1aMYkmg |
||
|
Dan Tasse
|
8b7e13b0c6 |
docs: add comments about metadata conventions (#4054)
In LanceDB Enterprise, we've adopted these conventions to give some "canonical" metadata paths. This lets us display them in a certain way in the UI or let agents standardize on them, to assume they'll find info in a certain place. This PR (only comments/docs) just documents those choices. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
35b5d015ac |
fix(node): preserve embedding registration in server bundles (#3806)
## Summary
- lazily initialize built-in OpenAI and Hugging Face providers when
consumers call the public embedding registry API
- choose automatic vector versus FTS search from embedding metadata on a
fresh pinned table revision for every execution
- expose automatic string searches as an `AutoQuery` with only
operations common to both native query families
- keep the registry shared and built-in registration safe across
duplicated module graphs
## Root cause
Nitro treats dependency modules as side-effect-free and removes the bare
OpenAI provider import from its generated route. Registration therefore
never runs, so `getRegistry().get("openai")` remains undefined even when
the registry itself is shared globally. Bundlers may also duplicate the
provider and registry module graphs.
The public embedding entry point now initializes built-in providers only
when `getRegistry()` is explicitly called, keeping initialization on a
live path that Nitro retains. Each terminal automatic-search execution
pins the exact table revision visible at dispatch, reads embedding
metadata and computes an embedding from that snapshot, replays the
builder operations, and constructs and executes the selected native
query against the same snapshot. Pinned native snapshots execute locally
when namespace pushdown cannot carry their revision, while remote
snapshots are seeded directly from one version-and-schema response. The
public `AutoQuery` builder exposes only the operations shared by FTS and
vector search, so runtime class narrowing cannot expose invalid
vector-only methods. Repeated built-in registration replaces stale
constructors from duplicated module graphs while public `register()`
retains its duplicate-alias error.
## Validation
- `cargo fmt --all`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test --runInBand` (783 passed, 5 skipped)
- serial examples suite with a local OpenAI mock (11 passed), including
`sentence-transformers.test.ts`
- packaged Nitro 2.13.4 server route using the reported imports returned
`{"registered":true}`
- fresh-process FTS fixture initialized both public built-ins and
confirmed automatic string search still returned the indexed row
- schema-consistency regressions cover read-consistency refresh,
checkout, checkoutLatest, restore, runtime class narrowing, concurrent
overwrite during embedding computation, and reused automatic-search
builders
- focused regressions confirm pinned native snapshots bypass unversioned
namespace pushdown and remote snapshots use one describe request
Fixes #2429
<!-- lance-gatekeeper-fix:v1 agent=2adf0f21b8bfb634606ed8897a849e30
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
Will JonesandClaude Opus 5
|
fce45ba9fc |
feat(nodejs): add listTables, deprecate tableNames (#4041)
`table_names` is being replaced by `list_tables` across the SDKs, but TypeScript only had `tableNames`. This PR adds `listTables`, which returns a page of table names together with the token that resumes after it, and marks `tableNames` and `TableNamesOptions` deprecated in favor of it. It binds the `Connection::list_tables` that already exists, so nothing in the Rust API changes and nothing existing breaks. `pageToken` is documented as opaque rather than as a table name, since what resumes a listing is the database's to decide — that keeps callers off a detail that is going to change. Stacked on #4040, which fixes a table being dropped at every page boundary. The page-walking test here needs that fix to pass. Review the last commit only until #4040 lands. ## Example ```ts const names = []; let pageToken = undefined; do { const page = await conn.listTables({ pageToken, limit: 100 }); names.push(...page.tables); pageToken = page.pageToken; } while (pageToken); ``` A namespace can be listed by passing its path first, mirroring `tableNames`: ```ts const page = await conn.listTables(["analytics"], { limit: 100 }); ``` Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Wyatt Alt
|
68749ecfa3 |
feat(nodejs): materialized view bindings (#3935)
Exposes materialized views to TypeScript: createMaterializedView,
openMaterializedView and listMaterializedViews on Connection, and a
MaterializedView handle carrying the parsed definition and
refresh({full, sourceVersion}), which returns the typed refresh result.
select accepts column names, [alias, expression] pairs, or a record of
the
same; the definition reads back off the stored schema, so a reopened
handle
needs no side channel. Remote connections surface the core's
not-supported
error up front.
The napi crate needed the same recursion-limit raise as the core crate:
the
refresh future's type graph overflows the default trait-recursion depth.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Drew Gallardo
|
e98d8ac685 |
feat!: rename branch merge to cherry_pick (#3986)
This PR is a **breaking** rename of #3686. merge reads like git merge w/ three-way, replay history, combine two lines of work. That is not this API. This call takes one additive change on a branch and lands it on main. New column, including a blob column. Main's existing columns are not rewritten. If it cannot land, you get `status="failed"` and `diff.errors`, not a merge conflict to resolve. Cherry-pick is terminology that aligns more with that. ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.cherry_pick("exp", dry_run=True) result = table.branches.cherry_pick("exp") if result["status"] == "cherryPicked": print("landed at", result["mainVersionAfter"]) elif result["status"] == "failed": print(result["diff"]["errors"]) ``` ### Behavior - Remote / Enterprise only. Local still NotSupported. - HTTP 409 is not an exception. It is Ok with status="failed" and diff.errors (CherryPickError). - Unknown error / status codes still parse as Unknown. - Requests are not retried. 409 is final and carries the body. - Endpoint is POST /v1/table/{id}/branches/cherry_pick/. - merge_insert and Table.merge are unchanged. ### Testing - `cargo test -p lancedb --features remote diff_branch` - `cargo test -p lancedb --features remote cherry_pick` - `pytest python/python/tests/test_remote_db.py -k cherry_pick` - node `remote.test.ts` diffs / cherry-picks path |
||
 Dan RammerandClaude Opus 5
|
f1c4967eeb |
feat: bring the MemWAL LSM surface to parity across the SDKs (#3962)
## Why Four of the eight LSM methods are **remote-only in the core**. `impl BaseTable for NativeTable` implements only `set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`, `compact_lsm` and `get_lsm_stats` fall through to trait defaults returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and `checkpoint_lsm` is built on all three. That explains the state of the bindings: Node had bound the four that work against a local table and stopped, so a Cloud user could install an LSM write spec but had no way to observe fresh-tier state or drive a checkpoint. Java had none of it at all. | SDK | set/unset/get spec | closeWriters | flush | compact | getStats | checkpoint | |---|---|---|---|---|---|---| | Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Node *(before)* | ✅ | ✅ | — | — | — | — | | **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** | | Java *(before)* | — | — | — | — | — | — | | **Java (after)** | **new** | n/a | **new** | **new** | **new** | **new** | Go and C are separate repos and are out of scope here. `closeLsmWriters` drains cached in-process shard writers, so it has no meaning for Java, which is a pure REST client. ## Node Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and `getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats` / `MemtableStats` objects — typed rather than a JSON blob, matching the existing `LsmWriteSpec` object in the same file, with `u64` cast to `i64` per that file's convention. Because these four are remote-only, the new tests assert each binding reaches the core and surfaces `NotSupported` against a local table. That covers the wiring; behavior against a real endpoint stays covered by the mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`. ## Python No new methods. All eight are on `LanceTable`, `AsyncTable` and `RemoteTable` — the last four landed on the sync `RemoteTable` in #3961, which is merged into this branch. What was missing here was reachability. `LsmWriteSpec` was importable only from the private `lancedb._lancedb`, appearing in `table.py` solely under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no mention of it, which per the repo's docs guidance means it rendered nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in `__all__`, and documented. ## Java Java reaches LanceDB purely over REST through the generated Lance Namespace client, and these routes are not in that spec, so they are issued through a small dedicated client rather than added to the spec. That call is revisitable — LSM is one of four unspecified route families alongside `multipart_write`, `page_cache/prewarm` and `branches/diff|merge`. If those are ever regularized into the spec as a group, `LanceDbTableLsm` is one file that gets deleted. `LsmWriteSpec` here is deliberately **not** `org.lance.memwal.InitializeMemWalParams`. That type defaults to maintaining *no* indexes where a spec here defaults to maintaining *every* index, and it cannot express the `null` that asks the server to resolve the set: | Value | On the wire | Meaning | |---|---|---| | unset (null) | `null` | Server resolves **every** maintainable index | | `Collections.emptyList()` | `[]` | Maintain **none** | | `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those | A dedicated test pins null and `[]` as distinct on the wire, since collapsing them is the failure mode that motivated a LanceDB-owned type. `checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs` with its constants and status semantics intact: 429/503 retried in place against an 8-budget, 421 restarting from flush against a 3-budget, 5s poll, and a target watermark fixed after the seal so it terminates under write load. `getLsmStats` returns typed `LsmStats` / `BucketStats` / `GenerationStats` / `MemtableStats`, mirroring the Rust structs in `rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes. Decoding is strict — see below. ## Review feedback Both gatekeeper findings were real. Each was reproduced against the scripted test server first, and each fix ships with the reproducer as a regression test. **The transport was doubling every checkpoint retry budget.** `HttpClients.createDefault()` installs Apache's default response retry strategy, whose retryable-status list is exactly 429 and 503 — the two statuses `isRetryable` owns. A 429 held against `flush_lsm` issued **18** wire requests where the loop intends 9, and `compact_lsm` was retried in place despite the loop being built to fall through to a fresh stats poll instead. Timing confirmed the mechanism: that run took 25.4s ≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry interval. Automatic retries are now disabled, so the checkpoint loop is the sole owner of the 421/429/503 transitions. A side effect worth noting: `testCheckpointRetriesRetryableStatusInPlace` was passing on a transport-absorbed 429 and never reaching `issue()`'s retry branch at all. It now exercises the real path. **Stats decoding failed open.** `getLsmStats` read the response with Jackson's `path()`, which yields a missing node that iterates as an empty array — making "malformed" indistinguishable from "no buckets", which is indistinguishable from "drained". Four separate payloads made `checkpointLsm()` report convergence for a checkpoint that never ran: | Response | Before | Now | |---|---|---| | `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ | | `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` | | empty response body | **reported success** | `IllegalStateException` | | bucket missing required fields | **reported success** | `IllegalStateException` | The empty-body row is the one to weight: a proxy 200 with no body is a realistic production event, and it silently reported a checkpoint that never happened. Decoding is now strict and fails closed, matching the serde contract on the Rust side exactly. One deliberate deviation from the review comment, which asked that *only* explicit JSON `null` count as disabled: Rust has `#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to `None` there too. Java now matches that. It is an absent-or-malformed **`buckets`** that fails closed, which is the case the comment was actually protecting. ## Testing - Java: **33 passing** (8 existing + 25 LSM) against a scripted `com.sun.net.httpserver.HttpServer` — no new test dependency. Wire assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`; checkpoint tests cover convergence, not piling onto a latched bucket, 421 restart-from-flush, 429 retry-in-place, terminal-status propagation, reissue exhaustion, the exact wire-request count against the retry budget, and five malformed stats payloads. - Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm run tsc`, `npm run lint`, `npm run docs` all clean. - Python: `ruff format --check` and `ruff check` clean. - Java formatting: `./mvnw -pl lancedb-core spotless:apply` and `spotless:check` both clean under a JDK 11 toolchain. ## Note: spotless needs a pre-16 JDK `./mvnw spotless:apply` fails on JDK 16+ with `JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7, pinned at `java/pom.xml:34`, predates JDK 16's compiler API change. **This is pre-existing** and reproduces on a pristine `main` checkout. It is not a blocker, just a toolchain requirement. Spotless was run against these sources under JDK 11 and both `spotless:apply` and `spotless:check` pass on the whole module: ```shell JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply ``` Bumping the plugin so it works on modern JDKs is still worth doing, but separately from this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Wyatt Alt
|
928c3dde2d |
feat: computed columns on remote tables (#3941)
LanceDB Cloud and Enterprise support computed columns through the REST
API,
so declaration dispatches per backend: local tables plan the expression
themselves, remote ones send {name, computed} entries for the server to
plan. A remote refresh is the server's backfill job --
refresh_column_async
submits it and returns a handle whose successful wait establishes a
read-freshness baseline on the submitting handle, unless a checkout has
pinned the handle by the time the job completes; the blocking form
refuses
rather than invent a fill count the server does not report.
Declaration entries are built from the namespace client's
AddColumnsEntry
model (lance-namespace 0.11.0, via the lance beta.13 pin), so the
payload
shape is compile-checked against the published contract.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Wyatt Alt
|
c429863122 |
feat: refresh_column_async returns a job handle (#3939)
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Wyatt Alt
|
fc0d917d32 |
feat: refresh computed columns (#3938)
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Wyatt Alt
|
def869bb78 |
feat: declare computed columns by SQL expression (#3937)
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.
The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Jack Ye
|
ffd35c1a8f |
feat: add asynchronous drop table API (#3936)
## Summary - add `drop_table_async` and return a job handle while preserving `drop_table` - consume remote 202 responses with cleanup job IDs and retain older-server compatibility - expose the API through Python and TypeScript connection wrappers |
||
|
Dan RammerandClaude Opus 5
|
706a9c327f |
feat: infer maintained indexes when an LsmWriteSpec omits them (#3748)
## What `LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`: | value | meaning | |---|---| | `None` (new default) | every index the MemWAL supports, resolved when the spec is installed | | `Some([])` | maintain nothing — a scan/filter-only WAL table | | `Some([..])` | exactly these, taken verbatim | `with_maintained_indexes` keeps its signature; `with_no_maintained_indexes()` is new. Surfaced through the remote path (null on the wire), Python, and Node. ## Why Callers had to state the maintained set by hand every time, which is both tedious and easy to get wrong — the common case is "maintain what I already built." Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating to lance's `is_maintainable_index_type`. This is load-bearing rather than cosmetic: lance does **not** skip an index type its memtable cannot build, it errors when the shard writer opens, so sweeping up a bitmap index would fail every memtable claim and leave the table unwritable. The inferred set excludes those, and an explicit list naming one is now rejected at spec time instead of at claim time. ## Behavior change A freshly constructed spec used to maintain **nothing**; it now maintains **everything supported**. This flipped because napi collapses `undefined` and `null` to `None`, so TypeScript cannot express "absent means nothing, null means all" — any other choice makes the bindings disagree with the wire. The error direction also favors it: an unwanted maintained index costs memory, while a silently unmaintained one degrades FTS to an unscored scan. Three existing tests encoded the old default and are updated rather than worked around. ## Caveat The resolved set is a snapshot, not a subscription. An index created after the spec is installed is not maintained until the spec is unset and set again. `get_lsm_write_spec` therefore always reports a concrete list — `None` never round-trips. ## Dependency Needs a lance release carrying `is_maintainable_index_type` (lance-format/lance#8095) before this builds against the pinned tag. Draft until then. ## Testing 38 Rust LSM tests and 10 Python tests pass against a local lance build, including new coverage that a bitmap index is excluded from inference and rejected when named, and that `[]` stays distinguishable from null on the wire. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Wyatt AltandClaude Fable 5
|
e3b472c212 |
feat: connection-level job operations (#3755)
Adds job operations to the connection surface, building on the Job handle from #3742: job(id), list_jobs, get_job, cancel_job, and job_history, plus a non-blocking Job.status(). Implemented on the Database trait (defaulting to NotSupported), the remote backend (/v1/jobs), and the Python and Node bindings; job_history returns Arrow batches. errors() and progress() are not included. Tested with mocked endpoints in all three languages. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Wyatt AltandClaude Fable 5
|
a6418b6cb9 |
feat: create_index returns a Job handle (#3742)
IndexBuilder::execute now returns a Job with wait and cancel methods. Local tables build the index synchronously and return an already-done job. Remote tables read the job id the server returns from create_index and track it through the /v1/jobs API: wait polls describe until the job reaches a terminal state and cancel posts a cancellation. Servers that return no job id yield a done job, so behavior against older servers is unchanged. The job id is not exposed on the handle. The Python and TypeScript bindings keep their current signatures and discard the handle; exposing Job there is left to follow-ups. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
 Heng Ge
|
f655f62e09 |
feat(query): add use_lsm to read MemWAL LSM data (#3489)
## What
MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).
## How
- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
- **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.
Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
|
||
  Drew GallardoandCursor
|
65cd142c7e |
feat: add remote branch diff and merge client APIs (#3686)
This PR adds some support for `diff` / `merge` in the remote client as for local tables we stay `NotSupported` until https://github.com/lance-format/lance/issues/7263. This wires the two review-and-land calls against the remote REST API: - `POST /v1/table/{id}/branches/diff` - `POST /v1/table/{id}/branches/merge` Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python returns the wire JSON, same shape as the REST response. Merge here means promoting a branch's added columns onto `main`. ### Behavior - Remote only. Local raises `NotSupported`. - A rejected merge is not an exception. HTTP 409 still returns `Ok` / a dict with `status="rejected"` and blockers in `diff.mergeBlockers`. - Unknown blocker / status codes parse as `Unknown` so a newer server does not break older clients. - `MergePreview` tolerates missing fields for the same reason. - Merge requests are not retried. 409 is final and carries the body you need. ### Example ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.merge("exp", dry_run=True) result = table.branches.merge("exp", dry_run=False) if result["status"] == "merged": print("landed at", result["mainVersionAfter"]) elif result["status"] == "rejected": print(result["diff"]["mergeBlockers"]) ``` ### Testing cargo test -p lancedb --features remote diff_branch cargo test -p lancedb --features remote merge_branch --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Jack Ye
|
37032151d3 |
feat: support distributed analyze plan metrics in clients (#3675)
Adds client-side support for analyze_plan distributed metrics modes across Rust, Python, and TypeScript clients. Defaults to aggregate for backward compatibility and sends the remote distributed_metrics parameter only when a non-default mode is requested. |
||
|
Jack Ye
|
06b53c97d6 |
feat: add table FTS query tokenization (#3659)
## Summary - add table-level FTS query tokenization returning token text and position - use the native index tokenizer for local tables and remote index metadata for remote tables - expose sync and async Python table wrappers with focused coverage |
||
|
Dan RammerandClaude Opus 4.8
|
6c066530e5 |
feat: add get_lsm_write_spec to read the installed LSM write spec (#3631)
## Summary Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the read counterpart to the existing `set_lsm_write_spec` / `unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is not enabled; otherwise reconstructs the spec (mode, shard column, `num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly as installed. ## Changes - **Rust core (`NativeTable`)** — reconstructs the spec from `mem_wal_index_details()`, resolving the shard column from its Lance field id via the dataset schema. This is a raw metadata read, so it is unaffected by `describe_indices` system-index filtering. - **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index through `index/list` with `include_system: true` (so the curated `list_indices` surface stays unchanged), then parses the index `details` JSON. It matches the index by name and ignores `index_type`, so no client `IndexType` variant is needed. It uses the **server-resolved `column` name** from the details (Lance field ids do not travel to the remote client). - **Python + TypeScript bindings** — sync and async, mirroring `set`/`unset`, with round-trip tests (bucket / identity / unsharded, plus `None` when unset). ## Tests - Rust: native round-trip unit test + remote mock-endpoint tests (present + absent). All green (`cargo test --features remote -p lancedb`). - Python/TS: round-trip tests added; binding-runtime execution runs in CI. ## Dependencies for the remote path The remote path is complete on the client side but depends on two out-of-repo pieces to work end-to-end: 1. **lance** — emit the server-resolved shard **`column`** name in the MemWAL index `details` JSON (field ids can't reach the client). See lance-format/lance#7667. 2. **server** — honor `include_system` on `index/list` so the `__lance_mem_wal` entry is returned for this read. Against an older server (no `include_system`), the remote getter degrades gracefully to `Ok(None)` rather than erroring. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Will JonesandClaude Opus 4.8
|
d889321b5e |
fix!: combine repeated where filters with AND instead of replacing (#3585)
BREAKING CHANGE: When passing multiple where clauses to a query, they now stack instead of replacing the previous filter. Previously, calling `where`/`only_if` more than once on a query silently replaced the previous filter, so only the last filter was applied. This was surprising and could return rows that an earlier filter should have excluded. This implements the alternative suggested in https://github.com/lancedb/lancedb/pull/3514#issuecomment-4664901580: instead of rejecting a second filter, repeated filters are combined with a logical AND (`(previous) AND (new)`). The combination happens in the Rust core (`QueryBase::only_if` and `only_if_expr`), so it applies to all SDKs at once (Rust, Python async, and TypeScript). The Python sync query builder keeps its own filter state, so it combines filters in the binding layer as well. SQL string and expression filters are combined within their own representation. When the two representations are mixed, the expression is lowered to SQL (via `expr_to_sql_string`) and the filters are combined as SQL strings, so chaining `where` works regardless of which form each filter takes. Fixes #2649 ## Tests - Rust: `cargo test --features remote -p lancedb --lib query` - Python: `uv run --extra tests pytest python/tests/test_query.py` - TypeScript: `pnpm test __test__/query.test.ts` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 Brendan Clement
|
f76b075d13 |
feat: add table branch support to remote tables and Python/TS bindings (#3540)
### Description Adding branch support for RemoteTable by threading a branch selector onto every operation the data plane accepts it on. Exposes the currentBranch to nodejs and python through the bindings. Matching the server handlers, the branch rides as: - a `?branch=` query parameter for Arrow-body and query-only ops (insert, merge_insert, multipart_*, version/list, drop_index) - a `branch` field in the JSON body for everything else (count_rows, query, update, delete, create_index, column ops, index list/stats, stats, restore, describe, tags create/update) A main-branch handle (`branch == None`) produces byte-identical requests to before: no `branch` field and no `?branch=` - Handle-per-branch: `create_branch` / `checkout_branch` return a new handle with fresh caches and reset version/freshness state, mirroring `NativeTable`. - `create_branch` maps 409 to already-exists, 400 to invalid, and 404 to not-found with source context, and sends without retry so the 409 stays observable. - `Ref` translation covers version, version-number (relative to the handle's branch), and tag (resolved via the tags endpoint); `"main"` and empty normalize to the main branch. - Python branch handles persist their branch (and pinned version) across pickle/fork, so a forked or pickled handle reopens on its branch rather than silently reverting to main. ### Tests - Rust mock tests per op category (query-param and body mechanisms, branch CRUD, error paths, backward-compat). - Python sync branch CRUD, `open_table(branch=)`, and a pickle round-trip regression test. |
||
|
Jack Ye
|
8373318e89 |
feat: support FM-Index scalar index for substring search (#3532)
Adds an FM-Index — a scalar index over string and binary columns that
accelerates substring search (`contains(col, 'needle')`), distinct from
the tokenized `FTS` index — across the Rust core and the Python and
TypeScript bindings.
## Rust
- `Index::Fm(FmIndexBuilder)` and `IndexType::Fm`.
- `make_index_params` maps `Index::Fm` to Lance's
`ScalarIndexParams::for_builtin(BuiltinIndexType::Fm)`.
- `supported_fm_data_type` validates
`Utf8`/`LargeUtf8`/`Binary`/`LargeBinary` columns.
- `list_indices` round-trips the type (`"Fm"` → `IndexType::Fm`); the
remote wire type is `"FM"`.
## Python
Adds `lancedb.index.Fm`, accepted by `create_index`:
```python
from lancedb.index import Fm
await tbl.create_index("text", config=Fm())
```
## TypeScript
Adds the `Index.fm()` factory:
```ts
await tbl.createIndex("text", { config: Index.fm() });
```
|
||
|
Brendan Clement
|
d9018067b3 |
feat: support checking out a version on a branch (#3504)
### Description Stacked on #3490. Adds an optional version to branch checkout across the Rust core and the Python and TypeScript SDKs, so you can open a specific version on a branch ("version V of branch B"), not just the branch's latest version Rust ```rust // Open version 3 of branch "exp" (a read-only view): check out from an // existing table, or open it directly from the connection. let exp_v3 = table.checkout_branch("exp", Some(3)).await?; let exp_v3 = db.open_table("items").branch("exp").version(3).execute().await?; // checkout_latest re-attaches to the branch's writable HEAD. exp_v3.checkout_latest().await?; // With no branch, a version opens main at that version. let main_v3 = db.open_table("items").version(3).execute().await?; ``` Python ```python # Open version 3 of branch "exp" (a read-only view): check out from an # existing table, or open it directly from the connection. branch_v3 = await table.branches.checkout("exp", version=3) branch_v3 = await db.open_table("items", branch="exp", version=3) # checkout_latest re-attaches to the branch's writable HEAD. await branch_v3.checkout_latest() # With no branch, a version opens main at that version. main_v3 = await db.open_table("items", version=3) ``` TypeScript ```typescript // Open version 3 of branch "exp" (a read-only view): check out from an // existing table, or open it directly from the connection. const branchV3 = await (await table.branches()).checkout("exp", 3); const opened = await db.openTable("items", undefined, { branch: "exp", version: 3 }); // checkoutLatest re-attaches to the branch's writable HEAD. await branchV3.checkoutLatest(); // With no branch, a version opens main at that version. const mainV3 = await db.openTable("items", undefined, { version: 3 }); ``` ### Testing - Added unit tests (Rust, Python sync + async, TypeScript): branch-scoped resolution at a version number shared with `main` and with another branch, read-only enforcement on a pinned handle, `checkout_latest` recovery to the branch's HEAD, fork-point reads, and the nonexistent-version/branch error paths. - Ran smoke tests against the Python and TypeScript SDKs on local machine. |