mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
e5a5b67212ccc75ca7b69f3ac53de3dc045ffb34
50
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
725eee8a47 |
fix: prevent unsafe write retries and bound Blob uploads (#4977)
Ambiguous gateway failures can occur after a write commits, so replaying the operation can append duplicate rows. Non-idempotent writes now retry only known precommit failures. Multipart retries reuse the same upload ID, part ID, and encoded bytes; ambiguous completion returns an unknown-outcome error without abort or replay. A definite 409/429 completion rejection triggers best-effort cleanup using the call's retry budget. Replayable parts and the shared byte-based partition target default to 64 MiB. Dictionary-bearing batches are compacted before size estimation and again when actual compressed IPC framing requires further splitting: keys are remapped without changing their types or schema, view backing buffers are collected, and ListView/LargeListView child ranges are compacted as an interval union so overlapping views retain sharing. Parts are prepared on demand; a single row and its schema can still exceed the target. Multipart progress starts at zero and counts each acknowledged IPC body once. Rust write builders and Python/TypeScript `add` expose per-call retry overrides. Atomic `create_table` pre-encodes to a temporary file and rejects compressed IPC over 256 MiB before sending any request; the schema-first import guidance now includes Python, TypeScript, and Rust examples. The shared partition target also affects local automatic partitioning. At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases pass on source verified against the committed four-file patch. With a 16 KiB target and 128 unique 1 KiB dictionary values, the original ordinary dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 / 143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 / 149,632. Encoded HTTP bodies are decoded to check values, nulls, slices, metadata, changing dictionaries, and overlapping views; additional cases cover framing-driven splitting, oversized rows, stable retry bytes/IDs, acknowledged progress, and atomic create. Real HTTP fault tests commit Blob payloads to a local Lance table and verify byte equality, no duplicate append, and zero abort/replay after ambiguous completion. The earlier 811 remote and 7 progress cases at `f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote database/options results (437 cases plus 8 fixture consumers), Node retry-binding evidence, and Rust/TypeScript documentation example checks remain historical validation. Python/Node extensions were not rebuilt for the dictionary follow-up; these results do not establish current binding integration or fresh cross-platform/minimum-dependency CI acceptance. A separate real Query Node lifecycle experiment used SDK `b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon `58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener forced a connection closed, a surviving QN accepted a retry with identical upload ID, part ID, and IPC bytes. Ordinary insert and complete response-loss scenarios sent no replay or abort; fresh reads observed exactly one commit and exact Blob bytes in all three scenarios. This local experiment used a 1-second drain grace, two 32 KiB payloads, and one distinct staged part. It validates acknowledgement-loss recovery at those revisions, not concurrent overlapping attempts, large uploads, physical orphan cleanup, the later dictionary encoding change, or the original sixth-insert EOF. The part target is not a process memory limit: source/ancestor buffers and temporary compaction allocations may coexist. Struct/FixedSizeList parent nulls can retain hidden child keys; this is not complete logical-reachability GC for every Arrow combination, and retained hidden values may affect wire size. The current local Lance 15.0.0-beta.17 rejects ListView/LargeListView persistence; the tests establish remote client IPC encoding behavior, not native or Cloud persistence support. Live Cloud, full workspace/Clippy, the original ordinary-insert disconnect root cause, and Sophon's asynchronous transaction cleanup behavior are not established. Cancelling during staging retains the existing server-session cleanup limitation. Function reader delivery is separate. |
||
|
|
e9774ed168 |
feat(nodejs): expose Function refresh publication intervals (#4957)
Expose `publicationIntervalRows` on TypeScript refresh options and
forward it as `publication_interval_rows`. Accept positive safe integers
and preserve whole-fragment publication when omitted.
```typescript
const job = await table.refreshColumnAsync("embedding", {
publicationIntervalRows: 10000,
});
await job.wait();
```
Document the threshold semantics and add coverage for serialization,
omission, invalid values, safe-integer boundaries, and Embedded
rejection.
Closes ENT-3755.
|
||
|
|
a356950bce |
feat: expose queued Function refresh status (#4959)
## Problem The existing Function `column_status` surface cannot decode the Agent scheduler's `queued` state or the `failure_threshold_reached` and `binding_revision_missing` blocked reasons. It also preserves only the latest refresh job, so a later cancellation can hide an earlier actual failure. ## Change - Adds the `queued`, `failure_threshold_reached`, and `binding_revision_missing` wire states in Rust, Python, and Node. The wire change is additive, while Rust callers must update exhaustive enum matches and public struct literals. - Adds optional `last_failure_job` alongside `last_refresh_job`. The latest cancellation and the most recent actual failure retain separate meanings. - Reuses the canonical Rust `JobFailure` and existing Python/Node `JobFailureInfo` models for nested job failure details instead of introducing a parallel status error model. - Maps the existing public `sql_job_id` field through Node's `JobFailureInfo`. - Keeps `column_status`, `column_statuses`, and `wait_fresh` as the existing API surfaces. There is no new endpoint, row-error list, durable ledger, or unknown-enum fallback. - Documents `queued` versus `running` and the separate latest-attempt and latest-failure jobs in the existing hidden freshness release draft. The new status values are additive but cross-service: `queued`, the two new blocked reasons, and `last_failure_job` require matching Agent/QN and SDK deployments before clients can rely on them. This SDK follow-up does not change the normal `7cc3904a` dependency pin or claim whole ENT-3487 completion. ## Validation The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`, based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The final product docs commit changes generated TypeDoc only. - Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust blobs are unchanged from their tested `788ef85b` source through `c9bac25e`. - Python: the PyO3 extension built successfully, and the selected native PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The test requests pinned version 42. - Node: the addon and TypeScript builds passed, Biome passed, and the selected Jest transport test passed 1/0 with 57 filtered tests. Addon SHA-256: `c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`. - Python extension SHA-256: `36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`. - TypeDoc was generated from the native `native.d.ts` produced by that Node build. A second generation produced no remaining diff. The public diff contains only product, test, and generated API documentation changes. Native runner scripts and preserved compile-failure logs remain private validation evidence. Agent/QN production mapping, mixed-version rollout, and the remaining ENT-3487 server-side work are tracked separately. |
||
|
|
7cc3904a7e |
feat: expose exact Function column freshness (#4958)
## Problem Function-backed columns need an exact `fresh`, `stale`, `never_computed`, and `failed` partition for the table version a caller is reading. The existing error listing is capped, and composing it with job and schema listings cannot prove which rows currently failed. ## Behavior - Adds a strict freshness reader beside the existing permissive planner. It distinguishes an absent certificate from legacy evidence that omits failures, and from a current certificate carrying failure coverage. - Extends the immutable freshness sidecar with per-fragment known and failed row bitmaps. Partial or `WHERE` publications cover only observed rows; untouched rows stay unknown. Deleted rows do not block classification. - Adds failure-aware whole-fragment and row-repair writers. A rejected older attempt cannot overwrite failure evidence already recorded for a newer input signature. - Adds `column_status`, `column_statuses`, and `wait_fresh` to Python and the corresponding Node methods. The client calls the server's exact status endpoint, validates the four-way partition and one-version guarantee, and preserves the complete Function version, last-job, and automatic-refresh metadata. - Adds an MDX-commented draft usage section to the Enterprise computed-column refresh guide, covering Python, Rust, and TypeScript and remaining hidden until the server integration ships. - Sends `column`, `branch`, and exact `version` selectors to the status route. A pinned handle rejects a nonempty response from any other table version; latest handles keep the existing read-freshness headers. - Retries `wait_fresh` only for HTTP 503 with `function_column_status_finalization_pending`. Existing failed rows remain visible and do not keep the wait pending; other errors propagate. - Models the shared scheduler projection, including `up_to_date`, real `blocked` states, their reason, failure streak, and effective cooldown. ## Validation This is the SDK side of ENT-3563. Sophon's failure-coverage publication and status endpoint are a separate integration; deployed end-to-end and whole-ticket acceptance remain pending. - Rust: 9 exact freshness/status tests passed, plus 1 exact latest-handle freshness-header test. The pinned transport test was extended with a wrong-version response and rerun at `ebd6649a` (1 passed). Coverage includes untouched partial rows, selected-row failure clearing, cross-generation evidence, typed and non-retried errors, `up_to_date`, blocked scheduler fields, and exact-version requests. - Python: 2 native PyO3 remote-transport tests passed together, including typed retry; one local facade test reaches the existing unsupported boundary. - Node: the NAPI addon built successfully, TypeScript and Biome passed, and the selected native remote-transport test passed 1/0, including exact-version selection and blocked scheduler metadata. - `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile, Biome, and TypeScript checks passed. Validation is mapped by the relevant unchanged source rather than treated as one final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header and Python remote tests at `31e68b7d`, the local Python test and Node build at `6f213584`, and the final TypeScript/Jest delta at `4f2d5c27`. The final product tree excludes those five private EC2 runner files and adds six generated TypeDoc files plus one hidden draft usage section in the computed-column refresh guide. The Node delta corrected a private TypeScript command and reused the unchanged native addon. |
||
|
|
cf8e6d0fd4 |
feat: expose Function call concurrency limits (#4956)
## Summary Adds an optional `max_concurrency` setting to Function registration and explicit refresh: - A registered Function version can define the default maximum number of user-code calls in flight for one logical execution. - A refresh can override that default for that execution only. - The value must be a positive `u32` and is independent from `resource_budget`; it is not an account-wide quota. - Omitting the option preserves the previous request payload. Local embedded refreshes reject this Enterprise-only option. Rust, Python sync/async, PyO3, and the existing Node refresh surface forward the option. Node has no Function registration surface today, so no registration API is added there. The generated TypeDoc pages are included for the new public option. ## Validation The native-tested source covers Function version decoding, registration validation and serialization, the Flight SQL registration ticket and returned version, remote refresh forwarding, local rejection, and Node's numeric boundary before conversion. Exact stage results are Rust `1+3` plus four remote cases at `1` each; Python `14/1/49/24`; Node `57` remote and `1` local rejection. Ruff, standard repository Pyright, and Biome passed. The tested Python extension was `21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the tested Node addon and generated native declaration were `f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and `4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The final 24-file diff preserves the 22 tested product/test/fixture/source-doc blobs and adds only two generated TypeDoc pages. |
||
|
|
4d22c0ce2b |
feat: expose Function refresh resource budgets in SDKs (#4951)
## Summary - Add the canonical Rust `FunctionWorkerResourceBudget` and `RefreshColumnOptions::resource_budget()` for explicit Enterprise Function refreshes. - Forward the nested budget through Python sync/async, PyO3, Rust remote requests, Node NAPI, and TypeScript. - Require positive CPU and memory values; normalize an omitted nested GPU count to numeric zero; reject the nested budget when combined with the top-level `gpu_count` shorthand. - Reject `resource_budget` on LanceDB Embedded refresh paths instead of silently ignoring it. - Document the Rust, Python, and TypeScript APIs, including that SQL servers currently require all three budget fields. The Rust, Python, and TypeScript additions use `cpu_millis` / `cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` / `gpuCount`. The top-level GPU shorthand keeps its existing behavior and remains mutually exclusive with the complete nested budget. ## Scope and compatibility This change configures only the explicit refresh call. It does not configure `add_columns` automatic refresh and does not introduce a fleet-wide call-cap interface. A LanceDB Enterprise server must expose the nested `resource_budget` backfill contract. LanceDB Embedded continues to support its existing expression refresh defaults and rejects the new complete budget explicitly. ## Validation Validated against the final signed product source: - Rust: resource-budget remote serialization test `2 passed`; validation test `1 passed`. - Python: actual PyO3 build; remote-request matrix `22 passed`; invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff check and format clean. - Repository-standard Pyright entry point: `0 errors`, within the repository's configured include set. - Node: native build and Biome lint passed; the existing remote-request test `1 passed`, including fractional, negative, overflow, NaN, and infinity nested GPU cases; the existing local-rejection test `1 passed`. - TypeDoc generation and generated-document cleanliness passed. - The native-tested product source differs from this head only in two semantically equivalent Node range checks required by Clippy. Current CI compiles and lints the final blobs. No dependency or lockfile changes are included. |
||
|
|
2af85bea89 |
feat: expose the remaining refresh options on the SDKs (#4890)
The server's backfill request takes a row filter, failure limits, per-call timeouts and worker sizing, and SQL exposes all of them through `REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send only batch_size, GPU placement and the retry and error policies. Anyone needing a filtered refresh or a bounded skip policy had to drop to SQL. This adds `where`, `max_failed_fraction`, `max_failed_rows`, `row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and `max_executor_processes_per_worker` to Python, Rust and Node, passed through unchanged. `RefreshColumnOptions` now serializes straight into the request body, so the three SDKs share one producer. Node gets a `RefreshColumnOptions` object, which also carries the options it lacked before. A LanceDB Embedded refresh rejects every option other than batch_size and GPU placement, naming the option. |
||
|
|
a9ad499781 |
feat: support remote table compaction via SQL OPTIMIZE TABLE statement (#4802)
`Table::optimize()` previously returned `NotSupported` for remote tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL. Only `OptimizeAction::Compact` without a custom `remap_options` or non-default `CompactionOptions` is supported remotely. `OptimizeAction::All`, `::Prune`, and `::Index` all return a clear `NotSupported` error instead of silently doing the wrong thing. A branch-scoped table handle submits a `WITH (branch = '<branch>')` clause; the statement doesn't validate the branch exists itself, so an unknown branch surfaces as an async job failure rather than an error from this call. The server's SQL job adapter doesn't expose compaction metrics, so `OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a placeholder rather than real numbers — it must still be `Some`, since callers (including the Python/Node bindings) unwrap it unconditionally on success. This mirrors how `refresh_materialized_view_async` already handles the same limitation for its own metrics. The Python and Node `optimize()` convenience methods call a new shared `Table::optimize_full`, which runs `Compact` and then, on a remote table, skips `Prune` and `Index` instead of attempting and failing on them — returning zeroed prune stats rather than submitting a compaction job and then raising. Python's `RemoteTable.optimize()` also now forwards `compaction_options` to the Rust layer, so a non-default value is rejected before submission instead of being silently dropped. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
f6612a5f2f |
test(nodejs): preserve Function worker observations in job progress (#4819)
Extend the existing job progress regression to cover Function worker counts, GPU capacity and wait reasons, and known or absent warm-reuse evidence through `Job.progress()` and `Job.wait()`. The callback assertions also preserve an unknown warm state instead of turning it into `false`. Validation on EC2: all 56 remote SDK tests, Biome lint/format, TypeScript build, and documentation generation pass. Native code and dependencies match the previously validated build. Refs ENT-3486. |
||
|
|
8d10f25121 |
feat(nodejs): expose structured job progress (#4815)
TypeScript job handles now expose structured progress through `progress()` and an optional synchronous callback to `wait()`. Snapshots preserve unknown fields and missing counters, and job listings include progress and pause state. Existing `status()` strings and no-argument `wait()` behavior remain compatible. Wait callbacks observe the existing backend poller. Observer exceptions are logged without changing the job outcome. A best-effort ETA is available only from compatible samples with positive progress, and is omitted during initialization, queueing, or pause. Part of ENT-3500. --------- Co-authored-by: EC2 Default User <ec2-user@ip-172-31-12-104.us-east-2.compute.internal> |
||
|
|
9e65bd2c27 |
docs: use Enterprise terminology across website and SDK documentation (#4787)
Use “LanceDB Enterprise” throughout the staged website guides, SDK reference sources, generated API pages, and user-facing SDK messages. A separate branded “LanceDB Distributed” name adds terminology without clarifying behavior; describe where operations run directly. - Update `docs/web`, including the architecture diagram title and alt text. Use Enterprise in headings, table labels, and prose; simplify redundant product/execution-model wording and name the product only once in the diagram alt text. - Update Python docstrings and the Python/Java reference pages, TypeScript comments and native declaration comments, and Rust API documentation. Regenerate the TypeScript reference, including 24 replacements across six pages. - Update SDK warnings/errors and existing test expectations to match, along with contributor guidance and Vale rules. Remove the obsolete rule distinguishing Enterprise from Distributed. - Keep lowercase “distributed” for technical descriptions. API identifiers, anchors, asset paths, and execution behavior are unchanged. No branded-name exceptions are proposed. Validation: - Python and Node native builds succeeded; focused Python tests passed (36 passed, 4 skipped), and focused Node tests passed (14). - TypeDoc generation and the complete MkDocs reference build passed. MkDocs retains an existing unresolved `lancedb.functions.SecretReference` cross-reference. - Rust formatting, Node lint/format, changed-file Python lint/format, 32 Vale tests, and whitespace checks passed. Repository-wide Python lint has 48 existing findings in six untouched files. - Combined documentation assembly passed. Mintlify validation and link/anchor/redirect checks previously passed for both site builds; no code snippets changed. Companion PRs: [saved editions, terminology guide, and shared rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview and writing rules](https://github.com/lancedb/sophon/pull/9511). --------- Co-authored-by: Brendan Clement <brendan.clement@lancedb.com> |
||
|
|
ac7930fa77 |
feat: use SQL for remote view lifecycle
Stacked on #4675. Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination. `drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`. |
||
|
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45dbf5eff5 |
fix: use SQL for all remote materialized view APIs (#4673)
Remote materialized-view APIs now use SQL for the full lifecycle. Create and refresh already submit SQL; this change moves describe, paginated listing, and drop onto `DESCRIBE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS`, and `DROP MATERIALIZED VIEW`. Async create, refresh, and drop return the existing Job handle with `job_type = "query"`. The handle has one `id`: the connection-scoped SQL query ID, also accepted by `describe_query`. Submission starts the query, the job drains its result once, and repeated waits reuse the completed result. Synchronous drop waits for catalog removal; physical cleanup proceeds in the background. Listing uses namespace-scoped keyset pagination with opaque resume tokens. Streamed and deferred SQL responses share the same path. Adds transport coverage and updates Python/TypeScript documentation for query-backed drop jobs. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
180eaaf68e |
fix!: reject unsupported remote connection and table options (#4486)
Fixes #4399. Remote connections currently discard local-only settings, including table write options that determine storage format. The synchronous Python client also binds the fourth positional `create_table` argument to `on_bad_vectors`, so an intended overwrite is sent as a create. This change aligns the remote Python signature with `DBConnection.create_table`, preserves the legacy remote timeout keywords, and raises `ValueError` for other unknown connection keywords. The shared Rust client rejects unsupported connection, create, and open options before sending HTTP requests; open validation also runs before a table-cache hit. Python reports these as `NotImplementedError`, including explicit `enable_v2_manifest_paths=False`. Supported Enterprise connection-level storage options remain available. This implements option 2 from the issue: explicit rejection. It deliberately does not add unverified server write parameters. Calls that previously supplied ignored options must omit them to use server-managed storage and caching. Database URI prefixes are already preserved on current main and are unchanged here. Regression tests cover Python sync/async, TypeScript create/open calls, and the Rust remote client, including empty-table creation, overwrite mode, and cached table handles. Python API documentation describes the behavior and corrected positional order. Reconciled with main at `0ea932da`, preserving upstream blob and error-handling tests and updating the Python test that previously expected open options to be ignored. Validation on the merged source: - Rebuilt the Python native extension; 304 tests passed across remote options, database, remote database, and session suites. - Rust remote database suite: 135 passed. Rebuilt the Node native extension; TypeScript remote suite: 62 passed. TypeScript compilation and documentation generation passed. - Full Python SDK lint and formatting pass (159 files); full TypeScript lint and formatting pass (66 files). Rust formatting and `git diff --check` pass. - Repository-wide Python lint/format reports existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust 1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci` build profile. Remote tests use HTTP fixtures; this does not claim validation against the original ARM64 single-node deployment. Documentation review follow-up: added a draft Enterprise migration note in `docs/web/basics/sdk.mdx`. It explains that `db://` connections reject local session/cache and per-table storage/write options, while supported connection-level Enterprise storage options remain available. Following the repository's release conventions, the note stays in an MDX comment below the migration list until the behavior is released. Vale 3.24.0 with the pinned LanceDB style package reports no findings; `git diff --check` passes. This follow-up changes documentation only. BREAKING CHANGE: Remote connections now reject unsupported local session/cache and per-table storage/write options instead of silently ignoring them. Callers must omit these options and use server-managed storage and caching; supported Enterprise connection-level storage options remain available. Integration with current main preserves its synchronous API signatures, including the remote `create_table(mode=None)` default, and the table compatibility defaults. The positional-order repair is already on main via #4608; this PR retains the distinct unsupported-option rejection behavior. CI follow-up: the inherited IVF search-effort test occasionally measured recall 0.375 against its unchanged 0.5 threshold after random index training. A matching local reproducer observed this once in 30 builds. The fixture now places the eight exact neighbors together, preserving the 16-partition setup, two writes, all probe/precedence assertions, and the recall threshold. The adjusted fixture returned recall 1.0 in 100 builds; the full query and remote suites passed (102 tests). This stabilizes test data and does not change search behavior or promise production recall. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
4a03859b2a | fix!: normalize blob layouts and support external byte ranges (#4589) | ||
|
|
0968c21f4c |
feat(remote): support set_unenforced_primary_key (#4158)
## Summary `RemoteTable::set_unenforced_primary_key` returned `NotSupported`, so the call failed against LanceDB Cloud and enterprise from every SDK — Python and TypeScript both forward to it. That also blocked sharded LSM writes, since the server rejects bucket/identity sharding on a table that declares no unenforced primary key, and the SDK had no way to declare one. No server-side support was missing. The unenforced primary key is Lance schema field metadata, and the existing `update_field_metadata` endpoint writes exactly that, so the remote table now installs the key through it. The commit layer behind that endpoint installs the position, enforces immutability and runs `verify_primary_key()` — the same code a native table reaches, so both paths agree on semantics and not just on error messages. ## Changes - `rust/lancedb/src/table/primary_key.rs`: the request validation (single column, column exists, supported dtype, key not already set) and the metadata edit (set the 1-indexed position, clear the legacy boolean key) move into `validate()` and `install_edit()`, shared by both table types so they cannot drift. Native behaviour is unchanged. - `rust/lancedb/src/remote/table.rs`: real implementation — `check_mutable()`, validate against the described schema, then one `update_field_metadata` request. - `python/python/lancedb/remote/table.py`: drop the "Not supported on LanceDB Cloud." docstring. No binding changes were needed; Python and TypeScript already forward to the Rust `BaseTable` method. ## Tests Three mocked-server tests in `rust/lancedb/src/remote/table.rs`: - the happy path, asserting the request body carries `lance-schema:unenforced-primary-key:position: "1"` and a null legacy key on the right field path; - the invalid requests a native table also rejects (empty, compound, unknown column, unsupported dtype), each failing before any write reaches the server; - the already-set case, rejected from the schema the server already reports. `cargo test --features remote -p lancedb --lib`: 1186 passed, 0 failed. `cargo clippy --features remote -p lancedb --lib --tests` is clean. ## Note for enterprise Sophon pins the lancedb submodule at `v0.39.0-beta.6`, so this reaches enterprise on the next bump. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011xV8EebEZDmDDV8hVBm6MD --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3336bdfbec | fix(remote): bound concurrent blob reads across handles (#4311) | ||
|
|
1e4662ac23 |
fix(remote): unify unsupported operation errors and warnings (#4414)
Remote SDKs report Cloud-specific errors against self-hosted servers, and some Python wrappers raise different exception types for the same unsupported operation. The sync index wrapper also warns that `replace` is ignored even though Rust forwards it to the server. Use backend-neutral table/server diagnostics and the existing `NotSupported` error conversions for unsupported manifest operations, blob ranges, primary keys, optimization, and HNSW-PQ indexes. Sync Python delegates to the async paths, so manifest operations and `optimize()` consistently raise `NotImplementedError`. Legacy and config-based HNSW-PQ creation share the same error and alternative-index advice. Remove the false replacement warning and simplify the ignored GPU-accelerator warning. Keep the deprecated cleanup/compaction no-ops, with backend-neutral warnings that do not assume automatic server maintenance. Embedding configuration forwarding and stale warning removal are now supplied by #4408 in `main`. Retain the regression that verifies full embedding metadata and warning-free remote creation alongside the diagnostics tests. Validation against the merged `main` revision `bbc4aebe` and Lance 14.0.0-beta.1: - Rebuilt the Python extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`; all 122 remote Python tests passed. Root Ruff checks and formatting passed using the uv-managed environment. - `pnpm build`, `pnpm lint`, formatting, and `pnpm run docs` passed; all 47 remote Node tests and the focused local vector-index regression passed. - All 583 Rust remote tests and 54 query unit tests passed, along with `cargo check --quiet --features remote --tests --examples`, `cargo clippy --quiet --features remote --tests --examples`, and `cargo fmt --all`. Fixes #4393 <!-- lance-gatekeeper-fix:v1 agent=10bafd337999f4910855c3a9b71e7dcf generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
de005f1d61 |
refactor!: implement lazy iterators for several APIs (#4459)
Implement lazy iterators for: - secrets - views - jobs - functions - materialized views |
||
|
|
069b2f7d31 |
feat: add delete rows to table stats (#3783)
add to table stats the number of rows that has been marked for deletion, but not yet deleted. note that when a table fragment is removed due to deletion of all of its rows, these rows are not counted. since the main use of this feature is to determine when table compaction is needed, the fact that these rows are not reported should not impact that decision (deleted table fragments don't need compaction). Fixes: https://github.com/lancedb/lancedb/issues/3761 Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com> |
||
|
|
3e0a1a8780 |
fix: reject zero vector search probes across SDKs (#4418)
`nprobes(0)` bypassed the validation used by the minimum and maximum probe setters, allowing zero into query requests and producing inconsistent local and remote results. Sync Python could additionally interpret the zero maximum as an unbounded search. Reject zero in the Rust vector-query builder and propagate its input error through the Python vector/hybrid and Node bindings. Validate nonpositive counts in the sync Python vector and hybrid builders so every SDK and backend reports `nprobes must be greater than 0` before execution. Update the Rust examples and document the positive-count requirement. Regression tests cover local and remote vector/hybrid builders, rejection before remote query requests, promised TypeScript query vectors, and positive counts replacing both probe bounds. Validation: - Rust probe regression tests: 2 passed. - Focused Python query, remote, hybrid, and serialization tests: 22 passed. The four sync regression cases fail with the original setters. - Focused TypeScript indexed local and mocked remote tests: 2 passed. - Workspace `cargo check --features remote --tests --examples` and Clippy passed. - Rust formatting, Python formatting/lint, and TypeScript build/lint/docs passed. Repository-wide `ruff check .` reports 29 existing errors in unchanged CI scripts; `ruff check python` passes. Fixes #4388 BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>` instead of `Self`; callers must handle the result, as with the existing minimum/maximum probe setters. <!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
7e979c1416 |
fix(node): preserve schemas for empty query results (#4423)
Empty local TypeScript queries returned a zero-row Arrow table with no columns because `toArrow()` built the table solely from record batches. Local streams can yield no batches, while remote streams often supply an empty batch carrying the schema. Expose the executed stream's schema through the native iterator and decode it into an empty Arrow table when no batches arrive. This preserves projected and search columns, types, nullability, and metadata without executing another query. Document the empty-result behavior. Validation: - Eleven regression cases failed with the original `toArrow()` implementation; all twelve now pass, including remote schema-only and empty-batch responses. - All 1,075 tests in the query, remote, Arrow, and table suites pass. - Native build, TypeScript compilation, Rust formatting and Clippy (`-D warnings`), TypeScript lint/formatting, and documentation generation pass. Fixes #4386 <!-- lance-gatekeeper-fix:v1 agent=7d589da1f7db5c7383f9600490e111a9 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
ef11e44adc |
feat(node): expose allowExternalBlobOutsideBases in Table.add (#4317)
## Summary - Add `allowExternalBlobOutsideBases` to the Node `AddDataOptions` and forward it through N-API to the Rust `AddDataBuilder`. - Document its default and local-only external blob behavior. - Cover URI strings and URI objects, option scoping, progress callbacks, overwrite mode, and remote rejection. ## Validation - `pnpm test --runInBand` (21 suites passed, 1 skipped; 908 tests passed, 6 skipped) - `pnpm lint-ci` - `pnpm tsc` - `pnpm run docs` - `cargo fmt --all -- --check` - `CC=clang cargo clippy --profile ci -p lancedb-nodejs --all-features -- -D warnings` Fixes #4267 |
||
|
|
604b1b583b |
feat: pause and resume jobs from the Python SDK (#4112)
Cancellation is the only job control the SDK exposes, and it is terminal, so a long-running server-side job cannot be parked and picked up again. This adds `pause_job` and `resume_job` to the Rust core connection and the Python bindings (sync and async), posting to the server's `/v1/jobs/pause` and `/v1/jobs/resume` endpoints. A pause parks the job until it is resumed: its workers drain and stop. The outcome strings mirror the server's answers — a job finalizing its results reports `"committing"` and cannot be parked, and a resume before the drain is confirmed reports `"still_pausing"`; both are retried rather than failed. Resuming re-queues the job and its workers pick their work back up from checkpoints. Local connections report the operations as unsupported, like the rest of the jobs API. |
||
|
|
e4d0f9e2da |
feat: return a cleanup job from drop_view (#4262)
Dropping a view unbinds its name and leaves the definition dataset to a server-side cleanup job, so the two are separate events a caller may want to wait on. `drop_view_async` returns that job — the same shape `drop_materialized_view_async` and `drop_function_async` already use: a `202` carries the job id, a `200` (nothing was bound to the name) yields an already-finished job with no id, and any other success status is an error rather than a silent no-op. ## `drop_view` waits `drop_view` now awaits the job before returning, so a caller who does not want to think about cleanup gets the stronger guarantee: when it returns, the definition really is deleted. That is deliberately **different** from `drop_materialized_view` and `drop_function`, which return as soon as the name is unbound and document that content may still be deleting. The view API is the newer one, and waiting is the semantic worth having; the other two are left alone rather than changing behaviour already released. ## Surfaces `Database` trait, the remote client, `Connection`, and the Python and Node bindings — matching where `drop_materialized_view_async` is already exposed. Four client tests cover the accepted case reporting its job id, the nothing-bound case reporting a finished job, a `202` without a usable `job_id`, and an unexpected success status. |
||
|
|
8541d6d9df |
feat: add view CRUD APIs (#4236)
A view is a named query a database stores and plans on every read. It
holds no rows, which is the whole difference from a materialized view.
## API
| Verb | Route |
| --- | --- |
| `create_view(name, query, namespace_path)` | `POST
/v1/view/{id}/create` |
| `describe_view(name, namespace_path)` | `POST /v1/view/{id}/describe`
|
| `drop_view(name, namespace_path)` | `POST /v1/view/{id}/drop` |
| `list_views(namespace_path)` | `GET /v1/namespace/{id}/view/list` |
On `Connection` and the `Database` trait, with the remote client, Python
(sync and async) and Node bindings. Local databases return
`NotSupported`: the server side is Sophon's, where a view is an object
of the database manifest.
`ViewDescription` carries the defining query, the database *and
namespace path* unqualified names in it resolve against, and the schema
the query resolved to. `create_view` returns one, so a caller has the
schema without a second call.
Both defaults travel with the view because it outlives the session that
declared it: the server re-plans the stored query on every read, so a
reader resolving an unqualified name against its own defaults would read
a different table. `default_namespace_path` crosses the wire as
`default_namespace`, a path like `namespace`, absent for the root.
There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create, each authorized against what it
actually touches.
Querying a view stays SQL's job. There are no rows behind a view, so
there is no `open_view` returning a `Table`.
|
||
|
|
5231d37f8d |
feat: list a table's per-row Function errors from the client (#4208)
A refresh running under a skip policy records each row it skipped, with the failing input and the error, but the client could not read that store: the server exposes it over SQL and, since recently, a REST route. A user who hit per-row failures still had to open a SQL session. `Table::function_errors` calls the route. The listing is table-addressed with optional job and column filters, the same addressing the SQL surface uses, so the two cannot disagree about what a table's errors are. The two non-record signals come back as their own fields rather than as rows: capped-fragment summaries, and whether the listing stopped at its limit. Local tables refuse rather than answer with an empty list. Python and Node expose the same call, with the same optional filters. |
||
|
|
f3ef21b8ca |
feat: use oauth2 crate for OAuth with configurable client auth (#4181)
Stacked on #4173 (`jack/restore-oidc-flows`, base branch mirrored to this repo so the diff shows only this change); context from review: https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100. Rebase to `main` once #4173 and #4179 merge. ## What moved to the `oauth2` crate (5.0, no default features) - Authorization URL generation and CSRF state (`authorize_url`, `CsrfToken`) - PKCE S256 challenge/verifier generation and code exchange - Client-credentials, authorization-code, refresh-token, and device-code grant request construction - Device authorization request and the device token polling loop (`authorization_pending`, `slow_down` +5s, expiry deadline, denial, network backoff capped at 10s) - Standard success/error response parsing (`RequestTokenError`) - Token-endpoint client authentication and standards-compliant parameter encoding (RFC 6749 2.3.1 Basic encoding) LanceDB keeps ownership of OIDC discovery (compared `openidconnect`: no measurable win for our 3-field metadata + strict validation, at real dependency cost), HTTPS-or-loopback endpoint enforcement, the loopback callback server, browser/stderr prompts, token caching and refresh orchestration, and the dedicated hardened Azure IMDS source, which is unchanged. ## Client authentication methods New `ClientAuthMethod` enum (`none` | `client_secret_basic` | `client_secret_post`), exposed in Rust, Python, and Node. Unset resolves to `client_secret_basic` when a secret is present (RFC 6749 2.3.1 recommendation and the normal Okta confidential-app default, so a default Okta app works without weakening its configuration) and to `none` for public clients (PKCE/device). Explicit `none` with a secret, or basic/post without one, is rejected. The method applies to client credentials, code exchange, refresh, and device requests. Deliberate behavior change: confidential clients previously always sent the secret in the POST body; they now default to Basic (Keycloak accepts both). No `audience`/`resource` parameters were added: the supported target is an Okta custom authorization server with the API audience configured server-side, so client-provided audience parameters are unnecessary; `add_extra_param` support exists if a concrete provider contract ever needs them. ## Device polling behavior changes (deliberate, tested) - The first token poll now happens immediately rather than after one interval (RFC 8628 allows both). - Transient failures (HTTP 429, 5xx, `temporarily_unavailable`, network errors) now retry with exponential backoff capped at 10s instead of retrying at the fixed interval; polling never spins faster than once per second even if a server reports a zero interval. ## Security and compatibility - Issuer and discovered endpoints (and device verification URIs) still require HTTPS except explicit loopback HTTP, enforced before any crate URL type is built - Token HTTP client keeps the hardened redirect policy that refuses insecure redirect targets; regression test added - Errors never embed raw response bodies (avoids leaking tokens through parse failures); all credential types stay redacted in Debug - Transient conditions (429/5xx/`temporarily_unavailable`) remain retryable in device polling and hard errors elsewhere; refresh keeps rotation and reauthentication semantics - Existing public APIs stay source-compatible except the added `OAuthConfig.client_auth_method` field ## Tests Rust: client-auth methods across code exchange/refresh/client-credentials/device (none/basic/post), auth-method resolution and validation, transient device retries, denial/expiry, redirect rejection, malformed-response leak check, PKCE URL assertions, redaction. Python and Node: enum values, conversion, unknown-method errors, config defaults. Manual Okta validation recipe (no automated Okta credentials): create a custom authorization server with an API audience, one confidential web app (Basic) for authorization-code, one native app (PKCE, no secret), one native device app; point `issuer_url` at the custom server, set `client_auth_method` only for the POST-required case; verify token acquisition, refresh after expiry, and `x-lancedb-credential-type: oidc` against a LanceDB deployment. Never commit tenant URLs or secrets. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
|
|
21f11b4463 |
feat!: replace get_job/job_history with describe_job/query_job_events (#4130)
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.
## Before
Four ways to ask about a job, none of which told you much.
```python
job = table.refresh_column_async("embedding")
job.status() # "running". That was the entire debug surface.
db.get_job(job_id) # state, and a spec. No result, no progress.
db.job_history(job_id) # raw record batches, no limit, no filter
db.job(job_id) # a handle that knew nothing
```
## After
Open a job the way you open a table; the handle answers everything.
```python
job = db.open_job(job_id) # raises JobNotFoundError if there is no such job
```
```python
>>> print(job)
Job(
id='job-1',
state='failed',
job_type='refresh_column',
creation_ms=1757000000000,
spec={
"column": "embedding",
"num_workers": 4
},
failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```
Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.
Per-fragment progress *while it is still running*:
```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist() # [5000, 5000, ...]
```
The handle an async action returns is the same object, one `refresh()`
away:
```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```
TypeScript is the same experience, down to `console.log`:
```ts
const job = await db.openJob(jobId); // rejects if there is no such job
console.log(job); // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```
## Why each piece matters
- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.
`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.
## Breaking
All shipped in 0.38.0. No deprecated aliases.
| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |
Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
|
||
|
|
a87cada90e |
feat(node)!: require Node >= 22 and drop npm lockfiles (#4074)
The bindings are built, installed and published with pnpm everywhere,
but a parallel npm dependency graph was still being maintained beside
it. This removes it, raises the supported Node floor to the versions we
actually test, and gives Dependabot the npm coverage it was missing.
## Dropping npm
`nodejs/package-lock.json` was regenerated by `ci/update_lockfiles.sh`
on every release commit and read by nothing — no workflow runs `npm ci`
or `npm install` in `nodejs/`, and npm never publishes a lockfile in a
package tarball. It could not even agree with the real install, since
npm does not see pnpm's `overrides`. Because GitHub's dependency graph
parses `package-lock.json`, it was also reporting vulnerabilities for a
tree we neither install nor ship.
`docs/package.json`, `docs/package-lock.json` and `docs/tsconfig.json`
go too. They depend on `file:../node` and
`file:../node/node_modules/apache-arrow` — the `node/` directory was
removed long ago — the tsconfig compiles `src/*.ts` where no TypeScript
files exist, and nothing installs any of it. `docs.yml` only referenced
the lockfile to configure an npm cache for an install it never ran.
Two `workflow_dispatch` workflows for regenerating those lockfiles are
removed as well. Both were already broken: they `uses:` composite
actions at `.github/workflows/update_package_lock{,_nodejs}` that do not
exist, so dispatching either failed immediately.
The remaining `npx` calls become direct `node_modules/.bin/...`
invocations. These were already running locally installed binaries
rather than resolving anything, but naming the binary removes the npm
CLI from the loop and does not depend on which Node version is active.
`dev.yml`'s commitlint check was the last place doing real npm
dependency resolution — an unpinned `npm install
@commitlint/config-conventional` that also bypassed the
`minimumReleaseAge` hold configured for `nodejs/` — and is now a pinned
`pnpm dlx`.
## Node support
Node 18 and 20 both reached end-of-life, in April 2025 and April 2026.
The matrix moves to 22, 24 and 26, and `engines` rises from `>= 18` to
`>= 22` so the declared floor is one the matrix actually covers. Node 22
is LTS until April 2027; 24 is LTS; 26 is Current and becomes LTS in
October 2026.
This also removes the reason the workflows reached for `npx` in the
first place: pnpm 11 requires Node >= 22.13, which every matrix version
now satisfies.
The prebuilt-binary smoke test in `npm-publish.yml` moves from Node 20
to Node 22 — the floor, where a napi ABI problem would surface first —
rather than fanning out across all three, to keep the publish matrix
from tripling.
## Dependabot
There were no npm-ecosystem entries at all, which is why the advisories
behind #4073 went unnoticed. Both pnpm lockfiles are now watched —
`nodejs/` and `nodejs/examples/`, which is a separate install — using
the same `lockfile-only` strategy as the existing cargo and pip entries,
so version ranges in `package.json` are left alone.
## Pre-commit biome
The hook ran `npx @biomejs/biome@1.8.3` while `nodejs/package.json`
resolved 1.9.4. The two disagree about formatting, so the hook rejected
code that `pnpm lint` accepts, and failed on unmodified `main` for
anyone touching `nodejs/`. It now uses the pnpm-managed biome, which
fixes the drift with no source changes.
## Testing
`dev.yml`'s commitlint job does not check out the repo, so it runs in an
empty workspace, and I could not verify `pnpm/action-setup` there
locally. It triggers on `pull_request_target`, so this PR exercises it
directly — worth confirming green before merge. I did verify the `pnpm
dlx` invocation itself locally: it accepts a conventional title and
rejects a non-conventional one with exit 1.
Node 26 is new enough that the examples job may surface gaps in prebuilt
native binaries (`onnxruntime-node`, `sharp`) before their maintainers
publish for it.
## Not included
`nodejs/examples/` still pins `sharp: "0.33.5"` and has its own audit
findings. Raising the Node floor unblocks that work — sharp 0.35
requires Node >= 20.9, which the matrix now satisfies — but it is a
dependency bump rather than tooling cleanup, so it is left separate.
## Breaking changes
`@lancedb/lancedb` now requires Node >= 22; previously >= 18. The
`@types/node` peer range moves from `>=18` to `>=22` to match. Users on
Node 18 or 20 must upgrade their runtime; both have been end-of-life for
some time. Existing installs are unaffected, since `engines` is only
checked on install.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
68749ecfa3 |
feat(nodejs): materialized view bindings (#3935)
Exposes materialized views to TypeScript: createMaterializedView,
openMaterializedView and listMaterializedViews on Connection, and a
MaterializedView handle carrying the parsed definition and
refresh({full, sourceVersion}), which returns the typed refresh result.
select accepts column names, [alias, expression] pairs, or a record of
the
same; the definition reads back off the stored schema, so a reopened
handle
needs no side channel. Remote connections surface the core's
not-supported
error up front.
The napi crate needed the same recursion-limit raise as the core crate:
the
refresh future's type graph overflows the default trait-recursion depth.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
|
e98d8ac685 |
feat!: rename branch merge to cherry_pick (#3986)
This PR is a **breaking** rename of #3686. merge reads like git merge w/ three-way, replay history, combine two lines of work. That is not this API. This call takes one additive change on a branch and lands it on main. New column, including a blob column. Main's existing columns are not rewritten. If it cannot land, you get `status="failed"` and `diff.errors`, not a merge conflict to resolve. Cherry-pick is terminology that aligns more with that. ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.cherry_pick("exp", dry_run=True) result = table.branches.cherry_pick("exp") if result["status"] == "cherryPicked": print("landed at", result["mainVersionAfter"]) elif result["status"] == "failed": print(result["diff"]["errors"]) ``` ### Behavior - Remote / Enterprise only. Local still NotSupported. - HTTP 409 is not an exception. It is Ok with status="failed" and diff.errors (CherryPickError). - Unknown error / status codes still parse as Unknown. - Requests are not retried. 409 is final and carries the body. - Endpoint is POST /v1/table/{id}/branches/cherry_pick/. - merge_insert and Table.merge are unchanged. ### Testing - `cargo test -p lancedb --features remote diff_branch` - `cargo test -p lancedb --features remote cherry_pick` - `pytest python/python/tests/test_remote_db.py -k cherry_pick` - node `remote.test.ts` diffs / cherry-picks path |
||
|
|
1493ece3de |
test(node): cover remote table server errors (#3841)
## Summary - add a public Node API regression test for JSON server errors from remote table operations - verify countRows reports the server message instead of an ArrayBuffer decoding TypeError ## Root cause and fix The former TypeScript remote HTTP client passed an Axios-decoded JSON error object to TextDecoder, which masked the server response with an ArrayBuffer TypeError. The current Rust-backed remote client consumes non-success response bodies as text and propagates them through the Node error chain. This test exercises that corrected path through countRows and prevents the original failure from regressing. ## Validation - pnpm build - pnpm lint-ci - pnpm test --runInBand __test__/remote.test.ts - pnpm run docs Fixes #825 <!-- lance-gatekeeper-fix:v1 agent=91591c3d6b065796e6166664ef638aa7 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
e3b472c212 |
feat: connection-level job operations (#3755)
Adds job operations to the connection surface, building on the Job handle from #3742: job(id), list_jobs, get_job, cancel_job, and job_history, plus a non-blocking Job.status(). Implemented on the Database trait (defaulting to NotSupported), the remote backend (/v1/jobs), and the Python and Node bindings; job_history returns Arrow batches. errors() and progress() are not included. Tested with mocked endpoints in all three languages. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
f7feed48c3 |
feat(fts): support custom stop-word lists (#3734)
## What Expose custom FTS stop-word lists in the Python and TypeScript public APIs, including their standalone tokenize helpers and remote index creation. This PR supports concrete string lists only. It does not add file or LanceDB-table stop-word sources. ## Why Rust already exposes Lance's custom stop-word list option. The Python and TypeScript APIs did not pass it through, and local index details did not retain the full tokenizer parameters needed by index-backed tokenization after reopening a table. ## How - Add `custom_stop_words` / `customStopWords` to the Python and TypeScript FTS and tokenize options. - Preserve `None` / `undefined`, empty lists, and list contents without normalization. - Load the persisted FTS segment parameters when returning local index details. - Serialize the concrete list in remote create-index requests. - Keep Python and TypeScript tests thin; behavior, persistence, query tokenization, and remote JSON coverage live primarily in Rust. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` - Python extension rebuild with `uv` and `maturin` - Targeted Python tests: 4 passed - Python `ruff format --check` and `ruff check` - TypeScript build, typecheck, Biome lint, generated docs, and targeted tests --------- Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local> |
||
|
|
9dc5ec03aa |
feat(fts): add block size configuration (#3691)
## What changed - add `block_size` to Python FTS configuration and the deprecated local/remote helpers - add `blockSize` to the TypeScript FTS options and propagate it through the NAPI binding - serialize the value as `block_size` for remote index creation - document the existing Rust builder API and generate the TypeScript API reference - add local, remote, metadata, search, and invalid-value regression coverage ## Why Lance supports configuring the number of documents per compressed FTS posting block, but LanceDB's Python and TypeScript APIs did not expose the setting. This made the experimental FTS V3 layout unavailable through those clients and allowed the value to be dropped before index creation. ## How it works The default remains `128`. Supported values are `128` and `256`; selecting `256` uses the experimental FTS V3 format. Invalid values are rejected by the Lance builder and surfaced as Python or JavaScript errors. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo +1.94.0 clippy --quiet --features remote --tests --examples -- -D warnings` - targeted Rust local and remote index tests - Rust doctests: 34 passed - Python Ruff checks, doctest, and targeted local/remote tests: 5 passed - TypeScript build, Biome lint, generated docs, and targeted Jest tests: 9 passed - `git diff --check` ## Limitations The Java client remains unchanged because its external remote REST model does not currently expose `block_size`. Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local> |
||
|
|
65cd142c7e |
feat: add remote branch diff and merge client APIs (#3686)
This PR adds some support for `diff` / `merge` in the remote client as for local tables we stay `NotSupported` until https://github.com/lance-format/lance/issues/7263. This wires the two review-and-land calls against the remote REST API: - `POST /v1/table/{id}/branches/diff` - `POST /v1/table/{id}/branches/merge` Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python returns the wire JSON, same shape as the REST response. Merge here means promoting a branch's added columns onto `main`. ### Behavior - Remote only. Local raises `NotSupported`. - A rejected merge is not an exception. HTTP 409 still returns `Ok` / a dict with `status="rejected"` and blockers in `diff.mergeBlockers`. - Unknown blocker / status codes parse as `Unknown` so a newer server does not break older clients. - `MergePreview` tolerates missing fields for the same reason. - Merge requests are not retried. 409 is final and carries the body you need. ### Example ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.merge("exp", dry_run=True) result = table.branches.merge("exp", dry_run=False) if result["status"] == "merged": print("landed at", result["mainVersionAfter"]) elif result["status"] == "rejected": print(result["diff"]["mergeBlockers"]) ``` ### Testing cargo test -p lancedb --features remote diff_branch cargo test -p lancedb --features remote merge_branch --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
f76b075d13 |
feat: add table branch support to remote tables and Python/TS bindings (#3540)
### Description Adding branch support for RemoteTable by threading a branch selector onto every operation the data plane accepts it on. Exposes the currentBranch to nodejs and python through the bindings. Matching the server handlers, the branch rides as: - a `?branch=` query parameter for Arrow-body and query-only ops (insert, merge_insert, multipart_*, version/list, drop_index) - a `branch` field in the JSON body for everything else (count_rows, query, update, delete, create_index, column ops, index list/stats, stats, restore, describe, tags create/update) A main-branch handle (`branch == None`) produces byte-identical requests to before: no `branch` field and no `?branch=` - Handle-per-branch: `create_branch` / `checkout_branch` return a new handle with fresh caches and reset version/freshness state, mirroring `NativeTable`. - `create_branch` maps 409 to already-exists, 400 to invalid, and 404 to not-found with source context, and sends without retry so the 409 stays observable. - `Ref` translation covers version, version-number (relative to the handle's branch), and tag (resolved via the tags endpoint); `"main"` and empty normalize to the main branch. - Python branch handles persist their branch (and pinned version) across pickle/fork, so a forked or pickled handle reopens on its branch rather than silently reverting to main. ### Tests - Rust mock tests per op category (query-param and body mechanisms, branch CRUD, error paths, backward-compat). - Python sync branch CRUD, `open_table(branch=)`, and a pickle round-trip regression test. |
||
|
|
d9018067b3 |
feat: support checking out a version on a branch (#3504)
### Description Stacked on #3490. Adds an optional version to branch checkout across the Rust core and the Python and TypeScript SDKs, so you can open a specific version on a branch ("version V of branch B"), not just the branch's latest version Rust ```rust // Open version 3 of branch "exp" (a read-only view): check out from an // existing table, or open it directly from the connection. let exp_v3 = table.checkout_branch("exp", Some(3)).await?; let exp_v3 = db.open_table("items").branch("exp").version(3).execute().await?; // checkout_latest re-attaches to the branch's writable HEAD. exp_v3.checkout_latest().await?; // With no branch, a version opens main at that version. let main_v3 = db.open_table("items").version(3).execute().await?; ``` Python ```python # Open version 3 of branch "exp" (a read-only view): check out from an # existing table, or open it directly from the connection. branch_v3 = await table.branches.checkout("exp", version=3) branch_v3 = await db.open_table("items", branch="exp", version=3) # checkout_latest re-attaches to the branch's writable HEAD. await branch_v3.checkout_latest() # With no branch, a version opens main at that version. main_v3 = await db.open_table("items", version=3) ``` TypeScript ```typescript // Open version 3 of branch "exp" (a read-only view): check out from an // existing table, or open it directly from the connection. const branchV3 = await (await table.branches()).checkout("exp", 3); const opened = await db.openTable("items", undefined, { branch: "exp", version: 3 }); // checkoutLatest re-attaches to the branch's writable HEAD. await branchV3.checkoutLatest(); // With no branch, a version opens main at that version. const mainV3 = await db.openTable("items", undefined, { version: 3 }); ``` ### Testing - Added unit tests (Rust, Python sync + async, TypeScript): branch-scoped resolution at a version number shared with `main` and with another branch, read-only enforcement on a pinned handle, `checkout_latest` recovery to the branch's HEAD, fork-point reads, and the nonexistent-version/branch error paths. - Ran smoke tests against the Python and TypeScript SDKs on local machine. |
||
|
|
4cb9147bbf |
feat(nodejs): add renameTable on Connection (#3386)
Adds `Connection.renameTable` to the Node SDK. Closes #3381. |
||
|
|
48e5caabda | ci(nodejs): lint for unused imports (#2673) | ||
|
|
8da74dcb37 |
feat: support per-request header override (#2631)
## Summary This PR introduces a `HeaderProvider` which is called for all remote HTTP calls to get the latest headers to inject. This is useful for features like adding the latest auth tokens where the header provider can auto-refresh tokens internally and each request always set the refreshed token. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
9391ad1450 |
feat: support mTLS for remote database (#2638)
This PR adds mTLS (mutual TLS) configuration support for the LanceDB remote HTTP client, allowing users to authenticate with client certificates and configure custom CA certificates for server verification. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
02595dc475 |
feat: add overall timeout parameter to remote client (#2550)
## Summary - Adds an overall `timeout` parameter to `TimeoutConfig` that limits the total time for the entire request - Can be set via config or `LANCE_CLIENT_TIMEOUT` environment variable - Exposed in Python and Node.js bindings - Includes comprehensive tests ## Test plan - [x] Unit tests for Rust TimeoutConfig - [x] Integration tests for Python bindings - [x] Integration tests for Node.js bindings - [x] All existing tests pass 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
16851389ea |
feat: extra headers parameter in client options (#2091)
Closes #1106 Unfortunately, these need to be set at the connection level. I investigated whether if we let users provide a callback they could use `AsyncLocalStorage` to access their context. However, it doesn't seem like NAPI supports this right now. I filed an issue: https://github.com/napi-rs/napi-rs/issues/2456 |
||
|
|
15f8f4d627 |
ci: check license headers (#2076)
Based on the same workflow in Lance. |
||
|
|
a324f4ad7a |
feat(node): enable logging and show full errors (#1775)
This exposes the `LANCEDB_LOG` environment variable in node, so that users can now turn on logging. In addition, fixes a bug where only the top-level error from Rust was being shown. This PR makes sure the full error chain is included in the error message. In the future, will improve this so the error chain is set on the [cause](https://nodejs.org/api/errors.html#errorcause) property of JS errors https://github.com/lancedb/lancedb/issues/1779 Fixes #1774 |
||
|
|
f3b6a1f55b |
feat(node): bind remote SDK to rust implementation (#1730)
Closes [#2509](https://github.com/lancedb/sophon/issues/2509) This is the Node.js analogue of #1700 |