mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
4e3ec7cc27587d4e64021ec5bd1b2607b724f5eb
264
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
0327618a62 |
feat(sql): add native vector dedup and partitioned mv refresh (#4303)
## Summary
Materialize a source-preserving dedup result from an indexed, pinned
snapshot with a short declaration. The default greedy policy keeps the
smallest unassigned snapshot row ID and removes only its directly
matching neighbors: For A < B < C, an A–B–C chain without an A–C edge
keeps A and C.
```python
from lancedb import vector_dedup
view = db.create_materialized_view(
"images_clean",
vector_dedup("images", column="phash", distance_threshold=4),
)
cleaned = view.table
```
The Python source expression captures the source version once at
creation and
waits for the materialized result by default. Use `dataset_version=...`
for an
explicit snapshot, `with_no_data=True` for declaration only, or
`create_materialized_view_async(...)` for a job handle. AsyncConnection
supports
the same expression. It uses the existing MV REST endpoints and registry
jobs,
without opening a SQL query connection. Source names are exact
root-namespace
table names. Additional select/where/limit options are rejected for this
source.
The SQL declaration remains available:
```sql
CREATE MATERIALIZED VIEW images_clean AS
SELECT * FROM vector_dedup('images', 7, 'phash', 4);
REFRESH MATERIALIZED VIEW images_clean;
```
This is the host-facing SQL entry point. The MV builder now accepts a
typed native vector source; the Rust SDK also supports the same
definition through `MaterializedViewDefinition::from_sql` and
`prepare_definition`; `view.refresh().execute()` executes it locally.
Hosts distribute its units using their existing job infrastructure. The
raw `vector_duplicate_pairs` source remains available separately when
callers want the pair relation.
## Implementation and contract
- Use actual Rust Lance `13.0.0-beta.15` (`db53dcf367`); the native
partition primitive computes pairs from index representations without
source-vector aggregation or a Python pair loop.
- Extend partitioned MV refresh with two dependent waves. Physical
segment/partition tasks stream pairs through a bounded external sort,
select direct representatives and write per-source-fragment bitmap
artifacts. Source-fragment tasks start after every selection receipt
completes and stream retained original rows, including blob payloads,
into uncommitted output fragments.
- Reuse normal blob write coercion and row-mask scans, including stable
row-ID translation after compaction. Null vectors, isolated rows and
source deletions are preserved correctly. Conflicting representative
decisions across overlapping index scopes fail before publication.
- Expose `dependency_units()` and `write_refresh_partition_with_inputs`
alongside the existing plan/commit API. Receipts bind
snapshot/configuration/policy/destination generation. Only a complete
accepted set can publish; unfinished units restart, completed units can
be reused. There is no pair-position checkpoint.
- Use MV format 4 for dedup results and format 3 for native pairs;
Python and TypeScript metadata readers recognize both. No source
deletion or source row-ID migration occurs.
Only same-segment/same-partition edges are covered. Quantized scores use
symmetric represented-vector distances. Result materialization requires
V2 data storage and is initially a native MV source, not an ordinary
SELECT table function. Compressed vertex masks stay in memory; edge
sorting spills to local disk. Unreferenced artifacts use normal Lance
retention/cleanup and must be retained for the job's recovery window.
## Validation
AWS Linux validation:
- Native pair/result integration matrix passed: multiple
segments/partitions, exact thresholds, empty output, cross-batch data,
single-process pair parity, source preservation, lost receipts and
publication guards.
- Blob/null-vector/pinned-deletion tests passed with both row-ID modes,
including compaction that changes physical row addresses.
- Representative policy, forced external-sort spill and
conflicting/corrupt artifact tests passed.
- Full MV regression suite: 153 passed, 2 pre-existing ignored. Rust
formatting and Clippy passed.
- Python MV suite: 47 passed; TypeScript MV suite: 11 passed. Native
bindings were rebuilt in project-local environments; formatting, lint
and TypeScript docs passed.
- Rust declaration doctests: 3 passed, including the complete local
dedup example.
- Companion host validation: 41 MV job tests passed (1 pre-existing
ignored), full workspace Clippy passed, and the complete process-level
CLI E2E suite passed (3 tests). Recovery exercises both selection and
materialization waves.
- A real CIFAR-10 pHash run used five actual registry-worker owners,
completed 32 selection + 24 materialization tasks, and retained 49,846
of 50,000 rows with identical original values and image payloads.
Host workflow performance measurements are recorded in the companion
host PR. Cross-platform CI is still running. The previous remote-test
bootstrap failed before Cargo tests with `401 Bad credentials`; the
workflow fix described below is being validated by CI. No deployment or
merge is performed.
## Python source validation
AWS tests cover automatic and explicit snapshots, physical/stable row
IDs, local
and remote creation, sync/job/async-connection APIs, delayed refresh,
strict
options and identifier quoting. The native Python extension was rebuilt
through
the project's uv/maturin backend. Ruff, Pyright and the full MkDocs
API-reference
build passed; both new public symbols are rendered.
A real 50,000-image FEv2 run disabled both Python SQL entry points and
supplied an
unreachable SQL endpoint. The new API still materialized the same 49,846
retained
rows and original image payloads. Three recorded creation/refresh jobs
completed
56 tasks each on all five registry-worker owners, and remained on source
snapshot
3 after source latest advanced to 4. No worker or server wire-protocol
changes
were needed for this client entry point.
Previous CI authentication failure:
https://github.com/lancedb/lancedb/actions/runs/36334679276/job/108663151265.
## Review and CI fixes
- Validate all JSON merge source fields before projection, rejecting
unknown and duplicate names at every struct level. Missing/reordered
fields and ordinary JSON merges remain supported. A checked-in 12-case
integration matrix covers root fields, structs, lists, large lists,
fixed-size lists and map values; rejected writes leave the table version
and row count unchanged.
- Match namespace connection MV method signatures to
`MaterializedViewSource` in all four sync/async methods, fixing the
Python 3.10 import failure from runtime override checks.
- Restore the remote-test script's encrypted-CLI path: use the Actions
token for artifact download and pass `SOPHON_READ_TOKEN` under the
variable the decryption script consumes. The previous workflow omitted
that variable and attempted the private release path, failing with HTTP
401. On the final head, remote CI passes: 1,464 Rust unit tests, the
integration targets and doctests pass (3 unit tests ignored).
AWS validation for these fixes: both review reproducers failed before
their fixes and pass afterward; 55 merge tests, 29 cast tests and the
standalone 7-case admission regression pass. Python 3.10.21 imports
successfully; 90 MV/namespace tests, 14 JSON tests and four real
namespace dedup cases pass. After `794b97a0`, the native binding was
rebuilt again and the JSON suite rerun. Rust formatting/Clippy, Ruff and
Pyright pass. Sophon's locked test compilation passes against the final
SDK source for all seven affected crates. Extended runners, logs and
benchmark artifacts remain local.
Gatekeeper recommends approval on `794b97a0`, and the `breaking-change`
label is set. Final CI: **49 checks passed, 3 expected skips, no
failures or pending checks**, including remote tests, Python 3.10,
minimum dependencies and Windows ARM Rust tests. All wheel builds pass
after retrying the Linux matrix affected by transient AlmaLinux
repository-metadata 404s. All code-review threads are resolved. The PR
remains open for maintainer review and merge.
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
Co-authored-by: Lance Release <lance-dev@lancedb.com>
|
||
 Xuanwo
|
a9e350c158 |
ci(python): build Windows release wheel on 8-core runner (#4264)
Since #4241 restored fat LTO / 1 codegen unit for the Windows wheel, the `windows` job in PyPI Publish has been cancelled at its 90-minute timeout on every run, including the v0.40.0-beta.6 and v0.40.0-beta.7 tags, so neither release was published. In those runs dependency compilation finishes after ~26 minutes and the `lancedb` crate compile + fat-LTO step was still running after 63 minutes. Move the job to the org's `windows-2025-8x-x64` larger runner and raise the timeout to 150 minutes. Fat LTO stays, since the thin-LTO wheel exceeds PyPI's 100 MiB limit. The final fat-LTO step is largely single-threaded, so the gain from 8 cores is mainly in the parallel dependency phase plus faster/larger hardware; the PR run of this workflow is the first measurement of fat LTO on this runner. |
||
|
Xuanwo
|
ff757b74fd |
fix(python): reduce Windows release wheel size (#4241)
The Windows wheel for 0.39.0 exceeded PyPI's 100 MiB file-size limit, preventing the original release upload. Restore the repository's release profile (`fat` LTO, 1 codegen unit) by removing the Windows-only `thin`/16 overrides introduced in #3716. Keep `rust-lld` as the linker. This recovers wheel-size headroom at the cost of the longer fat-LTO build. The already-published 0.39.0 Windows wheel was recovered separately by recompressing the original artifact; this change addresses the build configuration for future releases. Fixes #4239. ### Windows size comparison Built the same v0.38.0 source (`8c68e0c619f2b1febe92e51a29d72c968d24a5c9`) twice on one AWS `m7i.4xlarge` Windows Server 2025 machine, changing only the LTO/codegen-unit overrides: | Artifact | thin LTO / 16 units | fat LTO / 1 unit | | --- | ---: | ---: | | Installable Windows wheel | 104,129,447 bytes (99.31 MiB) | 73,875,399 bytes (70.45 MiB) | | Uncompressed native module | 309,969,408 bytes (295.61 MiB) | 206,609,408 bytes (197.04 MiB) | The thin/16 configuration increased wheel size by **40.95%** and native-module size by **50.03%** relative to fat/1. The compressed native module accounts for all but 3 bytes of the wheel increase. Both builds used Rust 1.97.0, maturin 1.12.4, Python 3.13.5, MSVC 14.44.35207, Windows SDK 10.0.26100.0, rust-lld, static CRT, default features, and `maturin build --release --strip --locked --verbose`. They ran sequentially with separate empty target directories and identical locked third-party dependencies. The tag's stale workspace package versions were normalized once before both builds. This controlled pair used VS2022 Build Tools; the historical GitHub runner used VS2026. Both wheels passed ZIP/RECORD integrity checks and installed successfully. Native smoke checks covered import, database creation, row count, nearest-vector query, and reopening the database. This establishes the combined configuration effect; it does not isolate LTO mode from codegen-unit count or establish a runtime-performance difference. The figures above are the v0.38.0 reproduction, **not measurements of this PR head**. The existing PyPI Publish pull-request workflow rebuilds the current revision without publishing; its result is pending. Workflow validation passed with `actionlint -shellcheck=`. Full actionlint reports the same three pre-existing ShellCheck diagnostics in the unchanged repository-selection step as on `main`. |
||
 Jack YeandXuanwo
|
6a07f88980 |
feat: add remote catalogs and Python and TypeScript bindings (#4195)
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
Jack YeandXuanwo
|
56bbd19ff4 |
test(python): cover namespace operations without pylance (#4174)
The "Test without pylance or pandas" CI job only ran `test_table.py`, so the regression fixed in #3606 had no guard: sync namespace operations used to route through the Python `lance_namespace` client, whose `dir` implementation ships in the optional `pylance` extra, so `lancedb.connect(path).list_namespaces()` failed with `No module named 'lance'` while the async API worked. Adds `python/python/tests/test_namespace_no_pylance.py` and runs it in that job. Its `without_pylance` fixture blocks `lance` imports, so the guard also fires in environments that do have `pylance` installed. Coverage: the original reproducer, nested namespace lifecycle, namespaced and root table lifecycle, the async path, and the one API that legitimately still needs `pylance` (`namespace_client()`). Verified the guard actually catches the regression: against `lancedb==0.33.0`, three of these tests fail with the original error; against a fixed build all pass. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
  Will JonesandClaude Sonnet 5
|
1da5876870 |
ci: add spell checking (#4148)
Adds [typos](https://github.com/crate-ci/typos) as a CI check and pre-commit hook, the same way Lance does it, so misspellings like the ones fixed in #4146 get caught automatically going forward. This also fixes the misspellings `typos` found across the repo (Rust, Python, TypeScript source, comments, and generated docs), and adds a small `.typos.toml` with `extend-words` entries for terms that are correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ` (a real quantization algorithm name), `mmaped` (the actual name of a `candle-core` API we call), and `Writeable` (from Python's `_typeshed.WriteableBuffer`). Third-party license files are excluded. Fixes #4147 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack Ye
|
e639b1b650 |
feat: add asynchronous remote SQL queries (#4070)
## Summary
Add SQL execution to remote LanceDB connections. On the standard
synchronous connection, `execute_query` waits for the initial result
stream and returns its Arrow reader. `execute_query_async` is called
without Python `await` and immediately returns a query handle for status
inspection, streaming, or cancellation. Local databases report that SQL
is not supported.
The transport and query lifecycle live in Rust. Python exposes
native-backed synchronous and asynchronous connection methods and query
wrappers; it does not use PyArrow's Flight client.
## User experience
The standard synchronous connection supports both direct reads and
background query execution:
```python
db = lancedb.connect(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
# Direct execution waits only until the initial result stream is available.
# Later batches continue streaming as the query progresses.
reader = db.execute_query(
"SELECT * FROM events",
default_namespace_path=["production"],
)
for batch in reader:
print(batch.num_rows)
# Background execution returns a query handle immediately. Despite the
# `_async` suffix, no Python `await` is needed on a synchronous connection.
query = db.execute_query_async("SELECT * FROM events")
print(query.id)
description = db.describe_query(query.id)
print(description.status)
print(description.progress)
print(description.expires_at)
# Start reading as soon as the service advertises partial results. The reader
# continues polling and yields newly available record batches until the query
# and all result endpoints are complete.
reader = query.reader()
for batch in reader:
print(batch.num_rows)
# Or cancel a different still-running query. Its status becomes "cancelling"
# while the server is still working, then "cancelled" once confirmed.
cancelled_query = db.execute_query_async("SELECT * FROM large_events")
cancelled_query.cancel()
```
The less commonly used asynchronous connection exposes the same
operations as coroutines:
```python
async_db = await lancedb.connect_async(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
query = await async_db.execute_query_async("SELECT * FROM events")
async for batch in await query.reader():
print(batch.num_rows)
```
The UUIDv7 query id is scoped to the connection that submitted it. The
connection retains lightweight shared query state used by
`query.describe()` and `db.describe_query(query.id)`; the id does not
encode SQL or a Flight continuation token and is not a cross-connection
resume token. Abandoned state has bounded retention, and terminal state
remains available briefly.
Unqualified table names use the connected database and the `public`
namespace by default. `default_namespace_path` accepts a list such as
`["production", "events"]`. SQL can still use qualified names to
reference other databases and namespaces available to the deployment.
## Design
- Uses Arrow Flight `PollFlightInfo` for submission and long polling,
`DoGet` for results, and `CancelFlightInfo` for cancellation. Each
`PollInfo.info` is treated as the cumulative set of currently available
endpoints, so advertised tickets are consumed once and batches can be
delivered before execution is complete.
- Serializes result completion and cancellation into one lifecycle. A
server-accepted request reports `cancelling` and wakes blocked
status/result work; a later retry can confirm `cancelled`. Result
retrieval is rejected after cancellation is accepted, while cancellation
after a result was already delivered is a no-op.
- Assigns a time-ordered UUIDv7 connection-scoped query id and retains
only shared evolving lifecycle state, keeping SQL, Flight continuation
tokens, and Arrow result data out of public ids and the registry.
- Leaves admission control to the server while honoring server
expiration and a local fallback retention window for abandoned entries.
- Retains terminal ids for five minutes so they remain available for
connection-level description.
- Keeps one lazily initialized SQL client on each remote database
connection and attaches fresh authentication, routing, namespace, and
request metadata to every operation.
- Applies the configured overall timeout to each execution, description,
reader, and cancellation operation. A result reader carries one absolute
deadline from `reader()` through the end of streaming; connect and read
timeouts continue to bound their individual phases.
- Returns a bounded, backpressured, single-consumer Arrow stream rather
than collecting the full result in memory. Dropping the reader stops
downloading but does not implicitly cancel the server query.
- Preserves typed schemas for empty result sets through the stream
schema.
- Accepts Flight result messages up to 1 GiB so a valid row containing a
large blob, string, or vector is not rejected by tonic's 4 MiB default
receive limit.
- Supports the Python client first while keeping the authoritative
implementation in the Rust core.
|
||
|
Will JonesandClaude Opus 5
|
a87cada90e |
feat(node)!: require Node >= 22 and drop npm lockfiles (#4074)
The bindings are built, installed and published with pnpm everywhere,
but a parallel npm dependency graph was still being maintained beside
it. This removes it, raises the supported Node floor to the versions we
actually test, and gives Dependabot the npm coverage it was missing.
## Dropping npm
`nodejs/package-lock.json` was regenerated by `ci/update_lockfiles.sh`
on every release commit and read by nothing — no workflow runs `npm ci`
or `npm install` in `nodejs/`, and npm never publishes a lockfile in a
package tarball. It could not even agree with the real install, since
npm does not see pnpm's `overrides`. Because GitHub's dependency graph
parses `package-lock.json`, it was also reporting vulnerabilities for a
tree we neither install nor ship.
`docs/package.json`, `docs/package-lock.json` and `docs/tsconfig.json`
go too. They depend on `file:../node` and
`file:../node/node_modules/apache-arrow` — the `node/` directory was
removed long ago — the tsconfig compiles `src/*.ts` where no TypeScript
files exist, and nothing installs any of it. `docs.yml` only referenced
the lockfile to configure an npm cache for an install it never ran.
Two `workflow_dispatch` workflows for regenerating those lockfiles are
removed as well. Both were already broken: they `uses:` composite
actions at `.github/workflows/update_package_lock{,_nodejs}` that do not
exist, so dispatching either failed immediately.
The remaining `npx` calls become direct `node_modules/.bin/...`
invocations. These were already running locally installed binaries
rather than resolving anything, but naming the binary removes the npm
CLI from the loop and does not depend on which Node version is active.
`dev.yml`'s commitlint check was the last place doing real npm
dependency resolution — an unpinned `npm install
@commitlint/config-conventional` that also bypassed the
`minimumReleaseAge` hold configured for `nodejs/` — and is now a pinned
`pnpm dlx`.
## Node support
Node 18 and 20 both reached end-of-life, in April 2025 and April 2026.
The matrix moves to 22, 24 and 26, and `engines` rises from `>= 18` to
`>= 22` so the declared floor is one the matrix actually covers. Node 22
is LTS until April 2027; 24 is LTS; 26 is Current and becomes LTS in
October 2026.
This also removes the reason the workflows reached for `npx` in the
first place: pnpm 11 requires Node >= 22.13, which every matrix version
now satisfies.
The prebuilt-binary smoke test in `npm-publish.yml` moves from Node 20
to Node 22 — the floor, where a napi ABI problem would surface first —
rather than fanning out across all three, to keep the publish matrix
from tripling.
## Dependabot
There were no npm-ecosystem entries at all, which is why the advisories
behind #4073 went unnoticed. Both pnpm lockfiles are now watched —
`nodejs/` and `nodejs/examples/`, which is a separate install — using
the same `lockfile-only` strategy as the existing cargo and pip entries,
so version ranges in `package.json` are left alone.
## Pre-commit biome
The hook ran `npx @biomejs/biome@1.8.3` while `nodejs/package.json`
resolved 1.9.4. The two disagree about formatting, so the hook rejected
code that `pnpm lint` accepts, and failed on unmodified `main` for
anyone touching `nodejs/`. It now uses the pnpm-managed biome, which
fixes the drift with no source changes.
## Testing
`dev.yml`'s commitlint job does not check out the repo, so it runs in an
empty workspace, and I could not verify `pnpm/action-setup` there
locally. It triggers on `pull_request_target`, so this PR exercises it
directly — worth confirming green before merge. I did verify the `pnpm
dlx` invocation itself locally: it accepts a conventional title and
rejects a non-conventional one with exit 1.
Node 26 is new enough that the examples job may surface gaps in prebuilt
native binaries (`onnxruntime-node`, `sharp`) before their maintainers
publish for it.
## Not included
`nodejs/examples/` still pins `sharp: "0.33.5"` and has its own audit
findings. Raising the Node floor unblocks that work — sharp 0.35
requires Node >= 20.9, which the matrix now satisfies — but it is a
dependency bump rather than tooling cleanup, so it is left separate.
## Breaking changes
`@lancedb/lancedb` now requires Node >= 22; previously >= 18. The
`@types/node` peer range moves from `>=18` to `>=22` to match. Users on
Node 18 or 20 must upgrade their runtime; both have been end-of-life for
some time. Existing installs are unaffected, since `engines` is only
checked on install.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Will JonesandClaude Opus 5
|
84f46df876 |
ci(nodejs): fix nightly OOM on the aarch64 publish legs (#4077)
The nightly `NPM Publish` run has failed every night since at least Aug 23, always on the same two legs: `aarch64-unknown-linux-gnu` and `aarch64-unknown-linux-musl`. The other five targets pass. rustc is OOM-killed during the fat-LTO codegen of the cdylib — `signal: 9` with no diagnostic, about 27 minutes in — and on the musl leg that takes the whole runner down with `The runner has received a shutdown signal`. Both legs now pass: | leg | before | peak memory | wall time | | --- | --- | --- | --- | | `aarch64-unknown-linux-gnu` | OOM-killed at ~27 min | 31391 → 22851 MiB | 38m43s → 22m04s | | `aarch64-unknown-linux-musl` | runner killed at ~28 min | >32 GiB → 16516 MiB | ~40 min → 20m50s | **ThinLTO** is most of that. Fat LTO is single-threaded, and its peak is consumed inside rustc's LLVM before any linker process is spawned — which is why it is the whole fix on musl, and why lld alone left the gnu leg still peaking at 31391 MiB against the runner's 32 GiB. Both legs now use the `lto: thin` / `codegen_units: 16` settings that darwin and both Windows legs already use, at a cost of a few percent runtime performance. **lld** covers the rest, on the gnu leg. arm64 Linux otherwise links through GNU `ld` where x86_64 already defaults to `rust-lld`, which is why only the arm64 legs hit this at all; on a comparable arm64 build (`lancedb/sophon#7313`) it cut the largest single linker process from 7.0 to 4.0 GiB and wall time by 35%. The flags live in a small wrapper script used as the linker rather than in `-C link-arg`, because the per-target rustflags variable does not reach every unit that links: dependency crates linking a dylib (`crc-fast`, `lance-arrow`) were invoked as bare `clang`, which targets the x86_64 host and fails with `Relocations in generic ELF (EM: 183)`. Separately, and affecting five legs rather than two: the three ThinLTO targets exported `CARGO_PROFILE_RELEASE_LTO` and `CARGO_PROFILE_RELEASE_CODEGEN_UNITS` from `pre_build`, which runs inside the build step — after the cache step. `Swatinem/rust-cache` computes its key when the action runs, before any step, so step-local values are invisible to it. The result is a loop that never converges: the key never changes, so restores are exact hits, an exact hit makes the post-run save a no-op, and cargo invalidates the restored artifacts anyway because the flags differ. Those legs have been rebuilding cold on every run. Both values move to job-level `env:` ahead of the cache step, driven by new `lto:`/`codegen_units:` matrix fields, and are forwarded into the containers with `-e` since `docker run` inherits nothing. Every leg's cache key shifts once as a result, so expect one cold rebuild. A `Report peak memory` step is added so whether these legs fit is a number rather than an inference from whether the runner survived. It produced the figures above. ## Not included Moving these legs to native arm64 runners. It would retire the zig cross path, the `AT_HWCAP2` workaround and the `TARGET_CC` override, and arm64 runners are billed roughly 37% below x64 at equal core count — but the `lts-debian-aarch64` image exists to link against the manylinux2014 sysroot's glibc 2.17, and building natively on ubuntu-24.04 would raise the minimum glibc for every published aarch64 binary. That is a user-facing decision, not a CI cleanup. Dropping these legs to smaller runners, which is where the real cost saving is — larger runners are billed even on public repos. On these numbers it is not available yet: musl at 16516 MiB is about 130 MiB over what a 16 GB standard runner has. Worth revisiting as a follow-up. ## Testing Cargo's rustflags precedence was checked locally rather than taken from the docs, since getting it wrong would silently change the published binaries. With a throwaway crate carrying both a `target.'cfg(all())'` and a per-target rustflags table: setting `RUSTFLAGS` discards both, and setting it to the empty string discards them too. That rules out routing the linker flag through a job-level `RUSTFLAGS`, because `env:` keys cannot be conditionally omitted and every other leg would then silently lose the `target-cpu`/`target-feature` settings in `.cargo/config.toml` — `+avx2` on x86_64 and `-crt-static` on aarch64-musl. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Will JonesandClaude Opus 5
|
217ea1a799 |
ci: use thin LTO and a larger runner for the Windows wheel build (#3716)
The Windows wheel job is the slowest job in the PyPI release workflow. Fat LTO of the cdylib is single-threaded and the peak-memory step of the build, so it does not get faster with more cores — and it has already caused rustc-LLVM OOM on the Windows runners for the nodejs builds. Switch the job to thin LTO with 16 codegen units on a `windows-2025-8x-x64` runner, trading some runtime performance on our least performance-sensitive platform for build time. This matches what the nodejs Windows builds in `npm-publish.yml` already do. `pypi-publish.yml` is in this workflow's `pull_request` paths filter, so this PR triggers a dry-run build that shows the new timing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 LanceDB RobotandYang Cen
|
cf27f6902e |
chore: update lance dependency to v11.0.0-beta.16 (#3992)
Updates Lance dependencies and Java lance-core to v11.0.0-beta.16. Also narrows the dependency updater's package matching so the local LanceDB crate remains a path dependency. Lance tag: https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.16 --------- Co-authored-by: Yang Cen <bubble-cal@outlook.com> |
||
|
Xuanwo
|
f76ee304b8 |
ci: isolate remote Rust tests (#3998)
The Linux Rust job can exhaust its disk after restoring a large fallback target cache and compiling multiple feature graphs into one target directory. Run remote tests in an independent job with registry-only caching, and run the simple example with all features so it reuses the preceding build artifacts. This preserves remote coverage and fork behavior while preventing all-features and remote-only artifacts from accumulating together. Failure evidence: https://github.com/lancedb/lancedb/actions/runs/32467317540/job/96726650990 |
||
|
Xuanwo
|
4ba2421254 |
refactor(python): require pydantic v2 (#3990)
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2 APIs throughout. This removes dual-version behavior from schema conversion, query serialization, embedding models, and Function wire models while preserving their existing public and canonical-wire behavior. The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared compatibility floor remains covered. |
||
 Wyatt AltandClaude Opus 5
|
790d0c684c |
docs(ci): clarify tag input on codex-update-lance-dependency (#3924)
Say what resolving "latest" actually does: pick the newest release, preferring stable over pre-release, and skip the run if it is not newer than the version pinned in Cargo.toml. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 LanceDB RobotandYang Cen
|
031c3585a8 |
chore: update lance dependency to v11.0.0-beta.7 (#3925)
Updates the Rust workspace Lance dependencies and Java lance-core dependency to v11.0.0-beta.7. No compatibility fixes were required; full-workspace Clippy passes with warnings denied. Lance tag: https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.7 --------- Co-authored-by: Yang Cen <159225399+BubbleCal@users.noreply.github.com> |
||
|
Xuanwo
|
5acce6782e | ci(docs): report link checker failures through issues (#3909) | ||
|
Xuanwo
|
b1cfe6edb1 |
ci(docs): add scheduled doc link check (#3888)
The docs have no link checking at all, so external links rot silently: a trial run already found `docs/src/python/python.md` pointing at `lancedb.github.io/lance-namespace`, which returns 404 since the repository moved to the lance-format org. Checking external links on the blocking path would be the wrong trade: third-party hosts rate-limit automated clients, reject non-browser user agents, and go down temporarily, so any of them having a bad minute would turn unrelated PRs red. Following lance-format/lance#8315, this adds a daily `lychee` run that reports broken links into a single tracking issue, rewritten in place on each run and closed automatically once every link resolves. The scan job runs the downloaded lychee binary with a read-only token; everything that writes lives in a separate report job, and a non-verdict lychee exit fails the run instead of publishing a bogus report. The check is restricted to http(s) links because much of `docs/src` is generated API reference (the `js/` tree comes from `npm run docs`) and the hand-written pages use mkdocstrings cross-references and nav-relative paths that only resolve in the site mkdocs builds, so relative links would be reported as broken on every run. The one broken link the trial run surfaced is fixed here; after the fix, a local run over all 154 files reports 0 errors across 216 unique links. |
||
|
001237c7a4 |
chore: update lance dependency to v11.0.0-beta.2 (#3886)
Updates the Lance dependencies and Java lance-core to v11.0.0-beta.2. Includes required compatibility fixes for the LanceFileVersion module move and the updated GooseFS/OpenDAL dependency. Lance tag: https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.2 --------- Co-authored-by: Daniel Rammer <hamersaw@protonmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Will JonesandClaude Opus 5
|
ff50e698cf |
ci: cut Actions cost by moving builds to free runners and fixing caches (#3735)
Standard GitHub-hosted runners are free on public repos, so all Actions spend here is on the `*-8x-*` / `4x` larger runners. Measured over 30 days at current (post-Jan-2026) larger-runner rates, that is ~$1,400/mo, and `npm-publish` is ~70% of it. ## Changes **Fat LTO was forcing builds onto large runners.** `[profile.release]` in `.cargo/config.toml` sets `lto = "fat"` with `codegen-units = 1`, which is single-threaded and the peak-memory step. The macOS `npm-publish` build was 111 of its 113 minutes in one `napi build` step, making it the critical path of the whole publish pipeline. The ThinLTO override already applied to Windows now covers macOS too, and both Windows builds move from `windows-2025-8x-x64` to the free standard `windows-2025`. **The npm-publish cargo cache never existed.** There are zero caches with its key prefix. The key was static, so `actions/cache` (which only writes on a miss) could never refresh it, and a multi-GB release `target/` per target could never fit the repo's 10 GB budget anyway. Now caches only the crate registry, keyed on `Cargo.lock`. The docker builds also mounted `.cargo/registry/*` while the cache saved `.cargo-cache`, so containers re-downloaded the registry every run. **Cache eviction thrash.** Repo cache usage is 10.4 GB against GitHub's 10 GB cap, so every PR run evicted main's warm entries. `rust.yml` and `nodejs.yml` now restore everywhere but only save from `main`. **npm-publish moves to nightly + tags** instead of every push to main (~90/month). The cross-compiled targets do need watching, so `report-failure` now fires on scheduled runs, and dedupes onto an existing open issue rather than filing one per night. **rust.yml aarch64-pc-windows-msvc** cross-compiled its tests and then skipped them, paying full codegen and link cost for a compile check. `windows-11-arm` is now GA and free on public repos, so it builds and tests natively. Its test step also passes `--target` — without it cargo used `target/ci/` rather than `target/<triple>/ci/` and rebuilt the entire dependency graph a second time. **pypi-publish.yml had no concurrency group**, so force-pushes left a ~74 minute Windows job running. ## What is cost vs. wall-clock | Change | Cost | Wall-clock | |---|---|---| | Windows npm-publish → free runners | **−$570/mo** | slower per job (8→4 cores) | | npm-publish nightly | **−$125/mo** | — | | pypi-publish concurrency | small | — | | macOS ThinLTO | $0 (already free) | **−~50 min** per release | | rust aarch64 Windows native | $0 (already free) | **−~25 min** | | rust `--target` on test step | $0 | large, avoids a second full build | | rust-cache `save-if` | small | faster via real cache hits | ## Risks - The two Windows builds now have 4 cores instead of 8 and ~14 GB of free disk. If they fail, it is most likely disk rather than memory; fallback is `windows-2025-4x-x64`, which still halves that line. - `windows-11-arm` has a thinner toolset (choco/vcpkg/protoc under emulation) and this enables a test step that has never run, so it may surface real aarch64 failures. That is the point, but it is the change most likely to need iteration. - ThinLTO applies to published macOS and Windows binaries, typically within a few percent of fat LTO. Linux release builds are untouched. ## Follow-ups - `python.yml` `pydantic1x` (37 min) and `Doctest` (33 min) each rebuild the extension from source via `pip install -e .` with no Rust cache; they should consume the wheel the `linux` job already builds. Worth ~$235/mo and ~70 min of compute per run. Separate PR. - The three `ubuntu-2404-8x-x64` npm-publish builds (~$420/mo at the old cadence) are the remaining large-runner spend; `aarch64-unknown-linux-gnu` could run natively on free `ubuntu-24.04-arm`. Worth doing after this lands so the ThinLTO change can be validated first. - The wheel composite actions declare `python-minor-version` as required but never use it, and every caller omits it (actionlint warns). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Will Jones
|
bf15655c83 |
chore: unify SDK versions and release tags on a single line (#3714)
Python was versioned and tagged separately from the Rust, Java, and Node.js SDKs, and had drifted three minor versions ahead (0.36 vs 0.33). Users had no way to tell which Python version corresponded to which Rust or Node release, and the gap had no meaning behind it. This unifies the two tracks so there is one version and one tag for all four SDKs. ## Version The shared version is set to `0.37.0-beta.0`. Python continues its own sequence (highest published: 0.36 → 0.37) while Rust, Java, and Node.js jump 0.33 → 0.37 to meet it. Picking Python's next minor means Python users see no discontinuity at all, and only the other SDKs skip forward. Note that `main` trails the `release/v0.32` branch on both lines (main is at 0.32.0-beta.3 / 0.35.0-beta.3; the release branch carries 0.33.0-beta.0 / 0.36.0-beta.0), so 0.37 is chosen to clear the highest tag on either branch. Every index stays monotonic: | index | publishes | last published | next | |---|---|---|---| | PyPI | stable only | 0.34.0 | 0.37.0 | | Fury | previews | 0.36.0b0 | 0.37.0-beta.1 | | npm | both | 0.33.0-beta.0 | 0.37.0-beta.1 | | crates.io | stable only | 0.31.0 | 0.37.0 | | Maven | both | 0.33.0-beta.0 | 0.37.0-beta.1 | A one-time jump for three SDKs, versus explaining the offset indefinitely. ## Mechanism * `python/.bumpversion.toml` is removed. `python/Cargo.toml` — the source of the Python package version, since `pyproject.toml` declares `dynamic = ["version"]` — becomes a tracked file of the root config. Its `cargo update -p lancedb-python` pre-commit hook is dropped as redundant: `ci/update_lockfiles.sh` already refreshes every workspace member version in `Cargo.lock`. * `pypi-publish.yml` triggers on `v*` instead of `python-v*`, so one tag releases all four packages. `ci/bump_version.sh` and `make-release-commit.yml` lose their now-dead tag-prefix and per-language plumbing, including the `python` / `other` dispatch inputs. * The two byte-identical GH release jobs in `npm-publish.yml` and `pypi-publish.yml` are replaced by a single `gh-release.yml`. One release per tag, named `LanceDB vX.Y.Z`, instead of separate "Python LanceDB" and "Node/Rust LanceDB" releases for the same commit. The trade-off: there is no longer a way to ship a Python-only patch without also releasing crates.io, Maven, and npm. That is the cost of making drift structurally impossible. ## Beta releases marked "Latest" (#3666) Both GH release jobs used: ```yaml prerelease: ${{ contains('beta', github.ref) }} ``` The arguments are reversed. `contains(search, item)` asks whether *`search`* contains *`item`*, so this evaluated "does the literal string `'beta'` contain `refs/tags/python-v0.35.0-beta.2`?" — always `false`. Every beta was published as a full release, and GitHub awards "Latest" to the newest non-prerelease. The new workflow derives the flag from the parsed version rather than the raw ref, and sets `make_latest` explicitly: ```yaml prerelease: ${{ steps.extract_version.outputs.prerelease }} make_latest: ${{ steps.extract_version.outputs.prerelease == 'false' }} ``` npm was never affected (`--tag preview` uses correct bash), and PyPI already excludes pre-releases from resolution. This only fixes releases published from here on. Already-published betas need a one-time backfill: ```shell gh api --paginate /repos/lancedb/lancedb/releases \ --jq '.[] | select(.prerelease == false) | select(.tag_name | test("beta")) | .id' \ | xargs -I{} gh api -X PATCH /repos/lancedb/lancedb/releases/{} -F prerelease=true ``` ## Verification Ran `ci/bump_version.sh` end-to-end against this branch with the release tooling installed: * `preview` → tags `v0.37.0-beta.1` (previous tag `v0.33.0-beta.0` detected, `pre_n` bump) * `stable` → tags `v0.37.0` * Both paths update `.bumpversion.toml`, `rust/lancedb/Cargo.toml`, `nodejs/Cargo.toml`, `python/Cargo.toml`, `nodejs/package.json`, the 7 `nodejs/npm/*/package.json` files, both Java poms, and `docs/src/java/java.md` together * `check_breaking_changes.py` resolves the last stable as `v0.31.0`, so the minor-version gate passes All five touched workflows parse as valid YAML and the pre-commit hooks pass. ## Notes for review * This targets `main` only, so it takes effect at the next release-branch cut. The in-flight `release/v0.32` branch still carries `v0.33.0-beta.0` / `python-v0.36.0-beta.0`; if we want the imminent stable to be 0.37.0, this needs to be applied there too. * Historical `python-v*` tags are left alone. The changelog builder scans `^v`, which does not match them, so the first unified release's notes will compute `fromTag` from the Rust/Node line only — a one-time gap in the Python-side changelog. * Pre-existing and not addressed here: `ci/update_lockfiles.sh --amend` amends the commit that `bump-my-version` has already tagged, so the lockfile update lands outside the tag on stable releases. Fixes #3666 |
||
|
Xuanwo
|
8450683b2a | chore: update lance dependency to v9.1.0-beta.4 (#3690) | ||
 Kobi Hikri
|
dfce767f4c |
ci: pin ad-m/github-push-action to a full commit SHA in the release job (#3677)
Hi, and thank you for LanceDB.
Small CI supply-chain hardening. In `make-release-commit.yml`, the
release job checks out with `LANCEDB_RELEASE_TOKEN` (a push-capable PAT)
and its final step pushes the version tag using a third-party action
pinned to a **mutable branch**:
```yaml
- name: Push new version tag
uses: ad-m/github-push-action@master
with:
github_token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
```
`@master` can move after review; whatever it points at then runs with
that release token in scope. This PR pins it to the commit behind the
current release (`v1.3.0` → `881a6320…`), keeping the version visible as
a comment. Behavior today is unchanged.
For transparency: I used AI assistance to spot and draft this; I
verified the workflow and resolved the SHA myself.
|
||
 Tobias
|
7b6ee0d655 |
feat(wheels): publish lancedb-compat for pre-Haswell x86_64 hosts (#3327)
Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge / Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled function. Per [westonpace's review](https://github.com/lancedb/lancedb/issues/3324#issuecomment-4328944354), the default `lancedb` wheel stays fast; pre-Haswell users get a separately-published `lancedb-compat` wheel. ## Summary - Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline). Same Python API (`import lancedb` works) — files install to the same namespace, so the two wheels conflict at install time and users pick one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` / `tensorflow-cpu`. - Generalizes `build_linux_wheel` and `upload_wheel` composites with optional `package-name` and `rustflags` inputs (defaults preserve the existing 4 `lancedb` matrix entries verbatim). - Documents the choice in `python/README.md`: `pip install lancedb-compat` for pre-Haswell hosts. The default `.cargo/config.toml` baseline is unchanged. ## Sequencing 1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in lance.~~ **Done — merged.** 2. lancedb's lance dep is bumped to a release that includes it (separate PR / normal cadence). 3. This PR's `lancedb-compat` wheel build path starts producing a wheel that runs on pre-Haswell hardware. **Maintainer setup**: register `lancedb-compat` on PyPI and configure trusted publishing. ## Verified end-to-end on Sandy Bridge Xeon E5-2609 Verification was done locally against a fork-pinned lance dep that includes the runtime dispatch implementation, using the same `RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI: ``` $ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release $ pip install ./target/wheels/lancedb-*.whl $ python verify.py PASS: import + simd dispatch + table create + vector search all work. ``` Pre-fix on the same CPU (default `pip install lancedb`): `Illegal instruction (core dumped)`. Full reproducer (deps + clone + build + verification): https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df. Fork-internal verification PR with the dep bump and full logs: [`tobocop2/lancedb#2`](https://github.com/tobocop2/lancedb/pull/2). ## Benchmarks — no regressions on modern CPUs from the lance-side change These are the numbers I ran for the lance PR, confirming the runtime dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that existing users install. Criterion, one machine, one session, base → PR, no `RUSTFLAGS` override. Full methodology, null experiments, and logs: [lance-format/lance#6630 benchmark comment](https://github.com/lance-format/lance/pull/6630#issuecomment-4933063394) and the [logs gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2). | benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade Lake (`avx512f`) | |---|---|---| | `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% | | `Cosine(f64, scalar)` | −0.34% | −1.94% | | `Cosine(u8, SIMD)` | +2.30% | +3.63% | | `Dot(f16, SIMD)` | −0.58% | +0.61% | | `Dot(f32, SIMD)` | +0.34% | **−6.08%** | | `Dot(f32, arrow_arity)` | +0.02% | −0.00% | | `L2(f32, scalar)` | −0.10% | −0.02% | | `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% | | **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** | | `L2(u8, SIMD)` | +0.42% | −3.11% | | `NormL2(f32, SIMD)` | −1.02% | −4.17% | | `NormL2(f64, SIMD)` | +3.51% | −0.58% | Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector width — improves 25–46%. --- To be transparent: this isn't my domain of expertise and the lance-side implementation is AI-generated. I verified it works end-to-end on the failing hardware. Happy to roll in feedback. |
||
|
LanceDB RobotandJack Ye
|
d8f0982ee8 |
chore: update lance dependency to v9.0.0-beta.23 (#3665)
Updates the Rust workspace Lance dependencies and Java lance-core from v9.0.0-beta.19 to v9.0.0-beta.23. No compatibility fixes were required; strict workspace Clippy and Rust formatting pass. Lance tag: https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.23 --------- Co-authored-by: Jack Ye <yezhaoqin@gmail.com> |
||
|
Will JonesandClaude Opus 4.8
|
0d9c87a079 |
ci(nodejs): move Windows builds to larger runner and use ThinLTO (#3634)
The `build - aarch64-pc-windows-msvc` node build job (and, marginally, the x86_64 one) had started hitting `rustc-LLVM ERROR: out of memory` while linking the `lancedb-nodejs` cdylib — most recently surfaced by #3526, which adds the goosefs backend (and its tonic/prost gRPC subtree) to the default node binary. The peak-memory step is the fat-LTO codegen (`lto=fat`, `codegen-units=1` from `.cargo/config.toml`), which merges the whole crate graph into a single LLVM module and runs single-threaded. It therefore neither parallelizes across cores nor fits in the 16 GB of the standard `windows-latest` runner as the dependency graph grows. This PR: - Moves both `*-pc-windows-msvc` node build jobs to `windows-2025-8x-x64` (more memory + cores). - Overrides the release profile to ThinLTO for just these jobs, via `CARGO_PROFILE_RELEASE_LTO=thin` / `CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16` in `pre_build`. ThinLTO parallelizes the cross-module optimization across the runner's cores and keeps peak memory well under the limit. Scoped so Python wheels and Rust release builds keep fat LTO. The larger runner alone would clear the OOM but waste the added cores on the single-threaded fat-LTO tail; ThinLTO is what makes the extra cores actually reduce wall-clock and gives durable memory headroom for future dependency growth. Tradeoff: ThinLTO can leave a small runtime-perf gap vs fat LTO for the node native binary, but it recovers most of it and is a common release configuration. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Xuanwo
|
22bf091de1 |
fix: avoid manifest writes for read-only directory namespace opens (#3635)
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687 for side-effect-free DirectoryNamespace read paths. This fixes root-level read-only table opens that previously could trigger `__manifest` creation through directory namespace construction, including Hugging Face bucket reads with read-only tokens. A LanceDB regression test now covers root listing operations without creating `__manifest`. Fixes #3633. |
||
 Raphael Malikian
|
f94673ae5e |
ci: update deprecated GitHub Actions to latest versions (Fixes #3577) (#3608)
Fixes #3577 ## Problem GitHub Actions is deprecating Node.js 20 on its runners. Multiple workflows in lancedb use action versions that target Node.js 20 (`actions/checkout@v4`, `actions/setup-node@v4`, `actions/cache@v4`, `actions/upload-artifact@v4`, `actions/download-artifact@v4`, `pnpm/action-setup@v4`). These are being force-run on Node.js 24, generating deprecation warnings. ## Solution Updated all deprecated actions to their latest major versions that support Node.js 24: | Action | Old Version | New Version | |--------|------------|-------------| | `actions/checkout` | @v4 | @v6 | | `actions/setup-node` | @v4 | @v6 | | `actions/cache` | @v4 | @v5 | | `actions/upload-artifact` | @v4 | @v7 | | `actions/download-artifact` | @v4 | @v8 | | `pnpm/action-setup` | @v4 | @v6 | Note: `actions/checkout@v6` and `actions/upload-artifact@v7` are already used in `pypi-publish.yml` — this PR extends the same versions to all remaining workflows. ### Files Changed - `.github/workflows/npm-publish.yml` — Updated checkout, setup-node, cache, upload-artifact, download-artifact, pnpm - `.github/workflows/nodejs.yml` — Updated checkout, setup-node, pnpm - `.github/workflows/python.yml` — Updated checkout - `.github/workflows/rust.yml` — Updated checkout - `.github/workflows/java.yml` — Updated checkout - `.github/workflows/java-publish.yml` — Updated checkout - `.github/workflows/cargo-publish.yml` — Updated checkout - `.github/workflows/docs.yml` — Updated checkout, setup-node - `.github/workflows/dev.yml` — Updated setup-node - `.github/workflows/codex-fix-ci.yml` — Updated checkout, setup-node, pnpm - `.github/workflows/codex-update-lance-dependency.yml` — Updated checkout, setup-node - `.github/workflows/license-header-check.yml` — Updated checkout - `.github/workflows/make-release-commit.yml` — Updated checkout - `.github/workflows/update_package_lock_run.yml` — Updated checkout - `.github/workflows/update_package_lock_run_nodejs.yml` — Updated checkout ## Verification - All 20 YAML files validated with `yaml.safe_load()` — no syntax errors - GitHub Actions CI will validate the actual action versions at runtime ## Changelog | Date | Change | Author | |------|--------|--------| | 2026-07-01 | Updated all deprecated Node 20 actions to latest versions across 15 workflow files | rtmalikian | --- **Disclosure:** This code was developed with assistance from DeepSeek-v4-pro (DeepSeek) via Hermes Agent (Nous Research). All changes were reviewed and verified for correctness. Signed-off-by: rtmalikian <rtmalikian@gmail.com> |
||
|
Xuanwo
|
7b874905fd |
ci: move Lance dependency bump flow into skill (#3475)
Moves the Lance dependency bump process into an in-repository skill so local agents and GitHub Actions share the same workflow definition. The update workflow is now an explicit, optional-tag entrypoint; latest-release resolution, duplicate PR handling, Java/Rust dependency updates, and Sophon follow-up are documented in the skill and backed by a small deterministic helper. |
||
|
Will JonesandClaude Opus 4.7
|
a3339b7bdd |
ci: drop manylinux2_17 wheel builds (#3455)
manylinux2_17 reached EOL in 2024 and pyarrow stopped publishing 2_17 wheels long ago. We already build manylinux2_28 wheels, so drop the 2_17 matrix entries. Fixes #3452 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Will Jones
|
b20cdc4f93 |
ci: fix pypi publish on mac/windows/arm (#3449)
The python-v0.32.0 publish run failed on every build matrix entry. Three independent issues: 1. **Mac and Windows**: `pypa/gh-action-pypi-publish` only runs on Linux, but was being called inline from each build job. 2. **Linux (all arches)**: `pypa/gh-action-pypi-publish` derives its docker image name from `github.action_repository`, which is empty when the action is invoked from inside a composite action (actions/runner#2473 — pypa's own `action.yml` references this bug). It falls back to `github.repository`, generating `docker://ghcr.io/lancedb/lancedb:<tag>`, which doesn't exist → `denied`. Only the ARM matrix entry surfaced this because it failed first and cancel-cascaded the rest. 3. **Windows**: `upload-artifact` in `build_windows_wheel` pointed at `python\target\wheels`, but maturin writes to the workspace-root `target/wheels`. The artifact was always empty. Also, `pypi-publish.yml` passed a `vcpkg_token` input that the composite doesn't declare. ## Changes - Build jobs (linux/mac/windows) now upload their wheels as `actions/upload-artifact` artifacts. - New Linux `publish` job downloads all wheel artifacts and runs the Fury or PyPA publish step directly (not via a composite), so `github.action_repository` resolves correctly. - Delete the unused `upload_wheel` composite action. - Drop the broken upload-artifact step inside `build_windows_wheel`. - Remove the bogus `vcpkg_token` input. - Fury upload now loops over all wheels instead of just the first. - Bump `actions/checkout`, `actions/upload-artifact`, `actions/download-artifact` to current major versions (Node 24) to clear deprecation warnings. - Bump Windows job timeout 60 → 90 minutes; previous run was cancel-timing-out on a 60m cap. - Use `rust-lld` as the Windows MSVC linker via `CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER`. `link.exe` is single-threaded and the long pole on Windows builds. Fixes #3445 ## Test plan - [x] Open this PR — `paths` filter triggers a dry-run build on all three platforms. - [x] Verify all three builds produce wheels. - [x] Confirm the `pypa/gh-action-pypi-publish` container actually starts (the actions/runner#2473 bug) via the `publish-dry-run` job pointed at TestPyPI. - [x] **REMOVE BEFORE MERGE**: drop the `publish-dry-run` job and the now-redundant `actions/upload-artifact` runs on PRs (currently always-on so the dry-run has wheels to publish). - [ ] After merge, cherry-pick onto `python-v0.32.0` and force-push the tag to re-trigger the publish. |
||
 
|
2d5298b6ee |
chore(deps): bump the rust-minor-patch group across 1 directory with 23 updates (#3382)
Weekly dependabot refresh of `Cargo.lock`. Dependabot's original PR also raised the lower-bound version requirements in `Cargo.toml` (arrow, tokio, aws-sdk-*, etc.) to match the new lockfile versions. That forces our library's consumers onto newer minimum versions and broke the MSRV check, which downgrades aws-sdk-* crates to verify they still build on Rust 1.91. Changes from the original: - Reverted all `Cargo.toml` requirement changes; `Cargo.lock` regenerated with `cargo update` within the existing ranges. The lockfile (and the binaries we ship) stays current on security fixes without bumping our public minimum versions. - Set `versioning-strategy: lockfile-only` in `.github/dependabot.yml` so future cargo dependabot PRs only touch `Cargo.lock`. Note: `aws-lc-rs` stays at 1.16.3 — `nodejs/Cargo.toml` pins it with `=`, which `lockfile-only` cannot move; bumping it needs a manual change. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Will Jones <will.jones127@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Will Jones <willjones127@gmail.com> |
||
|
Yang Cen
|
5d1c28922a |
feat(python): align to_pandas pandas kwargs (#3397)
## Feature This PR aligns LanceDB Python `to_pandas()` APIs with Lance pandas conversion capabilities while keeping LanceDB query-specific semantics intact. - Adds `blob_mode` and pandas `**kwargs` support to local table `to_pandas()`. - Delegates local `LanceTable.to_pandas()` to Lance dataset `to_pandas(blob_mode=..., **kwargs)`. - Keeps remote table `to_pandas()` unsupported with `NotImplementedError`. - Allows sync and async query `to_pandas()` to forward pandas kwargs after LanceDB `flatten` and `timeout` handling. Why we need this feature: Users can access Lance blob-aware pandas conversion from LanceDB local tables and can pass PyArrow pandas conversion options through table/query APIs without losing existing `flatten` or `timeout` behavior. How it works: The table API exposes a `BlobMode` literal type for `lazy`, `bytes`, and `descriptions`. Local tables call through to the backing Lance dataset. Query APIs do not add `blob_mode`; they materialize Arrow results, apply LanceDB flattening when requested, and then call `to_pandas(**kwargs)`. ## Validation - `uv run --frozen --extra tests pytest python/tests/test_table.py::test_table_to_pandas_default_matches_arrow python/tests/test_table.py::test_table_to_pandas_blob_bytes python/tests/test_table.py::test_table_to_pandas_kwargs python/tests/test_query.py::test_query_to_pandas_kwargs python/tests/test_query.py::test_query_timeout python/tests/test_remote_db.py::test_table_to_pandas_not_supported` - `uv run --frozen --extra dev ruff check python/lancedb/table.py python/lancedb/query.py python/lancedb/remote/table.py python/tests/test_table.py python/tests/test_query.py python/tests/test_remote_db.py` - `uv run --frozen --extra tests pytest python/tests/test_table.py python/tests/test_query.py python/tests/test_remote_db.py` Note: `python/uv.lock` was intentionally not committed in this branch. |
||
|
Will JonesandClaude Opus 4.7
|
81617fd3d9 |
ci(nodejs): switch from npm to pnpm 11 (#3373)
## Summary Switch the nodejs bindings and examples package from npm to pnpm 11 to pick up its stronger supply-chain defaults: - `minimumReleaseAge` defaults to 1 day, so newly-published (potentially compromised) versions aren't resolved into installs for at least 24h. - Install lifecycle scripts (`preinstall`/`install`/`postinstall`) are no longer run for arbitrary transitive deps; only an explicit allowlist may run them, and unapproved scripts cause install to fail (`strictDepBuilds: true`). - Audit uses GHSA IDs and `--fix=update` to add patched versions to `minimumReleaseAgeExclude`. This is the same class of protection that would have blunted the recent TanStack/`@uipath`/etc. compromise discussed in the [Aikido write-up](https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised). ## Changes - Replace `nodejs/package-lock.json` and `nodejs/examples/package-lock.json` with `pnpm-lock.yaml`. - Pin pnpm via `packageManager: pnpm@11.1.1` in both `package.json`s. - Add `pnpm-workspace.yaml` with the four build-script packages we actually need: `@biomejs/biome`, `onnxruntime-node`, `protobufjs`, `sharp`. Everything else is blocked from running install scripts. - Update package.json scripts (`npm run X` → `pnpm X`). - Update workflows: `.github/workflows/nodejs.yml`, `.github/workflows/npm-publish.yml`, and `.github/workflows/codex-fix-ci.yml` — install pnpm via `pnpm/action-setup@v4` and switch `setup-node` caches to `pnpm-lock.yaml`. - Refresh `nodejs/AGENTS.md`, `nodejs/CLAUDE.md`, and `nodejs/CONTRIBUTING.md`. `docs/package-lock.json` is **not** touched — out of scope for this PR. ## Test plan - [ ] `Lint` job (lint Rust/TS + examples lint) passes on CI. - [ ] `Linux (NodeJS 18/20)` build+test passes, including the examples test step. - [ ] `macos` build+test passes. - [ ] `NPM Publish` workflow's PR dry-run completes (build matrix + test matrix + dry `npm publish`). - [ ] No new install-script approvals are required at install time. ## Follow-ups - `update_package_lock_run_nodejs.yml` references a composite action path that doesn't exist (`./.github/workflows/update_package_lock_nodejs`); it was already broken pre-PR. We may want to either delete this workflow or rewrite it for pnpm in a follow-up. - Consider migrating `docs/` to pnpm in a separate PR. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
  Octopusandocto-patch
|
5338aeb006 |
ci: avoid passing GPG passphrase on command line in Java publish workflow (#3313)
Fixes #3299 ## Problem Two security issues exist in `.github/workflows/java-publish.yml`: 1. **`gpg-passphrase` input is misused**: `actions/setup-java`'s `gpg-passphrase` input expects the **name** of an environment variable (default: `GPG_PASSPHRASE`), not the secret value itself. The previous value `${{ secrets.GPG_PASSPHRASE }}` was setting the env var name to the actual secret, which is incorrect. 2. **Passphrase visible on the command line**: `-Dgpg.passphrase=${{ secrets.GPG_PASSPHRASE }}` passes the GPG passphrase as a Maven system property argument, making it visible in process listings and potentially echoed in debug logs — a supply-chain security risk for release workflows. ## Solution - Fix `gpg-passphrase: MAVEN_GPG_PASSPHRASE` — use the correct env var name so `actions/setup-java` generates a proper Maven `settings.xml` entry that reads from `MAVEN_GPG_PASSPHRASE`. - Remove `-Dgpg.passphrase=...` from the Maven CLI invocation. - Add `MAVEN_GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}` to the `env:` block of the Publish step, so the passphrase is available as an environment variable rather than a CLI argument. ## Testing The Java publish workflow only runs on tag pushes, so this cannot be exercised in a PR build. The logic change is straightforward: `actions/setup-java` is documented to write a `settings.xml` that reads `<gpg.passphrase>` from the named env var, and `maven-gpg-plugin` picks it up from there without any CLI argument. Co-authored-by: octo-patch <octo-patch@github.com> |
||
|
Will JonesandClaude Opus 4.7
|
d135c18db6 |
ci: add cargo-deny configuration and CI check (#3307)
Adds a `deny.toml` at the workspace root and a `deny` CI job that runs `cargo deny check` on every PR. Catches yanked crates, license drift, banned or wildcard dependencies, unapproved sources, and new RUSTSEC advisories. As part of wiring this up: - Updated `aws-lc-rs` 1.13.0 → 1.16.3 / `aws-lc-sys` 0.28.0 → 0.40.0 to clear four 2026 AWS-LC advisories (timing side-channel, PKCS7 bypass, CRL scope). Removed the `=0.28.0` workaround pin; the original build failure no longer reproduces. - Updated `bytes`, `zlib-rs`, `rand`, `rustls-webpki`, `lz4_flex` to clear their current advisories. - Marked `lancedb-nodejs` and `lancedb-python` as `publish = false` and pinned `lzma-sys` from `*` to `0.1` so `bans.wildcards = "deny"` can be enforced. 10 remaining advisories have no safe upgrade available (transitive via opendal, lance, datafusion, async-openai, aws-sdk on the legacy rustls 0.21 chain). Each is ignored in `deny.toml` with a per-entry rationale and a link to the RUSTSEC advisory. New advisories still fail CI. Fixes #3297 --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Will JonesandClaude Sonnet 4.6
|
ef399de092 |
ci: switch PyPI publish to OIDC trusted publishing (#3302)
## Summary - Replaces `LANCEDB_PYPI_API_TOKEN` (long-lived token) with OIDC trusted publishing via `pypa/gh-action-pypi-publish` - Adds `id-token: write` permission to linux/mac/windows jobs - Removes `twine`-based upload and the `pypi_token` input from `upload_wheel` composite action - Enables PEP 740 Sigstore attestations on published wheels as a bonus After merging, rotate/revoke the `LANCEDB_PYPI_API_TOKEN` secret. Closes #3294 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Xuanwo
|
c54888a83a |
refactor(python): remove legacy tantivy FTS support (#3282)
This follows the Rust-side Tantivy removal by deleting the remaining Python Tantivy runtime, tests, and packaging references. It also turns the legacy Python-only Tantivy parameters into explicit errors and stops reading legacy `_indices/fts` directories so Python FTS is fully native-only. |
||
|
Will Jones
|
ba6c44abc9 |
ci: add top-level permissions to GHA workflows (#3255)
Adds `permissions: contents: read` to the 10 workflows that had no top-level permissions block. Workflows that already declared permissions, or individual jobs that need elevated permissions (`issues: write`, `pull-requests: write`, `contents: write`), are left unchanged. Affected workflows: `dev.yml`, `java-publish.yml`, `java.yml`, `license-header-check.yml`, `nodejs.yml`, `pypi-publish.yml`, `python.yml`, `rust.yml`, `update_package_lock_run.yml`, `update_package_lock_run_nodejs.yml` |
||
|
Will JonesandClaude Sonnet 4.6
|
2807ad6854 |
chore: bump Rust toolchain from 1.91.0 to 1.94.0 (#3257)
Bumps the Rust toolchain to 1.94.0 (latest installed) to unblock CI failures caused by the AWS SDK's MSRV requirement. No lint fixes were needed. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
 aikido-autofix[bot]
|
4714598155 |
ci: mitigate template injection attack in build_linux_wheel (#3195)
This patch mitigates template injection vulnerabilities in GitHub Workflows by replacing direct references with an environment variable. Aikido used AI to generate this PR. High confidence: Aikido has a robust set of benchmarks for similar fixes, and they are proven to be effective. Co-authored-by: aikido-autofix[bot] <119856028+aikido-autofix[bot]@users.noreply.github.com> |
||
  Esteban GutierrezandEsteban Gutierrez
|
d8fc071a7d |
fix(ci): bump AWS SDK MSRV pins to March 2025 release (#3179)
Lance v4.1.0-beta requires the default-https-client feature on aws-sdk-dynamodb and aws-sdk-s3, which was introduced in the March 2025 AWS SDK release. Update all AWS SDK pins to versions from the same AWS SDK release to maintain internal dependency compatibility. Co-authored-by: Esteban Gutierrez <esteban@lancedb.com> |
||
|
Will JonesandClaude Opus 4.6
|
9a5b0398ec |
chore: fix ci (#3139)
* Move away from buildjet, which is shutting down runners for GHA [^1] * Add `Cargo.lock` to build jobs, so when we upgrade locked dependencies we check the builds actually pass. CI started failing because dependencies were changed in #3116 without running all build jobs. * Add fixes for aws-lc-rs build in NodeJS. [^1]: https://buildjet.com/for-github-actions/blog/we-are-shutting-down --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Esteban GutierrezandClaude Opus 4.6
|
6530d82690 |
chore: dependency updates and security fixes (#3116)
## Summary - Update dependencies across Rust, Python, Node.js, Java, Docker, and docs - Pin unpinned dependency lower bounds to prevent silent downgrades - Bump CI actions to current major versions 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Xuanwo
|
52ce2c995c |
fix(ci): only run npm publish on release tags (#3093)
This PR fixes the npm publish dry-run failure for prerelease versions without changing the existing workflow trigger behavior. The publish step now detects prerelease versions from `nodejs/package.json` and always appends `--tag preview` when needed. Context: - On `main` pushes, the workflow still runs `npm publish --dry-run` by design. - Recent failures were caused by prerelease versions (for example `0.27.0-beta.3`) running without `--tag`, which npm rejects. - The previous `refs/tags/v...-beta...` check did not apply on branch pushes, so dry-run could fail even though release tags worked. |
||
|
Will Jones
|
f91d2f5fec |
ci(python): pin maturin to work around bug (#3088)
Work around for https://github.com/PyO3/maturin/issues/3059 |
||
|
Will Jones
|
310967eceb | ci(rust): fix linux job (#3076) | ||
|
Will JonesandClaude Sonnet 4.6
|
367262662d |
feat(nodejs): upgrade napi-rs from v2 to v3 (#3057)
## Summary - Upgrades `@napi-rs/cli` from v2 to v3, `napi`/`napi-derive` Rust crates to 3.x - Fixes a bug ([napi-rs#1170](https://github.com/napi-rs/napi-rs/issues/1170)) where the CLI failed to locate the built `.node` binary when a custom Cargo target directory is set (via `config.toml`) ## Changes **package.json / CLI**: - `napi.name` → `napi.binaryName`, `napi.triples` → `napi.targets` - Removed `--no-const-enum` flag and fixed output dir arg - `napi universal` → `napi universalize` **Rust API migration**: - `#[napi::module_init]` → `#[napi_derive::module_init]` - `napi::JsObject` → `Object`, `.get::<_, T>()` → `.get::<T>()` - `ErrorStrategy` removed; `ThreadsafeFunction` now takes an explicit `Return` type with `CalleeHandled = false` const generic - `JsFunction` + `create_threadsafe_function` replaced by typed `Function<Args, Return>` + `build_threadsafe_function().build()` - `RerankerCallbacks` struct removed (`Function<'env,...>` can't be stored in structs); `VectorQuery::rerank` now accepts the function directly - `ClassInstance::clone()` now returns `ClassInstance`, fixed with explicit deref - `Vec<u8>` in `#[napi(object)]` now maps to `Array<number>` in v3; changed to `Buffer` to preserve the TypeScript `Buffer` type **TypeScript**: - `inner.rerank({ rerankHybrid: async (_, args) => ... })` → `inner.rerank(async (args) => ...)` - Header provider callback wrapped in `async` to match stricter typed constructor signature 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
LanceDB RobotandJack Ye
|
8cef8806e9 |
chore: update lance dependency to v3.0.0-beta.5 (#3058)
## Summary - Bump Lance Rust dependencies and Java `lance-core` to v3.0.0-beta.5 (refs/tags/v3.0.0-beta.5). - Update workspace toolchain and dependency defaults needed for the new Lance release. - Resolve new clippy lint defaults introduced by the toolchain update. ## Validation - `cargo clippy --workspace --tests --all-features -- -D warnings` - `cargo fmt --all` --------- Co-authored-by: Jack Ye <yezhaoqin@gmail.com> |
||
|
Will JonesandClaude Opus 4.6
|
a3cd7fce69 |
fix: update DatasetConsistencyWrapper to accept same-version updates (#3055)
## Summary `DatasetConsistencyWrapper::update()` only stored datasets with a strictly newer version. This caused `migrate_manifest_paths_v2` to silently drop its update since the migration renames files without bumping the dataset version. The subsequent `uses_v2_manifest_paths()` call would then return the stale cached dataset. Changed the version check from `>` to `>=` so same-version updates are accepted. ## Test plan - [x] Existing `test_create_table_v2_manifest_paths_async` Python test should pass - [x] Existing `should be able to migrate tables to the V2 manifest paths` NodeJS test should pass - [x] All dataset wrapper unit tests pass locally 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Jack Ye
|
dcb1443143 |
ci: add codex fix ci workflow (#3022)
Similar to the lance one added recently: https://github.com/lance-format/lance/actions/workflows/codex-fix-ci.yml |