Existing Rust and Python blob range calls must keep working when only
the SDK is upgraded. Add negotiated batched range reads across Rust,
Python sync/async and TypeScript, preserving ordered NULLs, duplicates
and branch/version reads with splitting and reassembly at 1,024 ranges
or 64 MiB per request.
Only a successful describe response advertising `x-lancedb-blob-ranges:
1` selects the batched endpoint. Missing or unknown capability values
select the existing descriptor-query plus BlobFile Range GET route
before the operation. Endpoint failures never switch protocols. HTTP 400
preserves the server message and request ID as Rust `InvalidInput` /
Python `ValueError`; other errors retain their existing semantics.
The operation binds both numeric version and immutable snapshot identity
from its producing query or first range response. Every later partition
and oversized-range piece sends that original identity and requires an
equal response identity before decoding. Same-name table recreation
therefore fails explicitly instead of returning mixed old/new payloads;
no partial array is returned on later failure.
Targets `main`. The read-buffering prerequisite #4979, safe-write
prerequisite #4977 and deferred-query identity prerequisite #4973 are
merged into main; #5000 also supplies the merged SQL timeout-test
prerequisite. Server support is [Sophon
#10219](https://github.com/lancedb/sophon/pull/10219), including #10211.
Exact SDK `1497b508324662180086a4d1e0a6eb9e0b3970db` and server
`f39712c2c150b220eb831e00f6663c9eef32dce5` passed a real local SDK →
REST → PE experiment: one Rust harness test, 14 assertion groups, 38
REST range requests and 34 PE RPCs. All four sync/async × warm/cold
table-recreation cases failed explicitly before the second descriptor
lookup or PE dispatch, with no mixed payload or partial result. The
other groups retained 1,025-range success, 70 MiB reassembly,
branch/version reads, overwrite snapshot consistency, error behavior and
TypeScript buffer ownership. Bindings were rebuilt and their loaded
hashes verified. This is local interoperability evidence, not Cloud,
mixed-fleet or deployed header-forwarding acceptance.
The server dependency upgrade, External/cache/URI-policy acceptance and
validated rollout remain required. Enforcing PE support must precede QN
and SDK enablement; identity does not retain versions or freeze External
objects. This PR does not claim deployed support or complete ENT-3100/GA
acceptance.
Published documentation retains the Early access badge and released
Enterprise instructions, including explicit fetches for copying. The new
Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example,
future streaming restrictions and GA wording remain source-only MDX
drafts until release. The correction distinguishing released Enterprise
SQL bytes from SDK descriptors remains visible. Documentation validation
checked comment boundaries, visible instructions, Python example syntax
and whitespace; the site was not built. The main integration keeps these
release gates unchanged.
Main integration at `0491d9bdcc65ce5d52ea93e1083c89e951ea4f9d` normally
merges main `9b13e8f72530c8b1950c09685f6920711154a539`, including its
Lance beta.18 dependency. Focused current-source validation passed: 493
remote-table/query/Blob/IPC/write-progress Rust tests, 23 shared
create/write-body tests, 59 native Blob integration tests and 7 public
HTTP range tests. Rebuilt Python passed 468 selected tests (one optional
PyAV test skipped), and rebuilt TypeScript passed 194 selected tests
across four suites; lint/format, TypeScript compilation and API
documentation generation passed. Shared tests include actual HTTP
multipart failures, acknowledgment boundaries, dictionary compaction and
native Blob compaction. The 598 unrelated Jest cases excluded by the
focused shared-suite selector were not run.
The retained 14-group joint REST/PE experiment used the earlier exact
candidates stated above. Operation-identity/range/IPC code is unchanged,
but dependencies and shared write interfaces changed with main; the new
local checks validate the SDK combination and do not constitute a fresh
final-head SDK/server joint, Cloud or rollout run. Early access and
every remaining server/External/cache/policy/dependency/rollout
condition remain in force.
CI snapshot on this head (2026-11-12 18:30 UTC): `build-no-lock` failed
in the standalone `ci/tests/test_build_no_lock.py` fixture at the
retained-rlib assertion, before the actual SDK no-lock build. The
fixture, helper and Rust workflow are byte-identical to main; the
fixture does not load SDK source. The underlying CI/environment cause is
not diagnosed here. Other jobs remained pending/running at the snapshot,
so CI is not reported as passing. Failure:
https://github.com/lancedb/lancedb/actions/runs/43280277756/job/131399418284.
Updates the Rust workspace and Java Lance dependencies to
[v13.1.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.1.0-beta.1)
and refreshes Cargo.lock. Removes a redundant `pub(crate)` declaration
in the private Node.js blob module to satisfy the required Clippy check.
Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `pnpm build`, and `pnpm test
__test__/blob.test.ts --runInBand` (26 tests passed) in `nodejs`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Closes#4288.
Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange`
and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an
optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When
the signal aborts, the call rejects with `signal.reason` (an
`AbortError` unless the caller gave a reason) and the native future is
dropped, so the read stops instead of running to completion. A signal
that is already aborted rejects before any read starts.
How it works:
- On the Rust side, these five methods are now sync `#[napi]` methods
that return `env.spawn_future(...)`. The future is wrapped in
`futures::future::abortable`, and the abort handle is registered with
napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so
it can't be an argument of an `async fn`, which is why the methods
changed shape. The TypeScript signatures are unchanged apart from the
new optional argument.
- napi-rs installs its own `onabort` on the signal it receives, which
would overwrite a handler the caller set. So the TypeScript wrapper
passes a private `AbortController` signal to the native call for each
request, forwards the caller's abort to it with `addEventListener(..., {
once: true })`, and removes the listener when the call settles.
Tests in `table.test.ts`:
- an already aborted signal rejects on all five methods, and the handle
still reads afterwards
- a custom abort reason is what the call rejects with
- a read with a signal that never aborts returns normally and leaves no
`abort` listener on the signal
- aborting a 32 MiB `readRange` right after it starts rejects with
`AbortError`, and the same handle reads again afterwards
The Node docs under `docs/src/js` are regenerated.
I didn't add a test for where the `read()` cursor ends up after a
mid-flight abort on a local table. The remote `BlobFile` documents that
a cancelled read leaves the cursor unchanged, but I haven't confirmed
the same for the native Lance handle, so the docs don't claim it.
Closes#4300.
The TypeScript `BlobFile` only had `size()`, `read()` and `readRange()`,
while the Rust `BlobFile` behind it also has `seek`, `tell`,
`read_up_to`, `read_ranges` and `close`. This exposes those on the Node
binding:
- `read(maxBytes?)` reads at most `maxBytes` from the cursor (Rust
`read_up_to`). Without the argument it reads to the end, as before.
- `readRanges([{ start, end }, ...])` reads several half-open ranges and
returns one buffer per range, in order. Like `readRange`, it does not
move the cursor.
- `seek(position)` and `tell()` move and report the cursor.
- `close()` releases the handle, and `isClosed()` reports it. Every
read, `seek` and `tell` after `close()` fails with "blob file is already
closed". A second `close()` does nothing. The binding keeps its own
`closed` flag, the same way the Python `BlobFile` does.
Offsets and lengths are `bigint`, like `readRange` and `size()`.
`BlobRange` is exported as a type.
The new tests in `table.test.ts` cover chunked reads with
`read(maxBytes)`, `seek` and `tell`, a negative seek position,
`readRanges` order and cursor, a `readRanges` end past the blob size,
and reads after `close()`. The Node docs under `docs/src/js` are
regenerated with `pnpm run docs`.
I left the `ReadableStream`/async iterator idea from the issue out of
this PR. The AbortSignal work in #4288 touches the same methods, so it
can follow on top of this.
Update the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v12.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.18).
Fix redundant visibility declarations in the Node.js Rust blob helpers
required by the workspace Clippy check.
Validated with workspace Clippy (all features and tests, warnings
denied), cargo fmt, pnpm build, and 13 targeted Node.js blob tests.
this PR blob v2 field helpers and reads to the Node SDK.
`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.
Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.
Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.
```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
"videos",
[{ id: 1n, video }],
{
schema: new Schema([
new Field("id", new Int64()),
blob("video"),
]),
},
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);
```
### Testing
- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
Xuanwo
Bilal Atique
Drew