mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-18 09:42:27 +00:00
feat(node): accept ArrayBuffer, Blob/File, and URL as blob column inputs (#4313)
Closes #4270. ## Summary Blob v2 columns now accept more of the containers TypeScript code commonly has, both as the column value and inside `{ data }` / `{ uri }`: | Input | Handling | |---|---| | `ArrayBuffer` | `new Uint8Array(value)`, no copy | | `URL` | stored as `value.href` | | `Blob` / `File` | `await value.arrayBuffer()` in the async write paths: `createTable`, `add`, `mergeInsert` | `makeArrowTable` is synchronous, so it can't read a `Blob`. Given one, it throws an error that names the field and row and points to the async APIs or `new Uint8Array(await value.arrayBuffer())`. It does not write a null blob. ### Types `BlobInput` is now the accepted input type. It is widened to the types above and exported from the package entry point with `BlobData` and `BlobUri`, so callers can annotate rows and `tsc` catches bad inputs. The struct forms are a union that requires exactly one of `data` / `uri`. The old normalized `{ data, uri }` shape is renamed to `BlobValue` (internal). ### Implementation - `blob.ts`: `normalizeBlobInput` (sync: `ArrayBuffer`/`URL`), `resolveBlobInput` (async: `Blob`/`File`). `coerceBlobValue` normalizes first. - `arrow.ts`: `mapBlobInputs` walks only the blob positions in the schema, including blobs nested in structs and lists. It copies records and arrays only where a value changed, so caller data is never mutated. `resolveBlobInputs` runs before `makeArrowTable` in `convertToTable` (used by `add` / `mergeInsert`) and in `createTable`. Each distinct `Blob` is read once. - `makeArrowTable` now coerces every blob value to the full `{ data, uri }` struct **before** schema inference. This also fixes two bugs that existed before this change: - Rows mixing plain bytes with `{ data }` failed inference. - A `{ uri }`-only struct produced a blob struct missing its `data` child. Blob validation errors now include the field path and row. - Also fixed along the way: - `add` / `mergeInsert` on a table with an embedding function rejected most blob values, because `ensureNestedFieldsExist` filled blob columns in like ordinary structs. It now skips them. - With a schema, a struct field that is `null` in one row and a record in a later row failed inference. Primitives and arrays given for a struct field are now rejected instead of being written as an all-null struct. - Blob structs with keys other than `data` / `uri` are rejected, which catches typos. Class instances with `data` / `uri` are handled like plain objects. ## Testing - Unit tests in `__test__/blob.test.ts`: coercion of the new types, rejection and error messages for `Blob` in the sync path, nested lists and structs, `resolveBlobInputs` (only blob columns touched, caller data untouched, shared `Blob` read once). - Table tests in `__test__/table.test.ts`: - `createTable`, `add`, and `mergeInsert` round-trip `ArrayBuffer` / `Blob` / `File` bytes through `fetchBlobs`. - `URL` and `{ uri: URL }` are added to the external-blob round-trip test from #4317 (`allowExternalBlobOutsideBases`). - Full `jest` suite passes locally after the rebase (938 passed). `pnpm lint` is clean and `pnpm run docs` has been regenerated. Streaming inputs (`ReadableStream`, Node `Readable`, async iterables) remain out of scope, as the issue says. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012U4KdHofT5edv3LcXHXbLv --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -157,9 +157,12 @@
|
||||
|
||||
- [AnalyzePlanDistributedMetrics](type-aliases/AnalyzePlanDistributedMetrics.md)
|
||||
- [BaseTokenizer](type-aliases/BaseTokenizer.md)
|
||||
- [BlobData](type-aliases/BlobData.md)
|
||||
- [BlobInput](type-aliases/BlobInput.md)
|
||||
- [BlobOptions](type-aliases/BlobOptions.md)
|
||||
- [BlobRange](type-aliases/BlobRange.md)
|
||||
- [BlobReadOptions](type-aliases/BlobReadOptions.md)
|
||||
- [BlobUri](type-aliases/BlobUri.md)
|
||||
- [Data](type-aliases/Data.md)
|
||||
- [DataLike](type-aliases/DataLike.md)
|
||||
- [DataTypeLike](type-aliases/DataTypeLike.md)
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BlobData
|
||||
|
||||
# Type Alias: BlobData
|
||||
|
||||
```ts
|
||||
type BlobData: Buffer | Uint8Array | ArrayBuffer | Blob;
|
||||
```
|
||||
|
||||
Bytes accepted for a blob value. `ArrayBuffer` is wrapped without copying.
|
||||
`Blob` and `File` are read with `arrayBuffer()`, which only the async write
|
||||
paths ([Connection.createTable](../classes/Connection.md#createtable), [Table.add](../classes/Table.md#add),
|
||||
[Table.mergeInsert](../classes/Table.md#mergeinsert)) can do.
|
||||
@@ -0,0 +1,36 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BlobInput
|
||||
|
||||
# Type Alias: BlobInput
|
||||
|
||||
```ts
|
||||
type BlobInput:
|
||||
| BlobData
|
||||
| BlobUri
|
||||
| object
|
||||
| object
|
||||
| null
|
||||
| undefined;
|
||||
```
|
||||
|
||||
A value accepted for a `lance.blob.v2` column (see [blob](../functions/blob.md)).
|
||||
|
||||
Either inline bytes, a URI pointing at external bytes, or a struct that sets
|
||||
exactly one of `data` or `uri`. `null` and `undefined` write a null blob.
|
||||
|
||||
## Example
|
||||
|
||||
```ts
|
||||
import { pathToFileURL } from "node:url";
|
||||
import type { BlobInput } from "@lancedb/lancedb";
|
||||
|
||||
const rows: { id: bigint; image: BlobInput }[] = [
|
||||
{ id: 1n, image: await (await fetch(url)).arrayBuffer() },
|
||||
{ id: 2n, image: pathToFileURL("/data/cat.png") },
|
||||
{ id: 3n, image: { data: new Blob(["hello"]) } },
|
||||
];
|
||||
await table.add(rows);
|
||||
```
|
||||
@@ -0,0 +1,13 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BlobUri
|
||||
|
||||
# Type Alias: BlobUri
|
||||
|
||||
```ts
|
||||
type BlobUri: string | URL;
|
||||
```
|
||||
|
||||
A URI accepted for a blob value. A `URL` is stored as its `href`.
|
||||
Reference in New Issue
Block a user