mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-21 20:45:57 +00:00
feat(node): add blob v2 fetch and field helpers (#4155)
this PR blob v2 field helpers and reads to the Node SDK.
`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.
Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.
Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.
```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
"videos",
[{ id: 1n, video }],
{
schema: new Schema([
new Field("id", new Int64()),
blob("video"),
]),
},
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);
```
### Testing
- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BlobFile
|
||||
|
||||
# Class: BlobFile
|
||||
|
||||
A lazy handle to blob bytes. Create one with [Table.fetchBlobFiles](Table.md#fetchblobfiles).
|
||||
|
||||
## Methods
|
||||
|
||||
### read()
|
||||
|
||||
```ts
|
||||
read(): Promise<Buffer>
|
||||
```
|
||||
|
||||
Reads from the cursor to the end and advances the cursor.
|
||||
|
||||
A second call returns an empty buffer. [BlobFile.readRange](BlobFile.md#readrange) does
|
||||
not move the cursor.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`Buffer`>
|
||||
|
||||
***
|
||||
|
||||
### readRange()
|
||||
|
||||
```ts
|
||||
readRange(start, end): Promise<Buffer>
|
||||
```
|
||||
|
||||
Reads the half-open byte range `[start, end)`.
|
||||
|
||||
Fails when `end` is past the blob size. Does not move the cursor.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **start**: `bigint`
|
||||
|
||||
* **end**: `bigint`
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`Buffer`>
|
||||
|
||||
***
|
||||
|
||||
### size()
|
||||
|
||||
```ts
|
||||
size(): bigint
|
||||
```
|
||||
|
||||
Returns the blob size in bytes.
|
||||
|
||||
#### Returns
|
||||
|
||||
`bigint`
|
||||
@@ -137,6 +137,20 @@ containing the new version number of the table after altering the columns.
|
||||
|
||||
***
|
||||
|
||||
### blobColumns()
|
||||
|
||||
```ts
|
||||
abstract blobColumns(): Promise<string[]>
|
||||
```
|
||||
|
||||
Blob v2 columns, including nested dotted paths.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`string`[]>
|
||||
|
||||
***
|
||||
|
||||
### branches()
|
||||
|
||||
```ts
|
||||
@@ -499,6 +513,54 @@ Drop an index from the table.
|
||||
|
||||
***
|
||||
|
||||
### fetchBlobFiles()
|
||||
|
||||
```ts
|
||||
abstract fetchBlobFiles(column, rowIds): Promise<(null | BlobFile)[]>
|
||||
```
|
||||
|
||||
Opens lazy blob handles for `column` at the given row IDs using the
|
||||
table's current checkout.
|
||||
|
||||
Preserves input order, duplicates, and nulls. Use this for large payloads.
|
||||
See [Table.fetchBlobs](Table.md#fetchblobs) for row-ID validity across versions.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **column**: `string`
|
||||
|
||||
* **rowIds**: readonly (`number` \| `bigint`)[]
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<(`null` \| [`BlobFile`](BlobFile.md))[]>
|
||||
|
||||
***
|
||||
|
||||
### fetchBlobs()
|
||||
|
||||
```ts
|
||||
abstract fetchBlobs(column, rowIds): Promise<(null | Buffer)[]>
|
||||
```
|
||||
|
||||
Bytes for `column` at row IDs from [Query.withRowId](Query.md#withrowid).
|
||||
|
||||
Reads the table's current checkout. IDs from another version can fail after
|
||||
compaction unless stable row ids are enabled. Results keep input order and
|
||||
duplicates. Null blobs are `null`. Empty blobs are empty buffers.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **column**: `string`
|
||||
|
||||
* **rowIds**: readonly (`number` \| `bigint`)[]
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<(`null` \| `Buffer`)[]>
|
||||
|
||||
***
|
||||
|
||||
### flushLsm()
|
||||
|
||||
```ts
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / blob
|
||||
|
||||
# Function: blob()
|
||||
|
||||
```ts
|
||||
function blob(name, options): Field
|
||||
```
|
||||
|
||||
Declares a `lance.blob.v2` column.
|
||||
|
||||
Query results are descriptors, not payload bytes. Use [Table.fetchBlobs](../classes/Table.md#fetchblobs)
|
||||
or [Table.fetchBlobFiles](../classes/Table.md#fetchblobfiles) to read bytes.
|
||||
|
||||
## Parameters
|
||||
|
||||
* **name**: `string`
|
||||
|
||||
* **options**: [`BlobOptions`](../type-aliases/BlobOptions.md) = `{}`
|
||||
|
||||
## Returns
|
||||
|
||||
`Field`
|
||||
|
||||
## Example
|
||||
|
||||
```ts
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { Field, Int64, Schema } from "apache-arrow";
|
||||
import { blob, connect } from "@lancedb/lancedb";
|
||||
|
||||
const db = await connect("./data");
|
||||
const video = await readFile("clip.mp4");
|
||||
const table = await db.createTable(
|
||||
"videos",
|
||||
[{ id: 1n, video }],
|
||||
{
|
||||
schema: new Schema([
|
||||
new Field("id", new Int64()),
|
||||
blob("video"),
|
||||
]),
|
||||
},
|
||||
);
|
||||
|
||||
const rows = await table.query().select(["id"]).withRowId().toArray();
|
||||
const rowIds = rows.map((row) => row._rowid as bigint);
|
||||
const bytes = await table.fetchBlobs("video", rowIds);
|
||||
|
||||
const [handle] = await table.fetchBlobFiles("video", rowIds);
|
||||
const size = handle!.size();
|
||||
const header = await handle!.readRange(0n, size < 65536n ? size : 65536n);
|
||||
```
|
||||
@@ -0,0 +1,22 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / isBlobField
|
||||
|
||||
# Function: isBlobField()
|
||||
|
||||
```ts
|
||||
function isBlobField(field): boolean
|
||||
```
|
||||
|
||||
Checks for the `lance.blob.v2` extension marker. Does not validate the
|
||||
field's storage type.
|
||||
|
||||
## Parameters
|
||||
|
||||
* **field**: `Field`<`any`>
|
||||
|
||||
## Returns
|
||||
|
||||
`boolean`
|
||||
@@ -19,6 +19,7 @@
|
||||
## Classes
|
||||
|
||||
- [AutoQuery](classes/AutoQuery.md)
|
||||
- [BlobFile](classes/BlobFile.md)
|
||||
- [BooleanQuery](classes/BooleanQuery.md)
|
||||
- [BoostQuery](classes/BoostQuery.md)
|
||||
- [BranchContents](classes/BranchContents.md)
|
||||
@@ -143,6 +144,7 @@
|
||||
|
||||
- [AnalyzePlanDistributedMetrics](type-aliases/AnalyzePlanDistributedMetrics.md)
|
||||
- [BaseTokenizer](type-aliases/BaseTokenizer.md)
|
||||
- [BlobOptions](type-aliases/BlobOptions.md)
|
||||
- [Data](type-aliases/Data.md)
|
||||
- [DataLike](type-aliases/DataLike.md)
|
||||
- [FieldLike](type-aliases/FieldLike.md)
|
||||
@@ -158,9 +160,11 @@
|
||||
## Functions
|
||||
|
||||
- [RecordBatchIterator](functions/RecordBatchIterator.md)
|
||||
- [blob](functions/blob.md)
|
||||
- [connect](functions/connect.md)
|
||||
- [connectNamespace](functions/connectNamespace.md)
|
||||
- [instrumentLanceDbMetrics](functions/instrumentLanceDbMetrics.md)
|
||||
- [isBlobField](functions/isBlobField.md)
|
||||
- [makeArrowTable](functions/makeArrowTable.md)
|
||||
- [packBits](functions/packBits.md)
|
||||
- [permutationBuilder](functions/permutationBuilder.md)
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BlobOptions
|
||||
|
||||
# Type Alias: BlobOptions
|
||||
|
||||
```ts
|
||||
type BlobOptions: object;
|
||||
```
|
||||
|
||||
## Type declaration
|
||||
|
||||
### dedicatedSizeThreshold?
|
||||
|
||||
```ts
|
||||
optional dedicatedSizeThreshold: number;
|
||||
```
|
||||
|
||||
Max payload bytes stored in a packed sidecar before a dedicated file. Must
|
||||
be a positive safe integer.
|
||||
|
||||
### inlineSizeThreshold?
|
||||
|
||||
```ts
|
||||
optional inlineSizeThreshold: number;
|
||||
```
|
||||
|
||||
Max payload bytes kept inline in the data file. Zero is allowed. Must be a
|
||||
safe integer.
|
||||
|
||||
### nullable?
|
||||
|
||||
```ts
|
||||
optional nullable: boolean;
|
||||
```
|
||||
|
||||
Defaults to true.
|
||||
|
||||
### packFileSizeThreshold?
|
||||
|
||||
```ts
|
||||
optional packFileSizeThreshold: number;
|
||||
```
|
||||
|
||||
Max bytes in one packed sidecar before starting another. Must be a positive
|
||||
safe integer.
|
||||
Reference in New Issue
Block a user