Will JonesandClaude
a68c7be202
Closes #4270. ## Summary Blob v2 columns now accept more of the containers TypeScript code commonly has, both as the column value and inside `{ data }` / `{ uri }`: | Input | Handling | |---|---| | `ArrayBuffer` | `new Uint8Array(value)`, no copy | | `URL` | stored as `value.href` | | `Blob` / `File` | `await value.arrayBuffer()` in the async write paths: `createTable`, `add`, `mergeInsert` | `makeArrowTable` is synchronous, so it can't read a `Blob`. Given one, it throws an error that names the field and row and points to the async APIs or `new Uint8Array(await value.arrayBuffer())`. It does not write a null blob. ### Types `BlobInput` is now the accepted input type. It is widened to the types above and exported from the package entry point with `BlobData` and `BlobUri`, so callers can annotate rows and `tsc` catches bad inputs. The struct forms are a union that requires exactly one of `data` / `uri`. The old normalized `{ data, uri }` shape is renamed to `BlobValue` (internal). ### Implementation - `blob.ts`: `normalizeBlobInput` (sync: `ArrayBuffer`/`URL`), `resolveBlobInput` (async: `Blob`/`File`). `coerceBlobValue` normalizes first. - `arrow.ts`: `mapBlobInputs` walks only the blob positions in the schema, including blobs nested in structs and lists. It copies records and arrays only where a value changed, so caller data is never mutated. `resolveBlobInputs` runs before `makeArrowTable` in `convertToTable` (used by `add` / `mergeInsert`) and in `createTable`. Each distinct `Blob` is read once. - `makeArrowTable` now coerces every blob value to the full `{ data, uri }` struct **before** schema inference. This also fixes two bugs that existed before this change: - Rows mixing plain bytes with `{ data }` failed inference. - A `{ uri }`-only struct produced a blob struct missing its `data` child. Blob validation errors now include the field path and row. - Also fixed along the way: - `add` / `mergeInsert` on a table with an embedding function rejected most blob values, because `ensureNestedFieldsExist` filled blob columns in like ordinary structs. It now skips them. - With a schema, a struct field that is `null` in one row and a record in a later row failed inference. Primitives and arrays given for a struct field are now rejected instead of being written as an all-null struct. - Blob structs with keys other than `data` / `uri` are rejected, which catches typos. Class instances with `data` / `uri` are handled like plain objects. ## Testing - Unit tests in `__test__/blob.test.ts`: coercion of the new types, rejection and error messages for `Blob` in the sync path, nested lists and structs, `resolveBlobInputs` (only blob columns touched, caller data untouched, shared `Blob` read once). - Table tests in `__test__/table.test.ts`: - `createTable`, `add`, and `mergeInsert` round-trip `ArrayBuffer` / `Blob` / `File` bytes through `fetchBlobs`. - `URL` and `{ uri: URL }` are added to the external-blob round-trip test from #4317 (`allowExternalBlobOutsideBases`). - Full `jest` suite passes locally after the rebase (938 passed). `pnpm lint` is clean and `pnpm run docs` has been regenerated. Streaming inputs (`ReadableStream`, Node `Readable`, async iterables) remain out of scope, as the issue says. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012U4KdHofT5edv3LcXHXbLv --------- Co-authored-by: Claude <noreply@anthropic.com>
LanceDB Documentation
LanceDB docs are available at docs.lancedb.com.
The SDK docs are built and deployed automatically by Github Actions
whenever a commit is pushed to the main branch. So it is possible for the docs to show
unreleased features.
Building the docs
Setup
- Install LanceDB Python. See setup in Python contributing guide.
Run
make developto install the Python package. - Install documentation dependencies. From LanceDB repo root:
pip install -r docs/requirements.txt
Preview the docs
cd docs
mkdocs serve
If you want to just generate the HTML files:
PYTHONPATH=. mkdocs build -f docs/mkdocs.yml
If successful, you should see a docs/site directory that you can verify locally.
Adding examples
To make sure examples are correct, we put examples in test files so they can be run as part of our test suites.
You can see the tests are at:
- Python:
python/python/tests/docs - Typescript:
nodejs/examples/
Checking python examples
cd python
pytest -vv python/tests/docs
Checking typescript examples
The examples depend on @lancedb/lancedb at file:../dist, so the package must be
built before running the tests. This uses pnpm; see the
Typescript contributing guide for the toolchain setup.
pushd nodejs
pnpm install
pnpm build
popd
Then you can run the examples by going to the nodejs/examples directory, which is a
separate pnpm package with its own lockfile:
pushd nodejs/examples
pnpm install
pnpm test
popd
API documentation
Python
The Python API documentation is organized based on the file docs/src/python/python.md.
We manually add entries there so we can control the organization of the reference page.
However, this means any new types must be manually added to the file. No additional
steps are needed to generate the API documentation.
Typescript
The typescript API documentation is generated from the typescript source code using typedoc.
When new APIs are added, you must manually re-run the typedoc command to update the API documentation. The new files should be checked into the repository.
pushd nodejs
# `pnpm docs` would invoke pnpm's built-in `docs` command, not the script.
pnpm run docs
popd