mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Will JonesandClaude
a68c7be202
Closes #4270. ## Summary Blob v2 columns now accept more of the containers TypeScript code commonly has, both as the column value and inside `{ data }` / `{ uri }`: | Input | Handling | |---|---| | `ArrayBuffer` | `new Uint8Array(value)`, no copy | | `URL` | stored as `value.href` | | `Blob` / `File` | `await value.arrayBuffer()` in the async write paths: `createTable`, `add`, `mergeInsert` | `makeArrowTable` is synchronous, so it can't read a `Blob`. Given one, it throws an error that names the field and row and points to the async APIs or `new Uint8Array(await value.arrayBuffer())`. It does not write a null blob. ### Types `BlobInput` is now the accepted input type. It is widened to the types above and exported from the package entry point with `BlobData` and `BlobUri`, so callers can annotate rows and `tsc` catches bad inputs. The struct forms are a union that requires exactly one of `data` / `uri`. The old normalized `{ data, uri }` shape is renamed to `BlobValue` (internal). ### Implementation - `blob.ts`: `normalizeBlobInput` (sync: `ArrayBuffer`/`URL`), `resolveBlobInput` (async: `Blob`/`File`). `coerceBlobValue` normalizes first. - `arrow.ts`: `mapBlobInputs` walks only the blob positions in the schema, including blobs nested in structs and lists. It copies records and arrays only where a value changed, so caller data is never mutated. `resolveBlobInputs` runs before `makeArrowTable` in `convertToTable` (used by `add` / `mergeInsert`) and in `createTable`. Each distinct `Blob` is read once. - `makeArrowTable` now coerces every blob value to the full `{ data, uri }` struct **before** schema inference. This also fixes two bugs that existed before this change: - Rows mixing plain bytes with `{ data }` failed inference. - A `{ uri }`-only struct produced a blob struct missing its `data` child. Blob validation errors now include the field path and row. - Also fixed along the way: - `add` / `mergeInsert` on a table with an embedding function rejected most blob values, because `ensureNestedFieldsExist` filled blob columns in like ordinary structs. It now skips them. - With a schema, a struct field that is `null` in one row and a record in a later row failed inference. Primitives and arrays given for a struct field are now rejected instead of being written as an all-null struct. - Blob structs with keys other than `data` / `uri` are rejected, which catches typos. Class instances with `data` / `uri` are handled like plain objects. ## Testing - Unit tests in `__test__/blob.test.ts`: coercion of the new types, rejection and error messages for `Blob` in the sync path, nested lists and structs, `resolveBlobInputs` (only blob columns touched, caller data untouched, shared `Blob` read once). - Table tests in `__test__/table.test.ts`: - `createTable`, `add`, and `mergeInsert` round-trip `ArrayBuffer` / `Blob` / `File` bytes through `fetchBlobs`. - `URL` and `{ uri: URL }` are added to the external-blob round-trip test from #4317 (`allowExternalBlobOutsideBases`). - Full `jest` suite passes locally after the rebase (938 passed). `pnpm lint` is clean and `pnpm run docs` has been regenerated. Streaming inputs (`ReadableStream`, Node `Readable`, async iterables) remain out of scope, as the issue says. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012U4KdHofT5edv3LcXHXbLv --------- Co-authored-by: Claude <noreply@anthropic.com>
The Multimodal AI Lakehouse
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- Open Source & Local: 100% open source, runs locally or in your cloud. No vendor lock-in.
- Cloud and Enterprise: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us