mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 03:58:26 +00:00
Will Jones
5d0a1ef66c
## Problem On the remote (LanceDB Cloud) write path, each write partition is uploaded as a **single** `/insert?upload_id=...` request that stays open until the whole partition has been streamed and the server has written it to object storage. For large bulk ingests a partition can be many GB, so a single request can run longer than the client read timeout (default 300s), surfacing as: ``` lancedb.remote.errors.HttpError: operation timed out ``` The server already supports staging **multiple** parts under one `upload_id` (each `/insert` writes a separate transaction that `complete` merges atomically), but the client never used that — it sent one part per partition. ## Change Split each partition into multiple parts of at most `max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded as its own `/insert?upload_id=...&upload_part_id=...` request. This bounds how long any single request stays open, independent of total data size or write parallelism. Key properties: - **Still streamed, not buffered.** Each part's body is driven through a bounded channel while the request is in flight (`futures::join!` of a producer + the send), so peak memory stays at a couple of batches per partition regardless of the part size. Backpressure from a slow/throttled server still propagates upstream. - **Correct part accounting.** An empty partition still sends exactly one (schema-only) part so `complete` has a transaction to commit; a size cut landing exactly on the end of input does not emit a trailing empty part. - **Multipart only.** The single-request (non-multipart) path is unchanged. ## Config New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable. **Default 1 GiB** (`Some(0)` disables splitting → one request per partition). Python users pick up the default/env automatically through the remote client. ## Tests - `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each batch in its own part → N requests, each carrying the shared `upload_id` and a distinct `upload_part_id`. - `test_multipart_single_part_when_under_budget`: a large budget keeps the partition in a single request. - Verified end-to-end against a live remote table: a forced-chunked multipart add (many parts) assembles to the correct row count. Related to ENT-1883. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LanceDB JavaScript SDK
A JavaScript library for LanceDB.
Installation
npm install @lancedb/lancedb
This will download the appropriate native library for your platform. We currently support:
- Linux (x86_64 and aarch64 on glibc and musl)
- MacOS (Intel and ARM/M1/M2)
- Windows (x86_64 and aarch64)
Usage
Basic Example
import * as lancedb from "@lancedb/lancedb";
const db = await lancedb.connect("data/sample-lancedb");
const table = await db.createTable("my_table", [
{ id: 1, vector: [0.1, 1.0], item: "foo", price: 10.0 },
{ id: 2, vector: [3.9, 0.5], item: "bar", price: 20.0 },
]);
const results = await table.vectorSearch([0.1, 0.3]).limit(20).toArray();
console.log(results);
The quickstart contains more complete examples.
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.