mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 03:58:26 +00:00
fix(rust): bound remote insert request size to avoid ingestion timeouts (#3630)
## Problem On the remote (LanceDB Cloud) write path, each write partition is uploaded as a **single** `/insert?upload_id=...` request that stays open until the whole partition has been streamed and the server has written it to object storage. For large bulk ingests a partition can be many GB, so a single request can run longer than the client read timeout (default 300s), surfacing as: ``` lancedb.remote.errors.HttpError: operation timed out ``` The server already supports staging **multiple** parts under one `upload_id` (each `/insert` writes a separate transaction that `complete` merges atomically), but the client never used that — it sent one part per partition. ## Change Split each partition into multiple parts of at most `max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded as its own `/insert?upload_id=...&upload_part_id=...` request. This bounds how long any single request stays open, independent of total data size or write parallelism. Key properties: - **Still streamed, not buffered.** Each part's body is driven through a bounded channel while the request is in flight (`futures::join!` of a producer + the send), so peak memory stays at a couple of batches per partition regardless of the part size. Backpressure from a slow/throttled server still propagates upstream. - **Correct part accounting.** An empty partition still sends exactly one (schema-only) part so `complete` has a transaction to commit; a size cut landing exactly on the end of input does not emit a trailing empty part. - **Multipart only.** The single-request (non-multipart) path is unchanged. ## Config New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable. **Default 1 GiB** (`Some(0)` disables splitting → one request per partition). Python users pick up the default/env automatically through the remote client. ## Tests - `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each batch in its own part → N requests, each carrying the shared `upload_id` and a distinct `upload_part_id`. - `test_multipart_single_part_when_under_budget`: a large budget keeps the partition in a single request. - Verified end-to-end against a live remote table: a forced-chunked multipart add (many parts) assembles to the correct row count. Related to ENT-1883. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -800,6 +800,10 @@ impl From<PyClientConfig> for lancedb::remote::ClientConfig {
|
||||
tls_config: value.tls_config.map(Into::into),
|
||||
header_provider,
|
||||
user_id: value.user_id,
|
||||
// Resolved from LANCE_CLIENT_MAX_BYTES_PER_REQUEST or the default.
|
||||
max_bytes_per_request: None,
|
||||
// Resolved from LANCE_CLIENT_MAX_REQUEST_DURATION or the read timeout.
|
||||
max_request_duration: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user