Will Jones
5d0a1ef66c
## Problem On the remote (LanceDB Cloud) write path, each write partition is uploaded as a **single** `/insert?upload_id=...` request that stays open until the whole partition has been streamed and the server has written it to object storage. For large bulk ingests a partition can be many GB, so a single request can run longer than the client read timeout (default 300s), surfacing as: ``` lancedb.remote.errors.HttpError: operation timed out ``` The server already supports staging **multiple** parts under one `upload_id` (each `/insert` writes a separate transaction that `complete` merges atomically), but the client never used that — it sent one part per partition. ## Change Split each partition into multiple parts of at most `max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded as its own `/insert?upload_id=...&upload_part_id=...` request. This bounds how long any single request stays open, independent of total data size or write parallelism. Key properties: - **Still streamed, not buffered.** Each part's body is driven through a bounded channel while the request is in flight (`futures::join!` of a producer + the send), so peak memory stays at a couple of batches per partition regardless of the part size. Backpressure from a slow/throttled server still propagates upstream. - **Correct part accounting.** An empty partition still sends exactly one (schema-only) part so `complete` has a transaction to commit; a size cut landing exactly on the end of input does not emit a trailing empty part. - **Multipart only.** The single-request (non-multipart) path is unchanged. ## Config New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable. **Default 1 GiB** (`Some(0)` disables splitting → one request per partition). Python users pick up the default/env automatically through the remote client. ## Tests - `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each batch in its own part → N requests, each carrying the shared `upload_id` and a distinct `upload_part_id`. - `test_multipart_single_part_when_under_budget`: a large budget keeps the partition in a single request. - Verified end-to-end against a live remote table: a forced-chunked multipart add (many parts) assembles to the correct row count. Related to ENT-1883. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LanceDB Python SDK
A Python library for LanceDB.
Installation
pip install lancedb
Pre-Haswell x86_64 hosts: lancedb-compat
The default lancedb wheel targets x86-64-haswell (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with Illegal instruction at import lancedb.
For those hosts, install the lancedb-compat package instead:
pip install lancedb-compat
Same Python API (import lancedb works as usual). The compat wheel is compiled at the x86-64-v2 baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use lance.simd_info() from Python to verify which tier was selected.
lancedb and lancedb-compat install to the same lancedb/ namespace and conflict at install time. Pick one. To switch, pip uninstall lancedb first, then pip install lancedb-compat (or vice-versa).
If you need a custom baseline (or lancedb-compat isn't yet published for your platform), build from source with the override:
RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
pip install ./target/wheels/lancedb-*.whl
Preview Releases
Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
Usage
Basic Example
import lancedb
db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>')
table = db.open_table('my_table')
results = table.search([0.1, 0.3]).limit(20).to_list()
print(results)
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.