Will Jones
7813907eb7
## Problem `table.add(dataset)` with a `pyarrow.dataset.Dataset` OOMs the client during bulk ingestion of wide rows (e.g. embedding columns), even against a remote table where the upload itself is streaming. The cause is in `to_scannable`: a `Dataset` is scanned with pyarrow's default scanner settings (`batch_size=131072` rows, `batch_readahead=16`, `fragment_readahead=4`). pyarrow's internal threads prefetch that read-ahead window independently of LanceDB's backpressure, so for wide rows a large fraction of the dataset is held in memory. On the remote path this is then multiplied across the multipart write partitions (one in-flight batch per partition, up to CPU-core count). Reproduced on a 10 GB / 1.55M-row dataset with two 768-dim float32 embeddings: peak client RSS ~11.7 GB for the scan alone (6.8 GB after consuming a *single* batch), ~15.4 GB for the full remote `add()`. ## Fix `to_scannable` now sizes the scanner from an estimate of bytes-per-row derived from the schema: - **Narrow datasets keep pyarrow's defaults** (empty scanner kwargs) — no throughput regression. The bound only engages above ~410 bytes/row. - **Wide rows** get a smaller `batch_size` (~16 MiB/batch) and reduced read-ahead (`batch_readahead=2`, `fragment_readahead=1`) so peak in-flight memory stays near a ~1 GiB budget. Read-ahead (not just batch size) has to drop, because pyarrow pins whole row-group buffers. On the 10 GB dataset this drops peak client RSS to ~1.4 GB, and it stays flat as the dataset grows. The `Dataset`/`LanceDataset` scannables remain rescannable (retry-safe). ## Also: expose `write_parallelism` on `add()` `AddDataBuilder::write_parallelism` already existed in Rust but was not exposed in Python. This PR forwards it through the async, sync, and remote `add()` methods, so users can cap the number of parallel write partitions (each buffers data in flight) to trade throughput for memory on large uploads. ## Tests - `test_scannable.py`: bytes-per-row estimation; narrow → defaults; wide → bounded; `Dataset` reader streams bounded batches and stays rescannable. - `test_table.py`: `write_parallelism` on sync and async `add()`, and that `write_parallelism=0` is rejected. Fixes ENT-1883 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LanceDB Python SDK
A Python library for LanceDB.
Installation
pip install lancedb
Pre-Haswell x86_64 hosts: lancedb-compat
The default lancedb wheel targets x86-64-haswell (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with Illegal instruction at import lancedb.
For those hosts, install the lancedb-compat package instead:
pip install lancedb-compat
Same Python API (import lancedb works as usual). The compat wheel is compiled at the x86-64-v2 baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use lance.simd_info() from Python to verify which tier was selected.
lancedb and lancedb-compat install to the same lancedb/ namespace and conflict at install time. Pick one. To switch, pip uninstall lancedb first, then pip install lancedb-compat (or vice-versa).
If you need a custom baseline (or lancedb-compat isn't yet published for your platform), build from source with the override:
RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
pip install ./target/wheels/lancedb-*.whl
Preview Releases
Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
Usage
Basic Example
import lancedb
db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>')
table = db.open_table('my_table')
results = table.search([0.1, 0.3]).limit(20).to_list()
print(results)
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.