mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Xuanwo
10f89b3c32
Remote Blob reads retain and concatenate intermediate IPC buffers across batches and HTTP requests, increasing peak memory. Decode fragmented IPC bodies using their declared length, assemble Blob values once across batches/requests, and collect large-object Range responses into known-size buffers. Impossible IPC/Range allocations return errors instead of panicking. Closing a remote Blob file releases cached bytes immediately, including while a cross-window read is waiting for the network. Rust and Python expose configurable Blob read buffers. Python sync/async `fetch_blob_files` and `open_blob_file(s)` accept `buffer_size`; the default is 512 KiB and zero disables prefetch. Explicit ranges bypass prefetch. New PE batch-endpoint adoption is separate. Malformed-length and retained-cache regressions fail on the previous implementation and pass after the correction. Real HTTP tests cover buffer propagation, close during blocked input, batch/request aggregation, and large-object fallback. The final extension was rebuilt before Python validation. Non-Blob Arrow arrays may still require alignment copies; a contiguous multi-batch result still requires its final allocation. ## Benchmark Peak client RSS increment in KiB, using `python/benchmarks/bench_fetch_blobs_rss.py`, independent clients and real loopback HTTP. Both extensions were rebuilt from clean, unchanged source with the same toolchain/configuration; loaded artifact hashes were checked. Baseline `42f11d13a2755e44b63876b5a5edd93901291150`, after `0a8a76de5286949d80368f95a5c529adf422c187`. | Equivalent fetch workload | Payload | Before | After | | --- | ---: | ---: | ---: | | Single response, one batch | 48 MiB | 63,812 | 63,896 | | Multiple responses, one batch per response | 144 MiB | 423,104 | 195,628 | | Single response, 12 rows per batch | 48 MiB | 130,520 | 71,348 | | Multiple responses, 12 rows per batch | 144 MiB | 363,144 | 172,748 | | One oversized row, Range fallback | 70 MiB | 182,816 | 85,548 | These are one observation per workload and build, not a statistical distribution or memory upper bound. The single-batch workload shows no improvement. Cloud/Sophon deployment, credentialed S3 and long-running stress have not been exercised.