fix(python): avoid deadlock for query-backed iterators (#3820)

## Summary

- move synchronous iterator preparation off LanceDB's singleton
background event loop
- keep the loop available when an input iterator executes a synchronous
table query
- cover query-backed iterators for both `Table.add` and `create_table`

## Root cause

The async implementations prepared input data on the singleton
background loop and peeked at the first item of iterators there. If that
iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the
query on the same loop and blocked waiting for it, so neither operation
could progress.

## Validation

- `ruff format` on the changed Python files
- `ruff check .`
- `cd python && uv run --frozen --extra tests pytest
python/tests/test_table.py -q` (138 passed)
- the broader non-slow suite was attempted but hit a native segmentation
fault in the external DataFusion `register_table_provider` path; an
isolated control using the pre-change inline preparation behavior
reproduced the same unrelated crash

Fixes #2107

<!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
This commit is contained in:
lancedb-gatefixer[bot]
2026-10-23 01:09:32 +08:00
committed by GitHub
parent 102518ebda
commit 40af9f350e
3 changed files with 47 additions and 6 deletions
+24
View File
@@ -518,6 +518,30 @@ def test_add(mem_db: DBConnection):
_add(table, schema)
def test_add_from_iterator_that_queries_table(mem_db: DBConnection):
source = mem_db.create_table("source", data=pa.table({"id": range(16)}))
target = mem_db.create_table("target", schema=source.schema)
def batches():
for _ in range(5):
yield source.search().limit(10).to_arrow()
target.add(batches())
assert target.count_rows() == 50
def test_create_table_from_iterator_that_queries_table(mem_db: DBConnection):
source = mem_db.create_table("source", data=pa.table({"id": range(16)}))
def batches():
yield source.search().limit(10).to_arrow()
target = mem_db.create_table("target", data=batches())
assert target.count_rows() == 10
def test_add_releases_arrow_buffers_without_gc(mem_db: DBConnection):
"""Regression test for https://github.com/lancedb/lancedb/issues/2512."""
schema = pa.schema([pa.field("x", pa.int64())])