mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
fix(python): avoid deadlock for query-backed iterators (#3820)
## Summary - move synchronous iterator preparation off LanceDB's singleton background event loop - keep the loop available when an input iterator executes a synchronous table query - cover query-backed iterators for both `Table.add` and `create_table` ## Root cause The async implementations prepared input data on the singleton background loop and peeked at the first item of iterators there. If that iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the query on the same loop and blocked waiting for it, so neither operation could progress. ## Validation - `ruff format` on the changed Python files - `ruff check .` - `cd python && uv run --frozen --extra tests pytest python/tests/test_table.py -q` (138 passed) - the broader non-slow suite was attempted but hit a native segmentation fault in the external DataFusion `register_table_provider` path; an isolated control using the pre-change inline preparation behavior reproduced the same unrelated crash Fixes #2107 <!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
This commit is contained in:
@@ -518,6 +518,30 @@ def test_add(mem_db: DBConnection):
|
||||
_add(table, schema)
|
||||
|
||||
|
||||
def test_add_from_iterator_that_queries_table(mem_db: DBConnection):
|
||||
source = mem_db.create_table("source", data=pa.table({"id": range(16)}))
|
||||
target = mem_db.create_table("target", schema=source.schema)
|
||||
|
||||
def batches():
|
||||
for _ in range(5):
|
||||
yield source.search().limit(10).to_arrow()
|
||||
|
||||
target.add(batches())
|
||||
|
||||
assert target.count_rows() == 50
|
||||
|
||||
|
||||
def test_create_table_from_iterator_that_queries_table(mem_db: DBConnection):
|
||||
source = mem_db.create_table("source", data=pa.table({"id": range(16)}))
|
||||
|
||||
def batches():
|
||||
yield source.search().limit(10).to_arrow()
|
||||
|
||||
target = mem_db.create_table("target", data=batches())
|
||||
|
||||
assert target.count_rows() == 10
|
||||
|
||||
|
||||
def test_add_releases_arrow_buffers_without_gc(mem_db: DBConnection):
|
||||
"""Regression test for https://github.com/lancedb/lancedb/issues/2512."""
|
||||
schema = pa.schema([pa.field("x", pa.int64())])
|
||||
|
||||
Reference in New Issue
Block a user