mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 15:52:26 +00:00
fix(python): warn when optimize cleans up all old versions (#4337)
## Why `optimize(cleanup_older_than=timedelta(0))` deletes the data files of every version except the latest. Any other handle still on an older version then fails with a missing-file error, and nothing tells the caller this can happen. #2470 asks for a docs note and a runtime warning. Fixes #2470 ## Scope - `AsyncTable.optimize` emits a `UserWarning` when `cleanup_older_than` rounds to 0 ms or less. `LanceTable.optimize` runs through it, so sync callers get the warning too. - The `cleanup_older_than` docstring on `Table.optimize`, `LanceTable.optimize`, and `AsyncTable.optimize` gets a `.. warning::` block, in the same style as the existing `delete_unverified` one. - Removed the function-local `import warnings` in `AsyncTable.optimize`. `warnings` is already imported at module level, and the local import would shadow it for the new check (`UnboundLocalError`). - `test_optimize` and `test_optimize_delete_unverified` intentionally use a 0 cleanup, so they now assert the warning with `pytest.warns`. Nodejs `cleanupOlderThan` is out of scope, since the issue is filed against the Python SDK. ## Tradeoffs Like the existing `retrain` deprecation warning, the warning is raised inside the async coroutine. For sync callers it is attributed to `asyncio/events.py` rather than the caller's line. Warning from both the sync and async entry points would duplicate the check for a small gain. ## Verification - Repro: a second handle checked out at an older version, then `optimize(cleanup_older_than=timedelta(0))` on the first. The second handle's read fails with `LanceError(IO) ... Not found: .../data/<file>.lance`. Before this change no warning is emitted. After it, the `UserWarning` fires. - New tests `test_optimize_warns_on_zero_cleanup` (async, and checks that `timedelta(days=1)` and the default don't warn) and `test_optimize_warns_on_zero_cleanup_sync` fail on main (`DID NOT WARN`) and pass with the fix. The failing tests are in their own commit before the fix. - `pytest python/tests/test_table.py -m "not slow and not s3_test"`: all pass except `test_merge_insert`. That test's `timeout=timedelta(0)` assertion fails the same way on unmodified main on my machine. - `ruff format --check` and `ruff check` are clean on the changed files. Tests ran against the source tree with the `_lancedb` extension from the matching `0.40.0b11` wheel. The change touches no Rust. This PR was written by an AI coding agent (Claude Code) and reviewed before submission. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Jay Zhou <190699450+jayzhou2309@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Jay Zhou
Claude Opus 5.5
parent
d8063545d9
commit
1a463f3633
@@ -4307,7 +4307,8 @@ async def test_optimize(mem_db_async: AsyncConnection):
|
||||
assert stats.prune.bytes_removed == 0
|
||||
assert stats.prune.old_versions_removed == 0
|
||||
|
||||
stats = await table.optimize(cleanup_older_than=timedelta(seconds=0))
|
||||
with pytest.warns(UserWarning, match="concurrent"):
|
||||
stats = await table.optimize(cleanup_older_than=timedelta(seconds=0))
|
||||
assert stats.prune.bytes_removed > 0
|
||||
assert stats.prune.old_versions_removed == 3
|
||||
|
||||
@@ -4335,12 +4336,35 @@ async def test_optimize_delete_unverified(tmp_db_async: AsyncConnection, tmp_pat
|
||||
|
||||
stats = await table.optimize(delete_unverified=False)
|
||||
assert stats.prune.old_versions_removed == 0
|
||||
stats = await table.optimize(
|
||||
cleanup_older_than=timedelta(seconds=0), delete_unverified=True
|
||||
)
|
||||
with pytest.warns(UserWarning, match="concurrent"):
|
||||
stats = await table.optimize(
|
||||
cleanup_older_than=timedelta(seconds=0), delete_unverified=True
|
||||
)
|
||||
assert stats.prune.old_versions_removed == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_optimize_warns_on_zero_cleanup(mem_db_async: AsyncConnection):
|
||||
table = await mem_db_async.create_table("test", data=[{"x": [1]}])
|
||||
with pytest.warns(UserWarning, match="concurrent"):
|
||||
await table.optimize(cleanup_older_than=timedelta(0))
|
||||
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error")
|
||||
await table.optimize(cleanup_older_than=timedelta(days=1))
|
||||
await table.optimize()
|
||||
|
||||
|
||||
def test_optimize_warns_on_zero_cleanup_sync(mem_db: DBConnection):
|
||||
table = mem_db.create_table("test", data=[{"x": [1]}])
|
||||
with warnings.catch_warnings(record=True) as seen:
|
||||
warnings.simplefilter("default")
|
||||
table.optimize(cleanup_older_than=timedelta(0))
|
||||
table.optimize(cleanup_older_than=timedelta(0))
|
||||
assert [w.filename for w in seen] == [__file__, __file__]
|
||||
assert all("concurrent" in str(w.message) for w in seen)
|
||||
|
||||
|
||||
def test_replace_field_metadata(tmp_path):
|
||||
db = lancedb.connect(tmp_path)
|
||||
table = db.create_table("my_table", data=[{"x": 0}])
|
||||
|
||||
Reference in New Issue
Block a user