fix(python): warn when optimize cleans up all old versions (#4337)

## Why

`optimize(cleanup_older_than=timedelta(0))` deletes the data files of
every version except the latest. Any other handle still on an older
version then fails with a missing-file error, and nothing tells the
caller this can happen. #2470 asks for a docs note and a runtime
warning.

Fixes #2470

## Scope

- `AsyncTable.optimize` emits a `UserWarning` when `cleanup_older_than`
rounds to 0 ms or less. `LanceTable.optimize` runs through it, so sync
callers get the warning too.
- The `cleanup_older_than` docstring on `Table.optimize`,
`LanceTable.optimize`, and `AsyncTable.optimize` gets a `.. warning::`
block, in the same style as the existing `delete_unverified` one.
- Removed the function-local `import warnings` in `AsyncTable.optimize`.
`warnings` is already imported at module level, and the local import
would shadow it for the new check (`UnboundLocalError`).
- `test_optimize` and `test_optimize_delete_unverified` intentionally
use a 0 cleanup, so they now assert the warning with `pytest.warns`.

Nodejs `cleanupOlderThan` is out of scope, since the issue is filed
against the Python SDK.

## Tradeoffs

Like the existing `retrain` deprecation warning, the warning is raised
inside the async coroutine. For sync callers it is attributed to
`asyncio/events.py` rather than the caller's line. Warning from both the
sync and async entry points would duplicate the check for a small gain.

## Verification

- Repro: a second handle checked out at an older version, then
`optimize(cleanup_older_than=timedelta(0))` on the first. The second
handle's read fails with `LanceError(IO) ... Not found:
.../data/<file>.lance`. Before this change no warning is emitted. After
it, the `UserWarning` fires.
- New tests `test_optimize_warns_on_zero_cleanup` (async, and checks
that `timedelta(days=1)` and the default don't warn) and
`test_optimize_warns_on_zero_cleanup_sync` fail on main (`DID NOT WARN`)
and pass with the fix. The failing tests are in their own commit before
the fix.
- `pytest python/tests/test_table.py -m "not slow and not s3_test"`: all
pass except `test_merge_insert`. That test's `timeout=timedelta(0)`
assertion fails the same way on unmodified main on my machine.
- `ruff format --check` and `ruff check` are clean on the changed files.

Tests ran against the source tree with the `_lancedb` extension from the
matching `0.40.0b11` wheel. The change touches no Rust.

This PR was written by an AI coding agent (Claude Code) and reviewed
before submission.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Jay Zhou <190699450+jayzhou2309@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jay Zhou
2026-09-28 20:11:47 -05:00
committed by GitHub
co-authored by Jay Zhou Claude Opus 5.5
parent d8063545d9
commit 1a463f3633
2 changed files with 87 additions and 21 deletions
+28 -4
View File
@@ -4307,7 +4307,8 @@ async def test_optimize(mem_db_async: AsyncConnection):
assert stats.prune.bytes_removed == 0
assert stats.prune.old_versions_removed == 0
stats = await table.optimize(cleanup_older_than=timedelta(seconds=0))
with pytest.warns(UserWarning, match="concurrent"):
stats = await table.optimize(cleanup_older_than=timedelta(seconds=0))
assert stats.prune.bytes_removed > 0
assert stats.prune.old_versions_removed == 3
@@ -4335,12 +4336,35 @@ async def test_optimize_delete_unverified(tmp_db_async: AsyncConnection, tmp_pat
stats = await table.optimize(delete_unverified=False)
assert stats.prune.old_versions_removed == 0
stats = await table.optimize(
cleanup_older_than=timedelta(seconds=0), delete_unverified=True
)
with pytest.warns(UserWarning, match="concurrent"):
stats = await table.optimize(
cleanup_older_than=timedelta(seconds=0), delete_unverified=True
)
assert stats.prune.old_versions_removed == 2
@pytest.mark.asyncio
async def test_optimize_warns_on_zero_cleanup(mem_db_async: AsyncConnection):
table = await mem_db_async.create_table("test", data=[{"x": [1]}])
with pytest.warns(UserWarning, match="concurrent"):
await table.optimize(cleanup_older_than=timedelta(0))
with warnings.catch_warnings():
warnings.simplefilter("error")
await table.optimize(cleanup_older_than=timedelta(days=1))
await table.optimize()
def test_optimize_warns_on_zero_cleanup_sync(mem_db: DBConnection):
table = mem_db.create_table("test", data=[{"x": [1]}])
with warnings.catch_warnings(record=True) as seen:
warnings.simplefilter("default")
table.optimize(cleanup_older_than=timedelta(0))
table.optimize(cleanup_older_than=timedelta(0))
assert [w.filename for w in seen] == [__file__, __file__]
assert all("concurrent" in str(w.message) for w in seen)
def test_replace_field_metadata(tmp_path):
db = lancedb.connect(tmp_path)
table = db.create_table("my_table", data=[{"x": 0}])