fix: keep tagged versions when pruning in optimize (#4544)

`optimize()` fails once a tagged version is older than the cleanup
cutoff, although the docs say tagged versions
are exempt from cleanup, and the failed prune also skips the
index-optimize step (repro and numbers in the issue).

Change: `OptimizeAction::Prune` already exposes
`error_if_tagged_old_versions: Option<bool>`, so no new API.
`cleanup_old_versions` and `cleanup_old_versions_before` in
`rust/lancedb/src/table/optimize.rs` now use
`unwrap_or(false)`, which covers `OptimizeAction::Prune`,
`OptimizeAction::All` and `optimize_prune_before`. The
Python and Node bindings pass `None`, which now means "retain tagged
versions"; I left `python/src/table.rs` and
`nodejs/src/table.rs` untouched (this also avoids a conflict with #3870,
which edits that region). `Some(true)`
still opts in to the error. Doc comments updated; the sentence in
`docs/web/management/table.mdx` is accurate once
this lands.

Tests: Rust `test_optimize_prune_retains_tagged_versions` (`Some(true)`
still errors; the default retains the tag;
`optimize_prune_before`; `checkout_tag` plus a data read), a Python sync
test with a scalar index (index step runs,
a second optimize works, untagged versions are pruned,
`checkout("release-1")` works), a Python async test and a
Node test.

**What I could and could not verify (please read):**
- I could NOT compile Rust here (not enough disk), so `cargo check`,
`cargo test`, the Python/Node binding builds
and therefore the fixed-path assertions of the new Rust, Python and Node
tests have not been run. This is why the
PR is a draft; I will mark it ready once CI is green and fix whatever it
finds.
- Ran: both new Python tests fail on the released 0.40.0 wheel with the
exact error above (so they do catch the
bug); pylance 13.0.0 confirms the semantics
(`cleanup_old_versions(error_if_tagged_old_versions=False)` removes
the untagged versions and keeps the tagged one, still readable with
`lance.dataset(version="keep")`); in Lance
v14.0.0-beta.7 `cleanup.rs` the policy field is `bool`, the builder
default is `true`, and Lance's own
`auto_cleanup_hook` uses `Some(false)`; `rustfmt --edition 2024 --check`
passes on `optimize.rs`.
- The Rust and Node tests rely on timestamp ordering
(`nextMillisecond()`, `days(0)`/`now()`), so they could be
  flaky: tell me if you prefer another approach.

Behaviour change: anyone who relied on the error to learn that a tag
blocks cleanup now gets a silent retain.
Not changed: the deprecated sync `LanceTable.cleanup_old_versions` calls
pylance directly and still errors on
tagged versions (the keyword may not exist in the oldest supported
pylance); `freshness.rs:1377` and
`refresh.rs:1077` still pass `None` (I did not check what those paths do
with tagged versions).

Fixes #4543

python/python/tests/test_table.py

@@ -4792,6 +4792,57 @@ def test_optimize_warns_on_zero_cleanup_sync(mem_db: DBConnection):
     assert all("concurrent" in str(w.message) for w in seen)
 
 
+def test_optimize_retains_tagged_versions(tmp_db: DBConnection):
+    # A tagged version older than the cleanup cutoff must be retained, not make
+    # optimize() fail part-way (after compaction, before the index step).
+    table = tmp_db.create_table(
+        "test", data=pa.table({"id": list(range(100)), "text": ["a"] * 100})
+    )
+    table.create_index("id", config=BTree())
+    table.tags.create("release-1", table.version)
+    tagged_version = table.version
+    for i in range(3):
+        table.add(pa.table({"id": [1000 + i], "text": ["new"]}))
+    assert table.stats()["fragment_stats"]["num_fragments"] == 4
+    assert table.index_stats("id_idx").num_unindexed_rows == 3
+    sleep(1.1)
+
+    table.optimize(cleanup_older_than=timedelta(seconds=1))
+
+    # The index step ran after the prune step
+    assert table.index_stats("id_idx").num_unindexed_rows == 0
+    # It stays optimizable
+    table.optimize(cleanup_older_than=timedelta(seconds=1))
+    assert table.stats()["fragment_stats"]["num_fragments"] == 1
+
+    # Untagged old versions were pruned, the tagged one was retained
+    versions = [v["version"] for v in table.list_versions()]
+    assert tagged_version in versions
+    for added in range(tagged_version + 1, tagged_version + 4):
+        assert added not in versions
+    assert len(table) == 103
+
+    table.checkout("release-1")
+    assert len(table) == 100
+    assert table.to_arrow().num_rows == 100
+
+
+@pytest.mark.asyncio
+async def test_optimize_retains_tagged_versions_async(tmp_db_async: AsyncConnection):
+    table = await tmp_db_async.create_table("test", data=[{"x": [1]}])
+    await table.tags.create("keep", await table.version())
+    await table.add(data=[{"x": [2]}])
+    await table.add(data=[{"x": [3]}])
+    sleep(1.1)
+
+    stats = await table.optimize(cleanup_older_than=timedelta(seconds=1))
+    assert stats.prune.old_versions_removed >= 1
+
+    await table.checkout("keep")
+    assert await table.count_rows() == 1
+    assert await table.query().to_arrow() == pa.table({"x": [[1]]})
+
+
 def test_replace_field_metadata(tmp_path):
     db = lancedb.connect(tmp_path)
     table = db.create_table("my_table", data=[{"x": 0}])