fix(python): expose optimize compaction options (#3870)

## Summary
- expose typed compaction options on synchronous and asynchronous table
optimization
- translate validated Python options into native Lance compaction
settings
- document the row-based steady-state limit and first-pass/source-budget
caveats
- carry the Lance beta.15 fragment-exclusion option through the typed
Python API

## Root cause
The native Python optimize binding always constructed default compaction
settings, and the public AsyncTable API offered no way to lower the
fragment row target. Wide-row fragments therefore remained compaction
candidates on every later optimize call, repeatedly rewriting the
accumulated table.

## Limitations
This is an explicit row-based mitigation, not byte-native task sizing or
a hard memory cap. Callers must derive target_rows_per_fragment from
expected row size. Existing oversized fragments are rewritten whole on
the first pass; max_bytes_per_file limits output file size rather than
compaction memory; and source budgets admit whole planned tasks, so a
budget below the first task performs no compaction. Splitting tasks by a
byte budget or returning a planner error requires a Lance-side change.

## Validation
- cargo check --release -p lancedb-python
- cargo clippy --quiet -p lancedb-python
- cargo fmt --all --check
- ruff format --check python tests
- ruff check python tests
- uv run --no-sync pytest python/tests/test_table.py -k
optimize_compaction -q (16 passed)

Fixes #2325

<!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>

docs/web-tests/py/test_doc_storage_examples.py

@@ -7,6 +7,36 @@ import pyarrow as pa
 import pytest
 
 
+def test_compaction_tuning(mem_db):
+    payload = b"x" * 1016
+    table = mem_db.create_table(
+        "compaction_tuning", [{"id": i, "payload": payload} for i in range(32)]
+    )
+    table.add([{"id": i, "payload": payload} for i in range(32, 64)])
+    assert table.stats()["fragment_stats"]["num_fragments"] == 2
+
+    # --8<-- [start:storage_compaction_tuning]
+    from lancedb import CompactionOptions
+
+    desired_fragment_bytes = 64 * 1024
+    average_row_bytes = 1024  # Estimate from representative rows.
+    options: CompactionOptions = {
+        "target_rows_per_fragment": max(1, desired_fragment_bytes // average_row_bytes),
+        "batch_size": 16,
+        "num_threads": 1,
+    }
+    table.optimize(compaction_options=options)
+    # --8<-- [end:storage_compaction_tuning]
+
+    assert table.stats()["fragment_stats"]["num_fragments"] == 1
+    assert table.count_rows() == 64
+
+    table.add([{"id": 64, "payload": payload}])
+    table.optimize(compaction_options=options)
+    assert table.stats()["fragment_stats"]["num_fragments"] == 2
+    assert sorted(table.to_arrow()["id"].to_pylist()) == list(range(65))
+
+
 def test_rebuild_vector_index(tmp_db):
     rng = np.random.default_rng(7)
     table = tmp_db.create_table(