refactor(table): shrink LsmStats to measured per-bucket fields

`get_lsm_stats` reported four table-level rollups, three per-bucket
derived fields, and two pieces of in-process state. All of it is either
arithmetic the caller can do on the per-bucket list (`bucket_count`,
`generations_total`, `l0_bytes_total`, `memtable_rows_total`, `l0_bytes`,
`wal_lag`) or a value that is true only for the microsecond it was read
(`fenced`, `compaction_in_progress`). Drop them.

What is left is measured: manifest metadata, one entry per flushed L0
generation, and one entry per in-memory memtable. Three shape changes
follow:

* `active_memtable` + `frozen_memtables` collapse into `memtables`,
  ordered oldest first with the active one last. Absent — not empty — for
  a `Sealed` bucket, whose writer is torn down.
* Generation `rows` is unconditional, so `LsmStatsOptions` and its
  `include_generation_rows` flag are gone. A row count that had to be
  asked for was routinely absent from the one report that needed it.
* `memtable_indexes` becomes a list of names on each memtable, since each
  carries its own index store and a frozen one predates an index created
  after it froze.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

python/python/lancedb/_lancedb.pyi

@@ -292,7 +292,7 @@ class Table:
     ) -> dict: ...
     async def flush_lsm(self) -> dict: ...
     async def compact_lsm(self, max_generations_per_bucket: Optional[int]) -> dict: ...
-    async def get_lsm_stats(self, include_generation_rows: bool) -> Optional[dict]: ...
+    async def get_lsm_stats(self) -> Optional[dict]: ...
     async def close_lsm_writers(self) -> None: ...
     @property
     def tags(self) -> Tags: ...

python/python/lancedb/table.py

@@ -3903,12 +3903,10 @@ class LanceTable(Table):
             )
         )
 
-    def get_lsm_stats(self, *, include_generation_rows: bool = False) -> Optional[dict]:
+    def get_lsm_stats(self) -> Optional[dict]:
         """Synchronous version of
         [`AsyncTable.get_lsm_stats`][lancedb.AsyncTable.get_lsm_stats]."""
-        return LOOP.run(
-            self._table.get_lsm_stats(include_generation_rows=include_generation_rows)
-        )
+        return LOOP.run(self._table.get_lsm_stats())
 
     def close_lsm_writers(self) -> None:
         """Close cached MemWAL shard writers. See
@@ -4676,10 +4674,8 @@ class AsyncTable:
         """
         return await self._inner.compact_lsm(max_generations_per_bucket)
 
-    async def get_lsm_stats(
-        self, *, include_generation_rows: bool = False
-    ) -> Optional[dict]:
-        """Read live LSM state: per-bucket detail plus a table-level aggregate.
+    async def get_lsm_stats(self) -> Optional[dict]:
+        """Read live per-bucket LSM state.
 
         Answers "how far behind is my fresh tier", "which bucket is hot", and
         "why is my fresh-tier vector search brute-force". Mutates no table
@@ -4688,14 +4684,8 @@ class AsyncTable:
 
         Returns ``None`` — and only — when the LSM write path is not enabled
         for the table.
-
-        Parameters
-        ----------
-        include_generation_rows
-            Report row counts per L0 generation. Off by default: each costs
-            one Lance manifest read.
         """
-        return await self._inner.get_lsm_stats(include_generation_rows)
+        return await self._inner.get_lsm_stats()
 
     async def close_lsm_writers(self) -> None:
         """Drain and close any cached MemWAL shard writers for this table.