Daniel RammerandClaude Opus 5 106f9ddb31 refactor(table): checkpoint on generation watermarks, not report bodies ...

`checkpoint_lsm` looped flush → compact and read `generations_remaining`
out of each compact response. Two problems. That count is stale the moment
a concurrent write lands, so on a table taking writes "L0 is empty" never
becomes true and the loop only ever exited on its deadline. And carrying
it back at all meant eight public types describing work already visible in
`get_lsm_stats`.

It now fixes a target watermark — the newest generation per bucket, read
after the opening flush — then triggers compaction and polls until no
generation at or below that target remains. Generations created *while* it
runs are deliberately not waited on, which is what makes it terminate
under sustained writes and is why the deadline could go.

* Deleted: `CheckpointOptions`, `CheckpointReport`, `CheckpointStopReason`,
  `CompactOptions`, `CompactReport`, `CompactBucketReport`, `FlushReport`,
  `FlushBucketReport`. `checkpoint_lsm`, `flush_lsm` and `compact_lsm` all
  return `()`; observability is `get_lsm_stats` before and after.
* Progress, not time, is the bound: `MAX_IDLE_POLLS` consecutive polls with
  neither a drop in outstanding generations nor a running pass is an error.
  A saturated pool or a fenced writer fails loudly instead of spinning; a
  slow table waits, since a running pass resets the counter. Wrap the call
  in `tokio::time::timeout` for a wall-clock bound.
* The metric counts generations, not buckets. One pass drains a bounded
  prefix, so a per-bucket boolean reads as "no progress" for every pass but
  the last — a bucket needing more passes than the bound would fail a
  checkpoint that was working.
* `BucketStats` gains `compacting`, and the loop never triggers against a
  bucket already compacting: it would only collect 429s from the latch
  doing the work.
* `get_lsm_stats` takes `include_generation_rows` again, off by default.
  Each count opens an uncached Lance dataset and this loop polls the route.
* `send_lsm_route` splits from its JSON decode — flush and compact answer
  202 with no body, and decoding an empty string is a parse error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-31 14:37:32 -05:00

..

lancedb

refactor(table): checkpoint on generation watermarks, not report bodies

2026-07-31 14:37:32 -05:00

tests

feat(fts): add block size configuration (#3691)

2026-07-24 15:02:38 -07:00