mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 20:18:37 +00:00
Dan Rammer
79ba076429
Converge a table's LSM write path into its base table, and inspect it. `checkpoint_lsm` is `flush` then `compact`, repeated until the fresh tier is empty — and the loop runs **client-side**. Putting it on the server would mean a background task, which means a single-flight intent, an intent that leaks on panic, a bounded-iteration policy, an "is it done" observable, and a story for every way a client can vanish mid-operation. None of that exists in this shape: each request does a bounded unit of work and reports what is left, so completion is *carried in the responses* rather than inferred from a shared counter that cannot distinguish "converged" from "hasn't started yet". Best-effort by construction. Nothing is frozen, so `converged` means L0 was empty as of the last pass. It is idempotent, abandonable at any point with zero consequence, and safe to run on a cadence — an already-converged table costs one round trip and zero compaction passes, because `flush` reports `generations_remaining` and the loop is never entered. ## The failure taxonomy is the load-bearing part Five distinct conditions used to arrive at a client as one 503. `Error::LsmRoute` carries a classification read from the response body's namespace error code **at the point of receipt** — before any generic helper folds the body into a string and keeps only the status. | condition | wire | client action | |---|---|---| | contention (latch held / pool saturated) | 429, code 21 | retry with backoff | | owning node draining | 503, code 19 `InvalidTableState` | **stop** | | fenced / no slot / transport | 503, code 17 | retry with backoff | | registry entry vanished | 404 | re-issue from `flush` (capped) | | table being dropped / not WAL-backed | 409 / 400 | stop | Draining is terminal because the drain gate is a one-way latch — retrying spins until the deadline to report a failure that was knowable on the first response. Transport retry is disabled on these routes for the same reason: it treats every 503 alike and would burn its budget before the classifier ever saw the body. `get_lsm_stats` returns `Option<LsmStats>`, matching `get_lsm_write_spec` — `None` only when the table has no LSM write path, since a struct of zeros would read as measurements. Python bindings mirror all four, preserving per-bucket detail rather than flattening to a table-level summary. ## Testing Six new unit tests against the mocked endpoint, plus the taxonomy round-trip: - flush into an empty L0 issues **zero** compact calls (asserts the call count — `generations_consumed: 0` is also true of a loop that ran a pointless pass) - the loop drives compact until the server reports zero remaining - **contention is not draining**: a 429 retries and converges; asserts the retry count - a draining node stops after **exactly one** request, no retries - stats round-trips fully populated; `include_generation_rows` off by default - every `(status, code)` pair classifies correctly, including unparseable 503 bodies falling back to *retryable* rather than terminal `cargo test -p lancedb --features remote --lib`: 723 passed. ## Notes for review - Depends on the sibling lance change returning `SealedGeneration` from `force_seal_active` only at the *server* level — no lance API is used here. - The branch is based on `codex/update-lance-10-0-0-beta-5`, so it carries one extra commit (`chore: update lance dependency to v10.0.0-beta.5`) that is not part of this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lancedb automation <robot@lancedb.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>