rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-08-18 12:08:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(table): checkpoint_lsm, flush_lsm, compact_lsm, get_lsm_stats (#3736)

Browse Source 
Converge a table's LSM write path into its base table, and inspect it.

`checkpoint_lsm` is `flush` then `compact`, repeated until the fresh
tier is empty — and the loop runs **client-side**. Putting it on the
server would mean a background task, which means a single-flight intent,
an intent that leaks on panic, a bounded-iteration policy, an "is it
done" observable, and a story for every way a client can vanish
mid-operation. None of that exists in this shape: each request does a
bounded unit of work and reports what is left, so completion is *carried
in the responses* rather than inferred from a shared counter that cannot
distinguish "converged" from "hasn't started yet".

Best-effort by construction. Nothing is frozen, so `converged` means L0
was empty as of the last pass. It is idempotent, abandonable at any
point with zero consequence, and safe to run on a cadence — an
already-converged table costs one round trip and zero compaction passes,
because `flush` reports `generations_remaining` and the loop is never
entered.

## The failure taxonomy is the load-bearing part

Five distinct conditions used to arrive at a client as one 503.
`Error::LsmRoute` carries a classification read from the response body's
namespace error code **at the point of receipt** — before any generic
helper folds the body into a string and keeps only the status.

| condition | wire | client action |
|---|---|---|
| contention (latch held / pool saturated) | 429, code 21 | retry with
backoff |
| owning node draining | 503, code 19 `InvalidTableState` | **stop** |
| fenced / no slot / transport | 503, code 17 | retry with backoff |
| registry entry vanished | 404 | re-issue from `flush` (capped) |
| table being dropped / not WAL-backed | 409 / 400 | stop |

Draining is terminal because the drain gate is a one-way latch —
retrying spins until the deadline to report a failure that was knowable
on the first response. Transport retry is disabled on these routes for
the same reason: it treats every 503 alike and would burn its budget
before the classifier ever saw the body.

`get_lsm_stats` returns `Option<LsmStats>`, matching
`get_lsm_write_spec` — `None` only when the table has no LSM write path,
since a struct of zeros would read as measurements.

Python bindings mirror all four, preserving per-bucket detail rather
than flattening to a table-level summary.

## Testing

Six new unit tests against the mocked endpoint, plus the taxonomy
round-trip:
- flush into an empty L0 issues **zero** compact calls (asserts the call
count — `generations_consumed: 0` is also true of a loop that ran a
pointless pass)
- the loop drives compact until the server reports zero remaining
- **contention is not draining**: a 429 retries and converges; asserts
the retry count
- a draining node stops after **exactly one** request, no retries
- stats round-trips fully populated; `include_generation_rows` off by
default
- every `(status, code)` pair classifies correctly, including
unparseable 503 bodies falling back to *retryable* rather than terminal

`cargo test -p lancedb --features remote --lib`: 723 passed.

## Notes for review

- Depends on the sibling lance change returning `SealedGeneration` from
`force_seal_active` only at the *server* level — no lance API is used
here.
- The branch is based on `codex/update-lance-10-0-0-beta-5`, so it
carries one extra commit (`chore: update lance dependency to
v10.0.0-beta.5`) that is not part of this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: lancedb automation <robot@lancedb.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dan Rammer
2026-08-07 13:44:49 -05:00
committed by GitHub
parent ec21e37040
commit 79ba076429
8 changed files with 1320 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/src/table.rs
+107 -1
View File
@@ -28,11 +28,72 @@ use pyo3::{
Bound, FromPyObject, Py, PyAny, PyRef, PyResult, Python,
exceptions::{PyRuntimeError, PyValueError},
pyclass, pyfunction, pymethods,
types::{IntoPyDict, PyAnyMethods, PyBytes, PyDict, PyDictMethods},
types::{IntoPyDict, PyAnyMethods, PyBytes, PyDict, PyDictMethods, PyList, PyListMethods},
};
mod scannable;
/// Convert `LsmStats` to a Python dict, preserving the per-bucket list.
///
/// Deliberately not flattened to a table-level summary: a table is N
/// buckets on one node, and the per-bucket detail is the reason the
/// endpoint exists — flattening hides the single hot bucket someone opened
/// it to find.
fn lsm_stats_to_py(py: Python<'_>, stats: &lancedb::table::LsmStats) -> PyResult<Py<PyDict>> {
let out = PyDict::new(py);
let buckets = PyList::empty(py);
for b in &stats.buckets {
let e = PyDict::new(py);
e.set_item("shard_id", &b.shard_id)?;
e.set_item("status", &b.status)?;
e.set_item("writer_epoch", b.writer_epoch)?;
e.set_item("manifest_version", b.manifest_version)?;
e.set_item("current_generation", b.current_generation)?;
e.set_item(
"replay_after_wal_entry_position",
b.replay_after_wal_entry_position,
)?;
e.set_item(
"wal_entry_position_last_seen",
b.wal_entry_position_last_seen,
)?;
let generations = PyList::empty(py);
for g in &b.generations {
let ge = PyDict::new(py);
ge.set_item("generation", g.generation)?;
ge.set_item("bytes", g.bytes)?;
ge.set_item("rows", g.rows)?;
generations.append(ge)?;
}
e.set_item("generations", generations)?;
e.set_item("compacting", b.compacting)?;
e.set_item(
"memtables",
b.memtables
.as_ref()
.map(|ms| {
let l = PyList::empty(py);
for m in ms {
let d = PyDict::new(py);
d.set_item("generation", m.generation)?;
d.set_item("rows", m.rows)?;
d.set_item("bytes", m.bytes)?;
d.set_item("batches", m.batches)?;
d.set_item("indexes", m.indexes.clone())?;
l.append(d)?;
}
PyResult::Ok(l.unbind())
})
.transpose()?,
)?;
buckets.append(e)?;
}
out.set_item("buckets", buckets)?;
Ok(out.unbind())
}
#[derive(FromPyObject)]
enum PredicateArg {
Expr(PyExpr),
@@ -1339,6 +1400,51 @@ impl Table {
})
}
/// Converge the table's LSM write path into its base table.
///
/// Best-effort: with writes flowing, new rows may land after the last
/// pass. Errors if the table stops making progress.
pub fn checkpoint_lsm(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {
inner.checkpoint_lsm().await.infer_error()
})
}
/// Seal every bucket's active memtable into L0.
pub fn flush_lsm(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(
self_.py(),
async move { inner.flush_lsm().await.infer_error() },
)
}
/// Trigger a background L0 → base pass per bucket. Returns once the
/// passes are dispatched, not once they finish — watch `get_lsm_stats`.
pub fn compact_lsm(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {
inner.compact_lsm().await.infer_error()
})
}
/// Live LSM state, or `None` when the LSM write path is not enabled.
#[pyo3(signature = (include_generation_rows=false))]
pub fn get_lsm_stats(
self_: PyRef<'_, Self>,
include_generation_rows: bool,
) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {
let stats = inner
.get_lsm_stats(include_generation_rows)
.await
.infer_error()?;
Python::attach(|py| stats.map(|s| lsm_stats_to_py(py, &s)).transpose())
})
}
pub fn close_lsm_writers(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {