mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-20 13:08:23 +00:00
Dan Rammer
f1c4967eeb
## Why Four of the eight LSM methods are **remote-only in the core**. `impl BaseTable for NativeTable` implements only `set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`, `compact_lsm` and `get_lsm_stats` fall through to trait defaults returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and `checkpoint_lsm` is built on all three. That explains the state of the bindings: Node had bound the four that work against a local table and stopped, so a Cloud user could install an LSM write spec but had no way to observe fresh-tier state or drive a checkpoint. Java had none of it at all. | SDK | set/unset/get spec | closeWriters | flush | compact | getStats | checkpoint | |---|---|---|---|---|---|---| | Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Node *(before)* | ✅ | ✅ | — | — | — | — | | **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** | | Java *(before)* | — | — | — | — | — | — | | **Java (after)** | **new** | n/a | **new** | **new** | **new** | **new** | Go and C are separate repos and are out of scope here. `closeLsmWriters` drains cached in-process shard writers, so it has no meaning for Java, which is a pure REST client. ## Node Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and `getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats` / `MemtableStats` objects — typed rather than a JSON blob, matching the existing `LsmWriteSpec` object in the same file, with `u64` cast to `i64` per that file's convention. Because these four are remote-only, the new tests assert each binding reaches the core and surfaces `NotSupported` against a local table. That covers the wiring; behavior against a real endpoint stays covered by the mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`. ## Python No new methods. All eight are on `LanceTable`, `AsyncTable` and `RemoteTable` — the last four landed on the sync `RemoteTable` in #3961, which is merged into this branch. What was missing here was reachability. `LsmWriteSpec` was importable only from the private `lancedb._lancedb`, appearing in `table.py` solely under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no mention of it, which per the repo's docs guidance means it rendered nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in `__all__`, and documented. ## Java Java reaches LanceDB purely over REST through the generated Lance Namespace client, and these routes are not in that spec, so they are issued through a small dedicated client rather than added to the spec. That call is revisitable — LSM is one of four unspecified route families alongside `multipart_write`, `page_cache/prewarm` and `branches/diff|merge`. If those are ever regularized into the spec as a group, `LanceDbTableLsm` is one file that gets deleted. `LsmWriteSpec` here is deliberately **not** `org.lance.memwal.InitializeMemWalParams`. That type defaults to maintaining *no* indexes where a spec here defaults to maintaining *every* index, and it cannot express the `null` that asks the server to resolve the set: | Value | On the wire | Meaning | |---|---|---| | unset (null) | `null` | Server resolves **every** maintainable index | | `Collections.emptyList()` | `[]` | Maintain **none** | | `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those | A dedicated test pins null and `[]` as distinct on the wire, since collapsing them is the failure mode that motivated a LanceDB-owned type. `checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs` with its constants and status semantics intact: 429/503 retried in place against an 8-budget, 421 restarting from flush against a 3-budget, 5s poll, and a target watermark fixed after the seal so it terminates under write load. `getLsmStats` returns typed `LsmStats` / `BucketStats` / `GenerationStats` / `MemtableStats`, mirroring the Rust structs in `rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes. Decoding is strict — see below. ## Review feedback Both gatekeeper findings were real. Each was reproduced against the scripted test server first, and each fix ships with the reproducer as a regression test. **The transport was doubling every checkpoint retry budget.** `HttpClients.createDefault()` installs Apache's default response retry strategy, whose retryable-status list is exactly 429 and 503 — the two statuses `isRetryable` owns. A 429 held against `flush_lsm` issued **18** wire requests where the loop intends 9, and `compact_lsm` was retried in place despite the loop being built to fall through to a fresh stats poll instead. Timing confirmed the mechanism: that run took 25.4s ≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry interval. Automatic retries are now disabled, so the checkpoint loop is the sole owner of the 421/429/503 transitions. A side effect worth noting: `testCheckpointRetriesRetryableStatusInPlace` was passing on a transport-absorbed 429 and never reaching `issue()`'s retry branch at all. It now exercises the real path. **Stats decoding failed open.** `getLsmStats` read the response with Jackson's `path()`, which yields a missing node that iterates as an empty array — making "malformed" indistinguishable from "no buckets", which is indistinguishable from "drained". Four separate payloads made `checkpointLsm()` report convergence for a checkpoint that never ran: | Response | Before | Now | |---|---|---| | `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ | | `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` | | empty response body | **reported success** | `IllegalStateException` | | bucket missing required fields | **reported success** | `IllegalStateException` | The empty-body row is the one to weight: a proxy 200 with no body is a realistic production event, and it silently reported a checkpoint that never happened. Decoding is now strict and fails closed, matching the serde contract on the Rust side exactly. One deliberate deviation from the review comment, which asked that *only* explicit JSON `null` count as disabled: Rust has `#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to `None` there too. Java now matches that. It is an absent-or-malformed **`buckets`** that fails closed, which is the case the comment was actually protecting. ## Testing - Java: **33 passing** (8 existing + 25 LSM) against a scripted `com.sun.net.httpserver.HttpServer` — no new test dependency. Wire assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`; checkpoint tests cover convergence, not piling onto a latched bucket, 421 restart-from-flush, 429 retry-in-place, terminal-status propagation, reissue exhaustion, the exact wire-request count against the retry budget, and five malformed stats payloads. - Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm run tsc`, `npm run lint`, `npm run docs` all clean. - Python: `ruff format --check` and `ruff check` clean. - Java formatting: `./mvnw -pl lancedb-core spotless:apply` and `spotless:check` both clean under a JDK 11 toolchain. ## Note: spotless needs a pre-16 JDK `./mvnw spotless:apply` fails on JDK 16+ with `JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7, pinned at `java/pom.xml:34`, predates JDK 16's compiler API change. **This is pre-existing** and reproduces on a pristine `main` checkout. It is not a blocker, just a toolchain requirement. Spotless was run against these sources under JDK 11 and both `spotless:apply` and `spotless:check` pass on the whole module: ```shell JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply ``` Bumping the plugin so it works on modern JDKs is still worth doing, but separately from this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
LanceDB Python SDK
A Python library for LanceDB.
Installation
pip install lancedb
Pre-Haswell x86_64 hosts: lancedb-compat
The default lancedb wheel targets x86-64-haswell (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with Illegal instruction at import lancedb.
For those hosts, install the lancedb-compat package instead:
pip install lancedb-compat
Same Python API (import lancedb works as usual). The compat wheel is compiled at the x86-64-v2 baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use lance.simd_info() from Python to verify which tier was selected.
lancedb and lancedb-compat install to the same lancedb/ namespace and conflict at install time. Pick one. To switch, pip uninstall lancedb first, then pip install lancedb-compat (or vice-versa).
If you need a custom baseline (or lancedb-compat isn't yet published for your platform), build from source with the override:
RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
pip install ./target/wheels/lancedb-*.whl
Preview Releases
Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
Usage
Basic Example
import lancedb
db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>')
table = db.open_table('my_table')
results = table.search([0.1, 0.3]).limit(20).to_list()
print(results)
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.