## Summary
The Java SDK pinned `jackson-databind` 2.17.1, and `jackson-core` and
the other Jackson modules came in transitively via `arrow-vector` and
`lance-namespace-apache-client` at 2.16.x–2.17.x, all affected by known
CVEs.
This imports `jackson-bom` 2.18.11 in the parent POM, and declares
`jackson-core`, `jackson-annotations`, `jackson-datatype-jsr310`, and
`jackson-jaxrs-json-provider` as direct (BOM-managed) dependencies of
`lancedb-core`. The direct declarations are needed because the parent's
dependency management doesn't apply to a consumer's transitive
resolution; without them, Maven's nearest-wins mediation still hands
downstream apps the older versions from `arrow-vector`.
## Evidence
`mvn dependency:list` in a separate consumer project depending only on
`lancedb-core`:
| Artifact | Before | After |
|---|---|---|
| jackson-core | 2.16.0 | 2.18.11 |
| jackson-annotations | 2.16.0 | 2.18.11 |
| jackson-databind | 2.17.1 | 2.18.11 |
| jackson-datatype-jsr310 | 2.16.0 | 2.18.11 |
| jackson-jaxrs-base / json-provider | 2.17.1 | 2.18.11 |
| jackson-module-jaxb-annotations | 2.17.1 | 2.18.11 |
## Merge Risk
**Door:** Two-way. Dependency versions only, and they stay within
Jackson 2.x.
**Blast Radius:** `java` dependencies. Downstream users of
`lancedb-core` now get Jackson 2.18.11 transitively.
Fixes OSS-2360, OSS-2337
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`maintained_indexes` on a MemWAL table has three values: a named list,
`[]` for none, and `null` for every index the table has, including ones
created later. This PR makes every client read all three back as they
were set.
- **Rust remote client** (also used by the Python and Node SDKs) reads
`null` as `None`.
- **Java client** reads `null` as `null`, not as an empty list.
- **Docs** for the setter, the field and the getter describe the three
values the same way in Rust, Python, Node and Java. The generated JS
docs are regenerated to match.
## Testing
- `rust/lancedb`: 73 LSM tests pass, including one case per value. With
the decoder typed as a list, the `null` case fails.
- Java: two cases for `null` and `[]` on the read path, run by the Java
CI job.
- `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean.
## Why
Four of the eight LSM methods are **remote-only in the core**. `impl
BaseTable for NativeTable` implements only
`set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`,
`compact_lsm` and `get_lsm_stats` fall through to trait defaults
returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and
`checkpoint_lsm` is built on all three.
That explains the state of the bindings: Node had bound the four that
work against a local table and stopped, so a Cloud user could install an
LSM write spec but had no way to observe fresh-tier state or drive a
checkpoint. Java had none of it at all.
| SDK | set/unset/get spec | closeWriters | flush | compact | getStats |
checkpoint |
|---|---|---|---|---|---|---|
| Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Node *(before)* | ✅ | ✅ | — | — | — | — |
| **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** |
| Java *(before)* | — | — | — | — | — | — |
| **Java (after)** | **new** | n/a | **new** | **new** | **new** |
**new** |
Go and C are separate repos and are out of scope here. `closeLsmWriters`
drains cached in-process shard writers, so it has no meaning for Java,
which is a pure REST client.
## Node
Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and
`getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats`
/ `MemtableStats` objects — typed rather than a JSON blob, matching the
existing `LsmWriteSpec` object in the same file, with `u64` cast to
`i64` per that file's convention.
Because these four are remote-only, the new tests assert each binding
reaches the core and surfaces `NotSupported` against a local table. That
covers the wiring; behavior against a real endpoint stays covered by the
mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`.
## Python
No new methods. All eight are on `LanceTable`, `AsyncTable` and
`RemoteTable` — the last four landed on the sync `RemoteTable` in #3961,
which is merged into this branch.
What was missing here was reachability. `LsmWriteSpec` was importable
only from the private `lancedb._lancedb`, appearing in `table.py` solely
under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no
mention of it, which per the repo's docs guidance means it rendered
nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in
`__all__`, and documented.
## Java
Java reaches LanceDB purely over REST through the generated Lance
Namespace client, and these routes are not in that spec, so they are
issued through a small dedicated client rather than added to the spec.
That call is revisitable — LSM is one of four unspecified route families
alongside `multipart_write`, `page_cache/prewarm` and
`branches/diff|merge`. If those are ever regularized into the spec as a
group, `LanceDbTableLsm` is one file that gets deleted.
`LsmWriteSpec` here is deliberately **not**
`org.lance.memwal.InitializeMemWalParams`. That type defaults to
maintaining *no* indexes where a spec here defaults to maintaining
*every* index, and it cannot express the `null` that asks the server to
resolve the set:
| Value | On the wire | Meaning |
|---|---|---|
| unset (null) | `null` | Server resolves **every** maintainable index |
| `Collections.emptyList()` | `[]` | Maintain **none** |
| `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those |
A dedicated test pins null and `[]` as distinct on the wire, since
collapsing them is the failure mode that motivated a LanceDB-owned type.
`checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs`
with its constants and status semantics intact: 429/503 retried in place
against an 8-budget, 421 restarting from flush against a 3-budget, 5s
poll, and a target watermark fixed after the seal so it terminates under
write load.
`getLsmStats` returns typed `LsmStats` / `BucketStats` /
`GenerationStats` / `MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes.
Decoding is strict — see below.
## Review feedback
Both gatekeeper findings were real. Each was reproduced against the
scripted test server first, and each fix ships with the reproducer as a
regression test.
**The transport was doubling every checkpoint retry budget.**
`HttpClients.createDefault()` installs Apache's default response retry
strategy, whose retryable-status list is exactly 429 and 503 — the two
statuses `isRetryable` owns. A 429 held against `flush_lsm` issued
**18** wire requests where the loop intends 9, and `compact_lsm` was
retried in place despite the loop being built to fall through to a fresh
stats poll instead. Timing confirmed the mechanism: that run took 25.4s
≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry
interval.
Automatic retries are now disabled, so the checkpoint loop is the sole
owner of the 421/429/503 transitions. A side effect worth noting:
`testCheckpointRetriesRetryableStatusInPlace` was passing on a
transport-absorbed 429 and never reaching `issue()`'s retry branch at
all. It now exercises the real path.
**Stats decoding failed open.** `getLsmStats` read the response with
Jackson's `path()`, which yields a missing node that iterates as an
empty array — making "malformed" indistinguishable from "no buckets",
which is indistinguishable from "drained". Four separate payloads made
`checkpointLsm()` report convergence for a checkpoint that never ran:
| Response | Before | Now |
|---|---|---|
| `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ |
| `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` |
| empty response body | **reported success** | `IllegalStateException` |
| bucket missing required fields | **reported success** |
`IllegalStateException` |
The empty-body row is the one to weight: a proxy 200 with no body is a
realistic production event, and it silently reported a checkpoint that
never happened.
Decoding is now strict and fails closed, matching the serde contract on
the Rust side exactly. One deliberate deviation from the review comment,
which asked that *only* explicit JSON `null` count as disabled: Rust has
`#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to
`None` there too. Java now matches that. It is an absent-or-malformed
**`buckets`** that fails closed, which is the case the comment was
actually protecting.
## Testing
- Java: **33 passing** (8 existing + 25 LSM) against a scripted
`com.sun.net.httpserver.HttpServer` — no new test dependency. Wire
assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`;
checkpoint tests cover convergence, not piling onto a latched bucket,
421 restart-from-flush, 429 retry-in-place, terminal-status propagation,
reissue exhaustion, the exact wire-request count against the retry
budget, and five malformed stats payloads.
- Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm
run tsc`, `npm run lint`, `npm run docs` all clean.
- Python: `ruff format --check` and `ruff check` clean.
- Java formatting: `./mvnw -pl lancedb-core spotless:apply` and
`spotless:check` both clean under a JDK 11 toolchain.
## Note: spotless needs a pre-16 JDK
`./mvnw spotless:apply` fails on JDK 16+ with
`JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7,
pinned at `java/pom.xml:34`, predates JDK 16's compiler API change.
**This is pre-existing** and reproduces on a pristine `main` checkout.
It is not a blocker, just a toolchain requirement. Spotless was run
against these sources under JDK 11 and both `spotless:apply` and
`spotless:check` pass on the whole module:
```shell
JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply
```
Bumping the plugin so it works on modern JDKs is still worth doing, but
separately from this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lance Release
Will JonesandClaude Opus 5.5
XY Zhan
Dan RammerandClaude Opus 5