**The maintained set is an intent, not a snapshot.** An unnamed set is
stored as
the intent to maintain every index, which lance re-resolves whenever it
builds a
MemTable, so an index created after the spec installs is maintained too.
A named
set is checked here, where the caller can be told which names exist; an
index of
a kind the MemWAL cannot mirror fails a named set and is skipped by an
unnamed
one.
**A newly created index reaches a writer that is already open.** lancedb
caches a
`ShardWriter` per shard and reuses it, so `create_index` installs the
current set
on the cached writer. A failure there is reported rather than
propagated: the
index is committed durably by then, and the writer this process holds
may be
fenced or poisoned, which is no reason to fail creating an index.
**A table carries one spec.** `set_lsm_write_spec` fails while one is
installed
and names `unset_lsm_write_spec` in the error, because the generations
already
written were homed under that spec. Accepting a repeat that changed only
the
maintained set would hand a caller who also passed a different sharding
a success
that changed nothing they asked for.
**The read path asks what is actually carried.** An unnamed set leaves
the stored
list empty, so the list alone cannot decide whether an index is
maintained. The
read asks the spec *and* every resident MemTable whether it carries the
index, so
a MemTable built before the index joined the set refuses the read rather
than
answering without its rows. Both spec routes report an unnamed set as
`null`
rather than `[]`: maintain whatever the table gains, and maintain
nothing, are
different settings.
## Known gap
The guard above covers resident MemTables. Once a writer flushes and
closes none
is resident, so a retained generation written before the index existed
still gets
an indexed read that omits its rows.
This window is **wider than on main**: a set resolved at install time
excluded an
index created later, so the read was refused; an intent includes it, and
the
refusal only holds while a MemTable is resident. Closing it needs the
generation's
own coverage, which this layer cannot see — its snapshot carries a
generation and
a path — so it belongs in lance's scanner or flush path.
## Testing
`cargo nextest run -p lancedb --lib --features remote` — **1479 passed,
3
skipped**. `cargo fmt --all --check`, and `cargo clippy --features
remote --lib
--tests --examples -- -D warnings`, both clean.
Covered: the round trip of an unnamed set, a set over an installed spec
refused,
a spec replaced through unset, a read refused while a resident MemTable
predates
the index, and an index created mid-write reaching the open writer.
Two Python binding tests move with the semantics: an unnamed set now
reads back
as `None` rather than the list it resolves to, and the refusal is
matched on
`already set`. Neither was run locally — no Python build here — so CI is
the
check on them.
PyO3 exposed `LsmWriteSpec` with its default `builtins` module, causing
mkdocstrings to resolve the public `lancedb.LsmWriteSpec` re-export as
`builtins.LsmWriteSpec` and fail the documentation build. Declare the
native extension module and pin the public re-export with a regression
test.
This also applies the repository's current Ruff formatter to seven
previously unformatted Python scripts.
## What
`LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`:
| value | meaning |
|---|---|
| `None` (new default) | every index the MemWAL supports, resolved when
the spec is installed |
| `Some([])` | maintain nothing — a scan/filter-only WAL table |
| `Some([..])` | exactly these, taken verbatim |
`with_maintained_indexes` keeps its signature;
`with_no_maintained_indexes()` is new. Surfaced through the remote path
(null on the wire), Python, and Node.
## Why
Callers had to state the maintained set by hand every time, which is
both tedious and easy to get wrong — the common case is "maintain what I
already built."
Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating
to lance's `is_maintainable_index_type`. This is load-bearing rather
than cosmetic: lance does **not** skip an index type its memtable cannot
build, it errors when the shard writer opens, so sweeping up a bitmap
index would fail every memtable claim and leave the table unwritable.
The inferred set excludes those, and an explicit list naming one is now
rejected at spec time instead of at claim time.
## Behavior change
A freshly constructed spec used to maintain **nothing**; it now
maintains **everything supported**. This flipped because napi collapses
`undefined` and `null` to `None`, so TypeScript cannot express "absent
means nothing, null means all" — any other choice makes the bindings
disagree with the wire. The error direction also favors it: an unwanted
maintained index costs memory, while a silently unmaintained one
degrades FTS to an unscored scan.
Three existing tests encoded the old default and are updated rather than
worked around.
## Caveat
The resolved set is a snapshot, not a subscription. An index created
after the spec is installed is not maintained until the spec is unset
and set again. `get_lsm_write_spec` therefore always reports a concrete
list — `None` never round-trips.
## Dependency
Needs a lance release carrying `is_maintainable_index_type`
(lance-format/lance#8095) before this builds against the pinned tag.
Draft until then.
## Testing
38 Rust LSM tests and 10 Python tests pass against a local lance build,
including new coverage that a bitmap index is excluded from inference
and rejected when named, and that `[]` stays distinguishable from null
on the wire.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the
read counterpart to the existing `set_lsm_write_spec` /
`unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is
not enabled; otherwise reconstructs the spec (mode, shard column,
`num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly
as installed.
## Changes
- **Rust core (`NativeTable`)** — reconstructs the spec from
`mem_wal_index_details()`, resolving the shard column from its Lance
field id via the dataset schema. This is a raw metadata read, so it is
unaffected by `describe_indices` system-index filtering.
- **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index
through `index/list` with `include_system: true` (so the curated
`list_indices` surface stays unchanged), then parses the index `details`
JSON. It matches the index by name and ignores `index_type`, so no
client `IndexType` variant is needed. It uses the **server-resolved
`column` name** from the details (Lance field ids do not travel to the
remote client).
- **Python + TypeScript bindings** — sync and async, mirroring
`set`/`unset`, with round-trip tests (bucket / identity / unsharded,
plus `None` when unset).
## Tests
- Rust: native round-trip unit test + remote mock-endpoint tests
(present + absent). All green (`cargo test --features remote -p
lancedb`).
- Python/TS: round-trip tests added; binding-runtime execution runs in
CI.
## Dependencies for the remote path
The remote path is complete on the client side but depends on two
out-of-repo pieces to work end-to-end:
1. **lance** — emit the server-resolved shard **`column`** name in the
MemWAL index `details` JSON (field ids can't reach the client). See
lance-format/lance#7667.
2. **server** — honor `include_system` on `index/list` so the
`__lance_mem_wal` entry is returned for this read.
Against an older server (no `include_system`), the remote getter
degrades gracefully to `Ok(None)` rather than erroring.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Bump lance dependency from `v7.0.0-beta.13` to `v7.0.0-rc.1`
- Remove PK constraint from `LsmWriteSpec::Bucket` docs and
`Table::set_lsm_write_spec` docs
- Remove test assertions that expected rejection when no PK is set or
when bucket column != PK
Closes https://github.com/lance-format/lance/issues/6917
## Summary
Split out from #3354
Adds `LsmWriteSpec` and `Table::set_lsm_write_spec` /
`unset_lsm_write_spec` to
install and clear the spec that selects Lance's MemWAL LSM-style write
path for
`merge_insert`.
`LsmWriteSpec` offers three sharding strategies, all built on Lance's
`InitializeMemWalBuilder`:
- `LsmWriteSpec::bucket(column, num_buckets)` — hash-bucket sharding by
the
single-column unenforced primary key.
- `LsmWriteSpec::identity(column)` — identity sharding by the raw value
of a
scalar column.
- `LsmWriteSpec::unsharded()` — a single MemWAL shard.
Each can be refined with `with_maintained_indexes(...)` (indexes the
MemWAL
keeps up to date as rows are appended) and
`with_writer_config_defaults(...)`
(default `ShardWriter` configuration recorded in the MemWAL index, so
every
writer starts from the same defaults). All variants require the table to
have
an unenforced primary key.
- `set_lsm_write_spec` installs the spec by initializing the MemWAL
index;
`unset_lsm_write_spec` removes it (dropping the MemWAL index), reverting
to
the standard `merge_insert` path. `unset` is idempotent.
- Bindings: Python (`LsmWriteSpec.bucket` / `.identity` / `.unsharded`,
`set_lsm_write_spec` / `unset_lsm_write_spec`) and TypeScript
(`setLsmWriteSpec` with `specType` `"bucket"` / `"identity"` /
`"unsharded"`). `RemoteTable` returns `NotSupported`.
The actual `merge_insert` LSM dispatch and `ShardWriter` write path are
a
follow-up — this PR only installs and clears the spec.
XY Zhan
Xuanwo
Dan RammerandClaude Opus 5
Jack Ye
Heng Ge