Heng Ge
f655f62e09
## What
MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).
## How
- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
- **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.
Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
4.5 KiB
@lancedb/lancedb • Docs
@lancedb/lancedb / MergeInsertBuilder
Class: MergeInsertBuilder
A builder used to create and run a merge insert operation
Constructors
new MergeInsertBuilder()
new MergeInsertBuilder(native, schema): MergeInsertBuilder
Construct a MergeInsertBuilder. Internal use only.
Parameters
-
native:
NativeMergeInsertBuilder -
schema:
Schema<any> |Promise<Schema<any>>
Returns
Methods
execute()
execute(data, execOptions?): Promise<MergeResult>
Executes the merge insert operation
Parameters
-
data:
Data -
execOptions?:
Partial<WriteExecutionOptions>
Returns
Promise<MergeResult>
the merge result
useIndex()
useIndex(useIndex): MergeInsertBuilder
Controls whether to use indexes for the merge operation.
When set to true (the default), the operation will use an index if available
on the join key for improved performance. When set to false, it forces a full
table scan even if an index exists. This can be useful for benchmarking or when
the query optimizer chooses a suboptimal path.
Parameters
- useIndex:
booleanWhether to use indices for the merge operation. Defaults totrue.
Returns
useLsm()
useLsm(enable): MergeInsertBuilder
Control MemWAL routing for this merge.
By default (unset), a mergeInsert on a table with an LSM write spec is
routed through Lance's MemWAL shard writer, and a table without one uses the
standard path.
Parameters
- enable:
booleantrueforces MemWAL routing and errors if the table has no LSM write spec.falseforces the standard write path even when a spec is set.
Returns
validateSingleShard()
validateSingleShard(validateSingleShard): MergeInsertBuilder
Controls how an LSM merge checks that its input targets a single shard.
When a table has an LSM write spec, every row in a mergeInsert call must
route to the same shard. When true (the default), every row is inspected
to verify this. When false, only the first row is inspected and the
shard it routes to is used for the whole input — a faster path for callers
that have already pre-sharded their input. Has no effect on tables without
an LSM write spec.
Parameters
- validateSingleShard:
booleanWhether to check every row routes to one shard. Defaults totrue.
Returns
whenMatchedUpdateAll()
whenMatchedUpdateAll(options?): MergeInsertBuilder
Rows that exist in both the source table (new data) and the target table (old data) will be updated, replacing the old row with the corresponding matching row.
If there are multiple matches then the behavior is undefined. Currently this causes multiple copies of the row to be created but that behavior is subject to change.
An optional condition may be specified. If it is, then only matched rows that satisfy the condtion will be updated. Any rows that do not satisfy the condition will be left as they are. Failing to satisfy the condition does not cause a "matched row" to become a "not matched" row.
The condition should be an SQL string. Use the prefix target. to refer to rows in the target table (old data) and the prefix source. to refer to rows in the source table (new data).
For example, "target.last_update < source.last_update"
Parameters
-
options?
-
options.where?:
string
Returns
whenNotMatchedBySourceDelete()
whenNotMatchedBySourceDelete(options?): MergeInsertBuilder
Rows that exist only in the target table (old data) will be deleted. An optional condition can be provided to limit what data is deleted.
Parameters
-
options?
-
options.where?:
stringAn optional condition to limit what data is deleted
Returns
whenNotMatchedInsertAll()
whenNotMatchedInsertAll(): MergeInsertBuilder
Rows that exist only in the source table (new data) should be inserted into the target table.