rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-26 13:36:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)

Browse Source 
## What

Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.

### 1. MemWAL APIs drop "LSM"

The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.

| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |

TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.

Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.

### 2. Stats use SSTable and table shard

Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.

**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.

**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.

| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |

Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.

`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.

## Compatibility

Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:

- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies

## Validation

- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dan Rammer
2026-10-16 13:33:57 -05:00
committed by GitHub
co-authored by Claude Opus 5
parent 0968c21f4c
commit fe34eb3549
73 changed files with 1853 additions and 1856 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rust/lancedb/src/dataloader/permutation/builder.rs
+9 -9
View File
@@ -250,17 +250,17 @@ impl PermutationBuilder {
}
// Unflushed rows have no row id, so a permutation cannot address them.
match self.base_table.base_table().get_lsm_write_spec().await {
match self.base_table.base_table().get_memwal_spec().await {
Ok(Some(_)) => {
return Err(Error::NotSupported {
message: "the data loader does not support tables with an LSM write \
message: "the data loader does not support tables with a MemWAL write \
spec: rows that have not been flushed to the base table \
have no row id, so a permutation cannot reference them"
.to_string(),
});
}
Ok(None) => {}
// No LSM write path means no spec.
// No MemWAL write path means no spec.
Err(Error::NotSupported { .. }) => {}
Err(err) => return Err(err),
}
@@ -489,9 +489,9 @@ mod tests {
)
.unwrap()
}
"/v1/table/remote_base/get_lsm_write_spec/" => http::Response::builder()
"/v1/table/remote_base/get_memwal_spec/" => http::Response::builder()
.status(200)
.body(br#"{"lsm_write_spec":null}"#.to_vec())
.body(br#"{"memwal_spec":null}"#.to_vec())
.unwrap(),
"/v1/table/remote_base/count_rows/" => {
let body = body.unwrap();
@@ -737,8 +737,8 @@ mod tests {
/// permutation cannot reference them. Reading the base table alone would drop
/// them from training without saying so, so the table is refused instead.
#[tokio::test]
async fn test_permutation_rejects_lsm_write_spec() {
use crate::table::LsmWriteSpec;
async fn test_permutation_rejects_memwal_spec() {
use crate::table::MemWalWriteSpec;
use arrow_array::{Int32Array, RecordBatchIterator};
use arrow_schema::{DataType, Field, Schema};
@@ -766,13 +766,13 @@ mod tests {
table.set_unenforced_primary_key(["idx"]).await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
let err = PermutationBuilder::new(table).build().await.unwrap_err();
assert!(
err.to_string().contains("LSM write spec"),
err.to_string().contains("MemWAL write spec"),
"expected the pre-check to refuse the table, got: {err}"
);
}
rust/lancedb/src/lib.rs
+1 -1
View File
@@ -167,7 +167,7 @@
//! # }
//! ```
// The MemWAL LSM read path (`table::query::lsm`) deepens the `create_plan` future's
// The MemWAL read path (`table::query::lsm`) deepens the `create_plan` future's
// type graph enough to overflow the default trait-recursion limit while evaluating
// auto-traits (`Send`) through the Linux io_uring build's moka cache. Raise it.
#![recursion_limit = "256"]
rust/lancedb/src/materialized_view/refresh.rs
+4 -2
View File
@@ -288,13 +288,15 @@ fn refresh_conflict_filter(dataset: &Dataset) -> Result<KeyExistenceFilter> {
Ok(filter.build())
}
/// Reject MemWAL/LSM state because an ordinary dataset scan cannot see its
/// Reject MemWAL/MemWAL state because an ordinary dataset scan cannot see its
/// un-compacted tiers.
pub(crate) async fn ensure_no_mem_wal(dataset: &Dataset, role: &str, name: &str) -> Result<()> {
let retained = !dataset.list_mem_wal_latest_shard_ids().await?.is_empty();
if retained || dataset.mem_wal_index_details().await?.is_some() {
return Err(Error::NotSupported {
message: format!("{role} '{name}' has an LSM write spec or retained un-compacted rows"),
message: format!(
"{role} '{name}' has a MemWAL write spec or retained un-compacted rows"
),
});
}
Ok(())
rust/lancedb/src/query.rs
+13 -13
View File
@@ -547,22 +547,22 @@ pub trait QueryBase {
/// Control MemWAL read routing for this query.
///
/// By default (unset), when the table carries a MemWAL write spec (see
/// [`crate::Table::set_lsm_write_spec`]), reads are routed through the LSM
/// [`crate::Table::enable_memwal`]), reads are routed through the LSM
/// scanner so they also return data written via the `merge_insert` LSM path
/// that has not yet been compacted into the base table (active/frozen
/// memtables and flushed generations); a table without a spec reads the base
/// memtables and SSTables); a table without a spec reads the base
/// table.
///
/// - `use_lsm(true)` forces LSM routing and errors if the table has no
/// - `use_memwal(true)` forces MemWAL routing and errors if the table has no
/// MemWAL write spec.
/// - `use_lsm(false)` bypasses the MemWAL and reads the base table only,
/// - `use_memwal(false)` bypasses the MemWAL and reads the base table only,
/// even when a spec is present.
///
/// Note: the LSM scanner does not support every query shape (e.g. reranking,
/// hybrid search, `order_by`). On a MemWAL table those shapes error unless
/// `use_lsm(false)` is set, because a base-only read would silently
/// `use_memwal(false)` is set, because a base-only read would silently
/// exclude un-compacted MemWAL data.
fn use_lsm(self, enable: bool) -> Self;
fn use_memwal(self, enable: bool) -> Self;
}
pub trait HasQuery {
@@ -634,8 +634,8 @@ impl<T: HasQuery> QueryBase for T {
self
}
fn use_lsm(mut self, enable: bool) -> Self {
self.mut_query().use_lsm = Some(enable);
fn use_memwal(mut self, enable: bool) -> Self {
self.mut_query().use_memwal = Some(enable);
self
}
}
@@ -900,17 +900,17 @@ pub struct QueryRequest {
/// Controls MemWAL read routing. When unset (the default), a query against a
/// table that carries a MemWAL write spec (see
/// [`crate::Table::set_lsm_write_spec`]) is routed through the LSM scanner so
/// [`crate::Table::enable_memwal`]) is routed through the LSM scanner so
/// it also sees data written via the `merge_insert` LSM path that has not yet
/// been compacted into the base table — the active and frozen in-memory
/// memtables and the flushed (L0) generations, deduplicated by primary key
/// memtables and the SSTables, deduplicated by primary key
/// against the base table (newest generation wins); a table without a spec
/// reads the base table.
///
/// - `Some(true)` forces LSM routing and errors if the table has no MemWAL
/// - `Some(true)` forces MemWAL routing and errors if the table has no MemWAL
/// write spec.
/// - `Some(false)` reads only the base table, bypassing the MemWAL.
pub use_lsm: Option<bool>,
pub use_memwal: Option<bool>,
}
impl Default for QueryRequest {
@@ -930,7 +930,7 @@ impl Default for QueryRequest {
norm: None,
disable_scoring_autoprojection: false,
order_by: None,
use_lsm: None,
use_memwal: None,
}
}
}
rust/lancedb/src/query/wal_fusion.rs
+2 -2
View File
@@ -54,7 +54,7 @@ fn wal_row_id_unsupported() -> Error {
Error::NotSupported {
message: "hybrid search on a MemWAL table cannot return _rowid: the fresh tier has no \
stable row id, and the ids the fusion joins on are synthesized from the \
primary key. Set use_lsm(false) to read the base table only (results will \
primary key. Set use_memwal(false) to read the base table only (results will \
exclude un-compacted MemWAL data)"
.to_string(),
}
@@ -80,7 +80,7 @@ where
{
// A previous query on this table was already refused, so skip straight to
// the key.
if query.request.base.use_lsm != Some(false) && query.parent.hybrid_pk_fusion_learned() {
if query.request.base.use_memwal != Some(false) && query.parent.hybrid_pk_fusion_learned() {
return run(Some(PkFusion::resolve(query).await?)).await;
}
match run(None).await {
rust/lancedb/src/remote/table.rs
+145 -167
View File
@@ -30,12 +30,12 @@ use crate::table::BranchDiff;
use crate::table::CherryPickResult;
use crate::table::DeleteResult;
use crate::table::DropColumnsResult;
use crate::table::LsmStats;
use crate::table::LsmWriteSpec;
use crate::table::MemWalStats;
use crate::table::MemWalWriteSpec;
use crate::table::MergeResult;
use crate::table::Tags;
use crate::table::UpdateResult;
use crate::table::lsm_stats::GetLsmStatsResponse;
use crate::table::memwal_stats::GetMemWalStatsResponse;
use crate::table::merge::MergeFilter;
use crate::table::primary_key;
use crate::table::query::create_multi_vector_plan;
@@ -1020,10 +1020,10 @@ impl<S: HttpSend> RemoteTable<S> {
) -> Result<()> {
params.check_filter()?;
body["prefilter"] = params.prefilter.into();
// Only forward use_lsm when explicitly set; a server that predates it
// Only forward use_memwal when explicitly set; a server that predates it
// ignores the field and routes as it would by default.
if let Some(use_lsm) = params.use_lsm {
body["use_lsm"] = serde_json::Value::Bool(use_lsm);
if let Some(use_memwal) = params.use_memwal {
body["use_memwal"] = serde_json::Value::Bool(use_memwal);
}
if let Some(offset) = params.offset {
body["offset"] = serde_json::Value::Number(serde_json::Number::from(offset));
@@ -3227,42 +3227,43 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
Ok(())
}
async fn flush_lsm(&self) -> Result<()> {
async fn flush_memwal(&self) -> Result<()> {
let request = self
.client
.post(&format!("/v1/table/{}/flush_lsm/", self.identifier));
.post(&format!("/v1/table/{}/flush_memwal/", self.identifier));
self.send_lsm_route(request).await?;
Ok(())
}
async fn compact_lsm(&self) -> Result<()> {
async fn compact_memwal(&self) -> Result<()> {
let request = self
.client
.post(&format!("/v1/table/{}/compact_lsm/", self.identifier));
.post(&format!("/v1/table/{}/compact_memwal/", self.identifier));
self.send_lsm_route(request).await?;
Ok(())
}
async fn get_lsm_stats(&self, include_generation_rows: bool) -> Result<Option<LsmStats>> {
// Read-semantics POST, like `get_lsm_write_spec`.
async fn get_memwal_stats(&self, include_sstable_rows: bool) -> Result<Option<MemWalStats>> {
// Read-semantics POST, like `get_memwal_spec`.
let request = self
.client
.post(&format!("/v1/table/{}/get_lsm_stats/", self.identifier))
.post(&format!("/v1/table/{}/get_memwal_stats/", self.identifier))
.json(&serde_json::json!({
"include_generation_rows": include_generation_rows,
"include_sstable_rows": include_sstable_rows,
}));
let (request_id, response) = self.send_lsm_route(request).await?;
let body = response.text().await.err_to_http(request_id.clone())?;
let parsed: GetLsmStatsResponse = serde_json::from_str(&body).map_err(|e| Error::Http {
source: format!("Failed to parse get_lsm_stats response: {e}").into(),
request_id,
status_code: None,
})?;
// `null` — and only — when the table has no LSM write path.
Ok(parsed.lsm_stats)
let parsed: GetMemWalStatsResponse =
serde_json::from_str(&body).map_err(|e| Error::Http {
source: format!("Failed to parse get_memwal_stats response: {e}").into(),
request_id,
status_code: None,
})?;
// `null` — and only — when the table has no MemWAL write path.
Ok(parsed.memwal_stats)
}
async fn set_lsm_write_spec(&self, spec: LsmWriteSpec) -> Result<()> {
async fn enable_memwal(&self, spec: MemWalWriteSpec) -> Result<()> {
self.check_mutable().await?;
// Map the spec onto the server's request DTO. `sharding` is internally
@@ -3270,7 +3271,7 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
// `maintained_indexes` asks the server to resolve every maintainable
// index at HEAD; a list is verbatim, an empty one meaning none.
let sharding = match &spec {
LsmWriteSpec::Bucket {
MemWalWriteSpec::Bucket {
column,
num_buckets,
..
@@ -3279,11 +3280,11 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
"column": column,
"num_buckets": num_buckets,
}),
LsmWriteSpec::Identity { column, .. } => serde_json::json!({
MemWalWriteSpec::Identity { column, .. } => serde_json::json!({
"mode": "identity",
"column": column,
}),
LsmWriteSpec::Unsharded { .. } => serde_json::json!({ "mode": "unsharded" }),
MemWalWriteSpec::Unsharded { .. } => serde_json::json!({ "mode": "unsharded" }),
};
let body = serde_json::json!({
"sharding": sharding,
@@ -3293,45 +3294,40 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
let request = self
.client
.post(&format!(
"/v1/table/{}/set_lsm_write_spec/",
self.identifier
))
.post(&format!("/v1/table/{}/enable_memwal/", self.identifier))
.json(&body);
let (request_id, response) = self.send(request, true).await?;
self.check_table_response(&request_id, response).await?;
Ok(())
}
async fn unset_lsm_write_spec(&self) -> Result<()> {
async fn disable_memwal(&self) -> Result<()> {
self.check_mutable().await?;
let request = self.client.post(&format!(
"/v1/table/{}/unset_lsm_write_spec/",
self.identifier
));
let request = self
.client
.post(&format!("/v1/table/{}/disable_memwal/", self.identifier));
let (request_id, response) = self.send(request, true).await?;
self.check_table_response(&request_id, response).await?;
self.wal_pk_fusion.forget(); // WAL-PK-FUSION: delete.
Ok(())
}
async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
async fn get_memwal_spec(&self) -> Result<Option<MemWalWriteSpec>> {
// Read counterpart to set/unset, resolved server-side against HEAD. The
// server reads the spec from the `__lance_mem_wal` system index (shard
// column mapped from its Lance field id against the current schema) and
// re-encodes it into the same sophon-owned shape the set endpoint
// accepts — no lance/lancedb types cross the wire. `lsm_write_spec` is
// null when the LSM write path is not enabled for the table.
let request = self.client.post(&format!(
"/v1/table/{}/get_lsm_write_spec/",
self.identifier
));
// accepts — no lance/lancedb types cross the wire. `memwal_spec` is
// null when the MemWAL write path is not enabled for the table.
let request = self
.client
.post(&format!("/v1/table/{}/get_memwal_spec/", self.identifier));
let (request_id, response) = self.send(request, true).await?;
let response = self.check_table_response(&request_id, response).await?;
let body = response.text().await.err_to_http(request_id.clone())?;
// Mirror of sophon's `Sharding` (internally tagged on `mode`) and
// `LsmWriteSpecBody` / `GetLsmWriteSpecResponse`.
// `MemWalWriteSpecBody` / `GetMemWalSpecResponse`.
#[derive(Deserialize)]
#[serde(tag = "mode", rename_all = "snake_case")]
enum Sharding {
@@ -3340,7 +3336,7 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
Identity { column: String },
}
#[derive(Deserialize)]
struct LsmWriteSpecBody {
struct MemWalWriteSpecBody {
sharding: Sharding,
/// `null` selects every index the table has; `[]` selects none.
#[serde(default)]
@@ -3349,19 +3345,19 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
writer_config_defaults: std::collections::HashMap<String, String>,
}
#[derive(Deserialize)]
struct GetLsmWriteSpecResponse {
lsm_write_spec: Option<LsmWriteSpecBody>,
struct GetMemWalSpecResponse {
memwal_spec: Option<MemWalWriteSpecBody>,
}
let parsed: GetLsmWriteSpecResponse =
let parsed: GetMemWalSpecResponse =
serde_json::from_str(&body).map_err(|e| Error::Http {
source: format!("Failed to parse get_lsm_write_spec response: {}", e).into(),
source: format!("Failed to parse get_memwal_spec response: {}", e).into(),
request_id,
status_code: None,
})?;
let Some(body) = parsed.lsm_write_spec else {
// The LSM write path is not enabled for this table.
let Some(body) = parsed.memwal_spec else {
// The MemWAL write path is not enabled for this table.
return Ok(None);
};
@@ -3369,9 +3365,9 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
Sharding::Bucket {
column,
num_buckets,
} => LsmWriteSpec::bucket(column, num_buckets),
Sharding::Identity { column } => LsmWriteSpec::identity(column),
Sharding::Unsharded => LsmWriteSpec::unsharded(),
} => MemWalWriteSpec::bucket(column, num_buckets),
Sharding::Identity { column } => MemWalWriteSpec::identity(column),
Sharding::Unsharded => MemWalWriteSpec::unsharded(),
}
.with_maintained_indexes(body.maintained_indexes)
.with_writer_config_defaults(body.writer_config_defaults);
@@ -3974,10 +3970,10 @@ pub struct MergeInsertRequest {
// (the default is true)
#[serde(skip_serializing_if = "is_true")]
use_index: bool,
// Only serialize use_lsm when explicitly set (Some); a server that predates
// Only serialize use_memwal when explicitly set (Some); a server that predates
// it ignores the field and routes as it would by default.
#[serde(skip_serializing_if = "Option::is_none")]
use_lsm: Option<bool>,
use_memwal: Option<bool>,
}
impl MergeInsertRequest {
@@ -4035,7 +4031,7 @@ impl TryFrom<MergeInsertBuilder> for MergeInsertRequest {
when_not_matched_by_source_delete_filt,
// Only serialize use_index when it's false for backwards compatibility
use_index: value.use_index,
use_lsm: value.use_lsm,
use_memwal: value.use_memwal,
})
}
}
@@ -10010,13 +10006,10 @@ mod tests {
}
#[tokio::test]
async fn test_set_lsm_write_spec_unsharded() {
async fn test_enable_memwal_unsharded() {
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(request.method(), "POST");
assert_eq!(
request.url().path(),
"/v1/table/my_table/set_lsm_write_spec/"
);
assert_eq!(request.url().path(), "/v1/table/my_table/enable_memwal/");
let body = request.body().unwrap().as_bytes().unwrap();
let body: serde_json::Value = serde_json::from_slice(body).unwrap();
assert_eq!(body["sharding"], serde_json::json!({ "mode": "unsharded" }));
@@ -10030,20 +10023,17 @@ mod tests {
.body(r#"{"maintained_indexes":["id_idx"]}"#)
.unwrap()
});
let spec = crate::table::LsmWriteSpec::unsharded()
let spec = crate::table::MemWalWriteSpec::unsharded()
.with_maintained_indexes(vec!["id_idx".to_string()])
.with_writer_config_defaults([("max_memtable_rows", "1000")]);
table.set_lsm_write_spec(spec).await.unwrap();
table.enable_memwal(spec).await.unwrap();
}
#[tokio::test]
async fn test_set_lsm_write_spec_bucket() {
async fn test_enable_memwal_bucket() {
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(request.method(), "POST");
assert_eq!(
request.url().path(),
"/v1/table/my_table/set_lsm_write_spec/"
);
assert_eq!(request.url().path(), "/v1/table/my_table/enable_memwal/");
let body = request.body().unwrap().as_bytes().unwrap();
let body: serde_json::Value = serde_json::from_slice(body).unwrap();
assert_eq!(
@@ -10055,14 +10045,14 @@ mod tests {
http::Response::builder().status(200).body("{}").unwrap()
});
table
.set_lsm_write_spec(crate::table::LsmWriteSpec::bucket("id", 16))
.enable_memwal(crate::table::MemWalWriteSpec::bucket("id", 16))
.await
.unwrap();
}
/// `[]` (none) must stay distinguishable on the wire from null (all).
#[tokio::test]
async fn test_set_lsm_write_spec_no_maintained_indexes() {
async fn test_enable_memwal_no_maintained_indexes() {
let table = Table::new_with_handler("my_table", |request| {
let body = request.body().unwrap().as_bytes().unwrap();
let body: serde_json::Value = serde_json::from_slice(body).unwrap();
@@ -10070,21 +10060,18 @@ mod tests {
http::Response::builder().status(200).body("{}").unwrap()
});
table
.set_lsm_write_spec(
crate::table::LsmWriteSpec::bucket("id", 16).with_maintained_indexes(Vec::new()),
.enable_memwal(
crate::table::MemWalWriteSpec::bucket("id", 16).with_maintained_indexes(Vec::new()),
)
.await
.unwrap();
}
#[tokio::test]
async fn test_set_lsm_write_spec_identity() {
async fn test_enable_memwal_identity() {
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(request.method(), "POST");
assert_eq!(
request.url().path(),
"/v1/table/my_table/set_lsm_write_spec/"
);
assert_eq!(request.url().path(), "/v1/table/my_table/enable_memwal/");
let body = request.body().unwrap().as_bytes().unwrap();
let body: serde_json::Value = serde_json::from_slice(body).unwrap();
assert_eq!(
@@ -10094,38 +10081,32 @@ mod tests {
http::Response::builder().status(200).body("{}").unwrap()
});
table
.set_lsm_write_spec(crate::table::LsmWriteSpec::identity("tenant"))
.enable_memwal(crate::table::MemWalWriteSpec::identity("tenant"))
.await
.unwrap();
}
#[tokio::test]
async fn test_unset_lsm_write_spec() {
async fn test_disable_memwal() {
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(request.method(), "POST");
assert_eq!(
request.url().path(),
"/v1/table/my_table/unset_lsm_write_spec/"
);
assert_eq!(request.url().path(), "/v1/table/my_table/disable_memwal/");
http::Response::builder().status(200).body("{}").unwrap()
});
table.unset_lsm_write_spec().await.unwrap();
table.disable_memwal().await.unwrap();
}
#[tokio::test]
async fn test_get_lsm_write_spec() {
async fn test_get_memwal_spec() {
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(request.method(), "POST");
assert_eq!(
request.url().path(),
"/v1/table/my_table/get_lsm_write_spec/"
);
assert_eq!(request.url().path(), "/v1/table/my_table/get_memwal_spec/");
// The server resolves the spec and re-encodes it into the same
// sophon-owned shape the set endpoint accepts (`Sharding` internally
// tagged on `mode`, wrapped in `lsm_write_spec`).
// tagged on `mode`, wrapped in `memwal_spec`).
let response = serde_json::json!({
"lsm_write_spec": {
"memwal_spec": {
"sharding": { "mode": "bucket", "column": "id", "num_buckets": 4 },
"maintained_indexes": ["id_idx"],
"writer_config_defaults": { "durable_write": "false" },
@@ -10138,12 +10119,12 @@ mod tests {
});
let spec = table
.get_lsm_write_spec()
.get_memwal_spec()
.await
.unwrap()
.expect("a spec should be reported");
match spec {
crate::table::LsmWriteSpec::Bucket {
crate::table::MemWalWriteSpec::Bucket {
column,
num_buckets,
maintained_indexes,
@@ -10170,13 +10151,13 @@ mod tests {
#[case::no_index(serde_json::json!([]), Some(vec![]))]
#[case::named(serde_json::json!(["id_idx"]), Some(vec!["id_idx".to_string()]))]
#[tokio::test]
async fn test_get_lsm_write_spec_round_trips_the_selection(
async fn test_get_memwal_spec_round_trips_the_selection(
#[case] reported: serde_json::Value,
#[case] expected: Option<Vec<String>>,
) {
let table = Table::new_with_handler("my_table", move |_| {
let response = serde_json::json!({
"lsm_write_spec": {
"memwal_spec": {
"sharding": { "mode": "unsharded" },
"maintained_indexes": reported,
"writer_config_defaults": {},
@@ -10189,7 +10170,7 @@ mod tests {
});
let spec = table
.get_lsm_write_spec()
.get_memwal_spec()
.await
.unwrap()
.expect("a spec should be reported");
@@ -10201,35 +10182,32 @@ mod tests {
}
#[tokio::test]
async fn test_get_lsm_write_spec_absent() {
async fn test_get_memwal_spec_absent() {
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(
request.url().path(),
"/v1/table/my_table/get_lsm_write_spec/"
);
// Null spec → the LSM write path is not enabled.
let response = serde_json::json!({ "lsm_write_spec": null });
assert_eq!(request.url().path(), "/v1/table/my_table/get_memwal_spec/");
// Null spec → the MemWAL write path is not enabled.
let response = serde_json::json!({ "memwal_spec": null });
http::Response::builder()
.status(200)
.body(response.to_string())
.unwrap()
});
assert!(table.get_lsm_write_spec().await.unwrap().is_none());
assert!(table.get_memwal_spec().await.unwrap().is_none());
}
/// Build a `get_lsm_stats` body for one bucket holding `generations`.
fn stats_body(generations: &[u64], compacting: bool) -> String {
/// Build a `get_memwal_stats` body for one table shard holding `sstables`.
fn stats_body(sstables: &[u64], compacting: bool) -> String {
serde_json::json!({
"lsm_stats": {
"buckets": [{
"memwal_stats": {
"table_shards": [{
"shard_id": "b0",
"status": "Active",
"writer_epoch": 1,
"manifest_version": 1,
"current_generation": generations.iter().max().copied().unwrap_or(0) + 1,
"current_generation": sstables.iter().max().copied().unwrap_or(0) + 1,
"replay_after_wal_entry_position": 0,
"wal_entry_position_last_seen": 0,
"generations": generations.iter()
"sstables": sstables.iter()
.map(|g| serde_json::json!({ "generation": g, "bytes": 1 }))
.collect::<Vec<_>>(),
"compacting": compacting,
@@ -10240,7 +10218,7 @@ mod tests {
.to_string()
}
/// `flush_lsm` / `compact_lsm` answer 202 with no body at all.
/// `flush_memwal` / `compact_memwal` answer 202 with no body at all.
fn accepted() -> http::Response<String> {
http::Response::builder()
.status(202)
@@ -10252,7 +10230,7 @@ mod tests {
http::Response::builder().status(200).body(body).unwrap()
}
/// A flush landing in an empty L0 finishes on the opening stats read
/// A flush landing in an empty SSTable tier finishes on the opening stats read
/// alone. Asserting zero compacts is the point: "it returned Ok" is also
/// true of a loop that ran a pointless pass.
#[tokio::test(start_paused = true)]
@@ -10261,18 +10239,18 @@ mod tests {
let seen = compacts.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("compact_lsm") {
if path.contains("compact_memwal") {
seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
panic!("an already-converged table must issue no compact calls");
}
if path.contains("flush_lsm") {
if path.contains("flush_memwal") {
return accepted();
}
assert_eq!(path, "/v1/table/my_table/get_lsm_stats/");
assert_eq!(path, "/v1/table/my_table/get_memwal_stats/");
ok_json(stats_body(&[], false))
});
table.checkpoint_lsm().await.unwrap();
table.checkpoint_memwal().await.unwrap();
assert_eq!(compacts.load(std::sync::atomic::Ordering::SeqCst), 0);
}
@@ -10284,10 +10262,10 @@ mod tests {
let seen = compacts.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("flush_lsm") {
if path.contains("flush_memwal") {
return accepted();
}
if path.contains("compact_lsm") {
if path.contains("compact_memwal") {
seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
return accepted();
}
@@ -10297,7 +10275,7 @@ mod tests {
ok_json(stats_body(&left, false))
});
table.checkpoint_lsm().await.unwrap();
table.checkpoint_memwal().await.unwrap();
assert_eq!(
compacts.load(std::sync::atomic::Ordering::SeqCst),
3,
@@ -10306,7 +10284,7 @@ mod tests {
}
/// Generations created *during* the checkpoint are not waited on, which
/// is what lets the loop terminate on a table taking writes where "L0 is
/// is what lets the loop terminate on a table taking writes where "the SSTable tier is
/// empty" never becomes true.
#[tokio::test(start_paused = true)]
async fn test_checkpoint_ignores_generations_created_while_it_runs() {
@@ -10314,10 +10292,10 @@ mod tests {
let seen = compacts.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("flush_lsm") {
if path.contains("flush_memwal") {
return accepted();
}
if path.contains("compact_lsm") {
if path.contains("compact_memwal") {
seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
return accepted();
}
@@ -10331,11 +10309,11 @@ mod tests {
ok_json(body)
});
table.checkpoint_lsm().await.unwrap();
table.checkpoint_memwal().await.unwrap();
assert_eq!(
compacts.load(std::sync::atomic::Ordering::SeqCst),
1,
"the loop must not chase generations written after it started"
"the loop must not chase SSTables written after it started"
);
}
@@ -10348,11 +10326,11 @@ mod tests {
let seen = compacts.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("flush_lsm") {
if path.contains("flush_memwal") {
return accepted();
}
if path.contains("compact_lsm") {
// First two triggers: every bucket already latched.
if path.contains("compact_memwal") {
// First two triggers: every table shard already latched.
if seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst) < 2 {
return http::Response::builder()
.status(429)
@@ -10373,7 +10351,7 @@ mod tests {
});
table
.checkpoint_lsm()
.checkpoint_memwal()
.await
.expect("contention must not abort the checkpoint");
assert_eq!(
@@ -10393,7 +10371,7 @@ mod tests {
let seen = polls.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("flush_lsm") || path.contains("compact_lsm") {
if path.contains("flush_memwal") || path.contains("compact_memwal") {
return accepted();
}
// The opening read lands; the next two polls are latched out.
@@ -10408,7 +10386,7 @@ mod tests {
});
table
.checkpoint_lsm()
.checkpoint_memwal()
.await
.expect("a contended poll must be retried, not surfaced");
assert_eq!(
@@ -10428,7 +10406,7 @@ mod tests {
let seen = flushes.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("flush_lsm") {
if path.contains("flush_memwal") {
if seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst) < 5 {
return http::Response::builder()
.status(429)
@@ -10437,20 +10415,20 @@ mod tests {
}
return accepted();
}
if path.contains("compact_lsm") {
if path.contains("compact_memwal") {
return accepted();
}
ok_json(stats_body(&[], false))
});
table
.checkpoint_lsm()
.checkpoint_memwal()
.await
.expect("contention must not be reported as a lost claim");
assert_eq!(
flushes.load(std::sync::atomic::Ordering::SeqCst),
6,
"five retries against one seal, then it lands"
"five retries against one freeze, then it lands"
);
}
@@ -10469,7 +10447,7 @@ mod tests {
.unwrap()
});
let err = table.checkpoint_lsm().await.unwrap_err();
let err = table.checkpoint_memwal().await.unwrap_err();
assert!(
matches!(&err, Error::Http { status_code: Some(s), .. } if s.as_u16() == 429),
"the fault that spent the budget must be the one reported: {err:?}"
@@ -10499,7 +10477,7 @@ mod tests {
.unwrap()
});
let err = table.checkpoint_lsm().await.unwrap_err();
let err = table.checkpoint_memwal().await.unwrap_err();
let message = err.to_string();
assert!(
matches!(&err, Error::Http { status_code: Some(s), .. } if s.as_u16() == 503),
@@ -10519,7 +10497,7 @@ mod tests {
/// A long stall with nothing compacting must keep waiting, not fail. The
/// client cannot judge this: a checkpoint queued behind unrelated tables
/// on the pod-wide compactor pool reports exactly these numbers — flat
/// generations, an idle latch — as one whose merges are failing. The
/// SSTable counts, an idle latch — as one whose merges are failing. The
/// deadline is the caller's.
#[tokio::test(start_paused = true)]
async fn test_checkpoint_waits_out_a_long_stall_rather_than_failing() {
@@ -10527,7 +10505,7 @@ mod tests {
let seen = polls.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("flush_lsm") || path.contains("compact_lsm") {
if path.contains("flush_memwal") || path.contains("compact_memwal") {
return accepted();
}
// Flat for far longer than any bound this loop ever had, with
@@ -10537,7 +10515,7 @@ mod tests {
});
table
.checkpoint_lsm()
.checkpoint_memwal()
.await
.expect("a stall is the server being slow, not the client's call to make");
assert!(
@@ -10546,7 +10524,7 @@ mod tests {
);
}
/// A pass already owns the latch on every outstanding bucket, so the loop
/// A pass already owns the latch on every outstanding table shard, so the loop
/// waits rather than piling on triggers it would only refuse. This is the
/// sole thing `compacting` is read for.
#[tokio::test(start_paused = true)]
@@ -10557,10 +10535,10 @@ mod tests {
let seen_compacts = compacts.clone();
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
if path.contains("flush_lsm") {
if path.contains("flush_memwal") {
return accepted();
}
if path.contains("compact_lsm") {
if path.contains("compact_memwal") {
seen_compacts.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
return accepted();
}
@@ -10574,32 +10552,32 @@ mod tests {
});
table
.checkpoint_lsm()
.checkpoint_memwal()
.await
.expect("a running pass is progress, not a stall");
assert_eq!(
compacts.load(std::sync::atomic::Ordering::SeqCst),
0,
"never trigger against a bucket already compacting"
"never trigger against a table shard already compacting"
);
}
/// WAL off ⇒ `None`; WAL on ⇒ a fully populated `Some` with no field
/// defaulting to a zero it did not measure. `include_generation_rows`
/// defaulting to a zero it did not measure. `include_sstable_rows`
/// rides in the body and is off unless asked for.
#[tokio::test]
async fn test_get_lsm_stats_round_trip() {
async fn test_get_memwal_stats_round_trip() {
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(request.url().path(), "/v1/table/my_table/get_lsm_stats/");
assert_eq!(request.url().path(), "/v1/table/my_table/get_memwal_stats/");
let body = request.body().unwrap().as_bytes().unwrap();
let body: serde_json::Value = serde_json::from_slice(body).unwrap();
assert_eq!(
body["include_generation_rows"], true,
body["include_sstable_rows"], true,
"the flag must reach the server, not be silently dropped"
);
let response = serde_json::json!({
"lsm_stats": {
"buckets": [{
"memwal_stats": {
"table_shards": [{
"shard_id": "b0",
"status": "Active",
"writer_epoch": 3,
@@ -10607,7 +10585,7 @@ mod tests {
"current_generation": 9,
"replay_after_wal_entry_position": 100,
"wal_entry_position_last_seen": 140,
"generations": [{ "generation": 8, "bytes": 4096, "rows": 30 }],
"sstables": [{ "generation": 8, "bytes": 4096, "rows": 30 }],
"compacting": false,
"memtables": [
{ "generation": 9, "rows": 12, "bytes": 900, "batches": 2,
@@ -10623,19 +10601,19 @@ mod tests {
});
let stats = table
.get_lsm_stats(true)
.get_memwal_stats(true)
.await
.unwrap()
.expect("a WAL-backed table reports Some");
let bucket = &stats.buckets[0];
assert_eq!(bucket.replay_after_wal_entry_position, 100);
assert_eq!(bucket.wal_entry_position_last_seen, 140);
assert!(!bucket.compacting);
assert_eq!(bucket.generations[0].generation, 8);
assert_eq!(bucket.generations[0].rows, Some(30));
let table_shard = &stats.table_shards[0];
assert_eq!(table_shard.replay_after_wal_entry_position, 100);
assert_eq!(table_shard.wal_entry_position_last_seen, 140);
assert!(!table_shard.compacting);
assert_eq!(table_shard.sstables[0].generation, 8);
assert_eq!(table_shard.sstables[0].rows, Some(30));
// The line that answers "why is my fresh-tier vector search
// brute-force" — an absent index name is the whole explanation.
let memtables = bucket.memtables.as_ref().unwrap();
let memtables = table_shard.memtables.as_ref().unwrap();
assert_eq!(memtables[0].indexes, vec!["vec_idx".to_string()]);
}
@@ -10655,7 +10633,7 @@ mod tests {
.unwrap()
});
let err = table.checkpoint_lsm().await.unwrap_err();
let err = table.checkpoint_memwal().await.unwrap_err();
assert!(
matches!(err, Error::TableNotFound { .. }),
"a missing table must say so: {err:?}"
@@ -10676,12 +10654,12 @@ mod tests {
let table = Table::new_with_handler("my_table", move |request| {
let path = request.url().path().to_string();
let n = seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
if path.contains("flush_lsm") {
if path.contains("flush_memwal") {
// First flush lands; the claim is then lost, and the
// re-issued flush succeeds.
return accepted();
}
if path.contains("compact_lsm") {
if path.contains("compact_memwal") {
if n < 4 {
return http::Response::builder()
.status(421)
@@ -10694,20 +10672,20 @@ mod tests {
});
table
.checkpoint_lsm()
.checkpoint_memwal()
.await
.expect("a lost claim must be recovered by re-flushing, not surfaced");
}
#[tokio::test]
async fn test_get_lsm_stats_absent_when_wal_off() {
async fn test_get_memwal_stats_absent_when_wal_off() {
let table = Table::new_with_handler("my_table", |_request| {
http::Response::builder()
.status(200)
.body(serde_json::json!({ "lsm_stats": null }).to_string())
.body(serde_json::json!({ "memwal_stats": null }).to_string())
.unwrap()
});
assert!(table.get_lsm_stats(false).await.unwrap().is_none());
assert!(table.get_memwal_stats(false).await.unwrap().is_none());
}
#[tokio::test]
@@ -14726,7 +14704,7 @@ mod tests {
/// Verbatim from sophon's `LSM_WITH_ROW_ID_UNSUPPORTED`; the fallback keys
/// off this sentence, so a test that paraphrased it would prove nothing.
const WAL_ROW_ID_REFUSAL_BODY: &str = r#"{"code":13,"error":"Bad request: the MemWAL LSM scanner does not support with_row_id (the LSM scanner exposes _rowaddr, not a stable _rowid); set use_lsm(false) to read the base table only (results will exclude un-compacted MemWAL data)"}"#;
const WAL_ROW_ID_REFUSAL_BODY: &str = r#"{"code":13,"error":"Bad request: the MemWAL LSM scanner does not support with_row_id (the LSM scanner exposes _rowaddr, not a stable _rowid); set use_memwal(false) to read the base table only (results will exclude un-compacted MemWAL data)"}"#;
use lance::dataset::ROW_ID;
@@ -14985,7 +14963,7 @@ mod tests {
// Translated, not passed through: the caller gets the reason and the
// way out, which a bare 400 does not carry.
assert!(matches!(err, Error::NotSupported { .. }), "{err:?}");
assert!(err.to_string().contains("use_lsm(false)"), "{err}");
assert!(err.to_string().contains("use_memwal(false)"), "{err}");
assert!(
bodies
.lock()
rust/lancedb/src/remote/table/blobs.rs
+2 -2
View File
@@ -685,7 +685,7 @@ impl<S: HttpSend> RemoteTable<S> {
select: Select::columns(&[column]),
with_row_id: true,
// Row ids name base-table rows, and the MemWAL scanner has no stable `_rowid`.
use_lsm: Some(false),
use_memwal: Some(false),
..Default::default()
});
let body = self
@@ -1121,7 +1121,7 @@ mod tests {
.unwrap();
assert_eq!(body["columns"], serde_json::json!(["image"]));
assert_eq!(body["with_row_id"], serde_json::json!(true));
assert_eq!(body["use_lsm"], serde_json::json!(false));
assert_eq!(body["use_memwal"], serde_json::json!(false));
let requested = filter_row_ids(body["filter"].as_str().unwrap());
captured.lock().unwrap().push(TakeMockRequest::Query {
row_ids: requested.clone(),
rust/lancedb/src/remote/table/insert.rs
+1 -1
View File
@@ -1497,7 +1497,7 @@ mod tests {
when_not_matched_by_source_delete: false,
when_not_matched_by_source_delete_filt: None,
use_index: true,
use_lsm: None,
use_memwal: None,
};
let input: Arc<dyn ExecutionPlan> = Arc::new(ErroringExec::new());
rust/lancedb/src/table.rs
+149 -145
View File
@@ -80,7 +80,7 @@ pub mod datafusion;
pub(crate) mod dataset;
pub mod delete;
pub mod freshness;
pub mod lsm_stats;
pub mod memwal_stats;
pub mod merge;
pub mod optimize;
pub(crate) mod primary_key;
@@ -107,7 +107,7 @@ use futures::future::join_all;
pub use lance::dataset::refs::{BranchContents, Ref, TagContents, Tags as LanceTags};
pub use lance::dataset::scanner::DatasetRecordBatchStream;
pub use lance_index::optimize::OptimizeOptions;
pub use lsm_stats::{BucketStats, GenerationStats, LsmStats, MemtableStats};
pub use memwal_stats::{MemWalStats, MemtableStats, SsTableStats, TableShardStats};
pub use optimize::{CompactionOptions, OptimizeAction, OptimizeStats};
pub use refresh::RefreshColumnResult;
pub use schema_evolution::{
@@ -327,20 +327,20 @@ pub use self::merge::MergeResult;
/// Specification selecting Lance's MemWAL LSM-style write path for
/// `merge_insert`.
///
/// Construct via [`LsmWriteSpec::bucket`], [`LsmWriteSpec::identity`], or
/// [`LsmWriteSpec::unsharded`], then optionally chain
/// [`LsmWriteSpec::with_maintained_indexes`] (indexes the MemWAL keeps up to
/// date) and [`LsmWriteSpec::with_writer_config_defaults`] (default
/// Construct via [`MemWalWriteSpec::bucket`], [`MemWalWriteSpec::identity`], or
/// [`MemWalWriteSpec::unsharded`], then optionally chain
/// [`MemWalWriteSpec::with_maintained_indexes`] (indexes the MemWAL keeps up to
/// date) and [`MemWalWriteSpec::with_writer_config_defaults`] (default
/// `ShardWriter` configuration recorded in the MemWAL index).
///
/// A fresh spec maintains every index on the table, including ones created
/// after it is installed.
///
/// Install a spec with [`Table::set_lsm_write_spec`] and remove it with
/// [`Table::unset_lsm_write_spec`]. The actual `merge_insert` dispatch
/// Install a spec with [`Table::enable_memwal`] and remove it with
/// [`Table::disable_memwal`]. The actual `merge_insert` dispatch
/// onto the MemWAL writer is a follow-up.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum LsmWriteSpec {
pub enum MemWalWriteSpec {
/// Hash-bucket sharding by a scalar column.
///
/// `column` must be a non-nested column with a supported scalar type.
@@ -384,7 +384,7 @@ pub enum LsmWriteSpec {
},
}
impl LsmWriteSpec {
impl MemWalWriteSpec {
/// Construct a hash-bucket sharding spec maintaining every index on the table.
pub fn bucket(column: impl Into<String>, num_buckets: u32) -> Self {
Self::Bucket {
@@ -401,7 +401,7 @@ impl LsmWriteSpec {
/// `column` must be a deterministic function of the unenforced primary
/// key: every row with a given primary key must always produce the same
/// `column` value. MemWAL dedups upserts by primary key but tracks
/// generations per shard, so if the same key is written with two
/// SSTables per shard, so if the same key is written with two
/// different `column` values its versions land in different shards and a
/// stale value can win. Typically `column` is the primary key itself, or
/// a stable attribute of it (e.g. a tenant id).
@@ -431,13 +431,13 @@ impl LsmWriteSpec {
/// any added with [`NativeTable::set_mem_index_registry`].
///
/// ```
/// # use lancedb::table::LsmWriteSpec;
/// # use lancedb::table::MemWalWriteSpec;
/// // Every index the table has, now and later:
/// LsmWriteSpec::unsharded().with_maintained_indexes(None);
/// MemWalWriteSpec::unsharded().with_maintained_indexes(None);
/// // Exactly these:
/// LsmWriteSpec::unsharded().with_maintained_indexes(vec!["id_idx".to_string()]);
/// MemWalWriteSpec::unsharded().with_maintained_indexes(vec!["id_idx".to_string()]);
/// // None at all:
/// LsmWriteSpec::unsharded().with_maintained_indexes(Vec::new());
/// MemWalWriteSpec::unsharded().with_maintained_indexes(Vec::new());
/// ```
pub fn with_maintained_indexes(mut self, indexes: impl Into<Option<Vec<String>>>) -> Self {
let indexes = indexes.into();
@@ -674,37 +674,37 @@ pub trait BaseTable: std::fmt::Display + std::fmt::Debug + Send + Sync {
message: "set_unenforced_primary_key is not supported on this table type".into(),
})
}
/// Install an [`LsmWriteSpec`] on this table.
/// Install an [`MemWalWriteSpec`] on this table.
///
/// The spec selects Lance's MemWAL LSM-style write path for future
/// `merge_insert` calls.
///
/// The default implementation returns `NotSupported`. Implementations
/// that support the MemWAL LSM write path must override this.
async fn set_lsm_write_spec(&self, _spec: LsmWriteSpec) -> Result<()> {
/// that support the MemWAL write path must override this.
async fn enable_memwal(&self, _spec: MemWalWriteSpec) -> Result<()> {
Err(Error::NotSupported {
message: "set_lsm_write_spec is not supported on this table type".into(),
message: "enable_memwal is not supported on this table type".into(),
})
}
/// Remove the [`LsmWriteSpec`] from this table.
/// Remove the [`MemWalWriteSpec`] from this table.
///
/// This is a no-op if no spec is currently set.
///
/// The default implementation returns `NotSupported`. Implementations
/// that support the MemWAL LSM write path must override this.
async fn unset_lsm_write_spec(&self) -> Result<()> {
/// that support the MemWAL write path must override this.
async fn disable_memwal(&self) -> Result<()> {
Err(Error::NotSupported {
message: "unset_lsm_write_spec is not supported on this table type".into(),
message: "disable_memwal is not supported on this table type".into(),
})
}
/// Read the [`LsmWriteSpec`] currently installed on this table, returning
/// `None` when the MemWAL LSM write path is not enabled.
/// Read the [`MemWalWriteSpec`] currently installed on this table, returning
/// `None` when the MemWAL write path is not enabled.
///
/// The default implementation returns `NotSupported`. Implementations that
/// support the MemWAL LSM write path must override this.
async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
/// support the MemWAL write path must override this.
async fn get_memwal_spec(&self) -> Result<Option<MemWalWriteSpec>> {
Err(Error::NotSupported {
message: "get_lsm_write_spec is not supported on this table type".into(),
message: "get_memwal_spec is not supported on this table type".into(),
})
}
/// Whether a hybrid query on this table has already been told it cannot
@@ -732,36 +732,36 @@ pub trait BaseTable: std::fmt::Display + std::fmt::Debug + Send + Sync {
/// the only cost of being late to notice is a base-only read that is still
/// correct.
fn note_hybrid_pk_fusion(&self) {}
/// Seal every bucket's active memtable into L0.
/// Freeze every table shard's active memtable into an SSTable.
///
/// The default implementation returns `NotSupported`.
async fn flush_lsm(&self) -> Result<()> {
async fn flush_memwal(&self) -> Result<()> {
Err(Error::NotSupported {
message: "flush_lsm is not supported on this table type".into(),
message: "flush_memwal is not supported on this table type".into(),
})
}
/// Trigger a background L0 → base compaction pass per bucket.
/// Trigger a background SSTable compaction pass per table shard.
///
/// The default implementation returns `NotSupported`.
async fn compact_lsm(&self) -> Result<()> {
async fn compact_memwal(&self) -> Result<()> {
Err(Error::NotSupported {
message: "compact_lsm is not supported on this table type".into(),
message: "compact_memwal is not supported on this table type".into(),
})
}
/// Read live LSM state, or `None` when the LSM write path is not
/// Read live MemWAL state, or `None` when the MemWAL write path is not
/// enabled for this table.
///
/// The default implementation returns `NotSupported`.
async fn get_lsm_stats(&self, _include_generation_rows: bool) -> Result<Option<LsmStats>> {
async fn get_memwal_stats(&self, _include_sstable_rows: bool) -> Result<Option<MemWalStats>> {
Err(Error::NotSupported {
message: "get_lsm_stats is not supported on this table type".into(),
message: "get_memwal_stats is not supported on this table type".into(),
})
}
/// Drain and close any cached MemWAL shard writers for this table.
///
/// The default implementation is a no-op; table types that maintain
/// MemWAL shard writers override it.
async fn close_lsm_writers(&self) -> Result<()> {
async fn close_memwal_writers(&self) -> Result<()> {
Ok(())
}
/// Names of the blob v2 columns in this table, in declaration order.
@@ -2006,50 +2006,50 @@ impl Table {
self.inner.set_unenforced_primary_key(&borrowed).await
}
/// Install an [`LsmWriteSpec`] on this table, selecting Lance's MemWAL
/// Install an [`MemWalWriteSpec`] on this table, selecting Lance's MemWAL
/// LSM-style write path for future `merge_insert` calls.
///
/// [`LsmWriteSpec`] chooses one of three sharding strategies:
/// [`MemWalWriteSpec`] chooses one of three sharding strategies:
///
/// - [`LsmWriteSpec::bucket`] — hash-bucket writes by a scalar column.
/// - [`LsmWriteSpec::identity`] — shard by the raw value of a scalar column.
/// - [`LsmWriteSpec::unsharded`] — route every write to a single shard.
/// - [`MemWalWriteSpec::bucket`] — hash-bucket writes by a scalar column.
/// - [`MemWalWriteSpec::identity`] — shard by the raw value of a scalar column.
/// - [`MemWalWriteSpec::unsharded`] — route every write to a single shard.
///
/// A table carries one spec: this fails while one is installed, since the
/// generations already written were homed under it.
/// [`Table::unset_lsm_write_spec`] removes one.
/// SSTables already written were homed under it.
/// [`Table::disable_memwal`] removes one.
///
/// # Example
///
/// ```
/// # use lancedb::table::{LsmWriteSpec, Table};
/// # use lancedb::table::{MemWalWriteSpec, Table};
/// # async fn example(table: &Table) -> Result<(), Box<dyn std::error::Error>> {
/// table
/// .set_lsm_write_spec(
/// LsmWriteSpec::bucket("id", 16).with_maintained_indexes(vec!["id_idx".to_string()]),
/// .enable_memwal(
/// MemWalWriteSpec::bucket("id", 16).with_maintained_indexes(vec!["id_idx".to_string()]),
/// )
/// .await?;
/// # Ok(())
/// # }
/// ```
pub async fn set_lsm_write_spec(&self, spec: LsmWriteSpec) -> Result<()> {
self.inner.set_lsm_write_spec(spec).await
pub async fn enable_memwal(&self, spec: MemWalWriteSpec) -> Result<()> {
self.inner.enable_memwal(spec).await
}
/// Remove the [`LsmWriteSpec`] from this table, reverting to the standard
/// Remove the [`MemWalWriteSpec`] from this table, reverting to the standard
/// `merge_insert` write path.
///
/// Errors if no spec is currently set.
pub async fn unset_lsm_write_spec(&self) -> Result<()> {
self.inner.unset_lsm_write_spec().await
pub async fn disable_memwal(&self) -> Result<()> {
self.inner.disable_memwal().await
}
/// Read the [`LsmWriteSpec`] currently installed on this table.
/// Read the [`MemWalWriteSpec`] currently installed on this table.
///
/// `Ok(None)` means the LSM write path is not enabled at all — no spec has
/// been set, or one was removed with [`Table::unset_lsm_write_spec`]. That
/// `Ok(None)` means the MemWAL write path is not enabled at all — no spec has
/// been set, or one was removed with [`Table::disable_memwal`]. That
/// is a different answer from a spec whose
/// [`LsmWriteSpec::maintained_indexes`] is `None`, which is an installed
/// [`MemWalWriteSpec::maintained_indexes`] is `None`, which is an installed
/// spec selecting indexes automatically.
///
/// The spec read back is the one that was installed, selection included:
@@ -2062,21 +2062,21 @@ impl Table {
/// ```
/// # use lancedb::table::Table;
/// # async fn example(table: &Table) -> Result<(), Box<dyn std::error::Error>> {
/// if let Some(spec) = table.get_lsm_write_spec().await? {
/// println!("LSM write path enabled: {:?}", spec);
/// if let Some(spec) = table.get_memwal_spec().await? {
/// println!("MemWAL write path enabled: {:?}", spec);
/// }
/// # Ok(())
/// # }
/// ```
pub async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
self.inner.get_lsm_write_spec().await
pub async fn get_memwal_spec(&self) -> Result<Option<MemWalWriteSpec>> {
self.inner.get_memwal_spec().await
}
/// Converge this table's LSM write path into its base table.
/// Converge this table's MemWAL write path into its base table.
///
/// One `flush` to seal every memtable into L0, then compaction triggers
/// One `flush` to freeze every memtable into an SSTable, then compaction triggers
/// until every generation that existed at that moment has reached base.
/// The loop runs client-side, reading progress from `get_lsm_stats`, so
/// The loop runs client-side, reading progress from `get_memwal_stats`, so
/// there is no held socket and nothing to reconcile if you drop this
/// future partway through.
///
@@ -2086,7 +2086,7 @@ impl Table {
/// already-converged table costs two round trips and triggers nothing.
///
/// **No deadline, and the caller owns that.** It returns when the target
/// generations are gone, propagates a terminal server fault, and
/// SSTables are gone, propagates a terminal server fault, and
/// otherwise waits however long the server takes. A slow table and a
/// stuck one are the same picture from here: the compactor pool is shared
/// across every table on the node, so a checkpoint queued behind
@@ -2099,66 +2099,69 @@ impl Table {
/// ```no_run
/// # use lancedb::Table;
/// # async fn example(table: &Table) -> Result<(), Box<dyn std::error::Error>> {
/// let before = table.get_lsm_stats(false).await?;
/// table.checkpoint_lsm().await?;
/// let after = table.get_lsm_stats(false).await?;
/// let before = table.get_memwal_stats(false).await?;
/// table.checkpoint_memwal().await?;
/// let after = table.get_memwal_stats(false).await?;
/// # Ok(())
/// # }
/// ```
pub async fn checkpoint_lsm(&self) -> Result<()> {
checkpoint::checkpoint_lsm(self).await
pub async fn checkpoint_memwal(&self) -> Result<()> {
checkpoint::checkpoint_memwal(self).await
}
/// Seal every bucket's active memtable into L0 without touching the
/// Freeze every table shard's active memtable into an SSTable without touching the
/// base table.
///
/// Independently useful: flushing makes memtable rows readable from L0 at
/// Independently useful: flushing makes memtable rows readable from an SSTable at
/// a lower per-query cost. On a node that has not claimed this table it
/// claims it and replays the WAL log first — reporting "nothing to flush"
/// without replaying would lie about durable data.
pub async fn flush_lsm(&self) -> Result<()> {
self.inner.flush_lsm().await
pub async fn flush_memwal(&self) -> Result<()> {
self.inner.flush_memwal().await
}
/// Run one bounded L0 → base compaction pass per bucket, reporting what
/// Run one bounded SSTable compaction pass per table shard, reporting what
/// it merged and what is left.
///
/// One pass, not convergence: that bounds each request's cost and gives a
/// caller driving its own cadence a progress signal per round trip.
pub async fn compact_lsm(&self) -> Result<()> {
self.inner.compact_lsm().await
pub async fn compact_memwal(&self) -> Result<()> {
self.inner.compact_memwal().await
}
/// Read live per-bucket LSM state.
/// Read live per-table-shard MemWAL state.
///
/// Answers "how far behind is my fresh tier", "which bucket is hot", and
/// Answers "how far behind is my fresh tier", "which table shard is hot", and
/// "why is my fresh-tier vector search brute-force". Mutates no table
/// state, though on a node that has not claimed this table it claims it,
/// exactly as a read would.
///
/// `include_generation_rows` reports a row count per L0 generation. Off by
/// `include_sstable_rows` reports a row count per SSTable. Off by
/// default: each count opens an uncached Lance dataset, and
/// `checkpoint_lsm` polls this needing only generation numbers.
/// `checkpoint_memwal` polls this needing only generation numbers.
///
/// `Ok(None)` only when the LSM write path is not enabled, matching
/// [`Table::get_lsm_write_spec`]. Stats is fresh-tier only, so with the
/// `Ok(None)` only when the MemWAL write path is not enabled, matching
/// [`Table::get_memwal_spec`]. Stats is fresh-tier only, so with the
/// WAL off there is no manifest to report and a struct of zeros would
/// read as measurements.
///
/// Do not build a checkpoint's termination on this: the completion
/// predicate lives in the `flush` and `compact` responses.
pub async fn get_lsm_stats(&self, include_generation_rows: bool) -> Result<Option<LsmStats>> {
self.inner.get_lsm_stats(include_generation_rows).await
pub async fn get_memwal_stats(
&self,
include_sstable_rows: bool,
) -> Result<Option<MemWalStats>> {
self.inner.get_memwal_stats(include_sstable_rows).await
}
/// Drain and close any cached MemWAL shard writers held for this table.
///
/// When an [`LsmWriteSpec`] is installed, `merge_insert` opens MemWAL shard
/// When an [`MemWalWriteSpec`] is installed, `merge_insert` opens MemWAL shard
/// writers and caches them for reuse across calls. This closes them,
/// flushing pending data; writers reopen lazily on the next `merge_insert`.
/// It is a no-op when no writers are cached.
pub async fn close_lsm_writers(&self) -> Result<()> {
self.inner.close_lsm_writers().await
pub async fn close_memwal_writers(&self) -> Result<()> {
self.inner.close_memwal_writers().await
}
/// Retrieve the version of the table
@@ -2776,8 +2779,8 @@ impl NativeTable {
}
/// Maintain the memtable index kinds in `registry` when this table writes
/// through its [`LsmWriteSpec`], and accept them when
/// [`LsmWriteSpec::with_maintained_indexes`] names one.
/// through its [`MemWalWriteSpec`], and accept them when
/// [`MemWalWriteSpec::with_maintained_indexes`] names one.
///
/// The registry belongs to this handle, its clones and its branch handles,
/// and is not stored with the table: a handle opened separately starts from
@@ -2788,7 +2791,7 @@ impl NativeTable {
/// ```
/// # use std::sync::Arc;
/// # use lancedb::Table;
/// # use lancedb::table::{LsmWriteSpec, MemIndexRegistry, mem_index::MemIndexPlugin};
/// # use lancedb::table::{MemWalWriteSpec, MemIndexRegistry, mem_index::MemIndexPlugin};
/// # async fn example(
/// # table: &Table,
/// # plugin: Arc<dyn MemIndexPlugin>,
@@ -2797,8 +2800,8 @@ impl NativeTable {
/// if let Some(native) = table.as_native() {
/// native.set_mem_index_registry(registry);
/// }
/// let spec = LsmWriteSpec::unsharded().with_maintained_indexes(vec!["region_bitmap".to_string()]);
/// table.set_lsm_write_spec(spec).await?;
/// let spec = MemWalWriteSpec::unsharded().with_maintained_indexes(vec!["region_bitmap".to_string()]);
/// table.enable_memwal(spec).await?;
/// # Ok(())
/// # }
/// ```
@@ -3691,20 +3694,20 @@ impl BaseTable for NativeTable {
primary_key::set_unenforced_primary_key(self, columns).await
}
async fn set_lsm_write_spec(&self, spec: LsmWriteSpec) -> Result<()> {
merge::lsm::set_lsm_write_spec(self, spec).await
async fn enable_memwal(&self, spec: MemWalWriteSpec) -> Result<()> {
merge::lsm::enable_memwal(self, spec).await
}
async fn unset_lsm_write_spec(&self) -> Result<()> {
merge::lsm::unset_lsm_write_spec(self).await
async fn disable_memwal(&self) -> Result<()> {
merge::lsm::disable_memwal(self).await
}
async fn get_lsm_write_spec(&self) -> Result<Option<LsmWriteSpec>> {
merge::lsm::get_lsm_write_spec(self).await
async fn get_memwal_spec(&self) -> Result<Option<MemWalWriteSpec>> {
merge::lsm::get_memwal_spec(self).await
}
async fn close_lsm_writers(&self) -> Result<()> {
merge::lsm::close_lsm_writers(self).await
async fn close_memwal_writers(&self) -> Result<()> {
merge::lsm::close_memwal_writers(self).await
}
async fn blob_columns(&self) -> Result<Vec<String>> {
@@ -5510,7 +5513,7 @@ mod tests {
}
#[tokio::test]
async fn test_set_lsm_write_spec() {
async fn test_enable_memwal() {
use arrow_array::StringArray;
use lance::dataset::mem_wal::DatasetMemWalExt;
@@ -5542,7 +5545,7 @@ mod tests {
// Reject num_buckets out of range.
for bad in [0u32, 1025] {
let err = table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", bad))
.enable_memwal(MemWalWriteSpec::bucket("id", bad))
.await
.expect_err("should reject");
assert!(matches!(err, Error::InvalidInput { .. }), "got {:?}", err);
@@ -5550,7 +5553,7 @@ mod tests {
// Happy path: install spec; verify MemWAL details record it.
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 4))
.enable_memwal(MemWalWriteSpec::bucket("id", 4))
.await
.unwrap();
@@ -5576,14 +5579,14 @@ mod tests {
// Mutation rejected.
let err = table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 8))
.enable_memwal(MemWalWriteSpec::bucket("id", 8))
.await
.expect_err("mutation should be rejected");
assert!(matches!(err, Error::InvalidInput { .. }), "got {:?}", err);
}
#[tokio::test]
async fn test_set_lsm_write_spec_unsharded() {
async fn test_enable_memwal_unsharded() {
use lance::dataset::mem_wal::DatasetMemWalExt;
let tmp_dir = tempdir().unwrap();
@@ -5605,7 +5608,7 @@ mod tests {
let table = conn.create_table("t", reader).execute().await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -5623,7 +5626,7 @@ mod tests {
}
#[tokio::test]
async fn test_set_lsm_write_spec_identity() {
async fn test_enable_memwal_identity() {
use lance::dataset::mem_wal::DatasetMemWalExt;
let tmp_dir = tempdir().unwrap();
@@ -5651,8 +5654,8 @@ mod tests {
let table = conn.create_table("t", reader).execute().await.unwrap();
table
.set_lsm_write_spec(
LsmWriteSpec::identity("region")
.enable_memwal(
MemWalWriteSpec::identity("region")
.with_writer_config_defaults([("durable_write", "false")]),
)
.await
@@ -5680,7 +5683,7 @@ mod tests {
}
#[tokio::test]
async fn test_unset_lsm_write_spec() {
async fn test_disable_memwal() {
use lance::dataset::mem_wal::DatasetMemWalExt;
let tmp_dir = tempdir().unwrap();
@@ -5702,11 +5705,11 @@ mod tests {
let table = conn.create_table("t", reader).execute().await.unwrap();
// unset errors when no spec is set.
table.unset_lsm_write_spec().await.unwrap_err();
table.disable_memwal().await.unwrap_err();
// Install a spec, then unset it.
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 4))
.enable_memwal(MemWalWriteSpec::bucket("id", 4))
.await
.unwrap();
{
@@ -5714,16 +5717,16 @@ mod tests {
assert!(dataset.mem_wal_index_details().await.unwrap().is_some());
}
table.unset_lsm_write_spec().await.unwrap();
table.disable_memwal().await.unwrap();
{
let dataset = table.as_native().unwrap().dataset.get().await.unwrap();
assert!(dataset.mem_wal_index_details().await.unwrap().is_none());
}
// A second unset errors; a fresh spec can still be installed afterwards.
table.unset_lsm_write_spec().await.unwrap_err();
table.disable_memwal().await.unwrap_err();
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 8))
.enable_memwal(MemWalWriteSpec::bucket("id", 8))
.await
.unwrap();
{
@@ -5733,7 +5736,7 @@ mod tests {
}
#[tokio::test]
async fn test_get_lsm_write_spec() {
async fn test_get_memwal_spec() {
let tmp_dir = tempdir().unwrap();
let uri = tmp_dir.path().to_str().unwrap();
@@ -5759,7 +5762,7 @@ mod tests {
let table = conn.create_table("t", reader).execute().await.unwrap();
// No spec installed yet.
assert_eq!(table.get_lsm_write_spec().await.unwrap(), None);
assert_eq!(table.get_memwal_spec().await.unwrap(), None);
// A real scalar index is needed to name it as a maintained index.
table
@@ -5771,35 +5774,35 @@ mod tests {
// Bucket spec round-trips exactly, including the routing column (recovered
// from its field id), maintained indexes, and writer config defaults.
let spec = LsmWriteSpec::bucket("id", 4)
let spec = MemWalWriteSpec::bucket("id", 4)
.with_maintained_indexes(vec![idx_name.clone()])
.with_writer_config_defaults([("durable_write", "false")]);
table.set_lsm_write_spec(spec.clone()).await.unwrap();
assert_eq!(table.get_lsm_write_spec().await.unwrap(), Some(spec));
table.enable_memwal(spec.clone()).await.unwrap();
assert_eq!(table.get_memwal_spec().await.unwrap(), Some(spec));
// After unset, no spec is reported.
table.unset_lsm_write_spec().await.unwrap();
assert_eq!(table.get_lsm_write_spec().await.unwrap(), None);
table.disable_memwal().await.unwrap();
assert_eq!(table.get_memwal_spec().await.unwrap(), None);
// Identity sharding round-trips (column recovered from the schema).
// A spec left at its default round-trips as the default: what is stored
// is the intent to maintain everything, not the set it resolves to now.
let spec = LsmWriteSpec::identity("region");
table.set_lsm_write_spec(spec.clone()).await.unwrap();
assert_eq!(table.get_lsm_write_spec().await.unwrap(), Some(spec));
table.unset_lsm_write_spec().await.unwrap();
let spec = MemWalWriteSpec::identity("region");
table.enable_memwal(spec.clone()).await.unwrap();
assert_eq!(table.get_memwal_spec().await.unwrap(), Some(spec));
table.disable_memwal().await.unwrap();
// Unsharded round-trips (no routing column).
let spec = LsmWriteSpec::unsharded();
table.set_lsm_write_spec(spec.clone()).await.unwrap();
assert_eq!(table.get_lsm_write_spec().await.unwrap(), Some(spec));
let spec = MemWalWriteSpec::unsharded();
table.enable_memwal(spec.clone()).await.unwrap();
assert_eq!(table.get_memwal_spec().await.unwrap(), Some(spec));
}
/// The maintained set defaults to every index on the table. A named index
/// the memtable cannot build fails the install, because the caller asked
/// for it; an unnamed one is skipped, because it arrived by existing.
#[tokio::test]
async fn test_set_lsm_write_spec_infers_maintained_indexes() {
async fn test_enable_memwal_infers_maintained_indexes() {
let tmp_dir = tempdir().unwrap();
let uri = tmp_dir.path().to_str().unwrap();
@@ -5839,8 +5842,9 @@ mod tests {
// Explicitly naming the bitmap index fails before anything commits.
let err = table
.set_lsm_write_spec(
LsmWriteSpec::unsharded().with_maintained_indexes(vec!["tag_bitmap".to_string()]),
.enable_memwal(
MemWalWriteSpec::unsharded()
.with_maintained_indexes(vec!["tag_bitmap".to_string()]),
)
.await
.unwrap_err();
@@ -5848,18 +5852,18 @@ mod tests {
matches!(err, Error::InvalidInput { ref message } if message.contains("tag_bitmap")),
"expected the bitmap index to be rejected, got {err:?}"
);
assert_eq!(table.get_lsm_write_spec().await.unwrap(), None);
assert_eq!(table.get_memwal_spec().await.unwrap(), None);
// The default names nothing, so the bitmap is skipped rather than
// refusing the table: it is maintained by existing, not by being asked
// for. The intent is what is stored, not the set it resolves to today.
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
assert_eq!(
table
.get_lsm_write_spec()
.get_memwal_spec()
.await
.unwrap()
.unwrap()
@@ -5870,16 +5874,16 @@ mod tests {
// Narrowing to a named subset: unset first, since an installed spec
// cannot be set over.
table.unset_lsm_write_spec().await.unwrap();
table.disable_memwal().await.unwrap();
table
.set_lsm_write_spec(
LsmWriteSpec::unsharded().with_maintained_indexes(vec!["id_btree".to_string()]),
.enable_memwal(
MemWalWriteSpec::unsharded().with_maintained_indexes(vec!["id_btree".to_string()]),
)
.await
.unwrap();
assert_eq!(
table
.get_lsm_write_spec()
.get_memwal_spec()
.await
.unwrap()
.unwrap()
@@ -5888,14 +5892,14 @@ mod tests {
);
// Opting out entirely is distinct from the default.
table.unset_lsm_write_spec().await.unwrap();
table.disable_memwal().await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded().with_maintained_indexes(Vec::new()))
.enable_memwal(MemWalWriteSpec::unsharded().with_maintained_indexes(Vec::new()))
.await
.unwrap();
assert_eq!(
table
.get_lsm_write_spec()
.get_memwal_spec()
.await
.unwrap()
.unwrap()
@@ -5906,7 +5910,7 @@ mod tests {
// An installed spec is never set over, whatever the new one asks for,
// and the refused call leaves it alone.
let err = table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 4))
.enable_memwal(MemWalWriteSpec::bucket("id", 4))
.await
.unwrap_err();
assert!(
@@ -5916,7 +5920,7 @@ mod tests {
);
assert_eq!(
table
.get_lsm_write_spec()
.get_memwal_spec()
.await
.unwrap()
.unwrap()
rust/lancedb/src/table/checkpoint.rs
+29 -29
View File
@@ -1,18 +1,18 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! Converging a table's LSM write path into its base table.
//! Converging a table's MemWAL write path into its base table.
//!
//! `checkpoint_lsm` seals once, then triggers compaction and watches
//! generation numbers until the L0 that existed at the start is gone.
//! `checkpoint_memwal` freezes once, then triggers compaction and watches
//! generation numbers until the SSTables that existed at the start are gone.
//!
//! The loop runs in the client, not the server: `compact_lsm` dispatches a
//! The loop runs in the client, not the server: `compact_memwal` dispatches a
//! pass and returns, so nothing holds a socket and a client can vanish
//! mid-operation with nothing to reconcile. Completion is read from
//! generation numbers in the shard manifest — durable state, unlike a count
//! in a compact response, which a concurrent write invalidates.
//!
//! The target set is fixed at the start, so generations created *during* the
//! The target set is fixed at the start, so SSTables created *during* the
//! checkpoint are ignored. That is what lets it terminate under write load,
//! and what makes it best-effort: it converges the fresh tier as of some
//! instant. Idempotent, abandonable at any point, safe on a cadence.
@@ -68,7 +68,7 @@ fn is_lost_claim(e: &Error) -> bool {
status_of(e) == Some(421)
}
/// Interval between `get_lsm_stats` polls. One interval is roughly one
/// Interval between `get_memwal_stats` polls. One interval is roughly one
/// compaction pass, the granularity at which the answer can change.
///
/// Fixed rather than configurable, matching `wait_for_index`. It costs
@@ -149,15 +149,15 @@ where
}
}
/// Drive [`Table::checkpoint_lsm`]: seal once, fix the target watermark
/// from the resulting L0, then trigger and poll until it drains.
pub(crate) async fn checkpoint_lsm(table: &Table) -> Result<()> {
/// Drive [`Table::checkpoint_memwal`]: freeze once, fix the target watermark
/// from the resulting SSTables, then trigger and poll until they drain.
pub(crate) async fn checkpoint_memwal(table: &Table) -> Result<()> {
for reissue in 0..=MAX_REISSUES {
// The seal turns everything written before this call into a
// generation, so the watermark has to be read after it. Idempotent:
// sealing an empty memtable is a no-op, so a re-issue does not churn
// empty generations.
match issue(|| table.flush_lsm()).await? {
// The freeze turns everything written before this call into an
// SSTable, so the watermark has to be read after it. Idempotent:
// freezing an empty memtable is a no-op, so a re-issue does not churn
// empty SSTables.
match issue(|| table.flush_memwal()).await? {
Attempt::Ok(()) => {}
Attempt::ReissueFromFlush => {
backoff(reissue).await;
@@ -165,7 +165,7 @@ pub(crate) async fn checkpoint_lsm(table: &Table) -> Result<()> {
}
}
let stats = match issue(|| table.get_lsm_stats(false)).await? {
let stats = match issue(|| table.get_memwal_stats(false)).await? {
Attempt::Ok(stats) => stats,
Attempt::ReissueFromFlush => {
backoff(reissue).await;
@@ -173,13 +173,13 @@ pub(crate) async fn checkpoint_lsm(table: &Table) -> Result<()> {
}
};
let Some(stats) = stats else {
// Not WAL-backed; `flush_lsm` would have errored first but for a race.
// Not WAL-backed; `flush_memwal` would have errored first but for a race.
return Ok(());
};
let targets: HashMap<String, u64> = stats
.buckets
.table_shards
.iter()
.filter_map(|b| Some((b.shard_id.clone(), b.newest_generation()?)))
.filter_map(|b| Some((b.shard_id.clone(), b.newest_sstable_generation()?)))
.collect();
if targets.is_empty() {
return Ok(());
@@ -194,18 +194,18 @@ pub(crate) async fn checkpoint_lsm(table: &Table) -> Result<()> {
}
}
Err(Error::Runtime {
message: "checkpoint_lsm: the owning node kept losing its claim; \
message: "checkpoint_memwal: the owning node kept losing its claim; \
re-issued from flush the maximum number of times"
.into(),
})
}
/// Trigger and poll until no bucket holds a generation at or below its
/// Trigger and poll until no table shard holds an SSTable generation at or below its
/// target.
///
/// No liveness bound, deliberately. The pod-wide compactor pool (a semaphore
/// of 2 by default, shared across every table on the node) is taken *inside*
/// the pass, after the bucket latch, so a checkpoint queued behind unrelated
/// the pass, after the table shard latch, so a checkpoint queued behind unrelated
/// tables is indistinguishable from one that is merging. An idle-poll counter
/// here could only ever have fired on a table that would have finished.
async fn drain_to_targets(
@@ -213,24 +213,24 @@ async fn drain_to_targets(
targets: &HashMap<String, u64>,
) -> Result<CheckpointOutcome> {
loop {
let stats = match issue(|| table.get_lsm_stats(false)).await? {
let stats = match issue(|| table.get_memwal_stats(false)).await? {
Attempt::Ok(stats) => stats,
Attempt::ReissueFromFlush => return Ok(CheckpointOutcome::ReissueFromFlush),
};
let Some(stats) = stats else {
return Ok(CheckpointOutcome::Done);
};
// `compacting` is the bucket's compaction latch, held from dispatch
// `compacting` is the table shard's compaction latch, held from dispatch
// until the pass ends — including while it waits on the pod-wide
// permit. So it answers one question only: do not pile on. Buckets
// with nothing outstanding are skipped, not counted as idle.
// permit. So it answers one question only: do not pile on. Table
// shards with nothing outstanding are skipped, not counted as idle.
let mut outstanding = 0;
let mut all_compacting = true;
for b in &stats.buckets {
for b in &stats.table_shards {
let Some(target) = targets.get(&b.shard_id) else {
continue;
};
let n = b.outstanding_generations(*target);
let n = b.outstanding_sstables(*target);
if n > 0 {
outstanding += n;
all_compacting &= b.compacting;
@@ -241,11 +241,11 @@ async fn drain_to_targets(
}
if !all_compacting {
match table.compact_lsm().await {
match table.compact_memwal().await {
Ok(()) => {}
Err(e) if is_lost_claim(&e) => return Ok(CheckpointOutcome::ReissueFromFlush),
Err(e) if !is_retryable(&e) => return Err(e),
// A 429 here means the server could latch no bucket at all,
// A 429 here means the server could latch no table shard at all,
// which the poll above already handles. Not retried in place:
// the latch it would contend for is the one doing the work, so
// fall through and re-read — `POLL_INTERVAL` is the backoff.
rust/lancedb/src/table/computed_columns.rs
+12 -12
View File
@@ -2197,7 +2197,7 @@ pub(crate) fn plan(schema: SchemaRef, columns: &[(String, String)]) -> Result<Ve
/// every declaration. For callers that stage declarations behind other work
/// and need those rejections before any of it lands.
///
/// Only the schema is consulted. Declaring also refuses a table with an LSM
/// Only the schema is consulted. Declaring also refuses a table with a MemWAL
/// write spec or retained SSTables; that is table state, checked at commit.
///
/// ```
@@ -3025,7 +3025,7 @@ mod tests {
/// The gate's reproducer: the reciprocal of the declare-under-spec check.
#[tokio::test]
async fn test_installing_an_lsm_spec_over_computed_columns_is_refused() {
use crate::table::LsmWriteSpec;
use crate::table::MemWalWriteSpec;
let tmp_dir = tempfile::tempdir().unwrap();
let conn = connect(tmp_dir.path().to_str().unwrap())
@@ -3053,14 +3053,14 @@ mod tests {
.unwrap();
let err = table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap_err();
assert!(
matches!(&err, Error::NotSupported { message } if message.contains("computed")),
"{err:?}"
);
assert!(table.get_lsm_write_spec().await.unwrap().is_none());
assert!(table.get_memwal_spec().await.unwrap().is_none());
}
/// The gate's reproducer: declaration metadata is admitted only through
@@ -3233,10 +3233,10 @@ mod tests {
);
}
/// Stale handles must not commit the computed/LSM state in either order.
/// Stale handles must not commit the computed/MemWAL state in either order.
#[tokio::test]
async fn test_stale_handles_cannot_mix_computed_and_lsm() {
use crate::table::LsmWriteSpec;
use crate::table::MemWalWriteSpec;
let tmp_dir = tempfile::tempdir().unwrap();
let uri = tmp_dir.path().to_str().unwrap();
@@ -3260,7 +3260,7 @@ mod tests {
.await
.unwrap();
let err = stale
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap_err();
assert!(matches!(err, Error::NotSupported { .. }), "install won");
@@ -3275,7 +3275,7 @@ mod tests {
table.set_unenforced_primary_key(["x"]).await.unwrap();
let stale = conn.open_table("mix2").execute().await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
let err = add_computed(&stale, &[("doubled".into(), "x * 2".into())])
@@ -3289,7 +3289,7 @@ mod tests {
/// flag is the durable marker; declaration refuses on it.
#[tokio::test]
async fn test_unset_with_retained_lsm_rows_cannot_admit_a_declaration() {
use crate::table::LsmWriteSpec;
use crate::table::MemWalWriteSpec;
use arrow_array::{Int64Array, RecordBatchIterator};
let tmp_dir = tempfile::tempdir().unwrap();
@@ -3316,7 +3316,7 @@ mod tests {
.unwrap();
table.set_unenforced_primary_key(["id"]).await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -3324,12 +3324,12 @@ mod tests {
merge
.when_matched_update_all(None)
.when_not_matched_insert_all()
.use_lsm(true);
.use_memwal(true);
merge
.execute(Box::new(RecordBatchIterator::new(vec![Ok(batch)], schema)))
.await
.unwrap();
table.unset_lsm_write_spec().await.unwrap();
table.disable_memwal().await.unwrap();
let err = add_computed(&table, &[("doubled".into(), "value * 2".into())])
.await
rust/lancedb/src/table/datafusion.rs
+7 -7
View File
@@ -166,7 +166,7 @@ pub struct BaseTableAdapter {
table: Arc<dyn BaseTable>,
schema: Arc<ArrowSchema>,
fts_query: Option<FullTextSearchQuery>,
use_lsm: Option<bool>,
use_memwal: Option<bool>,
}
impl BaseTableAdapter {
@@ -182,13 +182,13 @@ impl BaseTableAdapter {
table,
schema: Arc::new(schema),
fts_query: None,
use_lsm: None,
use_memwal: None,
})
}
/// Set the MemWAL read routing of every scan; see [`QueryRequest::use_lsm`].
pub fn with_use_lsm(mut self, use_lsm: Option<bool>) -> Self {
self.use_lsm = use_lsm;
/// Set the MemWAL read routing of every scan; see [`QueryRequest::use_memwal`].
pub fn with_use_memwal(mut self, use_memwal: Option<bool>) -> Self {
self.use_memwal = use_memwal;
self
}
@@ -204,7 +204,7 @@ impl BaseTableAdapter {
table: self.table.clone(),
schema,
fts_query: Some(fts_query),
use_lsm: self.use_lsm,
use_memwal: self.use_memwal,
}
}
}
@@ -232,7 +232,7 @@ impl TableProvider for BaseTableAdapter {
let mut query = QueryRequest {
full_text_search: self.fts_query.clone(),
disable_scoring_autoprojection: disable_scoring,
use_lsm: self.use_lsm,
use_memwal: self.use_memwal,
..Default::default()
};
rust/lancedb/src/table/lsm_stats.rs → rust/lancedb/src/table/memwal_stats.rs
+49 -43
View File
@@ -1,21 +1,21 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! Live per-bucket LSM state — the shape [`crate::Table::get_lsm_stats`]
//! Live per-table-shard MemWAL state — the shape [`crate::Table::get_memwal_stats`]
//! returns and [`super::checkpoint`] polls.
//!
//! Nothing here is derived: sums and differences (total L0 bytes, WAL lag)
//! Nothing here is derived: sums and differences (total SSTable bytes, WAL lag)
//! are the caller's to compute. There is no "WAL is off" shape — that case is
//! `None`, because a struct of zeros would read as measurements.
use serde::Deserialize;
/// One flushed L0 generation.
/// One SSTable.
#[derive(Debug, Clone, Deserialize)]
pub struct GenerationStats {
pub struct SsTableStats {
pub generation: u64,
pub bytes: u64,
/// Present only when `include_generation_rows` was requested. Off by
/// Present only when `include_sstable_rows` was requested. Off by
/// default because each count opens an uncached Lance dataset, and the
/// checkpoint loop polls this route needing only generation numbers.
#[serde(default)]
@@ -34,11 +34,11 @@ pub struct MemtableStats {
pub indexes: Vec<String>,
}
/// Live state of one bucket. A table is N buckets on one node; flattening to
/// a single number hides the one hot bucket that is usually why someone
/// Live state of one table shard. A table is N table shards on one node; flattening to
/// a single number hides the one hot table shard that is usually why someone
/// opened this endpoint.
#[derive(Debug, Clone, Deserialize)]
pub struct BucketStats {
pub struct TableShardStats {
pub shard_id: String,
/// `Active` | `Sealed` (drop-table 2PC in flight).
pub status: String,
@@ -47,68 +47,68 @@ pub struct BucketStats {
pub current_generation: u64,
pub replay_after_wal_entry_position: u64,
pub wal_entry_position_last_seen: u64,
pub generations: Vec<GenerationStats>,
/// Whether a pass owns this bucket's compaction latch right now. Says *a*
pub sstables: Vec<SsTableStats>,
/// Whether a pass owns this table shard's compaction latch right now. Says *a*
/// driver is running, not *whose*, and the latch is held from dispatch —
/// including while the pass queues for a pod-wide compactor permit. Read
/// it as "do not pile on", never as "mine is progressing".
pub compacting: bool,
/// Oldest first, active last. Absent for a `Sealed` bucket, whose
/// Oldest first, active last. Absent for a `Sealed` table shard, whose
/// in-memory state is torn down.
#[serde(default)]
pub memtables: Option<Vec<MemtableStats>>,
}
impl BucketStats {
/// The newest flushed generation, or `None` when L0 is empty.
pub(crate) fn newest_generation(&self) -> Option<u64> {
self.generations.iter().map(|g| g.generation).max()
impl TableShardStats {
/// The newest SSTable generation, or `None` when the tier is empty.
pub(crate) fn newest_sstable_generation(&self) -> Option<u64> {
self.sstables.iter().map(|g| g.generation).max()
}
/// How many generations at or below `target` are still in L0.
/// How many SSTables at or below `target` are still uncompacted.
///
/// A count, not a boolean: one pass drains a bounded prefix rather than
/// the whole target set, so a boolean would read as "no progress" for
/// every pass but the last. Compaction drains oldest-first, so this
/// decreases monotonically.
pub(crate) fn outstanding_generations(&self, target: u64) -> usize {
self.generations
pub(crate) fn outstanding_sstables(&self, target: u64) -> usize {
self.sstables
.iter()
.filter(|g| g.generation <= target)
.count()
}
}
/// Live LSM state, one entry per bucket.
/// Live MemWAL state, one entry per table shard.
#[derive(Debug, Clone, Deserialize)]
pub struct LsmStats {
pub buckets: Vec<BucketStats>,
pub struct MemWalStats {
pub table_shards: Vec<TableShardStats>,
}
/// Server-side JSON envelope for `get_lsm_stats`. `lsm_stats` is null when
/// the table has no LSM write path.
/// Server-side JSON envelope for `get_memwal_stats`. `memwal_stats` is null when
/// the table has no MemWAL write path.
#[derive(Debug, Deserialize)]
pub(crate) struct GetLsmStatsResponse {
pub(crate) struct GetMemWalStatsResponse {
#[serde(default)]
pub lsm_stats: Option<LsmStats>,
pub memwal_stats: Option<MemWalStats>,
}
#[cfg(test)]
mod tests {
use super::*;
fn bucket(shard: &str, generations: &[u64], compacting: bool) -> BucketStats {
BucketStats {
fn table_shard(shard: &str, sstables: &[u64], compacting: bool) -> TableShardStats {
TableShardStats {
shard_id: shard.into(),
status: "Active".into(),
writer_epoch: 1,
manifest_version: 1,
current_generation: generations.iter().max().copied().unwrap_or(0) + 1,
current_generation: sstables.iter().max().copied().unwrap_or(0) + 1,
replay_after_wal_entry_position: 0,
wal_entry_position_last_seen: 0,
generations: generations
sstables: sstables
.iter()
.map(|g| GenerationStats {
.map(|g| SsTableStats {
generation: *g,
bytes: 1,
rows: None,
@@ -123,40 +123,46 @@ mod tests {
/// generation created after it must not hold the loop open — that is why
/// the predicate terminates under write load.
#[test]
fn newer_generations_do_not_extend_the_target() {
let start = bucket("b0", &[7, 8], false);
let target = start.newest_generation().expect("L0 is non-empty");
fn newer_sstables_do_not_extend_the_target() {
let start = table_shard("b0", &[7, 8], false);
let target = start
.newest_sstable_generation()
.expect("the SSTable tier is non-empty");
assert_eq!(target, 8);
// Compaction drained 7 and 8; 9 and 10 arrived while it ran.
let later = bucket("b0", &[9, 10], false);
let later = table_shard("b0", &[9, 10], false);
assert_eq!(
later.outstanding_generations(target),
later.outstanding_sstables(target),
0,
"generations above the target are somebody else's problem"
"sstables above the target are somebody else's problem"
);
// Still holding 8 means still outstanding.
assert_eq!(
bucket("b0", &[8, 9], false).outstanding_generations(target),
table_shard("b0", &[8, 9], false).outstanding_sstables(target),
1
);
}
/// The metric counts generations, not buckets: a pass drains a bounded
/// prefix, so one bucket going 3 → 2 → 1 → 0 is three steps.
/// The metric counts SSTables, not table shards: a pass drains a bounded
/// prefix, so one table shard going 3 → 2 → 1 → 0 is three steps.
#[test]
fn progress_is_measured_in_generations() {
fn progress_is_measured_in_sstables() {
let target = 3;
let counts: Vec<usize> = [&[1u64, 2, 3][..], &[2, 3][..], &[3][..], &[][..]]
.iter()
.map(|gens| bucket("b0", gens, false).outstanding_generations(target))
.map(|gens| table_shard("b0", gens, false).outstanding_sstables(target))
.collect();
assert_eq!(counts, vec![3, 2, 1, 0]);
}
#[test]
fn empty_l0_has_no_target() {
assert!(bucket("b0", &[], false).newest_generation().is_none());
fn an_empty_sstable_tier_has_no_target() {
assert!(
table_shard("b0", &[], false)
.newest_sstable_generation()
.is_none()
);
}
}
rust/lancedb/src/table/merge.rs
+99 -98
View File
@@ -47,7 +47,7 @@ pub struct MergeResult {
/// Total number of rows written.
///
/// On the standard `merge_insert` path this equals
/// `num_inserted_rows + num_updated_rows`. On the MemWAL LSM write path the
/// `num_inserted_rows + num_updated_rows`. On the MemWAL write path the
/// insert/update breakdown is not known until compaction; in that mode
/// `num_inserted_rows`, `num_updated_rows`, `num_deleted_rows`, `version`
/// and `num_attempts` are all `0` and this field holds the total number of
@@ -76,7 +76,7 @@ pub struct MergeInsertBuilder {
pub(crate) when_not_matched_by_source_delete_filt: Option<MergeFilter>,
pub(crate) timeout: Option<Duration>,
pub(crate) use_index: bool,
pub(crate) use_lsm: Option<bool>,
pub(crate) use_memwal: Option<bool>,
pub(crate) validate_single_shard: bool,
}
@@ -92,7 +92,7 @@ impl MergeInsertBuilder {
when_not_matched_by_source_delete_filt: None,
timeout: None,
use_index: true,
use_lsm: None,
use_memwal: None,
validate_single_shard: true,
}
}
@@ -193,27 +193,27 @@ impl MergeInsertBuilder {
/// Control MemWAL routing for this `merge_insert`.
///
/// By default (unset), a `merge_insert` on a table with an
/// [`LsmWriteSpec`](super::LsmWriteSpec) installed is routed through Lance's
/// [`MemWalWriteSpec`](super::MemWalWriteSpec) installed is routed through Lance's
/// MemWAL shard writer; a table without one uses the standard path.
///
/// - `use_lsm(true)` forces MemWAL routing and errors if the table has no
/// LSM write spec.
/// - `use_lsm(false)` forces the standard write path even when a spec is set.
pub fn use_lsm(&mut self, enable: bool) -> &mut Self {
self.use_lsm = Some(enable);
/// - `use_memwal(true)` forces MemWAL routing and errors if the table has no
/// MemWAL write spec.
/// - `use_memwal(false)` forces the standard write path even when a spec is set.
pub fn use_memwal(&mut self, enable: bool) -> &mut Self {
self.use_memwal = Some(enable);
self
}
/// Controls how an LSM `merge_insert` checks that its input targets a
/// single shard.
///
/// When a table has an LSM write spec, every row in a `merge_insert` call
/// When a table has a MemWAL write spec, every row in a `merge_insert` call
/// must route to the same shard. When `true` (the default), every row is
/// inspected to verify this. When `false`, only the first row is inspected
/// and the shard it routes to is used for the whole input — a faster path
/// for callers that have already pre-sharded their input.
///
/// Has no effect on tables without an LSM write spec.
/// Has no effect on tables without a MemWAL write spec.
pub fn validate_single_shard(&mut self, validate_single_shard: bool) -> &mut Self {
self.validate_single_shard = validate_single_shard;
self
@@ -890,7 +890,7 @@ mod lsm_tests {
use crate::connect;
use crate::error::Error;
use crate::table::{LsmWriteSpec, Table};
use crate::table::{MemWalWriteSpec, Table};
/// A reader of `[id: Int64, value: Int64]` rows; `value` is `0..n`.
fn id_value_reader(ids: Vec<i64>) -> Box<dyn RecordBatchReader + Send> {
@@ -974,7 +974,7 @@ mod lsm_tests {
let table = id_value_table(&dir).await;
// num_buckets = 1: every row routes to the single bucket.
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 1))
.enable_memwal(MemWalWriteSpec::bucket("id", 1))
.await
.unwrap();
@@ -1001,7 +1001,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -1030,7 +1030,7 @@ mod lsm_tests {
.unwrap();
table.set_unenforced_primary_key(["id"]).await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::identity("region"))
.enable_memwal(MemWalWriteSpec::identity("region"))
.await
.unwrap();
@@ -1047,18 +1047,18 @@ mod lsm_tests {
}
#[tokio::test]
async fn lsm_merge_insert_use_lsm_false_falls_back() {
async fn lsm_merge_insert_use_memwal_false_falls_back() {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 1))
.enable_memwal(MemWalWriteSpec::bucket("id", 1))
.await
.unwrap();
// use_lsm(false) opts out: the standard path runs and commits even though
// use_memwal(false) opts out: the standard path runs and commits even though
// a spec is installed.
let mut builder = table.merge_insert(&["id"]);
builder.when_not_matched_insert_all().use_lsm(false);
builder.when_not_matched_insert_all().use_memwal(false);
let result = builder
.execute(id_value_reader(vec![3, 4, 5]))
.await
@@ -1069,17 +1069,17 @@ mod lsm_tests {
}
#[tokio::test]
async fn lsm_merge_insert_use_lsm_true_without_spec_errors() {
async fn lsm_merge_insert_use_memwal_true_without_spec_errors() {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
// use_lsm(true) demands MemWAL routing; without a write spec it errors
// use_memwal(true) demands MemWAL routing; without a write spec it errors
// rather than silently falling back to the standard path.
let mut builder = table.merge_insert(&["id"]);
builder
.when_matched_update_all(None)
.when_not_matched_insert_all()
.use_lsm(true);
.use_memwal(true);
let err = builder
.execute(id_value_reader(vec![3, 4, 5]))
.await
@@ -1092,7 +1092,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 1))
.enable_memwal(MemWalWriteSpec::bucket("id", 1))
.await
.unwrap();
@@ -1109,7 +1109,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 1))
.enable_memwal(MemWalWriteSpec::bucket("id", 1))
.await
.unwrap();
@@ -1125,7 +1125,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 1))
.enable_memwal(MemWalWriteSpec::bucket("id", 1))
.await
.unwrap();
@@ -1135,7 +1135,7 @@ mod lsm_tests {
.when_not_matched_insert_all();
builder.execute(id_value_reader(vec![7, 8])).await.unwrap();
table.close_lsm_writers().await.unwrap();
table.close_memwal_writers().await.unwrap();
// The writer reopens lazily on the next merge_insert.
let mut builder = table.merge_insert(&[]);
@@ -1160,7 +1160,7 @@ mod lsm_tests {
.unwrap();
table.set_unenforced_primary_key(["id"]).await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::identity("region"))
.enable_memwal(MemWalWriteSpec::identity("region"))
.await
.unwrap();
@@ -1197,10 +1197,10 @@ mod lsm_tests {
#[tokio::test]
async fn lsm_merge_insert_no_spec_uses_standard_path() {
let dir = tempdir().unwrap();
// id_value_table sets a primary key but no LSM write spec.
// id_value_table sets a primary key but no MemWAL write spec.
let table = id_value_table(&dir).await;
// Without a spec, a default merge_insert (use_lsm unset) simply uses
// Without a spec, a default merge_insert (use_memwal unset) simply uses
// the standard path and commits — no opt-out required, no error.
let mut builder = table.merge_insert(&["id"]);
builder
@@ -1228,7 +1228,7 @@ mod lsm_tests {
.unwrap();
table.set_unenforced_primary_key(["id"]).await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::identity("region"))
.enable_memwal(MemWalWriteSpec::identity("region"))
.await
.unwrap();
@@ -1254,7 +1254,7 @@ mod lsm_tests {
assert!(matches!(err, Error::InvalidInput { .. }), "got {err:?}");
// After closing the writer, a different shard can be written.
table.close_lsm_writers().await.unwrap();
table.close_memwal_writers().await.unwrap();
let mut builder = table.merge_insert(&[]);
builder
.when_matched_update_all(None)
@@ -1266,7 +1266,7 @@ mod lsm_tests {
}
// ---------------------------------------------------------------------
// LSM read path
// MemWAL read path
// ---------------------------------------------------------------------
use crate::arrow::SendableRecordBatchStream;
@@ -1325,7 +1325,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await; // base: ids 1,2,3 (value 0,1,2)
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -1340,8 +1340,8 @@ mod lsm_tests {
vec![1, 2, 3, 4, 5]
);
// use_lsm(false) bypasses the MemWAL and reads the base table only.
let base_only = table.query().use_lsm(false).execute().await.unwrap();
// use_memwal(false) bypasses the MemWAL and reads the base table only.
let base_only = table.query().use_memwal(false).execute().await.unwrap();
let rows = collect_id_value(base_only).await;
assert_eq!(
rows.iter().map(|(id, _)| *id).collect::<Vec<_>>(),
@@ -1350,14 +1350,14 @@ mod lsm_tests {
}
#[tokio::test]
async fn base_table_adapter_use_lsm_false_scans_the_base_only() {
async fn base_table_adapter_use_memwal_false_scans_the_base_only() {
use crate::table::datafusion::BaseTableAdapter;
use datafusion::prelude::SessionContext;
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await; // base: ids 1,2,3
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
// Ids 4,5 land in the active memtable, not the base table.
@@ -1366,7 +1366,7 @@ mod lsm_tests {
let adapter = BaseTableAdapter::try_new(table.base_table().clone())
.await
.unwrap()
.with_use_lsm(Some(false));
.with_use_memwal(Some(false));
let ctx = SessionContext::new();
ctx.register_table("base_only", Arc::new(adapter)).unwrap();
let batches = ctx
@@ -1389,7 +1389,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
lsm_upsert(&table, vec![4, 5]).await;
@@ -1407,7 +1407,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
lsm_upsert(&table, vec![4]).await;
@@ -1427,7 +1427,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await; // base: id 2 -> value 1
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -1445,7 +1445,7 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
lsm_upsert(&table, vec![2, 3, 4]).await; // id 2 -> value 0 (shadows base)
@@ -1460,14 +1460,14 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::bucket("id", 8))
.enable_memwal(MemWalWriteSpec::bucket("id", 8))
.await
.unwrap();
// Two single-row upserts that route to (likely) different buckets; each
// Two single-row upserts that route to (likely) different shards; each
// closes the writer so the next opens a fresh shard.
lsm_upsert(&table, vec![10]).await;
table.close_lsm_writers().await.unwrap();
table.close_memwal_writers().await.unwrap();
lsm_upsert(&table, vec![11]).await;
let lsm = table.query().execute().await.unwrap();
@@ -1482,14 +1482,14 @@ mod lsm_tests {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
lsm_upsert(&table, vec![4, 5]).await;
// close flushes the active memtable to an on-disk generation and drops
// the cached writer; the read must still see those rows via the shard
// manifest snapshot.
table.close_lsm_writers().await.unwrap();
table.close_memwal_writers().await.unwrap();
let lsm = table.query().execute().await.unwrap();
let ids: Vec<i64> = collect_id_value(lsm)
@@ -1503,11 +1503,11 @@ mod lsm_tests {
#[tokio::test]
async fn lsm_read_without_spec_reads_base() {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await; // no LSM write spec
let table = id_value_table(&dir).await; // no MemWAL write spec
// With no spec installed there is nothing to route: the default read and
// an explicit use_lsm(false) both read the base table without error.
for query in [table.query(), table.query().use_lsm(false)] {
// an explicit use_memwal(false) both read the base table without error.
for query in [table.query(), table.query().use_memwal(false)] {
let rows = collect_id_value(query.execute().await.unwrap()).await;
assert_eq!(
rows.iter().map(|(id, _)| *id).collect::<Vec<_>>(),
@@ -1517,11 +1517,11 @@ mod lsm_tests {
}
#[tokio::test]
async fn lsm_read_unsupported_shape_errors_without_use_lsm_false() {
async fn lsm_read_unsupported_shape_errors_without_use_memwal_false() {
let dir = tempdir().unwrap();
let table = id_value_table(&dir).await;
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
lsm_upsert(&table, vec![4]).await;
@@ -1538,12 +1538,12 @@ mod lsm_tests {
.expect("unsupported shape on a MemWAL table must error");
assert!(matches!(err, Error::NotSupported { .. }), "got {err:?}");
// use_lsm(false) is the escape hatch: it reads the base table only.
// use_memwal(false) is the escape hatch: it reads the base table only.
let rows = collect_id_value(
table
.query()
.with_row_id()
.use_lsm(false)
.use_memwal(false)
.execute()
.await
.unwrap(),
@@ -1555,7 +1555,7 @@ mod lsm_tests {
);
}
/// A table with `id`/`text` and a primary key, ready for an LSM write spec.
/// A table with `id`/`text` and a primary key, ready for a MemWAL write spec.
async fn lsm_text_table(dir: &tempfile::TempDir) -> crate::Table {
let conn = connect(dir.path().to_str().unwrap())
.execute()
@@ -1570,7 +1570,7 @@ mod lsm_tests {
table
}
/// Upsert `rows` through the LSM write path.
/// Upsert `rows` through the MemWAL write path.
async fn upsert_text(table: &crate::Table, rows: Vec<(i64, &str)>) {
let mut builder = table.merge_insert(&[]);
builder
@@ -1614,7 +1614,7 @@ mod lsm_tests {
let table = lsm_text_table(&dir).await;
// The default set, on a table with no FTS index yet.
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -1648,7 +1648,7 @@ mod lsm_tests {
/// A spec is replaced by unsetting it and setting the new one.
///
/// `unset_lsm_write_spec` drains the open writer, so the rows it held
/// `disable_memwal` drains the open writer, so the rows it held
/// survive the replacement, and the writer the next write opens builds its
/// MemTables from the new set.
#[tokio::test]
@@ -1668,15 +1668,15 @@ mod lsm_tests {
// Installed maintaining nothing, then written to: the writer opens with
// no FTS index.
table
.set_lsm_write_spec(LsmWriteSpec::unsharded().with_maintained_indexes(Vec::new()))
.enable_memwal(MemWalWriteSpec::unsharded().with_maintained_indexes(Vec::new()))
.await
.unwrap();
upsert_text(&table, vec![(99, "zebra")]).await;
// Setting over an installed spec is refused; unset, then set.
let err = table
.set_lsm_write_spec(
LsmWriteSpec::unsharded().with_maintained_indexes(vec![fts_index.clone()]),
.enable_memwal(
MemWalWriteSpec::unsharded().with_maintained_indexes(vec![fts_index.clone()]),
)
.await
.expect_err("an installed spec cannot be set over");
@@ -1684,9 +1684,9 @@ mod lsm_tests {
err.to_string().contains("already set"),
"unexpected error: {err}"
);
table.unset_lsm_write_spec().await.unwrap();
table.disable_memwal().await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded().with_maintained_indexes(vec![fts_index]))
.enable_memwal(MemWalWriteSpec::unsharded().with_maintained_indexes(vec![fts_index]))
.await
.unwrap();
@@ -1716,7 +1716,7 @@ mod lsm_tests {
/// A row written before the index exists is searchable once it does.
///
/// The MemTable holding it was built without the index. Creating the index
/// replaces the writer's configs, which seals that MemTable and waits for
/// replaces the writer's configs, which freezes that MemTable and waits for
/// its flush, so no resident MemTable is left that cannot answer the read.
#[tokio::test]
async fn lsm_a_row_written_before_the_index_is_searchable_after_it() {
@@ -1727,7 +1727,7 @@ mod lsm_tests {
let table = lsm_text_table(&dir).await;
// No index yet, and the default set: maintain whatever the table has.
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -1783,7 +1783,7 @@ mod lsm_tests {
.unwrap();
let fts_index = table.list_indices().await.unwrap()[0].name.clone();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded().with_maintained_indexes(vec![fts_index]))
.enable_memwal(MemWalWriteSpec::unsharded().with_maintained_indexes(vec![fts_index]))
.await
.unwrap();
@@ -1878,7 +1878,7 @@ mod lsm_tests {
}
let vec_index = table.list_indices().await.unwrap()[0].name.clone();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded().with_maintained_indexes(vec![vec_index]))
.enable_memwal(MemWalWriteSpec::unsharded().with_maintained_indexes(vec![vec_index]))
.await
.unwrap();
@@ -2129,8 +2129,8 @@ mod lsm_tests {
}
/// A kind Lance has no memtable index for is refused when named until the
/// table's registry maintains it; then it is maintained, flushed with each
/// generation, and answers searches on the flushed generations.
/// table's registry maintains it; then it is maintained, flushed into each
/// SSTable, and answers searches on the SSTables.
#[tokio::test]
async fn lsm_maintains_an_index_kind_from_the_tables_registry() {
let dir = tempdir().unwrap();
@@ -2150,10 +2150,11 @@ mod lsm_tests {
.execute()
.await
.unwrap();
let spec =
|| LsmWriteSpec::unsharded().with_maintained_indexes(vec!["region_bitmap".to_string()]);
let spec = || {
MemWalWriteSpec::unsharded().with_maintained_indexes(vec!["region_bitmap".to_string()])
};
let refused = table.set_lsm_write_spec(spec()).await.unwrap_err();
let refused = table.enable_memwal(spec()).await.unwrap_err();
assert!(
refused.to_string().contains("region_bitmap")
&& refused
@@ -2166,7 +2167,7 @@ mod lsm_tests {
.with_plugin(Arc::new(BitmapAsBTree))
.unwrap();
table.as_native().unwrap().set_mem_index_registry(registry);
table.set_lsm_write_spec(spec()).await.unwrap();
table.enable_memwal(spec()).await.unwrap();
// The registry is not stored with the table.
let other = conn.open_table("t").execute().await.unwrap();
@@ -2224,13 +2225,13 @@ mod lsm_tests {
active.index_store.index_names()
);
table.close_lsm_writers().await.unwrap();
table.close_memwal_writers().await.unwrap();
assert_eq!(fresh().await, vec![10, 11]);
assert_generations_answer_through_region_bitmap(&table, vec![10, 11]).await;
assert_sstables_answer_through_region_bitmap(&table, vec![10, 11]).await;
}
/// The generations each shard's manifest publishes.
async fn published_generations(table: &Table) -> Vec<lance::Dataset> {
/// The SSTables each shard's manifest publishes.
async fn published_sstables(table: &Table) -> Vec<lance::Dataset> {
use lance::dataset::mem_wal::ShardManifestStore;
let uri = table.uri().await.unwrap();
@@ -2238,7 +2239,7 @@ mod lsm_tests {
.await
.unwrap();
let mem_wal = std::path::Path::new(&uri).join("_mem_wal");
let mut generations = Vec::new();
let mut sstables = Vec::new();
for shard in std::fs::read_dir(&mem_wal).unwrap() {
let Ok(shard_id) = shard.unwrap().file_name().to_string_lossy().parse() else {
continue;
@@ -2250,30 +2251,30 @@ mod lsm_tests {
.expect("a shard directory has a manifest");
for sstable in manifest.sstables {
let path = mem_wal.join(shard_id.to_string()).join(&sstable.path);
generations.push(lance::Dataset::open(path.to_str().unwrap()).await.unwrap());
sstables.push(lance::Dataset::open(path.to_str().unwrap()).await.unwrap());
}
}
generations
sstables
}
/// Every published generation carries `region_bitmap` as a bitmap index on
/// Every published SSTable carries `region_bitmap` as a bitmap index on
/// `region`, and a search for `region = 'fresh'` on it uses that index.
async fn assert_generations_answer_through_region_bitmap(table: &Table, fresh: Vec<i64>) {
async fn assert_sstables_answer_through_region_bitmap(table: &Table, fresh: Vec<i64>) {
use lance::index::DatasetIndexExt;
let generations = published_generations(table).await;
let sstables = published_sstables(table).await;
assert!(
!generations.is_empty(),
"closing the writer flushed a generation"
!sstables.is_empty(),
"closing the writer flushed an SSTable"
);
let mut found = Vec::new();
for generation in generations {
let region = generation.schema().field("region").unwrap().id;
let indices = generation.load_indices().await.unwrap();
for sstable in sstables {
let region = sstable.schema().field("region").unwrap().id;
let indices = sstable.load_indices().await.unwrap();
let index = indices
.iter()
.find(|index| index.name == "region_bitmap")
.expect("the generation carries the index the plugin trained");
.expect("the SSTable carries the index the plugin trained");
assert_eq!(index.fields, vec![region]);
assert!(
index
@@ -2284,7 +2285,7 @@ mod lsm_tests {
index.index_details
);
let mut scan = generation.scan();
let mut scan = sstable.scan();
scan.filter("region = 'fresh'").unwrap();
let plan = scan.explain_plan(false).await.unwrap();
assert!(plan.contains("ScalarIndexQuery"), "{plan}");
@@ -2332,7 +2333,7 @@ mod lsm_tests {
"a clone shares the handle's registry"
);
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
@@ -2375,13 +2376,13 @@ mod lsm_tests {
active.index_store.index_names()
);
table.close_lsm_writers().await.unwrap();
// Only the generation written after the index exists carries it.
let generations = published_generations(&table).await;
table.close_memwal_writers().await.unwrap();
// Only the SSTable written after the index exists carries it.
let sstables = published_sstables(&table).await;
let mut carrying = 0;
for generation in &generations {
for sstable in &sstables {
use lance::index::DatasetIndexExt;
if generation
if sstable
.load_indices()
.await
.unwrap()
@@ -2393,8 +2394,8 @@ mod lsm_tests {
}
assert!(
carrying >= 1,
"{} generations, none carry the index",
generations.len()
"{} SSTables, none carry the index",
sstables.len()
);
}
}
rust/lancedb/src/table/merge/lsm.rs
+59 -59
View File
@@ -1,10 +1,10 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! MemWAL LSM write path for `merge_insert`.
//! MemWAL write path for `merge_insert`.
//!
//! [`set_lsm_write_spec`] installs an [`LsmWriteSpec`] on a table by creating
//! Lance's MemWAL index; [`unset_lsm_write_spec`] removes it. Once a spec is
//! [`enable_memwal`] installs an [`MemWalWriteSpec`] on a table by creating
//! Lance's MemWAL index; [`disable_memwal`] removes it. Once a spec is
//! installed, `merge_insert` upsert calls are dispatched through Lance's MemWAL
//! `ShardWriter` (LSM-style append) instead of the standard merge path — see
//! [`lsm_dispatch_decision`] and [`execute_lsm_merge_insert`].
@@ -14,7 +14,7 @@
//! unsharded). [`MergeInsertBuilder::validate_single_shard`] controls whether
//! every row is checked or only the first. A dataset writes to one shard at a
//! time; its writer is cached in the [`ShardWriterCache`] held alongside the
//! dataset, and [`close_lsm_writers`] closes it.
//! dataset, and [`close_memwal_writers`] closes it.
use std::collections::HashMap;
use std::sync::Arc;
@@ -40,9 +40,9 @@ use uuid::Uuid;
use crate::error::{Error, Result};
use crate::index::IndexConfig;
use crate::table::merge::{MergeInsertBuilder, MergeResult};
use crate::table::{BaseTable, LsmWriteSpec, NativeTable};
use crate::table::{BaseTable, MemWalWriteSpec, NativeTable};
/// Spec id of the sole sharding spec installed by [`set_lsm_write_spec`].
/// Spec id of the sole sharding spec installed by [`enable_memwal`].
/// Must match Lance's `InitializeMemWalBuilder` (`SHARDING_SPEC_ID`).
const SHARDING_SPEC_ID: u32 = 1;
@@ -61,23 +61,23 @@ const NUM_BUCKETS_PARAM: &str = "num_buckets";
const SHARD_NAMESPACE: Uuid = Uuid::from_u128(0x4c53_4d57_5249_5445_5f53_4841_5244_3031);
// =============================================================================
// set_lsm_write_spec
// enable_memwal
// =============================================================================
/// Install an [`LsmWriteSpec`] on the table.
/// Install an [`MemWalWriteSpec`] on the table.
///
/// The bucket / identity / unsharded sharding spec is constructed and validated
/// by Lance's
/// [`InitializeMemWalBuilder`](lance::dataset::mem_wal::InitializeMemWalBuilder).
#[allow(clippy::redundant_pub_crate)]
pub(crate) async fn set_lsm_write_spec(table: &NativeTable, spec: LsmWriteSpec) -> Result<()> {
pub(crate) async fn enable_memwal(table: &NativeTable, spec: MemWalWriteSpec) -> Result<()> {
table.dataset.ensure_mutable()?;
{
let dataset = table.dataset.get().await?;
if dataset.mem_wal_index_details().await?.is_some() {
return Err(Error::InvalidInput {
message: "set_lsm_write_spec: an LSM write spec is already set on this \
message: "enable_memwal: a MemWAL write spec is already set on this \
table and cannot be changed"
.into(),
});
@@ -115,14 +115,14 @@ pub(crate) async fn set_lsm_write_spec(table: &NativeTable, spec: LsmWriteSpec)
let schema = arrow_schema::Schema::from(dataset.schema());
if !crate::table::computed_columns::computed_columns(&schema).is_empty() {
return Err(Error::NotSupported {
message: "an LSM write spec cannot be installed on a table with computed \
message: "a MemWAL write spec cannot be installed on a table with computed \
columns: rows in un-compacted tiers are invisible to refresh"
.into(),
});
}
if crate::materialized_view::read_definition(&dataset.schema().metadata)?.is_some() {
return Err(Error::NotSupported {
message: "an LSM write spec cannot be installed on a materialized view: \
message: "a MemWAL write spec cannot be installed on a materialized view: \
rows in un-compacted tiers are invisible to refresh"
.into(),
});
@@ -131,7 +131,7 @@ pub(crate) async fn set_lsm_write_spec(table: &NativeTable, spec: LsmWriteSpec)
.initialize_mem_wal()
.mem_index_registry(table.mem_index_registry());
let writer_config_defaults = match spec {
LsmWriteSpec::Bucket {
MemWalWriteSpec::Bucket {
column,
num_buckets,
writer_config_defaults,
@@ -140,7 +140,7 @@ pub(crate) async fn set_lsm_write_spec(table: &NativeTable, spec: LsmWriteSpec)
builder = builder.bucket_sharding(column, num_buckets);
writer_config_defaults
}
LsmWriteSpec::Identity {
MemWalWriteSpec::Identity {
column,
writer_config_defaults,
..
@@ -148,7 +148,7 @@ pub(crate) async fn set_lsm_write_spec(table: &NativeTable, spec: LsmWriteSpec)
builder = builder.identity_sharding(column);
writer_config_defaults
}
LsmWriteSpec::Unsharded {
MemWalWriteSpec::Unsharded {
writer_config_defaults,
..
} => {
@@ -178,22 +178,22 @@ fn index_name_list(indices: &[IndexConfig]) -> String {
}
// =============================================================================
// unset_lsm_write_spec
// disable_memwal
// =============================================================================
/// Remove the [`LsmWriteSpec`] from the table by dropping the MemWAL index.
/// Remove the [`MemWalWriteSpec`] from the table by dropping the MemWAL index.
///
/// Any cached shard writers are drained and closed first. Errors if no spec is
/// currently set.
#[allow(clippy::redundant_pub_crate)]
pub(crate) async fn unset_lsm_write_spec(table: &NativeTable) -> Result<()> {
pub(crate) async fn disable_memwal(table: &NativeTable) -> Result<()> {
table.dataset.ensure_mutable()?;
{
let dataset = table.dataset.get().await?;
if dataset.mem_wal_index_details().await?.is_none() {
return Err(Error::InvalidInput {
message: "unset_lsm_write_spec: no LSM write spec is set on this table".into(),
message: "disable_memwal: no MemWAL write spec is set on this table".into(),
});
}
}
@@ -209,46 +209,46 @@ pub(crate) async fn unset_lsm_write_spec(table: &NativeTable) -> Result<()> {
}
// =============================================================================
// get_lsm_write_spec
// get_memwal_spec
// =============================================================================
/// Read the [`LsmWriteSpec`] currently installed on the table, if any.
/// Read the [`MemWalWriteSpec`] currently installed on the table, if any.
///
/// Reconstructs the spec from the MemWAL index's stored details — the same
/// metadata [`set_lsm_write_spec`] writes — read directly via
/// metadata [`enable_memwal`] writes — read directly via
/// [`mem_wal_index_details`](DatasetMemWalExt::mem_wal_index_details) (the raw
/// manifest record, not the `describe_indices` enrichment path, so it is
/// unaffected by system-index filtering there). Returns `Ok(None)` when no
/// MemWAL index is installed.
#[allow(clippy::redundant_pub_crate)]
pub(crate) async fn get_lsm_write_spec(table: &NativeTable) -> Result<Option<LsmWriteSpec>> {
pub(crate) async fn get_memwal_spec(table: &NativeTable) -> Result<Option<MemWalWriteSpec>> {
let dataset = table.dataset.get().await?;
let Some(details) = dataset.mem_wal_index_details().await? else {
return Ok(None);
};
let spec = lsm_write_spec_from_details(&details, dataset.schema())?;
let spec = memwal_spec_from_details(&details, dataset.schema())?;
Ok(Some(spec))
}
/// Reconstruct the public [`LsmWriteSpec`] from stored MemWAL index details.
/// Reconstruct the public [`MemWalWriteSpec`] from stored MemWAL index details.
///
/// The sharding transform and its parameters recover the mode; the routing
/// column is resolved from the sharding field's source id back to its schema
/// name (only bucket / identity carry a column). `maintained_indexes` and
/// `writer_config_defaults` are copied verbatim, so the result round-trips
/// what `set_lsm_write_spec` installed.
fn lsm_write_spec_from_details(
/// what `enable_memwal` installed.
fn memwal_spec_from_details(
details: &MemWalIndexDetails,
schema: &LanceSchema,
) -> Result<LsmWriteSpec> {
) -> Result<MemWalWriteSpec> {
let spec = details
.sharding_specs
.first()
.ok_or_else(|| Error::Runtime {
message: "get_lsm_write_spec: MemWAL index has no sharding spec".to_string(),
message: "get_memwal_spec: MemWAL index has no sharding spec".to_string(),
})?;
let field = spec.fields.first().ok_or_else(|| Error::Runtime {
message: "get_lsm_write_spec: MemWAL index has an empty sharding spec".to_string(),
message: "get_memwal_spec: MemWAL index has an empty sharding spec".to_string(),
})?;
let base = match field.transform.as_deref() {
@@ -259,16 +259,16 @@ fn lsm_write_spec_from_details(
.and_then(|s| s.parse::<u32>().ok())
.filter(|n| *n > 0)
.ok_or_else(|| Error::Runtime {
message: "get_lsm_write_spec: MemWAL bucket spec has a missing or invalid num_buckets parameter".to_string(),
message: "get_memwal_spec: MemWAL bucket spec has a missing or invalid num_buckets parameter".to_string(),
})?;
LsmWriteSpec::bucket(sharding_column(field, schema)?, num_buckets)
MemWalWriteSpec::bucket(sharding_column(field, schema)?, num_buckets)
}
Some(IDENTITY_TRANSFORM) => LsmWriteSpec::identity(sharding_column(field, schema)?),
Some(UNSHARDED_TRANSFORM) => LsmWriteSpec::unsharded(),
Some(IDENTITY_TRANSFORM) => MemWalWriteSpec::identity(sharding_column(field, schema)?),
Some(UNSHARDED_TRANSFORM) => MemWalWriteSpec::unsharded(),
other => {
return Err(Error::Runtime {
message: format!(
"get_lsm_write_spec: MemWAL index has an unsupported sharding transform {:?}",
"get_memwal_spec: MemWAL index has an unsupported sharding transform {:?}",
other
),
});
@@ -284,30 +284,30 @@ fn lsm_write_spec_from_details(
/// Resolve the single routing column name from a sharding field's source id.
///
/// `set_lsm_write_spec` records the shard column by its Lance field id, so the
/// `enable_memwal` records the shard column by its Lance field id, so the
/// name is recovered by looking that id up in the current schema.
fn sharding_column(field: &ShardingField, schema: &LanceSchema) -> Result<String> {
let source_id = *field.source_ids.first().ok_or_else(|| Error::Runtime {
message: "get_lsm_write_spec: sharding field has no source column".to_string(),
message: "get_memwal_spec: sharding field has no source column".to_string(),
})?;
schema
.field_by_id(source_id)
.map(|f| f.name.clone())
.ok_or_else(|| Error::Runtime {
message: format!(
"get_lsm_write_spec: sharding source field id {} not found in schema",
"get_memwal_spec: sharding source field id {} not found in schema",
source_id
),
})
}
// =============================================================================
// close_lsm_writers
// close_memwal_writers
// =============================================================================
/// Drain and close every cached MemWAL shard writer for the table.
#[allow(clippy::redundant_pub_crate)]
pub(crate) async fn close_lsm_writers(table: &NativeTable) -> Result<()> {
pub(crate) async fn close_memwal_writers(table: &NativeTable) -> Result<()> {
table.dataset.shard_writer().drain_and_close().await
}
@@ -321,7 +321,7 @@ pub(crate) async fn close_lsm_writers(table: &NativeTable) -> Result<()> {
/// so the writer lives where the dataset lives — cached for the session and
/// reused across `merge_insert` calls. A dataset writes to one shard at a
/// time; routing a `merge_insert` to a different shard requires closing the
/// current writer first via [`close_lsm_writers`]. `ShardWriter::put` takes
/// current writer first via [`close_memwal_writers`]. `ShardWriter::put` takes
/// `&self`, so concurrent puts on the cached writer are safe; `close` consumes
/// the writer, so the entry wraps it in `RwLock<Option<ShardWriter>>`.
#[derive(Default)]
@@ -370,8 +370,8 @@ impl ShardWriterEntry {
}
/// The cached writer's latest in-memory manifest (current generation +
/// flushed generations). `Ok(None)` if the writer was already closed.
/// Used by the LSM read path to snapshot this shard authoritatively
/// SSTables). `Ok(None)` if the writer was already closed.
/// Used by the MemWAL read path to snapshot this shard authoritatively
/// without re-reading the on-disk manifest.
async fn manifest(&self) -> Result<Option<lance_index::mem_wal::ShardManifest>> {
let guard = self.inner.read().await;
@@ -449,7 +449,7 @@ impl ShardWriterCache {
/// writer opened afterwards reads the committed set.
///
/// Holds the entry's read lock across the refresh, so a concurrent close
/// waits rather than retiring the writer mid-seal. Writes take the same
/// waits rather than retiring the writer mid-freeze. Writes take the same
/// lock first, so the order matches theirs.
pub(crate) async fn refresh_maintained_indexes(&self, dataset: &Dataset) {
let entry = {
@@ -463,13 +463,13 @@ impl ShardWriterCache {
let Some(writer) = guard.as_ref() else {
return;
};
// Replacing the configs seals the active memtable so the next one carries
// Replacing the configs freezes the active memtable so the next one carries
// the new indexes. The outgoing one keeps the set it was built with, so
// its flush has to land before a read: while it is still frozen, an
// indexed read sees a resident memtable that cannot answer and is
// refused.
let sealed = match dataset.refresh_mem_wal_index_configs(writer).await {
Ok(sealed) => sealed,
let frozen = match dataset.refresh_mem_wal_index_configs(writer).await {
Ok(frozen) => frozen,
Err(error) => {
log::warn!(
"the maintained index set changed but the open shard writer could \
@@ -478,18 +478,18 @@ impl ShardWriterCache {
return;
}
};
if let Some(fence) = sealed
if let Some(fence) = frozen
&& let Err(error) = fence.wait().await
{
log::warn!(
"the shard writer took the new maintained index set, but the memtable \
it sealed was not flushed, so an indexed read stays refused until \
it froze was not flushed, so an indexed read stays refused until \
it is: {error}"
);
}
}
/// Snapshot the cached writer's shard for the LSM read path: its shard id,
/// Snapshot the cached writer's shard for the MemWAL read path: its shard id,
/// authoritative in-memory manifest, and active + frozen memtable refs.
/// Returns `None` when no writer is currently cached (e.g. nothing has been
/// written this session, or the writer was closed).
@@ -537,7 +537,7 @@ fn check_shard_match(cached: Uuid, wanted: Uuid) -> Result<()> {
}
Err(Error::InvalidInput {
message: format!(
"merge_insert: a shard writer is already open for shard {} but this input routes to shard {}; call close_lsm_writers before writing to a different shard",
"merge_insert: a shard writer is already open for shard {} but this input routes to shard {}; call close_memwal_writers before writing to a different shard",
cached, wanted
),
})
@@ -569,7 +569,7 @@ pub(crate) struct LsmPlan {
/// Outcome of [`lsm_dispatch_decision`].
#[allow(clippy::redundant_pub_crate)]
pub(crate) enum LsmDispatch {
/// No LSM write spec applies; use the standard `merge_insert` path.
/// No MemWAL write spec applies; use the standard `merge_insert` path.
Standard,
/// Route the `merge_insert` through the MemWAL shard writer.
Lsm(LsmPlan),
@@ -583,18 +583,18 @@ pub(crate) async fn lsm_dispatch_decision(
params: &MergeInsertBuilder,
) -> Result<LsmDispatch> {
// Explicit opt-out: use the standard path regardless of any installed spec.
if params.use_lsm == Some(false) {
if params.use_memwal == Some(false) {
return Ok(LsmDispatch::Standard);
}
let dataset = table.dataset.get().await?;
let Some(details) = dataset.mem_wal_index_details().await? else {
// No write spec installed. `use_lsm(true)` demanded MemWAL routing, so
// No write spec installed. `use_memwal(true)` demanded MemWAL routing, so
// that is an error; otherwise fall back to the standard path.
if params.use_lsm == Some(true) {
if params.use_memwal == Some(true) {
return Err(Error::InvalidInput {
message: "use_lsm(true) was set but the table has no MemWAL write spec; \
install one with set_lsm_write_spec or leave use_lsm unset"
message: "use_memwal(true) was set but the table has no MemWAL write spec; \
install one with enable_memwal or leave use_memwal unset"
.to_string(),
});
}
@@ -616,7 +616,7 @@ pub(crate) async fn lsm_dispatch_decision(
if !params.on.is_empty() && params.on != pk_cols {
return Err(Error::InvalidInput {
message: format!(
"merge_insert: `on` columns {:?} must match the table's unenforced primary key {:?} when an LSM write spec is set; pass an empty `on` to default to the primary key",
"merge_insert: `on` columns {:?} must match the table's unenforced primary key {:?} when a MemWAL write spec is set; pass an empty `on` to default to the primary key",
params.on, pk_cols
),
});
@@ -624,7 +624,7 @@ pub(crate) async fn lsm_dispatch_decision(
if !is_upsert_only(params) {
return Err(Error::InvalidInput {
message: "merge_insert: when an LSM write spec is set, only the upsert form (when_matched_update_all without a filter + when_not_matched_insert_all, no by-source delete) is supported; call use_lsm(false) to use the standard merge_insert path".to_string(),
message: "merge_insert: when a MemWAL write spec is set, only the upsert form (when_matched_update_all without a filter + when_not_matched_insert_all, no by-source delete) is supported; call use_memwal(false) to use the standard merge_insert path".to_string(),
});
}
rust/lancedb/src/table/query.rs
+18 -18
View File
@@ -95,8 +95,8 @@ async fn can_execute_namespace_query(table: &NativeTable, query: &AnyQuery) -> R
return Ok(false);
}
// A MemWAL write spec means reads auto-route through the LSM scanner in
// `create_plan` even when `use_lsm` is unset. The namespace request has no
// use_lsm field, so pushing the default query down would silently omit
// `create_plan` even when `use_memwal` is unset. The namespace request has no
// use_memwal field, so pushing the default query down would silently omit
// un-compacted rows — force local execution whenever a spec is installed.
let dataset = table.dataset.get().await?;
if dataset.mem_wal_index_details().await?.is_some() {
@@ -106,10 +106,10 @@ async fn can_execute_namespace_query(table: &NativeTable, query: &AnyQuery) -> R
}
fn requires_local_namespace_execution(query: &AnyQuery) -> bool {
// The namespace QueryTable request cannot represent approx_mode, search_effort or use_lsm.
// For use_lsm that is worse than a tuning miss: MemWAL read routing lives only
// The namespace QueryTable request cannot represent approx_mode, search_effort or use_memwal.
// For use_memwal that is worse than a tuning miss: MemWAL read routing lives only
// in `create_plan`, so a pushed-down query would return stale base-only data.
if query.base().use_lsm.is_some() || query.base().take_offsets.is_some() {
if query.base().use_memwal.is_some() || query.base().take_offsets.is_some() {
return true;
}
matches!(
@@ -167,25 +167,25 @@ pub async fn create_plan(
let ds_ref = table.dataset.get().await?;
// MemWAL read routing driven by `use_lsm`:
// MemWAL read routing driven by `use_memwal`:
// * unset — route through the LSM scanner iff the table carries a write spec
// * Some(true) — force LSM routing; error if the table has no write spec
// * Some(true) — force MemWAL routing; error if the table has no write spec
// * Some(false) — read the base table only, bypassing the MemWAL
// The LSM scanner surfaces in-flight `merge_insert` data (active/frozen
// memtables + flushed generations); validation and dispatch live in `lsm`.
// memtables + SSTables); validation and dispatch live in `lsm`.
let has_spec = ds_ref.mem_wal_index_details().await?.is_some();
let use_lsm = match query.base.use_lsm {
let use_memwal = match query.base.use_memwal {
Some(true) if !has_spec => {
return Err(Error::InvalidInput {
message: "use_lsm(true) was set but the table has no MemWAL write spec; \
install one with set_lsm_write_spec or leave use_lsm unset"
message: "use_memwal(true) was set but the table has no MemWAL write spec; \
install one with enable_memwal or leave use_memwal unset"
.to_string(),
});
}
Some(enable) => enable,
None => has_spec,
};
if use_lsm {
if use_memwal {
return lsm::create_lsm_plan(table, ds_ref, query).await;
}
@@ -1195,7 +1195,7 @@ mod tests {
table.set_unenforced_primary_key(["id"]).await.unwrap();
table
.set_lsm_write_spec(crate::table::LsmWriteSpec::unsharded())
.enable_memwal(crate::table::MemWalWriteSpec::unsharded())
.await
.unwrap();
let lsm_error = table
@@ -1395,7 +1395,7 @@ mod tests {
}
#[tokio::test]
async fn test_execute_query_use_lsm_with_namespace_pushdown_runs_locally() {
async fn test_execute_query_use_memwal_with_namespace_pushdown_runs_locally() {
use crate::connect;
use crate::table::query::execute_query;
use arrow_array::{Int32Array, RecordBatch};
@@ -1418,7 +1418,7 @@ mod tests {
.unwrap();
let table = conn
.create_table("test_use_lsm_namespace_fallback", batch)
.create_table("test_use_memwal_namespace_fallback", batch)
.execute()
.await
.unwrap();
@@ -1429,13 +1429,13 @@ mod tests {
.pushdown_operations
.insert(NamespaceClientPushdownOperation::QueryTable);
// `use_lsm` set (even to false) must force local execution — the namespace
// request has no use_lsm field, so a pushdown would silently ignore it.
// `use_memwal` set (even to false) must force local execution — the namespace
// request has no use_memwal field, so a pushdown would silently ignore it.
let query_vector = Arc::new(Float32Array::from(vec![0.0, 0.0]));
let query = AnyQuery::VectorQuery(VectorQueryRequest {
base: QueryRequest {
limit: Some(1),
use_lsm: Some(false),
use_memwal: Some(false),
..Default::default()
},
column: Some("vector".to_string()),
rust/lancedb/src/table/query/lsm.rs
+23 -23
View File
@@ -1,14 +1,14 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! MemWAL LSM read path.
//! MemWAL read path.
//!
//! When a table has an LSM write spec installed (see [`set_lsm_write_spec`]),
//! When a table has a MemWAL write spec installed (see [`enable_memwal`]),
//! reads are routed through Lance's [`LsmScanner`] instead of the plain
//! base-table scan unless the query sets
//! [`use_lsm(false)`](crate::query::QueryBase::use_lsm). This makes data
//! [`use_memwal(false)`](crate::query::QueryBase::use_memwal). This makes data
//! written via the LSM `merge_insert` path — which lives in the active/frozen
//! in-memory memtables and the flushed SSTable generations until an external
//! in-memory memtables and the SSTables until an external
//! compaction merges it into the base table — visible to queries, deduplicated by
//! primary key (newest generation wins).
//!
@@ -17,10 +17,10 @@
//! three run through a single [`LsmScanner`], so a `where` filter is honored as a
//! prefilter uniformly — including for vector search, where `LsmScanner` threads
//! it into the vector planner's prefilter. Shapes the LSM path cannot honor are
//! rejected with [`Error::NotSupported`]; the caller must set `use_lsm(false)` to
//! rejected with [`Error::NotSupported`]; the caller must set `use_memwal(false)` to
//! run those against the base table.
//!
//! [`set_lsm_write_spec`]: crate::Table::set_lsm_write_spec
//! [`enable_memwal`]: crate::Table::enable_memwal
use std::collections::HashMap;
use std::sync::Arc;
@@ -57,7 +57,7 @@ const LSM_OVERFETCH_FACTOR: f64 = 2.0;
/// The caller guarantees `ds_ref` carries a MemWAL write spec (routing is decided
/// in [`create_plan`](super::create_plan)). Errors with [`Error::NotSupported`]
/// for query shapes the LSM scanner cannot honor — the caller must set
/// `use_lsm(false)` to run those against the base table.
/// `use_memwal(false)` to run those against the base table.
pub(super) async fn create_lsm_plan(
table: &NativeTable,
ds_ref: Arc<Dataset>,
@@ -67,11 +67,11 @@ pub(super) async fn create_lsm_plan(
// A time-traveled (checked-out) handle pins an older dataset version, but the
// WAL manifests and cached writer expose current live state — mixing them would
// surface WAL rows written after the requested version. `use_lsm(false)` reads
// surface WAL rows written after the requested version. `use_memwal(false)` reads
// the base table at the pinned version.
if table.dataset.time_travel_version().is_some() {
return Err(Error::NotSupported {
message: "the MemWAL LSM scanner cannot read from a time-traveled dataset version; set use_lsm(false) to read the base table at this version".to_string(),
message: "the MemWAL LSM scanner cannot read from a time-traveled dataset version; set use_memwal(false) to read the base table at this version".to_string(),
});
}
@@ -80,7 +80,7 @@ pub(super) async fn create_lsm_plan(
.mem_wal_index_details()
.await?
.ok_or_else(|| Error::Runtime {
message: "the MemWAL LSM write spec disappeared during read planning".to_string(),
message: "the MemWAL write spec disappeared during read planning".to_string(),
})?;
let pk_columns = pk_columns(&ds_ref)?;
@@ -135,10 +135,10 @@ pub(super) async fn create_lsm_plan(
restore_projection(plan, &query, &pk_columns)
}
/// Reject query shapes the LSM read path does not implement. On a MemWAL table
/// Reject query shapes the MemWAL read path does not implement. On a MemWAL table
/// reads route through the LSM scanner by default, so an unsupported shape is a
/// hard error rather than a silent fallback to the base-only scan — which would
/// exclude un-compacted MemWAL data. The caller must set `use_lsm(false)` to
/// exclude un-compacted MemWAL data. The caller must set `use_memwal(false)` to
/// run these against the base table, accepting that the results omit un-compacted
/// MemWAL data.
///
@@ -148,7 +148,7 @@ fn reject_unsupported(query: &VectorQueryRequest) -> Result<()> {
let unsupported = |what: &str| {
Err(Error::NotSupported {
message: format!(
"the MemWAL LSM scanner does not support {what}; set use_lsm(false) to read the base table only (results will exclude un-compacted MemWAL data)"
"the MemWAL LSM scanner does not support {what}; set use_memwal(false) to read the base table only (results will exclude un-compacted MemWAL data)"
),
})
};
@@ -275,7 +275,7 @@ fn exclusion_watermarks(
exclude
}
/// Assemble the per-shard snapshots (flushed SSTable generations) and the
/// Assemble the per-shard snapshots (SSTables) and the
/// in-memory memtables (active + frozen) for the table.
///
/// Snapshots for all shards come from their on-disk manifests; for the shard
@@ -413,7 +413,7 @@ fn base_scanner(
QueryFilter::Datafusion(expr) => scanner.filter_expr(expr.clone()),
QueryFilter::Substrait(_) => {
return Err(Error::NotSupported {
message: "the MemWAL LSM scanner does not support Substrait filters; set use_lsm(false) to read the base table only".to_string(),
message: "the MemWAL LSM scanner does not support Substrait filters; set use_memwal(false) to read the base table only".to_string(),
});
}
};
@@ -458,11 +458,11 @@ async fn fts_plan(
offset: Option<usize>,
) -> Result<Arc<dyn ExecutionPlan>> {
// Pre-check for a lancedb-flavored error; `LsmScanner` also validates the
// single-column requirement, but without the `use_lsm(false)` guidance.
// single-column requirement, but without the `use_memwal(false)` guidance.
let columns: Vec<String> = fts.columns().into_iter().collect();
if columns.len() > 1 {
return Err(Error::NotSupported {
message: "the MemWAL LSM scanner full-text search supports a single column; set use_lsm(false) to read the base table only".to_string(),
message: "the MemWAL LSM scanner full-text search supports a single column; set use_memwal(false) to read the base table only".to_string(),
});
}
let column = columns.first().ok_or_else(|| Error::NotSupported {
@@ -482,7 +482,7 @@ async fn fts_plan(
{
return Err(Error::NotSupported {
message: format!(
"the MemWAL LSM scanner full-text search requires the FTS index on '{column}' to be maintained by the write spec (LsmWriteSpec::with_maintained_indexes) and carried by every resident MemTable; otherwise un-compacted documents are omitted. set use_lsm(false) to read the base table only"
"the MemWAL LSM scanner full-text search requires the FTS index on '{column}' to be maintained by the write spec (MemWalWriteSpec::with_maintained_indexes) and carried by every resident MemTable; otherwise un-compacted documents are omitted. set use_memwal(false) to read the base table only"
),
});
}
@@ -633,7 +633,7 @@ fn resolve_single_index(
if names.len() > 1 {
return Err(Error::NotSupported {
message: format!(
"the MemWAL LSM scanner cannot resolve the {arm} index catch-up watermark for '{column}': it has multiple {arm} indexes; set use_lsm(false) to read the base table only"
"the MemWAL LSM scanner cannot resolve the {arm} index catch-up watermark for '{column}': it has multiple {arm} indexes; set use_memwal(false) to read the base table only"
),
});
}
@@ -682,7 +682,7 @@ fn restore_projection(
///
/// Note: the base and SSTable arms use `fast_search` (indexed data only), so a
/// base-table row not covered by a vector index is invisible here — it surfaces
/// only via the memtable or `use_lsm(false)`.
/// only via the memtable or `use_memwal(false)`.
#[allow(clippy::too_many_arguments)]
async fn vector_plan(
dataset: &Dataset,
@@ -717,7 +717,7 @@ async fn vector_plan(
if !index_maintained(dataset, &column, details, "VectorIndexDetails").await? {
return Err(Error::NotSupported {
message: format!(
"the MemWAL LSM scanner requires the vector index on '{column}' to be maintained by the write spec (LsmWriteSpec::with_maintained_indexes); otherwise compacted rows not yet re-indexed are omitted. set use_lsm(false) to read the base table only"
"the MemWAL LSM scanner requires the vector index on '{column}' to be maintained by the write spec (MemWalWriteSpec::with_maintained_indexes); otherwise compacted rows not yet re-indexed are omitted. set use_memwal(false) to read the base table only"
),
});
}
@@ -726,7 +726,7 @@ async fn vector_plan(
// clear error rather than failing deep in the planner.
if is_binary_vector_column(&arrow_schema, &column) {
return Err(Error::NotSupported {
message: "the MemWAL LSM scanner does not support binary (uint8) vector search; set use_lsm(false) to read the base table only".to_string(),
message: "the MemWAL LSM scanner does not support binary (uint8) vector search; set use_memwal(false) to read the base table only".to_string(),
});
}
@@ -855,7 +855,7 @@ mod tests {
fn exclusion_watermark_gates_on_lagging_index_catchup() {
let shard = Uuid::from_u128(1);
let details = MemWalIndexDetails {
// Compaction has drained generations through 5 into the base table...
// Compaction has drained SSTables through generation 5 into the base table...
compacted_sstables: vec![CompactedSsTable::new(shard, 5)],
// ...but the FTS index has only caught up through generation 2.
index_catchup: vec![IndexCatchupProgress::new(
rust/lancedb/src/table/refresh.rs
+11 -11
View File
@@ -87,7 +87,7 @@ async fn execute_refresh_column_with_source(
column: &str,
) -> Result<RefreshExecution> {
table.dataset.ensure_mutable()?;
ensure_no_lsm_write_spec(table).await?;
ensure_no_memwal_spec(table).await?;
let dataset = table.dataset.get().await?;
let expression = declared_expression(&dataset, column)?;
@@ -303,7 +303,7 @@ pub(crate) async fn execute_refresh_column_async(
// Validate before spawning so bad input is reported by this call rather
// than only by the job.
table.dataset.ensure_mutable()?;
ensure_no_lsm_write_spec(table).await?;
ensure_no_memwal_spec(table).await?;
let dataset = table.dataset.get().await?;
declared_expression(&dataset, column)?;
drop(dataset);
@@ -323,12 +323,12 @@ pub(crate) async fn execute_refresh_column_async(
})))
}
/// Refuse to refresh under an LSM write spec.
/// Refuse to refresh under a MemWAL write spec.
///
/// Refresh enumerates base fragments, and a write spec keeps visible rows in
/// un-compacted MemWAL tiers it cannot reach -- success would silently omit
/// readable rows.
async fn ensure_no_lsm_write_spec(table: &NativeTable) -> Result<()> {
async fn ensure_no_memwal_spec(table: &NativeTable) -> Result<()> {
use lance::dataset::mem_wal::DatasetMemWalExt;
// Unset drops the MemWAL index, so the spec alone stops describing a table
@@ -341,7 +341,7 @@ async fn ensure_no_lsm_write_spec(table: &NativeTable) -> Result<()> {
.list_mem_wal_latest_shard_ids()
.await?
.is_empty();
if retained_sstables || table.get_lsm_write_spec().await?.is_some() {
if retained_sstables || table.get_memwal_spec().await?.is_some() {
return Err(Error::NotSupported {
message: "refresh_column is not supported on a table with an LSM write \
spec: rows in un-compacted tiers are invisible to refresh"
@@ -1637,7 +1637,7 @@ mod tests {
/// writer left in that state.
#[tokio::test]
async fn test_refresh_refuses_a_foreign_lsm_state() {
use crate::table::LsmWriteSpec;
use crate::table::MemWalWriteSpec;
let tmp_dir = tempfile::tempdir().unwrap();
let conn = connect(tmp_dir.path().to_str().unwrap())
@@ -1655,7 +1655,7 @@ mod tests {
let table = conn.create_table("lsm", batch).execute().await.unwrap();
table.set_unenforced_primary_key(["x"]).await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
super::super::computed_columns::add_foreign_kind(&table, "doubled", "sql").await;
@@ -1673,7 +1673,7 @@ mod tests {
/// flag still marks retained SSTable rows; refresh refuses on the flag.
#[tokio::test]
async fn test_refresh_refuses_retained_catchup_state() {
use crate::table::LsmWriteSpec;
use crate::table::MemWalWriteSpec;
let tmp_dir = tempfile::tempdir().unwrap();
let conn = connect(tmp_dir.path().to_str().unwrap())
@@ -1697,14 +1697,14 @@ mod tests {
.unwrap();
table.set_unenforced_primary_key(["x"]).await.unwrap();
table
.set_lsm_write_spec(LsmWriteSpec::unsharded())
.enable_memwal(MemWalWriteSpec::unsharded())
.await
.unwrap();
let mut merge = table.merge_insert(&["x"]);
merge
.when_matched_update_all(None)
.when_not_matched_insert_all()
.use_lsm(true);
.use_memwal(true);
merge
.execute(Box::new(arrow_array::RecordBatchIterator::new(
vec![Ok(batch)],
@@ -1712,7 +1712,7 @@ mod tests {
)))
.await
.unwrap();
table.unset_lsm_write_spec().await.unwrap();
table.disable_memwal().await.unwrap();
super::super::computed_columns::add_foreign_kind(&table, "doubled", "sql").await;
let err = table.refresh_column("doubled").await.unwrap_err();
rust/lancedb/src/table/schema_evolution.rs
+2 -2
View File
@@ -131,7 +131,7 @@ pub(crate) async fn execute_declare(
) -> Result<AddColumnsResult> {
use lance::dataset::mem_wal::DatasetMemWalExt;
// An LSM write spec keeps visible rows in tiers refresh cannot reach;
// A MemWAL write spec keeps visible rows in tiers refresh cannot reach;
// checked against latest committed state, not this handle's snapshot.
table.checkout_latest().await?;
computed_columns::ensure_not_function_bound(
@@ -149,7 +149,7 @@ pub(crate) async fn execute_declare(
.list_mem_wal_latest_shard_ids()
.await?
.is_empty();
if retained_sstables || table.get_lsm_write_spec().await?.is_some() {
if retained_sstables || table.get_memwal_spec().await?.is_some() {
return Err(Error::NotSupported {
message: "computed columns are not supported on a table with an LSM write \
spec: rows in un-compacted tiers are invisible to refresh"