diff --git a/docs/rfcs/2026-09-06-object-store-wal.md b/docs/rfcs/2026-09-06-object-store-wal.md index c94abc073aa..be152130289 100644 --- a/docs/rfcs/2026-09-06-object-store-wal.md +++ b/docs/rfcs/2026-09-06-object-store-wal.md @@ -174,10 +174,51 @@ To move the WAL to another location, stop the instance, copy every object under The check does not catch every change of location. A location whose prefix already holds objects passes a region that never flushed, and a region whose watermark names an object at or below the newest object holding entries there, whatever history those objects hold. A refused start on an empty prefix leaves its start object behind when the process ends before the delete completes or when the delete fails, which is logged; the next start then finds a prefix with only start objects and passes a region that never flushed, while it still refuses every region whose watermark names an object. The same holds when the store created a region before the refusal. When it admitted an append or a release before the refusal, as it can when regions open while the instance already serves, the store withdraws nothing and the prefix keeps an object holding entries, which may already have collected the start object; the next start then also passes every region whose watermark names an object at or below that one. -**Metrics.** The store exports metrics under `greptime_logstore_object_store_wal_`. The write path reports `seal_to_durable_seconds`, the time from sealing a batch until its object is durable, `append_ack_seconds`, the time an append waits for its acknowledgement in the `durable` mode, `object_bytes` and `object_entries`, the size and the entry count of every object sealed, and `created_objects_total`, the creates that succeeded, an identical retry included; a create is counted and timed whether or not its batch is still there to be indexed, since an object a poisoned store gave up on is as durable as any other. `indexed_objects` and `indexed_bytes` are the objects and the bytes the catalog holds, set by a recovery that succeeded and moved by every object indexed or unindexed afterwards, so a collection takes the object it deleted off both, and a recovery that is abandoned publishes neither. A recovery that succeeds reports `recovery_seconds` and the objects it read footers from in `recovered_objects_total`, both of the same recoveries, so one that is abandoned at a fetch or rejected by the catalog contributes to neither; a read stream reports `read_seconds` once it has been read to its end, and one that fails or is dropped part way reports nothing. The failure paths are counted: `create_conflicts_total` for a create that found a conflicting object, `create_failures_total` for one the object store did not confirm, `poisoned_total` for a store that recorded a terminal error, `skipped_segments_total` for a segment a read skipped, `missing_history_accepted_total` for a region that `accept_missing_history` let open without its history, and `stalled_appends_total` with `stalled_append_seconds` for an append held back because the sealed batches were full, in either mode, or by the `enqueued` backlog thresholds, timed until it is admitted or refused, a stop, a terminal error, a failing store or the teardown of the actor releasing it included. The gauge `failing` counts the stores whose backlog the object store refuses with an error that requires intervention, and `rejected_appends_total` the appends and releases such a store rejected; the attempts of a failing store are counted in `create_failures_total`. The delete counters of a collection and the release record counter are described under *Entry ids*. The histogram buckets are chosen for the default flush interval of 100 ms and the default `max_batch_bytes` of 8 MiB. +**Metrics.** The store exports metrics under `greptime_logstore_object_store_wal_`. The write path reports `seal_to_durable_seconds`, the time from sealing a batch until its object is durable, `append_ack_seconds`, the time an append waits for its acknowledgement in the `durable` mode, `object_bytes` and `object_entries`, the size and the entry count of every object sealed, and `created_objects_total`, the creates that succeeded, an identical retry included; a create is counted and timed whether or not its batch is still there to be indexed, since an object a poisoned store gave up on is as durable as any other. `indexed_objects` and `indexed_bytes` are the objects and the bytes the catalog holds, set by a recovery that succeeded and moved by every object indexed or unindexed afterwards, so a collection takes the object it deleted off both, and a recovery that is abandoned publishes neither. A recovery that succeeds reports `recovery_seconds` and the objects it read footers from in `recovered_objects_total`, both of the same recoveries, so one that is abandoned at a fetch or rejected by the catalog contributes to neither; a read stream reports `read_seconds` once it has been read to its end, and one that fails or is dropped part way reports nothing. The failure paths are counted: `create_conflicts_total` for a create that found a conflicting object, `create_failures_total` for one the object store did not confirm, `poisoned_total` for a store that recorded a terminal error, `skipped_segments_total` for a segment a read skipped, `missing_history_accepted_total` for a region that `accept_missing_history` let open without its history, and `stalled_appends_total` with `stalled_append_seconds` for an append held back because the sealed batches were full, in either mode, or by the `enqueued` backlog thresholds, timed until it is admitted or refused, a stop, a terminal error, a failing store or the teardown of the actor releasing it included. The gauge `failing` counts the stores whose backlog the object store refuses with an error that requires intervention, and `rejected_appends_total` the appends and releases such a store rejected; the attempts of a failing store are counted in `create_failures_total`. The gauge `io_permission_unavailable` counts live stores observed without permission to initiate object-store I/O, including idle stores and those awaiting their first grant. Each store is counted once while permission is unavailable and removed from the count when permission returns or its actor exits; the gauge measures permission availability, not backlog size. The delete counters of a collection and the release record counter are described under *Entry ids*. The histogram buckets are chosen for the default flush interval of 100 ms and the default `max_batch_bytes` of 8 MiB. A `testing` cargo feature exposes hooks to wait for admitted appends, to seal the open batch, to hold and release creates and to make creates fail, so engine tests can flush deterministically and observe entries that are admitted but not durable. +### Node lease and recovery fencing + +A metasrv-controlled datanode accepts object-store WAL only when its plugins +contain `EnableObjectStoreWal`; OSS does not register this marker. The builder +installs an I/O permit backed by Mito's node lease before opening regions. The +permit exposes only whether new object-store I/O is allowed; the WAL does not +own deadlines or heartbeat renewal. Standalone installs no permit. The node +prefix retains the fixed `epochs/0` directory across restarts, independently of +the heartbeat node epoch and the store epoch in object headers. Metasrv +configuration, failover integration and Enterprise registration remain separate +prerequisites for enabling the distributed WAL. +Startup's initial object and its withdrawal after a missing-history failure are +exempt so startup does not depend on a heartbeat that has not started yet. + +Every new upload, including a retry or release-only object, and every collection +delete checks the permit immediately before issuance. Issued operations may +finish, including their transport retries. The flush timer checks expiry and +renewal without sealing while expired. Expiry fails durable open and unindexed +batches with a retryable error, ignores their late upload results and never +reuses their ids. Enqueued batches already acknowledged remain pending until +renewal. Release records remain available for a later batch, while their current +waiters fail. An admission that would wait for upload capacity fails immediately +while expired; ordinary admissions continue to rely on the region write lease. + +The datanode keeps the heartbeat running until the object-store WAL stops, then +closes the heartbeat and regions. If the lease expires during shutdown, the WAL +drops its unindexed backlog without waiting for uploads, and still waits for +issued deletes. Expiry is checked on the next flush timer tick, including during +shutdown, so detection can take one configured flush interval plus actor +scheduling delay. Dropping acknowledged enqueued entries has crash semantics. + +Recovery still selects the latest complete chain, but before writing its initial +object checks that the selected tip carries the maximum epoch among all listed +headers. Falling back to an older epoch fails without modifying storage, with +the skipped epochs, object counts and sequence ranges in the error. Explicit +`accept_missing_history` permits this fallback only as an operator recovery: +previous writers must have exited and acknowledged unflushed data may be lost. +The new initial object's epoch exceeds every listed epoch. An incomplete branch +within the selected epoch does not trigger this rejection. Remote readers always +reject an older-epoch fallback, independent of this setting. + ## Mito integration Mito maps `WalOptions::ObjectStore { prefix }` to `Provider::object_store_provider(region_id, prefix)`; resolving object store options against a Raft Engine or Kafka log store reports the existing incompatible-provider error. Reads go through the region-scoped reader used for Kafka, so a namespace shared by many regions never leaks entries across regions even if a log store returned them. Object store regions are opened individually rather than grouped by topic. After replay, `topic_latest_entry_id` follows the existing remote-WAL rule: it is taken from the store's latest entry id only when the region's memtables are empty, otherwise it stays at the replay start, because that value feeds pruning and must never advance past unflushed data. Before a flush, a full truncate or a discard of unflushed data publishes a manifest watermark, Mito waits on the durability barrier described under *Acknowledgement*; with the `durable` mode and with every other provider the wait returns at once. When Mito drops a region it calls `release_region` after `obsolete`, as described under *Entry ids*; the default of that method does nothing, so the other providers handle a dropped region as before. Neither a release nor an `obsolete` of an object store region that fails, the latter for instance on a store that the release of another region poisoned, fails the drop, since nothing sends the drop of a removed region again and a caller that drops several regions, such as the metric engine with its data and metadata regions, would leave the others undone: the region's runtime state is cleaned up and its files are scheduled for deletion as for any drop, and the failure is logged; an `obsolete` that fails for another provider fails the drop as before. A store that is neither stopped nor poisoned keeps the record and writes it with its next object; a stopped or poisoned store keeps none, and the region stays unreleased after a restart. The store carries the record in a later object, as described under *Entry ids*; a crash before any later create succeeds loses it, and the dropped region's segments then hold the reclaim boundary after the restart. Before Mito replays an existing region on open, on catch-up or for a create that finds the region present, it runs the history check described under *Log store*. @@ -271,7 +312,7 @@ The reference implementation was validated on the following evidence; the tests - Object store access: an unwritable store; a listing, a trailer fetch, a footer fetch, a segment fetch and a delete that fail; a delete that fails and is retried by the next collection; an object collected between the listing and its fetch, below and at the tip's reclaim boundary; an object store that answers slowly. - Store state: an object the catalog rejects, per-region entry id exhaustion, the catalog refusing a next sequence past the maximum, every stop path with several creates in flight that succeed, fail transiently or conflict after stop began, a stop with a delete in flight, an append held back by full sealed batches, and the appends the backlog thresholds were holding back released by admission, by a stop, by a terminal error and by the teardown of the actor. - Pipelined uploader: concurrent creates acknowledged in sequence order, every sealed batch linked to its pipeline predecessor, the open batch sealing at the threshold while creates are in flight, a transient failure failing every batch that is not indexed, created ones included, a permanent failure poisoning the store. -- Chain recovery: a linear chain, orphans below and above the tip, a broken tip falling back to the latest complete object, a late object of an earlier epoch next to the chain of a later one, a missing predecessor, objects present without a complete chain, orphans raising the next sequence, and the start object on an empty prefix and meeting objects of an earlier and of a later epoch. +- Chain recovery: a linear chain, orphans below and above the tip, an incomplete branch within the selected epoch, rejection of an older-epoch fallback before any mutation and its explicit acceptance with `accept_missing_history`, a late object of an earlier epoch next to the chain of a later one, a missing predecessor, objects present without a complete chain, orphans raising the next sequence, and the start object on an empty prefix and meeting objects of an earlier and of a later epoch. ### What the evidence does not cover diff --git a/src/cmd/src/standalone.rs b/src/cmd/src/standalone.rs index 0521aa9404f..9dbcb89abc4 100644 --- a/src/cmd/src/standalone.rs +++ b/src/cmd/src/standalone.rs @@ -54,7 +54,7 @@ use common_telemetry::{info, warn}; use common_time::timezone::set_default_timezone; use common_version::{short_version, verbose_version}; use common_wal::config::DatanodeWalConfig; -use common_wal::config::object_store::STANDALONE_GENERATION; +use common_wal::config::object_store::FIXED_DIRECTORY_GENERATION; use datanode::config::{DatanodeOptions, StorageConfig}; use datanode::datanode::{Datanode, DatanodeBuilder}; use datanode::region_server::RegionServer; @@ -98,7 +98,7 @@ pub async fn build_standalone_wal_provider( ) -> Result { match wal { DatanodeWalConfig::ObjectStore(config) => Ok(WalProvider::ObjectStore { - prefix: config.node_prefix(node_id.unwrap_or(0), STANDALONE_GENERATION), + prefix: config.node_prefix(node_id.unwrap_or(0), FIXED_DIRECTORY_GENERATION), }), DatanodeWalConfig::RaftEngine(_) | DatanodeWalConfig::Kafka(_) @@ -1220,7 +1220,7 @@ mod tests { ..Default::default() }; // The regions persist the node prefix the datanode runs its store under. - let node_prefix = config.node_prefix(0, STANDALONE_GENERATION); + let node_prefix = config.node_prefix(0, FIXED_DIRECTORY_GENERATION); assert_eq!(node_prefix, "cluster-a/wal/datanodes/0/epochs/0"); let provider = build_standalone_wal_provider( &DatanodeWalConfig::ObjectStore(config), diff --git a/src/common/wal/src/config/object_store.rs b/src/common/wal/src/config/object_store.rs index 9e4a371baae..c41adadcb69 100644 --- a/src/common/wal/src/config/object_store.rs +++ b/src/common/wal/src/config/object_store.rs @@ -17,12 +17,19 @@ use std::time::Duration; use common_base::readable_size::ReadableSize; use serde::{Deserialize, Serialize}; -/// Generation of the node prefix in standalone mode. +/// Fixed directory generation used by standalone and distributed WAL stores. /// -/// Standalone runs a single datanode that never changes generation, so the -/// prefix layout carries the constant. Distributed mode will take both the -/// node id and the generation from the metasrv. -pub const STANDALONE_GENERATION: u64 = 0; +/// The directory remains stable across process restarts. This is distinct from +/// both the heartbeat node epoch and the store epoch carried in object headers. +pub const FIXED_DIRECTORY_GENERATION: u64 = 0; + +/// Plugin marker allowing a metasrv-controlled datanode to use object-store WAL. +/// +/// OSS does not register this marker. Cluster integration registers it only after +/// the distributed WAL prerequisites are available; it does not itself configure +/// metasrv or enable region failover. +#[derive(Debug, Clone, Copy)] +pub struct EnableObjectStoreWal; /// When an append to the object store WAL returns. #[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] @@ -124,7 +131,7 @@ mod tests { fn test_node_prefix() { let config = ObjectStoreWalConfig::default(); assert_eq!( - config.node_prefix(0, STANDALONE_GENERATION), + config.node_prefix(0, FIXED_DIRECTORY_GENERATION), "wal/datanodes/0/epochs/0" ); diff --git a/src/datanode/src/datanode.rs b/src/datanode/src/datanode.rs index f035ecb92ea..f33188a8208 100644 --- a/src/datanode/src/datanode.rs +++ b/src/datanode/src/datanode.rs @@ -41,7 +41,9 @@ use common_stat::ResourceStatImpl; use common_telemetry::{error, info, warn}; use common_wal::config::DatanodeWalConfig; use common_wal::config::kafka::DatanodeKafkaConfig; -use common_wal::config::object_store::{ObjectStoreWalConfig, STANDALONE_GENERATION}; +use common_wal::config::object_store::{ + EnableObjectStoreWal, FIXED_DIRECTORY_GENERATION, ObjectStoreWalConfig, +}; use common_wal::config::raft_engine::RaftEngineConfig; use file_engine::engine::FileRegionEngine; use futures::future::join_all; @@ -162,15 +164,6 @@ impl Datanode { ); let _ = self.greptimedb_telemetry_task.stop().await; - if let Some(heartbeat_task) = &self.heartbeat_task { - record_shutdown_error( - &mut first_error, - heartbeat_task - .close() - .map_err(BoxedError::new) - .context(ShutdownInstanceSnafu), - ); - } // The object store WAL stops before the regions close: its stop // uploads the backlog or drops it, and answers every wait for its // entries, so a flush that waits for the WAL, queued or run by the @@ -185,6 +178,16 @@ impl Datanode { .context(ShutdownInstanceSnafu), ); } + // WAL draining still needs heartbeat renewal; stop it before the heartbeat. + if let Some(heartbeat_task) = &self.heartbeat_task { + record_shutdown_error( + &mut first_error, + heartbeat_task + .close() + .map_err(BoxedError::new) + .context(ShutdownInstanceSnafu), + ); + } record_shutdown_error(&mut first_error, self.region_server.stop().await); // The stopped engines no longer hold the raft-engine WAL, so it can be closed. if let Some(log_store) = self.raft_engine_log_store.take() { @@ -358,7 +361,10 @@ impl DatanodeBuilder { // writable upon open. let controlled_by_metasrv = meta_client.is_some(); if let DatanodeWalConfig::ObjectStore(config) = &self.opts.wal { - ensure!(!controlled_by_metasrv, ObjectStoreWalNotStandaloneSnafu); + ensure!( + !controlled_by_metasrv || self.plugins.get::().is_some(), + ObjectStoreWalNotStandaloneSnafu + ); validate_object_store_wal_config(config)?; } @@ -405,6 +411,12 @@ impl DatanodeBuilder { // the metasrv. if controlled_by_metasrv && let Some(mito) = region_server.mito_engine() { mito.require_write_lease(); + if let Some(log_store) = &self.object_store_log_store { + log_store + .set_io_permit(mito.object_store_wal_io_permit()) + .map_err(Box::new) + .context(OpenLogStoreSnafu)?; + } } // TODO(weny): Considering introducing a readonly kv_backend trait. @@ -887,7 +899,7 @@ impl DatanodeBuilder { } /// Builds [ObjectStoreLogStore] under the node prefix of `node_id` in the - /// standalone generation, the prefix the regions persist in their WAL options. + /// fixed directory generation, the prefix the regions persist in their WAL options. async fn build_object_store_log_store( object_store: ObjectStore, config: &ObjectStoreWalConfig, @@ -897,7 +909,7 @@ impl DatanodeBuilder { "Creating object store logstore with config: {:?}, node id: {}", config, node_id ); - ObjectStoreLogStore::try_new(object_store, config, node_id, STANDALONE_GENERATION) + ObjectStoreLogStore::try_new(object_store, config, node_id, FIXED_DIRECTORY_GENERATION) .await .map_err(Box::new) .context(OpenLogStoreSnafu) @@ -1207,7 +1219,7 @@ mod tests { use std::path::Path; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; - use std::time::Duration; + use std::time::{Duration, Instant}; use api::v1::{ColumnSchema, Rows}; use cache::build_datanode_layered_cache_registry; @@ -1226,7 +1238,9 @@ mod tests { use common_meta::node_manager::NodeManagerRef; use common_test_util::temp_dir::{create_named_temp_file, create_temp_dir}; use common_wal::config::DatanodeWalConfig; - use common_wal::config::object_store::{AckMode, ObjectStoreWalConfig, STANDALONE_GENERATION}; + use common_wal::config::object_store::{ + AckMode, EnableObjectStoreWal, FIXED_DIRECTORY_GENERATION, ObjectStoreWalConfig, + }; use common_wal::config::raft_engine::RaftEngineConfig; use common_wal::options::{ObjectStoreWalOptions, WAL_OPTIONS_KEY, WalOptions}; use log_store::ObjectStoreLogStore; @@ -1555,6 +1569,99 @@ mod tests { assert!(is_empty_dir(data_home.path())); } + #[tokio::test] + async fn test_cluster_object_store_wal_builder_installs_node_lease() { + let data_home = create_temp_dir("cluster-wal-node-lease"); + let config = ObjectStoreWalConfig { + prefix: "cluster-a/wal".to_string(), + ..enqueued_wal_config(Duration::from_secs(3600)) + }; + let mut builder = + object_store_wal_builder(data_home.path().to_str().unwrap(), config.clone()); + builder.opts.node_id = Some(7); + builder.plugins.insert(EnableObjectStoreWal); + builder.with_meta_client(Arc::new( + MetaClientBuilder::datanode_default_options(7).build(), + )); + let mut datanode = builder.build().await.unwrap(); + assert!(datanode.heartbeat_task.is_some()); + let log_store = datanode.object_store_log_store().unwrap(); + let engine = datanode.region_server.mito_engine().unwrap(); + let prefix = config.node_prefix(7, FIXED_DIRECTORY_GENERATION); + assert_eq!("cluster-a/wal/datanodes/7/epochs/0", prefix); + let provider = Provider::object_store_provider(REGION, prefix); + let entry = log_store.entry(vec![1], 0, REGION, &provider).unwrap(); + let response = log_store.append_batch(vec![entry]).await.unwrap(); + let entry_id = response.last_entry_ids[®ION]; + + // The builder installs the permit: no test-side set_io_permit call. + assert_matches!( + log_store.seal_open_batch().await.unwrap_err(), + LogStoreError::ObjectStoreWalLeaseExpired { .. } + ); + assert_eq!(0, log_store.latest_entry_id(&provider).unwrap()); + let deadline = Instant::now() + Duration::from_secs(60); + engine.renew_node_lease(deadline); + log_store.seal_open_batch().await.unwrap(); + assert_eq!(entry_id, log_store.latest_entry_id(&provider).unwrap()); + engine.freeze_lease_clock(deadline); + assert_matches!( + log_store.seal_open_batch().await.unwrap_err(), + LogStoreError::ObjectStoreWalLeaseExpired { .. } + ); + engine.renew_node_lease(deadline + Duration::from_secs(60)); + log_store.seal_open_batch().await.unwrap(); + datanode.shutdown().await.unwrap(); + } + + #[tokio::test] + async fn test_cluster_object_store_wal_shutdown_keeps_heartbeat_until_drained() { + let data_home = create_temp_dir("cluster-wal-shutdown"); + let config = enqueued_wal_config(Duration::from_secs(3600)); + let mut builder = + object_store_wal_builder(data_home.path().to_str().unwrap(), config.clone()); + builder.plugins.insert(EnableObjectStoreWal); + builder.with_meta_client(Arc::new( + MetaClientBuilder::datanode_default_options(0).build(), + )); + let mut datanode = builder.build().await.unwrap(); + let log_store = datanode.object_store_log_store().unwrap(); + let engine = datanode.region_server.mito_engine().unwrap(); + engine.renew_node_lease(Instant::now() + WAIT); + // Observe the real close flag without starting network heartbeats. + let heartbeat_running = datanode + .heartbeat_task + .as_ref() + .unwrap() + .running_flag_for_test(); + heartbeat_running.store(true, Ordering::Release); + let provider = Provider::object_store_provider( + REGION, + config.node_prefix(0, FIXED_DIRECTORY_GENERATION), + ); + let entry = log_store.entry(vec![1], 0, REGION, &provider).unwrap(); + let response = log_store.append_batch(vec![entry]).await.unwrap(); + log_store.hold_creates(); + let shutdown = tokio::spawn(async move { datanode.shutdown().await }); + tokio::time::timeout(WAIT, log_store.wait_for_parked_creates(1)) + .await + .unwrap() + .unwrap(); + assert!(!shutdown.is_finished()); + assert!(heartbeat_running.load(Ordering::Acquire)); + log_store.release_creates(); + tokio::time::timeout(WAIT, shutdown) + .await + .unwrap() + .unwrap() + .unwrap(); + assert!(!heartbeat_running.load(Ordering::Acquire)); + assert_eq!( + response.last_entry_ids[®ION], + log_store.latest_entry_id(&provider).unwrap() + ); + } + #[tokio::test] async fn test_build_rejects_unknown_wal_storage_provider() { let data_home = create_temp_dir("object-store-wal-unknown-provider"); @@ -1913,7 +2020,7 @@ mod tests { /// The options that put a region on the WAL of a datanode with `config`. fn region_wal_options(config: &ObjectStoreWalConfig) -> HashMap { - let prefix = config.node_prefix(0, STANDALONE_GENERATION); + let prefix = config.node_prefix(0, FIXED_DIRECTORY_GENERATION); let wal_options = serde_json::to_string(&WalOptions::ObjectStore(ObjectStoreWalOptions::new(prefix))) .unwrap(); @@ -1961,7 +2068,7 @@ mod tests { skipped_rows: usize, ) -> (Datanode, Arc) { let config = enqueued_wal_config(flush_interval); - let prefix = config.node_prefix(0, STANDALONE_GENERATION); + let prefix = config.node_prefix(0, FIXED_DIRECTORY_GENERATION); let mut request = CreateRequestBuilder::new().build(); request.options.extend(region_wal_options(&config)); let datanode = object_store_wal_builder(data_home, config) @@ -2162,7 +2269,7 @@ mod tests { fn test_node_prefix_passes_validation() { let config = ObjectStoreWalConfig::default(); let derived = ObjectStoreWalConfig { - prefix: config.node_prefix(0, STANDALONE_GENERATION), + prefix: config.node_prefix(0, FIXED_DIRECTORY_GENERATION), ..config }; validate_object_store_wal_config(&derived).unwrap(); diff --git a/src/datanode/src/error.rs b/src/datanode/src/error.rs index d5037c2874e..2a560393ad7 100644 --- a/src/datanode/src/error.rs +++ b/src/datanode/src/error.rs @@ -330,7 +330,9 @@ pub enum Error { location: Location, }, - #[snafu(display("Object store WAL is only supported in standalone mode"))] + #[snafu(display( + "Object store WAL supports standalone only in this build; metasrv-controlled deployment requires a build with distributed object-store WAL integration" + ))] ObjectStoreWalNotStandalone { #[snafu(implicit)] location: Location, diff --git a/src/datanode/src/heartbeat.rs b/src/datanode/src/heartbeat.rs index a00e3487f5a..7127dbbd6a1 100644 --- a/src/datanode/src/heartbeat.rs +++ b/src/datanode/src/heartbeat.rs @@ -501,6 +501,12 @@ impl HeartbeatTask { .collect() } + /// Exposes the lifecycle flag for shutdown-order tests without a network task. + #[cfg(test)] + pub(crate) fn running_flag_for_test(&self) -> Arc { + self.running.clone() + } + pub fn close(&self) -> Result<()> { let running = self.running.clone(); if running diff --git a/src/datanode/src/region_server.rs b/src/datanode/src/region_server.rs index 8c961e8c28b..fbfa7ad57de 100644 --- a/src/datanode/src/region_server.rs +++ b/src/datanode/src/region_server.rs @@ -2470,7 +2470,7 @@ mod tests { use common_error::ext::{ErrorExt, RetryHint}; use common_recordbatch::RecordBatches; use common_recordbatch::adapter::{RecordBatchMetrics, RegionWatermarkEntry}; - use common_wal::config::object_store::{ObjectStoreWalConfig, STANDALONE_GENERATION}; + use common_wal::config::object_store::{FIXED_DIRECTORY_GENERATION, ObjectStoreWalConfig}; use common_wal::options::{ObjectStoreWalOptions, WAL_OPTIONS_KEY, WalOptions}; use datatypes::prelude::{ConcreteDataType, VectorRef}; use datatypes::schema::{ColumnSchema, Schema}; @@ -3736,7 +3736,7 @@ mod tests { ObjectStore::new(Memory::default()).unwrap(), &wal_config, 0, - STANDALONE_GENERATION, + FIXED_DIRECTORY_GENERATION, ) .await .unwrap(); @@ -3746,7 +3746,7 @@ mod tests { let mut region_server = mock_region_server(); region_server.register_engine(Arc::new(engine)); let wal_options = serde_json::to_string(&WalOptions::ObjectStore( - ObjectStoreWalOptions::new(wal_config.node_prefix(0, STANDALONE_GENERATION)), + ObjectStoreWalOptions::new(wal_config.node_prefix(0, FIXED_DIRECTORY_GENERATION)), )) .unwrap(); (region_server, log_store, wal_options) diff --git a/src/log-store/src/error.rs b/src/log-store/src/error.rs index 3215db2ce21..8da6f8b71bb 100644 --- a/src/log-store/src/error.rs +++ b/src/log-store/src/error.rs @@ -93,6 +93,12 @@ pub enum Error { location: Location, }, + #[snafu(display("Object store WAL I/O permit is already installed"))] + WalIoPermitAlreadyInstalled { + #[snafu(implicit)] + location: Location, + }, + #[snafu(display("Log store not started yet"))] IllegalState { #[snafu(implicit)] @@ -482,6 +488,12 @@ pub enum Error { location: Location, }, + #[snafu(display("Object store WAL node lease is unavailable or expired"))] + ObjectStoreWalLeaseExpired { + #[snafu(implicit)] + location: Location, + }, + #[snafu(display("Object store WAL operation failed"))] ObjectStoreWal { source: Arc, @@ -556,11 +568,14 @@ impl ErrorExt for Error { | WalObjectConflict { .. } | WalObjectSequenceExhausted { .. } | WalEntryPositionExhausted { .. } => StatusCode::Unexpected, - WalObjectSequenceUnsettled { .. } => StatusCode::IllegalState, + WalObjectSequenceUnsettled { .. } | WalIoPermitAlreadyInstalled { .. } => { + StatusCode::IllegalState + } InvalidWalObject { source, .. } => source.status_code(), ObjectStoreWal { source, .. } => source.status_code(), ObjectStoreWalFailing { .. } => StatusCode::StorageUnavailable, + ObjectStoreWalLeaseExpired { .. } => StatusCode::RegionNotReady, // Object store related errors WalObjectStore { .. } @@ -602,6 +617,7 @@ impl ErrorExt for Error { | StaleWalObject { .. } | UnconfirmedWalEpochStart { .. } | ObjectStoreWalFailing { .. } => RetryHint::Retryable, + ObjectStoreWalLeaseExpired { .. } => RetryHint::RetryAfterRouteRefresh, ProduceRecord { error, .. } => match error { rskafka::client::producer::Error::Client(error) => { rskafka_client_error_to_retry_hint(error) diff --git a/src/log-store/src/metrics.rs b/src/log-store/src/metrics.rs index bdb123759eb..602db03147c 100644 --- a/src/log-store/src/metrics.rs +++ b/src/log-store/src/metrics.rs @@ -299,6 +299,15 @@ lazy_static! { ) .unwrap(); + /// Number of live object store logstores observed without permission to + /// initiate I/O, including before the first grant and while idle. This + /// measures permission availability, not retained backlog size. + pub static ref METRIC_OBJECT_STORE_WAL_IO_PERMISSION_UNAVAILABLE: IntGauge = register_int_gauge!( + "greptime_logstore_object_store_wal_io_permission_unavailable", + "live object store logstores observed without permission to initiate I/O, including idle stores and those awaiting their first grant", + ) + .unwrap(); + /// Number of object store logstores in the `enqueued` mode whose backlog /// the object store refuses with an error that repeating the create /// cannot settle, such as denied access or a missing bucket. Such a store diff --git a/src/log-store/src/object_store_wal.rs b/src/log-store/src/object_store_wal.rs index 6a24a756475..d99c1df1a7a 100644 --- a/src/log-store/src/object_store_wal.rs +++ b/src/log-store/src/object_store_wal.rs @@ -47,6 +47,21 @@ //! accepts writes, and its epoch is one above the sequence of that object, so //! no two instances share an epoch and a late object of an earlier instance //! never ends the chain. +//! Opening rejects a selected chain below the maximum listed epoch before +//! writing its start object. `accept_missing_history` permits an explicit +//! recovery with possible acknowledged data loss; remote readers never accept +//! this fallback. +//! +//! With the cluster acceptance plugin registered, the datanode installs a +//! node-lease I/O permit before opening regions. OSS registers no such plugin. +//! New creates and deletes check the permit immediately before issuance; +//! already issued operations may finish. Expiry fails unacknowledged durable +//! batches and pauses the acknowledged enqueued backlog until renewal. Stop +//! drops that backlog if the lease expires and still waits for issued deletes. +//! The next flush timer tick detects expiry, including during stop; detection +//! can take one configured flush interval plus actor scheduling delay. +//! Standalone stores do not install a permit. Startup's initial create and its +//! withdrawal on missing history are exempt, so startup needs no heartbeat. //! //! The reclaim boundary of an object states that every object with a lower //! sequence was fully flushed when the object was sealed, so the chain need @@ -79,7 +94,7 @@ //! per source prefix. Listing an upper bound reads no object indexes; preparing //! a range refreshes the listing and reuses cached indexes. Known last-entry //! bounds anchor their own chains, while listed bounds use the complete-chain -//! selection of startup recovery with an additional maximum-epoch check. +//! selection of startup recovery, requiring the maximum listed epoch. //! Neither path opens a writable store or modifies the source. The caller owns //! the lifetime of pending recovery ranges and //! publishes progress only after a successful read. diff --git a/src/log-store/src/object_store_wal/store.rs b/src/log-store/src/object_store_wal/store.rs index 46e76afdec5..7d58a8e2ba9 100644 --- a/src/log-store/src/object_store_wal/store.rs +++ b/src/log-store/src/object_store_wal/store.rs @@ -19,7 +19,7 @@ use std::fmt; #[cfg(any(test, feature = "testing"))] use std::ops::Range; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Mutex, PoisonError, RwLock}; +use std::sync::{Arc, Mutex, OnceLock, PoisonError, RwLock}; use std::time::{Duration, Instant}; use async_stream::try_stream; @@ -33,7 +33,9 @@ use object_store::{ErrorKind, ObjectStore}; use snafu::{IntoError, OptionExt, ResultExt, ensure}; use store_api::logstore::entry::{Entry, NaiveEntry}; use store_api::logstore::provider::{ObjectStoreProvider, Provider}; -use store_api::logstore::{AppendBatchResponse, EntryId, LogStore, SendableEntryStream}; +use store_api::logstore::{ + AppendBatchResponse, EntryId, LogStore, ObjectStoreWalIoPermit, SendableEntryStream, +}; use store_api::storage::RegionId; use tokio::sync::{Semaphore, mpsc, oneshot, watch}; use tokio::time::MissedTickBehavior; @@ -42,8 +44,8 @@ use crate::error::{ CorruptedWalObjectSnafu, Error, IncompleteWalEntrySnafu, InvalidProviderSnafu, InvalidWalObjectSnafu, InvalidWalObjectStoreSnafu, MismatchedWalPrefixSnafu, MismatchedWalRegionSnafu, MissingWalHistorySnafu, ObjectStoreWalFailingSnafu, - ObjectStoreWalSnafu, ObjectStoreWalStoppedSnafu, Result, StaleWalObjectSnafu, - UnconfirmedWalEpochStartSnafu, WalObjectSequenceExhaustedSnafu, + ObjectStoreWalLeaseExpiredSnafu, ObjectStoreWalSnafu, ObjectStoreWalStoppedSnafu, Result, + StaleWalObjectSnafu, UnconfirmedWalEpochStartSnafu, WalObjectSequenceExhaustedSnafu, WalObjectSequenceUnsettledSnafu, WalStartWithdrawnSnafu, }; use crate::metrics::{ @@ -51,7 +53,7 @@ use crate::metrics::{ METRIC_OBJECT_STORE_WAL_CREATE_FAILURES_TOTAL, METRIC_OBJECT_STORE_WAL_CREATED_OBJECTS_TOTAL, METRIC_OBJECT_STORE_WAL_DELETED_OBJECTS_TOTAL, METRIC_OBJECT_STORE_WAL_FAILED_DELETES_TOTAL, METRIC_OBJECT_STORE_WAL_FAILING, METRIC_OBJECT_STORE_WAL_INDEXED_BYTES, - METRIC_OBJECT_STORE_WAL_INDEXED_OBJECTS, + METRIC_OBJECT_STORE_WAL_INDEXED_OBJECTS, METRIC_OBJECT_STORE_WAL_IO_PERMISSION_UNAVAILABLE, METRIC_OBJECT_STORE_WAL_MISSING_HISTORY_ACCEPTED_TOTAL, METRIC_OBJECT_STORE_WAL_OBJECT_BYTES, METRIC_OBJECT_STORE_WAL_OBJECT_ENTRIES, METRIC_OBJECT_STORE_WAL_POISONED_TOTAL, METRIC_OBJECT_STORE_WAL_READ_SECONDS, METRIC_OBJECT_STORE_WAL_RECOVERED_OBJECTS_TOTAL, @@ -155,6 +157,9 @@ pub struct ObjectStoreLogStore { read_prefetch_budget: Arc, terminal_error: TerminalError, stopped: Arc, + /// Permission for new object-store creates and deletes. It is absent for + /// standalone stores and installed by a metasrv-controlled datanode. + io_permit: IoPermitSlot, command_tx: mpsc::Sender, append_tx: mpsc::Sender, #[cfg(any(test, feature = "testing"))] @@ -180,6 +185,7 @@ type ObsoleteEntryIds = Arc>>; type StartedRegions = Arc>>; type WalHoles = Arc>>>; +type IoPermitSlot = Arc>>; /// A segment a read skipped because it did not decode: the entries of one /// region in one object are missing from every read of that region. @@ -330,7 +336,32 @@ impl ObjectStoreLogStore { unindexed, released_entry_ids, recovered_objects, + skipped_epochs, } = recover(io.as_ref()).await?; + if let Some(tip) = tip + && tip.epoch < max_epoch + { + let skipped = skipped_epochs + .iter() + .map(|(epoch, objects)| { + format!( + "epoch {epoch}: {} objects, sequences {}..={}", + objects.objects, objects.first_object_seq, objects.last_object_seq + ) + }) + .collect::>() + .join("; "); + let reason = format!( + "WAL chain in {prefix} would fall back from epoch {max_epoch} to {}; skipped objects after the selected tip: {skipped}. Recover the missing objects, or set accept_missing_history only after all previous writers have exited and acknowledged unflushed data loss is accepted", + tip.epoch, + ); + if !config.accept_missing_history { + return CorruptedWalObjectSnafu { reason }.fail(); + } + warn!( + "{reason}; accepting the fallback because accept_missing_history is enabled; acknowledged unflushed data may be lost" + ); + } // Every epoch is one above the sequence of the start object its // instance created, and every object is at or above its start object, // so an epoch above the next sequence names no instance that ran. @@ -389,6 +420,7 @@ impl ObjectStoreLogStore { let deleting = Arc::new(DeletingObjects::default()); let terminal_error = TerminalError::default(); let stopped = Arc::new(AtomicBool::new(false)); + let io_permit = Arc::new(OnceLock::new()); let (command_tx, command_rx) = mpsc::channel(COMMAND_BUFFER); let (append_tx, append_rx) = mpsc::channel(APPEND_BUFFER); #[cfg(any(test, feature = "testing"))] @@ -414,6 +446,9 @@ impl ObjectStoreLogStore { wal_holes: wal_holes.clone(), terminal_error: terminal_error.clone(), stopped: stopped.clone(), + io_permit: io_permit.clone(), + io_blocked: false, + io_permission_unavailable: false, command_rx, append_rx, ack_mode: config.ack_mode, @@ -486,6 +521,7 @@ impl ObjectStoreLogStore { read_prefetch_budget: Arc::new(Semaphore::new(READ_PREFETCH_BYTES)), terminal_error, stopped, + io_permit, command_tx, append_tx, #[cfg(any(test, feature = "testing"))] @@ -518,6 +554,18 @@ impl ObjectStoreLogStore { catalog.region_max_entry_id(region_id).unwrap_or(0) } + /// Fences new object-store creates and deletes with `permit`. + /// + /// The current in-flight operation is not revoked. Supplying a permit is + /// a cluster integration concern; standalone stores keep the default + /// unrestricted behavior. + /// Installation is allowed once; renewals update the installed permit itself. + pub fn set_io_permit(&self, permit: Arc) -> Result<()> { + self.io_permit + .set(permit) + .map_err(|_| crate::error::WalIoPermitAlreadyInstalledSnafu.build()) + } + /// Returns the segments of the provider's region that reads of this store /// skipped because they did not decode, ordered by object sequence. Holes /// are kept in memory only: a store opened later on the same prefix @@ -731,6 +779,7 @@ impl ObjectStoreLogStore { /// Seals the open batch regardless of its size and age and returns once /// its object is durable and indexed, or with the error that failed it. + /// An empty batch does not wait for previously sealed objects. pub async fn seal_open_batch(&self) -> Result<()> { ensure!( !self.stopped.load(Ordering::Acquire), @@ -879,9 +928,13 @@ impl LogStore for ObjectStoreLogStore { .flush_candidates(FLUSH_OBJECT_THRESHOLD) } - /// Stops the store. Creates in flight run to completion and acknowledge - /// their entries if they succeed; in the `enqueued` mode the remaining + /// Stops the store. While the I/O permit remains valid, creates in flight + /// run to completion and acknowledge their entries if they succeed; in + /// the `enqueued` mode the remaining /// backlog is uploaded first, and a failure of that upload is returned. + /// If permission expires, unindexed batches are dropped without waiting + /// for uploads; discarded enqueued data or release records report an error. + /// Issued deletes are still awaited. async fn stop(&self) -> Result<()> { self.stopped.store(true, Ordering::Release); let (response_tx, response_rx) = oneshot::channel(); @@ -1494,7 +1547,9 @@ impl Failing { /// nothing is admitted and no create starts, except that the `enqueued` mode /// uploads its backlog and the `durable` mode creates one object with the /// pending bounded releases and release records of failed batches. Creates in -/// flight run to completion and acknowledge if they succeed. +/// flight run to completion and acknowledge if they succeed, while the I/O +/// permit remains valid. On expiry during stop, unindexed batches are dropped +/// without waiting for uploads; issued deletes are still awaited. /// /// Appends arrive on their own channel. The actor holds one back while the /// sealed batches leave no room for the two an admission can seal, or at a @@ -1522,6 +1577,11 @@ struct Actor { wal_holes: WalHoles, terminal_error: TerminalError, stopped: Arc, + io_permit: IoPermitSlot, + /// Work that must be retried on a timer tick after permission returns. + io_blocked: bool, + /// Whether this actor is counted as lacking I/O permission, even when idle. + io_permission_unavailable: bool, command_rx: mpsc::Receiver, append_rx: mpsc::Receiver, ack_mode: AckMode, @@ -1607,10 +1667,20 @@ impl Actor { let probe_at = self.failing.as_ref().and_then(|failing| failing.retry.at); tokio::select! { _ = interval.tick() => { - // Nothing starts after stop began; the `enqueued` mode - // sealed its backlog when stop was requested. - if !self.is_stopped() { - self.flush_open_batch(); + if !self.io_allowed() { + self.expire_io_permission(); + } else { + if std::mem::take(&mut self.io_permission_unavailable) { + METRIC_OBJECT_STORE_WAL_IO_PERMISSION_UNAVAILABLE.dec(); + info!("Object store WAL I/O permission available; {} sealed batches retained", self.sealed.len()); + } + if !self.is_stopped() { + self.flush_open_batch(); + } + if std::mem::take(&mut self.io_blocked) { + self.start_creates(); + self.start_deletes(); + } } } Some((object_seq, result)) = self.creates.next(), if !self.creates.is_empty() => { @@ -1684,6 +1754,10 @@ impl Actor { return; } if self.should_hold_back() { + if !self.io_allowed() { + let _ = response.send(Err(ObjectStoreWalLeaseExpiredSnafu.build())); + return; + } METRIC_OBJECT_STORE_WAL_STALLED_APPENDS_TOTAL.inc(); self.stalled = Some(((entries, response), Instant::now())); self.ensure_create_in_flight(); @@ -1822,7 +1896,8 @@ impl Actor { /// Seals the open batch together with `releases` and pending release /// records as the object `next_object_seq`, even when the open /// batch holds no entry, and starts its create when a slot is free. - /// Returns whether a batch was sealed. + /// Returns whether a batch was sealed. An invalid I/O permit keeps + /// release records pending and applies the lease-expiry policy to entries. /// /// A failing store or one at a backlog threshold keeps pending releases /// coalesced instead of sealing them alone on every tick. Stop seals them @@ -1832,6 +1907,13 @@ impl Actor { /// Ordinary entries bypass this delay; indexing a batch carrying releases /// resets it. fn seal(&mut self, releases: Vec) -> bool { + if !self.io_allowed() { + for release in releases { + self.keep_release(release); + } + self.expire_io_permission(); + return false; + } if self.open_batch.is_empty() && releases.is_empty() && (self.pending_releases.is_empty() @@ -1934,6 +2016,10 @@ impl Actor { /// next attempt once nothing is in flight, see [`Failing`]; after stop /// began it uploads its backlog at once. fn start_creates(&mut self) { + if !self.io_allowed() { + self.io_blocked = true; + return; + } if self.is_stopped() && self.ack_mode == AckMode::Durable && self.stop.is_empty() { return; } @@ -1974,6 +2060,10 @@ impl Actor { if self.is_stopped() { return; } + if !self.io_allowed() { + self.io_blocked = true; + return; + } if let Some(index) = self .sealed .iter() @@ -1995,6 +2085,7 @@ impl Actor { self.creating.insert(object_seq); let epoch = self.epoch; let ack_mode = self.ack_mode; + let io_permit = self.io_permit.clone(); #[cfg(any(test, feature = "testing"))] let mut creates_held = self.creates_held.clone(); #[cfg(any(test, feature = "testing"))] @@ -2023,6 +2114,9 @@ impl Actor { record_create(&result, Some(sealed_at)); return (object_seq, result); } + if !io_allowed(&io_permit) { + return (object_seq, Err(ObjectStoreWalLeaseExpiredSnafu.build())); + } let created = io.put_if_absent(object_seq, bytes).await; #[cfg(any(test, feature = "testing"))] let created = match created { @@ -2048,6 +2142,9 @@ impl Actor { fn on_create_completed(&mut self, object_seq: u64, result: Result) { self.creating.remove(&object_seq); + if !self.io_allowed() { + self.expire_io_permission(); + } // A batch that already failed, or that the store gave up on when it // poisoned itself: the object may exist, but it is off the chain, and // collection may delete it now. @@ -2061,6 +2158,13 @@ impl Actor { return; }; match result { + Err(Error::ObjectStoreWalLeaseExpired { .. }) => { + // No request was issued, so this is not a storage retry. + self.sealed[index].attempts -= 1; + self.sealed[index].state = CreateState::Pending; + self.io_blocked = true; + return; + } Ok(_) => { self.sealed[index].state = CreateState::Created; self.leave_failing("a create succeeded"); @@ -2274,6 +2378,7 @@ impl Actor { /// the next release-only seal. Failures are logged at a limited rate. fn roll_back(&mut self, error: Arc) { let stopped = self.is_stopped(); + let backlog_dropped = !self.open_batch.is_empty() || !self.sealed.is_empty(); let failure = || { if stopped { ObjectStoreWalStoppedSnafu.build() @@ -2286,7 +2391,13 @@ impl Actor { self.keep_releases(&mut batch); batch.fail(failure); } - if self.ack_mode == AckMode::Durable && !stopped && !self.pending_releases.is_empty() { + // Lease checks can repeat without attempting storage I/O. They must not + // extend the retry delay of retained releases on every expired tick. + if self.ack_mode == AckMode::Durable + && !stopped + && !self.pending_releases.is_empty() + && !matches!(error.as_ref(), Error::ObjectStoreWalLeaseExpired { .. }) + { let now = Instant::now(); if self .release_retry_logged_at @@ -2309,9 +2420,12 @@ impl Actor { // An acknowledged backlog was dropped: `stop` reports it, whether // its caller has arrived yet or not. So it does release records once // stop began, which no later create carries. - if self.ack_mode == AckMode::Enqueued + if (self.ack_mode == AckMode::Enqueued && backlog_dropped) || (!self.stop.is_empty() && !self.pending_releases.is_empty()) { + if stopped && self.stop_error.is_none() { + warn!(error; "Object store WAL discarded unpersisted backlog during stop"); + } self.stop_error.get_or_insert(error); } self.leave_failing("its backlog was dropped"); @@ -2616,9 +2730,11 @@ impl Actor { batch.release_waiters.push(response); return; } - // Sealing failed and poisoned the store. + // Sealing either poisoned the store or was held back by its lease. if let Some(error) = terminal(&self.terminal_error) { let _ = response.send(Err(shared(&error))); + } else { + let _ = response.send(Err(ObjectStoreWalLeaseExpiredSnafu.build())); } } @@ -2647,6 +2763,10 @@ impl Actor { if self.is_stopped() || terminal(&self.terminal_error).is_some() { return; } + if !self.io_allowed() { + self.io_blocked = true; + return; + } let free = MAX_IN_FLIGHT_DELETES.saturating_sub(self.deletes.len()); let boundary = self.durable_reclaim_boundary; let candidates = { @@ -2667,7 +2787,13 @@ impl Actor { for object_seq in candidates { self.deleting.start(object_seq); let io = self.io.clone(); + let permit = self.io_permit.clone(); + // Pushing does not poll the future; the lease may expire before + // the actor first polls it, even without an await inside the body. self.deletes.push(Box::pin(async move { + if !io_allowed(&permit) { + return (object_seq, Err(ObjectStoreWalLeaseExpiredSnafu.build())); + } (object_seq, io.delete(object_seq).await) })); } @@ -2727,6 +2853,7 @@ impl Actor { } METRIC_OBJECT_STORE_WAL_DELETED_OBJECTS_TOTAL.inc(); } + Err(Error::ObjectStoreWalLeaseExpired { .. }) => self.io_blocked = true, Err(error) => { warn!( error; @@ -2801,6 +2928,10 @@ impl Actor { /// [`finish_stop`](Self::finish_stop) once nothing is in flight. fn handle_stop(&mut self, response: oneshot::Sender>) { self.stop.push(response); + if !self.io_allowed() { + self.expire_io_permission(); + return; + } if let Some((_, response)) = self.take_stalled() { let _ = response.send(Err(ObjectStoreWalStoppedSnafu.build())); } @@ -2860,6 +2991,11 @@ impl Actor { let _ = response.send(Err(ObjectStoreWalStoppedSnafu.build())); return; } + if !self.io_allowed() { + self.expire_io_permission(); + let _ = response.send(Err(ObjectStoreWalLeaseExpiredSnafu.build())); + return; + } if self.flush_open_batch() && let Some(batch) = self.sealed.back_mut() { @@ -2888,6 +3024,59 @@ impl Actor { fn is_stopped(&self) -> bool { self.stopped.load(Ordering::Acquire) } + + /// Releases waiters while preserving the acknowledged enqueued backlog, + /// unless stop began: then the backlog is dropped and `stop_error` reports it. + /// Durable batches abandoned here must never be indexed by late completions. + fn expire_io_permission(&mut self) { + if !std::mem::replace(&mut self.io_permission_unavailable, true) { + METRIC_OBJECT_STORE_WAL_IO_PERMISSION_UNAVAILABLE.inc(); + info!( + "Object store WAL I/O permission unavailable; {} sealed batches, open batch empty: {}, {} pending releases", + self.sealed.len(), + self.open_batch.is_empty(), + self.pending_releases.len() + ); + } + self.io_blocked = true; + let stopped = self.is_stopped(); + if stopped || self.ack_mode == AckMode::Durable { + if !self.open_batch.is_empty() { + self.next_object_seq = self + .next_object_seq + .and_then(|seq| seq.checked_add(1)) + .filter(|seq| *seq < OBJECT_SEQ_LIMIT); + } + self.roll_back(Arc::new(ObjectStoreWalLeaseExpiredSnafu.build())); + } else { + for batch in &mut self.sealed { + for waiter in batch.release_waiters.drain(..) { + let _ = waiter.send(Err(ObjectStoreWalLeaseExpiredSnafu.build())); + } + } + } + self.fail_unacknowledged(|| { + if stopped { + ObjectStoreWalStoppedSnafu.build() + } else { + ObjectStoreWalLeaseExpiredSnafu.build() + } + }); + if stopped { + self.creates.clear(); + self.creating.clear(); + } + } + + fn io_allowed(&self) -> bool { + io_allowed(&self.io_permit) + } +} + +fn io_allowed(permit: &IoPermitSlot) -> bool { + permit + .get() + .is_none_or(|permit| permit.allows_object_store_io()) } impl Drop for Actor { @@ -2898,6 +3087,9 @@ impl Drop for Actor { self.deleting.abandon_all(); self.take_stalled(); self.leave_failing("its actor exited"); + if self.io_permission_unavailable { + METRIC_OBJECT_STORE_WAL_IO_PERMISSION_UNAVAILABLE.dec(); + } #[cfg(test)] LIVE_ACTORS.fetch_sub(1, Ordering::SeqCst); } @@ -3107,6 +3299,17 @@ struct Recovered { released_entry_ids: HashMap, /// Number of objects whose header and footer were read. recovered_objects: u64, + /// Objects after the selected tip in (epoch, object sequence) order, + /// grouped by epoch, including incomplete branches in the selected epoch. + skipped_epochs: BTreeMap, +} + +/// Listed objects in one epoch after the selected recovery tip. +#[derive(Debug)] +struct SkippedEpoch { + first_object_seq: u64, + last_object_seq: u64, + objects: usize, } /// Rebuilds the catalog from object headers and footers, @@ -3227,6 +3430,20 @@ fn finish_recovery(fetched: FetchedObjects) -> Result { .map(|header| header.epoch) .max() .unwrap_or(0); + let mut skipped_epochs = BTreeMap::::new(); + if let Some(tip) = tip { + for (&seq, header) in &headers { + if header.epoch > tip.epoch || (header.epoch == tip.epoch && seq > tip.object_seq) { + let skipped = skipped_epochs.entry(header.epoch).or_insert(SkippedEpoch { + first_object_seq: seq, + last_object_seq: seq, + objects: 0, + }); + skipped.last_object_seq = seq; + skipped.objects += 1; + } + } + } let last_listed = headers .keys() .copied() @@ -3271,6 +3488,7 @@ fn finish_recovery(fetched: FetchedObjects) -> Result { unindexed, released_entry_ids: HashMap::new(), recovered_objects, + skipped_epochs, }) } @@ -3455,6 +3673,368 @@ mod tests { config(Duration::from_secs(3600), u64::MAX) } + #[derive(Default)] + struct TestIoPermit(AtomicBool); + + impl ObjectStoreWalIoPermit for TestIoPermit { + fn allows_object_store_io(&self) -> bool { + self.0.load(Ordering::Acquire) + } + } + + #[tokio::test] + async fn test_node_lease_permission_metric_lifecycle() { + let _serialized = SERIALIZED.lock().await; + wait_until(|| LIVE_ACTORS.load(Ordering::SeqCst) == 0).await; + let unavailable = || METRIC_OBJECT_STORE_WAL_IO_PERMISSION_UNAVAILABLE.get(); + assert_eq!(0, unavailable()); + let config = config(MIN_FLUSH_INTERVAL, u64::MAX); + for config in [config.clone(), enqueued(config)] { + for stop in [false, true] { + let store = open(memory_store(), &config).await; + assert_eq!(0, unavailable()); + // The first heartbeat has not granted permission yet. This + // counts an idle store too, without calling the lease expired. + let permit = Arc::new(TestIoPermit::default()); + store.set_io_permit(permit.clone()).unwrap(); + wait_until(|| unavailable() == 1).await; + for _ in 0..3 { + assert!(store.seal_open_batch().await.is_err()); + assert_eq!(1, unavailable()); + } + permit.0.store(true, Ordering::Release); + wait_until(|| unavailable() == 0).await; + // Expiry after a renewal has the same permission contract. + permit.0.store(false, Ordering::Release); + wait_until(|| unavailable() == 1).await; + if stop { + store.stop().await.unwrap(); + } + drop(store); + wait_until(|| LIVE_ACTORS.load(Ordering::SeqCst) == 0).await; + assert_eq!(0, unavailable()); + } + } + } + + #[tokio::test] + async fn test_node_lease_retains_enqueued_entries_until_renewed() { + let _serialized = SERIALIZED.lock().await; + let store = open(memory_store(), &enqueued(manual())).await; + let permit = Arc::new(TestIoPermit::default()); + store.set_io_permit(permit.clone()).unwrap(); + let admitted = append(&store, region(1), "retained").await.unwrap(); + assert_eq!(id(1, 1), admitted.last_entry_ids[®ion(1)]); + assert!(matches!( + store.seal_open_batch().await, + Err(Error::ObjectStoreWalLeaseExpired { .. }) + )); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + assert_eq!(0, latest(&store, region(1))); + permit.0.store(true, Ordering::Release); + store.seal_open_batch().await.unwrap(); + assert_eq!(id(1, 1), latest(&store, region(1))); + assert_eq!( + expected_entries(region(1), &[(id(1, 1), "retained")]), + read_entries(&store, region(1), 0).await + ); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_rejects_durable_batch_without_reusing_ids() { + let _serialized = SERIALIZED.lock().await; + let store = open(memory_store(), &manual()).await; + let permit = Arc::new(TestIoPermit::default()); + store.set_io_permit(permit.clone()).unwrap(); + let first = spawn_append_batch(&store, vec![entry(&store, region(1), "expired")]); + store.wait_for_admitted_appends(1).await.unwrap(); + assert!(store.seal_open_batch().await.is_err()); + let error = timeout(WAIT, first).await.unwrap().unwrap().unwrap_err(); + assert_eq!(RetryHint::RetryAfterRouteRefresh, error.retry_hint()); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + permit.0.store(true, Ordering::Release); + let next = spawn_append_batch(&store, vec![entry(&store, region(1), "new")]); + store.wait_for_admitted_appends(2).await.unwrap(); + store.seal_open_batch().await.unwrap(); + assert_eq!( + id(2, 1), + next.await.unwrap().unwrap().last_entry_ids[®ion(1)] + ); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_empty_stop_and_single_permit_installation() { + let _serialized = SERIALIZED.lock().await; + for config in [manual(), enqueued(manual())] { + let store = open(memory_store(), &config).await; + store + .set_io_permit(Arc::new(TestIoPermit::default())) + .unwrap(); + let error = store + .set_io_permit(Arc::new(TestIoPermit(AtomicBool::new(true)))) + .unwrap_err(); + assert!(matches!(error, Error::WalIoPermitAlreadyInstalled { .. })); + let error = store.seal_open_batch().await.unwrap_err(); + assert_eq!(StatusCode::RegionNotReady, error.status_code()); + assert_eq!(RetryHint::RetryAfterRouteRefresh, error.retry_hint()); + store.stop().await.unwrap(); + } + } + + #[tokio::test] + async fn test_node_lease_denied_create_preserves_attempt_count() { + let _serialized = SERIALIZED.lock().await; + for previous_attempts in [0, 2] { + let store = open(memory_store(), &enqueued(eager())).await; + let permit = Arc::new(TestIoPermit(AtomicBool::new(true))); + store.set_io_permit(permit.clone()).unwrap(); + store.hold_creates(); + append(&store, region(1), "queued").await.unwrap(); + timeout(WAIT, store.wait_for_parked_creates(1)) + .await + .unwrap() + .unwrap(); + inspect_actor(&store, move |actor| { + // Model earlier issued attempts, then the current queued attempt. + actor.sealed[0].attempts = previous_attempts + 1; + }) + .await; + permit.0.store(false, Ordering::Release); + store.release_creates(); + timeout(WAIT, async { + loop { + let pending = inspect_actor(&store, move |actor| { + let batch = &actor.sealed[0]; + if matches!(batch.state, CreateState::Pending) { + assert_eq!(previous_attempts, batch.attempts); + true + } else { + false + } + }) + .await; + if pending { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + permit.0.store(true, Ordering::Release); + inspect_actor(&store, |actor| actor.start_creates()).await; + store + .wait_durable(&provider(region(1)), id(1, 1)) + .await + .unwrap(); + store.stop().await.unwrap(); + } + } + + #[tokio::test] + async fn test_node_lease_rechecks_before_polling_delete() { + let _serialized = SERIALIZED.lock().await; + let store = open(memory_store(), &enqueued(manual())).await; + let permit = Arc::new(TestIoPermit(AtomicBool::new(true))); + store.set_io_permit(permit.clone()).unwrap(); + let pending_permit = permit.clone(); + inspect_actor(&store, move |actor| { + // Queue collection of the empty start object, then expire the + // permit before returning control to the actor's select loop. + actor.durable_reclaim_boundary = 1; + actor.start_deletes(); + assert_eq!(1, actor.deletes.len()); + pending_permit.0.store(false, Ordering::Release); + }) + .await; + timeout(WAIT, async { + while !inspect_actor(&store, |actor| actor.deletes.is_empty()).await { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + inspect_actor(&store, |actor| { + assert!(actor.io_blocked); + actor.durable_reclaim_boundary = 0; + }) + .await; + permit.0.store(true, Ordering::Release); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_stopped_probe_does_not_mark_io_blocked() { + let _serialized = SERIALIZED.lock().await; + let store = open(memory_store(), &enqueued(manual())).await; + inspect_actor(&store, |actor| { + actor.enter_failing(ObjectStoreWalStoppedSnafu.build()); + actor.stopped.store(true, Ordering::Release); + actor.probe(); + assert!(!actor.io_blocked); + actor.stopped.store(false, Ordering::Release); + actor.leave_failing("test finished"); + }) + .await; + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_stop_drops_unpersisted_backlog() { + let _serialized = SERIALIZED.lock().await; + let object_store = memory_store(); + let store = open(object_store.clone(), &enqueued(manual())).await; + append(&store, region(1), "queued").await.unwrap(); + store + .set_io_permit(Arc::new(TestIoPermit::default())) + .unwrap(); + assert!(timeout(WAIT, store.stop()).await.unwrap().is_err()); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + let reopened = open(object_store, &eager()).await; + assert!(read_entries(&reopened, region(1), 0).await.is_empty()); + reopened.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_keeps_release_for_renewal() { + let _serialized = SERIALIZED.lock().await; + let store = open(memory_store(), &manual()).await; + let permit = Arc::new(TestIoPermit::default()); + store.set_io_permit(permit.clone()).unwrap(); + let error = store + .release_region(&provider(region(1)), region(1)) + .await + .unwrap_err(); + assert!(matches!(error, Error::ObjectStoreWalLeaseExpired { .. })); + // Repeated expiry checks must not accumulate storage retry backoff: + // renewal can seal the retained record immediately. + for _ in 0..8 { + assert!(matches!( + store.seal_open_batch().await, + Err(Error::ObjectStoreWalLeaseExpired { .. }) + )); + } + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + permit.0.store(true, Ordering::Release); + store.seal_open_batch().await.unwrap(); + assert_eq!( + vec![release(region(1), 0)], + releases_of(store.io.as_ref(), 1).await + ); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_discards_late_durable_upload() { + let _serialized = SERIALIZED.lock().await; + let (store, io, mut parked) = open_parking_creates(memory_store(), &eager()).await; + let permit = Arc::new(TestIoPermit(AtomicBool::new(true))); + store.set_io_permit(permit.clone()).unwrap(); + let first = spawn_append_batch(&store, vec![entry(&store, region(1), "late")]); + let (seq, release) = next_parked_operation(&mut parked).await; + assert_eq!(1, seq); + permit.0.store(false, Ordering::Release); + assert!(store.seal_open_batch().await.is_err()); + assert!( + first + .await + .unwrap() + .unwrap_err() + .retry_hint() + .is_retryable() + ); + // The issued create is allowed to finish, even after renewal. Its batch + // was already failed and must not become part of the live chain. + permit.0.store(true, Ordering::Release); + release.send(true).unwrap(); + io.creates_parked.store(false, Ordering::SeqCst); + let next = append(&store, region(1), "current").await.unwrap(); + assert_eq!(id(2, 1), next.last_entry_ids[®ion(1)]); + assert_eq!( + expected_entries(region(1), &[(id(2, 1), "current")]), + read_entries(&store, region(1), 0).await + ); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_rejects_backpressure_without_waiting() { + let _serialized = SERIALIZED.lock().await; + let config = ObjectStoreWalConfig { + max_unpersisted_bytes: ReadableSize(1), + ..enqueued(manual()) + }; + let store = open(memory_store(), &config).await; + store + .set_io_permit(Arc::new(TestIoPermit::default())) + .unwrap(); + append(&store, region(1), "queued").await.unwrap(); + let error = timeout(WAIT, append(&store, region(1), "blocked")) + .await + .unwrap() + .unwrap_err(); + assert!(matches!(error, Error::ObjectStoreWalLeaseExpired { .. })); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + assert!(store.stop().await.is_err()); + } + + #[tokio::test] + async fn test_node_lease_expires_during_stop_without_waiting_for_upload() { + let _serialized = SERIALIZED.lock().await; + let config = enqueued(config(MIN_FLUSH_INTERVAL, 1)); + let (store, _, mut parked) = open_parking_creates(memory_store(), &config).await; + let permit = Arc::new(TestIoPermit(AtomicBool::new(true))); + store.set_io_permit(permit.clone()).unwrap(); + append(&store, region(1), "queued").await.unwrap(); + let (_, release) = next_parked_operation(&mut parked).await; + let stop = { + let store = store.clone(); + tokio::spawn(async move { store.stop().await }) + }; + wait_until(|| store.stopped.load(Ordering::Acquire)).await; + permit.0.store(false, Ordering::Release); + assert!(timeout(WAIT, stop).await.unwrap().unwrap().is_err()); + assert!(release.is_closed()); + } + + #[tokio::test] + async fn test_node_lease_defers_collection_and_stop_waits_for_issued_delete() { + let _serialized = SERIALIZED.lock().await; + let config = enqueued(config(MIN_FLUSH_INTERVAL, 1)); + let (store, io, mut parked) = open_parking_creates(memory_store(), &config).await; + let permit = Arc::new(TestIoPermit(AtomicBool::new(true))); + store.set_io_permit(permit.clone()).unwrap(); + io.deletes_parked.store(true, Ordering::SeqCst); + append(&store, region(1), "first").await.unwrap(); + let (seq, release) = next_parked_operation(&mut parked).await; + assert_eq!(1, seq); + permit.0.store(false, Ordering::Release); + release.send(true).unwrap(); + // An issued enqueued create may finish, but its advanced reclaim + // boundary must not issue a delete under the expired lease. + wait_until(|| latest(&store, region(1)) == id(1, 1)).await; + assert_nothing_parks(&mut parked).await; + assert_eq!(vec![0, 1], object_seqs(io.as_ref()).await); + permit.0.store(true, Ordering::Release); + let (seq, release) = next_parked_operation(&mut parked).await; + assert_eq!(0, seq); + permit.0.store(false, Ordering::Release); + let stop = { + let store = store.clone(); + tokio::spawn(async move { store.stop().await }) + }; + wait_until(|| store.stopped.load(Ordering::Acquire)).await; + assert!(!stop.is_finished()); + release.send(true).unwrap(); + // All entries were indexed: expiry alone is not a shutdown failure. + timeout(WAIT, stop).await.unwrap().unwrap().unwrap(); + assert_eq!(vec![1], object_seqs(io.as_ref()).await); + } + async fn open( object_store: ObjectStore, config: &ObjectStoreWalConfig, @@ -5074,6 +5654,7 @@ mod tests { read_prefetch_budget: Arc::new(Semaphore::new(READ_PREFETCH_BYTES)), terminal_error: Arc::default(), stopped: Arc::new(AtomicBool::new(false)), + io_permit: Arc::default(), command_tx, append_tx, admitted_appends: watch::channel(0).1, @@ -7404,6 +7985,84 @@ mod tests { } } + #[tokio::test] + async fn test_recovery_rejects_older_epoch_until_explicitly_accepted() { + let _serialized = SERIALIZED.lock().await; + let object_store = memory_store(); + let io = ObjectStoreIo::new(object_store.clone(), PREFIX).unwrap(); + put_object_with_header(&io, chain_header(0, 1, None), &[]).await; + // Both branches have missing predecessors; epoch 1 remains complete + // only through object 0. + put_object_with_header(&io, chain_header(2, 1, Some((1, 1))), &[]).await; + put_object_with_header(&io, chain_header(3, 3, Some((2, 3))), &[]).await; + let before = io.list().await.unwrap(); + for _ in 0..2 { + let error = ObjectStoreLogStore::try_new(object_store.clone(), &eager(), 1, 2) + .await + .unwrap_err(); + assert!(matches!(error, Error::CorruptedWalObject { reason, .. } + if reason.contains("fall back from epoch 3 to 1") + && reason.contains("epoch 1: 1 objects, sequences 2..=2") + && reason.contains("epoch 3: 1 objects, sequences 3..=3"))); + assert_eq!(before, io.list().await.unwrap()); + } + let accepted = ObjectStoreWalConfig { + accept_missing_history: true, + ..eager() + }; + let store = open(object_store.clone(), &accepted).await; + store.stop().await.unwrap(); + let reopened = open(object_store, &eager()).await; + reopened.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_recovery_accepts_incomplete_branch_in_selected_epoch() { + let _serialized = SERIALIZED.lock().await; + let object_store = memory_store(); + let io = ObjectStoreIo::new(object_store.clone(), PREFIX).unwrap(); + put_object_with_header(&io, chain_header(0, 1, None), &[]).await; + // A failed upload of object 1 can leave a later, incomplete branch + // without evidence that a newer writer's complete history was lost. + put_object_with_header(&io, chain_header(2, 1, Some((1, 1))), &[]).await; + let store = open(object_store, &eager()).await; + let header = decode_header(&io.get(3).await.unwrap()).unwrap(); + assert_eq!(4, header.epoch); + assert_eq!( + Some(ChainLink { + object_seq: 0, + epoch: 1 + }), + header.prev + ); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_node_lease_rechecks_before_parked_upload() { + let _serialized = SERIALIZED.lock().await; + let store = open(memory_store(), &enqueued(config(MIN_FLUSH_INTERVAL, 1))).await; + let permit = Arc::new(TestIoPermit(AtomicBool::new(true))); + store.set_io_permit(permit.clone()).unwrap(); + store.hold_creates(); + append(&store, region(1), "queued").await.unwrap(); + timeout(WAIT, store.wait_for_parked_creates(1)) + .await + .unwrap() + .unwrap(); + permit.0.store(false, Ordering::Release); + store.release_creates(); + assert!(store.seal_open_batch().await.is_err()); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + permit.0.store(true, Ordering::Release); + timeout(WAIT, store.wait_durable(&provider(region(1)), id(1, 1))) + .await + .unwrap() + .unwrap(); + assert_eq!(id(1, 1), latest(&store, region(1))); + store.stop().await.unwrap(); + } + fn chain_of(headers: &[Header]) -> Vec { select_chain( &headers diff --git a/src/metric-engine/src/engine/drop.rs b/src/metric-engine/src/engine/drop.rs index f7db294478c..37868d4dfe6 100644 --- a/src/metric-engine/src/engine/drop.rs +++ b/src/metric-engine/src/engine/drop.rs @@ -189,7 +189,7 @@ impl MetricEngineInner { #[cfg(test)] mod tests { use api::v1::Rows; - use common_wal::config::object_store::{ObjectStoreWalConfig, STANDALONE_GENERATION}; + use common_wal::config::object_store::{FIXED_DIRECTORY_GENERATION, ObjectStoreWalConfig}; use common_wal::options::{ObjectStoreWalOptions, WAL_OPTIONS_KEY, WalOptions}; use log_store::ObjectStoreLogStore; use mito2::config::MitoConfig; @@ -238,7 +238,7 @@ mod tests { object_store.clone(), &config, 0, - STANDALONE_GENERATION, + FIXED_DIRECTORY_GENERATION, ) }; let wal = open_wal().await.unwrap(); diff --git a/src/mito2/src/engine.rs b/src/mito2/src/engine.rs index cc60af22f27..e257b046989 100644 --- a/src/mito2/src/engine.rs +++ b/src/mito2/src/engine.rs @@ -132,8 +132,8 @@ use snafu::{OptionExt, ResultExt, ensure}; use store_api::ManifestVersion; use store_api::codec::PrimaryKeyEncoding; use store_api::compaction_task::{self, CompactionTaskEntry}; -use store_api::logstore::LogStore; use store_api::logstore::provider::{KafkaProvider, Provider}; +use store_api::logstore::{LogStore, ObjectStoreWalIoPermit}; use store_api::metadata::{ColumnMetadata, RegionMetadataRef}; use store_api::metric_engine_consts::{ MANIFEST_INFO_EXTENSION_KEY, TABLE_COLUMN_METADATA_EXTENSION_KEY, @@ -346,6 +346,11 @@ impl MitoEngine { self.inner.workers.node_lease().enable(); } + /// Returns the node-lease permission used to fence object-store WAL I/O. + pub fn object_store_wal_io_permit(&self) -> Arc { + self.inner.workers.node_lease_permit() + } + /// Renews the lease of this node from a heartbeat response that carries region leases. pub fn renew_node_lease(&self, deadline: Instant) { self.inner.workers.node_lease().renew(deadline); diff --git a/src/mito2/src/engine/object_store_wal_recovery_test.rs b/src/mito2/src/engine/object_store_wal_recovery_test.rs index bd2cad053ee..8aaf938e296 100644 --- a/src/mito2/src/engine/object_store_wal_recovery_test.rs +++ b/src/mito2/src/engine/object_store_wal_recovery_test.rs @@ -31,7 +31,7 @@ use common_error::ext::{BoxedError, ErrorExt, RetryHint}; use common_error::status_code::StatusCode; use common_recordbatch::RecordBatches; use common_wal::config::object_store::{ - AckMode, CorruptedSegmentAction, ObjectStoreWalConfig, STANDALONE_GENERATION, + AckMode, CorruptedSegmentAction, FIXED_DIRECTORY_GENERATION, ObjectStoreWalConfig, }; use common_wal::options::{ObjectStoreWalOptions, WAL_OPTIONS_KEY, WalOptions}; use futures::TryStreamExt; @@ -99,7 +99,7 @@ async fn open_store(object_store: &ObjectStore, ack_mode: AckMode) -> Arc ObjectStoreWalConfig { /// The node prefix a standalone datanode derives from the configured root, /// which the regions persist in their WAL options. fn node_prefix() -> String { - wal_config().node_prefix(0, STANDALONE_GENERATION) + wal_config().node_prefix(0, FIXED_DIRECTORY_GENERATION) } pub(crate) fn wal_options() -> HashMap { @@ -624,12 +624,48 @@ fn held_wal_config() -> ObjectStoreWalConfig { } } +#[tokio::test] +async fn test_node_lease_fences_wal_and_renews_without_region_grants() { + let mut env = TestEnv::with_prefix("wal-node-lease").await; + let (engine, log_store) = env + .create_engine_with_object_store_wal(MitoConfig::default(), &held_wal_config()) + .await; + engine.require_write_lease(); + log_store + .set_io_permit(engine.object_store_wal_io_permit()) + .unwrap(); + let region_id = RegionId::new(1, 1); + let (_, schema, deadline) = create_leased_region_on(&engine, region_id).await; + engine.renew_node_lease(deadline); + put_range(&engine, region_id, &schema, 0, 3).await; + engine.freeze_lease_clock(deadline); + let error = log_store.seal_open_batch().await.unwrap_err(); + assert_eq!( + common_error::status_code::StatusCode::RegionNotReady, + error.status_code() + ); + assert_eq!( + common_error::ext::RetryHint::RetryAfterRouteRefresh, + error.retry_hint() + ); + assert!(stored_entry_ids(&log_store, region_id).await.is_empty()); + + // Node renewal, including an empty heartbeat grant list, unblocks WAL I/O + // independently of this region's expired Leader grant. + engine.renew_node_lease(deadline + Duration::from_secs(10)); + log_store.seal_open_batch().await.unwrap(); + assert_eq!(1, stored_entry_ids(&log_store, region_id).await.len()); + assert_eq!(3, count_rows(&engine, region_id).await); + log_store.stop().await.unwrap(); + engine.stop().await.unwrap(); +} + async fn new_log_store() -> Arc { ObjectStoreLogStore::try_new( ObjectStore::new(Memory::default()).unwrap(), &wal_config(), 0, - STANDALONE_GENERATION, + FIXED_DIRECTORY_GENERATION, ) .await .unwrap() diff --git a/src/mito2/src/engine/tail_replay_test.rs b/src/mito2/src/engine/tail_replay_test.rs index 66a77462208..9d2f18c3a8e 100644 --- a/src/mito2/src/engine/tail_replay_test.rs +++ b/src/mito2/src/engine/tail_replay_test.rs @@ -25,7 +25,7 @@ use common_base::readable_size::ReadableSize; use common_error::ext::ErrorExt; use common_error::status_code::StatusCode; use common_recordbatch::RecordBatches; -use common_wal::config::object_store::STANDALONE_GENERATION; +use common_wal::config::object_store::FIXED_DIRECTORY_GENERATION; use common_wal::options::WAL_OPTIONS_KEY; use datatypes::arrow::array::Array; use log_store::ObjectStoreLogStore; @@ -224,7 +224,7 @@ impl TestRegion { wal_store.clone(), &wal_config(), NODE_ID, - STANDALONE_GENERATION, + FIXED_DIRECTORY_GENERATION, ) .await .unwrap(); @@ -356,7 +356,7 @@ impl TestRegion { self.wal_store.clone(), &wal_config(), REMOTE_NODE_ID, - STANDALONE_GENERATION, + FIXED_DIRECTORY_GENERATION, ) .await .unwrap(); @@ -480,7 +480,7 @@ fn remote_tail(after: EntryId, upto: EntryId, upto_kind: UptoKind) -> PendingTai } fn remote_prefix() -> String { - wal_config().node_prefix(REMOTE_NODE_ID, STANDALONE_GENERATION) + wal_config().node_prefix(REMOTE_NODE_ID, FIXED_DIRECTORY_GENERATION) } #[tokio::test] @@ -624,7 +624,7 @@ async fn test_tail_read_stops_at_its_upto(#[case] upto_kind: UptoKind) { .reopen_with_tails(vec![own_tail(0, upto, upto_kind)]) .await; // Makes the entry after the tail unreadable. - let prefix = wal_config().node_prefix(NODE_ID, STANDALONE_GENERATION); + let prefix = wal_config().node_prefix(NODE_ID, FIXED_DIRECTORY_GENERATION); let objects = region .wal_store .list(&format!("{prefix}/objects/")) @@ -1535,10 +1535,14 @@ async fn test_replay_chunks_fire_sst_written_hook() { .await .with_node_id(NODE_ID); let wal_store = ObjectStore::new(Memory::default()).unwrap(); - let log_store = - ObjectStoreLogStore::try_new(wal_store, &wal_config(), NODE_ID, STANDALONE_GENERATION) - .await - .unwrap(); + let log_store = ObjectStoreLogStore::try_new( + wal_store, + &wal_config(), + NODE_ID, + FIXED_DIRECTORY_GENERATION, + ) + .await + .unwrap(); let hook = Arc::new(MockRegionHook::new()); let plugins = Plugins::new(); plugins.insert(hook.clone() as RegionHookRef); diff --git a/src/mito2/src/lease.rs b/src/mito2/src/lease.rs index 1544bd650ab..b6b7caaada9 100644 --- a/src/mito2/src/lease.rs +++ b/src/mito2/src/lease.rs @@ -17,6 +17,7 @@ use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{LazyLock, Mutex}; use std::time::Instant; +use store_api::logstore::ObjectStoreWalIoPermit; use store_api::storage::RegionId; /// The instant lease deadlines count from, so that a deadline fits in an [AtomicU64]. @@ -145,10 +146,36 @@ impl GrantedLeases { } } +impl ObjectStoreWalIoPermit for NodeLease { + fn allows_object_store_io(&self) -> bool { + !self.is_enabled() || self.now() < self.deadline.get() + } +} + #[cfg(test)] mod tests { + use std::time::Duration; + use super::*; + #[test] + fn test_object_store_permission_follows_node_lease() { + let lease = NodeLease::default(); + assert!(lease.allows_object_store_io()); + lease.enable(); + assert!(!lease.allows_object_store_io()); + let deadline = Instant::now() + Duration::from_secs(10); + lease.renew(deadline); + lease.freeze_clock(deadline - Duration::from_secs(1)); + assert!(lease.allows_object_store_io()); + lease.freeze_clock(deadline); + assert!(!lease.allows_object_store_io()); + lease.renew(deadline + Duration::from_secs(1)); + assert!(lease.allows_object_store_io()); + lease.renew(deadline - Duration::from_secs(1)); + assert!(lease.allows_object_store_io()); + } + #[test] fn test_record_checks_region_under_removal_lock() { let leases = GrantedLeases::default(); diff --git a/src/mito2/src/test_util.rs b/src/mito2/src/test_util.rs index 7af095b4ca6..04d56fb6499 100644 --- a/src/mito2/src/test_util.rs +++ b/src/mito2/src/test_util.rs @@ -42,7 +42,7 @@ use common_meta::kv_backend::KvBackendRef; use common_meta::kv_backend::memory::MemoryKvBackend; use common_telemetry::{debug, warn}; use common_test_util::temp_dir::{TempDir, create_temp_dir}; -use common_wal::config::object_store::{ObjectStoreWalConfig, STANDALONE_GENERATION}; +use common_wal::config::object_store::{FIXED_DIRECTORY_GENERATION, ObjectStoreWalConfig}; use common_wal::options::{KafkaWalOptions, WAL_OPTIONS_KEY, WalOptions}; use datatypes::arrow::array::{TimestampMillisecondArray, UInt8Array, UInt64Array}; use datatypes::extension::json::{Json2ExtensionType, JsonExtensionType}; @@ -683,10 +683,14 @@ impl TestEnv { wal_config: &ObjectStoreWalConfig, ) -> (MitoEngine, Arc) { let wal_object_store = ObjectStore::new(Memory::default()).unwrap(); - let log_store = - ObjectStoreLogStore::try_new(wal_object_store, wal_config, 0, STANDALONE_GENERATION) - .await - .unwrap(); + let log_store = ObjectStoreLogStore::try_new( + wal_object_store, + wal_config, + 0, + FIXED_DIRECTORY_GENERATION, + ) + .await + .unwrap(); let engine = self .create_engine_with_log_store(config, log_store.clone()) .await; diff --git a/src/mito2/src/worker.rs b/src/mito2/src/worker.rs index 89f779d503b..820596c62e7 100644 --- a/src/mito2/src/worker.rs +++ b/src/mito2/src/worker.rs @@ -441,6 +441,10 @@ impl WorkerGroup { handle_tail_replay::notify_writable(&self.worker(region.region_id).sender, region); } + pub(crate) fn node_lease_permit(&self) -> Arc { + self.node_lease.clone() + } + /// Asks the worker of `region` to cancel its tail replay, if one runs, without waiting. pub(crate) fn notify_cannot_replay(&self, region: &MitoRegionRef) { handle_tail_replay::notify_cannot_replay(&self.worker(region.region_id).sender, region); diff --git a/src/store-api/src/logstore.rs b/src/store-api/src/logstore.rs index 3716b914ef0..00f3f1a929f 100644 --- a/src/store-api/src/logstore.rs +++ b/src/store-api/src/logstore.rs @@ -30,6 +30,15 @@ pub use crate::logstore::entry::Id as EntryId; use crate::logstore::provider::Provider; use crate::storage::RegionId; +/// Allows a log store to initiate object-store I/O. +/// +/// A metasrv-controlled datanode supplies this from its node lease. The log +/// store deliberately does not know how the permission is renewed. +pub trait ObjectStoreWalIoPermit: Send + Sync + 'static { + /// Returns whether a new object-store create or delete may start now. + fn allows_object_store_io(&self) -> bool; +} + /// `LogStore` serves as a Write-Ahead-Log for storage engine. #[async_trait::async_trait] pub trait LogStore: Send + Sync + 'static + std::fmt::Debug {