mirror of
https://github.com/neondatabase/neon.git
synced 2026-05-18 13:40:37 +00:00
bd5cb9e86b
In safekeepers we have several background tasks. Previously `WAL backup` task was spawned by another task called `wal_backup_launcher`. That task received notifications via `wal_backup_launcher_rx` and decided to spawn or kill existing backup task associated with the timeline. This was inconvenient because each code segment that touched shared state was responsible for pushing notification into `wal_backup_launcher_tx` channel. This was error prone because it's easy to miss and could lead to deadlock in some cases, if notification pushing was done in the wrong order. We also had a similar issue with `is_active` timeline flag. That flag was calculated based on the state and code modifying the state had to call function to update the flag. We had a few bugs related to that, when we forgot to update `is_active` flag in some places where it could change. To fix these issues, this PR adds a new `timeline_manager` background task associated with each timeline. This task is responsible for managing all background tasks, including `is_active` flag which is used for pushing broker messages. It is subscribed for updates in timeline state in a loop and decides to spawn/kill background tasks when needed. There is a new structure called `TimelinesSet`. It stores a set of `Arc<Timeline>` and allows to copy the set to iterate without holding the mutex. This is what replaced `is_active` flag for the broker. Now broker push task holds a reference to the `TimelinesSet` with active timelines and use it instead of iterating over all timelines and filtering by `is_active` flag. Also added some metrics for manager iterations and active backup tasks. Ideally manager should be doing not too many iterations and we should not have a lot of backup tasks spawned at the same time. Fixes #7751 --------- Co-authored-by: Arseny Sher <sher-ars@yandex.ru>
91 lines
2.5 KiB
Rust
91 lines
2.5 KiB
Rust
use std::{collections::HashMap, sync::Arc};
|
|
|
|
use utils::id::TenantTimelineId;
|
|
|
|
use crate::timeline::Timeline;
|
|
|
|
/// Set of timelines, supports operations:
|
|
/// - add timeline
|
|
/// - remove timeline
|
|
/// - clone the set
|
|
///
|
|
/// Usually used for keeping subset of timelines. For example active timelines that require broker push.
|
|
pub struct TimelinesSet {
|
|
timelines: std::sync::Mutex<HashMap<TenantTimelineId, Arc<Timeline>>>,
|
|
}
|
|
|
|
impl Default for TimelinesSet {
|
|
fn default() -> Self {
|
|
Self {
|
|
timelines: std::sync::Mutex::new(HashMap::new()),
|
|
}
|
|
}
|
|
}
|
|
|
|
impl TimelinesSet {
|
|
pub fn insert(&self, tli: Arc<Timeline>) {
|
|
self.timelines.lock().unwrap().insert(tli.ttid, tli);
|
|
}
|
|
|
|
pub fn delete(&self, ttid: &TenantTimelineId) {
|
|
self.timelines.lock().unwrap().remove(ttid);
|
|
}
|
|
|
|
/// If present is true, adds timeline to the set, otherwise removes it.
|
|
pub fn set_present(&self, tli: Arc<Timeline>, present: bool) {
|
|
if present {
|
|
self.insert(tli);
|
|
} else {
|
|
self.delete(&tli.ttid);
|
|
}
|
|
}
|
|
|
|
pub fn is_present(&self, ttid: &TenantTimelineId) -> bool {
|
|
self.timelines.lock().unwrap().contains_key(ttid)
|
|
}
|
|
|
|
/// Returns all timelines in the set.
|
|
pub fn get_all(&self) -> Vec<Arc<Timeline>> {
|
|
self.timelines.lock().unwrap().values().cloned().collect()
|
|
}
|
|
|
|
/// Returns a timeline guard for easy presence control.
|
|
pub fn guard(self: &Arc<Self>, tli: Arc<Timeline>) -> TimelineSetGuard {
|
|
let is_present = self.is_present(&tli.ttid);
|
|
TimelineSetGuard {
|
|
timelines_set: self.clone(),
|
|
tli,
|
|
is_present,
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Guard is used to add or remove timeline from the set.
|
|
/// If the timeline present in set, it will be removed from it on drop.
|
|
/// Note: do not use more than one guard for the same timeline, it caches the presence state.
|
|
/// It is designed to be used in the manager task only.
|
|
pub struct TimelineSetGuard {
|
|
timelines_set: Arc<TimelinesSet>,
|
|
tli: Arc<Timeline>,
|
|
is_present: bool,
|
|
}
|
|
|
|
impl TimelineSetGuard {
|
|
/// Returns true if the state was changed.
|
|
pub fn set(&mut self, present: bool) -> bool {
|
|
if present == self.is_present {
|
|
return false;
|
|
}
|
|
self.is_present = present;
|
|
self.timelines_set.set_present(self.tli.clone(), present);
|
|
true
|
|
}
|
|
}
|
|
|
|
impl Drop for TimelineSetGuard {
|
|
fn drop(&mut self) {
|
|
// remove timeline from the map on drop
|
|
self.timelines_set.delete(&self.tli.ttid);
|
|
}
|
|
}
|