rust/neon
1
0
Fork 0
mirror of https://github.com/neondatabase/neon.git synced 2026-01-13 08:22:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Run gc on too many partial backup segments (#7700)

Browse Source 
The general partial backup idea is that each safekeeper keeps only one
partial segment in remote storage at a time. Sometimes this is not true,
for example if we uploaded object to S3 but got an error when tried to
remove the previous upload. In this case we still keep a list of all
potentially uploaded objects in safekeeper state.

This commit prints a warning to logs if there is too many objects in
safekeeper state. This is not expected and we should try to fix this
state, we can do this by running gc.

I haven't seen this being an issue anywhere, but printing a warning is
something that I wanted to do and forgot in initial PR.
This commit is contained in:
Arthur Petukhovsky
2024-06-01 00:18:56 +01:00
committed by GitHub
parent 7e60563910
commit e98bc4fd2b
1 changed files with 17 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
safekeeper/src/wal_backup_partial.rs
View File

@@ -24,7 +24,7 @@ use rand::Rng;
use remote_storage::RemotePath;
use serde::{Deserialize, Serialize};
use tracing::{debug, error, info, instrument};
use tracing::{debug, error, info, instrument, warn};
use utils::lsn::Lsn;
use crate::{
@@ -308,7 +308,23 @@ pub async fn main_task(tli: FullAccessTimeline, conf: SafeKeeperConf) {
debug!("state: {:?}", backup.state);
// The general idea is that each safekeeper keeps only one partial segment
// both in remote storage and in local state. If this is not true, something
// went wrong.
const MAX_SIMULTANEOUS_SEGMENTS: usize = 10;
'outer: loop {
if backup.state.segments.len() > MAX_SIMULTANEOUS_SEGMENTS {
warn!(
"too many segments in control_file state, running gc: {}",
backup.state.segments.len()
);
backup.gc().await.unwrap_or_else(|e| {
error!("failed to run gc: {:#}", e);
});
}
// wait until we have something to upload
let uploaded_segment = backup.state.uploaded_segment();
if let Some(seg) = &uploaded_segment {