mirror of
https://github.com/neondatabase/neon.git
synced 2026-01-13 00:12:54 +00:00
3a50d95b6d
## Problem Pageservers notify control plane directly when a shard import has completed. Control plane has to download the status of each shard from S3 and figure out if everything is truly done, before proceeding with branch activation. Issues with this approach are: * We can't control shard split behaviour on the storage controller side. It's unsafe to split during import. * Control plane needs to know about shards and implement logic to check all timelines are indeed ready. ## Summary of changes In short, storage controller coordinates imports, and, only when everything is done, notifies control plane. Big rocks: 1. Store timeline imports in the storage controller database. Each import stores the status of its shards in the database. We hook into the timeline creation call as our entry point for this. 2. Pageservers get a new upcall endpoint to notify the storage controller of shard import updates. 3. Storage controller handles these updates by updating persisted state. If an update finalizes the import, then poll pageservers until timeline activation, and, then, notify the control plane that the import is complete. Cplane side change with new endpoint is in https://github.com/neondatabase/cloud/pull/26166 Closes https://github.com/neondatabase/neon/issues/11566
64 lines
1.2 KiB
Rust
64 lines
1.2 KiB
Rust
use serde::Serialize;
|
|
use utils::seqwait::MonotonicCounter;
|
|
|
|
extern crate hyper0 as hyper;
|
|
|
|
mod auth;
|
|
mod background_node_operations;
|
|
mod compute_hook;
|
|
mod drain_utils;
|
|
mod heartbeater;
|
|
pub mod http;
|
|
mod id_lock_map;
|
|
mod leadership;
|
|
pub mod metrics;
|
|
mod node;
|
|
mod pageserver_client;
|
|
mod peer_client;
|
|
pub mod persistence;
|
|
mod reconciler;
|
|
mod safekeeper;
|
|
mod safekeeper_client;
|
|
mod scheduler;
|
|
mod schema;
|
|
pub mod service;
|
|
mod tenant_shard;
|
|
mod timeline_import;
|
|
|
|
#[derive(Ord, PartialOrd, Eq, PartialEq, Copy, Clone, Serialize)]
|
|
struct Sequence(u64);
|
|
|
|
impl Sequence {
|
|
fn initial() -> Self {
|
|
Self(0)
|
|
}
|
|
}
|
|
|
|
impl std::fmt::Display for Sequence {
|
|
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
|
|
write!(f, "{}", self.0)
|
|
}
|
|
}
|
|
|
|
impl std::fmt::Debug for Sequence {
|
|
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
|
|
write!(f, "{}", self.0)
|
|
}
|
|
}
|
|
|
|
impl MonotonicCounter<Sequence> for Sequence {
|
|
fn cnt_advance(&mut self, v: Sequence) {
|
|
assert!(*self <= v);
|
|
*self = v;
|
|
}
|
|
fn cnt_value(&self) -> Sequence {
|
|
*self
|
|
}
|
|
}
|
|
|
|
impl Sequence {
|
|
fn next(&self) -> Sequence {
|
|
Sequence(self.0 + 1)
|
|
}
|
|
}
|