Struct CompactionScheduler 

pub(crate) struct CompactionScheduler {
    scheduler: SchedulerRef,
    region_status: HashMap<RegionId, CompactionStatus>,
    request_sender: Sender<WorkerRequestWithTime>,
    cache_manager: CacheManagerRef,
    engine_config: Arc<MitoConfig>,
    memory_manager: Arc<CompactionMemoryManager>,
    memory_policy: OnExhaustedPolicy,
    listener: WorkerListener,
    plugins: Plugins,
    next_plan_id: u64,
}

Compaction scheduler tracks and manages compaction tasks.

Fields

scheduler: SchedulerRef

region_status: HashMap<RegionId, CompactionStatus>

All loaded regions, including idle regions and regions executing DDLs.

request_sender: Sender<WorkerRequestWithTime>

Request sender of the worker that this scheduler belongs to.

cache_manager: CacheManagerRef

engine_config: Arc<MitoConfig>

memory_manager: Arc<CompactionMemoryManager>

memory_policy: OnExhaustedPolicy

listener: WorkerListener

plugins: Plugins

Plugins for the compaction scheduler.

next_plan_id: u64

Scheduler-wide generation counter for compaction plans and executions. It outlives region statuses so close/reopen cannot reuse an old identity.

Implementations

impl CompactionScheduler

pub(crate) fn prepare_local_units( &mut self, region: CompactionRegion, units: Vec<CompactionUnit>, reserved: CompactingFiles, ) -> CompactionPhase

Builds unit tasks with fresh execution IDs and shares their already-reserved inputs.

fn local_units_mut( &mut self, region_id: RegionId, ) -> Option<&mut LocalCompaction>

Returns local unit state only while the region is in a local execution phase.

pub(crate) fn dispatch_local_units(&mut self, region_id: RegionId)

Submits pending units up to the request limit and arranges task-exit notifications.

pub(crate) fn can_apply_unit(&self, region_id: RegionId, id: u64) -> bool

Accepts an edit from any current unit that has started committing and is not finished.

pub(crate) fn on_unit_finished( &mut self, region_id: RegionId, id: u64, result: Result<bool, Arc<Error>>, )

Records apply progress or failure without ending sibling executions.

pub(crate) async fn on_unit_released( &mut self, region_id: RegionId, id: u64, manifest: &Arc<ManifestContext>, schemas: SchemaMetadataManagerRef, ) -> CompactionTransition

Accounts for task exit, starts pending units, and checks request completion.

pub(crate) async fn finish_local_units( &mut self, region_id: RegionId, manifest: &Arc<ManifestContext>, schemas: SchemaMetadataManagerRef, ) -> CompactionTransition

Completes the request only after all units exit, using the existing terminal paths.

impl CompactionScheduler

pub(super) fn dispatch_compaction_planning( &self, plan_id: u64, request: CompactionRequest, options: Options, time_range: Option<TimestampRange>, )

pub(super) async fn notify_planning_result( region_id: RegionId, plan_id: u64, request_sender: Sender<WorkerRequestWithTime>, planning: impl Future<Output = CompactionPlanningResult> + Send, )

Runs the planning future and always sends the planning result back to the worker, even if the planning panics.

The worker only leaves the picking phase after it receives the CompactionPickFinished notification. If a panicked planning task swallowed the notification, the region would be stuck in the picking phase forever, blocking all future compactions and pending DDLs (e.g. entering staging) of the region.

async fn prepare_compaction( request: CompactionRequest, options: Options, plugins: Plugins, max_background_compactions: usize, time_range: Option<TimestampRange>, ) -> CompactionPlanningResult

pub(super) async fn handle_compaction_pick_finished_inner( &mut self, finished: CompactionPickFinished, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, ) -> CompactionTransition

Applies a background planning result to the current compaction lifecycle.

Returns

Reports an automatic follow-up dispatched after Picking, or DDL requests released when Picking terminates without creating an execution. The owning worker must dispatch returned DDLs immediately because no execution callback will follow.

async fn finish_compaction_planning( &mut self, region_id: RegionId, err: Option<Arc<Error>>, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, ) -> CompactionTransition

async fn prepare_or_submit_compaction( &mut self, prepared: PreparedCompaction, files: CompactingFiles, waiters: Vec<OutputTx>, plan_id: u64, ) -> Result<Option<CompactionPhase>>

Submits a remote job, or prepares local unit tasks without submitting them. The caller dispatches local units after installing the returned phase.

fn exceeds_compaction_memory_limit(&self, estimated_bytes: u64) -> Option<u64>

impl CompactionScheduler

pub(crate) fn new( scheduler: SchedulerRef, request_sender: Sender<WorkerRequestWithTime>, cache_manager: CacheManagerRef, engine_config: Arc<MitoConfig>, listener: WorkerListener, plugins: Plugins, memory_manager: Arc<CompactionMemoryManager>, memory_policy: OnExhaustedPolicy, ) -> Self

Creates an empty scheduler bound to one region worker.

Effects

Stores the worker request sender and shared scheduling resources. No compaction is dispatched until a scheduling method is called.

Constraints

All lifecycle methods on the returned scheduler must be driven by the same worker so region state transitions remain serialized.

pub(crate) fn register_region( &mut self, version_control: &Arc<VersionControl>, access_layer: &AccessLayerRef, )

Registers a loaded region; replacing its VersionControl invalidates the old incarnation.

pub(crate) fn schedule_automatic_compaction( &mut self, compact_options: Options, version_control: &Arc<VersionControl>, access_layer: &AccessLayerRef, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, ) -> Result<bool>

Accepts an automatic compaction trigger for a region.

Effects

Starts planning when the registered region is idle. Otherwise, an active status coalesces the trigger into one unrestricted follow-up cycle. Returns true only when this call dispatches planning.

Constraints

The owning region worker must call this method serially. Automatic triggers have no waiter and may be ignored while a DDL fence is active. The region identity comes from version_control; access_layer and manifest_ctx must belong to the same region.

pub(crate) fn schedule_manual_compaction( &mut self, compact_options: Options, version_control: &Arc<VersionControl>, access_layer: &AccessLayerRef, waiter: OptionOutputTx, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, max_parallelism: usize, time_range: Option<TimestampRange>, ) -> Result<bool>

Accepts a manual compaction request for a region.

Effects

Starts planning when the registered region is idle. During an automatic compaction, queues the request and replaces any older pending manual request; during a manual compaction, rejects the new request through waiter. Returns true only when this call dispatches planning.

Constraints

The owning region worker must call this method serially. The supplied waiter is completed exactly once by cycle completion, replacement, cancellation, failure, or region teardown. The region identity comes from version_control; access_layer and manifest_ctx must belong to the same region.

pub(crate) async fn handle_compaction_pick_finished( &mut self, finished: CompactionPickFinished, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, ) -> CompactionTransition

Applies a background planning result to the current region state.

Effects

Rejects stale plans, submits a prepared local or remote execution, or completes the cycle when planning produced no executable plan. The returned transition reports a dispatched automatic follow-up or DDL requests released by the resulting terminal transition.

Constraints

The owning worker must call this method for planning notifications and must execute every returned DDL request. The notification may be stale; stale notifications intentionally have no effect.

pub(crate) fn is_current_execution( &self, region_id: RegionId, execution: &CompactionExecution, ) -> bool

Returns whether a terminal notification belongs to the installed execution.

Effects

Performs a read-only identity check against the region’s current plan.

Constraints

Callers must check this before applying a compaction edit. Matching only the region id is insufficient because close/reopen can replace a status.

pub(crate) async fn on_execution_finished( &mut self, region_id: RegionId, execution: &CompactionExecution, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, made_progress: bool, ) -> CompactionTransition

Completes the installed execution after its edit has been applied.

Effects

Notifies waiters, schedules a pending manual or automatic follow-up, or returns the region to idle. A follow-up is scheduled whenever the cycle latched an automatic trigger, or when made_progress is true (the execution produced output or reduced the file count): successful compaction keeps draining the region until the picker returns no plan. The returned transition reports a dispatched automatic follow-up or DDLs that are now safe to execute.

Constraints

The owning worker must call this only after accepting and applying the execution’s result, and must execute every returned DDL request. Stale executions are ignored.

pub(crate) fn on_execution_failed( &mut self, region_id: RegionId, execution: &CompactionExecution, err: Arc<Error>, )

Records failure of the installed execution.

Effects

Resets the matching region to idle and fails all compaction waiters and dependent DDL requests with the supplied error.

Constraints

A stale execution failure is ignored so it cannot tear down replacement state.

pub(crate) fn on_region_dropped(&mut self, region_id: RegionId)

Removes compaction state because the region was dropped.

Effects

Fails all compaction and DDL waiters owned by the status.

Constraints

The owning worker must invoke this as part of serialized region teardown.

pub(crate) fn on_region_closed(&mut self, region_id: RegionId)

Removes compaction state because the region was closed.

Effects

Fails all compaction and DDL waiters owned by the status.

Constraints

The owning worker must invoke this as part of serialized region teardown.

pub(crate) fn on_region_truncated(&mut self, region_id: RegionId)

Resets the cycle after truncate, retaining registration and outstanding DDL replies.

Effects

Fails all compaction and DDL waiters owned by the status.

Constraints

The owning worker must invoke this after truncate completes.

pub(crate) fn try_cancel_and_add_ddl<T>( &mut self, region_id: RegionId, sender: OptionOutputTx, request: T, into_ddl_request: impl FnOnce(T) -> DdlRequest, ) -> Result<(), (OptionOutputTx, T)>

Cancels a running compaction and queues its dependent DDL atomically.

Effects

Requests cancellation when the active phase can still stop, then queues the DDL. The queued DDL forms a scheduling fence for subsequent compaction triggers. If no compaction is running, returns the sender and typed request unchanged.

Constraints

Production callers use this only for DdlRequest::Truncate and DdlRequest::EnterStaging. Ok(()) means the caller must wait for a terminal callback to receive and execute the queued DDL; Err means the caller must execute the returned request directly.

impl CompactionScheduler

fn next_plan_id(counter: &mut u64) -> u64

Returns the current plan id and advances the counter.

Takes the counter instead of &mut self so callers can bump it while holding a mutable borrow of a region status.

fn schedule_compaction( &mut self, trigger: CompactionTrigger, compact_options: Options, version_control: &Arc<VersionControl>, access_layer: &AccessLayerRef, waiter: OptionOutputTx, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, max_parallelism: usize, time_range: Option<TimestampRange>, ) -> Result<bool>

fn handle_pending_compaction_request( &mut self, region_id: RegionId, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, ) -> bool

async fn on_compaction_finished( &mut self, region_id: RegionId, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, made_progress: bool, ) -> CompactionTransition

Notifies the scheduler that the compaction job is finished successfully.

fn schedule_automatic_followup( &mut self, region_id: RegionId, manifest_ctx: &Arc<ManifestContext>, schema_metadata_manager: SchemaMetadataManagerRef, ) -> bool

fn on_compaction_failed(&mut self, region_id: RegionId, err: Arc<Error>)

Notifies the scheduler that the compaction job is failed.

fn remove_region(&mut self, region_id: RegionId, err: Arc<Error>)

Unregisters a closed or dropped region and fails its queued requests.

fn fail_compaction(&mut self, region_id: RegionId, err: Arc<Error>)

Fails a cycle without discarding the loaded region’s registration.

fn finish_compaction_on_cancel( &mut self, region_id: RegionId, ) -> Vec<SenderDdlRequest>

Returns the cycle to idle and records any DDL handoff in the same resident entry.

fn take_ready_ddls( &mut self, region_id: RegionId, ) -> Option<Vec<SenderDdlRequest>>

Hands off pending DDLs after compaction ends and cancels remaining compaction waiters.

fn fence_ddl_replies( &mut self, region_id: RegionId, ddls: Vec<SenderDdlRequest>, ) -> Vec<SenderDdlRequest>

Hands off a drained batch of DDLs and fences new compaction until every reply lands.

Installs the fence, intercepts each DDL’s reply, and returns the same batch so the caller can execute it. The real result of each DDL is still forwarded to its original sender.

Constraints

Must only be called from a compaction terminal transition, and only by the worker that owns region_id.

pub(crate) fn on_ddl_complete( &mut self, region_id: RegionId, generation: u64, request_index: usize, )

Releases the DDL fence after its last reply, ignoring replies from an older generation.

Trait Implementations

impl Drop for CompactionScheduler

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.