Commit Graph
1 Commits
Author SHA1 Message Date
discord9andRuihang Xia 43c30d1446 feat(runtime): add weighted workload scheduler (#8736)
* feat(runtime): add weighted workload scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics

Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to dynamic-config revision

Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(perf): runtime-adjustable workload scheduler parameters

Expose dynamic adjustment of the experimental workload scheduler at
runtime:

- common-runtime: set_workload_scheduler_weights and
  set_workload_scheduler_max_concurrent_polls, which forward to the
  catio scheduler's set_weight/set_max_concurrent_polls when the
  scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
  /debug/workload_scheduler/max_concurrent_polls POST handlers, so
  operators can rebalance query/write shares or admission concurrency
  without restarting the datanode.

Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(perf): add GET /debug/workload_scheduler status endpoint

Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to time-accounting revision

Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to lock-free sampling revision

Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin catio to scheduler PR head

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): add scheduler bypass control

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: advance catio scheduler fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin merged catio scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: regenerate config docs for workload scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin catio scheduler test fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(http): satisfy scheduler lifecycle clippy

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: add distributed scheduler toggle coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: finalize workload scheduler runtime controls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin merged catio atomic weights

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: preserve unrelated lockfile resolution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(runtime): downsample scheduler time accounting

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(runtime): verify cross-runtime scheduler progress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): configure scheduler poll sampling

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(runtime): clarify scheduler activation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(runtime): explain scheduler use case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2026-09-02 07:02:39 +00:00