mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-08 22:48:58 +00:00
d9ebe5852c2ebf345e33c7cb8ce2d3ca57434b59
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  discord9andRuihang Xia
|
43c30d1446 |
feat(runtime): add weighted workload scheduler (#8736)
* feat(runtime): add weighted workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics Use the GreptimeTeam/catio fork (pinned c20eafc) which adds ClassStats::total_admission_wait and ClassStats::admitted, recorded at each QUEUED -> ADMITTED transition. This exposes the scheduler's own admission delay (excluding Tokio queueing and poll execution), enabling admission-wait based fairness gates. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to dynamic-config revision Bump the catio scheduler fork to 9f4b028 which adds Scheduler::set_weight and Scheduler::set_max_concurrent_polls for runtime configuration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): runtime-adjustable workload scheduler parameters Expose dynamic adjustment of the experimental workload scheduler at runtime: - common-runtime: set_workload_scheduler_weights and set_workload_scheduler_max_concurrent_polls, which forward to the catio scheduler's set_weight/set_max_concurrent_polls when the scheduler is enabled and reject zero values. - servers: /debug/workload_scheduler/weights and /debug/workload_scheduler/max_concurrent_polls POST handlers, so operators can rebalance query/write shares or admission concurrency without restarting the datanode. Both endpoints return 400 with a clear reason when the scheduler is disabled or the requested value is invalid. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): add GET /debug/workload_scheduler status endpoint Returns the current weights (per class), max_concurrent_polls, active_polls and per-class counters (queued, tasks, wakes, polls, completed, cancelled, admitted, total_admission_wait) as JSON. When the scheduler is disabled, returns enabled=false with the other fields omitted, so operators can distinguish 'disabled' from an error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to time-accounting revision Bump the catio scheduler fork to 257ba56 which replaces admission-count accounting with real execution-time accounting (pass += exec_time / (weight * concurrency)), so CPU share follows the configured weights regardless of poll length. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to lock-free sampling revision Bump the catio scheduler fork to efdc0a4 which adds an optional downsampled clock sampling mode (SchedulerBuilder::sample_every_polls, default off) with a lock-free per-class atomic counter, so the downsampled path costs one fetch_add per poll instead of a global mutex. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio to scheduler PR head Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): add scheduler bypass control Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: advance catio scheduler fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: regenerate config docs for workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio scheduler test fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(http): satisfy scheduler lifecycle clippy Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: add distributed scheduler toggle coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: finalize workload scheduler runtime controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio atomic weights Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: preserve unrelated lockfile resolution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(runtime): downsample scheduler time accounting Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(runtime): verify cross-runtime scheduler progress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): configure scheduler poll sampling Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): clarify scheduler activation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): explain scheduler use case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
discord9
|
3e63b8abf3 |
ci: automate compatibility version window (#8608)
* ci: automate compatibility version window Signed-off-by: discord9 <discord9@163.com> * ci: address compat version window review feedback Address all four review comments on the compat version window automation: - Keep the PR window to the sliding window only (latest patch of the two newest stable minor lines). Exact =vX.Y.Z anchors from case.toml are no longer unioned into from_versions; they are validated by the new --check-anchors mode and exercised by nightly runs via --nightly-window. - Add --published-only: the window is computed over stable git tags that have a published, non-draft GitHub release carrying the sqlness compat artifacts (greptime-linux-amd64 tar.gz and sha256sum), so failed releases cannot land in the window. - Run the updater Python tests plus the window/anchor consistency check in PR and merge-group CI (new compat-updater-check job in integration.yml). - Regenerate tests/compatibility/ci.toml to the current sliding window [v1.0.2, v1.1.4]. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Lei, HUANG
|
c55f297dec |
chore!: gate soft-drop table behind the enterprise feature (#8747)
* chore: gate soft-drop table behind the enterprise feature Soft-drop table becomes an enterprise-only feature: - metasrv rejects gc.experimental_soft_drop.enable=true at startup in non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled without the enterprise feature as a second line of defense - the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table() registration, and information_schema.recycle_bin registration are compiled out unless the enterprise feature is enabled - common-meta procedures, tombstone keys, and DdlTask serde stay unconditional for persisted-procedure recovery and wire compatibility - the [gc.experimental_soft_drop] section is removed from the OSS example config and generated docs (moving to the enterprise repo) - the soft-drop sqlness cases and their CI job are removed from OSS (moving to the enterprise repo); affected information_schema .result files are regenerated Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: limit unused_variables allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch accidental unused variables in register_admin_only. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: include the config key in the soft-drop enterprise gate error Addresses review comment: name gc.experimental_soft_drop.enable in the startup validation error so users can locate the setting quickly when it is set via env vars or layered config. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: limit unused_mut allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch unused mut in the table_ddl_event test setup. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: reject soft-drop DDL submissions in non-enterprise builds Addresses review comment: clients could bypass the SQL-level gates by submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly to the procedure service. Reject fresh submissions at the DdlManager boundary in non-enterprise builds while keeping the procedure loaders registered for crash recovery and wire compatibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: stop --enable-gc from enabling soft drop in the sqlness template Addresses review comment: the metasrv test template rendered [gc.experimental_soft_drop] enable = true under the generic --enable-gc flag, which non-enterprise metasrv now rejects at startup, making the documented --enable-gc mode unusable in OSS. Keep the flag scoped to plain GC; enterprise soft-drop coverage moves to the enterprise repo. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate fresh soft-drop procedures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: gate soft-drop fallback coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate soft-drop procedure implementation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate drop table soft-drop behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate expired soft-drop gc behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: test enterprise table ddl lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark purge_table as enterprise licensed The purge_table module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark recycle_bin as enterprise licensed The recycle_bin module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark soft-drop procedure sources as enterprise licensed The purge and undrop procedure implementations plus the recycle-bin test module compile only with the enterprise feature. Apply the Enterprise License header and register them with both license configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 Weny Xu
|
a6107fbe3d |
ci: optimize fuzz and split workflows (#8710)
* ci: batch fuzz targets in GitHub Actions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: improve fuzz test observability Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): preserve fuzz setup failure artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(ci): keep fuzz mock output in logs Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: optimize fuzz worker cache Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: warm fuzz target binaries Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: isolate fuzz workflow Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: centralize fuzz target preparation Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: split general workflows Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline docs required checks Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: transfer fuzz targets as artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve fuzz binary permissions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline fuzz workers Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: cache PR build dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: retain main build cache policy Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: address fuzz review feedback Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |