mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
Lei, HUANG
a0d798129a
* feat(mito2): schedule compaction units fairly across regions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): simplify compaction admission cost and ready queues Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): derive compaction coordinator capacity from its executor CompactionCoordinator::new accepted the executor concurrency and the admission capacity as independent values, so the two could drift and admit work that re-queues behind contenders in the executor's FIFO. Add CompactionCoordinator::new_local(concurrency), which builds the LocalScheduler and the capacity from one value, and use it in WorkerGroup. Tests keep the explicit constructor for injected executors. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep inflight compaction accounting unwind safe The local unit job incremented the inflight gauge, awaited the task and then decremented it. A panic escaping the task, or a started job future dropped before completion, skipped the decrement and leaked the gauge permanently. Own the gauge through an RAII guard created after the increment so every exit path decrements exactly once. Cover the unwind path with an isolated gauge. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): pin backlog weight cap and minimum positive cost The admission policy halves the charge of a region that has more than one ready unit and charges at least one half-byte. Neither property had a test that failed when the weight was constant, uncapped, or the minimum charge was removed. Add two scripted capacity-1 admission-order regressions: one for the capped backlog weight with merge-sized units, and one for zero-byte expiration units that must still accrue service. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): pin the idle service floor against stale credit A region that goes idle kept its low service value, so it could run a long burst on wake and starve the regions that served during its sleep. The only reentry tests left the virtual-time floor below that stale value, so removing the clamp changed no assertion. Add a capacity-1 admission-order regression where a busy region advances virtual time past the sleeping region's service before it returns with a three-unit batch. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document compaction admission accounting state The coordinator's state types carried no explanation of the ordering key, the service unit, or the coupling between service and virtual time, and the admission tick reused the crate's write-sequence term without saying so. Document the coordinator fields, the shared state, and each region queue field so the ordering contract can be reviewed without reverse-engineering the unit and the floor update. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document ReadyUnit ownership and completion ordering ReadyUnit's fields and constructor were undocumented, and the invariant that matters lived only in the executor trait and in AdmittedUnit's drop: the completion must never observe the job's live input leases. Document the fields and constructor, including that completion runs after the job is dropped on admission, withdrawal, rejection, panic and shutdown paths. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore(mito2): lowercase compaction admission metric help text The three new compaction admission metrics used sentence case inside a block whose other help strings are lowercase, so the exported /metrics output was inconsistent. Lowercase the help text only; names, labels and buckets are unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): align compaction test harness concurrency SchedulerEnv's default compaction scheduler was a single-runner LocalScheduler while the coordinator used the configured max_background_compactions as its admission capacity, a pairing production does not have. Up to capacity-1 units could be counted as running while permanently queued on one runner. Derive the default compaction executor from the configured concurrency, keep injected executors, and document that VecScheduler only collects jobs and cannot serve drain or shutdown assertions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): describe the widened max_background_compactions scope The setting now also bounds engine-wide admission of local compaction units and the number of independent units one automatic per-region pick may prepare, while manual compaction keeps its request parallelism. Only the pool size was documented, so raising the value silently widened both. Extend the field doc and both example TOML comments, then regenerate config/config.md with make config-docs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): make the inflight accounting regression testable The inflight test wrapped both its gauge assertion and the fixture panic in one catch_unwind and only checked that a panic happened, so a broken counter assertion could be mistaken for the expected unwind. It also exercised a copy of the accounting scope instead of the job wrapper. Run the production job through InflightCompaction::run so tests and callers share one accounting scope, assert the gauge outside the catch boundary, and verify the expected panic payload. Cover completion, unwind, an unpolled drop and a started drop. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): verify compaction capacity through execution The harness capacity test only read CompactionCoordinator::capacity(), so it still passed when the default executor was a single-runner LocalScheduler or when the injected executor was ignored. Remove that test-only getter and drive real units instead: one and four configured runners must each start exactly that many units, hold the rest, and drain; a separate case proves units reach an injected executor. Three scheduler tests also assumed a follow-up submission while the previous batch still held its admission slot. The harness never runs or releases those units, so they passed only on hosts whose CPU-derived capacity exceeded one. Pin their capacity to two so they no longer depend on the host. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify the max_background_compactions unit budget The field described the same limit as the number of independent units one automatic pick may prepare. In fact it bounds the merge-output budget: merged groups share inputs, and expired SSTs may add one expiration-only unit outside that budget while still consuming a slot in the same admission limit. Correct the rustdoc and both example TOMLs, then regenerate config/config.md with make config-docs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): describe ready unit ownership directly Explain that submission consumes the unit and prevents repeated admission, instead of referring to a nonexistent grant-message protocol. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): link the compaction admission wait metric Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): allow admission capacity to equal runner count State the non-strict capacity bound used by new_local, rather than implying that one executor slot must remain unused. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): use descriptive wording for region retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover queued compaction cancellation and DDL handoff Keep a prepared unit outside the executor behind another region's sole admission slot. Verify its actual Released notification completes the waiter as cancelled, releases inputs, and hands off truncate or enter-staging instead of failing the DDL. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover parallel automatic follow-up planning Keep two out-of-range windows eligible after a manual pick and assert that the automatic continuation prepares both with the configured parallelism. This detects a regression back to a single-output follow-up budget. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify completion handling for rejected submissions Document that accepted units dispatch completion after releasing job resources, while submission to a closed coordinator drops both futures unpolled and leaves error bookkeeping to the caller. Keep the existing notification behavior unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): observe rejected compaction units through release notifications Upstream now asserts release accounting right after admission, assuming the executor rejects a unit synchronously. Engine-wide admission queues units in the coordinator instead, so a rejection surfaces as the asynchronous `CompactionUnitNotification::Released` notification. Drive those notifications through the worker path before asserting, and drop a duplicated inflight accounting helper left behind by the conflict resolution. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(mito2): format rejected compaction release matching Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>