From ebb7e22566f8cb6246f39eab01e32488c682a2c8 Mon Sep 17 00:00:00 2001 From: Weny Xu Date: Wed, 28 Oct 2026 12:51:52 +0000 Subject: [PATCH] chore(ci): gate fuzz, E2E and KV backends by diff scope (#9927) * ci: gate the integration CI long tail by diff scope `integration.yml` starts its long tail on every pull request: the three fuzz families, the chaos legs, the export/import E2E job and the two sqlness KvBackend legs. Most diffs cannot affect them, so a `scope` job now classifies the pull request and tells those jobs whether to start. The classifier is deterministic and fail-open. Forced modes (merge queue, `schedule`, `workflow_dispatch`, `ci:full`, `CI_SCOPE_FORCE_ALL`) and three floors -- always-run paths, an uncovered `src/` change, and a PromQL/flow filter -- can only widen the result; everything else is a glob match against the suite catalog. Every step of the job is `continue-on-error` and every gate reads `needs.scope.outputs. != 'false'`, so a classifier crash runs the jobs instead of skipping them. The rules are checked out from the pull request base, so a pull request cannot relax its own gate. The policy is data: `.github/ci/scope-rules.toml` holds the path groups, the suite catalog and the always-run list, with a `reason` on every exclusion. `ci_scope_rules.py` loads it with the stdlib `tomllib` and validates the shape (unknown or cyclic `@group` references, a `family` that disagrees with the suite id, an empty reason) and that every glob's literal prefix still exists, so a renamed crate cannot leave a rule that matches nothing. `make check-ci-scope` runs that check locally; the `CI Scope Rules` job runs it on `ubuntu-slim`, the same image the classifier uses. `.github/ci/scope-cases.toml` holds golden cases that `test_ci_scope.py` replays through the real classifier. The PostgreSQL and MySQL sqlness legs move into their own `sqlness-kvbackend` job so they can be gated: a job-level `if:` cannot read the matrix, and gating them inside `sqlness` would skip required checks and block the merge. They run the same cases as the required Basic leg, so only the backend itself matters. The verdict is visible where triage looks: one log line per suite, the exact gate values, `::warning::` when a lookup fails and the gate widens, and a job summary that lists each suite with the rule that decided it and the changed files that matched. Measured on 200 merged and open pull requests: 27.5% always run, 41% decided by path rules, 31.5% skip every gated job; 995 of 2600 suite selections are skipped, with no selection that a path rule makes. The classifier itself runs in about ten seconds on a single-vCPU runner. Signed-off-by: WenyXu * fix(ci): let the scope job read pull requests The `scope` job declared `permissions: contents: read`, and an explicit block sets every unlisted permission to `none`. `GET /pulls/{n}` therefore answered 403, `build_plan` widened to a full run as designed, and every gate read an absent output as "run": the gate could never skip anything. - `pull-requests: read` on the `scope` job. - `tests/runner/**` joins `@kv-backend`: the runner implements `--setup-pg`/`--setup-mysql`, which only the KV-backend legs pass, so a change there must not skip the only job that exercises it. The `UNMAPPED_CRATES` excuse is gone with it. - The README no longer claims a queue failure blocks the merge: the gated jobs are not required checks. Signed-off-by: WenyXu * fix(ci): run every suite when the kind registry script changes The distributed and chaos fuzz jobs start their cluster through `.github/actions/setup-kind`, which runs `.github/scripts/kind-with-registry.sh`. `.github/actions/**` is in `ALWAYS_RUN` but the script is not, so a change confined to it selected no suite at all and every gate stayed `false`: the jobs that exercise the changed cluster setup were skipped. The script joins `ALWAYS_RUN`, consistent with the action that calls it, and a golden case pins it. Signed-off-by: WenyXu * fix(ci): widen the gate for the fuzz corpus seeding script `fuzz-test/action.yaml` runs `.github/scripts/prepare-fuzz-corpus.sh` before the cache restore in every fuzz job, but the script was not in `ALWAYS_RUN`. A change confined to it therefore selected no suite: `any_fuzz` stayed `false` and every fuzz job that runs it was skipped. The script joins `ALWAYS_RUN` next to the other scripts the fuzz jobs execute, with a golden case for it. Both misses so far (this one and `kind-with-registry.sh`) came through a composite action, which the workflow never names. `WorkflowContractTests` now resolves `uses: ./.github/actions/` transitively for the gated jobs and fails when one of the scripts they run is not in `ALWAYS_RUN`. Signed-off-by: WenyXu --------- Signed-off-by: WenyXu --- .github/ci/README.md | 168 ++++ .github/ci/ci_scope.py | 623 ++++++++++++ .github/ci/ci_scope_rules.py | 277 ++++++ .github/ci/scope-rules.toml | 284 ++++++ .github/ci/tests/fixtures/scope-cases.toml | 158 +++ .github/ci/tests/test_ci_scope.py | 1018 ++++++++++++++++++++ .github/scripts/tests/test_ci_slash.py | 2 +- .github/workflows/checks.yml | 19 + .github/workflows/integration.yml | 161 +++- Makefile | 4 + 10 files changed, 2675 insertions(+), 39 deletions(-) create mode 100644 .github/ci/README.md create mode 100755 .github/ci/ci_scope.py create mode 100755 .github/ci/ci_scope_rules.py create mode 100644 .github/ci/scope-rules.toml create mode 100644 .github/ci/tests/fixtures/scope-cases.toml create mode 100755 .github/ci/tests/test_ci_scope.py diff --git a/.github/ci/README.md b/.github/ci/README.md new file mode 100644 index 00000000000..07de1fcae7b --- /dev/null +++ b/.github/ci/README.md @@ -0,0 +1,168 @@ +# Integration CI admission gate + +`integration.yml` runs its expensive jobs — fuzz, export/import E2E, the sqlness +KvBackend legs — on every non-draft pull request. Most pull requests cannot +affect any of them. This directory decides which of those jobs start, using only +the diff. + +The gate can only *skip* work, never add it, and it fails open: when the verdict +is missing or uncertain, every gated job runs. + +| File | Role | +|---|---| +| `ci_scope.py` | the classifier: reads the diff, writes `$GITHUB_OUTPUT` | +| `scope-rules.toml` | the policy: which paths select which suite, as data | +| `ci_scope_rules.py` | loads and validates the policy; `--check-paths` | +| `tests/` | unit tests, and `fixtures/scope-cases.toml` golden cases | + +Two words appear everywhere below. A **suite** is one matrix leg of a gated job, +named `/`. A **gate** is one `$GITHUB_OUTPUT` key that switches a +whole job. + +## What is gated + +Only `integration.yml`, and only these jobs: + +| Gated job | Runs | +|---|---| +| `prepare-fuzz-targets` | the fuzz targets only | +| `fuzztest` | standalone fuzz, 2 legs | +| `unstable-fuzztest` | unstable standalone create-table fuzz, 1 leg | +| `distributed-fuzztest` | distributed fuzz with remote WAL, 4 legs | +| `distributed-fuzztest-with-chaos` | Chaos Mesh, 3 legs | +| `build-greptime-ci` | the profile-CI binary only | +| `export-import-v2-e2e` | CLI `COPY DATABASE` export/import against S3 | +| `sqlness-kvbackend` | sqlness with PostgreSQL and MySQL as the metadata store | + +The legs themselves are the `matrix` entries in `integration.yml`; the rules +file repeats their ids so a verdict can name them. + +Everything else always runs: `rust.yml`, `checks.yml`, `docs.yml`, the debug +`build`, the required `sqlness` job (Basic, Remote WAL, Flat format), `compat` +and `compat-updater-check`. None of the gated jobs is in the required-check +matrix in `docs.yml`, so a skipped job can never block a merge. + +Two choices follow from that rule. The two KvBackend legs sit in their own +`sqlness-kvbackend` job because a job-level `if:` cannot read `matrix`, so +gating them inside `sqlness` would skip required checks and make the pull +request unmergeable. The multi-language SDK job stays ungated: it takes about +three minutes, less than the cost of getting its verdict wrong. + +## How a verdict is made + +First match wins: + +1. **Forced** — `merge_group`, `schedule`, `workflow_dispatch`, the `ci:full` + label or `CI_SCOPE_FORCE_ALL`. Everything runs. +2. **Too large to judge** — more than 300 changed files. Everything runs. +3. **Always-run file changed** — `Cargo.toml`, `Cargo.lock`, `tests-fuzz/**`, + this directory, or anything else that changes what the suites build or how + they start. Everything runs. +4. **Uncovered change** — a path under `src/` that no suite claims. This is the + net for a crate added, split or renamed after the rules were written. + Everything runs. +5. **Path rules** — the suites whose globs match the changed files, and only + those. + +The code is the source of truth for the details; its module docstring is worth +reading before changing it. + +## Why a check was skipped + +The run page's summary pane answers this without opening a log: one row per +suite, with its verdict, the rule that decided it, and up to three of the +changed files that matched. + +The step log of the `CI Scope` job has the same information in text form. It +starts with `CI scope: mode=… full=…` (`grep '^CI scope'` finds it), then one +line per suite — `run` or `skip`, source, reason — then `gates:` with the exact +values the jobs read. + +A degraded decision is never silent. A failed label or file lookup emits +`::warning::` and widens the gate to every suite. + +## Forcing a full run + +Add the `ci:full` label to the pull request and click "Re-run all jobs". The +label is read through the API at run time, not from the event payload, so one +added after the first run still counts. `/ci` dispatch +(`.github/scripts/ci-slash.py`), the `CI_SCOPE_FORCE_ALL` repository variable +and every merge-queue run also run everything: the classifier returns a forced +plan there, so a wrong skip is still exercised before the change lands. None of +the gated jobs is a required check, so a failure in the queue does not block the +merge by itself. + +`CI_SCOPE_FORCE_ALL` is the switch to flip while debugging the rules themselves. + +## Changing the rules + +`scope-rules.toml` is data, not code: `ci_scope_rules.py` loads it with the +standard library's `tomllib`. Two things are validated: + +- the shape — an unknown or cyclic `@group` reference, a `gate` that disagrees + with the suite id, a gate missing from `[gates]`, or an exemption without a + `reason` all raise; +- that every glob still points at something that exists, so a renamed or + deleted crate fails the check instead of silently matching nothing. + +Run both with `make check-ci-scope`, or the path half alone with +`python3 .github/ci/ci_scope_rules.py --check-paths`. The same command runs in +the `CI Scope Rules` job of `checks.yml`, on `ubuntu-slim` — the image the +classifier itself uses, so a path that exists there also exists when the verdict +is computed. + +A malformed file fails the classify step, and because every step of that job is +`continue-on-error`, every gated job runs instead. + +`[[matching.exempt]]` lists paths outside the gate's remit: they select no suite +and never widen the gate. Each one carries a `reason`, and the loader rejects an +empty one — what CI never runs is the part a reader must be able to justify. + +`tests/fixtures/scope-cases.toml` is test data, not configuration: a diff shape +and the suites it must start. `ScopeCaseTests` replays each case through the +classifier, so a rule edit that changes behaviour fails with the case name +instead of silently rerouting CI. The cases carry no pull request numbers; +`git log` has those. + +## Adding a suite + +1. Add a `[suites."/"]` table to `scope-rules.toml` with `gate`, + `paths` (globs or `@group` references) and a one-line `description`. The id + is `/`, and `gate` is that id with `-` replaced by `_`; + the loader and `test_suite_metadata` both enforce it. +2. Add the matching matrix leg to the job in `integration.yml`. +3. Gate the job's `if:` with the matching output, and list `scope` in its + `needs`. A new gate also needs a key in `derive_outputs` and in the `scope` + job's `outputs`; `test_every_gate_routes_to_an_output` fails when that + routing is missing. +4. Add a case to `tests/fixtures/scope-cases.toml` and run + `python3 .github/ci/tests/test_ci_scope.py`. `CrateCoverageTests` separately + fails when a workspace crate matches no suite and is not excused in + `UNMAPPED_CRATES`: the runtime net would run everything, but the rule table + is what should be updated. + +## Ground rules for the workflow + +These six rules keep the gate failing open. Break one and either CI runs jobs it +should have skipped, or a classifier failure takes CI down with it. + +- **Read the output as `!= 'false'`.** An absent output means "run"; writing + `== 'true'` inverts the failure direction. +- **Start every gate with `!cancelled()`.** Without a status function the + condition carries an implicit `success()` over `needs`, so a `scope` job that + times out would skip the gates. Other dependencies are therefore required + explicitly with `needs..result == 'success'`. +- **Every fallible step of `scope` is `continue-on-error: true`.** GitHub skips + the dependents of a *failed* job, so a classifier crash must leave the job + green with no outputs rather than take CI down with it. +- **Rules come from the base revision.** The `scope` job checks out + `pull_request.base.sha` (or `merge_group.base_sha`), so a pull request cannot + weaken its own gate. On the first pull request that introduces this directory + the base has no `ci_scope.py`: the step fails, no outputs are written, and + every gate runs. +- **One gate per job, never per leg.** GitHub does not expose `matrix` to + `jobs..if`. Per-suite verdicts stay visible in the log. +- **Pass the run's `head`/`base` SHAs and let the classifier compare them with + the pull request's live revisions.** A re-run keeps the original SHAs while + GitHub serves the *current* file list; without the comparison the new diff + would be judged against the old tree. With it, a mismatch runs everything. diff --git a/.github/ci/ci_scope.py b/.github/ci/ci_scope.py new file mode 100755 index 00000000000..571e975325a --- /dev/null +++ b/.github/ci/ci_scope.py @@ -0,0 +1,623 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Decide which integration CI suites must run for one pull request. + +Pipeline, in priority order: + +1. forced mode -- `merge_group`, `schedule`, `workflow_dispatch`, + `CI_SCOPE_FORCE_ALL`, or the `ci:full` label. Every suite runs. +2. always-run floor -- a changed file matches `ALWAYS_RUN`. Every suite runs. +3. uncovered build change -- a path under `SRC_TREE` that no suite claims, so a + crate added or renamed later still runs everything. +4. static path matches -- glob match of the changed files against the suite + catalog in `scope-rules.toml`. Paths under `EXEMPT_TREES` are outside the + gate's remit and are dropped first: `@server` claims `src/query/**`, which + would otherwise put a PromQL-only change through the whole fleet. + +The verdict is deterministic: the same diff always produces the same plan, and +a change that no rule explains falls through to "run everything". + +The plan maps to `$GITHUB_OUTPUT` keys that the `integration.yml` gate jobs +consume; `--dry-run` prints them as JSON. +""" + +import argparse +import json +import os +import re +import sys +import urllib.error +import urllib.request + +from ci_scope_rules import ( + ALWAYS_RUN, + EXEMPT_TREES, + SRC_TREE, + SUITES, + suite_group, + suite_paths, +) + +MAX_FILES = 300 +HTTP_TIMEOUT = 45 +FORCED_EVENTS = {"merge_group", "schedule", "workflow_dispatch"} +# A diff with no code, config or workflow file never needs a suite. +CODE_FILE = re.compile( + r"\.(rs|toml|lock|sh|ya?ml)$|(^|/)Makefile$|(^|/)Cargo\.toml$|\.sql$" +) +FUZZ_GATES = ("fuzz_standalone", "fuzz_distributed", "fuzz_chaos") + + +class ScopeError(Exception): + """A handled failure: the caller falls back instead of crashing.""" + + +# --------------------------------------------------------------------------- +# HTTP seam (tests replace `_request`) +# --------------------------------------------------------------------------- + + +def _request(method, url, headers, body=None, timeout=HTTP_TIMEOUT): + request = urllib.request.Request(url, data=body, headers=headers, method=method) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + return response.status, response.read() + except urllib.error.HTTPError as err: + return err.code, err.read() + except (urllib.error.URLError, TimeoutError, OSError) as err: + raise ScopeError(f"{method} {url}: {err}") from err + + +def _get(url, headers, timeout=HTTP_TIMEOUT): + return _request("GET", url, headers, None, timeout) + + +def _post(url, headers, body, timeout=HTTP_TIMEOUT): + return _request("POST", url, headers, body, timeout) + + +def get_json(url, headers, timeout=HTTP_TIMEOUT): + """GET returning (status, parsed json); transport failures raise `ScopeError`.""" + status, raw = _get(url, headers, timeout) + try: + payload = json.loads(raw.decode("utf-8", "replace")) if raw else None + except json.JSONDecodeError: + payload = None + return status, payload + + +# --------------------------------------------------------------------------- +# Path matching +# --------------------------------------------------------------------------- + + +def glob_to_regex(pattern): + """Translate a repo glob into an anchored regexp. + + `**/` also matches zero directories, `**` spans `/`, `*` and `?` do not. + """ + out = ["^"] + index = 0 + while index < len(pattern): + char = pattern[index] + if char == "*": + end = index + while end < len(pattern) and pattern[end] == "*": + end += 1 + if end - index >= 2: + if end < len(pattern) and pattern[end] == "/": + out.append("(?:.*/)?") + index = end + 1 + continue + out.append(".*") + else: + out.append("[^/]*") + index = end + elif char == "?": + out.append("[^/]") + index += 1 + else: + out.append(re.escape(char)) + index += 1 + out.append("$") + return re.compile("".join(out)) + + +def compile_patterns(patterns): + return [glob_to_regex(pattern) for pattern in patterns] + + +def matches_any(filename, compiled): + return any(regex.match(filename) for regex in compiled) + + +# --------------------------------------------------------------------------- +# GitHub API +# --------------------------------------------------------------------------- + + +def github_api(method, path, token, payload=None): + base = os.environ.get("GITHUB_API_URL", "https://api.github.com").rstrip("/") + url = base + path + headers = { + "Accept": "application/vnd.github+json", + "User-Agent": "greptimedb-ci-scope", + } + if token: + headers["Authorization"] = "Bearer " + token + body = None + if payload is not None: + body = json.dumps(payload).encode() + headers["Content-Type"] = "application/json" + if method == "GET": + return get_json(url, headers) + status, raw = _request(method, url, headers, body, HTTP_TIMEOUT) + try: + parsed = json.loads(raw.decode("utf-8", "replace")) if raw else None + except json.JSONDecodeError: + parsed = None + return status, parsed + + +def fetch_changed_files(repo, number, token): + """Return (files, truncated). `truncated` means the PR has more than MAX_FILES.""" + files = [] + page = 1 + while True: + status, payload = github_api( + "GET", f"/repos/{repo}/pulls/{number}/files?per_page=100&page={page}", token + ) + if status != 200 or not isinstance(payload, list): + raise ScopeError(f"cannot list pull request files (HTTP {status})") + for item in payload: + if not isinstance(item, dict): + continue + patch = item.get("patch") + files.append( + { + "filename": item.get("filename", ""), + "previous_filename": item.get("previous_filename") or "", + "status": item.get("status", "modified"), + "additions": item.get("additions", 0), + "deletions": item.get("deletions", 0), + "patch": patch if isinstance(patch, str) else "", + "patch_included": isinstance(patch, str) and bool(patch), + } + ) + if len(payload) < 100: + return files, False + if len(files) >= MAX_FILES: + # Keep `per_page=100`: with `per_page=1`, page N is item N, so the + # probe would answer "is there a file 4?" instead of "file 301?". + status, extra = github_api( + "GET", + f"/repos/{repo}/pulls/{number}/files?per_page=100&page={page + 1}", + token, + ) + if status != 200 or not isinstance(extra, list): + raise ScopeError( + "cannot confirm the changed file list is complete " + f"(HTTP {status})" + ) + del files[MAX_FILES:] + return files, bool(extra) + page += 1 + + +def fetch_pull(repo, number, token): + """Return (labels, head sha, base sha) for one pull request.""" + status, payload = github_api("GET", f"/repos/{repo}/pulls/{number}", token) + if status != 200 or not isinstance(payload, dict): + raise ScopeError(f"cannot read pull request (HTTP {status})") + labels = payload.get("labels") or [] + head = ((payload.get("head") or {}).get("sha") or "").strip() + base = ((payload.get("base") or {}).get("sha") or "").strip() + names = {label.get("name", "") for label in labels if isinstance(label, dict)} + return names, head, base + + +def revision_drift(args, head, base): + """Return a note when the tested revisions are no longer the live ones. + + A re-run keeps the run's original SHAs while the file list served by GitHub + is the pull request's current one. After a force-push the classifier would + judge the new diff for the old tree, so a mismatch runs every suite instead. + """ + for supplied, live, name in ( + (args.head_sha, head, "head"), + (args.base_sha, base, "base"), + ): + if supplied and live and supplied.lower() != live.lower(): + return ( + f"pull request {name} moved from {supplied[:12]} to " + f"{live[:12]} after this run started" + ) + return "" + + +# --------------------------------------------------------------------------- +# Decisions +# --------------------------------------------------------------------------- + + +def file_paths(item): + """Every path the change touched; a rename keeps the deleted side too.""" + names = [item["filename"]] + previous = item.get("previous_filename") + if previous and previous != item["filename"]: + names.append(previous) + return names + + +def gated_paths(item): + """The paths of one changed file that the gate judges. + + A path under `EXEMPT_TREES` is outside the remit: it selects no suite, and + it is not a change that "no rule covers", so it cannot widen the gate. + """ + return [name for name in file_paths(item) if not name.startswith(EXEMPT_TREES)] + + +def path_matches(files): + """Return (always_run_hits, {suite_id: [matching file paths]}).""" + always_compiled = compile_patterns(ALWAYS_RUN) + suite_matchers = { + suite_id: compile_patterns(suite_paths(suite)) + for suite_id, suite in SUITES.items() + } + always_hits = set() + matches = {suite_id: set() for suite_id in SUITES} + for item in files: + for name in file_paths(item): + if matches_any(name, always_compiled): + always_hits.add(name) + for name in gated_paths(item): + for suite_id, compiled in suite_matchers.items(): + if matches_any(name, compiled): + matches[suite_id].add(name) + return sorted(always_hits), { + suite_id: sorted(matched) for suite_id, matched in matches.items() + } + + +def uncovered_changes(files, matches): + """True when a change can reach code that no suite path claims. + + This is the net for a crate that was added, split or renamed after the rules + were written: without it such a change matches no suite and would skip every + gated job. A single unmatched path is enough, even when other paths in the + same diff match a suite. + """ + return first_uncovered(files, matches) is not None + + +def first_uncovered(files, matches): + """The first changed path that no suite claims, or None. + + Naming it is what makes the widening explainable in the log. + """ + matched = {name for names in matches.values() for name in names} + return next( + ( + name + for item in files + for name in gated_paths(item) + if name.startswith(SRC_TREE) and name not in matched + ), + None, + ) + + +def has_meaningful_changes(files): + return any(CODE_FILE.search(item["filename"]) for item in files) + + +def finalize(plan): + """`full` means every gated suite runs, whatever produced the decision.""" + plan["full"] = all(entry["run"] for entry in plan["suites"].values()) + return plan + + +def all_suites(run, reason, source): + return { + suite_id: {"run": run, "source": source, "reason": reason} + for suite_id in SUITES + } + + +def full_plan(plan, reason, source, mode="static"): + """Every suite runs; `finalize` derives `full` from the verdicts.""" + plan.update(mode=mode, suites=all_suites(True, reason, source)) + return finalize(plan) + + +def forced_reason(args, labels): + """Return why every suite must run, or an empty string.""" + event = args.event or os.environ.get("GITHUB_EVENT_NAME") or "pull_request" + if event in FORCED_EVENTS: + return f"event={event}" + if os.environ.get("CI_SCOPE_FORCE_ALL", "").strip().lower() in { + "1", + "true", + "yes", + "on", + }: + return "CI_SCOPE_FORCE_ALL" + if "ci:full" in labels: + return "label ci:full" + return "" + + +def build_plan(args, token): + notes = [] + plan = { + "mode": "static", + "head_sha": args.head_sha or "", + "base_sha": args.base_sha or "", + "full": False, + "suites": {}, + "matching": {}, + "notes": notes, + } + + # Events and variables that force a run need no GitHub call at all; the + # `ci:full` label is the only reason to read the pull request early. + forced = forced_reason(args, set()) + labels, head_live, base_live = set(), "", "" + if args.pr and not forced: + try: + labels, head_live, base_live = fetch_pull( + args.repo, args.pr, token + ) + except ScopeError as err: + # A failed lookup must widen the gate, never narrow it. + notes.append(f"pull request lookup failed: {err}") + return full_plan(plan, "pull request lookup failed", "always-run") + forced = forced_reason(args, labels) + + if forced: + notes.append(f"forced: {forced}") + return full_plan(plan, f"forced by {forced}", "forced", mode="forced") + + if not args.pr: + raise ScopeError("--pr is required unless the event forces every suite") + + drift = revision_drift(args, head_live, base_live) + if drift: + notes.append(drift) + return full_plan(plan, drift, "always-run") + + files, truncated = fetch_changed_files(args.repo, args.pr, token) + if truncated: + notes.append(f"changed file list truncated at {MAX_FILES}") + return full_plan( + plan, + f"pull request has more than {MAX_FILES} changed files", + "always-run", + ) + + always_hits, matches = path_matches(files) + # Keep the evidence: the step summary explains a verdict by naming the files + # that matched, which is what triage asks for first. + plan["matching"] = {suite_id: sorted(names) for suite_id, names in matches.items()} + if always_hits: + notes.append(f"always-run: {', '.join(sorted(always_hits)[:5])}") + return full_plan( + plan, + "always-run path: " + ", ".join(sorted(always_hits)[:3]), + "always-run", + ) + + first = first_uncovered(files, matches) + if first is not None: + notes.append(f"uncovered build change: {first}") + return full_plan(plan, f"uncovered build change: {first}", "uncovered") + + static_run = {suite_id: bool(matched) for suite_id, matched in matches.items()} + + if not has_meaningful_changes(files) and not any(static_run.values()): + plan.update( + mode="static", + full=False, + suites={ + suite_id: { + "run": False, + "source": "static", + "reason": "no code, config or workflow file changed", + } + for suite_id in SUITES + }, + ) + notes.append("no meaningful changes: skip every suite") + return finalize(plan) + + plan.update( + mode="static", + suites={ + suite_id: { + "run": static_run[suite_id], + "source": "static", + "reason": ("path rule: " + ", ".join(matches[suite_id][:3])) + if matches[suite_id] + else "no path rule matched", + } + for suite_id in SUITES + }, + ) + return finalize(plan) + + +def derive_outputs(plan): + """Flatten the plan into the `$GITHUB_OUTPUT` keys the workflow gates on. + + + Every gate in `SUITES` must route to a key here, otherwise its suites are + classified but never switch a job; `test_every_gate_routes_to_an_output` + asks this function directly, so a new gate fails there until it is routed. + + + Job-level `if:` cannot read the `matrix` context, so the workflow gates a + whole fuzz gate instead of individual matrix legs; the per-suite verdicts + stay visible in the step log. + """ + + def suites_of(gate): + return [ + suite_id + for suite_id, entry in plan["suites"].items() + if entry["run"] and SUITES[suite_id]["gate"] == gate + ] + + outputs = {} + legs = {} + for gate in FUZZ_GATES: + running = suites_of(gate) + legs[gate] = [suite_group(suite_id) for suite_id in running] + legs[gate] = [leg for leg in legs[gate] if leg] + outputs[gate] = "true" if running else "false" + unstable = bool(suites_of("fuzz_unstable")) + # Every gate outside the fuzz group switches exactly one job of its own. + for gate in sorted({suite["gate"] for suite in SUITES.values()}): + if gate not in FUZZ_GATES: + outputs[gate] = "true" if suites_of(gate) else "false" + outputs["any_fuzz"] = ( + "true" + if unstable or any(outputs[gate] == "true" for gate in FUZZ_GATES) + else "false" + ) + distributed = outputs["fuzz_distributed"] == "true" or outputs["fuzz_chaos"] == "true" + outputs["build_ci_needed"] = "true" if unstable or distributed else "false" + return outputs, legs + + +def render_log(plan): + """One line per suite plus the emitted gates, for the step log. + + Triage starts in the run log, so the verdict has to be readable there + without any other artifact. + """ + head = f"CI scope: mode={plan['mode']} full={'true' if plan['full'] else 'false'}" + lines = [head] + width = max((len(suite_id) for suite_id in plan["suites"]), default=0) + for suite_id, entry in plan["suites"].items(): + lines.append( + f" {'run ' if entry['run'] else 'skip'} {suite_id:<{width}} " + f"{entry['source']:<12} {entry['reason']}" + ) + outputs, _ = derive_outputs(plan) + lines.append( + " gates: " + " ".join(f"{key}={value}" for key, value in outputs.items()) + ) + for note in plan["notes"]: + lines.append(f" note: {note}") + return "\n".join(lines) + + +SUMMARY_MAX_FILES = 3 + + +def render_summary(plan): + """Markdown for the job summary: the matching evidence behind each verdict. + + The step log answers "what ran"; this answers "why", which is the question + when a pull request unexpectedly skips or starts a suite. + """ + lines = [ + f"### CI scope: mode={plan['mode']}, full={'true' if plan['full'] else 'false'}", + "", + "| suite | verdict | rule | matched files |", + "| --- | --- | --- | --- |", + ] + matching = plan.get("matching") or {} + for suite_id, entry in plan["suites"].items(): + matched = matching.get(suite_id) or [] + shown = ", ".join(f"`{name}`" for name in matched[:SUMMARY_MAX_FILES]) + if len(matched) > SUMMARY_MAX_FILES: + shown += f" +{len(matched) - SUMMARY_MAX_FILES} more" + lines.append( + f"| `{suite_id}` | {'run' if entry['run'] else 'skip'} | " + f"{entry['source']} | {shown or '—'} |" + ) + if plan["notes"]: + lines += ["", "
notes", ""] + lines += [f"- {note}" for note in plan["notes"]] + lines += ["", "
"] + return "\n".join(lines) + + +def log_annotations(plan): + """Workflow-run annotations for every degraded decision. + + A lookup failure widens the gate; it must be visible in the run list without + opening the job log, otherwise it looks like a confident verdict. + """ + annotations = [] + for note in plan["notes"]: + if note.startswith("pull request lookup failed"): + suffix = "every suite runs" + level = "warning" + else: + continue + message = f"CI scope: {note}; {suffix}"[:300] + annotations.append(f"::{level}::{message}") + return annotations + + +def emit(plan, args): + outputs, _ = derive_outputs(plan) + if args.dry_run: + print(json.dumps(outputs, indent=2)) + elif args.github_output: + with open(args.github_output, "a", encoding="utf-8") as handle: + for key, value in outputs.items(): + handle.write(f"{key}={value}\n") + if args.summary: + with open(args.summary, "a", encoding="utf-8") as handle: + handle.write(render_summary(plan) + "\n") + + +def parse_args(argv): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY", "")) + parser.add_argument("--pr", type=int) + parser.add_argument("--head-sha", default="") + parser.add_argument("--base-sha", default="") + parser.add_argument( + "--event", + choices=["pull_request", "merge_group", "schedule", "workflow_dispatch"], + ) + parser.add_argument("--github-output") + parser.add_argument( + "--summary", + help="append the markdown matching table to this file ($GITHUB_STEP_SUMMARY)", + ) + parser.add_argument("--dry-run", action="store_true") + return parser.parse_args(argv) + + +def main(argv=None): + args = parse_args(argv) + token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN") or "" + plan = build_plan(args, token) + for annotation in log_annotations(plan): + print(annotation) + print(render_log(plan)) + emit(plan, args) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/ci/ci_scope_rules.py b/.github/ci/ci_scope_rules.py new file mode 100755 index 00000000000..c7453424848 --- /dev/null +++ b/.github/ci/ci_scope_rules.py @@ -0,0 +1,277 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Policy for the integration CI admission gate. + +The rules are data and live in `.github/ci/scope-rules.toml` so they can be +reviewed and changed without touching the classifier. This module parses that +file once at import, validates it, and exposes the names `ci_scope.py` uses. + +A malformed file raises at import: the classifier job's steps are +`continue-on-error`, so a broken rules file leaves the gate outputs empty and +every gated job runs. +""" + +import argparse +import pathlib +import sys +import tomllib + +RULES_PATH = pathlib.Path(__file__).resolve().with_name("scope-rules.toml") +# `.github/ci/scope-rules.toml`: parents[0] is `.github/ci`, parents[2] the repo. +REPO_ROOT = RULES_PATH.parents[2] + + + +class RulesError(ValueError): + """The rules file is missing, malformed or internally inconsistent.""" + + +def _require(table, key, where): + if key not in table: + raise RulesError(f"{where} is missing `{key}`") + return table[key] + + +def _strings(value, where): + if not isinstance(value, list) or not all(isinstance(item, str) for item in value): + raise RulesError(f"{where} must be an array of strings") + return list(value) + + +def _trees(entries, where): + """Normalise `[[matching.*_tree]]` tables into bare path prefixes. + + Each entry carries a `reason`: the exclusions decide what CI never runs, so + the justification is part of the contract, not a comment. + """ + if not isinstance(entries, list): + raise RulesError(f"{where} must be an array of tables") + paths = [] + for index, entry in enumerate(entries): + if not isinstance(entry, dict): + raise RulesError(f"{where}[{index}] is not a table") + path = _require(entry, "path", f"{where}[{index}]") + if not isinstance(path, str) or not path: + raise RulesError(f"{where}[{index}].path must be a non-empty string") + reason = _require(entry, "reason", f"{where}[{index}]") + if not isinstance(reason, str) or not reason.strip(): + raise RulesError(f"{where}[{index}].reason must explain the exclusion") + paths.append(path) + return paths + + +def validate(data): + """Check the shape of a parsed rules document and return it normalised.""" + if not isinstance(data, dict): + raise RulesError("rules document is not a table") + matching = _require(data, "matching", "the rules file") + gates_in = _require(data, "gates", "the rules file") + groups_in = _require(data, "groups", "the rules file") + suites_in = _require(data, "suites", "the rules file") + + groups = {} + for name, paths in groups_in.items(): + groups[name] = _strings(paths, f"groups.{name}") + + def expand(entries, stack=()): + out = [] + for entry in entries: + if not entry.startswith("@"): + out.append(entry) + continue + name = entry[1:] + if name not in groups: + raise RulesError(f"unknown path group `{entry}`") + if name in stack: + raise RulesError(f"path group cycle: {' -> '.join(stack + (name,))}") + out.extend(expand(groups[name], stack + (name,))) + return out + + for name in groups: + expand([f"@{name}"]) # every group must resolve on its own + + suites = {} + for suite_id, suite in suites_in.items(): + if not isinstance(suite, dict): + raise RulesError(f"suites.{suite_id} is not a table") + gate = _require(suite, "gate", f"suites.{suite_id}") + paths = _strings( + _require(suite, "paths", f"suites.{suite_id}"), f"suites.{suite_id}.paths" + ) + description = _require(suite, "description", f"suites.{suite_id}") + if not isinstance(description, str) or not description.strip(): + raise RulesError(f"suites.{suite_id}.description is empty") + # `gate/leg` for a matrix job, or a bare gate for a single-leg job. + prefix = suite_id.split("/", 1)[0].replace("-", "_") + if prefix != gate: + raise RulesError( + f"suites.{suite_id}.gate is `{gate}`, expected `{prefix}`" + ) + expand(paths) + suites[suite_id] = { + "gate": gate, + "paths": paths, + "description": description, + } + + gate_jobs = {} + for gate, job in gates_in.items(): + if not isinstance(job, str) or not job: + raise RulesError(f"gates.{gate} must name a job id") + gate_jobs[gate] = job + used = {suite["gate"] for suite in suites.values()} + unknown = sorted(used - set(gate_jobs)) + if unknown: + raise RulesError(f"suites use gates missing from [gates]: {', '.join(unknown)}") + dead = sorted(set(gate_jobs) - used) + if dead: + raise RulesError(f"[gates] names gates no suite uses: {', '.join(dead)}") + + return { + "always_run": _strings( + _require(matching, "always_run", "matching"), "matching.always_run" + ), + "src_tree": _require(matching, "src_tree", "matching"), + "exempt_trees": _trees( + _require(matching, "exempt", "matching"), "matching.exempt" + ), + "gates": gate_jobs, + "groups": groups, + "suites": suites, + } + + +def load(path=None): + """Parse and validate the rules file.""" + path = RULES_PATH if path is None else path + try: + raw = pathlib.Path(path).read_bytes() + except OSError as err: + raise RulesError(f"cannot read {path}: {err}") from err + try: + data = tomllib.loads(raw.decode("utf-8")) + except (tomllib.TOMLDecodeError, UnicodeDecodeError) as err: + raise RulesError(f"{path} is not valid TOML: {err}") from err + return validate(data) + + +_RULES = load() + +ALWAYS_RUN = _RULES["always_run"] +SRC_TREE = _RULES["src_tree"] +EXEMPT_TREES = tuple(_RULES["exempt_trees"]) +GATE_JOBS = _RULES["gates"] +PATH_GROUPS = _RULES["groups"] +SUITES = _RULES["suites"] + + +def main(argv=None): + """Validate the rules file and print what it holds; non-zero on failure.""" + parser = argparse.ArgumentParser(description="Validate the CI scope rules.") + parser.add_argument( + "--check-paths", + action="store_true", + help="also fail when a pattern's literal prefix does not exist", + ) + args = parser.parse_args(argv) + try: + rules = load() + except RulesError as err: + print(f"scope rules invalid: {err}", file=sys.stderr) + return 1 + if args.check_paths: + missing = missing_paths(rules) + for pattern, prefix in missing: + print(f"missing path: {pattern} (nothing under {prefix})", file=sys.stderr) + return 1 if missing else 0 + print( + f"scope rules ok: {len(rules['suites'])} suites, " + f"{len(rules['groups'])} path groups, " + f"{len(rules['always_run'])} always-run entries" + ) + return 0 + + +def suite_group(suite_id): + """Return the matrix group id encoded in a suite id, or None.""" + _, _, group = suite_id.partition("/") + return group or None + + +def expand_paths(entries, groups=None): + """Expand `@group` references, including groups nested in other groups.""" + groups = PATH_GROUPS if groups is None else groups + patterns = [] + for entry in entries: + if entry.startswith("@"): + patterns.extend(expand_paths(groups[entry[1:]], groups)) + else: + patterns.append(entry) + return patterns + + +def suite_paths(suite): + """Expand `@group` references in one suite definition into globs.""" + return expand_paths(suite["paths"]) + + +def _literal_prefix(pattern): + """Return the longest leading path that the glob cannot vary. + + Everything from the first `*`, `?` or `[` onwards is variable. When that + wildcard sits inside a segment the incomplete segment is dropped, so + `src/a/foo*.rs` yields `src/a/`; a wildcard in the very first segment (for + example `*.md`) yields `""` because the whole path is variable. + """ + cuts = [pattern.find(ch) for ch in "*?[" if ch in pattern] + if not cuts: + return pattern + prefix = pattern[: min(cuts)] + if "/" not in prefix: + return "" + return prefix.rsplit("/", 1)[0] + "/" + + +def _all_patterns(rules): + """Every glob the rules mention, with `@group` references expanded.""" + patterns = [] + for key in ("always_run", "exempt_trees"): + patterns.extend(expand_paths(rules[key], rules["groups"])) + for suite in rules["suites"].values(): + patterns.extend(expand_paths(suite["paths"], rules["groups"])) + return patterns + + +def missing_paths(rules=None, root=None): + """Return `(pattern, prefix)` for patterns whose literal prefix is absent. + + A rule that names a directory nobody renamed would otherwise match nothing + forever, silently narrowing the gate. A pattern with no fixed prefix is + skipped: there is nothing on disk to check. + """ + rules = _RULES if rules is None else rules + root = pathlib.Path(REPO_ROOT if root is None else root) + missing = set() + for pattern in _all_patterns(rules): + prefix = _literal_prefix(pattern) + if prefix and not (root / prefix).exists(): + missing.add((pattern, prefix)) + return sorted(missing) + + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/ci/scope-rules.toml b/.github/ci/scope-rules.toml new file mode 100644 index 00000000000..ba2b92deda2 --- /dev/null +++ b/.github/ci/scope-rules.toml @@ -0,0 +1,284 @@ +# Rules for the integration CI admission gate. +# +# Data only: `ci_scope_rules.py` loads this file with the stdlib `tomllib`, and +# `ci_scope.py` matches diffs against it. Edit this file, not the code, when a +# suite's paths change. +# +# Conventions +# ----------- +# - `**` crosses directories, `*` does not; a trailing `/**` covers everything +# under a directory. +# - `@name` inside `suites.*.paths` expands `[groups].name`, and groups may +# reference other groups. Expansion is recursive. +# - A suite id is `/`; `gate` is the `$GITHUB_OUTPUT` +# key that switches the job, and it must equal the id prefix with `-` +# replaced by `_`. +# - Everything here is a floor, never a filter: an unmatched `src/` path runs +# every suite, and a malformed file fails the classifier job, whose steps are +# `continue-on-error`, so the gated jobs run. + +[matching] +# A changed file matching one of these runs every suite. +# Put here anything that changes what the suites build or how they start. +always_run = [ + "Cargo.toml", + "Cargo.lock", + "rust-toolchain.toml", + ".cargo/**", + "tests-fuzz/**", + "tests-integration/**", + "config/**", + "docker/**", + ".github/ci/**", + ".github/workflows/integration.yml", + ".github/actions/**", + # Both run through a composite action, so neither appears in the workflow: + # `setup-kind` starts the cluster for the distributed and chaos jobs, and + # `fuzz-test` seeds the corpus every fuzz job starts from. + ".github/scripts/kind-with-registry.sh", + ".github/scripts/prepare-fuzz-corpus.sh", + ".github/scripts/run-fuzz-targets.sh", + ".github/scripts/collect-fuzz-target-artifacts.sh", + ".github/scripts/wait-kafka-ready.sh", + ".github/scripts/pull-test-deps-images.sh", +] + +# A change under `src/` that no suite claims runs every suite: that is the net +# for a crate added, split or renamed after these rules were written. +src_tree = "src/" + +# Paths outside the gate's remit. They select no suite, and they do not count as +# "no rule covers this" either, so they never widen the gate. Each needs a +# `reason`: what CI covers them instead. +[[matching.exempt]] +path = "src/promql/" +reason = "PromQL crate: rust.yml and the promql sqlness cases own it; the fuzz suites speak SQL." + +[[matching.exempt]] +path = "src/query/src/promql/" +reason = "PromQL module inside `src/query`, which `@server` claims as `src/query/**`." + +[[matching.exempt]] +path = "src/flow/" +reason = "Flow engine: rust.yml and the flow sqlness cases own it; no fuzz suite loads it." + +# Which job each gate switches. The keys are `$GITHUB_OUTPUT` names; the values +# are job ids in `integration.yml`. The classifier cannot read YAML, so the +# mapping is data here, and a test proves both directions line up. +[gates] +fuzz_standalone = "fuzztest" +fuzz_unstable = "unstable-fuzztest" +fuzz_distributed = "distributed-fuzztest" +fuzz_chaos = "distributed-fuzztest-with-chaos" +export_import_v2 = "export-import-v2-e2e" +sqlness_kvbackend = "sqlness-kvbackend" + +[groups] +server = [ + "src/operator/**", + "src/sql/**", + "src/catalog/**", + "src/datatypes/**", + "src/table/**", + "src/store-api/**", + "src/session/**", + "src/api/**", + "src/servers/**", + "src/frontend/**", + # Also covers `src/query/src/promql/`, which is exempt (see + # `[[matching.exempt]]`): a PromQL-only change must not run the fuzz fleet. + "src/query/**", + "src/partition/**", + "src/index/**", + "src/object-store/**", + "src/cache/**", + "src/puffin/**", + "src/auth/**", + "src/client/**", + "src/log-query/**", + "src/pipeline/**", + "src/plugins/**", + "src/cmd/**", + "src/standalone/**", + "src/datanode/**", + "src/common/base/**", + "src/common/batcher/**", + "src/common/catalog/**", + "src/common/config/**", + "src/common/datasource/**", + "src/common/decimal/**", + "src/common/error/**", + "src/common/event-recorder/**", + "src/common/frontend/**", + "src/common/function/**", + "src/common/grpc/**", + "src/common/grpc-expr/**", + "src/common/greptimedb-telemetry/**", + "src/common/macro/**", + "src/common/mem-prof/**", + "src/common/memory-manager/**", + "src/common/options/**", + "src/common/plugins/**", + "src/common/pprof/**", + "src/common/query/**", + "src/common/recordbatch/**", + "src/common/runtime/**", + "src/common/session/**", + "src/common/sql/**", + "src/common/stat/**", + "src/common/substrait/**", + "src/common/telemetry/**", + "src/common/test-util/**", + "src/common/time/**", + "src/common/version/**", + "src/common/workload/**", +] + +# Storage engines and WAL. +engine = [ + "src/mito2/**", + "src/mito-codec/**", + "src/metric-engine/**", + "src/file-engine/**", + "src/log-store/**", + "src/common/wal/**", +] + +# DDL and procedure code standalone executes in-process: it builds a +# `DdlManager` and runs `LocalProcedureExecutor` without any metasrv. +meta-core = [ + "src/common/meta/**", + "src/common/procedure/**", + "src/common/procedure-test/**", +] + +# Everything that sits on the KV backend the two sqlness KvBackend legs +# exercise: the backend implementations, the metadata service and the startup +# wiring that picks a backend. +kv-backend = [ + # The runner implements `--setup-pg`/`--setup-mysql`, which only these legs pass. + "tests/runner/**", + "src/common/meta/**", + "src/meta-srv/**", + "src/meta-client/**", + "src/common/procedure/**", + "src/cmd/**", + "src/common/config/**", + "src/common/options/**", +] + +# Adds the metasrv/meta-client crates on top of the in-process DDL code. +meta = [ + "src/meta-srv/**", + "src/meta-client/**", + "@meta-core", +] + +[suites."fuzz-standalone/database-and-regular-table"] +gate = "fuzz_standalone" +paths = [ + "@server", + "@engine", + "@meta-core", +] +description = "Standalone-server black-box fuzz of CREATE DATABASE/TABLE, ALTER, INSERT over MySQL." + +[suites."fuzz-standalone/logical-table"] +gate = "fuzz_standalone" +paths = [ + "@server", + "@engine", + "@meta-core", +] +description = "Standalone logical-table DDL/INSERT fuzz (mito2 logical tables)." + +[suites."fuzz-unstable/standalone"] +gate = "fuzz_unstable" +paths = [ + "@server", + "@engine", + "@meta-core", +] +description = "Unstable standalone create-table fuzz (nix build feature)." + +[suites."fuzz-distributed/database-and-regular-table"] +gate = "fuzz_distributed" +paths = [ + "@server", + "@engine", + "@meta", +] +description = "Distributed (frontend/datanode/metasrv, remote WAL) DDL/DML fuzz." + +[suites."fuzz-distributed/logical-table"] +gate = "fuzz_distributed" +paths = [ + "@server", + "@engine", + "@meta", +] +description = "Distributed logical-table fuzz." + +[suites."fuzz-distributed/repartition"] +gate = "fuzz_distributed" +paths = [ + "@server", + "@engine", + "@meta", +] +description = "Distributed repartition ALTER TABLE + GC fuzz." + +[suites."fuzz-distributed/logical-pruning"] +gate = "fuzz_distributed" +paths = [ + "@engine", + "@meta", +] +description = "Remote-WAL logical table pruning of stale manifests." + +[suites."fuzz-chaos/migration"] +gate = "fuzz_chaos" +paths = [ + "@server", + "@engine", + "@meta", +] +description = "Region migration under Chaos Mesh, mito and metric engines." + +[suites."fuzz-chaos/failover"] +gate = "fuzz_chaos" +paths = [ + "@server", + "@engine", + "@meta", +] +description = "Region failover under Chaos Mesh, mito and metric engines." + +[suites."fuzz-chaos/repartition"] +gate = "fuzz_chaos" +paths = [ + "@server", + "@engine", + "@meta", +] +description = "Repartition + GC under Chaos Mesh." + +[suites."export-import-v2"] +gate = "export_import_v2" +paths = [ + "src/cli/**", + "@server", + "@engine", + "@meta-core", +] +description = "CLI COPY DATABASE export/import v2 end-to-end against S3." + +[suites."sqlness-kvbackend/metasrv-postgres"] +gate = "sqlness_kvbackend" +paths = ["@kv-backend"] +description = "Sqlness with PostgreSQL as the metadata KV backend." + +[suites."sqlness-kvbackend/metasrv-mysql"] +gate = "sqlness_kvbackend" +paths = ["@kv-backend"] +description = "Sqlness with MySQL as the metadata KV backend." diff --git a/.github/ci/tests/fixtures/scope-cases.toml b/.github/ci/tests/fixtures/scope-cases.toml new file mode 100644 index 00000000000..d159bd3e32d --- /dev/null +++ b/.github/ci/tests/fixtures/scope-cases.toml @@ -0,0 +1,158 @@ +# Golden cases for the admission gate. +# +# `tests/test_ci_scope.py` replays every case through the real classifier, +# so an edit to `scope-rules.toml` that changes behaviour fails with the case +# name here. The cases encode intent, not implementation: each one says which +# suites a kind of change must start. +# +# `suites` is a list of globs over suite ids; `["*"]` means every suite and +# `[]` means none. + +[[case]] +name = "server crate: the whole fuzz fleet except the pruning suite" +note = "A change to `src/api` reaches every suite that builds the server binary." +paths = [ + "src/api/src/lib.rs", +] +suites = [ + "fuzz-standalone/database-and-regular-table", + "fuzz-standalone/logical-table", + "fuzz-unstable/standalone", + "fuzz-distributed/database-and-regular-table", + "fuzz-distributed/logical-table", + "fuzz-distributed/repartition", + "fuzz-chaos/migration", + "fuzz-chaos/failover", + "fuzz-chaos/repartition", + "export-import-v2", +] + +[[case]] +name = "storage engine: every fuzz suite, including remote-WAL pruning" +note = "logical-pruning needs engine and DDL code, so an engine change starts it." +paths = [ + "src/mito2/src/lib.rs", +] +suites = [ + "fuzz-standalone/*", + "fuzz-unstable/*", + "fuzz-distributed/*", + "fuzz-chaos/*", + "export-import-v2", +] + +[[case]] +name = "in-process DDL code registers the procedures standalone executes" +paths = [ + "src/common/meta/src/lib.rs", +] +suites = [ + "fuzz-standalone/*", + "fuzz-unstable/*", + "fuzz-distributed/*", + "fuzz-chaos/*", + "export-import-v2", + "sqlness-kvbackend/*", +] + +[[case]] +name = "metasrv only: distributed and chaos families" +note = "Standalone builds a DdlManager in-process and never links a metasrv." +paths = [ + "src/meta-srv/src/discovery/lease.rs", +] +suites = [ + "fuzz-distributed/database-and-regular-table", + "fuzz-distributed/logical-table", + "fuzz-distributed/repartition", + "fuzz-distributed/logical-pruning", + "fuzz-chaos/migration", + "fuzz-chaos/failover", + "fuzz-chaos/repartition", + "sqlness-kvbackend/*", +] + +[[case]] +name = "CLI only touches the export/import path" +paths = [ + "src/cli/src/lib.rs", +] +suites = [ + "export-import-v2", +] + +[[case]] +name = "PromQL crate: no suite" +note = "rust.yml and the promql sqlness cases own it; the fuzz suites speak SQL." +paths = [ + "src/promql/src/lib.rs", +] +suites = [] + +[[case]] +name = "PromQL module inside the query crate is not fuzzable either" +note = "`@server` claims `src/query/**`, so the module must be exempt." +paths = [ + "src/query/src/promql/src/lib.rs", +] +suites = [] + +[[case]] +name = "flow engine: no suite" +paths = [ + "src/flow/src/lib.rs", +] +suites = [] + +[[case]] +name = "mixed diff: the PromQL file subtracts nothing" +note = "The engine change still starts every suite." +paths = [ + "src/mito2/src/lib.rs", + "src/promql/src/lib.rs", +] +suites = [ + "fuzz-standalone/*", + "fuzz-unstable/*", + "fuzz-distributed/*", + "fuzz-chaos/*", + "export-import-v2", +] + +[[case]] +name = "crate added after the rules were written" +note = "No suite claims it, so the gate widens instead of skipping everything." +paths = [ + "src/brand-new/src/lib.rs", +] +suites = ["*"] + +[[case]] +name = "integration harness change" +paths = [ + "tests-integration/tests/fuzz.rs", +] +suites = ["*"] + +[[case]] +name = "documentation only" +paths = [ + "docs/readme.md", +] +suites = [] + +[[case]] +name = "kind registry setup: every suite" +note = "The distributed and chaos fuzz jobs start their cluster with this script." +paths = [ + ".github/scripts/kind-with-registry.sh", +] +suites = ["*"] + +[[case]] +name = "fuzz corpus seeding: every suite" +note = "Every fuzz job seeds its corpus through the `fuzz-test` action first." +paths = [ + ".github/scripts/prepare-fuzz-corpus.sh", +] +suites = ["*"] diff --git a/.github/ci/tests/test_ci_scope.py b/.github/ci/tests/test_ci_scope.py new file mode 100755 index 00000000000..77839aaaffa --- /dev/null +++ b/.github/ci/tests/test_ci_scope.py @@ -0,0 +1,1018 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Offline tests for the integration CI admission gate.""" + +import contextlib +import fnmatch +import io +import json +import os +import re +import pathlib +import sys +import tempfile +import tomllib +import unittest +from urllib.parse import parse_qs, urlparse + +GATE_DIR = pathlib.Path(__file__).resolve().parents[1] +WORKFLOWS = GATE_DIR.parent / "workflows" +REPO_ROOT = GATE_DIR.parents[1] +sys.path.insert(0, str(GATE_DIR)) + +import ci_scope # noqa: E402 +import ci_scope_rules # noqa: E402 + +SCOPE_CASES = pathlib.Path(__file__).resolve().parent / "fixtures" / "scope-cases.toml" +from ci_scope_rules import ( # noqa: E402 + ALWAYS_RUN, + PATH_GROUPS, + SUITES, + suite_paths, +) + +def file_entry(name, patch="@@ -1 +1 @@\n- old\n+ new\n", status="modified"): + entry = { + "filename": name, + "status": status, + "additions": 1, + "deletions": 1, + } + if patch is not None: + entry["patch"] = patch + return entry + + +class FakeHTTP: + """Records requests and serves canned GitHub API responses.""" + + def __init__(self): + self.calls = [] + self.file_pages = {1: []} + self.labels = [] + self.pull_status = 200 + self.draft = False + self.head_sha = "" + self.base_sha = "" + self.file_queries = [] + + def __call__(self, method, url, headers, body=None, timeout=None): + parsed = urlparse(url) + query = parse_qs(parsed.query) + self.calls.append((method, parsed.path, query.get("page", ["1"])[0])) + if parsed.path.endswith("/files"): + self.file_queries.append(dict(query)) + page = int(query.get("page", ["1"])[0]) + per_page = int(query.get("per_page", ["100"])[0]) + items = [] + for pageno in sorted(self.file_pages): + canned = self.file_pages[pageno] + if isinstance(canned, int): + # An int stands for an HTTP error status for that page. + if pageno == page: + return canned, b"{}" + continue + items.extend(canned) + # `file_pages` is stored at per_page=100; honour the requested + # window so a probe at per_page=1 cannot read past item 1. + start = (page - 1) * per_page + return 200, json.dumps(items[start : start + per_page]).encode() + if parsed.path.endswith("/pulls/1"): + if self.pull_status != 200: + return self.pull_status, b"{}" + return 200, json.dumps( + { + "labels": [{"name": name} for name in self.labels], + "head": {"sha": self.head_sha}, + "base": {"sha": self.base_sha}, + } + ).encode() + return 404, b"{}" + + +class ScopeTestCase(unittest.TestCase): + def setUp(self): + self.fake = FakeHTTP() + self._original = ci_scope._request + ci_scope._request = self.fake + self._environ = dict(os.environ) + for key in ["CI_SCOPE_FORCE_ALL"]: + os.environ.pop(key, None) + os.environ["GITHUB_API_URL"] = "https://api.github.com" + + def tearDown(self): + ci_scope._request = self._original + os.environ.clear() + os.environ.update(self._environ) + + def plan(self, files, pages=None, event="pull_request", extra_args=(), pr=1): + if pages is None: + pages = {1: files} + self.fake.file_pages = dict(pages) + args = ci_scope.parse_args( + ["--repo", "GreptimeTeam/greptimedb", "--pr", str(pr), "--event", event] + + list(extra_args) + ) + return ci_scope.build_plan(args, "token") + + def assert_all_run(self, plan): + for suite_id, entry in plan["suites"].items(): + self.assertTrue(entry["run"], suite_id) + + +class GlobTests(unittest.TestCase): + def test_double_star_crosses_directories(self): + regex = ci_scope.glob_to_regex("src/**/*.rs") + self.assertTrue(regex.match("src/a.rs")) + self.assertTrue(regex.match("src/a/b/c.rs")) + self.assertFalse(regex.match("tests/a.rs")) + + def test_single_star_stays_in_one_segment(self): + regex = ci_scope.glob_to_regex("src/*.rs") + self.assertTrue(regex.match("src/main.rs")) + self.assertFalse(regex.match("src/cmd/main.rs")) + self.assertFalse(regex.match("src/main.toml")) + + def test_question_mark_matches_one_character(self): + regex = ci_scope.glob_to_regex("a?.rs") + self.assertTrue(regex.match("ab.rs")) + self.assertFalse(regex.match("abc.rs")) + self.assertFalse(regex.match("a/.rs")) + + def test_suffix_glob(self): + regex = ci_scope.glob_to_regex(".cargo/**") + self.assertTrue(regex.match(".cargo/config.toml")) + self.assertFalse(regex.match("cargo/config.toml")) + + def test_suite_groups_expand(self): + for group in PATH_GROUPS: + self.assertTrue(PATH_GROUPS[group]) + self.assertTrue(any( + entry == "@" + group + for suite in SUITES.values() + for entry in suite["paths"] + ), group) + paths = suite_paths(SUITES["fuzz-distributed/repartition"]) + self.assertIn("src/mito2/**", paths) + self.assertIn("src/meta-srv/**", paths) + self.assertNotIn("@server", paths) + + def test_every_gate_routes_to_an_output(self): + # Ask the producer, not a copy of its key list: a new gate must fail + # here until `derive_outputs` routes it to a gate key. + plan = { + "mode": "static", + "full": True, + "notes": [], + "suites": { + suite_id: {"run": True, "source": "static", "reason": ""} + for suite_id in SUITES + }, + } + outputs, _ = ci_scope.derive_outputs(plan) + gates = {suite["gate"] for suite in SUITES.values()} + self.assertLessEqual(gates, set(outputs)) + + def test_suite_metadata(self): + for suite_id, suite in SUITES.items(): + self.assertEqual(suite_id, suite_id.strip()) + self.assertTrue(suite["description"]) + self.assertEqual(suite_id.split("/")[0].replace("-", "_"), suite["gate"]) + if suite["gate"] in ci_scope.FUZZ_GATES: + self.assertEqual(ci_scope.suite_group(suite_id), suite_id.split("/", 1)[1]) + + +class ForcedModeTests(ScopeTestCase): + def test_merge_group_forces_without_http(self): + plan = self.plan([], event="merge_group") + self.assertEqual(plan["mode"], "forced") + self.assert_all_run(plan) + self.assertEqual(self.fake.calls, []) + + def test_schedule_and_workflow_dispatch_force(self): + for event in ("schedule", "workflow_dispatch"): + plan = self.plan([], event=event) + self.assertEqual(plan["mode"], "forced", event) + + def test_force_all_env_forces(self): + os.environ["CI_SCOPE_FORCE_ALL"] = "true" + plan = self.plan([], event="pull_request") + self.assertEqual(plan["mode"], "forced") + + def test_ci_full_label_forces(self): + self.fake.labels = ["ci:full"] + plan = self.plan([file_entry("src/mito2/src/lib.rs")]) + self.assertEqual(plan["mode"], "forced") + self.assert_all_run(plan) + + +class UncoveredChangeTests(ScopeTestCase): + def test_uncovered_crate_runs_everything(self): + # `src/common/telemetry` is mapped by the server group; a crate added + # later is not, and must widen the gate. + plan = self.plan([file_entry("src/common/brand-new/src/lib.rs")]) + self.assertTrue(plan["full"], plan["notes"]) + self.assertEqual(plan["suites"]["export-import-v2"]["source"], "uncovered") + self.assertIn("uncovered build change", " ".join(plan["notes"])) + + def test_mapped_crate_keeps_the_path_decision(self): + plan = self.plan([file_entry("src/query/src/lib.rs")]) + self.assertFalse(plan["full"]) + self.assertTrue(plan["suites"]["fuzz-standalone/logical-table"]["run"]) + # `src/query` reaches `COPY DATABASE` and the chaos scenarios too. + self.assertTrue(plan["suites"]["export-import-v2"]["run"]) + self.assertTrue(plan["suites"]["fuzz-chaos/migration"]["run"]) + + def test_exempt_trees_do_not_widen(self): + plan = self.plan( + [file_entry("src/promql/src/lib.rs"), file_entry("src/flow/src/lib.rs")] + ) + self.assertFalse(plan["full"]) + for entry in plan["suites"].values(): + self.assertFalse(entry["run"], entry) + + def test_exempt_tree_is_not_claimed_by_a_broader_glob(self): + # `@server` claims `src/query/**`; the PromQL module under it must be + # dropped before matching, and must not count as uncovered either. + plan = self.plan([file_entry("src/query/src/promql/src/lib.rs")]) + self.assertFalse(plan["full"]) + for entry in plan["suites"].values(): + self.assertFalse(entry["run"], entry) + + +class StaticDecisionTests(ScopeTestCase): + def outputs(self, plan): + return ci_scope.derive_outputs(plan)[0] + + def test_always_run_forces_everything(self): + plan = self.plan([file_entry("Cargo.lock")]) + self.assertEqual(plan["mode"], "static") + self.assertTrue(plan["full"]) + self.assert_all_run(plan) + + def test_meta_only_skips_standalone_fuzz(self): + plan = self.plan([file_entry("src/meta-srv/src/lib.rs")]) + self.assertEqual(plan["mode"], "static") + outputs, legs = ci_scope.derive_outputs(plan) + self.assertEqual(outputs["fuzz_standalone"], "false") + self.assertEqual(outputs["fuzz_unstable"], "false") + self.assertIn("migration", legs["fuzz_chaos"]) + self.assertIn("logical-pruning", legs["fuzz_distributed"]) + self.assertEqual(outputs["export_import_v2"], "false") + self.assertEqual(outputs["any_fuzz"], "true") + self.assertEqual(outputs["build_ci_needed"], "true") + + def test_flow_and_promql_skip_every_gated_suite(self): + plan = self.plan( + [file_entry("src/flow/src/lib.rs"), file_entry("src/promql/src/lib.rs")] + ) + self.assertEqual(plan["mode"], "static") + for entry in plan["suites"].values(): + self.assertFalse(entry["run"], entry) + outputs, legs = ci_scope.derive_outputs(plan) + self.assertEqual(outputs["any_fuzz"], "false") + self.assertEqual(outputs["build_ci_needed"], "false") + self.assertEqual(outputs["export_import_v2"], "false") + for gate in ci_scope.FUZZ_GATES: + self.assertEqual(outputs[gate], "false") + self.assertTrue(all(not values for values in legs.values())) + + def test_docs_only_short_circuits(self): + plan = self.plan([file_entry("docs/readme.md")]) + self.assertEqual(plan["mode"], "static") + self.assertIn("no meaningful changes", " ".join(plan["notes"])) + + def test_mapped_change_records_the_path_rule_source(self): + plan = self.plan([file_entry("src/mito2/src/lib.rs")]) + self.assertEqual(plan["mode"], "static") + self.assertEqual(plan["suites"]["fuzz-standalone/logical-table"]["source"], "static") + + def test_file_cap_truncates(self): + files = [file_entry("src/mito2/f%d.rs" % index) for index in range(300)] + extra = [file_entry("src/mito2/overflow.rs")] + plan = self.plan(files, pages={1: files[:100], 2: files[100:200], 3: files[200:], 4: extra}) + self.assertTrue(plan["full"]) + self.assert_all_run(plan) + self.assertTrue(all(entry["source"] == "always-run" for entry in plan["suites"].values())) + + def test_exactly_max_files_is_not_truncated(self): + files = [file_entry("src/promql/f%d.rs" % index) for index in range(300)] + plan = self.plan([], pages={1: files[:100], 2: files[100:200], 3: files[200:], 4: []}) + self.assertFalse(plan["full"], plan["notes"]) + self.assertNotIn("truncated", " ".join(plan["notes"])) + self.assertEqual(len(self.fake.file_queries), 4) + self.assertEqual( + self.fake.file_queries[-1], {"per_page": ["100"], "page": ["4"]} + ) + + def test_force_push_after_the_run_widens_the_gate(self): + self.fake.head_sha = "b" * 40 + plan = self.plan( + [file_entry("src/promql/src/lib.rs")], + extra_args=("--head-sha", "a" * 40, "--base-sha", "c" * 40), + ) + self.assertTrue(plan["full"]) + self.assertIn("head moved from aaaaaaaaaaaa", " ".join(plan["notes"])) + + def test_matching_revisions_keep_the_static_verdict(self): + self.fake.head_sha = "a" * 40 + self.fake.base_sha = "c" * 40 + plan = self.plan( + [file_entry("src/promql/src/lib.rs")], + extra_args=("--head-sha", "a" * 40, "--base-sha", "c" * 40), + ) + self.assertFalse(plan["full"]) + self.assertNotIn("moved", " ".join(plan["notes"])) + + def test_pull_request_is_read_once_per_run(self): + self.plan([file_entry("src/mito2/src/lib.rs")]) + pulls = [ + path + for _, path, _ in self.fake.calls + if path.endswith("/pulls/1") + ] + self.assertEqual(len(pulls), 1) + + def test_failed_completeness_probe_fails_instead_of_narrowing(self): + # Raising leaves the gate outputs empty, which the workflow treats as + # "run everything"; returning False here would silently classify a + # truncated file list as complete. + files = [file_entry("src/mito2/f%d.rs" % index) for index in range(300)] + with self.assertRaises(ci_scope.ScopeError): + self.plan( + files, + pages={1: files[:100], 2: files[100:200], 3: files[200:], 4: 500}, + ) + + def test_file_list_pagination_stops_at_short_page(self): + files = [file_entry("src/mito2/f%d.rs" % index) for index in range(120)] + self.plan(files, pages={1: files[:100], 2: files[100:]}) + pages = [ + page for _, path, page in self.fake.calls if path.endswith("/files") + ] + self.assertEqual(pages, ["1", "2"]) + + +# Crates no gated suite exercises. Changing one either runs everything through +# the always-run floor, or reaches CI that is not gated here. +UNMAPPED_CRATES = { + "src/flow": "flow execution is covered by rust.yml and sqlness", + "src/promql": "PromQL execution is covered by rust.yml and sqlness", + "tests-fuzz": "in ALWAYS_RUN", + "tests-integration": "in ALWAYS_RUN", +} + + +def workspace_crates(): + """Every workspace member directory, so nested crates are not collapsed.""" + data = tomllib.loads((REPO_ROOT / "Cargo.toml").read_text(encoding="utf-8")) + return sorted(member.rstrip("/") for member in data["workspace"]["members"]) + + +def rules_doc(): + return { + "matching": { + "always_run": ["Cargo.toml"], + "src_tree": "src/", + "exempt": [{"path": "src/flow/", "reason": "test fixture"}], + }, + "gates": {"fuzz_x": "fuzztest"}, + "groups": {"server": ["src/a/**"]}, + "suites": { + "fuzz-x/g": { + "gate": "fuzz_x", + "paths": ["@server"], + "description": "a suite", + } + }, + } + + +class RulesFileTests(unittest.TestCase): + def test_repository_rules_are_valid(self): + rules = ci_scope_rules.load() + self.assertEqual(len(rules["suites"]), 13) + + def test_unknown_group_is_rejected(self): + doc = rules_doc() + doc["suites"]["fuzz-x/g"]["paths"] = ["@nope"] + with self.assertRaises(ci_scope_rules.RulesError): + ci_scope_rules.validate(doc) + + def test_group_cycle_is_rejected(self): + doc = rules_doc() + doc["groups"]["a"] = ["@b"] + doc["groups"]["b"] = ["@a"] + with self.assertRaises(ci_scope_rules.RulesError): + ci_scope_rules.validate(doc) + + def test_gate_must_match_the_suite_prefix(self): + doc = rules_doc() + doc["suites"]["fuzz-x/g"]["gate"] = "fuzz_y" + with self.assertRaises(ci_scope_rules.RulesError): + ci_scope_rules.validate(doc) + + def test_unknown_key_in_matching_is_ignored_but_missing_is_not(self): + doc = rules_doc() + del doc["matching"]["src_tree"] + with self.assertRaises(ci_scope_rules.RulesError): + ci_scope_rules.validate(doc) + + def test_validator_entry_point_succeeds_on_the_repository_rules(self): + out = io.StringIO() + with contextlib.redirect_stdout(out): + self.assertEqual(ci_scope_rules.main([]), 0) + self.assertIn("scope rules ok", out.getvalue()) + + def test_malformed_toml_names_the_file(self): + with tempfile.TemporaryDirectory() as tmp: + path = pathlib.Path(tmp) / "rules.toml" + path.write_text("this is not toml [[[") + with self.assertRaises(ci_scope_rules.RulesError) as ctx: + ci_scope_rules.load(path) + self.assertIn("not valid TOML", str(ctx.exception)) + + +def stub_rules(**overrides): + """A rules dict in the shape `validate` returns, naming only real paths.""" + rules = { + "always_run": ["Cargo.toml"], + "src_tree": "src/", + "exempt_trees": [], + "gates": {"fuzz_x": "fuzztest"}, + "groups": {"server": ["Cargo.toml"]}, + "suites": { + "fuzz-x": { + "gate": "fuzz_x", + "paths": ["@server"], + "description": "a suite", + } + }, + } + rules.update(overrides) + return rules + + +class PathCheckTests(unittest.TestCase): + def test_repository_rules_name_only_existing_paths(self): + self.assertEqual(ci_scope_rules.missing_paths(), []) + + def test_a_vanished_path_is_reported_with_its_prefix(self): + doc = stub_rules() + doc["suites"]["fuzz-x"]["paths"] = ["@server", "src/does-not-exist/**"] + self.assertEqual( + ci_scope_rules.missing_paths(doc), + [("src/does-not-exist/**", "src/does-not-exist/")], + ) + + def test_patterns_without_a_fixed_prefix_are_skipped(self): + doc = stub_rules(groups={"server": ["*.md"]}, always_run=["*.md"]) + self.assertEqual(ci_scope_rules.missing_paths(doc), []) + + + def test_cli_reports_a_missing_path(self): + text = """\ +[matching] +always_run = ["Cargo.toml"] +src_tree = "src/" + +[[matching.exempt]] +path = "src/flow/" +reason = "test fixture" + +[[matching.exempt]] +path = "src/promql/" +reason = "test fixture" + +[gates] +fuzz_x = "fuzztest" + +[groups] +server = ["src/does-not-exist/**"] + +[suites."fuzz-x"] +gate = "fuzz_x" +paths = ["@server"] +description = "a suite" +""" + with tempfile.TemporaryDirectory() as tmp: + path = pathlib.Path(tmp) / "scope-rules.toml" + path.write_text(text) + original = ci_scope_rules.RULES_PATH + ci_scope_rules.RULES_PATH = path + err = io.StringIO() + try: + with contextlib.redirect_stderr(err): + code = ci_scope_rules.main(["--check-paths"]) + finally: + ci_scope_rules.RULES_PATH = original + self.assertEqual(code, 1) + self.assertIn("missing path: src/does-not-exist/**", err.getvalue()) + + def test_cli_succeeds_on_the_repository_rules(self): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = ci_scope_rules.main(["--check-paths"]) + self.assertEqual(code, 0) + self.assertEqual(err.getvalue(), "") + + + +class RulesReasonTests(unittest.TestCase): + def test_exclusion_without_a_reason_is_rejected(self): + doc = rules_doc() + del doc["matching"]["exempt"][0]["reason"] + with self.assertRaises(ci_scope_rules.RulesError): + ci_scope_rules.validate(doc) + + def test_blank_reason_is_rejected(self): + doc = rules_doc() + doc["matching"]["exempt"][0]["reason"] = " " + with self.assertRaises(ci_scope_rules.RulesError): + ci_scope_rules.validate(doc) + + def test_repository_rules_explain_every_exclusion(self): + raw = tomllib.loads(ci_scope_rules.RULES_PATH.read_text()) + matching = raw["matching"] + self.assertTrue(matching["exempt"]) + for entry in matching["exempt"]: + self.assertTrue(entry["reason"].strip(), entry) + + +class ScopeCaseTests(ScopeTestCase): + """Replay `fixtures/scope-cases.toml` through the real classifier.""" + + @staticmethod + def cases(): + return tomllib.loads(SCOPE_CASES.read_text())["case"] + + def test_every_case_matches_the_classifier(self): + for case in self.cases(): + with self.subTest(case=case["name"]): + plan = self.plan([file_entry(path) for path in case["paths"]]) + expected = { + suite_id + for suite_id in ci_scope.SUITES + if any( + fnmatch.fnmatchcase(suite_id, pattern) + for pattern in case["suites"] + ) + } + actual = { + suite_id + for suite_id, entry in plan["suites"].items() + if entry["run"] + } + self.assertEqual(expected, actual, plan["notes"]) + + def test_cases_cover_every_suite(self): + covered = set() + for case in self.cases(): + covered |= { + suite_id + for suite_id in ci_scope.SUITES + if any( + fnmatch.fnmatchcase(suite_id, pattern) + for pattern in case["suites"] + ) + } + self.assertEqual(covered, set(ci_scope.SUITES)) + + +class SummaryTests(ScopeTestCase): + def summary(self, plan, path): + args = ci_scope.parse_args( + ["--repo", "GreptimeTeam/greptimedb", "--pr", "1", "--summary", path] + ) + ci_scope.emit(plan, args) + return pathlib.Path(path).read_text() + + def test_summary_names_the_files_that_matched(self): + plan = self.plan([file_entry("src/mito2/src/lib.rs")]) + with tempfile.TemporaryDirectory() as tmp: + text = self.summary(plan, str(pathlib.Path(tmp) / "summary.md")) + self.assertIn("### CI scope: mode=static", text) + row = next( + line + for line in text.splitlines() + if line.startswith("| `fuzz-standalone/logical-table`") + ) + self.assertIn("| run |", row) + self.assertIn("`src/mito2/src/lib.rs`", row) + + def test_summary_marks_a_skipped_suite_without_files(self): + plan = self.plan([file_entry("src/cli/src/lib.rs")]) + with tempfile.TemporaryDirectory() as tmp: + text = self.summary(plan, str(pathlib.Path(tmp) / "summary.md")) + row = next( + line + for line in text.splitlines() + if line.startswith("| `fuzz-standalone/database-and-regular-table`") + ) + self.assertIn("| skip |", row) + self.assertTrue(row.endswith("| — |"), row) + + def test_summary_truncates_long_file_lists(self): + files = [file_entry(f"src/mito2/src/f{index}.rs") for index in range(5)] + plan = self.plan(files) + with tempfile.TemporaryDirectory() as tmp: + text = self.summary(plan, str(pathlib.Path(tmp) / "summary.md")) + self.assertIn("+2 more", text) + + def test_summary_appends_instead_of_overwriting(self): + plan = self.plan([file_entry("src/mito2/src/lib.rs")]) + with tempfile.TemporaryDirectory() as tmp: + path = pathlib.Path(tmp) / "summary.md" + path.write_text("existing\n") + self.summary(plan, str(path)) + text = path.read_text() + self.assertTrue(text.startswith("existing\n")) + self.assertEqual(text.count("### CI scope"), 1) + + def test_summary_carries_the_notes(self): + plan = self.plan([file_entry("Cargo.lock")]) + with tempfile.TemporaryDirectory() as tmp: + text = self.summary(plan, str(pathlib.Path(tmp) / "summary.md")) + self.assertIn("
notes", text) + + +class CrateCoverageTests(unittest.TestCase): + """A new crate must not silently match no suite: the runtime net would + catch it by running everything, but the rule table should be updated.""" + + def test_every_workspace_crate_is_mapped_or_excused(self): + patterns = [ + ci_scope.glob_to_regex(path) + for suite in SUITES.values() + for path in suite_paths(suite) + ] + unmapped = [ + crate + for crate in workspace_crates() + if crate not in UNMAPPED_CRATES + and not any(p.match(crate + "/src/lib.rs") for p in patterns) + ] + self.assertEqual(unmapped, [], "crate matches no suite and no excuse") + + def test_excuses_reference_existing_crates(self): + stale = sorted(set(UNMAPPED_CRATES) - set(workspace_crates())) + self.assertEqual(stale, [], "excuse points at a crate that is gone") + + +class PathRuleCoverageTests(ScopeTestCase): + """A rule must select every suite the change can actually break.""" + + def test_copy_parser_selects_export_import(self): + plan = self.plan([file_entry("src/sql/src/parsers/copy_parser.rs")]) + self.assertTrue(plan["suites"]["export-import-v2"]["run"]) + + def test_shared_ddl_selects_standalone_fuzz(self): + plan = self.plan([file_entry("src/common/meta/src/ddl/create_table.rs")]) + for suite in ( + "fuzz-standalone/database-and-regular-table", + "fuzz-standalone/logical-table", + "fuzz-unstable/standalone", + ): + self.assertTrue(plan["suites"][suite]["run"], suite) + + def test_sqlness_runner_selects_the_kvbackend_legs(self): + # `--setup-pg` / `--setup-mysql` live in the runner and only these legs + # pass them, so a change there must not skip the only job that uses it. + plan = self.plan([file_entry("tests/runner/src/server_mode.rs")]) + for suite in ("sqlness-kvbackend/metasrv-postgres", "sqlness-kvbackend/metasrv-mysql"): + self.assertTrue(plan["suites"][suite]["run"], suite) + + def test_shared_ddl_selects_export_import(self): + # The E2E creates databases, tables and views and recreates them on + # import, all through the in-process DDL manager. + plan = self.plan([file_entry("src/common/procedure/src/lib.rs")]) + self.assertTrue(plan["suites"]["export-import-v2"]["run"]) + + def test_group_references_resolve_recursively(self): + self.assertIn("src/common/meta/**", ci_scope.suite_paths(SUITES["fuzz-chaos/migration"])) + + +class OutputTests(ScopeTestCase): + def test_flat_outputs(self): + plan = self.plan([file_entry("src/mito2/src/lib.rs")]) + outputs, legs = ci_scope.derive_outputs(plan) + self.assertEqual( + set(outputs), + { + "any_fuzz", + "build_ci_needed", + "fuzz_standalone", + "fuzz_distributed", + "fuzz_chaos", + "fuzz_unstable", + "export_import_v2", + "sqlness_kvbackend", + }, + ) + for key in outputs: + self.assertIn(outputs[key], ("true", "false"), key) + self.assertEqual( + sorted(legs["fuzz_standalone"]), + ["database-and-regular-table", "logical-table"], + ) + + def test_emit_writes_the_gate_outputs(self): + import tempfile + + with tempfile.TemporaryDirectory() as tmp: + out_path = os.path.join(tmp, "github_output") + args = ci_scope.parse_args( + [ + "--repo", + "GreptimeTeam/greptimedb", + "--pr", + "1", + "--event", + "merge_group", + "--github-output", + out_path, + ] + ) + ci_scope.emit(ci_scope.build_plan(args, "token"), args) + with open(out_path, encoding="utf-8") as handle: + text = handle.read() + self.assertIn("any_fuzz=true" + "\n", text) + self.assertIn("fuzz_unstable=true", text) + self.assertIn("fuzz_chaos=true", text) + self.assertEqual(text.count("\n"), 8) + + + +class GateJobContractTests(unittest.TestCase): + """The rules say which job each gate switches; prove the YAML agrees.""" + + WORKFLOW = WORKFLOWS / "integration.yml" + + def job_blocks(self): + blocks = {} + current = None + for line in self.WORKFLOW.read_text().splitlines(): + marker = re.match(r"^ ([A-Za-z0-9_-]+):$", line) + if marker: + current = marker.group(1) + blocks[current] = [] + elif current: + blocks[current].append(line) + return {name: "\n".join(lines) for name, lines in blocks.items()} + + def test_every_gate_names_a_job_that_reads_it(self): + blocks = self.job_blocks() + for gate, job in ci_scope_rules.GATE_JOBS.items(): + self.assertIn(job, blocks, gate) + self.assertIn(f"needs.scope.outputs.{gate} != 'false'", blocks[job], gate) + + def test_every_gated_job_is_named_by_a_gate(self): + derived = {"any_fuzz", "build_ci_needed"} + declared = set(ci_scope_rules.GATE_JOBS) + for name, block in self.job_blocks().items(): + for gate in re.findall(r"needs\.scope\.outputs\.([a-z0-9_]+)", block): + self.assertIn(gate, declared | derived, name) + + def test_a_suite_gate_missing_from_the_table_is_rejected(self): + doc = rules_doc() + del doc["gates"]["fuzz_x"] + with self.assertRaises(ci_scope_rules.RulesError) as ctx: + ci_scope_rules.validate(doc) + self.assertIn("missing from [gates]", str(ctx.exception)) + + def test_gates_no_suite_uses_are_rejected(self): + doc = rules_doc() + doc["gates"]["fuzz_y"] = "fuzztest" + with self.assertRaises(ci_scope_rules.RulesError) as ctx: + ci_scope_rules.validate(doc) + self.assertIn("no suite uses", str(ctx.exception)) + + +class WorkflowContractTests(unittest.TestCase): + """Keep the YAML gate contract in sync with the classifier outputs.""" + + WORKFLOW = WORKFLOWS / "integration.yml" + + def job_blocks(self): + text = self.WORKFLOW.read_text() + blocks = {} + current = None + for line in text.splitlines(): + marker = re.match(r"^ ([A-Za-z0-9_-]+):$", line) + if marker: + current = marker.group(1) + blocks[current] = [] + elif current: + blocks[current].append(line) + return {name: "\n".join(lines) for name, lines in blocks.items()} + + def test_every_script_a_gated_job_runs_is_always_run(self): + """A script a gated job executes must widen the gate. + + A change confined to such a script would otherwise select no suite, so + the jobs that run it are skipped exactly when they are worth running. + The reference is often not in the workflow: a composite action the job + `uses` can run the script, so the search follows those actions too. + """ + action_use = re.compile(r"uses:\s*\./(\.github/actions/[A-Za-z0-9_.-]+)") + script_ref = re.compile( + r"(?:\./)?(\.github/[A-Za-z0-9_./-]+\.(?:sh|py|js|cjs|ts))\b" + ) + always = [ci_scope.glob_to_regex(pattern) for pattern in ALWAYS_RUN] + + gated = [b for b in self.job_blocks().values() if "needs.scope.outputs" in b] + pending = {ref for block in gated for ref in script_ref.findall(block)} + pending |= {a for block in gated for a in action_use.findall(block)} + seen = set() + while pending: + reference = pending.pop() + if reference in seen: + continue + seen.add(reference) + if not reference.startswith(".github/actions/"): + continue + for suffix in ("action.yml", "action.yaml"): + path = REPO_ROOT / reference / suffix + if path.exists(): + text = path.read_text() + pending |= set(script_ref.findall(text)) + pending |= set(action_use.findall(text)) + break + uncovered = sorted( + ref for ref in seen if not any(pattern.match(ref) for pattern in always) + ) + self.assertEqual(uncovered, [], "a gated job runs a script ALWAYS_RUN misses") + + def test_gates_treat_a_missing_output_as_run(self): + blocks = self.job_blocks() + gated = [ + "export-import-v2-e2e", + "prepare-fuzz-targets", + "fuzztest", + "unstable-fuzztest", + "build-greptime-ci", + "distributed-fuzztest", + "distributed-fuzztest-with-chaos", + "sqlness-kvbackend", + ] + declared = set( + re.findall( + r"^ ([a-z0-9_]+): \$\{\{ steps\.classify\.outputs\.\1 \}\}$", + blocks["scope"], + re.MULTILINE, + ) + ) + self.assertIn("any_fuzz", declared) + seen = set() + for name in gated: + self.assertIn("scope", blocks[name], name) + match = re.search( + r"needs\.scope\.outputs\.([a-z0-9_]+) != 'false'", blocks[name] + ) + self.assertTrue(match, name) + seen.add(match.group(1)) + self.assertEqual(seen, declared) + + def test_gates_override_the_implicit_success_check(self): + # A condition without a status function carries an implicit success() + # over `needs`, so a scope job that times out or dies on the runner + # would skip the gates despite the fail-open output comparison. + blocks = self.job_blocks() + for name in ( + "export-import-v2-e2e", + "prepare-fuzz-targets", + "fuzztest", + "unstable-fuzztest", + "build-greptime-ci", + "distributed-fuzztest", + "distributed-fuzztest-with-chaos", + ): + block = blocks[name] + condition = re.search(r"^ if: \$\{\{ (.*?) \}\}$", block, re.M) + self.assertTrue(condition, name) + self.assertTrue(condition.group(1).startswith("!cancelled() && "), name) + needs = re.search(r"^ needs: \[(.*?)\]$", block, re.M) + self.assertTrue(needs, name) + for dep in [item.strip() for item in needs.group(1).split(",")]: + if not dep or dep == "scope": + continue + key = f"needs['{dep}'].result" if "-" in dep else f"needs.{dep}.result" + self.assertIn(key + " == 'success'", condition.group(1), name) + + def test_scope_job_cannot_fail_and_skip_the_gates(self): + block = self.job_blocks()["scope"] + self.assertNotIn("Fail-open defaults", block) + self.assertIn("github.event.merge_group.base_sha", block) + self.assertIn("CI_SCOPE_FORCE_ALL", block) + # Every step that can fail must be continue-on-error: a failed scope job + # would make GitHub skip the gated jobs instead of running them. + steps = block.split("\n - ") + guarded = [ + step + for step in steps + if step.startswith("uses: actions/checkout") + or step.startswith("name: Classify") + ] + self.assertEqual(len(guarded), 2) + for step in guarded: + self.assertIn("continue-on-error: true", step, step.splitlines()[0]) + + + +class LogRenderingTests(ScopeTestCase): + """The step log is where a triage starts; the verdict must be there.""" + + def sample_plan(self): + return { + "mode": "static", + "head_sha": "a" * 40, + "base_sha": "b" * 40, + "full": False, + "notes": ["always-run: Cargo.lock"], + "suites": { + "fuzz-standalone/logical-table": { + "run": True, + "source": "static", + "reason": "touches mito2", + }, + "fuzz-chaos/migration": { + "run": False, + "source": "static", + "reason": "meta docs only", + }, + }, + } + + def test_header_names_mode_and_full(self): + self.assertEqual( + ci_scope.render_log(self.sample_plan()).splitlines()[0], + "CI scope: mode=static full=false", + ) + + def test_one_line_per_suite_with_source_and_reason(self): + lines = ci_scope.render_log(self.sample_plan()).splitlines() + self.assertIn(" run fuzz-standalone/logical-table", lines[1]) + self.assertIn("static", lines[1]) + self.assertIn("touches mito2", lines[1]) + self.assertIn(" skip fuzz-chaos/migration", lines[2]) + self.assertIn("meta docs only", lines[2]) + + def test_gates_and_notes_are_printed(self): + log = ci_scope.render_log(self.sample_plan()) + self.assertIn(" gates: fuzz_standalone=true", log) + self.assertIn("any_fuzz=true", log) + self.assertIn("export_import_v2=false", log) + self.assertIn(" note: always-run: Cargo.lock", log) + + def test_a_failed_pull_request_lookup_is_annotated_and_runs_everything(self): + # Drive the real failure path: the label lookup and the revision + # comparison share one request, so a 500 here must widen the gate and + # still produce the warning the annotation promises. + self.fake.pull_status = 500 + plan = self.plan([file_entry("src/promql/src/lib.rs")]) + self.assertTrue(plan["full"]) + self.assertIn("pull request lookup failed", " ".join(plan["notes"])) + annotations = ci_scope.log_annotations(plan) + self.assertEqual(len(annotations), 1) + self.assertTrue(annotations[0].startswith("::warning::"), annotations) + self.assertIn("every suite runs", annotations[0]) + + def test_main_prints_the_log(self): + self.fake.file_pages = {1: [file_entry("src/mito2/src/lib.rs")]} + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + self.assertEqual( + ci_scope.main( + [ + "--repo", + "GreptimeTeam/greptimedb", + "--pr", + "1", + "--event", + "pull_request", + "--dry-run", + ] + ), + 0, + ) + text = stdout.getvalue() + self.assertIn("CI scope: mode=", text) + self.assertIn(" run fuzz-standalone/logical-table", text) + self.assertIn("any_fuzz=true", text) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/scripts/tests/test_ci_slash.py b/.github/scripts/tests/test_ci_slash.py index c8c01ceda15..e551a4f5297 100644 --- a/.github/scripts/tests/test_ci_slash.py +++ b/.github/scripts/tests/test_ci_slash.py @@ -136,7 +136,7 @@ class CiSlashTest(unittest.TestCase): def test_fork_rerun_gates_cover_standard_jobs(self): import re - for name, count in [("rust", 7), ("integration", 5), ("checks", 5), ("docs", 3)]: + for name, count in [("rust", 7), ("integration", 7), ("checks", 6), ("docs", 3)]: workflow = (SCRIPT.parents[1] / "workflows" / (name + ".yml")).read_text() gates = re.findall(r"^ if:.*github.run_attempt > 1.*$", workflow, re.MULTILINE) self.assertEqual(len(gates), count, name) diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 3bc54372d06..b3ff212ade4 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -92,6 +92,8 @@ jobs: run: .github/scripts/run-fuzz-targets-test.sh - name: Run CI comment-command tests run: python3 .github/scripts/tests/test_ci_slash.py + - name: Run CI scope gate tests + run: python3 .github/ci/tests/test_ci_scope.py - name: Run metric snapshot benchmark tests run: python3 -m unittest discover -s .github/scripts/tests -p 'test_metric_snapshot*.py' - name: Run query-regression Python tooling tests @@ -107,6 +109,23 @@ jobs: python3 tests/perf/test_agent_observability_summary.py python3 tests/perf/test_tracesbench_summary.py + scope-rules: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name != 'pull_request' || github.event.pull_request.draft == false || github.run_attempt > 1) }} + name: CI Scope Rules + # Same runner image as the `scope` job in integration.yml: if the rules file + # stops parsing there, the gate silently runs every job, so this check exists + # to fail the pull request instead. + runs-on: ubuntu-slim + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Show the interpreter + run: python3 -VV + - name: Validate .github/ci/scope-rules.toml + run: python3 .github/ci/ci_scope_rules.py --check-paths + check: if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name != 'pull_request' || github.event.pull_request.draft == false || github.run_attempt > 1) }} name: Check diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index afce4a8c796..6106e5a2860 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -42,6 +42,57 @@ concurrency: cancel-in-progress: true jobs: + # Which jobs this gate skips, and why: .github/ci/README.md. + scope: + name: CI Scope + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name != 'pull_request' || github.event.pull_request.draft == false || github.run_attempt > 1) }} + # ubuntu-slim ships python3, git and curl; this job only classifies. + runs-on: ubuntu-slim + timeout-minutes: 10 + permissions: + contents: read + # `/pulls/{n}` and `/pulls/{n}/files`. An explicit block sets every + # unlisted permission to none, and a 403 here would silently widen every + # gate instead of classifying. + pull-requests: read + outputs: + any_fuzz: ${{ steps.classify.outputs.any_fuzz }} + build_ci_needed: ${{ steps.classify.outputs.build_ci_needed }} + fuzz_standalone: ${{ steps.classify.outputs.fuzz_standalone }} + fuzz_distributed: ${{ steps.classify.outputs.fuzz_distributed }} + fuzz_chaos: ${{ steps.classify.outputs.fuzz_chaos }} + fuzz_unstable: ${{ steps.classify.outputs.fuzz_unstable }} + export_import_v2: ${{ steps.classify.outputs.export_import_v2 }} + sqlness_kvbackend: ${{ steps.classify.outputs.sqlness_kvbackend }} + steps: + # Rules come from the pull request base (the queue base on merge_group), + # so a pull request cannot weaken its own gate. If this job or the Classify + # step fails, the outputs stay empty and every gate below treats an absent + # value as "run", which keeps CI fail-open. + - uses: actions/checkout@v4 + # A checkout failure leaves the outputs empty, which every gate treats + # as "run"; letting the step fail would skip the gates instead. + continue-on-error: true + with: + ref: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha || github.sha }} + persist-credentials: false + - name: Classify + id: classify + continue-on-error: true + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + CI_SCOPE_FORCE_ALL: ${{ vars.CI_SCOPE_FORCE_ALL || 'false' }} + run: | + args=(--repo "$GITHUB_REPOSITORY" --event "$GITHUB_EVENT_NAME" + --github-output "$GITHUB_OUTPUT" + --summary "$GITHUB_STEP_SUMMARY") + if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then + args+=(--pr "${{ github.event.pull_request.number }}" + --head-sha "${{ github.event.pull_request.head.sha }}" + --base-sha "${{ github.event.pull_request.base.sha }}") + fi + python3 .github/ci/ci_scope.py "${args[@]}" + build: if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name != 'pull_request' || github.event.pull_request.draft == false || github.run_attempt > 1) }} name: Build GreptimeDB binaries @@ -116,28 +167,12 @@ jobs: - name: "Basic" opts: "" kafka: false - postgres: false - mysql: false - name: "Remote WAL" opts: "-w kafka -k 127.0.0.1:9092" kafka: true - postgres: false - mysql: false - - name: "PostgreSQL KvBackend" - opts: "--setup-pg postgresql://greptimedb:admin@127.0.0.1:5432/postgres" - kafka: false - postgres: true - mysql: false - - name: "MySQL KvBackend" - opts: "--setup-mysql mysql://greptimedb:admin@127.0.0.1:3306/mysql" - kafka: false - postgres: false - mysql: true - name: "Flat format" opts: "--enable-flat-format" kafka: false - postgres: false - mysql: false timeout-minutes: 60 steps: - uses: actions/checkout@v4 @@ -149,16 +184,6 @@ jobs: working-directory: tests-integration/fixtures run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait kafka - - if: matrix.mode.postgres - name: Setup PostgreSQL - working-directory: tests-integration/fixtures - run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait postgres - - - if: matrix.mode.mysql - name: Setup MySQL - working-directory: tests-integration/fixtures - run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait mysql - - name: Download pre-built binaries uses: actions/download-artifact@v4 with: @@ -183,10 +208,68 @@ jobs: /tmp/sqlness* retention-days: 3 + sqlness-kvbackend: + # The two foreign KV backends are the only sqlness legs that are not + # required checks, so they are the only ones this gate may skip. They run + # the same cases as the `Basic` leg, so the diff only matters when it can + # reach the backend itself; the merge queue runs them regardless. + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && (github.event_name != 'pull_request' || github.event.pull_request.draft == false || github.run_attempt > 1) && needs.build.result == 'success' && needs.scope.outputs.sqlness_kvbackend != 'false' }} + name: Sqlness Test (${{ matrix.mode.name }}) + needs: [scope, build] + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [ ubuntu-latest ] + mode: + - name: "Metasrv on PostgreSQL" + id: "metasrv-postgres" + opts: "--setup-pg postgresql://greptimedb:admin@127.0.0.1:5432/postgres" + postgres: true + mysql: false + - name: "Metasrv on MySQL" + id: "metasrv-mysql" + opts: "--setup-mysql mysql://greptimedb:admin@127.0.0.1:3306/mysql" + postgres: false + mysql: true + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + + - if: matrix.mode.postgres + name: Setup PostgreSQL + working-directory: tests-integration/fixtures + run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait postgres + + - if: matrix.mode.mysql + name: Setup MySQL + working-directory: tests-integration/fixtures + run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait mysql + + - name: Download pre-built binaries + uses: actions/download-artifact@v4 + with: + name: bins + path: . + - name: Unzip binaries + run: tar -xvf ./bins.tar.gz + - name: Run sqlness + run: RUST_BACKTRACE=1 ./bins/sqlness-runner bare ${{ matrix.mode.opts }} -c ./tests/cases --bins-dir ./bins --preserve-state + - name: Upload sqlness logs + if: failure() + uses: actions/upload-artifact@v4 + with: + name: sqlness-logs-${{ matrix.mode.name }} + path: | + /tmp/sqlness* + retention-days: 3 + export-import-v2-e2e: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name != 'pull_request' || github.event.pull_request.draft == false || github.run_attempt > 1) }} + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && (github.event_name != 'pull_request' || github.event.pull_request.draft == false || github.run_attempt > 1) && needs.build.result == 'success' && needs.scope.outputs.export_import_v2 != 'false' }} name: Export/Import V2 E2E Test - needs: build + needs: [scope, build] runs-on: ubuntu-latest timeout-minutes: 60 steps: @@ -319,8 +402,9 @@ jobs: retention-days: 3 prepare-fuzz-targets: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command)) && (github.event_name == 'pull_request' || inputs.fuzz_profile != 'none') }} + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command) || github.event_name == 'merge_group') && (github.event_name == 'pull_request' || github.event_name == 'merge_group' || inputs.fuzz_profile != 'none') && needs.scope.outputs.any_fuzz != 'false' }} name: Prepare Fuzz Targets + needs: [scope] runs-on: ubuntu-latest timeout-minutes: 60 steps: @@ -433,9 +517,9 @@ jobs: if-no-files-found: error fuzztest: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command)) && (github.event_name == 'pull_request' || inputs.fuzz_profile == 'standalone' || inputs.fuzz_profile == 'all') }} + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command) || github.event_name == 'merge_group') && (github.event_name == 'pull_request' || github.event_name == 'merge_group' || inputs.fuzz_profile == 'standalone' || inputs.fuzz_profile == 'all') && needs.build.result == 'success' && needs['prepare-fuzz-targets'].result == 'success' && needs.scope.outputs.fuzz_standalone != 'false' }} name: Fuzz Test (Standalone, ${{ matrix.group.name }}) - needs: [build, prepare-fuzz-targets] + needs: [build, prepare-fuzz-targets, scope] runs-on: ubuntu-latest timeout-minutes: 60 strategy: @@ -514,9 +598,9 @@ jobs: artifact-name: fuzz-standalone-${{ matrix.group.id }} unstable-fuzztest: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command)) && (github.event_name == 'pull_request' || inputs.fuzz_profile == 'standalone' || inputs.fuzz_profile == 'all') }} + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command) || github.event_name == 'merge_group') && (github.event_name == 'pull_request' || github.event_name == 'merge_group' || inputs.fuzz_profile == 'standalone' || inputs.fuzz_profile == 'all') && needs['build-greptime-ci'].result == 'success' && needs['prepare-fuzz-targets'].result == 'success' && needs.scope.outputs.fuzz_unstable != 'false' }} name: Unstable Fuzz Test - needs: [build-greptime-ci, prepare-fuzz-targets] + needs: [build-greptime-ci, prepare-fuzz-targets, scope] runs-on: ubuntu-latest timeout-minutes: 60 steps: @@ -574,8 +658,9 @@ jobs: artifact-name: fuzz-unstable-standalone build-greptime-ci: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command && inputs.fuzz_profile != 'none')) }} + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command && inputs.fuzz_profile != 'none') || github.event_name == 'merge_group') && needs.scope.outputs.build_ci_needed != 'false' }} name: Build GreptimeDB binary (profile-CI) + needs: [scope] runs-on: ${{ matrix.os }} strategy: matrix: @@ -634,10 +719,10 @@ jobs: version: current distributed-fuzztest: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command)) && (github.event_name == 'pull_request' || inputs.fuzz_profile == 'distributed' || inputs.fuzz_profile == 'all') }} + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command) || github.event_name == 'merge_group') && (github.event_name == 'pull_request' || github.event_name == 'merge_group' || inputs.fuzz_profile == 'distributed' || inputs.fuzz_profile == 'all') && needs['build-greptime-ci'].result == 'success' && needs['prepare-fuzz-targets'].result == 'success' && needs.scope.outputs.fuzz_distributed != 'false' }} name: Fuzz Test (Distributed, ${{ matrix.mode.name }}, ${{ matrix.group.name }}) runs-on: ubuntu-latest - needs: [build-greptime-ci, prepare-fuzz-targets] + needs: [build-greptime-ci, prepare-fuzz-targets, scope] timeout-minutes: 60 strategy: fail-fast: false @@ -873,10 +958,10 @@ jobs: docker system prune -f distributed-fuzztest-with-chaos: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command)) && (github.event_name == 'pull_request' || inputs.fuzz_profile == 'chaos' || inputs.fuzz_profile == 'all') }} + if: ${{ !cancelled() && github.repository == 'GreptimeTeam/greptimedb' && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' && inputs.ci_command) || github.event_name == 'merge_group') && (github.event_name == 'pull_request' || github.event_name == 'merge_group' || inputs.fuzz_profile == 'chaos' || inputs.fuzz_profile == 'all') && needs['build-greptime-ci'].result == 'success' && needs['prepare-fuzz-targets'].result == 'success' && needs.scope.outputs.fuzz_chaos != 'false' }} name: Fuzz Test with Chaos (Distributed, ${{ matrix.mode.name }}, ${{ matrix.group.name }}) runs-on: ubuntu-latest - needs: [build-greptime-ci, prepare-fuzz-targets] + needs: [build-greptime-ci, prepare-fuzz-targets, scope] timeout-minutes: 60 strategy: fail-fast: false diff --git a/Makefile b/Makefile index 21c87cc3707..073a95cb1fa 100644 --- a/Makefile +++ b/Makefile @@ -249,6 +249,10 @@ fmt-check: ## Check code format. check-enterprise-license: ## Check enterprise-gated files are listed in both license configs. python3 scripts/check-enterprise-license.py +.PHONY: check-ci-scope +check-ci-scope: ## Check the CI scope rules still match existing paths. + python3 .github/ci/ci_scope_rules.py --check-paths + .PHONY: start-etcd start-etcd: ## Start single node etcd for testing purpose. docker run --rm -d --network=host -p 2379-2380:2379-2380 ${ETCD_IMAGE}