diff --git a/.agents/skills/greptimedb-fuzz-ci-failure-investigation/SKILL.md b/.agents/skills/greptimedb-fuzz-ci-failure-investigation/SKILL.md index 3bbfe49d33..227561cbc1 100644 --- a/.agents/skills/greptimedb-fuzz-ci-failure-investigation/SKILL.md +++ b/.agents/skills/greptimedb-fuzz-ci-failure-investigation/SKILL.md @@ -15,7 +15,7 @@ comment on PRs, or delete artifacts unless the user explicitly asks. ## Scope -Use this skill for fuzz CI only. In `.github/workflows/develop.yml`, fuzz failures +Use this skill for fuzz CI only. In `.github/workflows/integration.yml`, fuzz failures are the CI jobs that use `.github/actions/fuzz-test`: - `fuzztest` — standalone fuzz targets. @@ -23,35 +23,36 @@ are the CI jobs that use `.github/actions/fuzz-test`: - `distributed-fuzztest` — distributed cluster fuzz targets. - `distributed-fuzztest-with-chaos` — distributed fuzz targets with Chaos Mesh. -Standalone fuzz jobs usually only provide the GitHub Actions job log, except -`unstable-fuzztest`, which uploads the `unstable-fuzz-logs` artifact from -`/tmp/unstable-greptime/` on failure. Distributed fuzz jobs additionally export -cluster artifacts on failure. +Each matrix job runs a semantic group of targets on one prepared environment. +The group defaults to fail-fast: after the first target failure it captures that +target's diagnostics and marks the remaining targets as skipped. The workflow +strategy still uses `fail-fast: false`, so failures do not cancel other groups. -The reusable action `.github/actions/fuzz-test/action.yaml` runs: +The reusable action `.github/actions/fuzz-test/action.yaml` delegates each target +to `.github/scripts/run-fuzz-targets.sh`. CI passes `FUZZ_BIN_DIR`, so the script +runs the prebuilt target executable: ```bash -cargo fuzz run --fuzz-dir tests-fuzz -D -s none ... -- -max_total_time= + -max_total_time= \ + -artifact_prefix=/libfuzzer/ ``` -Distributed fuzz jobs are the ones that export cluster artifacts. In the current -workflow they upload these names on failure: +Without `FUZZ_BIN_DIR`, such as during local reproduction, the script falls back +to `cargo fuzz run --fuzz-dir tests-fuzz -D -s none` with the same +libFuzzer arguments. -- `fuzz-tests-kind-logs-${{ matrix.mode.name }}-${{ matrix.target }}` from - `/tmp/kind` after `kind export logs /tmp/kind`. -- `fuzz-tests-monitor-dumps-${{ matrix.mode.name }}-${{ matrix.target }}` from - `/tmp/fuzz-monitor-dumps`, collected by - `.github/scripts/collect-fuzz-monitor-artifacts.sh`. -- `fuzz-tests-csv-dumps-${{ matrix.mode.name }}-${{ matrix.target }}` from - `/tmp/greptime-fuzz-dumps` for the non-chaos distributed fuzz job. - -Example: for run `29000666156`, job `86062894741`, target `fuzz_alter_table`, and -mode `Remote WAL`, the kind logs artifact is: +Failed jobs upload one group artifact. Its stable name identifies the job kind, +mode, and group, for example: ```text -fuzz-tests-kind-logs-Remote WAL-fuzz_alter_table +fuzz-distributed-remote-wal-database-and-regular-table ``` +The artifact contains `manifest.json`, `summary.md`, and one directory per target +under `targets//`. A failed distributed target can include `fuzz.log`, +libFuzzer reproducers, CSV/SQL traces, Kind logs, monitor dumps, and Kubernetes +state. Setup failures use `targets/setup/`. + ## Inputs The normal user input is a **CI target link**: a GitHub Actions job URL for one @@ -76,16 +77,16 @@ JOB_ID=86062894741 ``` When the input is a job URL, do not ask for the target name first. Use the job id -to fetch metadata, then derive the target and mode from the job name. A distributed -fuzz job name usually contains the CI target tuple: +to fetch metadata, then derive the group and mode from the job name. A distributed +fuzz job name usually contains the CI group tuple: ```text -Fuzz Test (Distributed, , ) -Fuzz Test with Chaos (Distributed, , ) +Fuzz Test (Distributed, , ) +Fuzz Test with Chaos (Distributed, , ) ``` If only a run URL is provided, list failed jobs and choose the fuzz job matching the -target/mode. Ask only when multiple fuzz jobs are plausible and there is no target, +group/mode. Ask only when multiple fuzz jobs are plausible and there is no group, mode, artifact, or job-url clue. ## Prerequisites @@ -143,12 +144,10 @@ jq -r --arg id "$JOB_ID" ' ' jobs.json > job-summary.txt ``` -Record the fuzz job kind, matrix mode, target, failed step, event type, branch, and -head SHA. Do not assume `main` for PR or merge-queue runs. - -For the common job-link input, parse `mode` and `target` from the captured job name -and use them to construct artifact names. If parsing is inconvenient, keep the raw -job name and match artifacts by the target substring first, then by mode substring. +Record the fuzz job kind, matrix mode, group, failed step, event type, branch, and +head SHA. Do not assume `main` for PR or merge-queue runs. The failed target is +authoritative in the downloaded artifact's `manifest.json`; do not guess it from +the group name. ## 2. Download CI output for the failed job @@ -193,22 +192,15 @@ gh api "repos/$REPO/actions/runs/$RUN_ID/artifacts?per_page=100" \ jq -r '.artifacts[] | [.id, .name, .expired, .size_in_bytes] | @tsv' artifacts.json ``` -Derive the expected artifact names from the failed job name: - -- Job name shape: `Fuzz Test (Distributed, , )` or - `Fuzz Test with Chaos (Distributed, , )`. -- Expected kind artifact: `fuzz-tests-kind-logs--`. -- Expected monitor artifact: `fuzz-tests-monitor-dumps--`. -- Expected CSV artifact: `fuzz-tests-csv-dumps--` when present. - -For a CI target/job link, this derivation is mandatory: the job URL gives `JOB_ID`, -`jobs.json` gives the full matrix job name, and the matrix job name gives -``/``. Do not rely on the user to provide the artifact name. +Match the one group artifact by job kind, mode slug, and group slug. If the display +name does not make the slug obvious, list the run artifacts and choose the unique +`fuzz-*` artifact whose suffix matches the group. Do not search for separate Kind, +monitor, or CSV artifacts; these are target subdirectories in the group artifact. Download exact artifacts when possible: ```bash -ARTIFACT='fuzz-tests-kind-logs-Remote WAL-fuzz_alter_table' +ARTIFACT='fuzz-distributed-remote-wal-database-and-regular-table' gh run download "$RUN_ID" --repo "$REPO" --name "$ARTIFACT" --dir artifacts/"$ARTIFACT" ``` @@ -225,30 +217,46 @@ mkdir -p artifacts/"$ARTIFACT" unzip -oq artifact-$ARTIFACT_ID.zip -d artifacts/"$ARTIFACT" ``` -For distributed fuzz failures, download at least the matching kind logs. Also -download matching monitor dumps and CSV dumps when available. Note expired or -missing artifacts explicitly; retention is currently short (`retention-days: 3`). +After download, read the manifest before the bulk logs: + +```bash +jq -r '.[] | [.target, .status, .exit_code, .after_prior_failure, .artifact_collection] | @tsv' \ + artifacts/"$ARTIFACT"/manifest.json +``` + +Select the first `failure` entry as the primary target. Entries with +`skipped_after_failure` did not run; entries with `after_prior_failure=true` ran +against a potentially contaminated shared environment. Note expired or missing +artifacts explicitly; retention is currently short (`retention-days: 3`). ## 4. Understand artifact contents -Kind logs come from `kind export logs /tmp/kind` and usually include Kubernetes -state plus container logs. Prioritize: +Evidence for target `` is rooted at +`artifacts/$ARTIFACT/targets//`. `result.json` records timestamps, status, +exit code, fuzz budget, prior-failure provenance, and artifact collection status. +`fuzz.log` is the complete libFuzzer output from the prebuilt target executable in +CI; local fallback runs it through `cargo fuzz run`. + +Kind logs under `targets//kind/` come from an immediate failure snapshot +and usually include Kubernetes state plus container logs. Prioritize: - GreptimeDB component logs: frontend, datanode, metasrv, flownode, monitor. - Kubernetes pod descriptions/events: restarts, OOMKilled, scheduling, image pull, volume, DNS, and network issues. - External service logs: etcd, Kafka, Minio, Chaos Mesh, Kafka WAL helper. -Monitor dumps are collected by `.github/scripts/collect-fuzz-monitor-artifacts.sh`. -They may include: +Monitor dumps under `targets//monitor/` are collected by +`.github/scripts/collect-fuzz-monitor-artifacts.sh`. They may include: - `state.log`, `sql.log`, `copy.log`, `port-forward.log`; - `*.show_create_table.sql` for `_gt_logs` and OpenTelemetry trace tables; - parquet dumps for `_gt_logs`, `opentelemetry_traces`, `opentelemetry_traces_operations`, and `opentelemetry_traces_services`. -CSV dumps come from fuzz targets under `/tmp/greptime-fuzz-dumps` and can contain -target-specific generated operations or state snapshots. +CSV and SQL traces are under `targets//csv/`. LibFuzzer crash or timeout +reproducers are under `targets//libfuzzer/`. Kubernetes descriptions and +events are under `targets//kubernetes/`. Standalone service logs are under +`targets//service/` when available. ## 5. Build a layered diagnosis model @@ -402,8 +410,8 @@ the run SHA. Useful entry points: - `tests-fuzz/` — fuzz targets, operation generators, checkers, dump paths. -- `.github/actions/fuzz-test/action.yaml` — exact `cargo fuzz` invocation. -- `.github/workflows/develop.yml` — fuzz matrix and artifact upload names. +- `.github/actions/fuzz-test/action.yaml` — exact CI fuzz runner configuration. +- `.github/workflows/integration.yml` — fuzz matrix and artifact upload names. - `.github/scripts/collect-fuzz-monitor-artifacts.sh` — monitor dump contents. - `src/mito2/AGENTS.md` — storage/WAL/region failures. - `src/metric-engine/AGENTS.md` — metric/logical table failures. diff --git a/.github/actions/fuzz-test/action.yaml b/.github/actions/fuzz-test/action.yaml index e3b4970d0c..b696d8099d 100644 --- a/.github/actions/fuzz-test/action.yaml +++ b/.github/actions/fuzz-test/action.yaml @@ -1,19 +1,42 @@ name: Fuzz Test -description: 'Fuzz test given setup and service' +description: 'Run one or more fuzz targets against the prepared service' inputs: - target: - description: "The fuzz target to test" + targets: + description: "Newline-separated fuzz targets to test" + required: true + group: + description: "Stable identifier for this group of fuzz targets" required: true max-total-time: - description: "Max total time(secs)" + description: "Maximum libFuzzer runtime per target in seconds" required: true unstable: default: 'false' - description: "Enable unstable feature" + description: "Enable the unstable feature" + fail-fast: + default: 'true' + description: "Stop this group after the first failed target" + collect-cluster-artifacts: + default: 'false' + description: "Collect Kind, Kubernetes, and monitor artifacts on target failure" + artifact-root: + default: '/tmp/greptime-fuzz-artifacts' + description: "Directory that stores target-scoped diagnostic artifacts" + prebuilt-binaries-dir: + default: '' + description: "Directory containing prebuilt fuzz target executables" runs: using: composite steps: - - name: Run Fuzz Test - shell: bash - run: cargo fuzz run ${{ inputs.target }} --fuzz-dir tests-fuzz -D -s none ${{ inputs.unstable == 'true' && '--features=unstable' || '' }} -- -max_total_time=${{ inputs.max-total-time }} - + - name: Run Fuzz Tests + shell: bash + env: + FUZZ_TARGETS: ${{ inputs.targets }} + FUZZ_GROUP: ${{ inputs.group }} + FUZZ_MAX_TOTAL_TIME: ${{ inputs.max-total-time }} + FUZZ_UNSTABLE: ${{ inputs.unstable }} + FUZZ_FAIL_FAST: ${{ inputs.fail-fast }} + FUZZ_COLLECT_CLUSTER_ARTIFACTS: ${{ inputs.collect-cluster-artifacts }} + FUZZ_ARTIFACT_ROOT: ${{ inputs.artifact-root }} + FUZZ_BIN_DIR: ${{ inputs.prebuilt-binaries-dir }} + run: .github/scripts/run-fuzz-targets.sh diff --git a/.github/actions/restore-fuzz-binaries/action.yaml b/.github/actions/restore-fuzz-binaries/action.yaml new file mode 100644 index 0000000000..69d7d5c797 --- /dev/null +++ b/.github/actions/restore-fuzz-binaries/action.yaml @@ -0,0 +1,41 @@ +name: Restore Fuzz Binaries +description: Unpack and verify prebuilt fuzz target executables +inputs: + targets: + description: Newline-separated fuzz targets expected in the archive + required: true + directory: + description: Directory containing the downloaded fuzz-target-binaries.tar + required: true +runs: + using: composite + steps: + - shell: bash + env: + FUZZ_TARGETS: ${{ inputs.targets }} + FUZZ_BIN_DIR: ${{ inputs.directory }}/greptime-fuzz-binaries + FUZZ_BIN_ARCHIVE: ${{ inputs.directory }}/fuzz-target-binaries.tar + run: | + set -euo pipefail + + echo "Downloaded fuzz archive:" + ls -lh "${FUZZ_BIN_ARCHIVE}" + echo "Archive contents:" + tar -tvf "${FUZZ_BIN_ARCHIVE}" + + rm -rf "${FUZZ_BIN_DIR}" + tar -C "$(dirname "${FUZZ_BIN_DIR}")" -xf "${FUZZ_BIN_ARCHIVE}" + + echo "Restored fuzz binaries:" + find "${FUZZ_BIN_DIR}" -maxdepth 1 -type f \ + -printf '%f %s bytes mode=%m\n' | sort + + while IFS= read -r target; do + [[ -z "${target}" ]] && continue + binary="${FUZZ_BIN_DIR}/${target}" + if [[ ! -x "${binary}" ]]; then + echo "Expected executable fuzz binary is missing: ${binary}" >&2 + exit 1 + fi + file "${binary}" + done <<<"${FUZZ_TARGETS}" diff --git a/.github/actions/upload-fuzz-artifacts/action.yaml b/.github/actions/upload-fuzz-artifacts/action.yaml new file mode 100644 index 0000000000..ca0de82730 --- /dev/null +++ b/.github/actions/upload-fuzz-artifacts/action.yaml @@ -0,0 +1,50 @@ +name: Upload Fuzz Artifacts +description: 'Upload target-scoped fuzz diagnostics and link them from the job summary' +inputs: + artifact-name: + description: "Name of the fuzz diagnostic artifact" + required: true + artifact-root: + default: '/tmp/greptime-fuzz-artifacts' + description: "Directory containing the group manifest and target diagnostics" + retention-days: + default: '3' + description: "Number of days to retain the artifact" +outputs: + artifact-id: + description: "GitHub ID of the uploaded artifact" + value: ${{ steps.upload.outputs.artifact-id }} + artifact-url: + description: "URL of the uploaded artifact" + value: ${{ steps.upload.outputs.artifact-url }} + artifact-digest: + description: "SHA-256 digest of the uploaded artifact" + value: ${{ steps.upload.outputs.artifact-digest }} +runs: + using: composite + steps: + - name: Upload fuzz diagnostics + id: upload + uses: actions/upload-artifact@v4 + with: + name: ${{ inputs.artifact-name }} + path: ${{ inputs.artifact-root }} + if-no-files-found: warn + retention-days: ${{ inputs.retention-days }} + - name: Add fuzz diagnostics to job summary + shell: bash + env: + FUZZ_ARTIFACT_NAME: ${{ inputs.artifact-name }} + FUZZ_ARTIFACT_ID: ${{ steps.upload.outputs.artifact-id }} + FUZZ_ARTIFACT_URL: ${{ steps.upload.outputs.artifact-url }} + FUZZ_ARTIFACT_DIGEST: ${{ steps.upload.outputs.artifact-digest }} + FUZZ_ARTIFACT_RETENTION_DAYS: ${{ inputs.retention-days }} + run: | + { + printf '### Fuzz diagnostics\n\n' + printf -- '- Artifact: [%s](%s)\n' "${FUZZ_ARTIFACT_NAME}" "${FUZZ_ARTIFACT_URL}" + printf -- '- Artifact ID: `%s`\n' "${FUZZ_ARTIFACT_ID}" + printf -- '- SHA-256: `%s`\n' "${FUZZ_ARTIFACT_DIGEST}" + printf -- '- Retention: %s days\n' "${FUZZ_ARTIFACT_RETENTION_DAYS}" + } >>"${GITHUB_STEP_SUMMARY}" + echo "::notice title=Fuzz diagnostics uploaded::artifact=${FUZZ_ARTIFACT_NAME}, digest=${FUZZ_ARTIFACT_DIGEST}" diff --git a/.github/scripts/collect-fuzz-target-artifacts.sh b/.github/scripts/collect-fuzz-target-artifacts.sh new file mode 100755 index 0000000000..adc058fe41 --- /dev/null +++ b/.github/scripts/collect-fuzz-target-artifacts.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash + +set -uo pipefail + +if [[ "$#" -ne 2 ]]; then + echo "Usage: $0 " >&2 + exit 2 +fi + +target="$1" +target_dir="$2" +namespace="${GT_FUZZ_NS:-my-greptimedb}" +collection_log="${target_dir}/artifact-collection.log" +monitor_collector="${FUZZ_MONITOR_COLLECTOR:-.github/scripts/collect-fuzz-monitor-artifacts.sh}" +failed=false + +mkdir -p "${target_dir}" +exec > >(tee -a "${collection_log}") 2>&1 + +log() { + printf '[collect-fuzz-target-artifacts] %s\n' "$*" +} + +collect() { + local name="$1" + shift + + log "collect ${name}" + if "$@"; then + return + fi + + log "failed to collect ${name}" + failed=true +} + +write_setup_failure_manifest() { + local artifact_root + local result_path="${target_dir}/result.json" + local manifest_path + local summary_path + local collection_status + local completed_at + + artifact_root="$(dirname "$(dirname "${target_dir}")")" + manifest_path="${artifact_root}/manifest.json" + summary_path="${artifact_root}/summary.md" + completed_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + if [[ "${failed}" == true ]]; then + collection_status="failed" + else + collection_status="success" + fi + + jq -n \ + --arg target "${target}" \ + --arg group "${FUZZ_GROUP:-setup-failure}" \ + --arg git_sha "${GITHUB_SHA:-}" \ + --arg completed_at "${completed_at}" \ + --arg artifact_collection "${collection_status}" \ + '{ + target: $target, + group: $group, + git_sha: $git_sha, + status: "failure", + phase: "setup", + started_at: null, + completed_at: $completed_at, + duration_secs: null, + exit_code: null, + max_total_time_secs: null, + after_prior_failure: false, + artifact_collection: $artifact_collection + }' >"${result_path}" + jq -s '.' "${result_path}" >"${manifest_path}" + + { + printf '## Fuzz target results: `%s`\n\n' "${FUZZ_GROUP:-setup-failure}" + printf -- '- Failure phase: `setup`\n' + printf -- '- Artifact collection: `%s`\n\n' "${collection_status}" + printf '| Target | Status | Exit code | Artifacts |\n' + printf '| --- | --- | ---: | --- |\n' + printf '| `%s` | failure | - | %s |\n' "${target}" "${collection_status}" + } >"${summary_path}" + + if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then + cat "${summary_path}" >>"${GITHUB_STEP_SUMMARY}" + fi +} + +log "collect artifacts for target ${target} under ${target_dir}" + +if [[ -n "${FUZZ_SERVICE_LOG:-}" && -f "${FUZZ_SERVICE_LOG}" ]]; then + mkdir -p "${target_dir}/service" + collect "service log" cp "${FUZZ_SERVICE_LOG}" "${target_dir}/service/greptime.log" +fi + +case "${FUZZ_COLLECT_CLUSTER_ARTIFACTS:-false}" in + false) + ;; + true) + mkdir -p "${target_dir}/kind" "${target_dir}/kubernetes" "${target_dir}/monitor" + collect "Kubernetes nodes" \ + bash -c 'kubectl describe nodes >"$1" 2>&1' _ \ + "${target_dir}/kubernetes/nodes.txt" + collect "Kubernetes pods" \ + bash -c 'kubectl get pods -A -o wide >"$1" 2>&1 && kubectl describe pod -n "$2" >>"$1" 2>&1' _ \ + "${target_dir}/kubernetes/pods.txt" "${namespace}" + collect "Kubernetes events" \ + bash -c 'kubectl get events -A --sort-by=.lastTimestamp >"$1" 2>&1' _ \ + "${target_dir}/kubernetes/events.txt" + collect "Kind logs" kind export logs "${target_dir}/kind" + collect "monitor dumps" \ + env \ + GT_FUZZ_NS="${namespace}" \ + GT_FUZZ_CLUSTER="${GT_FUZZ_CLUSTER:-my-greptimedb}" \ + GT_MONITOR_HTTP_LOCAL_PORT="${GT_MONITOR_HTTP_LOCAL_PORT:-14000}" \ + GT_MONITOR_ARTIFACT_DIR="${target_dir}/monitor" \ + GT_MONITOR_SERVER_EXPORT_DIR="${GT_MONITOR_SERVER_EXPORT_DIR:-/tmp/gt-monitor-dump/${target}}" \ + bash "${monitor_collector}" + ;; + *) + log "FUZZ_COLLECT_CLUSTER_ARTIFACTS must be true or false" + exit 2 + ;; +esac + +if [[ "${target}" == setup ]]; then + write_setup_failure_manifest +fi + +if [[ "${failed}" == true ]]; then + exit 1 +fi + +log "artifact collection completed" diff --git a/.github/scripts/run-fuzz-targets-test.sh b/.github/scripts/run-fuzz-targets-test.sh new file mode 100755 index 0000000000..5e4cda6436 --- /dev/null +++ b/.github/scripts/run-fuzz-targets-test.sh @@ -0,0 +1,371 @@ +#!/usr/bin/env bash + +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +RUNNER="${SCRIPT_DIR}/run-fuzz-targets.sh" +COLLECTOR="${SCRIPT_DIR}/collect-fuzz-target-artifacts.sh" + +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +assert_eq() { + local expected="$1" + local actual="$2" + local message="$3" + [[ "${expected}" == "${actual}" ]] || \ + fail "${message}: expected=${expected@Q}, actual=${actual@Q}" +} + +assert_file() { + [[ -f "$1" ]] || fail "expected file: $1" +} + +new_fixture() { + fixture="$(mktemp -d -t fuzz-runner-test.XXXXXX)" + mkdir -p "${fixture}/bin" + + cat >"${fixture}/bin/cargo" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\t%s\t%s\n' "${GT_FUZZ_DUMP_DIR}" "$*" "${MOCK_CARGO_MARKER:-}" >>"${MOCK_CARGO_LOG}" +target="$3" +case " ${MOCK_FAIL_TARGETS:-} " in + *" ${target} "*) exit 17 ;; +esac +EOF + + cat >"${fixture}/collector" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\t%s\t%s\n' "$1" "$2" "${FUZZ_COLLECT_CLUSTER_ARTIFACTS}" >>"${MOCK_COLLECTOR_LOG}" +mkdir -p "$2/mock-artifacts" +printf 'collected\n' >"$2/mock-artifacts/state.txt" +EOF + + chmod +x "${fixture}/bin/cargo" "${fixture}/collector" +} + +cleanup_fixture() { + rm -rf "${fixture}" +} + +run_fixture() { + local targets="$1" + local fail_fast="$2" + local unstable="$3" + local fail_targets="$4" + local artifact_root="${fixture}/artifacts" + local -a runner_env=( + "PATH=${fixture}/bin:${PATH}" + "MOCK_CARGO_LOG=${fixture}/cargo.log" + "MOCK_COLLECTOR_LOG=${fixture}/collector.log" + "MOCK_FAIL_TARGETS=${fail_targets}" + "MOCK_CARGO_MARKER=fixture" + "FUZZ_TARGETS=${targets}" + "FUZZ_GROUP=test-group" + "FUZZ_MAX_TOTAL_TIME=120" + "FUZZ_UNSTABLE=${unstable}" + "FUZZ_COLLECT_CLUSTER_ARTIFACTS=true" + "FUZZ_ARTIFACT_ROOT=${artifact_root}" + "FUZZ_ARTIFACT_COLLECTOR=${fixture}/collector" + "GT_FUZZ_BINARY_PATH=${GT_FUZZ_BINARY_PATH:-}" + "GT_FUZZ_INSTANCE_ROOT_DIR=${GT_FUZZ_INSTANCE_ROOT_DIR:-}" + "GITHUB_SHA=deadbeef" + "GITHUB_STEP_SUMMARY=${fixture}/github-step-summary.md" + ) + if [[ "${fail_fast}" != unset ]]; then + runner_env+=("FUZZ_FAIL_FAST=${fail_fast}") + fi + + set +e + env -u FUZZ_FAIL_FAST "${runner_env[@]}" \ + "${RUNNER}" >"${fixture}/stdout.log" 2>&1 + fixture_status=$? + set -e +} + +run_prebuilt_fixture() { + new_fixture + mkdir -p "${fixture}/prebuilt" + cat >"${fixture}/prebuilt/fuzz_create_table" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\t%s\t%s\n' "${GT_FUZZ_DUMP_DIR}" "$*" "${MOCK_BINARY_MARKER:-}" >>"${MOCK_BINARY_LOG}" +EOF + chmod +x "${fixture}/prebuilt/fuzz_create_table" + + set +e + env \ + "MOCK_BINARY_LOG=${fixture}/binary.log" \ + "MOCK_BINARY_MARKER=fixture" \ + "FUZZ_TARGETS=fuzz_create_table" \ + "FUZZ_GROUP=test-group" \ + "FUZZ_MAX_TOTAL_TIME=120" \ + "FUZZ_UNSTABLE=false" \ + "FUZZ_FAIL_FAST=true" \ + "FUZZ_COLLECT_CLUSTER_ARTIFACTS=false" \ + "FUZZ_ARTIFACT_ROOT=${fixture}/artifacts" \ + "FUZZ_ARTIFACT_COLLECTOR=${fixture}/collector" \ + "FUZZ_BIN_DIR=${fixture}/prebuilt" \ + "GITHUB_SHA=deadbeef" \ + "GITHUB_STEP_SUMMARY=${fixture}/github-step-summary.md" \ + "${RUNNER}" >"${fixture}/stdout.log" 2>&1 + fixture_status=$? + set -e +} + +test_successful_targets_run_in_order() { + new_fixture + run_fixture $'fuzz_create_table\nfuzz_insert' false true "" + + assert_eq 0 "${fixture_status}" "successful run status" + assert_eq 2 "$(wc -l <"${fixture}/cargo.log" | tr -d ' ')" "cargo invocation count" + assert_eq \ + $'fuzz_create_table\nfuzz_insert' \ + "$(awk -F '\t' '{print $2}' "${fixture}/cargo.log" | sed -E 's/^fuzz run ([^ ]+).*/\1/')" \ + "target order" + grep -q -- '--features=unstable' "${fixture}/cargo.log" || fail "unstable feature missing" + grep -q -- '-max_total_time=120' "${fixture}/cargo.log" || fail "fuzz time missing" + grep -q -- '-artifact_prefix=.*/targets/fuzz_create_table/libfuzzer/' "${fixture}/cargo.log" || \ + fail "target-scoped libFuzzer prefix missing" + assert_eq success "$(jq -r '.[0].status' "${fixture}/artifacts/manifest.json")" "first status" + assert_eq success "$(jq -r '.[1].status' "${fixture}/artifacts/manifest.json")" "second status" + assert_file "${fixture}/artifacts/summary.md" + grep -q 'Policy: `continue-after-failure`' "${fixture}/artifacts/summary.md" || \ + fail "group policy missing from summary" + grep -q 'Results: \*\*2 passed\*\*, \*\*0 failed\*\*, \*\*0 skipped\*\* / 2 total' \ + "${fixture}/artifacts/summary.md" || fail "group counts missing from summary" + grep -q 'title=Fuzz target completed' "${fixture}/stdout.log" || \ + fail "target completion notice missing" + grep -q 'title=Fuzz group completed' "${fixture}/stdout.log" || \ + fail "group completion notice missing" + assert_file "${fixture}/github-step-summary.md" + cleanup_fixture +} + +test_prebuilt_binary_runs_without_cargo() { + run_prebuilt_fixture + + assert_eq 0 "${fixture_status}" "prebuilt run status" + [[ ! -e "${fixture}/cargo.log" ]] || fail "cargo ran for prebuilt binary" + assert_eq 1 "$(wc -l <"${fixture}/binary.log" | tr -d ' ')" "prebuilt invocation count" + grep -q -- '-max_total_time=120' "${fixture}/binary.log" || fail "prebuilt fuzz time missing" + grep -q -- '-artifact_prefix=.*/targets/fuzz_create_table/libfuzzer/' "${fixture}/binary.log" || \ + fail "prebuilt target-scoped libFuzzer prefix missing" + cleanup_fixture +} + +test_fail_fast_stops_after_first_failure() { + new_fixture + run_fixture $'fuzz_create_table\nfuzz_insert\nfuzz_alter_table' true false "fuzz_insert" + + assert_eq 1 "${fixture_status}" "fail-fast run status" + assert_eq 2 "$(wc -l <"${fixture}/cargo.log" | tr -d ' ')" "fail-fast cargo count" + assert_eq 1 "$(wc -l <"${fixture}/collector.log" | tr -d ' ')" "collector count" + assert_eq failure "$(jq -r '.[1].status' "${fixture}/artifacts/manifest.json")" "failed status" + assert_eq skipped_after_failure "$(jq -r '.[2].status' "${fixture}/artifacts/manifest.json")" "skipped status" + assert_eq true "$(jq -r '.[2].after_prior_failure' "${fixture}/artifacts/manifest.json")" "skipped provenance" + assert_file "${fixture}/artifacts/targets/fuzz_insert/mock-artifacts/state.txt" + grep -q 'title=Fuzz target failed.*target=fuzz_insert' "${fixture}/stdout.log" || \ + fail "target failure annotation missing" + grep -q 'title=Fuzz target skipped.*target=fuzz_alter_table' "${fixture}/stdout.log" || \ + fail "skipped target annotation missing" + grep -q '### Reproduce failed targets' "${fixture}/artifacts/summary.md" || \ + fail "reproduction section missing" + grep -q 'cargo fuzz run fuzz_insert' "${fixture}/artifacts/summary.md" || \ + fail "reproduction command missing" + cleanup_fixture +} + +test_fail_fast_defaults_to_true() { + new_fixture + GT_FUZZ_BINARY_PATH=./bin/greptime \ + GT_FUZZ_INSTANCE_ROOT_DIR=/tmp/greptime-fuzz-artifacts/targets/fuzz_insert/service/instance/ \ + run_fixture $'fuzz_create_table\nfuzz_insert\nfuzz_alter_table' unset true "fuzz_insert" + + assert_eq 1 "${fixture_status}" "default fail-fast run status" + assert_eq 2 "$(wc -l <"${fixture}/cargo.log" | tr -d ' ')" "default fail-fast cargo count" + assert_eq skipped_after_failure \ + "$(jq -r '.[2].status' "${fixture}/artifacts/manifest.json")" \ + "default fail-fast skipped status" + grep -q -- '--features=unstable' "${fixture}/artifacts/summary.md" || \ + fail "unstable reproduction argument missing" + grep -q -- 'GT_FUZZ_BINARY_PATH=./bin/greptime' \ + "${fixture}/artifacts/summary.md" || fail "unstable binary path missing from reproduction" + grep -q -- 'GT_FUZZ_INSTANCE_ROOT_DIR=/tmp/greptime-fuzz-artifacts/targets/fuzz_insert/service/instance/' \ + "${fixture}/artifacts/summary.md" || fail "unstable instance root missing from reproduction" + cleanup_fixture +} + +test_non_fail_fast_marks_later_targets() { + new_fixture + run_fixture $'fuzz_create_table\nfuzz_insert' false false "fuzz_create_table" + + assert_eq 1 "${fixture_status}" "non-fail-fast run status" + assert_eq 2 "$(wc -l <"${fixture}/cargo.log" | tr -d ' ')" "non-fail-fast cargo count" + assert_eq true "$(jq -r '.[1].after_prior_failure' "${fixture}/artifacts/manifest.json")" "later target provenance" + assert_eq success "$(jq -r '.[1].status' "${fixture}/artifacts/manifest.json")" "later target status" + cleanup_fixture +} + +test_invalid_configuration_fails_before_cargo() { + new_fixture + run_fixture 'fuzz target' true false "" + + assert_eq 2 "${fixture_status}" "invalid target status" + [[ ! -e "${fixture}/cargo.log" ]] || fail "cargo ran for invalid target" + cleanup_fixture + + new_fixture + run_fixture 'fuzz_create_table' sometimes false "" + + assert_eq 2 "${fixture_status}" "invalid fail-fast status" + [[ ! -e "${fixture}/cargo.log" ]] || fail "cargo ran for invalid fail-fast" + cleanup_fixture +} + +test_collector_keeps_target_scopes_separate() { + new_fixture + service_log="${fixture}/greptime.log" + printf 'service output\n' >"${service_log}" + + FUZZ_COLLECT_CLUSTER_ARTIFACTS=false \ + FUZZ_SERVICE_LOG="${service_log}" \ + "${COLLECTOR}" fuzz_insert "${fixture}/target-a" + FUZZ_COLLECT_CLUSTER_ARTIFACTS=false \ + FUZZ_SERVICE_LOG="${service_log}" \ + "${COLLECTOR}" fuzz_alter_table "${fixture}/target-b" + + assert_file "${fixture}/target-a/service/greptime.log" + assert_file "${fixture}/target-b/service/greptime.log" + assert_file "${fixture}/target-a/artifact-collection.log" + assert_file "${fixture}/target-b/artifact-collection.log" + cleanup_fixture +} + +test_cluster_collector_honors_target_scope_and_namespace() { + new_fixture + + cat >"${fixture}/bin/kubectl" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf 'kubectl %s\n' "$*" >>"${MOCK_COMMAND_LOG}" +printf 'mock kubectl output\n' +EOF + cat >"${fixture}/bin/kind" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf 'kind %s\n' "$*" >>"${MOCK_COMMAND_LOG}" +EOF + cat >"${fixture}/monitor-collector" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\t%s\t%s\t%s\n' \ + "${GT_FUZZ_NS}" "${GT_FUZZ_CLUSTER}" "${GT_MONITOR_ARTIFACT_DIR}" \ + "${GT_MONITOR_SERVER_EXPORT_DIR}" \ + >>"${MOCK_MONITOR_LOG}" +printf 'monitor output\n' >"${GT_MONITOR_ARTIFACT_DIR}/monitor.txt" +EOF + chmod +x "${fixture}/bin/kubectl" "${fixture}/bin/kind" + + PATH="${fixture}/bin:${PATH}" \ + MOCK_COMMAND_LOG="${fixture}/commands.log" \ + MOCK_MONITOR_LOG="${fixture}/monitor.log" \ + FUZZ_COLLECT_CLUSTER_ARTIFACTS=true \ + FUZZ_MONITOR_COLLECTOR="${fixture}/monitor-collector" \ + GT_FUZZ_NS=test-namespace \ + GT_FUZZ_CLUSTER=test-cluster \ + "${COLLECTOR}" fuzz_insert "${fixture}/target" + + grep -q 'kubectl describe pod -n test-namespace' "${fixture}/commands.log" || \ + fail "collector ignored configured namespace" + grep -q "kind export logs ${fixture}/target/kind" "${fixture}/commands.log" || \ + fail "kind logs are not target-scoped" + assert_eq \ + "test-namespace"$'\t'"test-cluster"$'\t'"${fixture}/target/monitor"$'\t'"/tmp/gt-monitor-dump/fuzz_insert" \ + "$(cat "${fixture}/monitor.log")" \ + "monitor collector scope" + assert_file "${fixture}/target/kubernetes/nodes.txt" + assert_file "${fixture}/target/kubernetes/pods.txt" + assert_file "${fixture}/target/kubernetes/events.txt" + assert_file "${fixture}/target/monitor/monitor.txt" + cleanup_fixture +} + +test_setup_failure_writes_artifact_contract() { + new_fixture + artifact_root="${fixture}/artifacts" + setup_summary="${fixture}/setup-step-summary.md" + + FUZZ_COLLECT_CLUSTER_ARTIFACTS=false \ + FUZZ_GROUP=setup-group \ + GITHUB_SHA=deadbeef \ + GITHUB_STEP_SUMMARY="${setup_summary}" \ + "${COLLECTOR}" setup "${artifact_root}/targets/setup" + + assert_file "${artifact_root}/targets/setup/result.json" + assert_file "${artifact_root}/manifest.json" + assert_file "${artifact_root}/summary.md" + assert_file "${setup_summary}" + assert_eq setup-group "$(jq -r '.[0].group' "${artifact_root}/manifest.json")" \ + "setup manifest group" + assert_eq setup "$(jq -r '.[0].target' "${artifact_root}/manifest.json")" \ + "setup manifest target" + assert_eq failure "$(jq -r '.[0].status' "${artifact_root}/manifest.json")" \ + "setup manifest status" + assert_eq setup "$(jq -r '.[0].phase' "${artifact_root}/manifest.json")" \ + "setup manifest phase" + grep -q 'Failure phase: `setup`' "${artifact_root}/summary.md" || \ + fail "setup failure phase missing from summary" + cmp -s "${artifact_root}/summary.md" "${setup_summary}" || \ + fail "setup summary was not appended to the job summary" + cleanup_fixture +} + +test_setup_failure_keeps_manifest_when_collection_fails() { + new_fixture + artifact_root="${fixture}/artifacts" + + cat >"${fixture}/bin/kubectl" <<'EOF' +#!/usr/bin/env bash +exit 1 +EOF + cat >"${fixture}/bin/kind" <<'EOF' +#!/usr/bin/env bash +exit 1 +EOF + chmod +x "${fixture}/bin/kubectl" "${fixture}/bin/kind" + + set +e + PATH="${fixture}/bin:${PATH}" \ + FUZZ_COLLECT_CLUSTER_ARTIFACTS=true \ + FUZZ_GROUP=setup-group \ + FUZZ_MONITOR_COLLECTOR="${fixture}/missing-monitor-collector" \ + GITHUB_STEP_SUMMARY="${fixture}/setup-step-summary.md" \ + "${COLLECTOR}" setup "${artifact_root}/targets/setup" \ + >"${fixture}/collector-stdout.log" 2>&1 + collector_status=$? + set -e + + assert_eq 1 "${collector_status}" "failed setup collection status" + assert_file "${artifact_root}/manifest.json" + assert_eq failed "$(jq -r '.[0].artifact_collection' "${artifact_root}/manifest.json")" \ + "failed setup collection marker" + cleanup_fixture +} + +test_successful_targets_run_in_order +test_prebuilt_binary_runs_without_cargo +test_fail_fast_stops_after_first_failure +test_fail_fast_defaults_to_true +test_non_fail_fast_marks_later_targets +test_invalid_configuration_fails_before_cargo +test_collector_keeps_target_scopes_separate +test_cluster_collector_honors_target_scope_and_namespace +test_setup_failure_writes_artifact_contract +test_setup_failure_keeps_manifest_when_collection_fails + +printf 'All fuzz orchestration script tests passed.\n' diff --git a/.github/scripts/run-fuzz-targets.sh b/.github/scripts/run-fuzz-targets.sh new file mode 100755 index 0000000000..226a393e16 --- /dev/null +++ b/.github/scripts/run-fuzz-targets.sh @@ -0,0 +1,272 @@ +#!/usr/bin/env bash + +set -euo pipefail + +FUZZ_ARTIFACT_ROOT="${FUZZ_ARTIFACT_ROOT:-/tmp/greptime-fuzz-artifacts}" +FUZZ_ARTIFACT_COLLECTOR="${FUZZ_ARTIFACT_COLLECTOR:-.github/scripts/collect-fuzz-target-artifacts.sh}" +FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS="${FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS:-180}" + +log() { + printf '[run-fuzz-targets] %s\n' "$*" +} + +parse_bool() { + case "$2" in + true | false) + printf -v "$1" '%s' "$2" + ;; + *) + log "$1 must be true or false, got: $2" + exit 2 + ;; + esac +} + +write_result() { + local path="$1" + local target="$2" + local status="$3" + local started_at="$4" + local completed_at="$5" + local duration_secs="$6" + local exit_code="$7" + local after_prior_failure="$8" + local artifact_collection="$9" + + jq -n \ + --arg target "${target}" \ + --arg group "${FUZZ_GROUP}" \ + --arg git_sha "${GITHUB_SHA:-}" \ + --arg status "${status}" \ + --arg started_at "${started_at}" \ + --arg completed_at "${completed_at}" \ + --arg duration_secs "${duration_secs}" \ + --arg exit_code "${exit_code}" \ + --arg max_total_time "${FUZZ_MAX_TOTAL_TIME}" \ + --argjson after_prior_failure "${after_prior_failure}" \ + --arg artifact_collection "${artifact_collection}" \ + '{ + target: $target, + group: $group, + git_sha: $git_sha, + status: $status, + started_at: (if $started_at == "" then null else $started_at end), + completed_at: (if $completed_at == "" then null else $completed_at end), + duration_secs: (if $duration_secs == "" then null else ($duration_secs | tonumber) end), + exit_code: (if $exit_code == "" then null else ($exit_code | tonumber) end), + max_total_time_secs: ($max_total_time | tonumber), + after_prior_failure: $after_prior_failure, + artifact_collection: $artifact_collection + }' >"${path}" +} + +write_summary() { + local manifest="$1" + local summary="$2" + local elapsed_secs="$3" + local total_count="$4" + local success_count="$5" + local failure_count="$6" + local skipped_count="$7" + local policy + local target reproduce_command + local -a reproduce_args + + if [[ "${fuzz_fail_fast}" == true ]]; then + policy="fail-fast" + else + policy="continue-after-failure" + fi + + { + printf '## Fuzz target results: `%s`\n\n' "${FUZZ_GROUP}" + printf -- '- Policy: `%s`\n' "${policy}" + printf -- '- Results: **%s passed**, **%s failed**, **%s skipped** / %s total\n' \ + "${success_count}" "${failure_count}" "${skipped_count}" "${total_count}" + printf -- '- Group elapsed time: **%ss**\n' "${elapsed_secs}" + if [[ -n "${GITHUB_SHA:-}" ]]; then + printf -- '- Commit: `%s`\n' "${GITHUB_SHA}" + fi + printf '\n' + printf '| Target | Status | Duration (s) | Exit code | After prior failure | Artifacts |\n' + printf '| --- | --- | ---: | ---: | --- | --- |\n' + jq -r '.[] | "| `\(.target)` | \(.status) | \(.duration_secs // "-") | \(.exit_code // "-") | \(.after_prior_failure) | \(.artifact_collection) |"' "${manifest}" + + if [[ "${failure_count}" -gt 0 ]]; then + printf '\n### Reproduce failed targets\n\n' + while IFS= read -r target; do + reproduce_args=() + if [[ "${fuzz_unstable}" == true ]]; then + if [[ -n "${GT_FUZZ_BINARY_PATH:-}" ]]; then + reproduce_args+=("GT_FUZZ_BINARY_PATH=${GT_FUZZ_BINARY_PATH}") + fi + if [[ -n "${GT_FUZZ_INSTANCE_ROOT_DIR:-}" ]]; then + reproduce_args+=("GT_FUZZ_INSTANCE_ROOT_DIR=${GT_FUZZ_INSTANCE_ROOT_DIR}") + fi + fi + reproduce_args+=(cargo fuzz run "${target}" --fuzz-dir tests-fuzz -D -s none) + if [[ "${fuzz_unstable}" == true ]]; then + reproduce_args+=(--features=unstable) + fi + reproduce_args+=(-- "-max_total_time=${FUZZ_MAX_TOTAL_TIME}") + printf -v reproduce_command '%q ' "${reproduce_args[@]}" + printf -- '- `%s`\n' "${reproduce_command% }" + done < <(jq -r '.[] | select(.status == "failure") | .target' "${manifest}") + fi + } >"${summary}" + + if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then + cat "${summary}" >>"${GITHUB_STEP_SUMMARY}" + fi +} + +: "${FUZZ_TARGETS:?FUZZ_TARGETS is required}" +: "${FUZZ_GROUP:?FUZZ_GROUP is required}" +: "${FUZZ_MAX_TOTAL_TIME:?FUZZ_MAX_TOTAL_TIME is required}" + +if [[ ! "${FUZZ_GROUP}" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then + log "FUZZ_GROUP must be a lowercase slug, got: ${FUZZ_GROUP}" + exit 2 +fi +if [[ ! "${FUZZ_MAX_TOTAL_TIME}" =~ ^[1-9][0-9]*$ ]]; then + log "FUZZ_MAX_TOTAL_TIME must be a positive integer, got: ${FUZZ_MAX_TOTAL_TIME}" + exit 2 +fi +if [[ ! "${FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS}" =~ ^[1-9][0-9]*$ ]]; then + log "FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS must be a positive integer" + exit 2 +fi + +parse_bool fuzz_unstable "${FUZZ_UNSTABLE:-false}" +parse_bool fuzz_fail_fast "${FUZZ_FAIL_FAST:-true}" +parse_bool fuzz_collect_cluster_artifacts "${FUZZ_COLLECT_CLUSTER_ARTIFACTS:-false}" + +if [[ -n "${FUZZ_BIN_DIR:-}" && ! -d "${FUZZ_BIN_DIR}" ]]; then + log "FUZZ_BIN_DIR does not exist: ${FUZZ_BIN_DIR}" + exit 2 +fi + +targets=() +while IFS= read -r target; do + target="${target%$'\r'}" + [[ -z "${target}" ]] && continue + if [[ ! "${target}" =~ ^[a-zA-Z0-9_]+$ ]]; then + log "invalid fuzz target: ${target}" + exit 2 + fi + targets+=("${target}") +done <<<"${FUZZ_TARGETS}" + +if [[ "${#targets[@]}" -eq 0 ]]; then + log "FUZZ_TARGETS contains no targets" + exit 2 +fi + +mkdir -p "${FUZZ_ARTIFACT_ROOT}/targets" +result_paths=() +failed_targets=() +prior_failure=false +group_started_epoch="$(date +%s)" + +for ((index = 0; index < ${#targets[@]}; index++)); do + target="${targets[index]}" + target_dir="${FUZZ_ARTIFACT_ROOT}/targets/${target}" + result_path="${target_dir}/result.json" + mkdir -p "${target_dir}/csv" "${target_dir}/libfuzzer" + result_paths+=("${result_path}") + + if [[ "${prior_failure}" == true && "${fuzz_fail_fast}" == true ]]; then + write_result \ + "${result_path}" "${target}" "skipped_after_failure" \ + "" "" "" "" true "not_requested" + echo "::notice title=Fuzz target skipped::target=${target}, group=${FUZZ_GROUP}, reason=prior target failure" + continue + fi + + started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + started_epoch="$(date +%s)" + echo "::group::Fuzz target: ${target}" + log "run target ${target}" + if [[ -n "${FUZZ_BIN_DIR:-}" ]]; then + fuzz_binary="${FUZZ_BIN_DIR}/${target}" + if [[ ! -x "${fuzz_binary}" ]]; then + log "prebuilt fuzz binary does not exist or is not executable: ${fuzz_binary}" + exit 2 + fi + run_args=("${fuzz_binary}" "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-artifact_prefix=${target_dir}/libfuzzer/") + else + run_args=(cargo fuzz run "${target}" --fuzz-dir tests-fuzz -D -s none) + if [[ "${fuzz_unstable}" == true ]]; then + run_args+=(--features=unstable) + fi + run_args+=(-- "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-artifact_prefix=${target_dir}/libfuzzer/") + fi + if GT_FUZZ_DUMP_DIR="${target_dir}/csv" \ + "${run_args[@]}" 2>&1 | tee "${target_dir}/fuzz.log"; then + exit_code=0 + status="success" + artifact_collection="not_requested" + else + exit_code=$? + status="failure" + artifact_collection="success" + fi + + completed_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + completed_epoch="$(date +%s)" + duration_secs=$((completed_epoch - started_epoch)) + + if [[ "${status}" == failure ]]; then + failed_targets+=("${target}") + echo "::error title=Fuzz target failed::target=${target}, group=${FUZZ_GROUP}, exit_code=${exit_code}, duration_secs=${duration_secs}; collecting diagnostics" + log "target ${target} failed with exit code ${exit_code}; collect diagnostics" + if timeout --signal=TERM --kill-after=10s \ + "${FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS}s" \ + env \ + FUZZ_COLLECT_CLUSTER_ARTIFACTS="${fuzz_collect_cluster_artifacts}" \ + "${FUZZ_ARTIFACT_COLLECTOR}" "${target}" "${target_dir}"; then + : + else + collector_exit_code=$? + if [[ "${collector_exit_code}" -eq 124 ]]; then + artifact_collection="timeout" + else + artifact_collection="failed" + fi + echo "::warning title=Fuzz artifact collection failed::target=${target}, status=${artifact_collection}, exit_code=${collector_exit_code}" + fi + fi + echo "::endgroup::" + + write_result \ + "${result_path}" "${target}" "${status}" \ + "${started_at}" "${completed_at}" "${duration_secs}" "${exit_code}" \ + "${prior_failure}" "${artifact_collection}" + + if [[ "${status}" == failure ]]; then + prior_failure=true + else + echo "::notice title=Fuzz target completed::target=${target}, group=${FUZZ_GROUP}, status=success, duration_secs=${duration_secs}, after_prior_failure=${prior_failure}" + fi +done + +manifest="${FUZZ_ARTIFACT_ROOT}/manifest.json" +summary="${FUZZ_ARTIFACT_ROOT}/summary.md" +jq -s '.' "${result_paths[@]}" >"${manifest}" +group_completed_epoch="$(date +%s)" +group_elapsed_secs=$((group_completed_epoch - group_started_epoch)) +total_count="$(jq 'length' "${manifest}")" +success_count="$(jq '[.[] | select(.status == "success")] | length' "${manifest}")" +failure_count="$(jq '[.[] | select(.status == "failure")] | length' "${manifest}")" +skipped_count="$(jq '[.[] | select(.status == "skipped_after_failure")] | length' "${manifest}")" +write_summary \ + "${manifest}" "${summary}" "${group_elapsed_secs}" \ + "${total_count}" "${success_count}" "${failure_count}" "${skipped_count}" +echo "::notice title=Fuzz group completed::group=${FUZZ_GROUP}, passed=${success_count}, failed=${failure_count}, skipped=${skipped_count}, elapsed_secs=${group_elapsed_secs}" + +if [[ "${#failed_targets[@]}" -gt 0 ]]; then + log "failed targets: ${failed_targets[*]}" + exit 1 +fi + +log "all targets passed: ${targets[*]}" diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml new file mode 100644 index 0000000000..71b1ee4c0f --- /dev/null +++ b/.github/workflows/checks.yml @@ -0,0 +1,125 @@ +on: + schedule: + - cron: "0 15 * * 1-5" + merge_group: + pull_request: + types: [ opened, synchronize, reopened, ready_for_review ] + paths-ignore: + - 'docs/**' + - 'config/**' + - '**.md' + - '.dockerignore' + - 'docker/**' + - '.gitignore' + - 'grafana/**' + - 'Makefile' + workflow_dispatch: + +name: Checks + +env: + TAPLO_VERSION: "0.9.3" + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +jobs: + check-typos-and-docs: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: Check typos and docs + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: crate-ci/typos@master + - name: Check the config docs + run: | + make config-docs && \ + git diff --name-only --exit-code ./config/config.md \ + || (echo "'config/config.md' is not up-to-date, please run 'make config-docs'." && exit 1) + + license-header-check: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + runs-on: ubuntu-latest + name: Check License Header + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: korandoru/hawkeye@v5 + - name: Check enterprise license header + uses: korandoru/hawkeye@v5 + with: + config: licenserc-enterprise.toml + + github-script-tests: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: GitHub Script Tests + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-node@v4 + with: + node-version: 24 + - name: Run Node GitHub script tests + run: node --test .github/scripts/query-regression-comment.test.cjs + - name: Setup uv + uses: astral-sh/setup-uv@v3 + with: + version: "latest" + - name: Run check-version script tests + run: uv run --no-project python .github/scripts/check-version-test.py + - name: Run fuzz orchestration script tests + run: .github/scripts/run-fuzz-targets-test.sh + + check: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: Check + runs-on: ${{ matrix.os }} + strategy: + matrix: + os: [ ubuntu-latest ] + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions-rust-lang/setup-rust-toolchain@v1 + - name: Rust Cache + uses: Swatinem/rust-cache@v2 + with: + # Shares across multiple jobs + # Shares with `Clippy` job + shared-key: "check-lint" + cache-all-crates: "true" + save-if: ${{ github.ref == 'refs/heads/main' }} + - name: Run cargo check + run: cargo check --locked --workspace --all-targets + - name: Run cargo check (all features) + run: cargo check --locked --workspace --all-targets --all-features + + toml: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: Toml Check + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions-rust-lang/setup-rust-toolchain@v1 + - name: Install taplo + uses: taiki-e/install-action@v2 + with: + tool: taplo@${{ env.TAPLO_VERSION }} + fallback: none + - name: Run taplo + run: taplo format --check diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 71812a35bf..097852be12 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -25,7 +25,7 @@ on: - 'Makefile' workflow_dispatch: -name: CI +name: Docs CI # To pass the required status check, see: # https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks @@ -49,49 +49,21 @@ jobs: persist-credentials: false - uses: korandoru/hawkeye@v5 - check: - name: Check - runs-on: ubuntu-latest - steps: - - run: 'echo "No action required"' - - fmt: - name: Rustfmt - runs-on: ubuntu-latest - steps: - - run: 'echo "No action required"' - - clippy: - name: Clippy - runs-on: ubuntu-latest - steps: - - run: 'echo "No action required"' - - check-udeps: - name: Unused Dependencies - runs-on: ubuntu-latest - steps: - - run: 'echo "No action required"' - - coverage: - runs-on: ubuntu-latest - steps: - - run: 'echo "No action required"' - - test: - runs-on: ubuntu-latest - steps: - - run: 'echo "No action required"' - - sqlness: - name: Sqlness Test (${{ matrix.mode.name }}) - runs-on: ${{ matrix.os }} + required-checks: + name: ${{ matrix.check }} + runs-on: ubuntu-slim strategy: matrix: - os: [ ubuntu-latest ] - mode: - - name: "Basic" - - name: "Remote WAL" - - name: "Flat format" + check: + - Check + - Rustfmt + - Clippy + - Unused Dependencies + - Sqlness Test (Basic) + - Sqlness Test (Remote WAL) + - test + - coverage + - Sqlness Test (Flat format) + - Compatibility Test (recent releases) steps: - run: 'echo "No action required"' diff --git a/.github/workflows/develop.yml b/.github/workflows/integration.yml similarity index 53% rename from .github/workflows/develop.yml rename to .github/workflows/integration.yml index 15792779a5..32b32d1ae9 100644 --- a/.github/workflows/develop.yml +++ b/.github/workflows/integration.yml @@ -15,107 +15,16 @@ on: - 'Makefile' workflow_dispatch: -name: CI +name: Integration CI + +env: + CARGO_FUZZ_VERSION: "0.13.2" concurrency: group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} cancel-in-progress: true jobs: - check-typos-and-docs: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Check typos and docs - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: crate-ci/typos@master - - name: Check the config docs - run: | - make config-docs && \ - git diff --name-only --exit-code ./config/config.md \ - || (echo "'config/config.md' is not up-to-date, please run 'make config-docs'." && exit 1) - - license-header-check: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - runs-on: ubuntu-latest - name: Check License Header - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: korandoru/hawkeye@v5 - - name: Check enterprise license header - uses: korandoru/hawkeye@v5 - with: - config: licenserc-enterprise.toml - - github-script-tests: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: GitHub Script Tests - runs-on: ubuntu-latest - timeout-minutes: 5 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: actions/setup-node@v4 - with: - node-version: 24 - - name: Run Node GitHub script tests - run: node --test .github/scripts/query-regression-comment.test.cjs - - name: Setup uv - uses: astral-sh/setup-uv@v3 - with: - version: "latest" - - name: Run check-version script tests - run: uv run --no-project python .github/scripts/check-version-test.py - - check: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Check - runs-on: ${{ matrix.os }} - strategy: - matrix: - os: [ ubuntu-latest ] - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - name: Rust Cache - uses: Swatinem/rust-cache@v2 - with: - # Shares across multiple jobs - # Shares with `Clippy` job - shared-key: "check-lint" - cache-all-crates: "true" - save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Run cargo check - run: cargo check --locked --workspace --all-targets - - name: Run cargo check (all features) - run: cargo check --locked --workspace --all-targets --all-features - - toml: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Toml Check - runs-on: ubuntu-latest - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - name: Install taplo - run: cargo +stable install taplo-cli --version ^0.9 --locked --force - - name: Run taplo - run: taplo format --check - build: if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} name: Build GreptimeDB binaries @@ -132,19 +41,37 @@ jobs: with: repo-token: ${{ secrets.GITHUB_TOKEN }} - uses: actions-rust-lang/setup-rust-toolchain@v1 - - uses: Swatinem/rust-cache@v2 + - name: Build Binaries Cache + id: build-binaries-cache + uses: Swatinem/rust-cache@v2 with: # Shares across multiple jobs shared-key: "build-binaries" cache-all-crates: "true" + cache-workspace-crates: "true" save-if: ${{ github.ref == 'refs/heads/main' }} + - name: Report build binaries cache lookup + shell: bash + run: | + { + printf '## Build binaries cache lookup\n\n' + printf -- '- Dependency cache: `%s`\n' "${{ steps.build-binaries-cache.outputs.cache-hit }}" + } >>"${GITHUB_STEP_SUMMARY}" - name: Install cargo-gc-bin shell: bash - run: cargo install cargo-gc-bin --force + run: | + started_epoch="$(date +%s)" + cargo install cargo-gc-bin --force + elapsed_secs=$(( $(date +%s) - started_epoch )) + printf -- '- cargo-gc-bin install: **%ss**\n' "${elapsed_secs}" >>"${GITHUB_STEP_SUMMARY}" - name: Build greptime binaries shell: bash # `cargo gc` will invoke `cargo build` with specified args - run: cargo gc -- --bin greptime --bin sqlness-runner --features "pg_kvbackend,mysql_kvbackend,vector_index" + run: | + started_epoch="$(date +%s)" + cargo gc -- --bin greptime --bin sqlness-runner --features "pg_kvbackend,mysql_kvbackend,vector_index" + elapsed_secs=$(( $(date +%s) - started_epoch )) + printf -- '- Binary build: **%ss**\n' "${elapsed_secs}" >>"${GITHUB_STEP_SUMMARY}" - name: Pack greptime binaries shell: bash run: | @@ -159,587 +86,6 @@ jobs: artifacts-dir: bins version: current - fuzztest: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Fuzz Test - needs: build - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - max-parallel: 4 - matrix: - target: [ "fuzz_create_table", "fuzz_alter_table", "fuzz_create_database", "fuzz_create_logical_table", "fuzz_alter_logical_table", "fuzz_insert", "fuzz_insert_logical_table" ] - steps: - - name: Remove unused software - run: | - echo "Disk space before:" - df -h - [[ -d /usr/share/dotnet ]] && sudo rm -rf /usr/share/dotnet - [[ -d /usr/local/lib/android ]] && sudo rm -rf /usr/local/lib/android - [[ -d /opt/ghc ]] && sudo rm -rf /opt/ghc - [[ -d /opt/hostedtoolcache/CodeQL ]] && sudo rm -rf /opt/hostedtoolcache/CodeQL - sudo docker image prune --all --force - sudo docker builder prune -a - echo "Disk space after:" - df -h - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - name: Set Rust Fuzz - shell: bash - run: | - sudo apt-get install -y libfuzzer-14-dev - rustup install nightly - cargo +nightly install cargo-fuzz cargo-gc-bin --force - - name: Download pre-built binaries - uses: actions/download-artifact@v4 - with: - name: bins - path: . - - name: Unzip binaries - run: | - tar -xvf ./bins.tar.gz - rm ./bins.tar.gz - - name: Run GreptimeDB - run: | - ./bins/greptime standalone start& - - name: Fuzz Test - uses: ./.github/actions/fuzz-test - env: - CUSTOM_LIBFUZZER_PATH: /usr/lib/llvm-14/lib/libFuzzer.a - GT_MYSQL_ADDR: 127.0.0.1:4002 - with: - target: ${{ matrix.target }} - max-total-time: 120 - - unstable-fuzztest: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Unstable Fuzz Test - needs: build-greptime-ci - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - max-parallel: 1 - matrix: - target: [ "unstable_fuzz_create_table_standalone" ] - steps: - - name: Remove unused software - run: | - echo "Disk space before:" - df -h - [[ -d /usr/share/dotnet ]] && sudo rm -rf /usr/share/dotnet - [[ -d /usr/local/lib/android ]] && sudo rm -rf /usr/local/lib/android - [[ -d /opt/ghc ]] && sudo rm -rf /opt/ghc - [[ -d /opt/hostedtoolcache/CodeQL ]] && sudo rm -rf /opt/hostedtoolcache/CodeQL - sudo docker image prune --all --force - sudo docker builder prune -a - echo "Disk space after:" - df -h - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - name: Set Rust Fuzz - shell: bash - run: | - sudo apt update && sudo apt install -y libfuzzer-14-dev - cargo install cargo-fuzz cargo-gc-bin --force - - name: Download pre-built binary - uses: actions/download-artifact@v4 - with: - name: bin - path: . - - name: Unzip binary - run: | - tar -xvf ./bin.tar.gz - rm ./bin.tar.gz - - name: Run Fuzz Test - uses: ./.github/actions/fuzz-test - env: - CUSTOM_LIBFUZZER_PATH: /usr/lib/llvm-14/lib/libFuzzer.a - GT_MYSQL_ADDR: 127.0.0.1:4002 - GT_FUZZ_BINARY_PATH: ./bin/greptime - GT_FUZZ_INSTANCE_ROOT_DIR: /tmp/unstable-greptime/ - with: - target: ${{ matrix.target }} - max-total-time: 120 - unstable: 'true' - - name: Upload unstable fuzz test logs - if: failure() - uses: actions/upload-artifact@v4 - with: - name: unstable-fuzz-logs - path: /tmp/unstable-greptime/ - retention-days: 3 - - name: Describe pods - if: failure() - shell: bash - run: | - kubectl describe pod -n my-greptimedb - - build-greptime-ci: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Build GreptimeDB binary (profile-CI) - runs-on: ${{ matrix.os }} - strategy: - matrix: - os: [ ubuntu-latest ] - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - uses: Swatinem/rust-cache@v2 - with: - # Shares across multiple jobs - shared-key: "build-greptime-ci" - cache-all-crates: "true" - save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Install cargo-gc-bin - shell: bash - run: cargo install cargo-gc-bin --force - - name: Build greptime binary - shell: bash - # `cargo gc` will invoke `cargo build` with specified args - run: cargo gc --profile ci -- --bin greptime --features "pg_kvbackend,mysql_kvbackend" - - name: Pack greptime binary - shell: bash - run: | - mkdir bin && \ - mv ./target/ci/greptime bin - - name: Print greptime binaries info - run: ls -lh bin - - name: Upload artifacts - uses: ./.github/actions/upload-artifacts - with: - artifacts-dir: bin - version: current - - distributed-fuzztest: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Fuzz Test (Distributed, ${{ matrix.mode.name }}, ${{ matrix.target }}) - runs-on: ubuntu-latest - needs: build-greptime-ci - timeout-minutes: 60 - strategy: - fail-fast: false - max-parallel: 5 - matrix: - target: [ "fuzz_create_table", "fuzz_alter_table", "fuzz_create_database", "fuzz_create_logical_table", "fuzz_alter_logical_table", "fuzz_insert", "fuzz_insert_logical_table" ] - mode: - - name: "Remote WAL" - minio: true - kafka: true - values: "with-remote-wal.yaml" - include: - - target: "fuzz_repartition_table" - mode: - name: "Local WAL mito table repartition" - minio: true - kafka: false - values: "with-minio-repartition-gc.yaml" - - target: "fuzz_repartition_metric_table" - mode: - name: "Local WAL metric table repartition" - minio: true - kafka: false - values: "with-minio-repartition-gc.yaml" - - target: "fuzz_remote_wal_logical_prune" - mode: - name: "Remote WAL logical pruning" - minio: true - kafka: true - values: "with-remote-wal-logical-prune.yaml" - kafkaWalHelper: true - steps: - - name: Remove unused software - run: | - echo "Disk space before:" - df -h - [[ -d /usr/share/dotnet ]] && sudo rm -rf /usr/share/dotnet - [[ -d /usr/local/lib/android ]] && sudo rm -rf /usr/local/lib/android - [[ -d /opt/ghc ]] && sudo rm -rf /opt/ghc - [[ -d /opt/hostedtoolcache/CodeQL ]] && sudo rm -rf /opt/hostedtoolcache/CodeQL - sudo docker image prune --all --force - sudo docker builder prune -a - echo "Disk space after:" - df -h - - uses: actions/checkout@v4 - with: - persist-credentials: false - - name: Setup Kind - uses: ./.github/actions/setup-kind - - if: matrix.mode.minio - name: Setup Minio - uses: ./.github/actions/setup-minio - - if: matrix.mode.kafka - name: Setup Kafka cluster - uses: ./.github/actions/setup-kafka-cluster - - name: Setup Etcd cluster - uses: ./.github/actions/setup-etcd-cluster - # Prepares for fuzz tests - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - name: Set Rust Fuzz - shell: bash - run: | - sudo apt-get install -y libfuzzer-14-dev - rustup install nightly - cargo +nightly install cargo-fuzz cargo-gc-bin --force - # Downloads ci image - - name: Download pre-built binariy - uses: actions/download-artifact@v4 - with: - name: bin - path: . - - name: Unzip binary - run: | - tar -xvf ./bin.tar.gz - rm ./bin.tar.gz - - name: Build and push GreptimeDB image - uses: ./.github/actions/build-and-push-ci-image - - if: matrix.mode.kafkaWalHelper - name: Build Kafka WAL helper binary - shell: bash - run: | - cargo build --bin kafka_wal_helper - mkdir -p amd64 - cp target/debug/kafka_wal_helper amd64/kafka_wal_helper - - if: matrix.mode.kafkaWalHelper - name: Build and push Kafka WAL helper image - uses: docker/build-push-action@v5 - with: - context: . - file: ./docker/ci/ubuntu/Dockerfile.kafka-wal-helper - push: true - tags: localhost:5001/greptime/kafka-wal-helper:latest - build-args: | - TARGETARCH=amd64 - TARGET_BIN=kafka_wal_helper - - name: Wait for etcd - run: | - kubectl wait \ - --for=condition=Ready \ - pod -l app.kubernetes.io/instance=etcd \ - --timeout=120s \ - -n etcd-cluster - - if: matrix.mode.minio - name: Wait for minio - run: | - kubectl wait \ - --for=condition=Ready \ - pod -l app=minio \ - --timeout=120s \ - -n minio - - if: matrix.mode.kafka - name: Wait for kafka - run: | - kubectl wait \ - --for=condition=Ready \ - pod -l app.kubernetes.io/instance=kafka \ - --timeout=180s \ - -n kafka-cluster - - if: matrix.mode.kafkaWalHelper - name: Setup Kafka WAL helper - shell: bash - run: | - kubectl -n kafka-cluster apply \ - -f .github/actions/setup-greptimedb-cluster/kafka-wal-helper.yaml - - kubectl -n kafka-cluster wait \ - --for=condition=Ready \ - pod -l app=kafka-wal-helper \ - --timeout=120s - - name: Print etcd info - shell: bash - run: kubectl get all --show-labels -n etcd-cluster - # Setup cluster for test - - name: Setup GreptimeDB cluster - uses: ./.github/actions/setup-greptimedb-cluster - with: - image-registry: localhost:5001 - values-filename: ${{ matrix.mode.values }} - - name: Port forward (mysql) - run: | - kubectl port-forward service/my-greptimedb-frontend 4002:4002 -n my-greptimedb& - - if: matrix.mode.kafkaWalHelper - name: Port forward Kafka WAL helper - run: | - kubectl port-forward service/kafka-wal-helper 8080:8080 -n kafka-cluster& - - if: matrix.mode.kafka - name: Check Kafka readiness - shell: bash - run: | - .github/scripts/wait-kafka-ready.sh - - name: Fuzz Test - uses: ./.github/actions/fuzz-test - env: - CUSTOM_LIBFUZZER_PATH: /usr/lib/llvm-14/lib/libFuzzer.a - GT_MYSQL_ADDR: 127.0.0.1:4002 - GT_KAFKA_WAL_HELPER_URL: http://127.0.0.1:8080 - GT_KAFKA_ADDR: kafka.kafka-cluster.svc.cluster.local:9092 - GT_WAL_TOPIC_PREFIX: greptimedb_wal_topic - GT_WAL_NUM_TOPICS: 3 - GT_FUZZ_OVERRIDE_DELETE_AFTER_SECS: 300 - with: - target: ${{ matrix.target }} - max-total-time: ${{ matrix.mode.kafkaWalHelper && 600 || 120 }} - - name: Describe Nodes - if: failure() - shell: bash - run: | - kubectl describe nodes - - name: Describe pod - if: failure() - shell: bash - run: | - kubectl describe pod -n my-greptimedb - - name: Export kind logs - if: failure() - shell: bash - run: | - kind export logs /tmp/kind - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v4 - with: - name: fuzz-tests-kind-logs-${{ matrix.mode.name }}-${{ matrix.target }} - path: /tmp/kind - retention-days: 3 - - name: Collect monitor dumps - if: failure() - shell: bash - env: - GT_FUZZ_NS: my-greptimedb - GT_FUZZ_CLUSTER: my-greptimedb - GT_MONITOR_HTTP_LOCAL_PORT: 14000 - GT_MONITOR_ARTIFACT_DIR: /tmp/fuzz-monitor-dumps - run: | - bash .github/scripts/collect-fuzz-monitor-artifacts.sh - - name: Upload monitor dumps - if: failure() - uses: actions/upload-artifact@v4 - with: - name: fuzz-tests-monitor-dumps-${{ matrix.mode.name }}-${{ matrix.target }} - path: /tmp/fuzz-monitor-dumps - if-no-files-found: warn - retention-days: 3 - - name: Upload CSV dumps - if: failure() - uses: actions/upload-artifact@v4 - with: - name: fuzz-tests-csv-dumps-${{ matrix.mode.name }}-${{ matrix.target }} - path: /tmp/greptime-fuzz-dumps - if-no-files-found: warn - retention-days: 3 - - name: Delete cluster - if: success() - shell: bash - run: | - kind delete cluster - docker stop $(docker ps -a -q) - docker rm $(docker ps -a -q) - docker system prune -f - - distributed-fuzztest-with-chaos: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Fuzz Test with Chaos (Distributed, ${{ matrix.mode.name }}, ${{ matrix.target }}) - runs-on: ubuntu-latest - needs: build-greptime-ci - timeout-minutes: 60 - strategy: - fail-fast: false - max-parallel: 4 - matrix: - target: ["fuzz_migrate_mito_regions", "fuzz_migrate_metric_regions", "fuzz_failover_mito_regions", "fuzz_failover_metric_regions"] - mode: - - name: "Remote WAL" - minio: true - kafka: true - values: "with-remote-wal.yaml" - include: - - target: "fuzz_migrate_mito_regions" - mode: - name: "Local WAL" - minio: true - kafka: false - values: "with-minio.yaml" - - target: "fuzz_migrate_metric_regions" - mode: - name: "Local WAL" - minio: true - kafka: false - values: "with-minio.yaml" - - target: "fuzz_repartition_table_chaos" - mode: - name: "Local WAL repartition chaos" - minio: true - kafka: false - values: "with-minio-repartition-gc.yaml" - steps: - - name: Remove unused software - run: | - echo "Disk space before:" - df -h - [[ -d /usr/share/dotnet ]] && sudo rm -rf /usr/share/dotnet - [[ -d /usr/local/lib/android ]] && sudo rm -rf /usr/local/lib/android - [[ -d /opt/ghc ]] && sudo rm -rf /opt/ghc - [[ -d /opt/hostedtoolcache/CodeQL ]] && sudo rm -rf /opt/hostedtoolcache/CodeQL - sudo docker image prune --all --force - sudo docker builder prune -a - echo "Disk space after:" - df -h - - uses: actions/checkout@v4 - with: - persist-credentials: false - - name: Setup Kind - uses: ./.github/actions/setup-kind - - name: Setup Chaos Mesh - uses: ./.github/actions/setup-chaos - - if: matrix.mode.minio - name: Setup Minio - uses: ./.github/actions/setup-minio - - if: matrix.mode.kafka - name: Setup Kafka cluster - uses: ./.github/actions/setup-kafka-cluster - - name: Setup Etcd cluster - uses: ./.github/actions/setup-etcd-cluster - # Prepares for fuzz tests - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - name: Set Rust Fuzz - shell: bash - run: | - sudo apt-get install -y libfuzzer-14-dev - rustup install nightly - cargo +nightly install cargo-fuzz cargo-gc-bin --force - # Downloads ci image - - name: Download pre-built binariy - uses: actions/download-artifact@v4 - with: - name: bin - path: . - - name: Unzip binary - run: | - tar -xvf ./bin.tar.gz - rm ./bin.tar.gz - - name: Build and push GreptimeDB image - uses: ./.github/actions/build-and-push-ci-image - - name: Wait for etcd - run: | - kubectl wait \ - --for=condition=Ready \ - pod -l app.kubernetes.io/instance=etcd \ - --timeout=120s \ - -n etcd-cluster - - if: matrix.mode.minio - name: Wait for minio - run: | - kubectl wait \ - --for=condition=Ready \ - pod -l app=minio \ - --timeout=120s \ - -n minio - - if: matrix.mode.kafka - name: Wait for kafka - run: | - kubectl wait \ - --for=condition=Ready \ - pod -l app.kubernetes.io/instance=kafka \ - --timeout=180s \ - -n kafka-cluster - - name: Print etcd info - shell: bash - run: kubectl get all --show-labels -n etcd-cluster - # Setup cluster for test - - name: Setup GreptimeDB cluster - uses: ./.github/actions/setup-greptimedb-cluster - with: - image-registry: localhost:5001 - values-filename: ${{ matrix.mode.values }} - enable-region-failover: ${{ matrix.mode.kafka }} - - name: Port forward (mysql) - run: | - kubectl port-forward service/my-greptimedb-frontend 4002:4002 -n my-greptimedb& - - if: matrix.mode.kafka - name: Check Kafka readiness - shell: bash - run: | - .github/scripts/wait-kafka-ready.sh - - name: Fuzz Test - uses: ./.github/actions/fuzz-test - env: - CUSTOM_LIBFUZZER_PATH: /usr/lib/llvm-14/lib/libFuzzer.a - GT_MYSQL_ADDR: 127.0.0.1:4002 - with: - target: ${{ matrix.target }} - max-total-time: 120 - - name: Describe Nodes - if: failure() - shell: bash - run: | - kubectl describe nodes - - name: Describe pods - if: failure() - shell: bash - run: | - kubectl describe pod -n my-greptimedb - - name: Export kind logs - if: failure() - shell: bash - run: | - kind export logs /tmp/kind - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v4 - with: - name: fuzz-tests-kind-logs-${{ matrix.mode.name }}-${{ matrix.target }} - path: /tmp/kind - retention-days: 3 - - name: Collect monitor dumps - if: failure() - shell: bash - env: - GT_FUZZ_NS: my-greptimedb - GT_FUZZ_CLUSTER: my-greptimedb - GT_MONITOR_HTTP_LOCAL_PORT: 14000 - GT_MONITOR_ARTIFACT_DIR: /tmp/fuzz-monitor-dumps - run: | - bash .github/scripts/collect-fuzz-monitor-artifacts.sh - - name: Upload monitor dumps - if: failure() - uses: actions/upload-artifact@v4 - with: - name: fuzz-tests-monitor-dumps-${{ matrix.mode.name }}-${{ matrix.target }} - path: /tmp/fuzz-monitor-dumps - if-no-files-found: warn - retention-days: 3 - - name: Delete cluster - if: success() - shell: bash - run: | - kind delete cluster - docker stop $(docker ps -a -q) - docker rm $(docker ps -a -q) - docker system prune -f - sqlness: if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} name: Sqlness Test (${{ matrix.mode.name }}) @@ -889,207 +235,6 @@ jobs: if-no-files-found: warn retention-days: 3 - fmt: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Rustfmt - runs-on: ubuntu-latest - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - with: - components: rustfmt - - name: Check format - run: make fmt-check - - clippy: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Clippy - runs-on: ubuntu-latest - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - with: - components: clippy - - name: Rust Cache - uses: Swatinem/rust-cache@v2 - with: - # Shares across multiple jobs - # Shares with `Check` job - shared-key: "check-lint" - cache-all-crates: "true" - save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Run cargo clippy - run: make clippy - - check-udeps: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Check Unused Dependencies - runs-on: ubuntu-latest - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 - - name: Rust Cache - uses: Swatinem/rust-cache@v2 - with: - shared-key: "check-udeps" - cache-all-crates: "true" - save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Install cargo-udeps - run: cargo install cargo-udeps --locked - - name: Check unused dependencies - run: make check-udeps - - conflict-check: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} - name: Check for conflict - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - name: Merge Conflict Finder - uses: olivernybroe/action-conflict-finder@v4.0 - - test: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name != 'merge_group' }} - runs-on: ubuntu-22.04-arm - timeout-minutes: 60 - needs: [conflict-check, clippy, fmt, check-udeps] - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: rui314/setup-mold@v1 - - name: Install toolchain - uses: actions-rust-lang/setup-rust-toolchain@v1 - with: - cache: false - - name: Rust Cache - uses: Swatinem/rust-cache@v2 - with: - # Shares cross multiple jobs - shared-key: "coverage-test" - cache-all-crates: "true" - save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Install latest nextest release - uses: taiki-e/install-action@nextest - - - name: Setup external services - working-directory: tests-integration/fixtures - run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait - - - name: Run nextest cases - run: cargo nextest run --workspace -F dashboard -F pg_kvbackend -F mysql_kvbackend -F vector_index - env: - CARGO_BUILD_RUSTFLAGS: "-C link-arg=-fuse-ld=mold" - RUST_BACKTRACE: 1 - RUST_MIN_STACK: 8388608 # 8MB - CARGO_INCREMENTAL: 0 - GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} - GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} - GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - GT_MINIO_BUCKET: greptime - GT_MINIO_ACCESS_KEY_ID: superpower_ci_user - GT_MINIO_ACCESS_KEY: superpower_password - GT_MINIO_REGION: us-west-2 - GT_MINIO_ENDPOINT_URL: http://127.0.0.1:9000 - GT_ETCD_TLS_ENDPOINTS: https://127.0.0.1:2378 - GT_ETCD_ENDPOINTS: http://127.0.0.1:2379 - GT_POSTGRES_ENDPOINTS: postgres://greptimedb:admin@127.0.0.1:5432/postgres - GT_POSTGRES15_ENDPOINTS: postgres://test_user:test_password@127.0.0.1:5433/postgres - GT_POSTGRES15_SCHEMA: test_schema - GT_MYSQL_ENDPOINTS: mysql://greptimedb:admin@127.0.0.1:3306/mysql - GT_KAFKA_ENDPOINTS: 127.0.0.1:9092 - GT_KAFKA_SASL_ENDPOINTS: 127.0.0.1:9093 - UNITTEST_LOG_DIR: "__unittest_logs" - - coverage: - if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'merge_group' }} - runs-on: ubuntu-22.04-8-cores - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - - uses: arduino/setup-protoc@v3 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: rui314/setup-mold@v1 - - name: Install toolchain - uses: actions-rust-lang/setup-rust-toolchain@v1 - with: - components: llvm-tools - cache: false - - name: Rust Cache - uses: Swatinem/rust-cache@v2 - with: - # Shares cross multiple jobs - shared-key: "coverage-test" - save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Install latest nextest release - uses: taiki-e/install-action@nextest - - name: Install cargo-llvm-cov - uses: taiki-e/install-action@cargo-llvm-cov - - - name: Setup external services - working-directory: tests-integration/fixtures - run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait - - - name: Run nextest cases - run: cargo llvm-cov nextest --workspace --lcov --output-path lcov.info -F dashboard -F pg_kvbackend -F mysql_kvbackend -F vector_index - env: - CARGO_BUILD_RUSTFLAGS: "-C link-arg=-fuse-ld=mold" - RUST_BACKTRACE: 1 - CARGO_INCREMENTAL: 0 - GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} - GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} - GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - GT_MINIO_BUCKET: greptime - GT_MINIO_ACCESS_KEY_ID: superpower_ci_user - GT_MINIO_ACCESS_KEY: superpower_password - GT_MINIO_REGION: us-west-2 - GT_MINIO_ENDPOINT_URL: http://127.0.0.1:9000 - GT_ETCD_TLS_ENDPOINTS: https://127.0.0.1:2378 - GT_ETCD_ENDPOINTS: http://127.0.0.1:2379 - GT_POSTGRES_ENDPOINTS: postgres://greptimedb:admin@127.0.0.1:5432/postgres - GT_POSTGRES15_ENDPOINTS: postgres://test_user:test_password@127.0.0.1:5433/postgres - GT_POSTGRES15_SCHEMA: test_schema - GT_MYSQL_ENDPOINTS: mysql://greptimedb:admin@127.0.0.1:3306/mysql - GT_KAFKA_ENDPOINTS: 127.0.0.1:9092 - GT_KAFKA_SASL_ENDPOINTS: 127.0.0.1:9093 - UNITTEST_LOG_DIR: "__unittest_logs" - - name: Codecov upload - uses: codecov/codecov-action@v4 - with: - token: ${{ secrets.CODECOV_TOKEN }} - files: ./lcov.info - flags: rust - fail_ci_if_error: false - verbose: true - run-multi-lang-tests: if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} name: Run Multi-language SDK Tests @@ -1124,3 +269,722 @@ jobs: name: compatibility-logs path: /tmp/sqlness-compat* retention-days: 3 + + prepare-fuzz-targets: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request' }} + name: Prepare Fuzz Targets + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false + - name: Fuzz Dependency Cache + id: fuzz-dependencies-cache + uses: Swatinem/rust-cache@v2 + with: + prefix-key: "v1-fuzz-${{ env.CARGO_FUZZ_VERSION }}" + shared-key: "fuzz-dependencies" + workspaces: ". -> target" + cache-targets: "true" + cache-all-crates: "true" + save-if: ${{ github.event_name == 'pull_request' }} + - name: Look up prebuilt fuzz binaries + id: fuzz-binaries + uses: actions/cache/restore@v4 + with: + path: ${{ runner.temp }}/greptime-fuzz-binaries + key: v1-fuzz-binaries-${{ env.CARGO_FUZZ_VERSION }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', '.cargo/**', 'tests-fuzz/**', 'src/**') }} + - name: Report fuzz cache lookup + shell: bash + run: | + { + printf '## Fuzz cache lookup\n\n' + printf -- '- Dependency cache: `%s`\n' "${{ steps.fuzz-dependencies-cache.outputs.cache-hit }}" + printf -- '- Prebuilt binary cache: `%s`\n' "${{ steps.fuzz-binaries.outputs.cache-hit }}" + } >>"${GITHUB_STEP_SUMMARY}" + - name: Set Rust Fuzz + if: steps.fuzz-binaries.outputs.cache-hit != 'true' + shell: bash + run: | + sudo apt-get install -y libfuzzer-14-dev + cargo install cargo-fuzz --version "${CARGO_FUZZ_VERSION}" --locked + - name: Build fuzz target binaries + if: steps.fuzz-binaries.outputs.cache-hit != 'true' + shell: bash + env: + CUSTOM_LIBFUZZER_PATH: /usr/lib/llvm-14/lib/libFuzzer.a + FUZZ_TARGETS: | + fuzz_create_database + fuzz_create_table + fuzz_alter_table + fuzz_insert + fuzz_create_logical_table + fuzz_alter_logical_table + fuzz_insert_logical_table + fuzz_repartition_table + fuzz_repartition_metric_table + fuzz_repartition_table_chaos + fuzz_remote_wal_logical_prune + fuzz_migrate_mito_regions + fuzz_migrate_metric_regions + fuzz_failover_mito_regions + fuzz_failover_metric_regions + FUZZ_UNSTABLE_TARGETS: unstable_fuzz_create_table_standalone + FUZZ_BIN_DIR: ${{ runner.temp }}/greptime-fuzz-binaries + run: | + set -euo pipefail + target_dir="$(cargo metadata --manifest-path tests-fuzz/Cargo.toml --no-deps --format-version 1 | jq -r '.target_directory')" + mkdir -p "${FUZZ_BIN_DIR}" + build_targets() { + local unstable="$1" + local targets="$2" + while IFS= read -r target; do + [[ -z "${target}" ]] && continue + args=(fuzz build "${target}" --fuzz-dir tests-fuzz -D -s none) + if [[ "${unstable}" == true ]]; then + args+=(--features unstable) + fi + cargo "${args[@]}" + binary="$(find "${target_dir}" -type f -path "*/debug/${target}" -perm -u+x -print -quit)" + if [[ -z "${binary}" ]]; then + echo "Unable to locate built fuzz target: ${target}" >&2 + exit 1 + fi + cp "${binary}" "${FUZZ_BIN_DIR}/${target}" + done <<<"${targets}" + } + build_targets false "${FUZZ_TARGETS}" + build_targets true "${FUZZ_UNSTABLE_TARGETS}" + - name: Save prebuilt fuzz binaries + if: steps.fuzz-binaries.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: ${{ runner.temp }}/greptime-fuzz-binaries + key: v1-fuzz-binaries-${{ env.CARGO_FUZZ_VERSION }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', '.cargo/**', 'tests-fuzz/**', 'src/**') }} + - name: Pack prebuilt fuzz binaries + shell: bash + run: | + tar -C "${{ runner.temp }}" -cf "${{ runner.temp }}/fuzz-target-binaries.tar" \ + greptime-fuzz-binaries + - name: Upload prebuilt fuzz binaries + uses: actions/upload-artifact@v4 + with: + name: fuzz-target-binaries + path: ${{ runner.temp }}/fuzz-target-binaries.tar + compression-level: 0 + retention-days: 1 + if-no-files-found: error + + fuzztest: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request' }} + name: Fuzz Test (Standalone, ${{ matrix.group.name }}) + needs: [build, prepare-fuzz-targets] + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + max-parallel: 2 + matrix: + group: + - name: "Database and regular table" + id: "database-and-regular-table" + targets: | + fuzz_create_database + fuzz_create_table + fuzz_alter_table + fuzz_insert + - name: "Logical table" + id: "logical-table" + targets: | + fuzz_create_logical_table + fuzz_alter_logical_table + fuzz_insert_logical_table + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Download prebuilt fuzz binaries + uses: actions/download-artifact@v4 + with: + name: fuzz-target-binaries + path: ${{ runner.temp }} + - name: Unpack and verify prebuilt fuzz binaries + uses: ./.github/actions/restore-fuzz-binaries + with: + directory: ${{ runner.temp }} + targets: ${{ matrix.group.targets }} + - name: Download pre-built binaries + uses: actions/download-artifact@v4 + with: + name: bins + path: . + - name: Unzip binaries + run: | + tar -xvf ./bins.tar.gz + rm ./bins.tar.gz + - name: Run GreptimeDB + run: | + mkdir -p /tmp/greptime-fuzz-runtime + ./bins/greptime standalone start >/tmp/greptime-fuzz-runtime/greptime.log 2>&1 & + - name: Fuzz Test + uses: ./.github/actions/fuzz-test + env: + GT_MYSQL_ADDR: 127.0.0.1:4002 + FUZZ_SERVICE_LOG: /tmp/greptime-fuzz-runtime/greptime.log + with: + targets: ${{ matrix.group.targets }} + group: ${{ matrix.group.id }} + max-total-time: 120 + fail-fast: 'true' + prebuilt-binaries-dir: ${{ runner.temp }}/greptime-fuzz-binaries + - name: Collect setup failure artifacts + if: ${{ failure() && !cancelled() }} + shell: bash + env: + FUZZ_COLLECT_CLUSTER_ARTIFACTS: 'false' + FUZZ_SERVICE_LOG: /tmp/greptime-fuzz-runtime/greptime.log + FUZZ_GROUP: ${{ matrix.group.id }} + run: | + if [[ ! -f /tmp/greptime-fuzz-artifacts/manifest.json ]]; then + .github/scripts/collect-fuzz-target-artifacts.sh \ + setup /tmp/greptime-fuzz-artifacts/targets/setup || true + fi + - name: Upload fuzz test artifacts + if: ${{ failure() && !cancelled() }} + uses: ./.github/actions/upload-fuzz-artifacts + with: + artifact-name: fuzz-standalone-${{ matrix.group.id }} + + unstable-fuzztest: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request' }} + name: Unstable Fuzz Test + needs: [build-greptime-ci, prepare-fuzz-targets] + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Download prebuilt fuzz binaries + uses: actions/download-artifact@v4 + with: + name: fuzz-target-binaries + path: ${{ runner.temp }} + - name: Unpack and verify prebuilt fuzz binaries + uses: ./.github/actions/restore-fuzz-binaries + with: + directory: ${{ runner.temp }} + targets: unstable_fuzz_create_table_standalone + - name: Download pre-built binary + uses: actions/download-artifact@v4 + with: + name: bin + path: . + - name: Unzip binary + run: | + tar -xvf ./bin.tar.gz + rm ./bin.tar.gz + - name: Run Fuzz Test + uses: ./.github/actions/fuzz-test + env: + GT_MYSQL_ADDR: 127.0.0.1:4002 + GT_FUZZ_BINARY_PATH: ./bin/greptime + GT_FUZZ_INSTANCE_ROOT_DIR: /tmp/greptime-fuzz-artifacts/targets/unstable_fuzz_create_table_standalone/service/instance/ + with: + targets: unstable_fuzz_create_table_standalone + group: unstable-fuzz-create-table + max-total-time: 120 + unstable: 'true' + fail-fast: 'true' + prebuilt-binaries-dir: ${{ runner.temp }}/greptime-fuzz-binaries + - name: Collect setup failure artifacts + if: ${{ failure() && !cancelled() }} + shell: bash + env: + FUZZ_COLLECT_CLUSTER_ARTIFACTS: 'false' + FUZZ_GROUP: unstable-fuzz-create-table + run: | + if [[ ! -f /tmp/greptime-fuzz-artifacts/manifest.json ]]; then + .github/scripts/collect-fuzz-target-artifacts.sh \ + setup /tmp/greptime-fuzz-artifacts/targets/setup || true + fi + - name: Upload fuzz test artifacts + if: ${{ failure() && !cancelled() }} + uses: ./.github/actions/upload-fuzz-artifacts + with: + artifact-name: fuzz-unstable-standalone + + build-greptime-ci: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request' }} + name: Build GreptimeDB binary (profile-CI) + runs-on: ${{ matrix.os }} + strategy: + matrix: + os: [ ubuntu-latest ] + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions-rust-lang/setup-rust-toolchain@v1 + - name: Build Profile-CI Cache + id: build-greptime-ci-cache + uses: Swatinem/rust-cache@v2 + with: + # Shares across multiple jobs + shared-key: "build-greptime-ci" + cache-all-crates: "true" + cache-workspace-crates: "true" + save-if: ${{ github.event_name == 'pull_request' }} + - name: Report profile-CI cache lookup + shell: bash + run: | + { + printf '## Build profile-CI cache lookup\n\n' + printf -- '- Dependency cache: `%s`\n' "${{ steps.build-greptime-ci-cache.outputs.cache-hit }}" + } >>"${GITHUB_STEP_SUMMARY}" + - name: Install cargo-gc-bin + shell: bash + run: | + started_epoch="$(date +%s)" + cargo install cargo-gc-bin --force + elapsed_secs=$(( $(date +%s) - started_epoch )) + printf -- '- cargo-gc-bin install: **%ss**\n' "${elapsed_secs}" >>"${GITHUB_STEP_SUMMARY}" + - name: Build greptime binary + shell: bash + # `cargo gc` will invoke `cargo build` with specified args + run: | + started_epoch="$(date +%s)" + cargo gc --profile ci -- --bin greptime --features "pg_kvbackend,mysql_kvbackend" + elapsed_secs=$(( $(date +%s) - started_epoch )) + printf -- '- Profile-CI binary build: **%ss**\n' "${elapsed_secs}" >>"${GITHUB_STEP_SUMMARY}" + - name: Pack greptime binary + shell: bash + run: | + mkdir bin && \ + mv ./target/ci/greptime bin + - name: Print greptime binaries info + run: ls -lh bin + - name: Upload artifacts + uses: ./.github/actions/upload-artifacts + with: + artifacts-dir: bin + version: current + + distributed-fuzztest: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request' }} + name: Fuzz Test (Distributed, ${{ matrix.mode.name }}, ${{ matrix.group.name }}) + runs-on: ubuntu-latest + needs: [build-greptime-ci, prepare-fuzz-targets] + timeout-minutes: 60 + strategy: + fail-fast: false + max-parallel: 4 + matrix: + group: + - name: "Database and regular table" + id: "database-and-regular-table" + targets: | + fuzz_create_database + fuzz_create_table + fuzz_alter_table + fuzz_insert + - name: "Logical table" + id: "logical-table" + targets: | + fuzz_create_logical_table + fuzz_alter_logical_table + fuzz_insert_logical_table + mode: + - name: "Remote WAL" + id: "remote-wal" + minio: true + kafka: true + values: "with-remote-wal.yaml" + include: + - group: + name: "Repartition" + id: "repartition" + targets: | + fuzz_repartition_table + fuzz_repartition_metric_table + mode: + name: "Local WAL" + id: "local-wal" + minio: true + kafka: false + values: "with-minio-repartition-gc.yaml" + - group: + name: "Logical pruning" + id: "logical-pruning" + targets: "fuzz_remote_wal_logical_prune" + mode: + name: "Remote WAL logical pruning" + id: "remote-wal-logical-pruning" + minio: true + kafka: true + values: "with-remote-wal-logical-prune.yaml" + kafkaWalHelper: true + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Setup Kind + uses: ./.github/actions/setup-kind + - if: matrix.mode.minio + name: Setup Minio + uses: ./.github/actions/setup-minio + - if: matrix.mode.kafka + name: Setup Kafka cluster + uses: ./.github/actions/setup-kafka-cluster + - name: Setup Etcd cluster + uses: ./.github/actions/setup-etcd-cluster + # Only the Kafka WAL helper needs the Rust toolchain. + - if: matrix.mode.kafkaWalHelper + uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - if: matrix.mode.kafkaWalHelper + uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false + - if: matrix.mode.kafkaWalHelper + name: Kafka WAL Helper Cache + id: kafka-wal-helper-cache + uses: Swatinem/rust-cache@v2 + with: + prefix-key: "v1-kafka-wal-helper" + shared-key: "kafka-wal-helper" + workspaces: ". -> target" + cache-targets: "true" + cache-all-crates: "true" + save-if: ${{ github.event_name == 'pull_request' }} + - if: matrix.mode.kafkaWalHelper + name: Report Kafka WAL helper cache lookup + shell: bash + run: | + { + printf '## Kafka WAL helper cache lookup\n\n' + printf -- '- Dependency cache: `%s`\n' "${{ steps.kafka-wal-helper-cache.outputs.cache-hit }}" + } >>"${GITHUB_STEP_SUMMARY}" + - name: Download prebuilt fuzz binaries + uses: actions/download-artifact@v4 + with: + name: fuzz-target-binaries + path: ${{ runner.temp }} + - name: Unpack and verify prebuilt fuzz binaries + uses: ./.github/actions/restore-fuzz-binaries + with: + directory: ${{ runner.temp }} + targets: ${{ matrix.group.targets }} + # Downloads ci image + - name: Download pre-built binariy + uses: actions/download-artifact@v4 + with: + name: bin + path: . + - name: Unzip binary + run: | + tar -xvf ./bin.tar.gz + rm ./bin.tar.gz + - name: Build and push GreptimeDB image + uses: ./.github/actions/build-and-push-ci-image + - if: matrix.mode.kafkaWalHelper + name: Build Kafka WAL helper binary + shell: bash + run: | + started_epoch="$(date +%s)" + cargo build --bin kafka_wal_helper + elapsed_secs=$(( $(date +%s) - started_epoch )) + printf -- '- Kafka WAL helper build: **%ss**\n' "${elapsed_secs}" >>"${GITHUB_STEP_SUMMARY}" + mkdir -p amd64 + cp target/debug/kafka_wal_helper amd64/kafka_wal_helper + - if: matrix.mode.kafkaWalHelper + name: Build and push Kafka WAL helper image + uses: docker/build-push-action@v5 + with: + context: . + file: ./docker/ci/ubuntu/Dockerfile.kafka-wal-helper + push: true + tags: localhost:5001/greptime/kafka-wal-helper:latest + build-args: | + TARGETARCH=amd64 + TARGET_BIN=kafka_wal_helper + - name: Wait for etcd + run: | + kubectl wait \ + --for=condition=Ready \ + pod -l app.kubernetes.io/instance=etcd \ + --timeout=120s \ + -n etcd-cluster + - if: matrix.mode.minio + name: Wait for minio + run: | + kubectl wait \ + --for=condition=Ready \ + pod -l app=minio \ + --timeout=120s \ + -n minio + - if: matrix.mode.kafka + name: Wait for kafka + run: | + kubectl wait \ + --for=condition=Ready \ + pod -l app.kubernetes.io/instance=kafka \ + --timeout=180s \ + -n kafka-cluster + - if: matrix.mode.kafkaWalHelper + name: Setup Kafka WAL helper + shell: bash + run: | + kubectl -n kafka-cluster apply \ + -f .github/actions/setup-greptimedb-cluster/kafka-wal-helper.yaml + + kubectl -n kafka-cluster wait \ + --for=condition=Ready \ + pod -l app=kafka-wal-helper \ + --timeout=120s + - name: Print etcd info + shell: bash + run: kubectl get all --show-labels -n etcd-cluster + # Setup cluster for test + - name: Setup GreptimeDB cluster + uses: ./.github/actions/setup-greptimedb-cluster + with: + image-registry: localhost:5001 + values-filename: ${{ matrix.mode.values }} + - name: Port forward (mysql) + run: | + kubectl port-forward service/my-greptimedb-frontend 4002:4002 -n my-greptimedb& + - if: matrix.mode.kafkaWalHelper + name: Port forward Kafka WAL helper + run: | + kubectl port-forward service/kafka-wal-helper 8080:8080 -n kafka-cluster& + - if: matrix.mode.kafka + name: Check Kafka readiness + shell: bash + run: | + .github/scripts/wait-kafka-ready.sh + - name: Fuzz Test + uses: ./.github/actions/fuzz-test + env: + GT_MYSQL_ADDR: 127.0.0.1:4002 + GT_KAFKA_WAL_HELPER_URL: http://127.0.0.1:8080 + GT_KAFKA_ADDR: kafka.kafka-cluster.svc.cluster.local:9092 + GT_WAL_TOPIC_PREFIX: greptimedb_wal_topic + GT_WAL_NUM_TOPICS: 3 + GT_FUZZ_OVERRIDE_DELETE_AFTER_SECS: 300 + with: + targets: ${{ matrix.group.targets }} + group: ${{ matrix.group.id }} + max-total-time: ${{ matrix.mode.kafkaWalHelper && 600 || 120 }} + fail-fast: 'true' + collect-cluster-artifacts: 'true' + prebuilt-binaries-dir: ${{ runner.temp }}/greptime-fuzz-binaries + - name: Collect setup failure artifacts + if: ${{ failure() && !cancelled() }} + shell: bash + env: + FUZZ_COLLECT_CLUSTER_ARTIFACTS: 'true' + GT_FUZZ_NS: my-greptimedb + GT_FUZZ_CLUSTER: my-greptimedb + GT_MONITOR_HTTP_LOCAL_PORT: 14000 + FUZZ_GROUP: ${{ matrix.group.id }} + run: | + if [[ ! -f /tmp/greptime-fuzz-artifacts/manifest.json ]]; then + timeout --signal=TERM --kill-after=10s 180s \ + .github/scripts/collect-fuzz-target-artifacts.sh \ + setup /tmp/greptime-fuzz-artifacts/targets/setup || true + fi + - name: Upload fuzz test artifacts + if: ${{ failure() && !cancelled() }} + uses: ./.github/actions/upload-fuzz-artifacts + with: + artifact-name: fuzz-distributed-${{ matrix.mode.id }}-${{ matrix.group.id }} + - name: Delete cluster + if: success() + shell: bash + run: | + kind delete cluster + docker stop $(docker ps -a -q) + docker rm $(docker ps -a -q) + docker system prune -f + + distributed-fuzztest-with-chaos: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request' }} + name: Fuzz Test with Chaos (Distributed, ${{ matrix.mode.name }}, ${{ matrix.group.name }}) + runs-on: ubuntu-latest + needs: [build-greptime-ci, prepare-fuzz-targets] + timeout-minutes: 60 + strategy: + fail-fast: false + max-parallel: 4 + matrix: + group: + - name: "Migration" + id: "migration" + targets: | + fuzz_migrate_mito_regions + fuzz_migrate_metric_regions + - name: "Failover" + id: "failover" + targets: | + fuzz_failover_mito_regions + fuzz_failover_metric_regions + mode: + - name: "Remote WAL" + id: "remote-wal" + minio: true + kafka: true + values: "with-remote-wal.yaml" + include: + - group: + name: "Migration" + id: "migration" + targets: | + fuzz_migrate_mito_regions + fuzz_migrate_metric_regions + mode: + name: "Local WAL" + id: "local-wal" + minio: true + kafka: false + values: "with-minio.yaml" + - group: + name: "Repartition" + id: "repartition" + targets: "fuzz_repartition_table_chaos" + mode: + name: "Local WAL repartition chaos" + id: "local-wal-repartition-chaos" + minio: true + kafka: false + values: "with-minio-repartition-gc.yaml" + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Setup Kind + uses: ./.github/actions/setup-kind + - name: Setup Chaos Mesh + uses: ./.github/actions/setup-chaos + - if: matrix.mode.minio + name: Setup Minio + uses: ./.github/actions/setup-minio + - if: matrix.mode.kafka + name: Setup Kafka cluster + uses: ./.github/actions/setup-kafka-cluster + - name: Setup Etcd cluster + uses: ./.github/actions/setup-etcd-cluster + - name: Download prebuilt fuzz binaries + uses: actions/download-artifact@v4 + with: + name: fuzz-target-binaries + path: ${{ runner.temp }} + - name: Unpack and verify prebuilt fuzz binaries + uses: ./.github/actions/restore-fuzz-binaries + with: + directory: ${{ runner.temp }} + targets: ${{ matrix.group.targets }} + # Downloads ci image + - name: Download pre-built binariy + uses: actions/download-artifact@v4 + with: + name: bin + path: . + - name: Unzip binary + run: | + tar -xvf ./bin.tar.gz + rm ./bin.tar.gz + - name: Build and push GreptimeDB image + uses: ./.github/actions/build-and-push-ci-image + - name: Wait for etcd + run: | + kubectl wait \ + --for=condition=Ready \ + pod -l app.kubernetes.io/instance=etcd \ + --timeout=120s \ + -n etcd-cluster + - if: matrix.mode.minio + name: Wait for minio + run: | + kubectl wait \ + --for=condition=Ready \ + pod -l app=minio \ + --timeout=120s \ + -n minio + - if: matrix.mode.kafka + name: Wait for kafka + run: | + kubectl wait \ + --for=condition=Ready \ + pod -l app.kubernetes.io/instance=kafka \ + --timeout=180s \ + -n kafka-cluster + - name: Print etcd info + shell: bash + run: kubectl get all --show-labels -n etcd-cluster + # Setup cluster for test + - name: Setup GreptimeDB cluster + uses: ./.github/actions/setup-greptimedb-cluster + with: + image-registry: localhost:5001 + values-filename: ${{ matrix.mode.values }} + enable-region-failover: ${{ matrix.mode.kafka }} + - name: Port forward (mysql) + run: | + kubectl port-forward service/my-greptimedb-frontend 4002:4002 -n my-greptimedb& + - if: matrix.mode.kafka + name: Check Kafka readiness + shell: bash + run: | + .github/scripts/wait-kafka-ready.sh + - name: Fuzz Test + uses: ./.github/actions/fuzz-test + env: + GT_MYSQL_ADDR: 127.0.0.1:4002 + with: + targets: ${{ matrix.group.targets }} + group: ${{ matrix.group.id }} + max-total-time: 120 + fail-fast: 'true' + collect-cluster-artifacts: 'true' + prebuilt-binaries-dir: ${{ runner.temp }}/greptime-fuzz-binaries + - name: Collect setup failure artifacts + if: ${{ failure() && !cancelled() }} + shell: bash + env: + FUZZ_COLLECT_CLUSTER_ARTIFACTS: 'true' + GT_FUZZ_NS: my-greptimedb + GT_FUZZ_CLUSTER: my-greptimedb + GT_MONITOR_HTTP_LOCAL_PORT: 14000 + FUZZ_GROUP: ${{ matrix.group.id }} + run: | + if [[ ! -f /tmp/greptime-fuzz-artifacts/manifest.json ]]; then + timeout --signal=TERM --kill-after=10s 180s \ + .github/scripts/collect-fuzz-target-artifacts.sh \ + setup /tmp/greptime-fuzz-artifacts/targets/setup || true + fi + - name: Upload fuzz test artifacts + if: ${{ failure() && !cancelled() }} + uses: ./.github/actions/upload-fuzz-artifacts + with: + artifact-name: fuzz-distributed-chaos-${{ matrix.mode.id }}-${{ matrix.group.id }} + - name: Delete cluster + if: success() + shell: bash + run: | + kind delete cluster + docker stop $(docker ps -a -q) + docker rm $(docker ps -a -q) + docker system prune -f diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml new file mode 100644 index 0000000000..b5bfa1a296 --- /dev/null +++ b/.github/workflows/rust.yml @@ -0,0 +1,230 @@ +on: + schedule: + - cron: "0 15 * * 1-5" + merge_group: + pull_request: + types: [ opened, synchronize, reopened, ready_for_review ] + paths-ignore: + - 'docs/**' + - 'config/**' + - '**.md' + - '.dockerignore' + - 'docker/**' + - '.gitignore' + - 'grafana/**' + - 'Makefile' + workflow_dispatch: + +name: Rust CI + +env: + CARGO_UDEPS_VERSION: "0.1.61" + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +jobs: + fmt: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: Rustfmt + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + components: rustfmt + - name: Check format + run: make fmt-check + + clippy: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: Clippy + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + components: clippy + - name: Rust Cache + uses: Swatinem/rust-cache@v2 + with: + # Shares across multiple jobs + # Shares with `Check` job + shared-key: "check-lint" + cache-all-crates: "true" + save-if: ${{ github.ref == 'refs/heads/main' }} + - name: Run cargo clippy + run: make clippy + + check-udeps: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: Check Unused Dependencies + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions-rust-lang/setup-rust-toolchain@v1 + - name: Rust Cache + uses: Swatinem/rust-cache@v2 + with: + shared-key: "check-udeps" + cache-all-crates: "true" + save-if: ${{ github.ref == 'refs/heads/main' }} + - name: Install cargo-udeps + uses: taiki-e/install-action@v2 + with: + tool: cargo-udeps@${{ env.CARGO_UDEPS_VERSION }} + fallback: none + - name: Check unused dependencies + run: make check-udeps + + conflict-check: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + name: Check for conflict + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Merge Conflict Finder + uses: olivernybroe/action-conflict-finder@v4.0 + + test: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name != 'merge_group' }} + runs-on: ubuntu-22.04-arm + timeout-minutes: 60 + needs: [conflict-check, clippy, fmt, check-udeps] + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: rui314/setup-mold@v1 + - name: Install toolchain + uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false + - name: Rust Cache + uses: Swatinem/rust-cache@v2 + with: + # Shares cross multiple jobs + shared-key: "coverage-test" + cache-all-crates: "true" + save-if: ${{ github.ref == 'refs/heads/main' }} + - name: Install latest nextest release + uses: taiki-e/install-action@nextest + + - name: Setup external services + working-directory: tests-integration/fixtures + run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait + + - name: Run nextest cases + run: cargo nextest run --workspace -F dashboard -F pg_kvbackend -F mysql_kvbackend -F vector_index + env: + CARGO_BUILD_RUSTFLAGS: "-C link-arg=-fuse-ld=mold" + RUST_BACKTRACE: 1 + RUST_MIN_STACK: 8388608 # 8MB + CARGO_INCREMENTAL: 0 + GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} + GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} + GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} + GT_MINIO_BUCKET: greptime + GT_MINIO_ACCESS_KEY_ID: superpower_ci_user + GT_MINIO_ACCESS_KEY: superpower_password + GT_MINIO_REGION: us-west-2 + GT_MINIO_ENDPOINT_URL: http://127.0.0.1:9000 + GT_ETCD_TLS_ENDPOINTS: https://127.0.0.1:2378 + GT_ETCD_ENDPOINTS: http://127.0.0.1:2379 + GT_POSTGRES_ENDPOINTS: postgres://greptimedb:admin@127.0.0.1:5432/postgres + GT_POSTGRES15_ENDPOINTS: postgres://test_user:test_password@127.0.0.1:5433/postgres + GT_POSTGRES15_SCHEMA: test_schema + GT_MYSQL_ENDPOINTS: mysql://greptimedb:admin@127.0.0.1:3306/mysql + GT_KAFKA_ENDPOINTS: 127.0.0.1:9092 + GT_KAFKA_SASL_ENDPOINTS: 127.0.0.1:9093 + UNITTEST_LOG_DIR: "__unittest_logs" + + coverage: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'merge_group' }} + runs-on: ubuntu-22.04-8-cores + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: rui314/setup-mold@v1 + - name: Install toolchain + uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + components: llvm-tools + cache: false + - name: Rust Cache + uses: Swatinem/rust-cache@v2 + with: + # Shares cross multiple jobs + shared-key: "coverage-test" + save-if: ${{ github.ref == 'refs/heads/main' }} + - name: Install latest nextest release + uses: taiki-e/install-action@nextest + - name: Install cargo-llvm-cov + uses: taiki-e/install-action@cargo-llvm-cov + + - name: Setup external services + working-directory: tests-integration/fixtures + run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait + + - name: Run nextest cases + run: cargo llvm-cov nextest --workspace --lcov --output-path lcov.info -F dashboard -F pg_kvbackend -F mysql_kvbackend -F vector_index + env: + CARGO_BUILD_RUSTFLAGS: "-C link-arg=-fuse-ld=mold" + RUST_BACKTRACE: 1 + CARGO_INCREMENTAL: 0 + GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} + GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} + GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} + GT_MINIO_BUCKET: greptime + GT_MINIO_ACCESS_KEY_ID: superpower_ci_user + GT_MINIO_ACCESS_KEY: superpower_password + GT_MINIO_REGION: us-west-2 + GT_MINIO_ENDPOINT_URL: http://127.0.0.1:9000 + GT_ETCD_TLS_ENDPOINTS: https://127.0.0.1:2378 + GT_ETCD_ENDPOINTS: http://127.0.0.1:2379 + GT_POSTGRES_ENDPOINTS: postgres://greptimedb:admin@127.0.0.1:5432/postgres + GT_POSTGRES15_ENDPOINTS: postgres://test_user:test_password@127.0.0.1:5433/postgres + GT_POSTGRES15_SCHEMA: test_schema + GT_MYSQL_ENDPOINTS: mysql://greptimedb:admin@127.0.0.1:3306/mysql + GT_KAFKA_ENDPOINTS: 127.0.0.1:9092 + GT_KAFKA_SASL_ENDPOINTS: 127.0.0.1:9093 + UNITTEST_LOG_DIR: "__unittest_logs" + - name: Codecov upload + uses: codecov/codecov-action@v4 + with: + token: ${{ secrets.CODECOV_TOKEN }} + files: ./lcov.info + flags: rust + fail_ci_if_error: false + verbose: true diff --git a/README.md b/README.md index 99c7881460..66b6be8fdb 100644 --- a/README.md +++ b/README.md @@ -27,8 +27,8 @@ replacing Prometheus, Loki, and Elasticsearch Docker Pulls - -GitHub Actions + +GitHub Actions Codecov diff --git a/licenses/README.md b/licenses/README.md index 3fadc6d14c..45ccd167a8 100644 --- a/licenses/README.md +++ b/licenses/README.md @@ -13,7 +13,7 @@ Enterprise License header generated from [`enterprise-header.txt`](enterprise-he ## How it's enforced -CI (`license-header-check` in `.github/workflows/develop.yml`) runs +CI (`license-header-check` in `.github/workflows/checks.yml`) runs [hawkeye](https://github.com/korandoru/hawkeye) twice: 1. Default config — applies the Apache-2.0 header to all sources, **excluding** diff --git a/tests/compatibility/AGENTS.md b/tests/compatibility/AGENTS.md index 2d2e9952b2..41880813eb 100644 --- a/tests/compatibility/AGENTS.md +++ b/tests/compatibility/AGENTS.md @@ -63,4 +63,4 @@ would be selected before a real run. for a sampled version pair. - `.github/scripts/run-compat.py` owns the CI-side window parsing and compat invocation. Keep workflow YAML thin; update the script instead of embedding - parsing or loops in `.github/workflows/develop.yml`. + parsing or loops in `.github/workflows/integration.yml`.