diff --git a/.agents/skills/greptimedb-release-note/SKILL.md b/.agents/skills/greptimedb-release-note/SKILL.md index 893b99445ce..026ad90cd8d 100644 --- a/.agents/skills/greptimedb-release-note/SKILL.md +++ b/.agents/skills/greptimedb-release-note/SKILL.md @@ -1,6 +1,6 @@ --- name: greptimedb-release-note -description: Generate a GreptimeDB release changelog with git cliff (correct range, subtract already-released patch PRs, rebuild contributors, add human-curated highlights), output to a file, and prepare the docs-repo blog PR. Use when asked to write/generate a GreptimeDB release note or changelog. +description: Generate a GreptimeDB release changelog (minor, patch, or alpha/beta/rc pre-release) with git cliff (correct range, subtract already-released patch PRs, rebuild contributors, add human-curated highlights and upgrade notes), output to a file, and prepare the docs-repo blog PR. Use when asked to write/generate a GreptimeDB release note or changelog. --- # GreptimeDB Release Note / Changelog @@ -27,41 +27,54 @@ a token it still runs, but may be rate-limited / incomplete. ## 2. Determine the previous version (skip nightlies) -From `gh release list --repo GreptimeTeam/greptimedb`, ignore `*-nightly-*`, `-rc.*`, `-beta.*`, -and build-suffixed tags; focus on formal `vX.Y.Z`. +From `gh release list --repo GreptimeTeam/greptimedb`, ignore `*-nightly-*` and build-suffixed +tags (e.g. `v1.3.0-alpha.1-3c0e2a8d5-20260928-1790639248`). - Patch `vX.Y.Z` (Z>0): previous = `vX.Y.(Z-1)`. - New minor `vX.Y.0`: previous = the latest `vX.(Y-1).*` (for `v1.0.0`, the biggest `0.x`). +- Pre-release `vX.Y.Z-{alpha,beta,rc}.N`: previous = the latest pre-release tag of the same + `vX.Y.Z` (e.g. `v1.3.0-beta.1` for `v1.3.0-beta.2`). If there is none (the first alpha), + see "First pre-release" in §3: the previous minor's `.0` may not be published yet, and since + v1.2 a `.0` tag sits on `release/vX.Y`, not on `main`. Double-check with the user. ## 3. Pick the range and generate -Key topology fact: a **minor tag** (e.g. `v1.0.0`) is an ancestor of `main`; **patch tags** -(`v1.0.1`, `v1.0.2`) live on the `release/v1.0` branch and are **NOT** ancestors of `main` -(they are cherry-picks with different SHAs). Verify with +Key topology fact: **patch tags** (`v1.0.1`, `v1.0.2`) live on `release/vX.Y` and are **NOT** +ancestors of `main` (they are cherry-picks with different SHAs). Up to v1.1 the `.0` tag was +the point where `release/vX.Y` left `main`; from v1.2 the branch is cut before the `.0`, and +pre-releases and the `.0` itself are tagged on the branch with backports. Verify with `git merge-base --is-ancestor /main`. -### New minor (`X.Y.0`, cut from main) -Note the two different tags here: the `git cliff` **base is the previous minor `.0` tag** -(e.g. `v1.0.0`, an ancestor of `main`) — **not** the "previous version" from §2 (the latest -patch, e.g. `v1.0.2`), which is only used below to decide which patch PRs to subtract. +### New minor (`X.Y.0`) +The note compares against the whole previous line: everything new since `release/vX.(Y-1)` +left `main`, minus what any published `vX.(Y-1).*` release (pre-releases, `.0`, patches) +already shipped. The line's own pre-releases (`vX.Y.0-beta.*`) are not subtracted; the `.0` +note covers the whole minor. -Base = the **previous minor tag** (e.g. `v1.0.0`); tip = the release commit (= `release/vX.Y` -tip, usually `/main`): +Base = the **previous minor tag** (e.g. `v1.1.0`); tip = the tagged `vX.Y.0` commit, resolved as +in "Pre-release" below. Split the range at the point where `release/vX.Y` left `main`: ``` -GITHUB_TOKEN=$(gh auth token) git cliff .. --tag vX.Y.0 \ - -o /path/CHANGELOG-vX.Y.0.md +bp=$(git merge-base /release/vX.Y /main) ``` -`cliff.toml`'s `ignore_tags` folds in-range nightly tags into the single section. +- `..$bp` is on `main`; generate it with `git cliff`: + ``` + GITHUB_TOKEN=$(gh auth token) git cliff ..$bp --tag vX.Y.0 \ + -o /path/CHANGELOG-vX.Y.0.md + ``` + `cliff.toml`'s `ignore_tags` folds in-range nightly tags into the single section. +- `$bp..` is the release branch: backports, squashed picks, and commits without a PR. + `git cliff` fails on such commits (`Filter call 'sort' failed`), so build this part as a + pre-release: collect the original PRs with the loop in "Pre-release" and add their bullets. + List the commits that carry no PR with + `git log --format='%h %s' $bp.. | grep -vE '\(#[0-9]+\)$'`; drop mechanical ones, and + for a manual port of a `main` PR (the body says `cherry picked from commit` a SHA that is not + on `main`), find that PR by title and cite it. -**Then subtract PRs already shipped in the intermediate patch releases** (`vX.(Y-1).1`, -`.2`, …) — their main-branch commits are inside the range and would duplicate, and the -audience cares about what's new vs the latest patch. Collect the PR set from the patch release -bodies and remove matching lines: -``` -gh release view vX.(Y-1).Z --repo GreptimeTeam/greptimedb | grep -oE 'pull/[0-9]+' -``` -Remove every changelog bullet whose `#NNNN` is in that combined set (a small Python script is -the reliable way — match `pull/)` on lines starting with `*`). +**Then subtract PRs already shipped in published `vX.(Y-1).*` releases**: their `main` commits +are inside the range. Build the set as for the first pre-release (below): the PR links in +every published `vX.(Y-1).*` release body, plus the `(#N)` line endings in the history of the +newest published `vX.(Y-1).*` tag. Remove every changelog bullet whose `#NNNN` is in that set (a +small Python script is the reliable way — match `pull/)` on lines starting with `*`). ### Patch (`X.Y.Z`, Z>0, cherry-picks on the release branch) The previous patch tag is an ancestor of the release branch, so a plain range works **when the @@ -99,11 +112,78 @@ user which they prefer**): If the branch history is otherwise messy, the same "identify the picked PRs and keep only those" rule applies. +### First pre-release (`X.Y.0-alpha.1`, cut from main) +The base is the commit where the previous line's release branch left `main`. It exists when the +alpha is cut, even if `vX.(Y-1).0` does not: +``` +base=$(git merge-base /release/vX.(Y-1) ) +``` +If there is no `release/vX.(Y-1)`, use the newest published release tag that is an ancestor of +the tag (`git merge-base --is-ancestor `). Run `git cliff ..`, then +subtract every PR that already shipped in a published `vX.(Y-1).*` release, pre-releases +included. Take the union of the PR links in those release bodies and the `(#N)` line endings in +the history of the newest published `vX.(Y-1).*` tag: +``` +gh release view --repo GreptimeTeam/greptimedb | grep -oE 'pull/[0-9]+' +git log --format=%B | grep -oE '\(#[0-9]+\)$' +``` +Either source alone misses PRs: a release body can omit a backport that its tag contains, and a +backport commit can name only the backport PR. + +### Pre-release (`X.Y.Z-{alpha,beta,rc}.N` after an earlier pre-release) +**The tip is the tagged commit, not the release branch tip.** Resolve it from the remote and +write this SHA in the docs PR body (§8) as what the note was checked against: +``` +git ls-remote --tags 'vX.Y.Z-beta.N*' +``` +Use the `refs/tags/vX.Y.Z-beta.N` line (its `^{}` line if the tag is annotated), not the +build-suffixed tags it also lists. PRs that land on `release/vX.Y` after the tag +(`git log --oneline ../release/vX.Y`) belong to the next release, even if they are +already merged when you write the note. + +The range `..` usually mixes two kinds of commits: individual backports +(`... [Backport release/vX.Y] (#NNNN)`) and squashed pick commits (`chore: backport selected +changes for ... (#NNNN)`). In both, the original PRs are the `(#NN)` at the end of body lines; +a backport commit can also carry prerequisite PRs. Collect the PR set, falling back to the +subject for commits whose body lists none: +``` +for c in $(git rev-list ..); do + prs=$(git log -1 --format=%b "$c" | grep -oE '\(#[0-9]+\)$') + [ -n "$prs" ] || prs=$(git log -1 --format=%s "$c" | grep -oE '\(#[0-9]+\)$') + echo "$prs" +done | tr -dc '0-9\n' | sort -un +``` +List the bullets under the **original** PRs (title and author of the PR merged to `main`), not +the backport PRs. Build them with patch (a) or (b) above; for (b), fetch +`gh api repos/GreptimeTeam/greptimedb/pulls/ --jq '{number,title,author:.user.login,merged_at}'`, +group by the `cliff.toml` parsers (first match wins), order by `merged_at`, and put `!` titles +under Breaking changes (§6). + +A squashed pick PR can also contain release-only commits that have no PR. List them: +``` +gh api repos/GreptimeTeam/greptimedb/pulls//commits --paginate \ + --jq '.[].commit.message | split("\n")[0]' | grep -vE '\(#[0-9]+\)$' +``` +Mechanical ones (version bump, toolchain or lint adaptations; example: +`fix: adapt release OptionMap iteration to Rust 1.98`) get no bullet. A release-only +commit that changes behavior gets a bullet in its group. + > Flaky-network note for (a): `git cliff` fetches the repo's full commit history from the GitHub > API for author enrichment (paginates back thousands of commits, even for a tiny range). On a > flaky connection it can time out mid-pagination; it caches pages, so just **re-run until it > completes** — each retry resumes from the cache. If it keeps failing, fall back to (b). +### Groups (all paths) +`git cliff` drops PRs whose title isn't a conventional commit (`filter_unconventional`). List +them and add the user-facing ones by hand: +``` +git log --format=%s | grep -E '\(#[0-9]+\)$' | grep -vE '^[a-z]+(\([^)]*\))?!?: ' +``` +Use only the groups in `cliff.toml`. `git cliff` prints a `### Build` section for `build:` commits +because no parser matches them; move a user-facing `build(deps)` fix to Bug Fixes and the rest to +Miscellaneous Tasks. Keep each bullet list free of blank lines; a blank line turns it into a +loose list that renders with different spacing. + ## 4. Rebuild contributor sections after subtracting `cliff` computes `New Contributors` / `All Contributors` over the full range, so after removing @@ -114,6 +194,21 @@ lines, recompute from the **remaining** commit bullets: in-range PR shipped in a patch). Do this in the same script that subtracts the PRs. +When the bullets are built by hand (§3 (b), pre-releases), `git cliff` computes no New +Contributors, and GitHub's generated notes don't either (they only see the backport PRs). An +author is new if none of their merged PRs shipped before, i.e. none appears in the history of the +base tag: +``` +git log --format=%B | grep -oE '\(#[0-9]+\)$' | tr -dc '0-9\n' | sort -u > /tmp/shipped.txt +gh search prs --repo GreptimeTeam/greptimedb --author --merged --limit 1000 \ + --json number --jq '.[].number' | sort -u > /tmp/author.txt +comm -12 /tmp/author.txt /tmp/shipped.txt # empty => new contributor +``` +Run it for every non-bot author in the release, and cite their earliest-merged PR in this +release. "Earliest merged PR is in range" is not the same test: a contributor whose first PR +merged on `main` but was not picked into any release is still new when a later PR of theirs +ships. + ## 5. Verify Cross-check the result against `git log ..`: no subtracted PR remains, and no @@ -122,17 +217,71 @@ line (ask the user; some prefer to keep it). ## 6. Human-curated sections (insert after the `Release date:` line) +**Style.** Follow the Google developer documentation style guide: sentence-case headings, active voice, present +tense for behavior, American spelling, serial comma, code font for identifiers, bold only for UI +names. Keep it short: +- Intro: two sentences at most, what the release covers and its main theme. +- Highlight: one paragraph of up to four sentences, plus an example if it earns one (see below). +- Upgrade note or breaking change: one bullet, one or two sentences: what changes, who is + affected, what to do. +- Every sentence states a fact the reader acts on or checks. Delete sentences that restate the + heading or the previous sentence, transitions, and summaries. +- No marketing words (`significantly`, `powerful`, `seamless`, `robust`, `enhanced`), no vague + verbs (`supports`, `handles`, `improves`; say what happens), no `several`/`many` when the number + is known. Keep the source's hedges; don't add new ones. + +Example: +> Bad: This release significantly enhances write reliability by introducing a robust retry +> mechanism that seamlessly handles transient failures. +> +> Good: In distributed clusters, the frontend retries a failed write up to three times with +> exponential backoff. Set `max_retries = 0` to disable retries. + Mirror past release bodies (`gh release view v1.0.0 --repo GreptimeTeam/greptimedb`): -- **Short intro**, terse, engineer-written tone (no marketing adjectives). -- **`### 👍 Highlights`** — *few, deep* highlights, each with a **working example** - (SQL / TOML config). To get examples right, **read the highlight PR and the related docs - PR/page** in the docs repo (`GreptimeTeam/docs`, often checked out locally) — verify syntax - in `docs/reference/sql/*.md`, `config/*.example.toml`, `config/config.md`. Do **not** mention - implementation details, tiny features, or unfinished/experimental-but-not-ready features. - Always have the user review and edit the highlights; iterate. +- **Short intro** (see Style). +- **`### 👍 Highlights`** — *few, deep* highlights, ranked by user impact. Fixes for silently + wrong query results, lost or invisible data, and security issues rank with new features and + can be highlights. + - Write a fix as its symptom and the affected deployments or usage, not the PR title. Example: + "with no traffic, `errors / requests > 0.05` was true, so the alert fired". + - State whether a feature is on by default. If an example config uses the default values, say + so. + - Give an example (SQL / TOML config) only where it helps readers act. Don't pick highlights, + or the example, by what is easy to demonstrate. + - One highlight covers one change or one user-visible problem. Don't merge unrelated PRs into + one entry because they touch the same component. + - Sources: the PR body's sections on impact, compatibility, and behavior changes, plus the + code and config it touches, not the title alone. Verify examples against the related docs + PR/page in the docs repo (`GreptimeTeam/docs`, often checked out locally), + `docs/reference/sql/*.md`, `config/*.example.toml`, `config/config.md`. + Do **not** mention implementation details, tiny features, or unfinished/experimental-but-not-ready + features. Always have the user review and edit the highlights; iterate. - **Dashboard** subsection — don't just bump the version. Read the dashboard PRs in the bundled release (`gh release view --repo GreptimeTeam/dashboard`; then the linked PRs) and - describe the user-facing change. + describe the user-facing change. Use the names the UI shows, not the PR titles; check them in + the dashboard's locale strings at the bundled tag: + ``` + gh api 'repos/GreptimeTeam/dashboard/contents/src/locale/en-US/menu.ts?ref=' \ + --jq .content | base64 -d + ``` + Screenshots need absolute `https://` URLs: the body is copied to the GitHub release, where + relative and site-root (`/img/...`) paths don't resolve. +- **`### Upgrade notes`** and **`### Breaking changes`** — check every PR in the release for: + `!` titles; `Compatibility`, `Behavior change`, `Breaking` paragraphs in the body (often a + plain `Compatibility:` line, not a heading); unchecked "API changes are backward compatible" / + "Schema or data changes are backward compatible" boxes; new config options and their defaults; + changed defaults; persisted format changes; rolling-upgrade order; new hard limits. These + signals set reading order, not a filter: a limit can sit in plain prose with every box + checked, so read the body of every user-facing `feat`/`fix` PR. Examples of what is easy to + miss: a 512 MiB cap on decompressed request bodies (#9264); a default step for stepless + subqueries (#9481). + - **Upgrade notes**: steps operators take — upgrade order, index rebuilds, settings to review. + - **Breaking changes**: behavior that used to work and now errors or returns different + results. Put the `!` PR bullets here too, after these entries (see `v1.3.0-beta.1`). +- **Benchmark results** (optional) — state the baseline version (often the previous formal + release, not the previous beta), data size, cache state, and hardware. Don't attribute results + to single PRs. Leave out results the benchmark marks as not for publication, and smoke-level + runs. ## 7. Output @@ -144,7 +293,8 @@ it**. The release runbook deletes it after the release + docs PR are done. The release note is also published as a blog post in **`GreptimeTeam/docs`**. - **Ask the user for their local `GreptimeTeam/docs` checkout path**; if they have none, offer to clone (`git clone git@github.com:GreptimeTeam/docs.git `). -- File: `blog/release-X-Y-Z.md` (version with dashes; see `blog/release-1-0-0.md`). +- File: `blog/release-X-Y-Z.md` (version with dashes; see `blog/release-1-0-0.md`). Pre-releases + use dashes throughout: `blog/release-1-3-0-beta-2.md`. - Content = docs frontmatter (below) followed by the GitHub release body, **keeping the `# vX.Y.Z` H1 directly under the frontmatter**. The blog frontmatter has no `title:`, so Docusaurus uses that H1 as the page title and sidebar label — omitting it makes the page @@ -156,11 +306,13 @@ The release note is also published as a blog post in **`GreptimeTeam/docs`**. description: GreptimeDB vX.Y.Z Changelog date: YYYY-MM-DD --- - # vX.Y.Z Release date: ... ``` + The docs CI check `check-release-note-sync` requires the post minus its frontmatter to equal + the GitHub release body byte for byte, so the H1 goes on the line right after the closing + `---`, with no blank line. - The docs working tree may have unrelated WIP — **don't disturb it**: use a `git worktree` off `origin/main`: ``` @@ -169,8 +321,19 @@ The release note is also published as a blog post in **`GreptimeTeam/docs`**. ``` - **Read and follow the current docs repo PR template** (`.github/pull_request_template.md`) from the checkout used for the release. - Fill in every required section and leave reviewer-owned checklist boxes unchecked; - do not rely on hard-coded section names from an older template. + Fill in every required section; do not rely on hard-coded section names from an older + template. The checklist items are the author's to confirm: check the ones you did, and for one + that doesn't apply, check it and say why. For pre-releases, state the tagged SHA (§3) under + Verification. +- **Build the site before opening the PR.** A new worktree has no `node_modules`. Install with + the pnpm version pinned in `package.json` (`packageManager`), as CI does: + ``` + cd /tmp/docs-release-note + pm=$(jq -r .packageManager package.json) # e.g. pnpm@8.6.0 + npx -y "$pm" install --frozen-lockfile + npx -y "$pm" build + ``` + If the build fails, fix it or stop; don't mark the PR ready for review. - Commit with sign-off, push, open a **draft** PR, then remove the worktree: ``` git -C /tmp/docs-release-note add blog/release-X-Y-Z.md diff --git a/.github/actions/build-linux-artifacts/action.yml b/.github/actions/build-linux-artifacts/action.yml index 26028485061..a16c1a72405 100644 --- a/.github/actions/build-linux-artifacts/action.yml +++ b/.github/actions/build-linux-artifacts/action.yml @@ -50,6 +50,11 @@ inputs: runs: using: composite steps: + - name: Set up Linux runner + uses: ./.github/actions/setup-linux-runner + with: + tools: make shasum + - name: Run integration test if: ${{ inputs.disable-run-tests == 'false' }} shell: bash diff --git a/.github/actions/fuzz-test/action.yaml b/.github/actions/fuzz-test/action.yaml index b696d8099dc..b93634b16a2 100644 --- a/.github/actions/fuzz-test/action.yaml +++ b/.github/actions/fuzz-test/action.yaml @@ -28,7 +28,38 @@ inputs: runs: using: composite steps: + - name: Prepare Fuzz Corpus + id: corpus + shell: bash + env: + FUZZ_TARGETS: ${{ inputs.targets }} + run: bash .github/scripts/prepare-fuzz-corpus.sh + + - name: Restore Fuzz Corpus + id: restore-corpus + uses: actions/cache/restore@v4 + continue-on-error: true + with: + path: ${{ steps.corpus.outputs.paths }} + key: ${{ steps.corpus.outputs.key-prefix }}${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + restore-keys: ${{ steps.corpus.outputs.key-prefix }} + + - name: Report Fuzz Corpus Cache + shell: bash + env: + CACHE_OUTCOME: ${{ steps.restore-corpus.outcome }} + CACHE_MATCHED_KEY: ${{ steps.restore-corpus.outputs.cache-matched-key }} + run: | + if [[ "${CACHE_OUTCOME}" == failure ]]; then + echo '::warning::Fuzz corpus restore failed; running with available inputs' + elif [[ -n "${CACHE_MATCHED_KEY}" ]]; then + printf 'Restored fuzz corpus: %s\n' "${CACHE_MATCHED_KEY}" + else + echo 'No cached fuzz corpus found; starting with an empty corpus' + fi + - name: Run Fuzz Tests + id: run-fuzz shell: bash env: FUZZ_TARGETS: ${{ inputs.targets }} @@ -39,4 +70,13 @@ runs: FUZZ_COLLECT_CLUSTER_ARTIFACTS: ${{ inputs.collect-cluster-artifacts }} FUZZ_ARTIFACT_ROOT: ${{ inputs.artifact-root }} FUZZ_BIN_DIR: ${{ inputs.prebuilt-binaries-dir }} + FUZZ_CORPUS_ROOT: ${{ github.workspace }}/tests-fuzz/corpus run: .github/scripts/run-fuzz-targets.sh + + - name: Save Fuzz Corpus + if: ${{ !cancelled() && (steps.run-fuzz.outcome == 'success' || steps.run-fuzz.outcome == 'failure') }} + uses: actions/cache/save@v4 + continue-on-error: true + with: + path: ${{ steps.corpus.outputs.paths }} + key: ${{ steps.corpus.outputs.key-prefix }}${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} diff --git a/.github/actions/github-script-tests/action.yml b/.github/actions/github-script-tests/action.yml index c7aba309383..2d2a310a9ef 100644 --- a/.github/actions/github-script-tests/action.yml +++ b/.github/actions/github-script-tests/action.yml @@ -8,7 +8,7 @@ runs: node-version: 24 - name: Run Node GitHub script tests shell: bash - run: node --test .github/scripts/query-regression-comment.test.cjs .github/scripts/heavy-ci-policy.test.cjs + run: node --test .github/scripts/tests/ci-cache-policy.test.cjs .github/scripts/query-regression-comment.test.cjs .github/scripts/heavy-ci-policy.test.cjs .github/scripts/tests/nightly-merge.test.cjs - name: Setup uv uses: astral-sh/setup-uv@v3 with: @@ -44,6 +44,7 @@ runs: python3 tests/perf/test_query_regression_runner_otlp_trace_load.py python3 tests/perf/test_query_regression_summary_otlp.py python3 tests/perf/test_query_regression_case_selection.py + python3 tests/perf/test_query_regression_helpers.py python3 tests/perf/test_query_regression_nightly_refs.py python3 tests/perf/test_query_regression_slash.py python3 tests/perf/test_aliyun_ecs_runner_scripts.py diff --git a/.github/actions/setup-benchmark/action.yml b/.github/actions/setup-linux-runner/action.yml similarity index 60% rename from .github/actions/setup-benchmark/action.yml rename to .github/actions/setup-linux-runner/action.yml index a446a78a2c3..95de8e465c5 100644 --- a/.github/actions/setup-benchmark/action.yml +++ b/.github/actions/setup-linux-runner/action.yml @@ -1,19 +1,41 @@ -name: Set up benchmark dependencies +name: Set up Linux runner dependencies description: Reuse installed tools and install missing dependencies on an Ubuntu self-hosted runner +inputs: + tools: + description: Space-separated commands to install if missing (Ubuntu package names, except shasum) + required: false + default: curl + expected-architecture: + description: Optional uname architecture required by the workload + required: false + default: '' runs: using: composite steps: - name: Check host and install missing tools shell: bash + env: + REQUIRED_TOOLS: ${{ inputs.tools }} + EXPECTED_ARCH: ${{ inputs.expected-architecture }} run: | set -euo pipefail - [[ "$(uname -m)" == x86_64 ]] + if [[ -n "$EXPECTED_ARCH" ]]; then + [[ "$(uname -m)" == "$EXPECTED_ARCH" ]] + fi command -v git missing=() - for tool in curl jq python3 rsync; do - command -v "$tool" >/dev/null || missing+=("$tool") + read -r -a tools <<< "$REQUIRED_TOOLS" + for tool in "${tools[@]}"; do + if ! command -v "$tool" >/dev/null; then + case "$tool" in + shasum) missing+=(libdigest-sha-perl) ;; + *) missing+=("$tool") ;; + esac + fi done if ((${#missing[@]})); then + . /etc/os-release + [[ "$ID" == ubuntu ]] sudo -n apt-get update sudo -n env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get install -y --no-install-recommends "${missing[@]}" fi @@ -24,6 +46,8 @@ runs: if ! command -v docker >/dev/null; then . /etc/os-release [[ "$ID" == ubuntu ]] + sudo -n apt-get update + sudo -n env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get install -y --no-install-recommends curl ca-certificates sudo -n install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo -n tee /etc/apt/keyrings/docker.asc >/dev/null sudo -n chmod a+r /etc/apt/keyrings/docker.asc diff --git a/.github/actions/start-runner/action.yml b/.github/actions/start-runner/action.yml index 44a5ff3feb3..17f07ae6324 100644 --- a/.github/actions/start-runner/action.yml +++ b/.github/actions/start-runner/action.yml @@ -20,6 +20,9 @@ inputs: image-id: description: The EC2 image id required: true + ec2-volume-size: + description: Root volume size in GiB (uses the AMI default when omitted) + required: false security-group-id: description: The EC2 security group id required: true @@ -61,6 +64,7 @@ runs: with: mode: start ec2-image-id: ${{ inputs.image-id }} + ec2-volume-size: ${{ inputs.ec2-volume-size }} ec2-instance-type: ${{ steps.get-ec2-instance-type.outputs.instance-type }} subnet-id: ${{ inputs.subnet-id }} security-group-id: ${{ inputs.security-group-id }} diff --git a/.github/ci/cache-policy.md b/.github/ci/cache-policy.md new file mode 100644 index 00000000000..d1eb053ec1c --- /dev/null +++ b/.github/ci/cache-policy.md @@ -0,0 +1,45 @@ +# CI cache policy + +Checks, Rust CI, Integration CI, and Nightly CI use one explicit Rust cache per +build job. Disable the implicit cache in `setup-rust-toolchain` when adding an +explicit cache: otherwise its independent save policy can write PR-scoped +caches even when the explicit cache only saves on main. + +Integration CI's existing weekday schedule runs the fuzz builds and suites, +including the profile-CI binary and Kafka helper, to populate main caches. +To warm these caches manually after deployment, run Integration CI on main with +`fuzz_profile: all`; no internal `ci_command` marker is required. + +Rust CI's existing weekday schedule warms the Windows and RISC-V compilation +caches. Nightly CI's existing schedule warms its Windows runtime test caches. +A manual Rust CI run on main can warm the compilation caches sooner. These jobs +keep their separate keys and existing runner types; no extra warmup job is added. +Coverage caches are not warmed by these schedules. +PR, merge-group, and non-main manual runs restore caches without saving them. +Fuzz binaries still flow to downstream jobs through workflow artifacts on every +run; only the persistent binary cache is main-written. Re-running a PR cannot +reuse a newly saved PR-specific binary cache under this policy. + +Keep helper version variables outside the `CARGO*` and `RUST*` prefixes unless +they affect the compilation being cached. rust-cache includes those prefixes in +its environment hash. Fuzz tool versions remain explicitly included in the fuzz +cache keys. Compiler versions, Cargo manifests, flags, OS, and architecture +remain part of the existing cache isolation. + +This policy reduces duplicate and PR-only writes; it does not guarantee that all +build caches fit in the repository quota. Renamed environment variables cause +an initial cold restore until main produces the new keys. Existing caches are +not deleted by this change. Coverage warming and additional runner capacity are +not introduced. + +Validate the policy with: + +```sh +node --test .github/scripts/tests/ci-cache-policy.test.cjs +``` + +After deployment, compare the same commit and runner image across repeated runs. +Record exact versus fallback restores, archive size, restore/save duration, and +compilation duration. Verify main writes the keys consumed by PRs and that no +implicit post-action uploads remain. Local policy tests do not establish a +runtime speedup or long-term cache hit rate. diff --git a/.github/ci/scope-rules.toml b/.github/ci/scope-rules.toml index 690a398db2e..05ee736d625 100644 --- a/.github/ci/scope-rules.toml +++ b/.github/ci/scope-rules.toml @@ -32,8 +32,11 @@ always_run = [ ".github/ci/**", ".github/workflows/integration.yml", ".github/actions/**", - # These helpers run through composite actions and affect test execution. + # Both run through a composite action, so neither appears in the workflow: + # `setup-kind` starts the cluster for the distributed and chaos jobs, and + # `fuzz-test` seeds the corpus every fuzz job starts from. ".github/scripts/kind-with-registry.sh", + ".github/scripts/prepare-fuzz-corpus.sh", ".github/scripts/run-fuzz-targets.sh", ".github/scripts/collect-fuzz-target-artifacts.sh", ".github/scripts/wait-kafka-ready.sh", @@ -100,6 +103,7 @@ server = [ "src/datanode/**", "src/common/base/**", "src/common/batcher/**", + "src/common/cache/**", "src/common/catalog/**", "src/common/config/**", "src/common/datasource/**", @@ -137,6 +141,7 @@ engine = [ "src/mito-codec/**", "src/metric-engine/**", "src/file-engine/**", + "src/relational-engine/**", "src/log-store/**", "src/common/wal/**", ] diff --git a/.github/ci/tests/fixtures/scope-cases.toml b/.github/ci/tests/fixtures/scope-cases.toml index 3b52024a956..d159bd3e32d 100644 --- a/.github/ci/tests/fixtures/scope-cases.toml +++ b/.github/ci/tests/fixtures/scope-cases.toml @@ -150,9 +150,9 @@ paths = [ suites = ["*"] [[case]] -name = "fuzz target orchestration: every suite" -note = "Fuzz jobs execute targets through the `fuzz-test` action." +name = "fuzz corpus seeding: every suite" +note = "Every fuzz job seeds its corpus through the `fuzz-test` action first." paths = [ - ".github/scripts/run-fuzz-targets.sh", + ".github/scripts/prepare-fuzz-corpus.sh", ] suites = ["*"] diff --git a/.github/heavy-ci.md b/.github/heavy-ci.md index 01179861ced..4b0d17bc3a3 100644 --- a/.github/heavy-ci.md +++ b/.github/heavy-ci.md @@ -5,8 +5,8 @@ action before expensive builds and tests. A failed, skipped, or cancelled script-test job blocks those jobs, including manually dispatched runs. Each workflow runs its own preflight so it does not wait on another workflow run. -Rust CI runs Windows and RISC-V compilation checks only on schedule or manual -dispatch, not on PRs or merge groups. Windows nextest and Windows SQLness run in the existing Nightly CI workflow, which also +Rust CI keeps the Windows compilation check on PRs and merge groups. Windows +nextest and Windows SQLness run in the existing Nightly CI workflow, which also supports manual dispatch. Its existing schedule is Tuesday through Friday at 07:00 Asia/Shanghai, not every day. No new Windows test job is added. @@ -26,12 +26,3 @@ for same-repository draft PRs with their existing authorization rules. Validate job admission with `node --test .github/scripts/heavy-ci-policy.test.cjs`. These tests verify triggering decisions, not runner availability or performance. - -## Narthex branch - -GitHub schedules run only on the default branch. They do not test `narthex` -automatically. To validate this branch, manually dispatch Rust CI, Integration -CI (selecting a fuzz profile), or Nightly CI with `narthex` as the ref. - -Land these scope rules on `narthex` before updating dependent PR branches: -the classifier checks out the PR base revision, not the PR head. diff --git a/.github/nightly-merge.md b/.github/nightly-merge.md new file mode 100644 index 00000000000..cc9419ec652 --- /dev/null +++ b/.github/nightly-merge.md @@ -0,0 +1,49 @@ +# Nightly merge + +The PR template opts new PRs into the `merge nightly` label. The Nightly Merge +workflow reconciles the label on PR creation, edits, and reopening. Existing PRs +without the checkbox can opt in by adding the label manually; label sync leaves +their labels unchanged. Unchecking removes the label; it does not +request an immediate merge or cancel an existing queue entry/auto-merge request. + +Every day at 02:10, 03:10, 04:10, 05:10, 06:10, 07:10, and 08:10 +Asia/Shanghai (UTC+8), the workflow enqueues up to five eligible PRs targeting +`main` per run, oldest first. Already queued PRs do not count toward this limit. +GitHub may delay scheduled runs. +The label alone controls nightly opt-in; the enqueue job does not check the PR +body. PRs must be approved and clean, and have all +review threads resolved. Drafts are skipped. +If GitHub rejects enqueueing because a PR belongs to a stack, the workflow skips +it and continues without failing the run or counting it toward the five-PR limit. +Other API errors still fail the run. Dry runs do not call the enqueue API, so +they cannot detect this stack rejection. +Already queued PRs and manually enabled auto-merge requests are left alone; +Maintainers can use **Merge when ready** to proceed earlier. Unready PRs wait for a later +scheduled run; this workflow never enables auto-merge on their behalf. + +## Repository setup + +- Create the `merge nightly` label before enabling the workflow. +- Require the merge queue, reviews, status checks, and conversation resolution + on `main`. Conversation resolution must also be enforced by GitHub to cover + new comments arriving between the script's check and enqueueing. +- Ensure the existing `GH_PERSONAL_ACCESS_TOKEN` automation secret can read PRs + and review threads and enqueue PRs in this repository. Do not grant it a + branch-rule bypass. This credential is already referenced by other workflows; + its actual permissions have not been validated by the local tests. +- Run **Nightly Merge** manually with the default `dry_run: true` first and + inspect its job summary. Setting it to false explicitly enqueues outside the + scheduled time as well. Merge queue build concurrency is configured separately + in GitHub; this workflow does not change it. + +The workflow never checks out or executes PR code. Label reconciliation uses +`GITHUB_TOKEN`; enqueueing uses the automation credential so downstream CI can +run. No runner remains allocated while waiting for the nightly schedule. + +## Local validation + +Run `actionlint .github/workflows/nightly-merge.yml .github/workflows/checks.yml` +and `node --test .github/scripts/tests/nightly-merge.test.cjs`. +The tests execute the actual inline workflow scripts with mocked GitHub APIs, +including one-factor comparisons against an eligible PR. They do not validate +live GitHub permissions, branch rules, or runner savings. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index a33531c2733..5b26f64ceb3 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -25,3 +25,14 @@ Please convert it to a draft if some of the following conditions are not met. - [ ] API changes are backward compatible. - [ ] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-` labels (e.g. `backport-v1.3` targets `release/v1.3`). + +## Merge timing + +- [x] Merge during the nightly window + + diff --git a/.github/runner-scale-sets/query-regression/README.md b/.github/runner-scale-sets/query-regression/README.md index 44500a8e50f..748821966d6 100644 --- a/.github/runner-scale-sets/query-regression/README.md +++ b/.github/runner-scale-sets/query-regression/README.md @@ -80,8 +80,13 @@ after its TTL, so finish the inspection within that window. Trust model on the ECS path: the instance receives only the one-hour runner registration token via user data and holds no cloud credentials; the Aliyun AK/SK exist only in the control-plane jobs. Instance tags -(`managed-by=query-regression-ci`, `query-regression-run-id`) feed the janitor -sweep. +(`managed-by=query-regression-ci`, `query-regression-run-id`, +`github-repository`) feed the janitor sweep. The sweep only selects instances +whose `github-repository` matches the requested repository. Older instances +without that tag are excluded: verify their ownership before adding the tag, +or clean them up by explicit instance ID. Per-run teardown remains unchanged. +Update every repository sharing the account and region to the scoped helper; +an older janitor can still sweep instances belonging to other repositories. ### Building and updating the ECS image diff --git a/.github/scripts/agent-observability-summary.py b/.github/scripts/agent-observability-summary.py index 33ba3ad7e6e..29e3e995536 100644 --- a/.github/scripts/agent-observability-summary.py +++ b/.github/scripts/agent-observability-summary.py @@ -133,9 +133,10 @@ def benchmark(root, targets): return '\n'.join(lines) -def lifecycle(root, targets): - manifest = read(root / 'run-manifest.json') - corpus = read(root / 'corpus' / 'summary.json') +def lifecycle(root, targets, data_root=None): + data_root = data_root or root + manifest = read(data_root / 'run-manifest.json') + corpus = read(data_root / 'corpus' / 'summary.json') lines = ['# Load / lifecycle summary', '', f'Dataset: **{cell(manifest.get("profile", corpus.get("data_profile", {}).get("id")))}**; rows: **{cell(corpus.get("line_count"))}**; ' f'seed: `{cell(corpus.get("seed"))}`.', @@ -161,12 +162,13 @@ def lifecycle(root, targets): physical_rows = [] load_rows = [] details = ['
', 'DDL, index configuration and image provenance', ''] - raw_bytes = read(root / 'corpus' / 'size.json').get('raw_jsonl_bytes') + raw_bytes = read(data_root / 'corpus' / 'size.json').get('raw_jsonl_bytes') for t in targets: directory = result_dir(root, t) physical = read(directory / 'physical-evidence.json') storage = physical.get('storage', {}) - total = storage.get('engine_total_bytes') + s3 = read(root / f'{t}-1' / 'storage.json').get('type') == 's3' + total = None if s3 else storage.get('engine_total_bytes') stats = read(root / f'{t}-1' / 'post-load-docker-stats.json') ratio = f'{raw_bytes / total:.2f}×' if (physical.get('passed') is True and isinstance(raw_bytes, (int, float)) and raw_bytes > 0 and @@ -186,6 +188,7 @@ def lifecycle(root, targets): lines += ['## Load and physical evidence', '', table(['DB', 'Write transport', 'Batches', 'Load smoke rows'], load_rows), '', f'Raw JSONL size: **{size(raw_bytes)}**. Ratio = raw JSONL / engine bytes at collection time; ' 'not a settled-storage guarantee. CPU/memory are post-load snapshots, not peaks. ' + 'S3 total/compression is omitted: local files exclude remote objects. ' 'Missing artifacts remain blank; pre-cleanup stats are not substituted.', '', table(['DB', 'Engine size', 'Size ratio', 'Materialization', 'SSTs / active parts', 'Post-load CPU', 'Post-load memory / limit'], physical_rows), '', *details, '
', ''] diff --git a/.github/scripts/aliyun-ecs-runner-provision.py b/.github/scripts/aliyun-ecs-runner-provision.py index 46efbffd3ec..2d026c00bc0 100644 --- a/.github/scripts/aliyun-ecs-runner-provision.py +++ b/.github/scripts/aliyun-ecs-runner-provision.py @@ -59,6 +59,7 @@ MANAGED_BY_TAG_KEY = "managed-by" MANAGED_BY_TAG_VALUE = "query-regression-ci" RUN_TAG_KEY = "query-regression-run-id" TTL_TAG_KEY = "runner-ttl-hours" +REPO_TAG_KEY = "github-repository" # Runner cache paths on the instance system disk. They are created empty # every provision and deleted with the VM. @@ -359,6 +360,7 @@ def run_instance(client, config: ProvisionConfig, user_data: str) -> str: key=MANAGED_BY_TAG_KEY, value=MANAGED_BY_TAG_VALUE ), ecs_models.RunInstancesRequestTag(key=RUN_TAG_KEY, value=config.run_id), + ecs_models.RunInstancesRequestTag(key=REPO_TAG_KEY, value=config.repo), ] + ( [ diff --git a/.github/scripts/aliyun-ecs-runner-teardown.py b/.github/scripts/aliyun-ecs-runner-teardown.py index d5b30b2922d..85742c1f694 100644 --- a/.github/scripts/aliyun-ecs-runner-teardown.py +++ b/.github/scripts/aliyun-ecs-runner-teardown.py @@ -32,8 +32,8 @@ Two modes: runner by name. Both steps are idempotent and best-effort; a missing instance or runner is not an error. - Sweep (scheduled janitor): delete every instance tagged as managed by - query-regression CI whose creation time is older than its tagged TTL (or - the given fallback TTL for untagged instances), and + query-regression CI in the requested repository whose creation time is older + than its tagged TTL (or the fallback TTL for instances without a TTL tag), and deregister the matching runners. This is the safety net for runs whose teardown job never executed. """ @@ -162,7 +162,7 @@ def delete_instance(client, instance_id: str, region_id: str | None = None) -> b def list_managed_instances( - client, region_id: str + client, region_id: str, repo: str ) -> list[tuple[str, str, str, str | None]]: from alibabacloud_ecs20140526 import models as ecs_models @@ -175,6 +175,9 @@ def list_managed_instances( ecs_models.DescribeInstancesRequestTag( key=provision.MANAGED_BY_TAG_KEY, value=provision.MANAGED_BY_TAG_VALUE, + ), + ecs_models.DescribeInstancesRequestTag( + key=provision.REPO_TAG_KEY, value=repo ) ], max_results=100, @@ -201,7 +204,7 @@ def list_managed_instances( def sweep(client, region_id: str, repo: str, github_token: str, ttl: timedelta) -> int: - instances = list_managed_instances(client, region_id) + instances = list_managed_instances(client, region_id, repo) print(f"Found {len(instances)} managed instance(s) in {region_id}", flush=True) expired = expired_instance_names(instances, datetime.now(timezone.utc), ttl) ok = True @@ -227,7 +230,7 @@ def main() -> int: parser.add_argument( "--sweep", action="store_true", - help="Janitor mode: delete all managed instances older than --sweep-ttl-hours.", + help="Janitor mode: delete expired managed instances tagged for --repo.", ) parser.add_argument("--sweep-ttl-hours", type=float, default=SWEEP_TTL.total_seconds() / 3600) args = parser.parse_args() diff --git a/.github/scripts/benchmark-version-comparison.py b/.github/scripts/benchmark-version-comparison.py index 1aa744df9ad..527797a2cb5 100644 --- a/.github/scripts/benchmark-version-comparison.py +++ b/.github/scripts/benchmark-version-comparison.py @@ -62,11 +62,12 @@ def row(query, metric, old, new, compatible, reason): improvement_percent=delta, status='comparable' if delta is not None else reason) -def agent(root): +def agent(root, storage="file"): + candidate, baseline = (root, root / "baseline") if storage == "file" else (root / "s3", root / "baseline/s3") def load(r): return read(r / 'greptimedb-1/greptimedb-1/measured-summary.json') - a, b = load(root / 'baseline'), load(root) - finished = [r / 'greptimedb-1/exit-code.txt' for r in (root / 'baseline', root)] + a, b = load(baseline), load(candidate) + finished = [r / 'greptimedb-1/exit-code.txt' for r in (baseline, candidate)] compatible = (all(p.is_file() and p.read_text().strip() == '0' for p in finished) and a.get('passed') is True and b.get('passed') is True and same(a, b, ('contract_sha256', 'data_profile', 'query_window', 'warmup_runs', 'measured_runs'))) @@ -81,8 +82,9 @@ def agent(root): return rows -def traces(root): - a, b = (read(r / 'runs/greptimedb/run.json') for r in (root / 'baseline', root)) +def traces(root, storage="file"): + candidate, baseline = (root, root / "baseline") if storage == "file" else (root / "s3", root / "baseline/s3") + a, b = (read(r / 'runs/greptimedb/run.json') for r in (baseline, candidate)) compatible = (a.get('success') is True and b.get('success') is True and same(a, b, ('expected', 'protocol', 'audit_expected', 'corpus')) and same(a.get('execution', {}), b.get('execution', {}), @@ -110,18 +112,22 @@ def traces(root): return rows -def long_range(root): +def long_range(root, storage="file"): + names = ("greptimedb-baseline", "greptimedb") if storage == "file" else ("greptimedb-s3-baseline", "greptimedb-s3") + def target_root(name): + nested = root / "o11ybench" / ("o11ybench-" + name) + return nested if nested.exists() else root rows = [] - manifest = read(root / 'run-manifest.json') + manifests = {name: read(target_root(name) / 'run-manifest.json') for name in names} healthy = all( - all(field in (manifest.get(phase, {}).get(target, {}).get('docker', {}) if phase == 'post_load_results' - else manifest.get(phase, {}).get(target, {})).get('state', '').split() + all(field in (manifests[target].get(phase, {}).get(target, {}).get('docker', {}) if phase == 'post_load_results' + else manifests[target].get(phase, {}).get(target, {})).get('state', '').split() for field in ('Running=true', 'ExitCode=0', 'OOMKilled=false')) for phase in ('post_load_results', 'post_query_results') - for target in ('greptimedb-baseline', 'greptimedb')) + for target in names) for mode in ('serial', 'concurrent-duration-c1', 'concurrent-duration-c4'): - a, b = (read(root / name / 'results-vmbench-long-range' / (mode + '-summary.json')) - for name in ('greptimedb-baseline', 'greptimedb')) + a, b = (read(target_root(name) / name / 'results-vmbench-long-range' / (mode + '-summary.json')) + for name in names) fields = ('workload', 'query_file_sha256', 'start', 'end', 'step', 'profile', 'data_duration', 'scrape_interval', 'cache_policy_effective', 'mode', 'connection_mode', 'request_timeout_seconds') fields += ('tries', 'warmup_runs') if mode == 'serial' else ('concurrency', 'duration_seconds', 'warmup_seconds') @@ -178,7 +184,13 @@ def main(): parser.error('report requires --root and --kind') document = {'kind': args.kind, 'baseline_tag': os.environ.get('BASELINE_GREPTIMEDB_TAG', ''), 'candidate_tag': os.environ.get('GREPTIMEDB_TAG', ''), - 'rows': {'agent': agent, 'traces': traces, 'long-range': long_range}[args.kind](args.root)} + 'rows': []} + mode = os.environ.get('GREPTIMEDB_STORAGE', 'file') + for storage in (['file', 's3'] if mode == 'both' else [mode]): + rows = {'agent': agent, 'traces': traces, 'long-range': long_range}[args.kind](args.root, storage) + for item in rows: + item['query'] = storage.upper() + '/' + item['query'] + document['rows'].extend(rows) text = render(document) # Preserve partial comparisons even if a target failed before creating output. args.root.mkdir(parents=True, exist_ok=True) diff --git a/.github/scripts/build-query-regression-helpers.sh b/.github/scripts/build-query-regression-helpers.sh new file mode 100644 index 00000000000..6b6cdb6fcde --- /dev/null +++ b/.github/scripts/build-query-regression-helpers.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -euo pipefail +: "${CARGO_TARGET_DIR:?CARGO_TARGET_DIR must be configured by the runner}" +revision="$(git rev-parse HEAD)" +tag="query-regression-helpers:${revision}" +staging="$(mktemp -d)" +trap 'rm -rf -- "$staging"' EXIT +cargo build --profile nightly -p cmd --features dev-tools \ + --bin query_perf_fixture --bin query_regression_runner +for binary in query_perf_fixture query_regression_runner; do + "$CARGO_TARGET_DIR/nightly/$binary" --help >/dev/null + cp "$CARGO_TARGET_DIR/nightly/$binary" "$staging/$binary" +done +mkdir -p helpers-image helpers-validation +docker build --provenance=false --build-arg "SOURCE_REVISION=$revision" \ + --file docker/query-regression-helpers/Dockerfile --tag "$tag" "$staging" +container="$(docker create "$tag" /query_perf_fixture --help)" +trap 'docker rm --force --volumes "$container"; rm -rf -- "$staging"' EXIT +mkdir -p "$staging/extracted" +for binary in query_perf_fixture query_regression_runner; do + docker cp "$container:/$binary" "$staging/extracted/$binary" + chmod +x "$staging/extracted/$binary" + "$staging/extracted/$binary" --help >/dev/null + cmp "$staging/$binary" "$staging/extracted/$binary" +done +# Hold the revision and cases constant: compare source-built versus extracted +# fixture normalization twice. Keep the raw outputs as workflow artifacts. +printf '%s\n' "$revision" > helpers-validation/source-revision.txt +for iteration in 1 2; do + while IFS= read -r case; do + name="$(basename "$(dirname "$case")")" + for mode in source image; do + binary="$staging/query_perf_fixture" + [[ "$mode" != image ]] || binary="$staging/extracted/query_perf_fixture" + "$binary" plan --case "$case" > "helpers-validation/${name}-${iteration}-${mode}.json" + done + cmp "helpers-validation/${name}-${iteration}-source.json" \ + "helpers-validation/${name}-${iteration}-image.json" + done < <(find tests/perf/query_cases -name case.toml -type f | sort) +done +(cd "$staging" && sha256sum query_perf_fixture query_regression_runner) > helpers-image/SHA256SUMS +printf '%s\n' "$revision" > helpers-image/source-revision.txt +docker save "$tag" --output helpers-image/image.tar diff --git a/.github/scripts/ci-runners.md b/.github/scripts/ci-runners.md index 6e4b2b3a4f7..f21bdbdb0af 100644 --- a/.github/scripts/ci-runners.md +++ b/.github/scripts/ci-runners.md @@ -13,8 +13,12 @@ AWS uses `aws-ec2-runner-provision.py` and `aws-ec2-runner-teardown.py` directly Repository variables supply `EC2_RUNNER_REGION`, `EC2_RUNNER_SUBNET_ID`, and `EC2_RUNNER_SECURITY_GROUP_ID`; credentials use `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `GH_PERSONAL_ACCESS_TOKEN` secrets. -The current region is Singapore (`ap-southeast-1`). By default the script selects -the newest available Canonical Ubuntu 24.04 amd64 server AMI in that region +The runner region comes from `EC2_RUNNER_REGION` (currently `us-east-1`). +The subnet and security group must belong to the selected AWS account and region. +For benchmarks using real S3, `AWS_CI_TEST_BUCKET_REGION` must match the runner +region; `AWS_CI_TEST_BUCKET` must be accessible with the configured credentials. +By default the script selects the newest available Canonical Ubuntu 24.04 amd64 +server AMI in that region (owner `099720109477`) and logs its ID. Set `BENCHMARK_EC2_IMAGE_ID` to pin a prepared Ubuntu AMI. Existing build CI's `EC2_RUNNER_LINUX_AMD64_IMAGE_ID` is unaffected. Both workflows default `system_disk_gib` to `auto` (also accepting @@ -31,9 +35,12 @@ swappiness 10, disables systemd-oomd and background automatic updates, and sets `OOMPolicy=continue` on the runner service. Explicit setup package installation remains enabled. -Both benchmark jobs call `.github/actions/setup-benchmark` after checkout. It -reuses existing tools, installs missing curl/jq/python3/rsync, installs Docker CE -if absent, and checks Docker access. Missing packages require passwordless sudo; +Benchmark jobs call `.github/actions/setup-linux-runner` after checkout with +`tools: curl jq python3 rsync` and require x86_64. Linux artifact builds reuse +the same action with `tools: make shasum` on either amd64 or arm64; the RISC-V +cross-build calls it on amd64 too. The action reuses installed commands, maps +`shasum` to Ubuntu package `libdigest-sha-perl`, installs Docker CE if absent, +and checks Docker access. Missing packages require passwordless sudo; Aliyun's prepared image normally provides the tools and Docker already. Existing Docker installations are preserved, so exact versions need not match between providers. Provisioning must establish Docker permissions before starting the diff --git a/.github/scripts/metric-snapshot-s3-bench.py b/.github/scripts/metric-snapshot-s3-bench.py index 7bb01a410b1..6542a18ba04 100644 --- a/.github/scripts/metric-snapshot-s3-bench.py +++ b/.github/scripts/metric-snapshot-s3-bench.py @@ -105,8 +105,9 @@ class Benchmark: self.endpoint = os.environ.get("S3_ENDPOINT", "") if self.endpoint and not local_smoke: raise ValueError("Custom S3 endpoints are restricted to local smoke") - if not local_smoke and os.environ["S3_REGION"] != "ap-southeast-1": - raise ValueError("Cloud benchmark requires ap-southeast-1") + if not local_smoke and (not os.environ["S3_REGION"].strip() + or os.environ["S3_REGION"] != os.environ["RUNNER_REGION"]): + raise ValueError("Cloud benchmark requires the same nonempty bucket and runner region") self.containers = set() self.network_created = False self.phases = [] @@ -324,7 +325,7 @@ level = "info" lines = ["# Metric packed snapshot benchmark", "", f"Image: `{self.image}`", f"Source commit: `{self.commit}`", "", "Local smoke only; timings are not AWS results." if self.local_smoke else - "AWS S3 in ap-southeast-1; source and target use local File storage.", "", + f"S3 in {os.environ['S3_REGION']}; source and target use local File storage.", "", "| Round | Phase | Seconds | Result |", "| --- | --- | ---: | --- |"] for phase in self.phases: lines.append(f"| {phase['round']} | {phase['phase']} | {phase['wall_seconds']:.3f} | {phase['outcome']} |") diff --git a/.github/scripts/metric-snapshot-s3-ops.py b/.github/scripts/metric-snapshot-s3-ops.py index f1e34defb89..f639ec90979 100644 --- a/.github/scripts/metric-snapshot-s3-ops.py +++ b/.github/scripts/metric-snapshot-s3-ops.py @@ -21,7 +21,7 @@ """Budget preflight and strictly scoped S3 operations for the metric snapshot benchmark.""" import argparse -from decimal import Decimal +from decimal import Decimal, InvalidOperation import json import os from pathlib import Path @@ -29,8 +29,8 @@ import re import time import urllib.request -REGION = "ap-southeast-1" -HOURLY_USD = Decimal("0.9408") +# Historical budget estimate; dispatch can supply a rate for the configured region. +DEFAULT_HOURLY_USD = "0.9408" OTHER_RESERVE_USD = Decimal("1.50") @@ -40,6 +40,16 @@ def owned_prefix(prefix): return prefix +def hourly_estimate(env): + try: + estimate = Decimal(env.get("EC2_HOURLY_USD", DEFAULT_HOURLY_USD)) + except InvalidOperation as error: + raise ValueError("EC2_HOURLY_USD must be a positive finite estimate") from error + if not estimate.is_finite() or estimate <= 0: + raise ValueError("EC2_HOURLY_USD must be a positive finite estimate") + return estimate + + def plan(env): image = env["GREPTIMEDB_IMAGE"] if not re.fullmatch(r"greptime/greptimedb(?:-nightly)?@sha256:[0-9a-f]{64}", image): @@ -49,21 +59,22 @@ def plan(env): tables, minutes = int(env["LOGICAL_TABLES_PER_GROUP"]), int(env["BENCHMARK_TIMEOUT_MINUTES"]) if not 50 <= tables <= 10_000 or not 30 <= minutes <= 240: raise ValueError("logical tables per group must be 50..10000; timeout must be 30..240 minutes") - if env["S3_REGION"] != REGION or env["RUNNER_REGION"] != REGION: - raise ValueError("bucket and runner must both use ap-southeast-1") + if not env["S3_REGION"].strip() or env["S3_REGION"] != env["RUNNER_REGION"]: + raise ValueError("bucket and runner must use the same nonempty region") if not env["S3_BUCKET"]: raise ValueError("S3 bucket is required") prior = Decimal(env["PRIOR_SPEND_USD"]) # Reserve 45m provision, 10m queue allowance and 20m cleanup beyond the job timeout. - reservation = HOURLY_USD * Decimal(minutes + 75) / 60 + OTHER_RESERVE_USD + hourly = hourly_estimate(env) + reservation = hourly * Decimal(minutes + 75) / 60 + OTHER_RESERVE_USD if not prior.is_finite() or prior < 0 or prior >= 15 or prior + reservation > 20: raise ValueError("campaign budget exhausted: prior spend must be below $15 and leave the full attempt reserve within $20") return {"image": image, "expected_commit": env["EXPECTED_COMMIT"], "logical_tables_per_group": tables, "smoke_logical_tables": 500, "timeout_minutes": minutes, "prefix": owned_prefix(env["RUN_PREFIX"]), - "instance_type": "c6id.4xlarge", "region": REGION, + "instance_type": "c6id.4xlarge", "region": env["S3_REGION"], "prior_estimated_usd": str(prior), "attempt_reserve_usd": str(reservation), - "ec2_hourly_usd": str(HOURLY_USD), "other_reserve_usd": str(OTHER_RESERVE_USD), + "ec2_hourly_usd": str(hourly), "other_reserve_usd": str(OTHER_RESERVE_USD), "cost_note": "Estimate, not an AWS billing total or a hard spending cap."} @@ -71,8 +82,8 @@ def client(region): import boto3 from botocore.config import Config - if region != REGION: - raise ValueError("S3 region must be ap-southeast-1") + if not region.strip(): + raise ValueError("S3 region is required") return boto3.client("s3", region_name=region, config=Config( connect_timeout=10, read_timeout=30, retries={"mode": "standard", "max_attempts": 3})) @@ -193,8 +204,10 @@ def main(): if args.started_at is None: parser.error("cost requires --started-at") seconds = max(0, int(time.time()) - args.started_at) - estimate = HOURLY_USD * Decimal(seconds) / 3600 - result = {"allocation_window_seconds": seconds, "ec2_estimate_usd": str(round(estimate, 4)), + hourly = hourly_estimate(os.environ) + estimate = hourly * Decimal(seconds) / 3600 + result = {"allocation_window_seconds": seconds, "ec2_hourly_usd": str(hourly), + "ec2_estimate_usd": str(round(estimate, 4)), "campaign_estimate_with_other_reserve_usd": str(round( Decimal(os.environ["PRIOR_SPEND_USD"]) + estimate + OTHER_RESERVE_USD, 4)), "note": "Allocation wall-time estimate plus $1.50 reserve for S3, disk and IPv4; not an AWS bill."} diff --git a/.github/scripts/o11ybench-storage.py b/.github/scripts/o11ybench-storage.py new file mode 100644 index 00000000000..e050e933d95 --- /dev/null +++ b/.github/scripts/o11ybench-storage.py @@ -0,0 +1,316 @@ +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Select File/S3 benchmark arms without changing workload adapters.""" +import argparse +import copy +import importlib.util +import json +from pathlib import Path +import sys +import os +import subprocess + + +def validate(env): + mode = env.get('GREPTIMEDB_STORAGE', 'file') + if mode not in ('file', 's3', 'both'): + raise ValueError('GREPTIMEDB_STORAGE must be file, s3 or both') + if mode != 'file': + if env.get('RUN_GREPTIMEDB') != 'true': + raise ValueError('S3 requires Run GreptimeDB') + if env.get('PROVIDER') != 'AWS': + raise ValueError('S3 comparisons require an AWS runner') + if not env.get('S3_BUCKET') or not env.get('S3_REGION'): + raise ValueError('AWS_CI_TEST_BUCKET and AWS_CI_TEST_BUCKET_REGION must be configured') + if env['S3_REGION'] != env.get('RUNNER_REGION'): + raise ValueError('S3 bucket and EC2 runner regions must match') + return mode + + +def replace(command, key, value): + result = list(command) + if key in result: + result[result.index(key) + 1] = value + else: + result += [key, value] + return result + + +def commands(command, mode): + """Recovery visits each owned root; execution uses explicit workflow steps.""" + stage = command[command.index('--stage') + 1] + if stage != 'cleanup': + raise ValueError('only cleanup may expand multiple arms') + result = [list(command)] + if mode != 'file': + root = command[command.index('--root') + 1] + owner = command[command.index('--owner') + 1] + secondary = replace(command, '--root', root + '/s3') + result.append(replace(secondary, '--owner', owner + '-s3')) + return result + + +def module(path): + sys.path.insert(0, str(path.parent)) + spec = importlib.util.spec_from_file_location(path.stem.replace('-', '_'), path) + result = importlib.util.module_from_spec(spec) + spec.loader.exec_module(result) + return result + + +def summary(kind, root, targets, section, mode): + filename = 'agent-observability-summary.py' if kind == 'agent' else 'tracesbench-summary.py' + renderer = module(Path(__file__).with_name(filename)) + primary = [t for t in targets if t != 'greptimedb' or mode != 's3'] + arms = [(root, primary, 'File')] + if mode != 'file' and 'greptimedb' in targets: + arms.append((root / 's3', ['greptimedb'], 'S3')) + output = [] + for arm, selected, label in arms: + if not selected: + continue + output.append('## ' + ('GreptimeDB S3' if label == 'S3' else 'Primary targets (GreptimeDB File)')) + if kind == 'agent': + body = (renderer.benchmark(arm, selected) if section == 'benchmark' + else renderer.lifecycle(arm, selected, data_root=root.parent if root.name == 'baseline' else root)) + else: + report, runs = renderer.load(arm) + body = (renderer.benchmark(report, runs, selected) if section == 'benchmark' + else renderer.lifecycle(report, runs, arm, selected)) + body = body.replace('| greptimedb |', '| GreptimeDB ' + label + ' |') + body = body.replace('**greptimedb**', '**GreptimeDB ' + label + '**') + output.append(body) + if label == 'S3': + directories = [arm / 'greptimedb-1'] if kind == 'agent' else [arm / 'runs/greptimedb'] + for directory in directories: + for action in ('probe', 'inventory', 'cleanup'): + path = directory / ('s3-' + action + '.json') + data = json.loads(path.read_text()) if path.exists() else {} + status = 'passed' if data.get('passed') is True else 'failed / unavailable' + count = data.get('object_count', data.get('deleted_objects', '—')) + output.append(f'- S3 {action}: **{status}**; objects: {count}.') + return '\n\n'.join(output) + + +def split_metrics(plan): + """One executor root per target, with the immutable dataset shared by all.""" + result = {} + for target in plan['targets']: + child = copy.deepcopy(plan) + name = target['name'] + dest = str(Path(plan['output_root']) / 'o11ybench' / ('o11ybench-' + name)) + child['output_root'] = dest + child['run_id'] = plan['run_id'] + '-' + name + child['targets'] = [copy.deepcopy(target)] + child['data']['generate'] = False + child['report'].update(root=dest + '/report', index_html=dest + '/report/index.html') + child['artifacts'] = {k: dest + '/' + Path(v).name for k, v in plan['artifacts'].items()} + result[name] = child + return result + + +def metrics(action, args): + parser = argparse.ArgumentParser() + parser.add_argument('--plan', type=Path, required=True) + parser.add_argument('--tools', type=Path, required=True) + parser.add_argument('--runtime-image', required=True) + parser.add_argument('--target') + a = parser.parse_args(args) + plan = json.loads(a.plan.read_text()) + plan['runtime_image'] = a.runtime_image + executor = module(a.tools / 'scripts/run_benchmark_plan.py') + root = Path(plan['output_root']) + if action == 'metrics-prepare': + executor.require_safe_execute_root(root) + root.mkdir(parents=True, exist_ok=False) + manifest = executor.build_manifest(plan) + executor.run_checked([str(executor.DOWNLOAD_DATA)], manifest, + env=executor.generation_environment(plan)) + (root / 'generation.json').write_text(json.dumps(manifest, indent=2) + '\n') + for name, child in split_metrics(plan).items(): + (a.plan.parent / (name + '.json')).write_text(json.dumps(child, indent=2) + '\n') + elif action == 'metrics-run': + path = a.plan.parent / (a.target + '.json') + child = json.loads(path.read_text()) + executor.execute_plan(child, executor.build_manifest(child), replace=False) + subprocess.run([sys.executable, str(a.tools / 'scripts/check_run_benchmark_plan.py'), + '--artifacts', child['output_root']], check=True) + else: + # Validate every selected target together, retaining cross-target gates. + manifests = [executor.build_manifest(child) for child in split_metrics(plan).values()] + combined = executor.build_manifest(plan) + combined['targets'] = [target for m in manifests for target in m['targets']] + for command in executor.validation_commands(combined): + subprocess.run(command, check=True) + + +def overview(path): + """Render the same validated report data into the GitHub job summary.""" + data = json.loads(path.read_text()) + renderer = module(Path(__file__).with_name('agent-observability-summary.py')) + lines = ['# Storage comparison', '', 'Smoke results are diagnostic; see classification per arm. ' + 'No cold-object-store performance claim. Missing results are not passes.', ''] + for workload in data['workloads']: + labels = {p['id']: p['label'] for p in workload['participants']} + for scenario in workload['scenarios']: + lines += ['## ' + workload['title'] + ' / ' + scenario['label'], ''] + rows = [] + for result in scenario['results']: + lines += [labels[result['participant']] + ': ' + str(result.get('classification')), ''] + for query_id, query in result['queries'].items(): + m = query['metrics'] + rows.append([query_id, labels[result['participant']], query['status'], + renderer.duration(m.get('p50_ms', m.get('median_ms'))), + renderer.duration(m.get('p95_ms')), renderer.duration(m.get('p99_ms'))]) + lines.append(renderer.table(['Query', 'DB / storage', 'Status', 'P50 / median', 'P95', 'P99'], rows)) + return '\n'.join(lines) + + +def audit(root, report, targets, mode, baseline=False, allow_trace_query_failures=False): + data = json.loads(report.read_text()) + actual = {(p['target'], p.get('storage', {}).get('type'), p['role']) + for w in data['workloads'] for p in w['participants']} + expected = {(t, None, 'comparison') for t in targets if t != 'greptimedb'} + modes = ['file', 's3'] if mode == 'both' else [mode] + if 'greptimedb' in targets: + expected |= {('greptimedb', m, role) for m in modes + for role in (['candidate', 'baseline'] if baseline else ['candidate'])} + if actual != expected: + raise ValueError(f'participant mismatch: expected {sorted(map(str, expected))}, got {sorted(map(str, actual))}') + query_failures = [] + for w in data['workloads']: + partial = allow_trace_query_failures and w.get('id') == 'traces' + if partial and not w['scenarios']: + raise ValueError('missing trace scenarios') + for scenario in w['scenarios']: + if partial and {r['participant'] for r in scenario['results']} != {p['id'] for p in w['participants']}: + raise ValueError('missing trace participant results') + for result in scenario['results']: + if partial and result.get('health') != 'recorded healthy snapshots': + raise ValueError('missing or unhealthy trace evidence: ' + result['participant']) + queries = result['queries'] + if not any(q['status'] == 'passed' for q in queries.values()): + raise ValueError('no successful measured queries: ' + result['participant']) + if any(q['status'] == 'unavailable' or (q['status'] == 'failed' and not partial) + for q in queries.values()): + raise ValueError('failed or unavailable queries: ' + result['participant']) + for query_id, query in queries.items(): + if query['status'] == 'failed': + query_failures.append(dict(workload=w['id'], scenario=scenario['id'], + participant=result['participant'], query=query_id, + reason=query.get('reason', ''))) + for manifest_path in root.glob('o11ybench/*/run-manifest.json'): + manifest = json.loads(manifest_path.read_text()) + if manifest.get('status') != 'passed': + raise ValueError('incomplete target manifest: ' + str(manifest_path)) + probes = list(root.rglob('s3-probe.json')) + expected_s3 = sum(m == 's3' for _, m, _ in expected) + if len(probes) != expected_s3: + raise ValueError('missing or duplicate S3 evidence') + for probe in probes: + for action in ('probe', 'inventory', 'cleanup'): + payload = json.loads(probe.with_name('s3-' + action + '.json').read_text()) + if payload.get('passed') is not True or (action == 'inventory' and payload.get('object_count', 0) <= 0): + raise ValueError('failed S3 ' + action) + text = (root / 'ci-summary.md').read_text() + for m in modes if 'greptimedb' in targets else []: + if m.upper() not in text: + raise ValueError('storage mode missing from saved summary: ' + m) + result = dict(passed=True, participants=[dict(target=t, storage=m, role=role) + for t, m, role in sorted(expected, key=str)], s3_arms=len(probes), query_failures=query_failures) + (root / 'artifact-audit.json').write_text(json.dumps(result, indent=2) + '\n') + lines = ['## Artifact validation', '', '- Selected participants: ' + str(len(expected)) + '; all present.', + '- Query result coverage and S3 probe/inventory/cleanup: passed.', ''] + if query_failures: + lines += ['- Collection completed with query failures; this is not an all-pass query run.', ''] + lines += [f"- {f['participant']} / {f['query']}: failed (excluded from comparison)." + for f in query_failures] + lines.append('') + renderer = module(Path(__file__).with_name('agent-observability-summary.py')) + rows = [] + for probe in probes: + inv = json.loads(probe.with_name('s3-inventory.json').read_text()) + cleanup = json.loads(probe.with_name('s3-cleanup.json').read_text()) + rows.append([str(probe.parent.relative_to(root)), inv['object_count'], inv['object_bytes'], + cleanup.get('deleted_objects'), 'passed']) + if rows: + lines.append(renderer.table(['S3 arm', 'Objects', 'Remote bytes', 'Deleted current objects', 'Cleanup'], rows)) + text = '\n'.join(lines) + '\n' + with (root / 'ci-summary.md').open('a') as output: + output.write(text) + if os.environ.get('GITHUB_STEP_SUMMARY'): + with Path(os.environ['GITHUB_STEP_SUMMARY']).open('a') as output: + output.write(text) + return result + + +def main(): + if len(sys.argv) > 1 and sys.argv[1] == 'audit': + parser = argparse.ArgumentParser() + parser.add_argument('--root', type=Path, required=True) + parser.add_argument('--report', type=Path, required=True) + parser.add_argument('--targets', required=True) + parser.add_argument('--allow-trace-query-failures', action='store_true', + help='Retain failed Traces queries after native validation; missing results still fail') + args = parser.parse_args(sys.argv[2:]) + print(json.dumps(audit(args.root, args.report, args.targets.split(','), + os.environ.get('GREPTIMEDB_STORAGE', 'file'), + os.environ.get('COMPARE_GREPTIMEDB') == 'true', args.allow_trace_query_failures))) + return + if len(sys.argv) > 1 and sys.argv[1] == 'normalize-metrics': + path = Path(sys.argv[2]) + plan = json.loads(path.read_text()) + for target in plan['targets']: + if target.get('storage', {}).get('type') == 's3' and '-s3' not in target['name']: + target['name'] = target['name'].replace('-baseline', '-s3-baseline') if target['name'].endswith('-baseline') else target['name'] + '-s3' + path.write_text(json.dumps(plan, indent=2) + '\n') + return + if len(sys.argv) > 1 and sys.argv[1].startswith('metrics-'): + metrics(sys.argv[1], sys.argv[2:]) + return + if len(sys.argv) > 1 and sys.argv[1] == 'overview': + print(overview(Path(sys.argv[2]))) + return + if len(sys.argv) > 1 and sys.argv[1] == 'summary': + parser = argparse.ArgumentParser() + parser.add_argument('--kind', choices=['agent', 'traces'], required=True) + parser.add_argument('--root', type=Path, required=True) + parser.add_argument('--targets', required=True) + parser.add_argument('--section', choices=['benchmark', 'lifecycle'], required=True) + args = parser.parse_args(sys.argv[2:]) + print(summary(args.kind, args.root, args.targets.split(','), args.section, + os.environ.get('GREPTIMEDB_STORAGE', 'file'))) + return + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('action', choices=['validate', 'run']) + parser.add_argument('command', nargs=argparse.REMAINDER) + args = parser.parse_args() + mode = validate(os.environ) + if args.action == 'validate': + print('GreptimeDB storage: ' + mode) + return + command = args.command[1:] if args.command[:1] == ['--'] else args.command + planned = commands(command, mode) + failed = 0 + for cmd in planned: + result = subprocess.run(cmd) + failed = failed or result.returncode + # Cleanup must try both arms even if one failed. Target failures remain + # visible, while the other arm can still produce diagnostic evidence. + raise SystemExit(failed) + + +if __name__ == '__main__': + main() diff --git a/.github/scripts/prepare-fuzz-corpus.sh b/.github/scripts/prepare-fuzz-corpus.sh new file mode 100644 index 00000000000..e647cf9ceaf --- /dev/null +++ b/.github/scripts/prepare-fuzz-corpus.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +set -euo pipefail + +: "${FUZZ_TARGETS:?FUZZ_TARGETS is required}" +: "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE is required}" +: "${GITHUB_OUTPUT:?GITHUB_OUTPUT is required}" + +targets=() +while IFS= read -r target; do + target="${target%$'\r'}" + [[ -z "${target}" ]] && continue + if [[ ! "${target}" =~ ^[a-zA-Z0-9_]+$ ]]; then + printf 'Invalid fuzz target: %s\n' "${target}" >&2 + exit 2 + fi + targets+=("${target}") +done <<<"${FUZZ_TARGETS}" + +if [[ "${#targets[@]}" -eq 0 ]]; then + printf 'FUZZ_TARGETS contains no targets\n' >&2 + exit 2 +fi + +# Corpus identity depends on the targets, not the source revision or their order. +target_set="$(printf '%s\n' "${targets[@]}" | LC_ALL=C sort -u)" +target_hash="$(printf '%s\n' "${target_set}" | sha256sum | cut -d ' ' -f 1)" +corpus_root="${GITHUB_WORKSPACE}/tests-fuzz/corpus" + +{ + printf 'key-prefix=fuzz-corpus-v1-%s-\n' "${target_hash}" + printf 'paths<>"${GITHUB_OUTPUT}" diff --git a/.github/scripts/publish-query-regression-helpers.sh b/.github/scripts/publish-query-regression-helpers.sh new file mode 100644 index 00000000000..b867455f498 --- /dev/null +++ b/.github/scripts/publish-query-regression-helpers.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +set -euo pipefail +repository=greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/query-regression-helpers +revision="$(cat helpers-image/source-revision.txt)" +[[ "$revision" == "$GITHUB_SHA" ]] +tag="${repository}:${revision}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" +# Match release-dev-builder-images.yaml: publish ACR images with Skopeo from +# a hosted runner. A failed publish can reuse the completed build artifact. +mkdir -p helpers-validation +docker run --rm \ + -v "${DOCKER_CONFIG:-$HOME/.docker}:/root/.docker:ro" \ + -v "$PWD/helpers-image:/image:ro" \ + -v "$PWD/helpers-validation:/result" \ + -e REGISTRY_AUTH_FILE=/root/.docker/config.json \ + quay.io/skopeo/stable:latest --command-timeout 20m copy --retry-times 3 \ + --digestfile /result/digest.txt \ + docker-archive:/image/image.tar "docker://$tag" +digest="$(cat helpers-validation/digest.txt)" +[[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]] +image="$repository@$digest" +# The ECS build tested these exact bytes. Verify them again from the registry; +# hosted runners need not share the ECS runtime library versions. +python3 .github/scripts/query-regression-helpers.py "$image" helpers-validation/extracted \ + --checksums helpers-image/SHA256SUMS +jq -e --arg revision "$revision" '.source_revision == $revision' \ + helpers-validation/extracted/helpers-image.json >/dev/null +printf 'image=%s\n' "$image" >> "$GITHUB_OUTPUT" +printf 'Published helpers: `%s` (source `%s`)\n' "$image" "$revision" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/scripts/query-regression-helpers.py b/.github/scripts/query-regression-helpers.py new file mode 100644 index 00000000000..7f1e22a5c83 --- /dev/null +++ b/.github/scripts/query-regression-helpers.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Extract digest-pinned query regression tools from an artifact image.""" + +import argparse +import hashlib +import json +import os +import re +import subprocess +from pathlib import Path + +BINARIES = ("query_perf_fixture", "query_regression_runner") + + +def resolve_image(value): + if value == "pinned": + value = os.environ.get("QUERY_REGRESSION_HELPERS_IMAGE", "") + if not re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9._/:@-]*@sha256:[0-9a-f]{64}", value): + raise ValueError("helpers_image requires an image@sha256 digest; set QUERY_REGRESSION_HELPERS_IMAGE by publishing a verified image before using 'pinned'") + return value + + +def run(*args): + return subprocess.check_output(args, text=True).strip() + + +def extract(image, destination, checksums=None): + expected = None + if checksums is not None: + entries = [line.split() for line in checksums.read_text().splitlines()] + expected = {name: digest for digest, name in entries} + if set(expected) != set(BINARIES) or len(entries) != len(BINARIES): + raise ValueError("Checksums must contain exactly the two helper binaries") + image = resolve_image(image) + destination.mkdir(parents=True, exist_ok=True) + run("docker", "pull", image) + metadata = json.loads(run("docker", "image", "inspect", image))[0] + revision = (metadata.get("Config", {}).get("Labels") or {}).get("org.opencontainers.image.revision", "") + if not re.fullmatch(r"[0-9a-f]{40}", revision): + raise ValueError("Helper image is missing its source revision label") + container = run("docker", "create", image, "/query_perf_fixture", "--help") + try: + for binary in BINARIES: + target = destination / binary + run("docker", "cp", f"{container}:/{binary}", str(target)) + target.chmod(0o755) + if expected is None: + run(str(target.resolve()), "--help") + else: + with target.open("rb") as file: + actual = hashlib.file_digest(file, "sha256").hexdigest() + if actual != expected[binary]: + raise ValueError(f"Published helper checksum mismatch: {binary}") + finally: + run("docker", "rm", "--force", "--volumes", container) + provenance = {"image": image, "image_id": metadata["Id"], "source_revision": revision} + (destination / "helpers-image.json").write_text(json.dumps(provenance, indent=2) + "\n") + print(json.dumps(provenance)) + return provenance + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("image") + parser.add_argument("destination", type=Path) + parser.add_argument("--checksums", type=Path, help="Verify build checksums instead of executing on this host") + args = parser.parse_args() + extract(args.image, args.destination, args.checksums) diff --git a/.github/scripts/query-regression-run.py b/.github/scripts/query-regression-run.py index d488cee1594..e59dcdfb709 100644 --- a/.github/scripts/query-regression-run.py +++ b/.github/scripts/query-regression-run.py @@ -21,10 +21,13 @@ import argparse import json import os import re +import shlex import signal import socket import subprocess +import sys import time +import uuid import urllib.request from dataclasses import dataclass from pathlib import Path @@ -130,6 +133,7 @@ class RunTarget: datanode_http_port: int datanode_data_dir: Path frontend_config: Path | None = None + containerized: bool = False def allocate_ports(n: int) -> list[int]: @@ -151,6 +155,7 @@ def make_target( root: Path, ports: list[int], frontend_config: Path | None = None, + containerized: bool = False, ) -> RunTarget: work_dir = root / name return RunTarget( @@ -167,6 +172,7 @@ def make_target( datanode_http_port=ports[3], datanode_data_dir=work_dir / "datanode-0" / "data", frontend_config=frontend_config, + containerized=containerized, ) @@ -258,7 +264,11 @@ def stop_component( procs: dict[tuple[str, str], subprocess.Popen[bytes]], ) -> None: proc = procs.pop((target.name, name), None) - if proc is None or proc.poll() is not None: + if proc is None: + return + if proc.poll() is not None: + if target.containerized and proc.returncode != 0: + raise RuntimeError(f"Docker target {target.name}/{name} exited with {proc.returncode}") return try: os.killpg(proc.pid, signal.SIGTERM) @@ -272,6 +282,8 @@ def stop_component( except ProcessLookupError: return proc.wait(timeout=20) + if target.containerized and proc.returncode not in (0, 128 + signal.SIGTERM): + raise RuntimeError(f"Docker target {target.name}/{name} cleanup failed: {proc.returncode}") def restart_component( @@ -284,9 +296,15 @@ def restart_component( def stop_all(targets: list[RunTarget], procs: dict[tuple[str, str], subprocess.Popen[bytes]]) -> None: + errors = [] for target in reversed(targets): for name in ("frontend", "datanode", "metasrv"): - stop_component(target, name, procs) + try: + stop_component(target, name, procs) + except Exception as err: + errors.append(err) + if errors: + raise RuntimeError(f"component cleanup failed: {errors}") from errors[0] def load_plan(fixture_generator: Path, case_path: Path) -> dict[str, Any]: @@ -313,8 +331,8 @@ def run_direct_case( ) -> int: ports = allocate_ports(16) targets = [ - make_target("base", base_bin, work_dir, ports[:8]), - make_target("candidate", candidate_bin, work_dir, ports[8:]), + make_target("base", base_bin, work_dir, ports[:8], containerized=bool(getattr(args, "base_image", ""))), + make_target("candidate", candidate_bin, work_dir, ports[8:], containerized=bool(getattr(args, "candidate_image", ""))), ] for target in targets: if target.work_dir.exists() and any(target.work_dir.iterdir()): @@ -407,6 +425,7 @@ def run_remote_case( work_dir, ports[:8], work_dir / "base" / "frontend-prom-store.toml", + containerized=bool(getattr(args, "base_image", "")), ), make_target( "candidate", @@ -414,6 +433,7 @@ def run_remote_case( work_dir, ports[8:], work_dir / "candidate" / "frontend-prom-store.toml", + containerized=bool(getattr(args, "candidate_image", "")), ), ] for target in targets: @@ -495,8 +515,8 @@ def run_otlp_case( raise ValueError("--otelgen-bin is required for otlp_trace_load") ports = allocate_ports(16) targets = [ - make_target("base", base_bin, work_dir, ports[:8]), - make_target("candidate", candidate_bin, work_dir, ports[8:]), + make_target("base", base_bin, work_dir, ports[:8], containerized=bool(getattr(args, "base_image", ""))), + make_target("candidate", candidate_bin, work_dir, ports[8:], containerized=bool(getattr(args, "candidate_image", ""))), ] for target in targets: if target.work_dir.exists() and any(target.work_dir.iterdir()): @@ -634,6 +654,106 @@ def write_failed_report(work_dir: Path, case_path: Path, error: Exception) -> No ) +def docker_target(image: str, root: Path, command: list[str]) -> int: + """Expose an image as a binary, including read-bench subcommands. + + The caller terminates a process group. Keep Docker clients in separate + sessions so cleanup can stop the container before SST inspection starts. + """ + name = f"query-regression-{uuid.uuid4().hex}" + + def interrupted(signum, _frame): + raise SystemExit(128 + signum) + + previous = {sig: signal.signal(sig, interrupted) for sig in (signal.SIGTERM, signal.SIGINT)} + try: + subprocess.run([ + "docker", "create", "--name", name, "--network", "host", + "--user", f"{os.getuid()}:{os.getgid()}", + "--volume", f"{root}:{root}", "--workdir", str(root), + "--entrypoint", "/greptime/bin/greptime", image, *command, + ], check=True, stdout=subprocess.DEVNULL, start_new_session=True) + attached = subprocess.run( + ["docker", "start", "--attach", name], check=False, start_new_session=True, + ) + if attached.returncode != 0: + return attached.returncode + # docker start's status is not the contract for the container's exit code. + result = subprocess.run( + ["docker", "inspect", "--format", "{{.State.ExitCode}}", name], + check=True, capture_output=True, text=True, start_new_session=True, + ) + return int(result.stdout.strip()) + finally: + for sig in previous: + signal.signal(sig, signal.SIG_IGN) + try: + subprocess.run( + ["docker", "stop", "--time", "5", name], check=False, + stdout=subprocess.DEVNULL, timeout=8, start_new_session=True, + ) + except subprocess.TimeoutExpired: + # Force-removal below still waits for the container to stop. + pass + finally: + try: + subprocess.run( + ["docker", "rm", "--force", "--volumes", name], check=True, + stdout=subprocess.DEVNULL, timeout=5, start_new_session=True, + ) + finally: + for sig, handler in previous.items(): + signal.signal(sig, handler) + + +def prepare_targets(args: argparse.Namespace) -> dict[str, Any]: + """Resolve images once and create binary-compatible launchers for all cases.""" + metadata = {} + images = {} + for name in ("base", "candidate"): + image = getattr(args, f"{name}_image") + ref = getattr(args, f"{name}_ref") + if not image: + metadata[name] = {"kind": "source", "ref": os.environ.get(f"VERIFIED_{name.upper()}_SHA") or ref} + continue + if sys.platform != "linux": + raise ValueError("Docker targets require Linux host networking and a local Docker daemon") + if image.startswith("-") or any(c.isspace() for c in image): + raise ValueError(f"invalid {name} image reference: {image!r}") + if image not in images: + result = subprocess.run( + ["docker", "image", "inspect", image], check=False, capture_output=True, text=True, + ) + if result.returncode != 0: + subprocess.run(["docker", "pull", image], check=True) + result = subprocess.run( + ["docker", "image", "inspect", image], check=True, capture_output=True, text=True, + ) + images[image] = json.loads(result.stdout)[0] + info = images[image] + image_id = info["Id"] + if not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id): + raise ValueError(f"invalid Docker image ID: {image_id!r}") + args.work_dir = args.work_dir.resolve() + if ":" in str(args.work_dir): + raise ValueError("Docker target work directory cannot contain ':'") + launchers = args.work_dir / ".targets" + launchers.mkdir(parents=True, exist_ok=True) + launcher = launchers / name + command = [sys.executable, str(Path(__file__).resolve()), + "docker-target", image_id, str(args.work_dir)] + launcher.write_text("#!/bin/sh\nexec " + shlex.join(command) + ' "$@"\n', encoding="utf-8") + launcher.chmod(0o755) + setattr(args, f"{name}_bin", launcher) + setattr(args, f"{name}_ref", f"{image} ({image_id})") + metadata[name] = { + "kind": "docker", "image": image, "image_id": image_id, + "repo_digests": info.get("RepoDigests", []), + "network": "host", "user": f"{os.getuid()}:{os.getgid()}", + } + return metadata + + def main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--cases", action="append", help="'all', 'heavy', or comma/space separated case paths") @@ -663,6 +783,8 @@ def main() -> int: parser.add_argument("--base-ref", default=os.environ.get("BASE_REF", "")) parser.add_argument("--candidate-ref", default=os.environ.get("CANDIDATE_REF", "")) parser.add_argument("--github-output", default=os.environ.get("GITHUB_OUTPUT")) + parser.add_argument("--base-image", default=os.environ.get("BASE_IMAGE", "")) + parser.add_argument("--candidate-image", default=os.environ.get("CANDIDATE_IMAGE", "")) args = parser.parse_args() try: cases = split_cases(args.cases or [os.environ.get("CASE_PATHS", "all")]) @@ -671,6 +793,12 @@ def main() -> int: append_github_output(args.github_output, 1) return 0 + preparation_error = None + metadata = {} + try: + metadata = prepare_targets(args) + except Exception as err: + preparation_error = err reports: list[Path] = [] status = 0 for case in cases: @@ -678,11 +806,18 @@ def main() -> int: work_dir = args.work_dir / case_slug(case_path) reports.append(work_dir / "query-regression-report.json") try: + if preparation_error is not None: + raise preparation_error case_status = run_case(args, case_path, work_dir) except Exception as err: # noqa: BLE001 - preserve an aggregate report for the summary step print(f"error: query regression case {case_path}: {err}", flush=True) write_failed_report(work_dir, case_path, err) case_status = 1 + report_path = reports[-1] + if (args.base_image or args.candidate_image) and report_path.exists(): + report = json.loads(report_path.read_text(encoding="utf-8")) + report["execution_targets"] = metadata + report_path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") if case_status != 0: status = case_status or 1 @@ -695,4 +830,6 @@ def main() -> int: if __name__ == "__main__": + if len(sys.argv) > 1 and sys.argv[1] == "docker-target": + raise SystemExit(docker_target(sys.argv[2], Path(sys.argv[3]), sys.argv[4:])) raise SystemExit(main()) diff --git a/.github/scripts/run-fuzz-targets-test.sh b/.github/scripts/run-fuzz-targets-test.sh index 9705b6bc5af..ddc559c8db2 100755 --- a/.github/scripts/run-fuzz-targets-test.sh +++ b/.github/scripts/run-fuzz-targets-test.sh @@ -5,6 +5,7 @@ set -euo pipefail SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" RUNNER="${SCRIPT_DIR}/run-fuzz-targets.sh" COLLECTOR="${SCRIPT_DIR}/collect-fuzz-target-artifacts.sh" +PREPARE_CORPUS="${SCRIPT_DIR}/prepare-fuzz-corpus.sh" fail() { printf 'FAIL: %s\n' "$*" >&2 @@ -32,6 +33,13 @@ new_fixture() { set -euo pipefail printf '%s\t%s\t%s\n' "${GT_FUZZ_DUMP_DIR}" "$*" "${MOCK_CARGO_MARKER:-}" >>"${MOCK_CARGO_LOG}" target="$4" +corpus="$5" +[[ -d "${corpus}" ]] || exit 18 +printf '%s\t%s\n' "${target}" "${corpus}" >>"${MOCK_CORPUS_LOG}" +if [[ -f "${corpus}/seed" ]]; then + cat "${corpus}/seed" >>"${MOCK_SEED_LOG}" +fi +printf '%s\n' "${target}" >"${corpus}/learned" case " ${MOCK_FAIL_TARGETS:-} " in *" ${target} "*) exit 17 ;; esac @@ -64,12 +72,15 @@ run_fixture() { "MOCK_COLLECTOR_LOG=${fixture}/collector.log" "MOCK_FAIL_TARGETS=${fail_targets}" "MOCK_CARGO_MARKER=fixture" + "MOCK_CORPUS_LOG=${fixture}/corpus.log" + "MOCK_SEED_LOG=${fixture}/seeds.log" "FUZZ_TARGETS=${targets}" "FUZZ_GROUP=test-group" "FUZZ_MAX_TOTAL_TIME=120" "FUZZ_UNSTABLE=${unstable}" "FUZZ_COLLECT_CLUSTER_ARTIFACTS=true" "FUZZ_ARTIFACT_ROOT=${artifact_root}" + "FUZZ_CORPUS_ROOT=${fixture}/corpus with spaces" "FUZZ_ARTIFACT_COLLECTOR=${fixture}/collector" "GT_FUZZ_BINARY_PATH=${GT_FUZZ_BINARY_PATH:-}" "GT_FUZZ_INSTANCE_ROOT_DIR=${GT_FUZZ_INSTANCE_ROOT_DIR:-}" @@ -81,7 +92,7 @@ run_fixture() { fi set +e - env -u FUZZ_FAIL_FAST "${runner_env[@]}" \ + env -u FUZZ_FAIL_FAST -u FUZZ_BIN_DIR "${runner_env[@]}" \ "${RUNNER}" >"${fixture}/stdout.log" 2>&1 fixture_status=$? set -e @@ -89,18 +100,30 @@ run_fixture() { run_prebuilt_fixture() { new_fixture - mkdir -p "${fixture}/prebuilt" + mkdir -p "${fixture}/prebuilt" "${fixture}/corpus with spaces/fuzz_create_table" + printf 'restored prebuilt seed\n' >"${fixture}/corpus with spaces/fuzz_create_table/seed" cat >"${fixture}/prebuilt/fuzz_create_table" <<'EOF' #!/usr/bin/env bash set -euo pipefail printf '%s\t%s\t%s\n' "${GT_FUZZ_DUMP_DIR}" "$*" "${MOCK_BINARY_MARKER:-}" >>"${MOCK_BINARY_LOG}" +[[ -d "$1" ]] || exit 18 +printf '%s\n' "$1" >>"${MOCK_CORPUS_LOG}" +cat "$1/seed" >>"${MOCK_SEED_LOG}" +printf 'new prebuilt input\n' >"$1/learned" +exit "${MOCK_BINARY_EXIT_CODE}" EOF chmod +x "${fixture}/prebuilt/fuzz_create_table" set +e env \ + "PATH=${fixture}/bin:${PATH}" \ + "MOCK_CARGO_LOG=${fixture}/cargo.log" \ "MOCK_BINARY_LOG=${fixture}/binary.log" \ "MOCK_BINARY_MARKER=fixture" \ + "MOCK_BINARY_EXIT_CODE=${1:-0}" \ + "MOCK_CORPUS_LOG=${fixture}/corpus.log" \ + "MOCK_SEED_LOG=${fixture}/seeds.log" \ + "MOCK_COLLECTOR_LOG=${fixture}/collector.log" \ "FUZZ_TARGETS=fuzz_create_table" \ "FUZZ_GROUP=test-group" \ "FUZZ_MAX_TOTAL_TIME=120" \ @@ -108,6 +131,7 @@ EOF "FUZZ_FAIL_FAST=true" \ "FUZZ_COLLECT_CLUSTER_ARTIFACTS=false" \ "FUZZ_ARTIFACT_ROOT=${fixture}/artifacts" \ + "FUZZ_CORPUS_ROOT=${fixture}/corpus with spaces" \ "FUZZ_ARTIFACT_COLLECTOR=${fixture}/collector" \ "FUZZ_BIN_DIR=${fixture}/prebuilt" \ "GITHUB_SHA=deadbeef" \ @@ -143,6 +167,7 @@ test_successful_targets_run_in_order() { "target order" grep -q -- '--features=unstable' "${fixture}/cargo.log" || fail "unstable feature missing" grep -q -- '-max_total_time=120' "${fixture}/cargo.log" || fail "fuzz time missing" + grep -q -- '-print_final_stats=1' "${fixture}/cargo.log" || fail "fuzz statistics missing" grep -q -- '-artifact_prefix=.*/targets/fuzz_create_table/libfuzzer/' "${fixture}/cargo.log" || \ fail "target-scoped libFuzzer prefix missing" assert_eq success "$(jq -r '.[0].status' "${fixture}/artifacts/manifest.json")" "first status" @@ -167,8 +192,49 @@ test_prebuilt_binary_runs_without_cargo() { [[ ! -e "${fixture}/cargo.log" ]] || fail "cargo ran for prebuilt binary" assert_eq 1 "$(wc -l <"${fixture}/binary.log" | tr -d ' ')" "prebuilt invocation count" grep -q -- '-max_total_time=120' "${fixture}/binary.log" || fail "prebuilt fuzz time missing" + grep -q -- '-print_final_stats=1' "${fixture}/binary.log" || fail "prebuilt fuzz statistics missing" grep -q -- '-artifact_prefix=.*/targets/fuzz_create_table/libfuzzer/' "${fixture}/binary.log" || \ fail "prebuilt target-scoped libFuzzer prefix missing" + assert_eq "${fixture}/corpus with spaces/fuzz_create_table" "$(cat "${fixture}/corpus.log")" \ + "prebuilt corpus argument" + assert_eq 'restored prebuilt seed' "$(cat "${fixture}/seeds.log")" "prebuilt restored input" + assert_file "${fixture}/corpus with spaces/fuzz_create_table/learned" + cleanup_fixture +} + +test_cargo_reuses_separate_target_corpora() { + new_fixture + mkdir -p "${fixture}/corpus with spaces/fuzz_create_table" "${fixture}/corpus with spaces/fuzz_insert" + printf 'create seed\n' >"${fixture}/corpus with spaces/fuzz_create_table/seed" + printf 'insert seed\n' >"${fixture}/corpus with spaces/fuzz_insert/seed" + run_fixture $'fuzz_create_table\nfuzz_insert' false false "" + + assert_eq 0 "${fixture_status}" "restored cargo run status" + assert_eq $'create seed\ninsert seed' "$(cat "${fixture}/seeds.log")" "target-scoped restored inputs" + assert_eq \ + "fuzz_create_table"$'\t'"${fixture}/corpus with spaces/fuzz_create_table"$'\n'"fuzz_insert"$'\t'"${fixture}/corpus with spaces/fuzz_insert" \ + "$(cat "${fixture}/corpus.log")" "cargo corpus arguments" + assert_eq fuzz_create_table "$(cat "${fixture}/corpus with spaces/fuzz_create_table/learned")" \ + "first target learned input" + assert_eq fuzz_insert "$(cat "${fixture}/corpus with spaces/fuzz_insert/learned")" \ + "second target learned input" + + run_fixture fuzz_create_table true false "" + assert_eq 0 "${fixture_status}" "repeated cargo run status" + assert_file "${fixture}/corpus with spaces/fuzz_create_table/seed" + assert_file "${fixture}/corpus with spaces/fuzz_insert/learned" + cleanup_fixture +} + +test_prebuilt_failure_preserves_corpus_and_artifacts() { + run_prebuilt_fixture 17 + + assert_eq 1 "${fixture_status}" "prebuilt failure status" + assert_eq 17 "$(jq -r '.[0].exit_code' "${fixture}/artifacts/manifest.json")" \ + "prebuilt failure exit code" + assert_file "${fixture}/corpus with spaces/fuzz_create_table/seed" + assert_file "${fixture}/corpus with spaces/fuzz_create_table/learned" + assert_file "${fixture}/artifacts/targets/fuzz_create_table/mock-artifacts/state.txt" cleanup_fixture } @@ -183,6 +249,8 @@ test_fail_fast_stops_after_first_failure() { assert_eq skipped_after_failure "$(jq -r '.[2].status' "${fixture}/artifacts/manifest.json")" "skipped status" assert_eq true "$(jq -r '.[2].after_prior_failure' "${fixture}/artifacts/manifest.json")" "skipped provenance" assert_file "${fixture}/artifacts/targets/fuzz_insert/mock-artifacts/state.txt" + assert_file "${fixture}/corpus with spaces/fuzz_insert/learned" + [[ ! -e "${fixture}/corpus with spaces/fuzz_alter_table" ]] || fail "skipped target corpus created" grep -q 'title=Fuzz target failed.*target=fuzz_insert' "${fixture}/stdout.log" || \ fail "target failure annotation missing" grep -q 'title=Fuzz target skipped.*target=fuzz_alter_table' "${fixture}/stdout.log" || \ @@ -191,6 +259,8 @@ test_fail_fast_stops_after_first_failure() { fail "reproduction section missing" grep -q 'cargo +nightly fuzz run fuzz_insert' "${fixture}/artifacts/summary.md" || \ fail "reproduction command missing" + grep -q 'fuzz_insert --fuzz-dir tests-fuzz' "${fixture}/artifacts/summary.md" || \ + fail "reproduction command depends on the CI corpus path" cleanup_fixture } @@ -241,6 +311,56 @@ test_invalid_configuration_fails_before_cargo() { cleanup_fixture } +prepare_corpus_fixture() { + local targets="$1" + local output="$2" + set +e + FUZZ_TARGETS="${targets}" \ + GITHUB_WORKSPACE="${fixture}/workspace with spaces" \ + GITHUB_OUTPUT="${output}" \ + bash "${PREPARE_CORPUS}" >"${fixture}/prepare.log" 2>&1 + prepare_status=$? + set -e +} + +test_corpus_cache_identity_and_paths() { + new_fixture + prepare_corpus_fixture $'fuzz_insert\r\n\nfuzz_create_table\r\nfuzz_insert\r' "${fixture}/first-output" + assert_eq 0 "${prepare_status}" "corpus preparation status" + prepare_corpus_fixture $'fuzz_create_table\nfuzz_insert' "${fixture}/second-output" + assert_eq 0 "${prepare_status}" "reordered corpus preparation status" + cmp -s "${fixture}/first-output" "${fixture}/second-output" || \ + fail "cache identity depends on target order, duplicates, or line endings" + assert_eq \ + "${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_create_table"$'\n'"${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_insert" \ + "$(sed '1,2d; $d' "${fixture}/first-output")" "selected corpus cache paths" + [[ -d "${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_insert" ]] || \ + fail "corpus directory not created" + [[ ! -e "${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_alter_table" ]] || \ + fail "unselected corpus directory created" + + GITHUB_SHA=another-revision prepare_corpus_fixture $'fuzz_create_table\nfuzz_insert' "${fixture}/new-revision-output" + cmp -s "${fixture}/first-output" "${fixture}/new-revision-output" || \ + fail "source revision changed cache identity" + prepare_corpus_fixture fuzz_insert "${fixture}/subset-output" + assert_eq 0 "${prepare_status}" "subset corpus preparation status" + [[ "$(head -n 1 "${fixture}/first-output")" != "$(head -n 1 "${fixture}/subset-output")" ]] || \ + fail "different target sets share cache identity" + cleanup_fixture +} + +test_corpus_preparation_rejects_invalid_targets_before_writing() { + local targets + for targets in $'fuzz_insert\n../escape' 'fuzz target' $'\n\r\n'; do + new_fixture + prepare_corpus_fixture "${targets}" "${fixture}/output" + assert_eq 2 "${prepare_status}" "invalid corpus target status" + [[ ! -e "${fixture}/output" ]] || fail "invalid targets wrote cache outputs" + [[ ! -e "${fixture}/workspace with spaces" ]] || fail "invalid targets created directories" + cleanup_fixture + done +} + test_collector_keeps_target_scopes_separate() { new_fixture service_log="${fixture}/greptime.log" @@ -373,10 +493,14 @@ EOF test_successful_targets_run_in_order test_prebuilt_binary_runs_without_cargo +test_cargo_reuses_separate_target_corpora +test_prebuilt_failure_preserves_corpus_and_artifacts test_fail_fast_stops_after_first_failure test_fail_fast_defaults_to_true test_non_fail_fast_marks_later_targets test_invalid_configuration_fails_before_cargo +test_corpus_cache_identity_and_paths +test_corpus_preparation_rejects_invalid_targets_before_writing test_collector_keeps_target_scopes_separate test_cluster_collector_honors_target_scope_and_namespace test_setup_failure_writes_artifact_contract diff --git a/.github/scripts/run-fuzz-targets.sh b/.github/scripts/run-fuzz-targets.sh index 205ba58c419..1489d20b7f0 100755 --- a/.github/scripts/run-fuzz-targets.sh +++ b/.github/scripts/run-fuzz-targets.sh @@ -5,6 +5,7 @@ set -euo pipefail FUZZ_ARTIFACT_ROOT="${FUZZ_ARTIFACT_ROOT:-/tmp/greptime-fuzz-artifacts}" FUZZ_ARTIFACT_COLLECTOR="${FUZZ_ARTIFACT_COLLECTOR:-.github/scripts/collect-fuzz-target-artifacts.sh}" FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS="${FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS:-180}" +FUZZ_CORPUS_ROOT="${FUZZ_CORPUS_ROOT:-tests-fuzz/corpus}" log() { printf '[run-fuzz-targets] %s\n' "$*" @@ -187,19 +188,22 @@ for ((index = 0; index < ${#targets[@]}; index++)); do started_epoch="$(date +%s)" echo "::group::Fuzz target: ${target}" log "run target ${target}" + mkdir -p "${FUZZ_CORPUS_ROOT}/${target}" + corpus_dir="$(cd -- "${FUZZ_CORPUS_ROOT}/${target}" && pwd)" + log "corpus directory: ${corpus_dir}" if [[ -n "${FUZZ_BIN_DIR:-}" ]]; then fuzz_binary="${FUZZ_BIN_DIR}/${target}" if [[ ! -x "${fuzz_binary}" ]]; then log "prebuilt fuzz binary does not exist or is not executable: ${fuzz_binary}" exit 2 fi - run_args=("${fuzz_binary}" "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-artifact_prefix=${target_dir}/libfuzzer/") + run_args=("${fuzz_binary}" "${corpus_dir}" "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-print_final_stats=1" "-artifact_prefix=${target_dir}/libfuzzer/") else - run_args=(cargo +"${RUSTUP_TOOLCHAIN:-nightly}" fuzz run "${target}" --fuzz-dir tests-fuzz -D -s none) + run_args=(cargo +"${RUSTUP_TOOLCHAIN:-nightly}" fuzz run "${target}" "${corpus_dir}" --fuzz-dir tests-fuzz -D -s none) if [[ "${fuzz_unstable}" == true ]]; then run_args+=(--features=unstable) fi - run_args+=(-- "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-artifact_prefix=${target_dir}/libfuzzer/") + run_args+=(-- "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-print_final_stats=1" "-artifact_prefix=${target_dir}/libfuzzer/") fi if GT_FUZZ_DUMP_DIR="${target_dir}/csv" \ "${run_args[@]}" 2>&1 | tee "${target_dir}/fuzz.log"; then diff --git a/.github/scripts/tests/ci-cache-policy.test.cjs b/.github/scripts/tests/ci-cache-policy.test.cjs new file mode 100644 index 00000000000..ba6817424c6 --- /dev/null +++ b/.github/scripts/tests/ci-cache-policy.test.cjs @@ -0,0 +1,155 @@ +const assert = require('node:assert/strict'); +const { readFileSync } = require('node:fs'); +const { resolve } = require('node:path'); +const { test } = require('node:test'); + +const directory = resolve(__dirname, '../../workflows'); +const workflows = ['checks', 'rust', 'integration', 'nightly-ci'].map(name => ({ + name, + text: readFileSync(resolve(directory, name + '.yml'), 'utf8'), +})); +function jobs(text) { + return [...text.matchAll(/^ ([\w-]+):\n([\s\S]*?)(?=^ [\w-]+:\n|(?![\s\S]))/gm)]; +} +function steps(body) { + return body.split(/^ - /m).slice(1); +} + +test('build caches have one controller and lightweight checks do not cache targets', () => { + let checked = 0; + for (const { name, text } of workflows) { + for (const [, job, body] of jobs(text)) { + if (!body.includes('Swatinem/rust-cache@') && !['fmt', 'check-udeps', 'toml'].includes(job)) continue; + for (const step of steps(body).filter(s => s.includes('uses: actions-rust-lang/setup-rust-toolchain@'))) { + assert.match(step, /^ cache: false$/m, name + '/' + job); + checked++; + } + } + } + assert.ok(checked >= 10, 'expected build and lightweight jobs to be covered'); +}); + +test('only main writes caches, and the nightly read-only cache stays read-only', () => { + let checked = 0; + for (const { name, text } of workflows) { + for (const [, job, body] of jobs(text)) { + for (const step of steps(body)) { + if (!step.includes('uses: Swatinem/rust-cache@') && !step.includes('uses: actions/cache/save@')) continue; + const condition = step.includes('uses: Swatinem/rust-cache@') + ? step.match(/^ save-if: (.+)$/m)?.[1] + : step.match(/^ if: (.+)$/m)?.[1]; + assert.ok(condition, name + '/' + job); + const expression = condition.replace(/^\$\{\{\s*|\s*\}\}$/g, '') + .replaceAll('steps.fuzz-binaries.outputs.cache-hit', 'hit'); + const enabled = new Function('github', 'hit', 'return (' + expression + ');'); + for (const [event, ref] of [ + ['pull_request', 'refs/pull/123/merge'], + ['merge_group', 'refs/heads/gh-readonly-queue/main/pr-123'], + ['workflow_dispatch', 'refs/heads/feature'], + ]) assert.equal(enabled({ event_name: event, ref }, 'false'), false, name + '/' + job); + assert.equal(enabled({ event_name: 'schedule', ref: 'refs/heads/main' }, 'false'), + condition !== 'false', name + '/' + job); + if (step.includes('uses: actions/cache/save@')) { + assert.equal(enabled({ ref: 'refs/heads/main' }, 'true'), false); + } + checked++; + } + } + } + assert.ok(checked >= 10); +}); + +test('helper tool versions do not fragment unrelated Rust caches', () => { + for (const { text } of workflows) { + assert.doesNotMatch(text, /CARGO_(SHEAR|FUZZ)_VERSION/); + } + const integration = workflows.find(w => w.name === 'integration').text; + assert.match(integration, /prefix-key: "v1-fuzz-\$\{\{ env.FUZZ_VERSION \}\}"/); + assert.match(integration, /cargo install cargo-fuzz --version "\$\{FUZZ_VERSION\}"/); +}); + +// Evaluate the actual job guard as well as the cache's save condition. A valid +// save-if alone cannot prove that a scheduled cache producer is reachable. +function warmupJob(workflow, job, { + event = 'schedule', ref = 'refs/heads/main', + profile = event === 'workflow_dispatch' ? 'all' : undefined, + outputs = {}, failed = '', cancelled = false, draft = false, +} = {}) { + const text = workflows.find(w => w.name === workflow).text; + assert.match(text, /^ schedule:$/m); + const body = jobs(text).find(([, name]) => name === job)?.[2]; + assert.ok(body, workflow + '/' + job); + const github = { repository: 'GreptimeTeam/greptimedb', event_name: event, ref, + event: { pull_request: { draft } }, run_attempt: 1 }; + const needs = { scope: { outputs }, ...Object.fromEntries( + ['github-script-tests', 'build', 'prepare-fuzz-targets', 'build-greptime-ci'].map(name => + [name, { result: name === failed ? 'failure' : 'success' }])) }; + const evaluate = expression => new Function('github', 'inputs', 'needs', 'cancelled', + 'return (' + expression.replace(/^\$\{\{\s*|\s*\}\}$/g, '') + ');')( + github, { fuzz_profile: profile, ci_command: false }, needs, () => cancelled); + const admitted = evaluate(body.match(/^ if: (.+)$/m)[1]); + const saves = [...body.matchAll(/^ save-if: (.+)$/gm)] + .map(match => admitted && evaluate(match[1])); + return { admitted, saves }; +} + +const fuzzJobs = ['prepare-fuzz-targets', 'build-greptime-ci', 'fuzztest', + 'unstable-fuzztest', 'distributed-fuzztest', 'distributed-fuzztest-with-chaos']; + +test('main schedules can execute cache producers and their fuzz consumers', () => { + for (const job of fuzzJobs) { + // Scheduled events do not supply workflow_dispatch inputs. + const result = warmupJob('integration', job, { profile: undefined }); + assert.equal(result.admitted, true, job); + assert.ok(result.saves.every(Boolean), job); + } + for (const [workflow, job] of [ + ['integration', 'build'], ['integration', 'export-import-v2-e2e'], + ['integration', 'prepare-fuzz-targets'], ['integration', 'build-greptime-ci'], + ['integration', 'distributed-fuzztest'], + ['rust', 'check-windows'], ['rust', 'check-riscv64'], + ['nightly-ci', 'sqlness-windows'], ['nightly-ci', 'test-on-windows'], + ]) { + for (const event of ['schedule', 'workflow_dispatch']) { + assert.deepEqual(warmupJob(workflow, job, { event }), + { admitted: true, saves: [true] }, workflow + '/' + job + '/' + event); + } + } +}); + +test('manual warming respects selected profiles without an internal command marker', () => { + const expected = { + none: [], + standalone: fuzzJobs.slice(0, 4), + distributed: [fuzzJobs[0], fuzzJobs[1], fuzzJobs[4]], + chaos: [fuzzJobs[0], fuzzJobs[1], fuzzJobs[5]], + all: fuzzJobs, + }; + for (const [profile, wanted] of Object.entries(expected)) { + assert.deepEqual(fuzzJobs.filter(job => warmupJob('integration', job, + { event: 'workflow_dispatch', profile }).admitted), wanted, profile); + } + for (const job of fuzzJobs) { + const result = warmupJob('integration', job, + { event: 'workflow_dispatch', ref: 'refs/heads/feature' }); + assert.ok(result.saves.every(value => value === false), job); + assert.equal(warmupJob('integration', job, + { event: 'pull_request', ref: 'refs/pull/1/merge', draft: true }).admitted, false, job); + } +}); + +test('warmup keeps scope exclusions, dependency failures and cancellation gates', () => { + const gates = ['any_fuzz', 'build_ci_needed', 'fuzz_standalone', + 'fuzz_unstable', 'fuzz_distributed', 'fuzz_chaos']; + for (const [index, job] of fuzzJobs.entries()) { + assert.equal(warmupJob('integration', job, { failed: 'github-script-tests' }).admitted, false, job); + assert.equal(warmupJob('integration', job, { cancelled: true }).admitted, false, job); + assert.equal(warmupJob('integration', job, + { outputs: { [gates[index]]: 'false' } }).admitted, false, job); + } + for (const job of fuzzJobs.slice(2)) { + for (const failed of ['github-script-tests', 'prepare-fuzz-targets', job === 'fuzztest' ? 'build' : 'build-greptime-ci']) { + assert.equal(warmupJob('integration', job, { failed }).admitted, false, job); + } + } +}); diff --git a/.github/scripts/tests/nightly-merge.test.cjs b/.github/scripts/tests/nightly-merge.test.cjs new file mode 100644 index 00000000000..d6c8792f7fe --- /dev/null +++ b/.github/scripts/tests/nightly-merge.test.cjs @@ -0,0 +1,153 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Run with: node --test .github/scripts/tests/nightly-merge.test.cjs +const { readFileSync } = require('node:fs'); +const { resolve } = require('node:path'); +const assert = require('node:assert/strict'); +const { test } = require('node:test'); +const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor; +const workflow = readFileSync(resolve(__dirname, '../../workflows/nightly-merge.yml'), 'utf8'); +const scripts = [...workflow.matchAll(/ script: \|\n((?: .*\n|\n)+)/g)] + .map(match => new AsyncFunction('github', 'context', 'core', 'process', match[1].replace(/^ /gm, ''))); +assert.equal(scripts.length, 2); +const checked = '- [x] Merge during the nightly window '; + +async function run({ sync = false, body = checked, labels = ['merge nightly'], + detail = {}, current = {}, threads = [[true]], count = 1, dryRun = false, + failFirst = false, enqueueError = Error('head changed') } = {}) { + const calls = []; + let page = 0; + const github = { + rest: { + pulls: { get: async () => ({ data: { state: 'open', draft: false, base: { ref: 'main' }, + body, labels: labels.map(name => ({ name })), ...current } }) }, + issues: { + listForRepo: () => {}, + addLabels: async args => calls.push(['add', args.labels]), + removeLabel: async args => calls.push(['remove', args.name]) + } + }, + paginate: async () => Array.from({ length: count }, (_, i) => ({ number: i + 1, pull_request: {} })), + graphql: async (query, args) => { + if (query.includes('mutation')) { + calls.push(['enqueue', args]); + if (failFirst && calls.filter(call => call[0] === 'enqueue').length === 1) throw enqueueError; + return {}; + } + if (query.includes('reviewThreads')) { + const index = page++ % threads.length; + calls.push(['threads', args.cursor]); + return { repository: { pullRequest: { reviewThreads: { + nodes: threads[index].map(isResolved => ({ isResolved })), + pageInfo: { hasNextPage: index + 1 < threads.length, endCursor: `page-${index}` } + } } } }; + } + return { repository: { pullRequest: { id: `PR_${args.number}`, headRefOid: 'abc', + reviewDecision: 'APPROVED', mergeStateStatus: 'CLEAN', ...detail } } }; + } + }; + const summary = { addHeading() { return this; }, addTable(rows) { calls.push(['summary', rows]); return this; }, async write() {} }; + await scripts[sync ? 0 : 1](github, + { repo: { owner: 'GreptimeTeam', repo: 'greptimedb' }, payload: { pull_request: { number: 1 } } }, + { summary, info() {}, warning() {}, setFailed: message => calls.push(['failed', message]) }, + { env: { DRY_RUN: String(dryRun) } }); + return calls; +} + +test('label sync grants PR label write permission without checking out PR code', () => { + const syncJob = workflow.split(' sync-label:\n')[1].split(' enqueue:\n')[0]; + assert.match(syncJob, /^ pull-requests: write$/m); + assert.doesNotMatch(syncJob, /uses: actions\/checkout/); +}); + +test('template opts in and checkbox edits reconcile only the nightly label', async () => { + assert.ok(readFileSync(resolve(__dirname, '../../pull_request_template.md'), 'utf8').includes(checked)); + assert.deepEqual(await run({ sync: true, labels: ['bug'] }), [['add', ['merge nightly']]]); + assert.deepEqual(await run({ sync: true }), []); + for (const body of [checked.replace('[x]', '[ ]')]) { + assert.deepEqual(await run({ sync: true, body }), [['remove', 'merge nightly']]); + } + assert.deepEqual(await run({ sync: true, body: checked.replace('[x]', '[X]'), labels: [] }), [['add', ['merge nightly']]]); +}); + +test('eligible PR is enqueued with its checked head; dry run performs no mutation', async () => { + assert.deepEqual((await run()).filter(c => c[0] === 'enqueue'), [['enqueue', { id: 'PR_1', sha: 'abc' }]]); + assert.equal((await run({ dryRun: true })).some(c => c[0] === 'enqueue'), false); +}); + +test('each eligibility gate independently prevents enqueueing', async () => { + for (const options of [ + { labels: [] }, + { current: { draft: true } }, { current: { state: 'closed' } }, { current: { base: { ref: 'release/v1.3' } } }, + { detail: { reviewDecision: 'REVIEW_REQUIRED' } }, { detail: { reviewDecision: 'CHANGES_REQUESTED' } }, + { detail: { mergeStateStatus: 'BLOCKED' } }, { detail: { mergeStateStatus: 'UNKNOWN' } }, + { detail: { mergeQueueEntry: { id: 'entry' } } }, { detail: { autoMergeRequest: { enabledAt: 'now' } } }, + { threads: [[false]] }, { threads: [[true], [false]] } + ]) { + assert.equal((await run(options)).some(c => c[0] === 'enqueue'), false, JSON.stringify(options)); + } +}); + +test('all thread pages are checked before enqueueing', async () => { + const calls = await run({ threads: [[true], [true]] }); + assert.deepEqual(calls.filter(c => c[0] === 'threads'), [['threads', null], ['threads', 'page-0']]); + assert.equal(calls.filter(c => c[0] === 'enqueue').length, 1); +}); + +test('per-run cap and individual failures do not prevent later candidates', async () => { + assert.equal((await run({ count: 8 })).filter(c => c[0] === 'enqueue').length, 5); + const calls = await run({ count: 2, failFirst: true }); + assert.equal(calls.filter(c => c[0] === 'enqueue').length, 2); + assert.equal(calls.filter(c => c[0] === 'failed').length, 1); +}); + +test('label-only opt-in survives PRs without a template and does not require a checkbox', async () => { + for (const body of ['', null, 'Existing PR description', `> ${checked}`]) { + assert.deepEqual(await run({ sync: true, body }), []); + assert.deepEqual(await run({ sync: true, body, labels: [] }), []); + assert.equal((await run({ body })).filter(call => call[0] === 'enqueue').length, 1); + assert.equal((await run({ body, labels: [] })).some(call => call[0] === 'enqueue'), false); + } + // The label is authoritative for admission, even before checkbox synchronization. + assert.equal((await run({ body: checked.replace('[x]', '[ ]') })).some(call => call[0] === 'enqueue'), true); +}); + +const stackMessage = 'This pull request is part of a stack and must be enqueued using the asynchronous merge REST API. For more information, see https://docs.github.com/rest/pulls/pulls#merge-a-pull-request-asynchronously'; + +test('stacked PR rejection is skipped without consuming the successful enqueue cap', async () => { + const calls = await run({ count: 7, failFirst: true, + enqueueError: Object.assign(Error('GraphQL request failed'), { errors: [{ message: stackMessage }] }) }); + assert.equal(calls.some(c => c[0] === 'failed'), false); + assert.deepEqual(calls.find(c => c[0] === 'summary')[1], [ + ['PR', 'Result'], ['1', 'Skipped: stacked PR'], + ...[2, 3, 4, 5, 6].map(number => [String(number), 'Enqueued']) + ]); + assert.deepEqual(calls.filter(c => c[0] === 'enqueue').map(c => c[1].id), + ['PR_1', 'PR_2', 'PR_3', 'PR_4', 'PR_5', 'PR_6']); +}); + +test('mixed or unstructured errors are not mistaken for a stacked PR rejection', async () => { + for (const enqueueError of [ + Error(stackMessage), + Object.assign(Error('GraphQL request failed'), { errors: [] }), + Object.assign(Error('GraphQL request failed'), { errors: [{ message: stackMessage }, { message: 'Forbidden' }] }) + ]) { + const calls = await run({ count: 2, failFirst: true, enqueueError }); + assert.equal(calls.filter(c => c[0] === 'failed').length, 1); + assert.deepEqual(calls.find(c => c[0] === 'summary')[1], [ + ['PR', 'Result'], ['1', 'Failed: see workflow logs'], ['2', 'Enqueued'] + ]); + } +}); diff --git a/.github/scripts/tests/test_metric_snapshot_s3_bench.py b/.github/scripts/tests/test_metric_snapshot_s3_bench.py index c662ecbff76..092771e1154 100644 --- a/.github/scripts/tests/test_metric_snapshot_s3_bench.py +++ b/.github/scripts/tests/test_metric_snapshot_s3_bench.py @@ -28,6 +28,25 @@ SPEC.loader.exec_module(bench) class PhaseTests(unittest.TestCase): + def test_cloud_region_configuration(self): + for bucket, runner, accepted in (("us-east-1", "us-east-1", True), + ("ap-southeast-1", "ap-southeast-1", True), + ("us-east-1", "ap-southeast-1", False), + ("", "", False)): + with self.subTest(bucket=bucket, runner=runner), tempfile.TemporaryDirectory() as directory: + env = dict(GREPTIMEDB_IMAGE="greptime/greptimedb@sha256:" + "a" * 64, + EXPECTED_COMMIT="b" * 40, RUN_PREFIX="metric-snapshot-bench/123-1/", + RESULTS=str(Path(directory) / "results"), DATA_ROOT=str(Path(directory) / "data"), + S3_REGION=bucket, RUNNER_REGION=runner, S3_ENDPOINT="") + with patch.dict(bench.os.environ, env): + if accepted: + benchmark = bench.Benchmark(local_smoke=False) + benchmark.summary() + self.assertIn(f"S3 in {bucket};", (benchmark.results / "summary.md").read_text()) + else: + with self.assertRaisesRegex(ValueError, "same nonempty"): + bench.Benchmark(local_smoke=False) + def test_docker_nanosecond_timestamps(self): start = bench.docker_timestamp("2026-10-14T06:06:44.72000818Z") end = bench.docker_timestamp("2026-10-14T06:06:46.123456789Z") diff --git a/.github/scripts/tests/test_metric_snapshot_s3_ops.py b/.github/scripts/tests/test_metric_snapshot_s3_ops.py index d22c09044b2..5c1a265af0f 100644 --- a/.github/scripts/tests/test_metric_snapshot_s3_ops.py +++ b/.github/scripts/tests/test_metric_snapshot_s3_ops.py @@ -20,6 +20,7 @@ import io import json from pathlib import Path import tempfile +import types import unittest from unittest import mock @@ -82,6 +83,45 @@ class OperationsTest(unittest.TestCase): with self.subTest(changes=changes), self.assertRaises(ValueError): ops.plan(self.env(**changes)) + def test_regions_follow_configuration_and_reject_mismatch_or_empty(self): + for region in ("us-east-1", "ap-southeast-1"): + with self.subTest(region=region): + self.assertEqual(ops.plan(self.env(S3_REGION=region, RUNNER_REGION=region))["region"], region) + for bucket, runner in (("us-east-1", "ap-southeast-1"), ("", ""), (" ", " ")): + with self.subTest(bucket=bucket, runner=runner), self.assertRaises(ValueError): + ops.plan(self.env(S3_REGION=bucket, RUNNER_REGION=runner)) + + def test_client_uses_configured_region(self): + boto = types.ModuleType("boto3") + boto.client = mock.Mock() + config = types.ModuleType("botocore.config") + config.Config = mock.Mock() + with mock.patch.dict("sys.modules", {"boto3": boto, "botocore.config": config}): + for region in ("us-east-1", "ap-southeast-1"): + ops.client(region) + self.assertEqual(boto.client.call_args.kwargs["region_name"], region) + with self.assertRaises(ValueError): + ops.client("") + + def test_hourly_estimate_controls_reservation_and_cost(self): + env = self.env(EC2_HOURLY_USD="2", BENCHMARK_TIMEOUT_MINUTES="30") + plan = ops.plan(env) + self.assertEqual(plan["ec2_hourly_usd"], "2") + self.assertEqual(plan["attempt_reserve_usd"], "5.00") + with mock.patch.dict(ops.os.environ, env), mock.patch.object(ops.time, "time", return_value=4600), \ + mock.patch("sys.argv", ["ops", "cost", "--started-at", "1000"]), \ + mock.patch("sys.stdout", new_callable=io.StringIO) as output: + ops.main() + cost = json.loads(output.getvalue()) + self.assertEqual(cost["ec2_hourly_usd"], "2") + self.assertEqual(cost["ec2_estimate_usd"], "2.0000") + self.assertEqual(cost["campaign_estimate_with_other_reserve_usd"], "3.5000") + for rate in ("", "invalid", "0", "-1", "NaN", "Infinity", "-Infinity"): + with self.subTest(rate=rate), self.assertRaises(ValueError): + ops.plan(self.env(EC2_HOURLY_USD=rate)) + with self.assertRaises(ValueError): + ops.plan(self.env(EC2_HOURLY_USD="4")) + def watcher_env(self): return self.env(GITHUB_REPOSITORY="GreptimeTeam/greptimedb", GITHUB_RUN_ID="123", GITHUB_RUN_ATTEMPT="2", GH_TOKEN="test-token", BENCHMARK_TIMEOUT_MINUTES="30") diff --git a/.github/scripts/update-query-regression-helpers-image.sh b/.github/scripts/update-query-regression-helpers-image.sh new file mode 100644 index 00000000000..9ef9122da74 --- /dev/null +++ b/.github/scripts/update-query-regression-helpers-image.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +set -euo pipefail +: "${HELPERS_IMAGE:?Verified image digest is required}" +: "${GITHUB_REPOSITORY:?Repository is required}" +repository=greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/query-regression-helpers +[[ "$HELPERS_IMAGE" == "$repository@sha256:"* ]] +[[ "${HELPERS_IMAGE#"$repository@"}" =~ ^sha256:[0-9a-f]{64}$ ]] +# Like QUERY_REGRESSION_ECS_IMAGE_ID, this is CI configuration, not a source +# change. Create on first publication, then update only after image verification. +gh variable set QUERY_REGRESSION_HELPERS_IMAGE \ + --repo "$GITHUB_REPOSITORY" --body "$HELPERS_IMAGE" +gh api "repos/$GITHUB_REPOSITORY/actions/variables/QUERY_REGRESSION_HELPERS_IMAGE" \ + | jq -e --arg expected "$HELPERS_IMAGE" '.value == $expected' >/dev/null +printf 'QUERY_REGRESSION_HELPERS_IMAGE updated to `%s`\n' "$HELPERS_IMAGE" diff --git a/.github/workflows/agent-observability.yml b/.github/workflows/agent-observability.yml index d265801d4a0..055af1d0f26 100644 --- a/.github/workflows/agent-observability.yml +++ b/.github/workflows/agent-observability.yml @@ -2,6 +2,11 @@ name: Agent Observability Benchmark on: workflow_dispatch: inputs: + greptimedb_storage: + description: 'GreptimeDB data storage (S3 requires AWS in the bucket region; WAL stays local)' + type: choice + options: [file, s3, both] + default: file run_greptimedb: description: Run GreptimeDB type: boolean @@ -56,7 +61,7 @@ on: type: string required: false runtime_image: - description: 'Runtime image: auto = AWS ECR Public or Aliyun ACR by provider; explicit tag/digest overrides' + description: 'Runtime image: auto = pinned S3 test runtime on ACR; explicit tag/digest overrides' type: string default: auto required: true @@ -99,6 +104,11 @@ on: permissions: contents: read env: + GREPTIMEDB_STORAGE: ${{ inputs.greptimedb_storage || 'file' }} + S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} + S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} + RUNNER_REGION: ${{ vars.EC2_RUNNER_REGION }} + RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }} COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }} BASELINE_GREPTIMEDB_TAG: ${{ inputs.baseline_greptimedb_tag }} PROFILE: ${{ inputs.dataset }} @@ -106,7 +116,7 @@ env: CLICKHOUSE_TAG: ${{ inputs.clickhouse_tag }} VICTORIALOGS_TAG: ${{ inputs.victorialogs_tag }} O11YBENCH_REF: ${{ inputs.o11ybench_ref }} - RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || inputs.provider == 'AWS' && 'public.ecr.aws/i8k6a5e1/greptime/o11ybenchruntime:2f40fb3fdbdc@sha256:8940782216bf5d066308fd71f6e346d950073c41cfb5756c5f3dc8aacf069cbc' || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:2f40fb3fdbdc@sha256:8940782216bf5d066308fd71f6e346d950073c41cfb5756c5f3dc8aacf069cbc' }} + RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:s3-test-b0cf5204d2fc@sha256:c2bf622db6056a2029842f4dc1cc1f37b82773bd45b134a0a83ffe2a8cd62214' }} PROVIDER: ${{ inputs.provider }} REQUESTED_INSTANCE_TYPE: ${{ inputs.instance_type }} SYSTEM_DISK_GIB: ${{ inputs.system_disk_gib }} @@ -130,6 +140,10 @@ jobs: with: path: greptimedb persist-credentials: false + - name: Validate GreptimeDB storage + run: | + python3 greptimedb/tests/perf/test_o11ybench_storage.py + python3 greptimedb/.github/scripts/o11ybench-storage.py validate - name: Validate version comparison env: RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }} @@ -144,6 +158,18 @@ jobs: python3 greptimedb/tests/perf/test_aliyun_ecs_runner_scripts.py python3 greptimedb/tests/perf/test_ci_runner_scripts.py python3 greptimedb/tests/perf/test_agent_observability_summary.py + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Check runtime supports shared version-comparison data if: inputs.compare_greptimedb run: | @@ -275,7 +301,10 @@ jobs: path: greptimedb persist-credentials: false - name: Set up benchmark dependencies - uses: ./greptimedb/.github/actions/setup-benchmark + uses: ./greptimedb/.github/actions/setup-linux-runner + with: + tools: curl jq python3 rsync + expected-architecture: x86_64 - name: Check Docker on ephemeral Linux host shell: bash run: | @@ -285,6 +314,18 @@ jobs: docker info # The workspace (and Docker state) must not use a small /tmp filesystem. df -h "$GITHUB_WORKSPACE" /var/lib/docker + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Resolve image tags and verify runtime revision shell: bash run: | @@ -343,24 +384,48 @@ jobs: --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" \ --root "$ARTIFACT_ROOT" --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" cp "$MANIFEST_PATH" "$ARTIFACT_ROOT/run-manifest.json" - - name: Load and benchmark baseline GreptimeDB - id: baseline_greptimedb - if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.compare_greptimedb }} + - name: Load and benchmark baseline GreptimeDB (File) + id: baseline_greptimedb_file + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' && inputs.compare_greptimedb }} run: |- - GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --execute \ + GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage file --execute \ --owner "$OWNER-base" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT/baseline" --data-root "$ARTIFACT_ROOT" \ --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \ --target greptimedb --image "$BASELINE_GREPTIMEDB_IMAGE" \ --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT" - - name: Load and benchmark greptimedb - id: greptimedb - if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' }} + - name: Load and benchmark baseline GreptimeDB (S3) + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} + id: baseline_greptimedb_s3 + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' && inputs.compare_greptimedb }} run: |- - GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --execute \ + GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage s3 --execute \ + --owner "$OWNER-base-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT/baseline/s3" --data-root "$ARTIFACT_ROOT" \ + --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \ + --target greptimedb --image "$BASELINE_GREPTIMEDB_IMAGE" \ + --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT" + - name: Load and benchmark greptimedb (File) + id: greptimedb_file + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' }} + run: |- + GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage file --execute \ --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT" \ --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \ --target greptimedb --image "$GREPTIMEDB_IMAGE" \ --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT" + - name: Load and benchmark greptimedb (S3) + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} + id: greptimedb_s3 + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' }} + run: |- + GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage s3 --execute \ + --owner "$OWNER-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT/s3" --data-root "$ARTIFACT_ROOT" \ + --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \ + --target greptimedb --image "$GREPTIMEDB_IMAGE" \ + --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT" - name: Load and benchmark clickhouse id: clickhouse if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.clickhouse == 'true' }} @@ -380,7 +445,7 @@ jobs: --target victorialogs --image "$VICTORIALOGS_IMAGE" \ --victorialogs-http-port "$VICTORIALOGS_HTTP_PORT" - name: Compare all three targets - if: ${{ !cancelled() && steps.greptimedb.outcome == 'success' && steps.clickhouse.outcome == 'success' && steps.victorialogs.outcome + if: ${{ !cancelled() && steps.greptimedb_file.outcome == 'success' && steps.clickhouse.outcome == 'success' && steps.victorialogs.outcome == 'success' }} run: |- docker run --rm --user "$(id -u):$(id -g)" --label "o11ybench.owner=$OWNER" \ @@ -391,45 +456,61 @@ jobs: --victorialogs-summary /results/victorialogs-1/victorialogs-1/measured-summary.json \ --output /results/comparison.json - name: Clean up owned containers + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} if: always() run: |- if [[ -f o11ybench/scripts/run-agent-observability.sh ]]; then - GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute \ + GITHUB_STEP_SUMMARY= python3 greptimedb/.github/scripts/o11ybench-storage.py run -- bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute --runtime-image "$RESOLVED_RUNTIME_IMAGE" \ --owner "$OWNER" --root "$ARTIFACT_ROOT" fi - name: Clean up baseline containers + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} if: always() && inputs.compare_greptimedb run: |- if [[ -f o11ybench/scripts/run-agent-observability.sh ]]; then - GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute \ + GITHUB_STEP_SUMMARY= python3 greptimedb/.github/scripts/o11ybench-storage.py run -- bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute --runtime-image "$RESOLVED_RUNTIME_IMAGE" \ --owner "$OWNER-base" --root "$ARTIFACT_ROOT/baseline" fi - name: Benchmark summary if: always() run: | - python3 greptimedb/.github/scripts/agent-observability-summary.py \ - --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section benchmark >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind agent \ + --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section benchmark | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" - name: Load / lifecycle summary if: always() run: | - python3 greptimedb/.github/scripts/agent-observability-summary.py \ - --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section lifecycle >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind agent \ + --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section lifecycle | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" - name: Baseline load / lifecycle summary if: always() && inputs.compare_greptimedb run: | - echo '## Baseline GreptimeDB' >> "$GITHUB_STEP_SUMMARY" - python3 greptimedb/.github/scripts/agent-observability-summary.py \ - --root "$ARTIFACT_ROOT/baseline" --targets greptimedb --section lifecycle >> "$GITHUB_STEP_SUMMARY" + echo '## Baseline GreptimeDB' | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind agent \ + --root "$ARTIFACT_ROOT/baseline" --targets greptimedb --section lifecycle | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" - name: GreptimeDB version comparison if: always() && inputs.compare_greptimedb run: | - python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind agent --root "$ARTIFACT_ROOT" >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind agent --root "$ARTIFACT_ROOT" | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + - name: Render storage comparison overview + if: always() && steps.generate.outcome == 'success' + run: | + python3 o11ybench/scripts/build_query_performance_report.py --agent "$ARTIFACT_ROOT" \ + --output "$ARTIFACT_ROOT-query-report.html" --json-output "$ARTIFACT_ROOT-query-report.json" + python3 greptimedb/.github/scripts/o11ybench-storage.py overview "$ARTIFACT_ROOT-query-report.json" \ + | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/o11ybench-storage.py audit --root "$ARTIFACT_ROOT" \ + --report "$ARTIFACT_ROOT-query-report.json" --targets "$TARGETS" - name: Upload results (no dataset) if: always() uses: actions/upload-artifact@v4 with: name: agent-observability-${{ github.run_id }}-${{ github.run_attempt }} path: |- + ${{ env.ARTIFACT_ROOT }}-query-report.* ${{ env.MANIFEST_PATH }} ${{ env.ARTIFACT_ROOT }} !${{ env.ARTIFACT_ROOT }}/corpus/agent_observations.jsonl diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 1f14d8851cc..0d38a844539 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -107,6 +107,8 @@ jobs: with: tool: protoc@3.23.4 - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Rust Cache uses: Swatinem/rust-cache@v2 with: @@ -130,6 +132,8 @@ jobs: with: persist-credentials: false - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Install taplo uses: taiki-e/install-action@v2 with: diff --git a/.github/workflows/dev-build.yml b/.github/workflows/dev-build.yml index 563435b708e..0295c5d2779 100644 --- a/.github/workflows/dev-build.yml +++ b/.github/workflows/dev-build.yml @@ -4,6 +4,11 @@ name: GreptimeDB Development Build on: workflow_dispatch: # Allows you to run this workflow manually. inputs: + ec2_volume_size: + description: Root volume size in GiB for each EC2 build runner + type: number + required: false + default: 500 large-page-size: description: Build GreptimeDB with large page size (65536). type: boolean @@ -133,6 +138,7 @@ jobs: aws-region: ${{ vars.EC2_RUNNER_REGION }} github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} image-id: ${{ vars.EC2_RUNNER_LINUX_AMD64_IMAGE_ID }} + ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }} security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }} subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }} @@ -147,6 +153,7 @@ jobs: aws-region: ${{ vars.EC2_RUNNER_REGION }} github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} image-id: ${{ vars.EC2_RUNNER_LINUX_ARM64_IMAGE_ID }} + ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }} security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }} subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }} @@ -185,8 +192,8 @@ jobs: large-page-size: ${{ inputs.large-page-size }} aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }} aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} build-linux-arm64-artifacts: name: Build linux-arm64 artifacts @@ -223,8 +230,8 @@ jobs: large-page-size: ${{ inputs.large-page-size }} aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }} aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} release-images-to-dockerhub: name: Build and push images to DockerHub diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index c9a6e8caaf3..fa8bc7af2d9 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -74,5 +74,10 @@ jobs: - coverage - Sqlness Test (Flat format) - Compatibility Test (recent releases) + - GitHub Script Tests (Rust CI) + - GitHub Script Tests (Integration CI) + - Check for conflict + - Check Unused Dependencies + - Build GreptimeDB binaries (ubuntu-latest) steps: - run: 'echo "No action required"' diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 00c3f52a034..134fdfde6ba 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -30,7 +30,7 @@ on: name: Integration CI env: - CARGO_FUZZ_VERSION: "0.13.2" + FUZZ_VERSION: "0.13.2" # Fuzz targets require a nightly toolchain (cargo-fuzz needs # `-Zsanitizer=fuzzer`). This pin decouples the fuzz workflow from the # workspace default in rust-toolchain.toml, so it keeps working when the @@ -121,6 +121,8 @@ jobs: with: tool: protoc@3.23.4 - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Build Binaries Cache id: build-binaries-cache uses: Swatinem/rust-cache@v2 @@ -292,6 +294,8 @@ jobs: with: tool: protoc@3.23.4 - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Rust Cache uses: Swatinem/rust-cache@v2 with: @@ -436,18 +440,18 @@ jobs: id: fuzz-dependencies-cache uses: Swatinem/rust-cache@v2 with: - prefix-key: "v1-fuzz-${{ env.CARGO_FUZZ_VERSION }}" + prefix-key: "v1-fuzz-${{ env.FUZZ_VERSION }}" shared-key: "fuzz-dependencies" workspaces: ". -> target" cache-targets: "true" cache-all-crates: "true" - save-if: ${{ github.event_name == 'pull_request' }} + save-if: ${{ github.ref == 'refs/heads/main' }} - name: Look up prebuilt fuzz binaries id: fuzz-binaries uses: actions/cache/restore@v4 with: path: ${{ runner.temp }}/greptime-fuzz-binaries - key: v1-fuzz-binaries-${{ env.CARGO_FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }} + key: v1-fuzz-binaries-${{ env.FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }} - name: Report fuzz cache lookup shell: bash run: | @@ -462,7 +466,7 @@ jobs: env: RUSTUP_TOOLCHAIN: ${{ env.FUZZ_RUST_TOOLCHAIN }} run: | - cargo install cargo-fuzz --version "${CARGO_FUZZ_VERSION}" --locked + cargo install cargo-fuzz --version "${FUZZ_VERSION}" --locked - name: Build fuzz target binaries if: steps.fuzz-binaries.outputs.cache-hit != 'true' shell: bash @@ -511,11 +515,11 @@ jobs: build_targets false "${FUZZ_TARGETS}" build_targets true "${FUZZ_UNSTABLE_TARGETS}" - name: Save prebuilt fuzz binaries - if: steps.fuzz-binaries.outputs.cache-hit != 'true' + if: github.ref == 'refs/heads/main' && steps.fuzz-binaries.outputs.cache-hit != 'true' uses: actions/cache/save@v4 with: path: ${{ runner.temp }}/greptime-fuzz-binaries - key: v1-fuzz-binaries-${{ env.CARGO_FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }} + key: v1-fuzz-binaries-${{ env.FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }} - name: Pack prebuilt fuzz binaries shell: bash run: | @@ -688,6 +692,8 @@ jobs: with: tool: protoc@3.23.4 - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Build Profile-CI Cache id: build-greptime-ci-cache uses: Swatinem/rust-cache@v2 @@ -696,7 +702,7 @@ jobs: shared-key: "build-greptime-ci" cache-all-crates: "true" cache-workspace-crates: "true" - save-if: ${{ github.event_name == 'pull_request' }} + save-if: ${{ github.ref == 'refs/heads/main' }} - name: Report profile-CI cache lookup shell: bash run: | @@ -819,7 +825,7 @@ jobs: workspaces: ". -> target" cache-targets: "true" cache-all-crates: "true" - save-if: ${{ github.event_name == 'pull_request' }} + save-if: ${{ github.ref == 'refs/heads/main' }} - if: matrix.mode.kafkaWalHelper name: Report Kafka WAL helper cache lookup shell: bash diff --git a/.github/workflows/metric-snapshot-s3-bench.yml b/.github/workflows/metric-snapshot-s3-bench.yml index e4a12dde69f..8b961b1d4fa 100644 --- a/.github/workflows/metric-snapshot-s3-bench.yml +++ b/.github/workflows/metric-snapshot-s3-bench.yml @@ -18,6 +18,11 @@ on: description: 'Setup, 500-table smoke and full run timeout, 30..240 minutes' type: number default: 240 + ec2_hourly_usd: + description: 'Positive EC2 hourly USD budget estimate; historical default, not live regional pricing' + required: true + type: string + default: '0.9408' prior_spend_usd: description: 'Cumulative estimated cloud spend for all earlier attempts in this $20 campaign' required: true @@ -34,6 +39,7 @@ env: LOGICAL_TABLES_PER_GROUP: ${{ inputs.logical_tables_per_group }} BENCHMARK_TIMEOUT_MINUTES: ${{ inputs.benchmark_timeout_minutes }} PRIOR_SPEND_USD: ${{ inputs.prior_spend_usd }} + EC2_HOURLY_USD: ${{ inputs.ec2_hourly_usd }} RUN_PREFIX: metric-snapshot-bench/${{ github.run_id }}-${{ github.run_attempt }}/ S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} @@ -66,8 +72,8 @@ jobs: - uses: astral-sh/setup-uv@v6 - name: Check S3 access before allocating EC2 env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} run: uv run greptimedb/.github/scripts/metric-snapshot-s3-ops.py probe - name: Resolve the fixed benchmark runner id: runner @@ -125,7 +131,10 @@ jobs: path: greptimedb persist-credentials: false - name: Set up benchmark dependencies - uses: ./greptimedb/.github/actions/setup-benchmark + uses: ./greptimedb/.github/actions/setup-linux-runner + with: + tools: curl jq python3 rsync + expected-architecture: x86_64 - uses: astral-sh/setup-uv@v6 - name: Mount the temporary local File data disk run: | @@ -148,20 +157,20 @@ jobs: findmnt -no SOURCE,FSTYPE -T "$DISK_ROOT" - name: Probe the real same-region S3 bucket env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} run: uv run greptimedb/.github/scripts/metric-snapshot-s3-ops.py probe - name: Run the 500-table smoke, then one full packed export and restore env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} run: python3 greptimedb/.github/scripts/metric-snapshot-s3-bench.py run - name: Check results for credentials id: scrub if: always() env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} run: python3 greptimedb/.github/scripts/metric-snapshot-s3-ops.py scrub - name: Publish the readable result if: ${{ always() && steps.scrub.outcome == 'success' }} @@ -220,8 +229,8 @@ jobs: - name: Remove and verify only this run's S3 prefix if: always() env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} run: uv run greptimedb/.github/scripts/metric-snapshot-s3-ops.py cleanup - name: Estimate the cumulative cloud cost if: ${{ always() && needs.provision.outputs.started_at != '' }} diff --git a/.github/workflows/nightly-build.yml b/.github/workflows/nightly-build.yml index c78ca4dc5b9..81627e9fa4c 100644 --- a/.github/workflows/nightly-build.yml +++ b/.github/workflows/nightly-build.yml @@ -9,6 +9,11 @@ on: - cron: '0 0 * * 1-5' workflow_dispatch: # Allows you to run this workflow manually. inputs: + ec2_volume_size: + description: Root volume size in GiB for each EC2 build runner + type: number + required: false + default: 500 linux_amd64_runner: type: choice description: The runner uses to build linux-amd64 artifacts @@ -113,6 +118,7 @@ jobs: aws-region: ${{ vars.EC2_RUNNER_REGION }} github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} image-id: ${{ vars.EC2_RUNNER_LINUX_AMD64_IMAGE_ID }} + ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }} security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }} subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }} @@ -127,6 +133,7 @@ jobs: aws-region: ${{ vars.EC2_RUNNER_REGION }} github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} image-id: ${{ vars.EC2_RUNNER_LINUX_ARM64_IMAGE_ID }} + ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }} security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }} subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }} @@ -153,8 +160,8 @@ jobs: image-namespace: ${{ vars.ECR_IMAGE_NAMESPACE }} aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }} aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} build-linux-arm64-artifacts: name: Build linux-arm64 artifacts @@ -179,8 +186,8 @@ jobs: image-namespace: ${{ vars.ECR_IMAGE_NAMESPACE }} aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }} aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} release-images-to-dockerhub: name: Build and push images to DockerHub diff --git a/.github/workflows/nightly-ci.yml b/.github/workflows/nightly-ci.yml index f6158a8d7ad..2b6862bb827 100644 --- a/.github/workflows/nightly-ci.yml +++ b/.github/workflows/nightly-ci.yml @@ -30,8 +30,8 @@ jobs: data-root: sqlness-test aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }} aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - name: Upload sqlness logs if: failure() uses: actions/upload-artifact@v4 @@ -57,8 +57,12 @@ jobs: with: tool: protoc@3.23.4 - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Rust Cache uses: Swatinem/rust-cache@v2 + with: + save-if: ${{ github.ref == 'refs/heads/main' }} - name: Run sqlness run: make sqlness-test env: @@ -109,8 +113,8 @@ jobs: RUST_MIN_STACK: 8388608 # 8MB CARGO_INCREMENTAL: 0 GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} - GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + GT_S3_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} GT_ETCD_TLS_ENDPOINTS: https://127.0.0.1:2378 GT_ETCD_ENDPOINTS: http://127.0.0.1:2379 @@ -143,9 +147,12 @@ jobs: - name: Install Rust toolchain uses: actions-rust-lang/setup-rust-toolchain@v1 with: + cache: false components: llvm-tools-preview - name: Rust Cache uses: Swatinem/rust-cache@v2 + with: + save-if: ${{ github.ref == 'refs/heads/main' }} - name: Install Cargo Nextest uses: taiki-e/install-action@nextest - name: Install WSL distribution @@ -160,8 +167,8 @@ jobs: RUST_MIN_STACK: 8388608 # 8 MiB, aligned with Rust CI and release tests. CARGO_INCREMENTAL: 0 GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} - GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + GT_S3_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} UNITTEST_LOG_DIR: "__unittest_logs" diff --git a/.github/workflows/nightly-jsonbench.yaml b/.github/workflows/nightly-jsonbench.yaml index d5db6f9ba4a..c246ee1f02c 100644 --- a/.github/workflows/nightly-jsonbench.yaml +++ b/.github/workflows/nightly-jsonbench.yaml @@ -112,6 +112,8 @@ jobs: id: start-linux-runner with: runner: ${{ vars.LARGE_IOPS_RUNNER }} + # Database data, WAL, and compaction files use the root volume. + ec2-volume-size: "500" aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ${{ vars.EC2_RUNNER_REGION }} diff --git a/.github/workflows/nightly-merge.yml b/.github/workflows/nightly-merge.yml new file mode 100644 index 00000000000..05f5154358b --- /dev/null +++ b/.github/workflows/nightly-merge.yml @@ -0,0 +1,153 @@ +# Setup and behavior: .github/nightly-merge.md +name: Nightly Merge + +on: + pull_request_target: + types: [opened, edited, reopened] + schedule: + # Hourly from 02:10 through 08:10 Asia/Shanghai (UTC+8), inclusive. + # GitHub may delay scheduled runs. + - cron: '10 0,18-23 * * *' + workflow_dispatch: + inputs: + dry_run: + description: List eligible PRs without adding them to the merge queue + type: boolean + default: true + +permissions: {} + +jobs: + sync-label: + if: github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request_target' + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + # PR label mutations require pull-request write permission. + pull-requests: write + issues: write + concurrency: + group: nightly-merge-label-${{ github.event.pull_request.number }} + cancel-in-progress: false + steps: + # No checkout: PR contents are untrusted data, never executable code. + - uses: actions/github-script@v7 + with: + script: | + const { owner, repo } = context.repo; + const issue_number = context.payload.pull_request.number; + const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: issue_number }); + const checkbox = /^- \[([ xX])\] Merge during the nightly window \s*$/m.exec(pr.body || ''); + // PRs without the template manage their opt-in through the label. + if (!checkbox) return; + const selected = checkbox[1].toLowerCase() === 'x'; + const label = 'merge nightly'; + const labeled = pr.labels.some(item => item.name === label); + if (selected && !labeled) { + await github.rest.issues.addLabels({ owner, repo, issue_number, labels: [label] }); + } else if (!selected && labeled) { + await github.rest.issues.removeLabel({ owner, repo, issue_number, name: label }); + } + + enqueue: + if: github.repository == 'GreptimeTeam/greptimedb' && github.event_name != 'pull_request_target' + runs-on: ubuntu-latest + timeout-minutes: 10 + concurrency: + group: nightly-merge-enqueue + cancel-in-progress: false + steps: + # Reuses the automation credential used by backport and compat updates. + # A PAT (or App token) lets subsequent merge_group workflows run. + - uses: actions/github-script@v7 + env: + DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }} + with: + github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + script: | + const { owner, repo } = context.repo; + const dryRun = process.env.DRY_RUN === 'true'; + const candidates = await github.paginate(github.rest.issues.listForRepo, { + owner, repo, state: 'open', labels: 'merge nightly', + sort: 'created', direction: 'asc', per_page: 100 + }); + const rows = []; + let enqueued = 0; + for (const candidate of candidates) { + if (!candidate.pull_request || enqueued >= 5) continue; + const number = candidate.number; + try { + // Fetch current labels and PR state, not the earlier list snapshot. + const { data: current } = await github.rest.pulls.get({ owner, repo, pull_number: number }); + if (current.state !== 'open' || current.draft || current.base.ref !== 'main' || + !current.labels.some(label => label.name === 'merge nightly')) { + rows.push([String(number), 'Skipped: not opted in, draft, closed, or not targeting main']); + continue; + } + const { repository: { pullRequest: pr } } = await github.graphql(` + query($owner: String!, $repo: String!, $number: Int!) { + repository(owner: $owner, name: $repo) { + pullRequest(number: $number) { + id headRefOid reviewDecision mergeStateStatus + mergeQueueEntry { id } + autoMergeRequest { enabledAt } + } + } + }`, { owner, repo, number }); + if (pr.mergeQueueEntry || pr.autoMergeRequest || + pr.reviewDecision !== 'APPROVED' || pr.mergeStateStatus !== 'CLEAN') { + rows.push([String(number), 'Skipped: already queued/auto-merge enabled, or not approved and clean']); + continue; + } + let cursor = null; + let unresolved = false; + do { + const { repository: { pullRequest: { reviewThreads } } } = await github.graphql(` + query($owner: String!, $repo: String!, $number: Int!, $cursor: String) { + repository(owner: $owner, name: $repo) { + pullRequest(number: $number) { + reviewThreads(first: 100, after: $cursor) { + nodes { isResolved } + pageInfo { hasNextPage endCursor } + } + } + } + }`, { owner, repo, number, cursor }); + unresolved = reviewThreads.nodes.some(thread => !thread.isResolved); + cursor = reviewThreads.pageInfo.hasNextPage ? reviewThreads.pageInfo.endCursor : null; + } while (cursor && !unresolved); + if (unresolved) { + rows.push([String(number), 'Skipped: unresolved review threads']); + continue; + } + if (!dryRun) { + // GitHub enforces branch rules; never bypass or enable auto-merge. + try { + await github.graphql(` + mutation($id: ID!, $sha: GitObjectID!) { + enqueuePullRequest(input: {pullRequestId: $id, expectedHeadOid: $sha}) { + mergeQueueEntry { id } + } + }`, { id: pr.id, sha: pr.headRefOid }); + } catch (error) { + // Stacked PRs require a different API and are not handled by this workflow. + const stackError = 'This pull request is part of a stack and must be enqueued using the asynchronous merge REST API.'; + if (error.errors?.length && error.errors.every(item => item.message?.startsWith(stackError))) { + core.warning(`PR #${number}: skipped stacked PR`); + rows.push([String(number), 'Skipped: stacked PR']); + continue; + } + throw error; + } + } + enqueued++; + rows.push([String(number), dryRun ? 'Would enqueue' : 'Enqueued']); + } catch (error) { + core.warning(`PR #${number}: ${error.message}`); + rows.push([String(number), 'Failed: see workflow logs']); + core.setFailed('Some PRs could not be processed; see workflow logs.'); + } + } + for (const [number, result] of rows) core.info(`PR #${number}: ${result}`); + await core.summary.addHeading(dryRun ? 'Nightly merge (dry run)' : 'Nightly merge') + .addTable([['PR', 'Result'], ...rows]).write(); diff --git a/.github/workflows/oswal-s3-bench.yml b/.github/workflows/oswal-s3-bench.yml index e61754cb1f1..655582ff634 100644 --- a/.github/workflows/oswal-s3-bench.yml +++ b/.github/workflows/oswal-s3-bench.yml @@ -117,6 +117,18 @@ jobs: --timeout-minutes "$BENCHMARK_TIMEOUT_MINUTES" --ttl-hours "$JANITOR_TTL_HOURS" python3 greptimedb/.github/scripts/oswal-s3-bench.py image --tag "$GREPTIMEDB_TAG" --image "$GREPTIMEDB_IMAGE_INPUT" \ --images "$OSWAL_IMAGES" + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Pull every GreptimeDB image and record its version id: images env: @@ -193,13 +205,28 @@ jobs: path: greptimedb persist-credentials: false - name: Set up benchmark dependencies - uses: ./greptimedb/.github/actions/setup-benchmark + uses: ./greptimedb/.github/actions/setup-linux-runner + with: + tools: curl jq python3 rsync + expected-architecture: x86_64 - uses: astral-sh/setup-uv@v6 - name: Start the MinIO fixture if: inputs.smoke run: | bash greptimedb/.github/scripts/pull-benchmark-image.sh greptime/minio:2024 docker compose -f greptimedb/tests-integration/fixtures/docker-compose.yml up -d --wait minio + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Pull and check images, write the plan run: | bash greptimedb/.github/scripts/oswal-s3-bench-run.sh pull @@ -211,8 +238,8 @@ jobs: --runtime-image "$RUNTIME_IMAGE" --instance-type "$INSTANCE_TYPE" --region "$S3_REGION" --prefix "$RUN_PREFIX" - name: Probe the bucket env: - AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }} run: uv run greptimedb/.github/scripts/oswal-s3-bench.py probe --bucket "$S3_BUCKET" --region "$S3_REGION" --endpoint "$S3_ENDPOINT" --prefix "$RUN_PREFIX" - name: Put the Raft Engine WAL on the instance store run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh wal-disk "$WAL_ROOT" @@ -220,8 +247,8 @@ jobs: run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh generate - name: Load every cell env: - AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }} run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh cells - name: Summary if: always() @@ -229,14 +256,14 @@ jobs: - name: Delete the run's S3 prefix if: always() env: - AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }} run: uv run greptimedb/.github/scripts/oswal-s3-bench.py cleanup --bucket "$S3_BUCKET" --region "$S3_REGION" --endpoint "$S3_ENDPOINT" --prefix "$RUN_PREFIX" - name: Check the results hold no credentials if: always() env: - AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }} run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh scrub - name: Upload results (no corpus) if: always() diff --git a/.github/workflows/publish-query-regression-helpers.yml b/.github/workflows/publish-query-regression-helpers.yml new file mode 100644 index 00000000000..eca58cff96d --- /dev/null +++ b/.github/workflows/publish-query-regression-helpers.yml @@ -0,0 +1,166 @@ +name: Publish query regression helpers + +on: + workflow_dispatch: + workflow_call: + push: + branches: [main] + paths: + - src/cmd/src/bin/query_perf_fixture.rs + - src/cmd/src/bin/query_perf_fixture/** + - src/cmd/src/bin/query_regression_runner.rs + - src/cmd/src/bin/query_regression_runner/** + - docker/query-regression-helpers/** + - .github/workflows/publish-query-regression-helpers.yml + - .github/scripts/publish-query-regression-helpers.sh + - .github/scripts/build-query-regression-helpers.sh + - .github/scripts/update-query-regression-helpers-image.sh + - Cargo.lock + +permissions: + contents: read + +# Serialize publication and updates of the repository image variable. +concurrency: + group: publish-query-regression-helpers + cancel-in-progress: false + +jobs: + provision: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} + runs-on: ubuntu-latest + timeout-minutes: 45 + outputs: + label: ${{ steps.provision.outputs.label }} + instance_id: ${{ steps.provision.outputs.instance_id }} + runner_name: ${{ steps.provision.outputs.runner_name }} + steps: + - name: Checkout trusted provisioning scripts + uses: actions/checkout@v4 + with: + repository: ${{ github.repository }} + ref: ${{ github.sha }} + persist-credentials: false + + - name: Provision ECS runner + id: provision + uses: ./.github/actions/aliyun-ecs-create + with: + access-key-id: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }} + access-key-secret: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }} + github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + region-id: ${{ vars.ALIYUN_ECS_REGION_ID }} + vswitch-id: ${{ vars.ALIYUN_ECS_VSWITCH_ID }} + security-group-id: ${{ vars.ALIYUN_ECS_SECURITY_GROUP_ID }} + instance-type: ${{ vars.ALIYUN_ECS_INSTANCE_TYPE }} + # Preserve query regression's original resource and janitor budgets. + system-disk-gib: '80' + ttl-hours: '4' + enable-docker: 'true' + resource-group-id: ${{ vars.ALIYUN_ECS_RESOURCE_GROUP_ID }} + image-id: ${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }} + runner-uid: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }} + runner-gid: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }} + + build: + needs: provision + runs-on: ${{ needs.provision.outputs.label }} + timeout-minutes: 150 + env: + CARGO_HOME: /home/runner/.cargo + CARGO_TARGET_DIR: /home/runner/query-regression-target + RUSTUP_HOME: /opt/rustup + RUSTUP_TOOLCHAIN: nightly-2026-03-21 + RUSTUP_AUTO_INSTALL: "0" + CARGO_INCREMENTAL: "0" + RUSTFLAGS: -D warnings -C link-arg=-fuse-ld=mold + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Build and validate helpers + run: bash .github/scripts/build-query-regression-helpers.sh + - name: Upload reusable helper image + uses: actions/upload-artifact@v4 + with: + name: helpers-image + path: helpers-image/ + compression-level: 1 + retention-days: 7 + if-no-files-found: error + - name: Upload build validation evidence + if: always() + uses: actions/upload-artifact@v4 + with: + name: helpers-build-validation + path: helpers-validation/ + if-no-files-found: warn + + publish: + needs: build + runs-on: ubuntu-latest + timeout-minutes: 45 + outputs: + image: ${{ steps.publish.outputs.image }} + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/download-artifact@v4 + with: + name: helpers-image + path: helpers-image/ + - name: Login to registry + uses: docker/login-action@v3 + with: + registry: greptime-registry.cn-hangzhou.cr.aliyuncs.com + username: ${{ secrets.ALICLOUD_USERNAME }} + password: ${{ secrets.ALICLOUD_PASSWORD }} + - name: Publish and verify helpers + id: publish + run: bash .github/scripts/publish-query-regression-helpers.sh + - name: Upload publication validation evidence + if: always() + uses: actions/upload-artifact@v4 + with: + name: helpers-publish-validation + path: | + helpers-validation/digest.txt + helpers-validation/extracted/helpers-image.json + if-no-files-found: warn + + update-variable: + needs: publish + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Update helpers image repository variable + env: + GH_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + HELPERS_IMAGE: ${{ needs.publish.outputs.image }} + run: bash .github/scripts/update-query-regression-helpers-image.sh + + teardown: + if: ${{ always() && needs.provision.outputs.instance_id != '' }} + needs: [provision, build] + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout trusted teardown scripts + uses: actions/checkout@v4 + with: + repository: ${{ github.repository }} + ref: ${{ github.sha }} + persist-credentials: false + + - name: Teardown ECS runner + uses: ./.github/actions/aliyun-ecs-delete + with: + access-key-id: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }} + access-key-secret: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }} + github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + region-id: ${{ vars.ALIYUN_ECS_REGION_ID }} + instance-id: ${{ needs.provision.outputs.instance_id }} + runner-name: ${{ needs.provision.outputs.runner_name }} diff --git a/.github/workflows/query-regression.yml b/.github/workflows/query-regression.yml index 6a368c455d6..d65d6666658 100644 --- a/.github/workflows/query-regression.yml +++ b/.github/workflows/query-regression.yml @@ -26,6 +26,21 @@ on: description: Candidate ref/sha to build; reusable release calls must provide an immutable full SHA required: true type: string + base_image: + description: Base Docker image tag/digest (empty builds base_ref from source) + required: false + type: string + default: "" + candidate_image: + description: Candidate Docker image tag/digest (empty builds candidate_ref from source) + required: false + type: string + default: "" + helpers_image: + description: Helpers image@sha256 or pinned from repository variable (empty builds from source) + required: false + type: string + default: "" cargo_profile: description: Cargo profile to build base and candidate binaries with required: false @@ -80,6 +95,21 @@ on: description: Runner HTTP timeout seconds required: true default: "300" + base_image: + description: Base Docker image tag/digest (empty builds base_ref from source) + required: false + type: string + default: "" + candidate_image: + description: Candidate Docker image tag/digest (empty builds candidate_ref from source) + required: false + type: string + default: "" + helpers_image: + description: Helpers image@sha256 or pinned from repository variable (empty builds from source) + required: false + type: string + default: "" cargo_profile: description: Cargo profile to build base and candidate binaries with required: true @@ -132,6 +162,7 @@ jobs: python3 tests/perf/test_query_regression_runner_otlp_trace_load.py python3 tests/perf/test_query_regression_summary_otlp.py python3 tests/perf/test_query_regression_case_selection.py + python3 tests/perf/test_query_regression_helpers.py python3 tests/perf/test_query_regression_nightly_refs.py python3 tests/perf/test_query_regression_slash.py python3 tests/perf/test_aliyun_ecs_runner_scripts.py @@ -171,6 +202,7 @@ jobs: # Preserve query regression's original resource and janitor budgets. system-disk-gib: '80' ttl-hours: '4' + enable-docker: ${{ inputs.base_image != '' || inputs.candidate_image != '' || inputs.helpers_image != '' }} resource-group-id: ${{ vars.ALIYUN_ECS_RESOURCE_GROUP_ID }} image-id: ${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }} runner-uid: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }} @@ -214,6 +246,10 @@ jobs: PR_NUMBER: ${{ inputs.pr_number }} HEAD_REPO: ${{ inputs.head_repo }} BASE_REPO: ${{ inputs.base_repo }} + HELPERS_IMAGE: ${{ inputs.helpers_image }} + QUERY_REGRESSION_HELPERS_IMAGE: ${{ vars.QUERY_REGRESSION_HELPERS_IMAGE }} + BASE_IMAGE: ${{ inputs.base_image }} + CANDIDATE_IMAGE: ${{ inputs.candidate_image }} BASE_REF: ${{ inputs.base_ref }} CANDIDATE_REF: ${{ inputs.candidate_ref || github.ref }} EVENT_MERGE_SHA: ${{ inputs.pr_number != '' && inputs.candidate_ref || '' }} @@ -224,6 +260,16 @@ jobs: EXPECTED_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }} EXPECTED_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }} steps: + - name: Validate image target admission + run: | + if [[ "${PR_ADMISSION}" == "true" && ( -n "${BASE_IMAGE}" || -n "${CANDIDATE_IMAGE}" || -n "${HELPERS_IMAGE}" ) ]]; then + echo "PR admission requires source targets for verified commit attribution." >&2 + exit 1 + fi + if [[ -n "${BASE_IMAGE}" || -n "${CANDIDATE_IMAGE}" || -n "${HELPERS_IMAGE}" ]]; then + docker info >/dev/null + fi + - name: Report provisioned ECS runner if: ${{ needs.provision.outputs.instance_id != '' }} shell: bash @@ -418,6 +464,7 @@ jobs: > "${GITHUB_WORKSPACE}/query-regression-trusted-scripts/query-regression-summary.py" - name: Verify runner image tools + if: ${{ inputs.base_image == '' || inputs.candidate_image == '' || inputs.helpers_image == '' }} shell: bash run: | set -euo pipefail @@ -534,6 +581,7 @@ jobs: fi - name: Prepare query regression cache + if: ${{ inputs.base_image == '' || inputs.candidate_image == '' || inputs.helpers_image == '' }} shell: bash working-directory: src run: | @@ -549,7 +597,7 @@ jobs: readonly EXPECTED_SCCACHE_DIR="/home/runner/.cache/sccache" readonly EXPECTED_RUSTC_WRAPPER="/usr/local/bin/sccache" readonly RUNNER_IMAGE_DIGEST="sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11" - readonly RUNNER_IMAGE_EPOCH="7" + readonly RUNNER_IMAGE_EPOCH="8" require_expected_root() { local name="$1" @@ -694,6 +742,7 @@ jobs: sccache --show-stats - name: Build base greptime + if: ${{ inputs.base_image == '' }} working-directory: src run: | set -euo pipefail @@ -728,14 +777,27 @@ jobs: git reset --hard "${VERIFIED_CANDIDATE_SHA}" git clean -ffdx + - name: Extract precompiled query regression helpers + if: ${{ inputs.helpers_image != '' }} + working-directory: src + run: | + python3 .github/scripts/query-regression-helpers.py "$HELPERS_IMAGE" \ + "$GITHUB_WORKSPACE/query-regression-bins/candidate" + cat "$GITHUB_WORKSPACE/query-regression-bins/candidate/helpers-image.json" >> "$GITHUB_STEP_SUMMARY" + - name: Build candidate greptime and query regression helpers + if: ${{ inputs.candidate_image == '' || inputs.helpers_image == '' }} working-directory: src run: | set -euo pipefail SECONDS=0 - cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime - cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools - cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_regression_runner --features dev-tools + if [[ -z "${CANDIDATE_IMAGE}" ]]; then + cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime + fi + if [[ -z "${HELPERS_IMAGE}" ]]; then + cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools + cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_regression_runner --features dev-tools + fi candidate_build_elapsed="${SECONDS}" printf 'Candidate greptime and query regression helper cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}" printf -- '- Candidate greptime and query regression helper cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}" @@ -745,12 +807,16 @@ jobs: target_dir="debug" fi mkdir -p "${GITHUB_WORKSPACE}/query-regression-bins/candidate" - cp "${CARGO_TARGET_DIR}/${target_dir}/greptime" \ - "${GITHUB_WORKSPACE}/query-regression-bins/candidate/greptime" - cp "${CARGO_TARGET_DIR}/${target_dir}/query_perf_fixture" \ - "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_perf_fixture" - cp "${CARGO_TARGET_DIR}/${target_dir}/query_regression_runner" \ - "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_regression_runner" + if [[ -z "${CANDIDATE_IMAGE}" ]]; then + cp "${CARGO_TARGET_DIR}/${target_dir}/greptime" \ + "${GITHUB_WORKSPACE}/query-regression-bins/candidate/greptime" + fi + if [[ -z "${HELPERS_IMAGE}" ]]; then + cp "${CARGO_TARGET_DIR}/${target_dir}/query_perf_fixture" \ + "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_perf_fixture" + cp "${CARGO_TARGET_DIR}/${target_dir}/query_regression_runner" \ + "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_regression_runner" + fi - name: Run query regression id: run @@ -792,6 +858,7 @@ jobs: query-regression-work/**/logs/** query-regression-work/**/otelgen/** query-regression-summary.md + query-regression-bins/candidate/helpers-image.json machine-telemetry.log if-no-files-found: warn retention-days: 7 diff --git a/.github/workflows/release-dev-builder-images.yaml b/.github/workflows/release-dev-builder-images.yaml index 2e1041c1ff4..9f74035f8c9 100644 --- a/.github/workflows/release-dev-builder-images.yaml +++ b/.github/workflows/release-dev-builder-images.yaml @@ -35,6 +35,11 @@ on: description: Rebuild the query-regression ECS runner image (and update QUERY_REGRESSION_ECS_IMAGE_ID + RUNNER_IMAGE_EPOCH) required: false default: false + release_query_regression_helpers_image: + type: boolean + description: Publish precompiled query regression helpers and update the image repository variable + required: false + default: false update_dev_builder_image_tag: type: boolean description: Update the DEV_BUILDER_IMAGE_TAG in Makefile and create a PR @@ -451,3 +456,8 @@ jobs: echo "## Lockstep updates" >> "$GITHUB_STEP_SUMMARY" echo "- QUERY_REGRESSION_ECS_IMAGE_ID → \`${image_id}\`" >> "$GITHUB_STEP_SUMMARY" echo "- RUNNER_IMAGE_EPOCH → \`${new_epoch}\` (PR on \`${BRANCH}\`, pending review/merge)" >> "$GITHUB_STEP_SUMMARY" + + release-query-regression-helpers: + if: ${{ github.repository == 'GreptimeTeam/greptimedb' && inputs.release_query_regression_helpers_image }} + uses: ./.github/workflows/publish-query-regression-helpers.yml + secrets: inherit diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b292b586453..c4489f9aabd 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -20,6 +20,11 @@ on: - cron: '0 0 * * 1-5' workflow_dispatch: # Allows you to run this workflow manually. inputs: + ec2_volume_size: + description: Root volume size in GiB for each EC2 build runner + type: number + required: false + default: 500 linux_amd64_runner: type: choice description: The runner uses to build linux-amd64 artifacts @@ -215,6 +220,7 @@ jobs: aws-region: ${{ vars.EC2_RUNNER_REGION }} github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} image-id: ${{ vars.EC2_RUNNER_LINUX_AMD64_IMAGE_ID }} + ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }} security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }} subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }} @@ -229,6 +235,7 @@ jobs: aws-region: ${{ vars.EC2_RUNNER_REGION }} github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} image-id: ${{ vars.EC2_RUNNER_LINUX_ARM64_IMAGE_ID }} + ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }} security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }} subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }} @@ -255,8 +262,8 @@ jobs: image-namespace: ${{ vars.ECR_IMAGE_NAMESPACE }} aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }} aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} build-linux-arm64-artifacts: name: Build linux-arm64 artifacts @@ -282,8 +289,8 @@ jobs: large-page-size: true aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }} aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} - aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }} + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} build-linux-riscv64-artifacts: name: Build linux-riscv64 artifacts @@ -298,6 +305,11 @@ jobs: fetch-depth: 0 persist-credentials: false + - name: Set up Linux runner + uses: ./.github/actions/setup-linux-runner + with: + tools: make shasum + - name: Prepare riscv64 dev-builder image shell: bash env: diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 842fa306c4e..d66bc042b5b 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -24,7 +24,7 @@ on: name: Rust CI env: - CARGO_SHEAR_VERSION: "1.13.4" + SHEAR_VERSION: "1.13.4" concurrency: group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} @@ -56,6 +56,7 @@ jobs: tool: protoc@3.23.4 - uses: actions-rust-lang/setup-rust-toolchain@v1 with: + cache: false components: rustfmt - name: Check format run: make fmt-check @@ -75,6 +76,7 @@ jobs: tool: protoc@3.23.4 - uses: actions-rust-lang/setup-rust-toolchain@v1 with: + cache: false components: clippy - name: Rust Cache uses: Swatinem/rust-cache@v2 @@ -97,10 +99,12 @@ jobs: with: persist-credentials: false - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Install cargo-shear uses: taiki-e/install-action@v2 with: - tool: cargo-shear@${{ env.CARGO_SHEAR_VERSION }} + tool: cargo-shear@${{ env.SHEAR_VERSION }} fallback: none - name: Check unused dependencies run: make check-udeps @@ -122,6 +126,8 @@ jobs: - name: Install riscv64 cross toolchain run: sudo apt-get update && sudo apt-get install -y gcc-riscv64-linux-gnu g++-riscv64-linux-gnu - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Add riscv64 target run: rustup target add riscv64gc-unknown-linux-gnu - name: Rust Cache @@ -153,6 +159,8 @@ jobs: with: version: "14.0" - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache: false - name: Rust Cache uses: Swatinem/rust-cache@v2 with: diff --git a/.github/workflows/tracesbench.yml b/.github/workflows/tracesbench.yml index 049faaef1a7..73fe62001df 100644 --- a/.github/workflows/tracesbench.yml +++ b/.github/workflows/tracesbench.yml @@ -2,6 +2,11 @@ name: Tracesbench on: workflow_dispatch: inputs: + greptimedb_storage: + description: 'GreptimeDB data storage (S3 requires AWS in the bucket region; WAL stays local)' + type: choice + options: [file, s3, both] + default: file run_greptimedb: description: Run GreptimeDB type: boolean @@ -39,7 +44,7 @@ on: default: sanity required: true runtime_image: - description: 'Runtime image: auto = AWS ECR Public or Aliyun ACR by provider; explicit tag/digest overrides' + description: 'Runtime image: auto = pinned S3 test runtime on ACR; explicit tag/digest overrides' type: string default: auto required: true @@ -107,6 +112,11 @@ on: permissions: contents: read env: + GREPTIMEDB_STORAGE: ${{ inputs.greptimedb_storage || 'file' }} + S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} + S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} + RUNNER_REGION: ${{ vars.EC2_RUNNER_REGION }} + RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }} COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }} BASELINE_GREPTIMEDB_TAG: ${{ inputs.baseline_greptimedb_tag }} DATASET: ${{ inputs.dataset }} @@ -114,7 +124,7 @@ env: RUNTIME_CPUS: ${{ inputs.runtime_cpus }} RUNTIME_MEMORY: ${{ inputs.runtime_memory }} BENCHMARK_ROOT: ${{ inputs.benchmark_root }} - RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || inputs.provider == 'AWS' && 'public.ecr.aws/i8k6a5e1/greptime/o11ybenchruntime:77a84e1c8136@sha256:c482993ec177b026235d6cd0c61d4fb6d6f1088bc78bdc0e5964a5398de6dba0' || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:77a84e1c8136@sha256:c482993ec177b026235d6cd0c61d4fb6d6f1088bc78bdc0e5964a5398de6dba0' }} + RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:s3-test-b0cf5204d2fc@sha256:c2bf622db6056a2029842f4dc1cc1f37b82773bd45b134a0a83ffe2a8cd62214' }} PROVIDER: ${{ inputs.provider }} REQUESTED_INSTANCE_TYPE: ${{ inputs.instance_type }} SYSTEM_DISK_GIB: ${{ inputs.system_disk_gib }} @@ -162,6 +172,10 @@ jobs: [[ "$GREPTIMEDB_TAG" =~ ^[a-zA-Z0-9_][a-zA-Z0-9_.-]{0,127}$ ]] || { echo 'Invalid GreptimeDB image tag' >&2; exit 1; } fi printf 'targets=%s\n' "$selected" >> "$GITHUB_OUTPUT" + - name: Validate GreptimeDB storage + run: | + python3 greptimedb/tests/perf/test_o11ybench_storage.py + python3 greptimedb/.github/scripts/o11ybench-storage.py validate - name: Validate version comparison env: RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }} @@ -180,6 +194,18 @@ jobs: esac fi python3 greptimedb/.github/scripts/ci-runner-config.py --provider "$PROVIDER" --instance-type "$instance_type" + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Validate budgets and shared runtime id: runtime run: | @@ -305,7 +331,10 @@ jobs: path: greptimedb persist-credentials: false - name: Set up benchmark dependencies - uses: ./greptimedb/.github/actions/setup-benchmark + uses: ./greptimedb/.github/actions/setup-linux-runner + with: + tools: curl jq python3 rsync + expected-architecture: x86_64 - name: Checkout matching benchmark tools uses: actions/checkout@v4 with: @@ -314,6 +343,18 @@ jobs: ref: ${{ env.RESOLVED_O11YBENCH_REF }} path: o11ybench persist-credentials: false + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Check runner and runtime identity id: runtime run: | @@ -362,22 +403,44 @@ jobs: python3 o11ybench/scripts/run_tracesbench_docker.py --stage generate \ --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" - - name: Baseline GreptimeDB load, correctness and benchmark - if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.compare_greptimedb }} + - name: Baseline GreptimeDB load, correctness and benchmark (File) + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' && inputs.compare_greptimedb }} run: | - python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --target greptimedb --port 14000 --greptimedb-image "$BASELINE_GREPTIMEDB_IMAGE" \ + python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage file --target greptimedb --port 14000 --greptimedb-image "$BASELINE_GREPTIMEDB_IMAGE" \ --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \ --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ --root "$RUN_ROOT/baseline" --data-root "$RUN_ROOT" --owner "$OWNER-base" --runtime-image "$RESOLVED_RUNTIME_IMAGE" python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/baseline/runs/greptimedb" - - name: GreptimeDB load, correctness and benchmark - if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' }} + - name: Baseline GreptimeDB load, correctness and benchmark (S3) + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' && inputs.compare_greptimedb }} run: | - python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --target greptimedb --port 14000 --greptimedb-image "$GREPTIMEDB_IMAGE" \ + python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage s3 --target greptimedb --port 14000 --greptimedb-image "$BASELINE_GREPTIMEDB_IMAGE" \ + --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \ + --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ + --root "$RUN_ROOT/baseline/s3" --data-root "$RUN_ROOT" --owner "$OWNER-base-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE" + python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/baseline/s3/runs/greptimedb" + - name: GreptimeDB load, correctness and benchmark (File) + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' }} + run: | + python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage file --target greptimedb --port 14000 --greptimedb-image "$GREPTIMEDB_IMAGE" \ --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \ --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/runs/greptimedb" + - name: GreptimeDB load, correctness and benchmark (S3) + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} + if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' }} + run: | + python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage s3 --target greptimedb --port 14000 --greptimedb-image "$GREPTIMEDB_IMAGE" \ + --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \ + --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ + --root "$RUN_ROOT/s3" --data-root "$RUN_ROOT" --owner "$OWNER-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE" + python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/s3/runs/greptimedb" - name: VictoriaTraces load, correctness and benchmark if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.victoriatraces == 'true' }} run: | @@ -395,49 +458,80 @@ jobs: - name: GreptimeDB version comparison if: always() && inputs.compare_greptimedb run: | - python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind traces --root "$RUN_ROOT" >> "$GITHUB_STEP_SUMMARY" - - name: Validate and render report + python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind traces --root "$RUN_ROOT" | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + - name: Validate and render primary report + if: always() && steps.generate.outcome == 'success' env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }} run: | + if [[ "$GREPTIMEDB_STORAGE" == s3 ]]; then + TARGETS=$(tr "," "\n" <<< "$TARGETS" | sed "/^greptimedb$/d" | paste -sd, -) + [[ -n "$TARGETS" ]] || exit 0 + fi comparison_args=() - if [[ "$COMPARE_GREPTIMEDB" == true ]]; then + if [[ "$COMPARE_GREPTIMEDB" == true && "$GREPTIMEDB_STORAGE" != s3 ]]; then comparison_args+=(--baseline-run "$RUN_ROOT/baseline/runs/greptimedb/run.json") fi - python3 o11ybench/scripts/run_tracesbench_docker.py --stage report --targets "$TARGETS" "${comparison_args[@]}" \ + python3 o11ybench/scripts/run_tracesbench_docker.py --stage report --targets "${TARGETS}" "${comparison_args[@]}" \ --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Validate and render S3 report + if: always() && steps.generate.outcome == 'success' && inputs.greptimedb_storage != 'file' + run: | + python3 o11ybench/scripts/run_tracesbench_docker.py --stage report --targets greptimedb \ + --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ + --root "$RUN_ROOT/s3" --data-root "$RUN_ROOT" --owner "$OWNER-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE" - name: Clean up owned containers + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} if: always() && steps.runtime.outcome == 'success' run: | - python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \ + python3 greptimedb/.github/scripts/o11ybench-storage.py run -- python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \ --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \ --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" - name: Clean up baseline containers + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} if: always() && steps.runtime.outcome == 'success' && inputs.compare_greptimedb run: | - python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \ + python3 greptimedb/.github/scripts/o11ybench-storage.py run -- python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \ --root "$RUN_ROOT/baseline" --owner "$OWNER-base" --runtime-image "$RESOLVED_RUNTIME_IMAGE" - name: Benchmark summary if: always() - run: python3 greptimedb/.github/scripts/tracesbench-summary.py --root "$RUN_ROOT" --targets "$TARGETS" --section benchmark >> "$GITHUB_STEP_SUMMARY" + run: python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind traces --root "$RUN_ROOT" --targets "$TARGETS" --section benchmark | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" - name: Load / lifecycle summary if: always() - run: python3 greptimedb/.github/scripts/tracesbench-summary.py --root "$RUN_ROOT" --targets "$TARGETS" --section lifecycle >> "$GITHUB_STEP_SUMMARY" + run: python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind traces --root "$RUN_ROOT" --targets "$TARGETS" --section lifecycle | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" - name: Baseline load / lifecycle summary if: always() && inputs.compare_greptimedb run: | - echo '## Baseline GreptimeDB' >> "$GITHUB_STEP_SUMMARY" - python3 greptimedb/.github/scripts/tracesbench-summary.py \ - --root "$RUN_ROOT/baseline" --targets greptimedb --section lifecycle >> "$GITHUB_STEP_SUMMARY" + echo '## Baseline GreptimeDB' | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind traces \ + --root "$RUN_ROOT/baseline" --targets greptimedb --section lifecycle | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + - name: Render storage comparison overview + if: always() && steps.generate.outcome == 'success' + run: | + mkdir -p "$RUN_ROOT/metadata" + cp "$METADATA_ROOT"/*-image.json "$RUN_ROOT/metadata/" + python3 o11ybench/scripts/build_query_performance_report.py --traces "$RUN_ROOT" \ + --output "$RUN_ROOT-query-report.html" --json-output "$RUN_ROOT-query-report.json" + python3 greptimedb/.github/scripts/o11ybench-storage.py overview "$RUN_ROOT-query-report.json" \ + | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/o11ybench-storage.py audit --root "$RUN_ROOT" \ + --report "$RUN_ROOT-query-report.json" --targets "$TARGETS" --allow-trace-query-failures - name: Stage benchmark artifacts if: always() run: | mkdir -p "$RUNNER_TEMP/tracesbench-artifacts" + cp "$RUN_ROOT"-query-report.* "$RUNNER_TEMP/tracesbench-artifacts/" 2>/dev/null || true if [[ -d "$METADATA_ROOT" ]]; then rsync -a --no-owner --no-group "$METADATA_ROOT" "$RUNNER_TEMP/tracesbench-artifacts/" fi - for entry in runs report report.json report.md version-comparison.json version-comparison.md; do + for entry in artifact-audit.json metadata runs report report.json report.md ci-summary.md version-comparison.json version-comparison.md; do if [[ -e "$RUN_ROOT/$entry" ]]; then rsync -a --no-owner --no-group "$RUN_ROOT/$entry" "$RUNNER_TEMP/tracesbench-artifacts/" fi @@ -446,6 +540,12 @@ jobs: mkdir -p "$RUNNER_TEMP/tracesbench-artifacts/baseline" rsync -a --no-owner --no-group "$RUN_ROOT/baseline/runs" "$RUNNER_TEMP/tracesbench-artifacts/baseline/" fi + for arm in s3 baseline/s3; do + if [[ -d "$RUN_ROOT/$arm" ]]; then + mkdir -p "$RUNNER_TEMP/tracesbench-artifacts/$arm" + rsync -a --no-owner --no-group --exclude scratch --exclude corpora "$RUN_ROOT/$arm/" "$RUNNER_TEMP/tracesbench-artifacts/$arm/" + fi + done - name: Upload HTML and results (no corpus) if: always() uses: actions/upload-artifact@v4 diff --git a/.github/workflows/vmbench-long-range.yml b/.github/workflows/vmbench-long-range.yml index aaf2dd7f437..625156b6652 100644 --- a/.github/workflows/vmbench-long-range.yml +++ b/.github/workflows/vmbench-long-range.yml @@ -2,6 +2,11 @@ name: VMBench Long Range on: workflow_dispatch: inputs: + greptimedb_storage: + description: 'GreptimeDB data storage (S3 requires AWS in the bucket region; WAL stays local)' + type: choice + options: [file, s3, both] + default: file run_greptimedb: description: Run GreptimeDB type: boolean @@ -12,6 +17,11 @@ on: type: boolean default: false required: false + run_prometheus: + description: Run Prometheus (requires a runtime image with managed Prometheus support) + type: boolean + default: false + required: false range_preset: description: Long-range query window (fixed dataset per preset) type: choice @@ -40,8 +50,13 @@ on: type: string default: v1.114.0@sha256:227ad53fd57a4b4430624898a1be0cac39431e1235fa0dc92f567cce38969344 required: true + prometheus_tag: + description: Docker Hub prom/prometheus tag or tag@sha256 digest + type: string + default: v3.2.1@sha256:6927e0919a144aa7616fd0137d4816816d42f6b816de3af269ab065250859a62 + required: true runtime_image: - description: 'Runtime image: auto = AWS ECR Public or Aliyun ACR by provider; explicit tag/digest overrides' + description: 'Runtime image: auto = pinned Prometheus/File/S3 runtime on ACR; explicit tag/digest overrides' type: string default: auto required: true @@ -89,13 +104,19 @@ on: permissions: contents: read env: + GREPTIMEDB_STORAGE: ${{ inputs.greptimedb_storage || 'file' }} + S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }} + S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }} + RUNNER_REGION: ${{ vars.EC2_RUNNER_REGION }} + RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }} COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }} BASELINE_GREPTIMEDB_TAG: ${{ inputs.baseline_greptimedb_tag }} BENCHMARK_ROOT: ${{ inputs.benchmark_root }} RANGE_PRESET: ${{ inputs.range_preset }} GREPTIMEDB_TAG: ${{ inputs.greptimedb_tag }} VICTORIAMETRICS_TAG: ${{ inputs.victoriametrics_tag }} - RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || inputs.provider == 'AWS' && 'public.ecr.aws/i8k6a5e1/greptime/o11ybenchruntime:b43785b56f42@sha256:e38811eb68caf3682ba56f5591a3e8e27e78e7983a923abb29fa2d7cd8fdbd46' || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:b43785b56f42@sha256:e38811eb68caf3682ba56f5591a3e8e27e78e7983a923abb29fa2d7cd8fdbd46' }} + PROMETHEUS_TAG: ${{ inputs.prometheus_tag }} + RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:345b6f7c-prometheus-s3@sha256:39d207f4a9c479b8614b8283e2d049fc8f430fcbd2b3362ce3035cfbf294edf9' }} PROVIDER: ${{ inputs.provider }} REQUESTED_INSTANCE_TYPE: ${{ inputs.instance_type }} SYSTEM_DISK_GIB: ${{ inputs.system_disk_gib }} @@ -119,6 +140,10 @@ jobs: with: path: greptimedb persist-credentials: false + - name: Validate GreptimeDB storage + run: | + python3 greptimedb/tests/perf/test_o11ybench_storage.py + python3 greptimedb/.github/scripts/o11ybench-storage.py validate - name: Validate version comparison env: RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }} @@ -132,11 +157,24 @@ jobs: run: | python3 greptimedb/tests/perf/test_aliyun_ecs_runner_scripts.py python3 greptimedb/tests/perf/test_ci_runner_scripts.py + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Validate budgets and build warm plan id: plan env: RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }} RUN_VICTORIAMETRICS: ${{ inputs.run_victoriametrics }} + RUN_PROMETHEUS: ${{ inputs.run_prometheus }} run: | set -euo pipefail [[ "$BENCHMARK_ROOT" == /* && "$BENCHMARK_ROOT" != / && "$BENCHMARK_ROOT" != */ ]] @@ -161,17 +199,18 @@ jobs: [[ "$revision" =~ ^[0-9a-f]{40}$ ]] printf 'runtime_image=%s\no11ybench_ref=%s\nsystem_disk_gib=%s\n' "$resolved" "$revision" "$SYSTEM_DISK_GIB" >> "$GITHUB_OUTPUT" printf 'run_id=long-range-%s-%s\nplan_artifact=vmbench-plan-%s-%s\n' "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT" "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT" >> "$GITHUB_OUTPUT" - args=(--workload vmbench-long-range --range-preset "$RANGE_PRESET" + args=(--workload vmbench-long-range --range-preset "$RANGE_PRESET" --greptimedb-storage "$GREPTIMEDB_STORAGE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --run-classification controlled_server_benchmark --no-open-browser) if [[ "$COMPARE_GREPTIMEDB" == true ]]; then args+=(--target "greptimedb:name=greptimedb-baseline,image=greptime/greptimedb:$BASELINE_GREPTIMEDB_TAG,url=http://127.0.0.1:14000/v1/prometheus/api/v1/query_range") fi # Tags become target-spec fields; reject separators before passing them to the planner. - for target in greptimedb victoriametrics; do + for target in greptimedb victoriametrics prometheus; do case "$target" in greptimedb) enabled=$RUN_GREPTIMEDB; tag=$GREPTIMEDB_TAG; repo=greptime/greptimedb; url=http://127.0.0.1:14000/v1/prometheus/api/v1/query_range ;; victoriametrics) enabled=$RUN_VICTORIAMETRICS; tag=$VICTORIAMETRICS_TAG; repo=victoriametrics/victoria-metrics; url=http://127.0.0.1:18428/api/v1/query_range ;; + prometheus) enabled=$RUN_PROMETHEUS; tag=$PROMETHEUS_TAG; repo=prom/prometheus; url=http://127.0.0.1:19090/api/v1/query_range ;; esac [[ "$enabled" == true ]] || continue [[ "$tag" =~ ^[a-zA-Z0-9_][a-zA-Z0-9_.-]{0,127}(@sha256:[0-9a-f]{64})?$ ]] @@ -180,10 +219,11 @@ jobs: # The packaged planner owns profile, target and resource-limit validation. mkdir -p "$RUNNER_TEMP/vmbench-plans" # Keep the original warm defaults: serial 5/1 and c1/c4 30s/5s. - docker run --rm "$resolved" scripts/o11ybench_plan.py "${args[@]}" \ + docker run --rm -e S3_BUCKET -e S3_REGION "$resolved" scripts/o11ybench_plan.py "${args[@]}" \ --run-id "long-range-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT-warm" \ --output-root "$BENCHMARK_ROOT/o11ybench/o11ybench-long-range-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT-warm" \ --cache-policy warm > "$RUNNER_TEMP/vmbench-plans/warm.json" + python3 greptimedb/.github/scripts/o11ybench-storage.py normalize-metrics "$RUNNER_TEMP/vmbench-plans/warm.json" - uses: actions/upload-artifact@v4 with: name: vmbench-plan-${{ github.run_id }}-${{ github.run_attempt }} @@ -266,7 +306,10 @@ jobs: path: greptimedb persist-credentials: false - name: Set up benchmark dependencies - uses: ./greptimedb/.github/actions/setup-benchmark + uses: ./greptimedb/.github/actions/setup-linux-runner + with: + tools: curl jq python3 rsync + expected-architecture: x86_64 - name: Checkout matching benchmark tools uses: actions/checkout@v4 with: @@ -279,6 +322,18 @@ jobs: with: name: ${{ needs.validate.outputs.plan_artifact }} path: ${{ env.METADATA_ROOT }} + - name: Configure AWS credentials for ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + - name: Login to Amazon ECR Public + if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/') + uses: aws-actions/amazon-ecr-login@v2 + with: + registry-type: public - name: Check runner and resolve database images run: | set -euo pipefail @@ -313,25 +368,92 @@ jobs: --plan "$METADATA_ROOT/warm.json" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --dry-run \ > "$METADATA_ROOT/warm-execution-plan.json" cat "$METADATA_ROOT/warm-execution-plan.json" - - name: Warm serial and concurrent benchmarks + - name: Generate shared dataset and split target plans + id: generate run: | - set -euo pipefail - python3 -u o11ybench/scripts/run_benchmark_plan.py \ - --plan "$METADATA_ROOT/warm.json" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --execute \ - 2>&1 | tee "$METADATA_ROOT/warm-execution.log" - python3 o11ybench/scripts/check_run_benchmark_plan.py --artifacts "$ARTIFACT_ROOT-warm" + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-prepare \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Load and benchmark Baseline GreptimeDB File + if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.compare_greptimedb && inputs.greptimedb_storage != 's3' }} + run: | + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb-baseline \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Load and benchmark Baseline GreptimeDB S3 + if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.compare_greptimedb && inputs.greptimedb_storage != 'file' }} + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + run: | + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb-s3-baseline \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Load and benchmark GreptimeDB File + if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_greptimedb && inputs.greptimedb_storage != 's3' }} + run: | + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Load and benchmark GreptimeDB S3 + if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_greptimedb && inputs.greptimedb_storage != 'file' }} + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + run: | + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb-s3 \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Load and benchmark VictoriaMetrics + if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_victoriametrics }} + run: | + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target victoriametrics \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Load and benchmark Prometheus + if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_prometheus }} + run: | + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target prometheus \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" + - name: Validate all selected target results + if: always() && steps.generate.outcome == 'success' + run: | + python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-validate \ + --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE" - name: Clean up owned containers + env: + AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }} + AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }} if: always() run: | docker ps -aq --filter "label=o11ybench.run-id=$RUN_ID-warm" | xargs -r docker rm -fv + for name in greptimedb greptimedb-s3 greptimedb-baseline greptimedb-s3-baseline victoriametrics prometheus; do + plan="$METADATA_ROOT/$name.json" + [[ -f "$plan" ]] || continue + run_id=$(jq -r '.run_id // empty' "$plan") + [[ -n "$run_id" ]] || continue + docker ps -aq --filter "label=o11ybench.run-id=$run_id" | xargs -r docker rm -fv + done + if [[ -d "$ARTIFACT_ROOT-warm" ]]; then + for probe in "$ARTIFACT_ROOT-warm"/o11ybench/*/*/s3-probe.json; do + [[ -f "$probe" ]] || continue + python3 o11ybench/scripts/greptimedb_storage.py recover --runtime-image "$RESOLVED_RUNTIME_IMAGE" --output "${probe%/*}" + done + fi - name: GreptimeDB version comparison if: always() && inputs.compare_greptimedb run: | - python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind long-range --root "$ARTIFACT_ROOT-warm" >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind long-range --root "$ARTIFACT_ROOT-warm" | tee -a "$ARTIFACT_ROOT-warm/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + - name: Render storage comparison overview + if: always() && steps.generate.outcome == 'success' + run: | + mkdir -p "$ARTIFACT_ROOT-warm/metadata" + cp "$METADATA_ROOT"/*-image.json "$ARTIFACT_ROOT-warm/metadata/" + python3 o11ybench/scripts/build_query_performance_report.py --metrics "$ARTIFACT_ROOT-warm" \ + --output "$ARTIFACT_ROOT-warm-query-report.html" --json-output "$ARTIFACT_ROOT-warm-query-report.json" + python3 greptimedb/.github/scripts/o11ybench-storage.py overview "$ARTIFACT_ROOT-warm-query-report.json" \ + | tee -a "$ARTIFACT_ROOT-warm/ci-summary.md" >> "$GITHUB_STEP_SUMMARY" + python3 greptimedb/.github/scripts/o11ybench-storage.py audit --root "$ARTIFACT_ROOT-warm" \ + --report "$ARTIFACT_ROOT-warm-query-report.json" --targets "$(jq -r '[.targets[].kind] | unique | join(",")' "$METADATA_ROOT/warm.json")" - name: Stage benchmark artifacts if: always() run: | mkdir -p "$RUNNER_TEMP/vmbench-artifacts" + cp "$ARTIFACT_ROOT-warm"-query-report.* "$RUNNER_TEMP/vmbench-artifacts/" 2>/dev/null || true if [[ -d "$METADATA_ROOT" ]]; then rsync -a --no-owner --no-group "$METADATA_ROOT" "$RUNNER_TEMP/vmbench-artifacts/" fi diff --git a/.gitignore b/.gitignore index dd60d96224e..dba299862bb 100644 --- a/.gitignore +++ b/.gitignore @@ -79,3 +79,5 @@ docs/specs/ # Claude Code personal/local overrides (keep shared .claude/ content tracked) .claude/*.local.* + +output/ diff --git a/Cargo.lock b/Cargo.lock index 34d2e06170c..8db5c0b3339 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -663,8 +663,8 @@ dependencies = [ [[package]] name = "arrow-pg" -version = "0.15.0" -source = "git+https://github.com/datafusion-contrib/datafusion-postgres.git?rev=ffb14a52b3d7c7489559812e1ec38096f225ce93#ffb14a52b3d7c7489559812e1ec38096f225ce93" +version = "0.16.0" +source = "git+https://github.com/datafusion-contrib/datafusion-postgres.git?rev=3ae0a6cb30b1c5131687f4637247ff4f84ac11e4#3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -920,7 +920,7 @@ checksum = "37672978ae0febce7516ae0a85b53e6185159a9a28787391eb63fc44ec36037d" dependencies = [ "async-fs", "futures-lite", - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -974,6 +974,7 @@ dependencies = [ "hmac 0.12.1", "pbkdf2", "rand 0.9.4", + "serde", "sha1 0.10.6", "sha2 0.10.9", "snafu 0.8.6", @@ -1002,9 +1003,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "aws-lc-rs" -version = "1.17.0" +version = "1.18.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00" +checksum = "faac5829c2b74c28f830747e7818ccfb684261b5f48a1118b1e2a13d36dfab13" dependencies = [ "aws-lc-sys", "untrusted 0.7.1", @@ -1013,14 +1014,15 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.41.0" +version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4" +checksum = "f1622d8446a2d4b2ce0c7eefc73dd43a99779028d5ee5c2dd8073a658ba8a2bc" dependencies = [ "cc", "cmake", "dunce", "fs_extra", + "pkg-config", ] [[package]] @@ -1200,6 +1202,12 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d27c3610c36aee21ce8ac510e6224498de4228ad772a171ed65643a24693a5a8" +[[package]] +name = "base16ct" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd307490d624467aa6f74b0eabb77633d1f758a7b25f12bceb0b22e08d9726f6" + [[package]] name = "base62" version = "2.2.1" @@ -1672,7 +1680,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a1d8af896b707212cd0e99c112a78c9497dd32994192a463ed2f7419d29bd8c6" dependencies = [ "serde", - "thiserror 2.0.17", + "thiserror 2.0.21", "toml 0.8.23", ] @@ -2259,7 +2267,7 @@ version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0fa961b519f0b462e3a3b4a34b64d119eeaca1d59af726fe450bbba07a9fc0a1" dependencies = [ - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -2341,6 +2349,19 @@ dependencies = [ "tokio", ] +[[package]] +name = "common-cache" +version = "1.4.0-alpha.0" +dependencies = [ + "async-trait", + "bytes", + "common-base", + "futures", + "moka", + "prometheus 0.14.0", + "tokio", +] + [[package]] name = "common-catalog" version = "1.4.0-alpha.0" @@ -2489,6 +2510,7 @@ dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", "async-trait", + "auth", "axum 0.8.4", "bincode", "catalog", @@ -2679,6 +2701,7 @@ dependencies = [ "common-wal", "common-workload", "datafusion-common", + "datafusion-expr", "datatypes", "deadpool", "deadpool-postgres", @@ -2706,7 +2729,7 @@ dependencies = [ "serde_json", "serde_with", "snafu 0.8.6", - "sqlx", + "sqlx 0.9.0", "store-api", "strum 0.27.1", "table", @@ -3462,6 +3485,20 @@ version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" +[[package]] +name = "crypto-bigint" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a52aa3fcda4e6302a9f48734f234d35d4721b96f8fe07d073f07ce9df4f0271" +dependencies = [ + "cpubits", + "ctutils", + "num-traits", + "rand_core 0.10.0", + "serdect", + "zeroize", +] + [[package]] name = "crypto-common" version = "0.1.6" @@ -3482,6 +3519,16 @@ dependencies = [ "hybrid-array", ] +[[package]] +name = "crypto-primes" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3633a51a39c69ebbaa4feaa694bd83d241e4093901c84a0963b19d9bb3f0cf8f" +dependencies = [ + "crypto-bigint", + "rand_core 0.10.0", +] + [[package]] name = "crypto_secretbox" version = "0.1.1" @@ -3732,7 +3779,7 @@ checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" [[package]] name = "datafusion" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -3784,7 +3831,7 @@ dependencies = [ [[package]] name = "datafusion-catalog" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "async-trait", @@ -3808,7 +3855,7 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "async-trait", @@ -3831,7 +3878,7 @@ dependencies = [ [[package]] name = "datafusion-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-ipc 59.2.0", @@ -3857,7 +3904,7 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "futures", "log", @@ -3867,7 +3914,7 @@ dependencies = [ [[package]] name = "datafusion-datasource" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "async-compression", @@ -3903,7 +3950,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-ipc 59.2.0", @@ -3927,7 +3974,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "async-trait", @@ -3950,7 +3997,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "async-trait", @@ -3973,7 +4020,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -4005,12 +4052,12 @@ dependencies = [ [[package]] name = "datafusion-doc" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" [[package]] name = "datafusion-execution" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-buffer 59.2.0", @@ -4035,7 +4082,7 @@ dependencies = [ [[package]] name = "datafusion-expr" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -4059,7 +4106,7 @@ dependencies = [ [[package]] name = "datafusion-expr-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4070,7 +4117,7 @@ dependencies = [ [[package]] name = "datafusion-functions" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-buffer 59.2.0", @@ -4101,7 +4148,7 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4121,7 +4168,7 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4132,7 +4179,7 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-ord 59.2.0", @@ -4156,7 +4203,7 @@ dependencies = [ [[package]] name = "datafusion-functions-table" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "async-trait", @@ -4171,7 +4218,7 @@ dependencies = [ [[package]] name = "datafusion-functions-window" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4187,7 +4234,7 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -4196,7 +4243,7 @@ dependencies = [ [[package]] name = "datafusion-macros" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "datafusion-doc", "quote", @@ -4206,7 +4253,7 @@ dependencies = [ [[package]] name = "datafusion-optimizer" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "chrono", @@ -4239,8 +4286,8 @@ dependencies = [ [[package]] name = "datafusion-pg-catalog" -version = "0.18.3" -source = "git+https://github.com/datafusion-contrib/datafusion-postgres.git?rev=ffb14a52b3d7c7489559812e1ec38096f225ce93#ffb14a52b3d7c7489559812e1ec38096f225ce93" +version = "0.19.0" +source = "git+https://github.com/datafusion-contrib/datafusion-postgres.git?rev=3ae0a6cb30b1c5131687f4637247ff4f84ac11e4#3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" dependencies = [ "arrow-pg", "async-trait", @@ -4254,7 +4301,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -4277,7 +4324,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4291,7 +4338,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "chrono", @@ -4308,7 +4355,7 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4327,7 +4374,7 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "arrow-data 59.2.0", @@ -4364,7 +4411,7 @@ dependencies = [ [[package]] name = "datafusion-proto" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-catalog", @@ -4390,7 +4437,7 @@ dependencies = [ [[package]] name = "datafusion-proto-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4400,7 +4447,7 @@ dependencies = [ [[package]] name = "datafusion-proto-models" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "datafusion-common", "datafusion-proto-common", @@ -4410,7 +4457,7 @@ dependencies = [ [[package]] name = "datafusion-pruning" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4425,7 +4472,7 @@ dependencies = [ [[package]] name = "datafusion-session" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow-schema 59.2.0", "async-trait", @@ -4439,7 +4486,7 @@ dependencies = [ [[package]] name = "datafusion-sql" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "arrow 59.2.0", "bigdecimal 0.4.8", @@ -4458,7 +4505,7 @@ dependencies = [ [[package]] name = "datafusion-substrait" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=e3221b8f74b15c7192621576e5d1e354a6cf097a#e3221b8f74b15c7192621576e5d1e354a6cf097a" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" dependencies = [ "async-recursion", "async-trait", @@ -4480,6 +4527,7 @@ version = "1.4.0-alpha.0" dependencies = [ "api", "arrow-flight", + "async-stream", "async-trait", "bytes", "cache", @@ -4529,6 +4577,8 @@ dependencies = [ "prometheus 0.14.0", "prost 0.14.1", "query", + "rand 0.9.4", + "relational-engine", "reqwest 0.13.4", "serde", "serde_json", @@ -4574,7 +4624,6 @@ dependencies = [ "parquet-variant-compute", "parquet-variant-json", "paste", - "regex", "serde", "serde_json", "snafu 0.8.6", @@ -4643,7 +4692,18 @@ dependencies = [ "const-oid 0.9.6", "der_derive", "flagset", - "pem-rfc7468", + "pem-rfc7468 0.7.0", + "zeroize", +] + +[[package]] +name = "der" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a878c850e9e421b20262e9b41f9c860e4785fa07541c266b62ff9d1ef998a80a" +dependencies = [ + "const-oid 0.10.2", + "pem-rfc7468 1.0.0", "zeroize", ] @@ -4981,7 +5041,7 @@ dependencies = [ "chrono", "rust_decimal", "serde", - "thiserror 2.0.17", + "thiserror 2.0.21", "time", "winnow 0.6.26", ] @@ -5177,6 +5237,16 @@ dependencies = [ "windows-sys 0.48.0", ] +[[package]] +name = "etcetera" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de48cc4d1c1d97a20fd819def54b890cadde72ed3ad0c614822a0a433361be96" +dependencies = [ + "cfg-if", + "windows-sys 0.61.2", +] + [[package]] name = "ethnum" version = "1.5.3" @@ -5519,6 +5589,17 @@ dependencies = [ "spin 0.9.8", ] +[[package]] +name = "flume" +version = "0.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e139bc46ca777eb5efaf62df0ab8cc5fd400866427e56c68b22e414e53bd3be" +dependencies = [ + "futures-core", + "futures-sink", + "spin 0.9.8", +] + [[package]] name = "fnv" version = "1.0.7" @@ -5612,7 +5693,6 @@ dependencies = [ "common-test-util", "common-time", "common-version", - "dashmap", "datafusion", "datafusion-expr", "datatypes", @@ -5785,9 +5865,9 @@ dependencies = [ [[package]] name = "futures-channel" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10" +checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4" dependencies = [ "futures-core", "futures-sink", @@ -5795,15 +5875,15 @@ dependencies = [ [[package]] name = "futures-core" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e" +checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" [[package]] name = "futures-executor" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e28d1d997f585e54aebc3f97d39e72338912123a67330d723fdbb564d646c9f" +checksum = "031b47cf1a3c6cc8bc2fc76cd437f521619387907d469316e7c0bc278f1f5432" dependencies = [ "futures-core", "futures-task", @@ -5823,9 +5903,9 @@ dependencies = [ [[package]] name = "futures-io" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6" +checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed" [[package]] name = "futures-lite" @@ -5842,26 +5922,26 @@ dependencies = [ [[package]] name = "futures-macro" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" +checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 3.0.5", ] [[package]] name = "futures-sink" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7" +checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d" [[package]] name = "futures-task" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" [[package]] name = "futures-timer" @@ -5871,9 +5951,9 @@ checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24" [[package]] name = "futures-util" -version = "0.3.31" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" dependencies = [ "futures-channel", "futures-core", @@ -5883,7 +5963,6 @@ dependencies = [ "futures-task", "memchr", "pin-project-lite", - "pin-utils", "slab", ] @@ -6101,7 +6180,7 @@ dependencies = [ [[package]] name = "greptime-proto" version = "0.1.0" -source = "git+https://github.com/GreptimeTeam/greptime-proto.git?rev=f51cb5312bf7feaf1199361ce16edc63a0ad40b2#f51cb5312bf7feaf1199361ce16edc63a0ad40b2" +source = "git+https://github.com/discord9/greptime-proto.git?rev=b9159cd680ac2b88d155cbc9effd6df9694e8597#b9159cd680ac2b88d155cbc9effd6df9694e8597" dependencies = [ "prost 0.14.1", "prost-types 0.14.1", @@ -6194,11 +6273,11 @@ dependencies = [ [[package]] name = "halfbrown" -version = "0.3.0" +version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa2c385c6df70fd180bbb673d93039dbd2cd34e41d782600bdf6e1ca7bce39aa" +checksum = "0c7ed2f2edad8a14c8186b847909a41fbb9c3eafa44f88bd891114ed5019da09" dependencies = [ - "hashbrown 0.15.4", + "hashbrown 0.16.1", "serde", ] @@ -6272,6 +6351,15 @@ dependencies = [ "hashbrown 0.15.4", ] +[[package]] +name = "hashlink" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "824e001ac4f3012dd16a264bec811403a67ca9deb6c102fc5049b32c4574b35f" +dependencies = [ + "hashbrown 0.16.1", +] + [[package]] name = "hdfs-native" version = "0.14.6" @@ -6306,7 +6394,7 @@ dependencies = [ "regex", "roxmltree", "socket2 0.6.4", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "url", "uuid", @@ -6399,6 +6487,15 @@ dependencies = [ "hmac 0.12.1", ] +[[package]] +name = "hkdf" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4aaa26c720c68b866f2c96ef5c1264b3e6f473fe5d4ce61cd44bbe913e553018" +dependencies = [ + "hmac 0.13.0", +] + [[package]] name = "hmac" version = "0.12.1" @@ -6649,7 +6746,7 @@ dependencies = [ "tokio", "tokio-rustls", "tower-service", - "webpki-roots 1.0.9", + "webpki-roots 1.0.10", ] [[package]] @@ -6971,6 +7068,7 @@ dependencies = [ "common-telemetry", "common-test-util", "criterion 0.7.0", + "datatypes", "fastbloom", "fst", "futures", @@ -7423,7 +7521,7 @@ dependencies = [ "jni-sys", "log", "simd_cesu8", - "thiserror 2.0.17", + "thiserror 2.0.21", "walkdir", "windows-link 0.2.1", ] @@ -7508,7 +7606,7 @@ dependencies = [ [[package]] name = "jsonb" version = "0.5.6" -source = "git+https://github.com/GreptimeTeam/jsonb?rev=dd1e2341e3df35e9de38959328caa7628e81b6b1#dd1e2341e3df35e9de38959328caa7628e81b6b1" +source = "git+https://github.com/GreptimeTeam/jsonb?rev=5a6472858d860ac034075c17253f41617bdfa956#5a6472858d860ac034075c17253f41617bdfa956" dependencies = [ "byteorder", "ethnum", @@ -7550,7 +7648,7 @@ dependencies = [ "pest_derive", "regex", "serde_json", - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -8078,7 +8176,7 @@ checksum = "656b3b27f8893f7bbf9485148ff9a65f019e3f33bd5cdc87c83cab16b3fd9ec8" dependencies = [ "libc", "neli", - "thiserror 2.0.17", + "thiserror 2.0.21", "windows-sys 0.59.0", ] @@ -8119,6 +8217,7 @@ dependencies = [ "bytes", "chrono", "common-base", + "common-cache", "common-config", "common-error", "common-macro", @@ -8314,7 +8413,7 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bb4bdc8b0ce69932332cf76d24af69c3a155242af95c226b2ab6c2e371ed1149" dependencies = [ - "thiserror 2.0.17", + "thiserror 2.0.21", "zerocopy", "zerocopy-derive", ] @@ -8546,7 +8645,7 @@ dependencies = [ "serde_json", "servers", "snafu 0.8.6", - "sqlx", + "sqlx 0.9.0", "store-api", "strum 0.27.1", "table", @@ -8613,6 +8712,7 @@ dependencies = [ "humantime-serde", "itertools 0.14.0", "lazy_static", + "log-store", "mito-codec", "mito2", "moka", @@ -8723,6 +8823,7 @@ dependencies = [ "bytes", "chrono", "common-base", + "common-cache", "common-datasource", "common-error", "common-function", @@ -8898,7 +8999,7 @@ dependencies = [ "twox-hash", "url", "webpki", - "webpki-roots 1.0.9", + "webpki-roots 1.0.10", ] [[package]] @@ -8934,7 +9035,7 @@ dependencies = [ "quote", "syn 2.0.117", "termcolor", - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -8958,13 +9059,13 @@ dependencies = [ "rustls", "serde", "socket2 0.6.4", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tokio-rustls", "tokio-util", "twox-hash", "url", - "webpki-roots 1.0.9", + "webpki-roots 1.0.10", ] [[package]] @@ -9030,7 +9131,7 @@ dependencies = [ "serde_json", "sha1 0.10.6", "sha2 0.10.9", - "thiserror 2.0.17", + "thiserror 2.0.21", "time", "uuid", ] @@ -9059,7 +9160,7 @@ dependencies = [ "serde_json", "sha1 0.11.0", "sha2 0.11.0", - "thiserror 2.0.17", + "thiserror 2.0.21", "uuid", ] @@ -9601,7 +9702,7 @@ dependencies = [ "itertools 0.14.0", "parking_lot 0.12.4", "percent-encoding", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tracing", "url", @@ -9942,7 +10043,7 @@ dependencies = [ "asyncband", "opendal-core", "serde", - "sqlx", + "sqlx 0.8.6", ] [[package]] @@ -10015,7 +10116,7 @@ dependencies = [ "futures-sink", "js-sys", "pin-project-lite", - "thiserror 2.0.17", + "thiserror 2.0.21", "tracing", ] @@ -10045,7 +10146,7 @@ dependencies = [ "opentelemetry_sdk", "prost 0.14.1", "reqwest 0.13.4", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tonic 0.14.2", "tonic-types", @@ -10092,7 +10193,7 @@ dependencies = [ "percent-encoding", "portable-atomic", "rand 0.9.4", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tokio-stream", ] @@ -10104,6 +10205,7 @@ dependencies = [ "ahash 0.8.12", "api", "arrow 59.2.0", + "arrow-schema 59.2.0", "async-stream", "async-trait", "axum 0.8.4", @@ -10507,9 +10609,11 @@ dependencies = [ "common-error", "common-macro", "common-meta", + "common-time", "criterion 0.7.0", "datafusion-common", "datafusion-expr", + "datafusion-optimizer", "datafusion-physical-expr", "datatypes", "moka", @@ -10596,7 +10700,7 @@ checksum = "500fa4cdeacd98997c5865e3d0d1cb8fe7e9d7d75ecc775e07989a433a9a9a59" dependencies = [ "arrayvec", "bitflags 2.12.1", - "thiserror 2.0.17", + "thiserror 2.0.21", "zerocopy", "zerocopy-derive", ] @@ -10626,6 +10730,15 @@ dependencies = [ "base64ct", ] +[[package]] +name = "pem-rfc7468" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6305423e0e7738146434843d1694d621cce767262b2a86910beab705e4493d9" +dependencies = [ + "base64ct", +] + [[package]] name = "percent-encoding" version = "2.3.2" @@ -10645,7 +10758,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1db05f56d34358a8b1066f67cbb203ee3e7ed2ba674a6263a1d5ec6db2204323" dependencies = [ "memchr", - "thiserror 2.0.17", + "thiserror 2.0.21", "ucd-trie", ] @@ -10727,9 +10840,9 @@ dependencies = [ [[package]] name = "pgwire" -version = "0.41.0" +version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c667f8f4046889d52ed11e17aae954b1816d3df48dd23bdebcf199cef332df93" +checksum = "69354d96ef951739b5ccc75a99a8a1b09bc164afde265be9656ddab87395b481" dependencies = [ "async-trait", "aws-lc-rs", @@ -10751,7 +10864,7 @@ dependencies = [ "serde_json", "smol_str", "stringprep", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tokio-rustls", "tokio-util", @@ -10920,6 +11033,8 @@ dependencies = [ "snafu 0.8.6", "sql", "table", + "temp-env", + "tempfile", "tokio", "urlencoding", "vrl", @@ -10943,9 +11058,19 @@ version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8ffb9f10fa047879315e6625af03c164b16962a5368d724ed16323b68ace47f" dependencies = [ - "der", - "pkcs8", - "spki", + "der 0.7.10", + "pkcs8 0.10.2", + "spki 0.7.3", +] + +[[package]] +name = "pkcs1" +version = "0.8.0-rc.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f485dbafe293ea6ccaa7d82bc591651a796eb8a9aec3df040b057b706073dbc3" +dependencies = [ + "der 0.8.2", + "spki 0.8.1", ] [[package]] @@ -10956,11 +11081,11 @@ checksum = "e847e2c91a18bfa887dd028ec33f2fe6f25db77db3619024764914affe8b69a6" dependencies = [ "aes 0.8.4", "cbc 0.1.2", - "der", + "der 0.7.10", "pbkdf2", "scrypt", "sha2 0.10.9", - "spki", + "spki 0.7.3", ] [[package]] @@ -10969,10 +11094,20 @@ version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" dependencies = [ - "der", + "der 0.7.10", "pkcs5", "rand_core 0.6.4", - "spki", + "spki 0.7.3", +] + +[[package]] +name = "pkcs8" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "451913da69c775a56034ea8d9003d27ee8948e12443eae7c038ba100a4f21cb7" +dependencies = [ + "der 0.8.2", + "spki 0.8.1", ] [[package]] @@ -11140,8 +11275,7 @@ checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" [[package]] name = "pprof" version = "0.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38a01da47675efa7673b032bf8efd8214f1917d89685e07e395ab125ea42b187" +source = "git+https://github.com/tikv/pprof-rs?rev=01cff82dbe6fe110a707bf2b38d8ebb1d14a18f8#01cff82dbe6fe110a707bf2b38d8ebb1d14a18f8" dependencies = [ "aligned-vec", "backtrace", @@ -11165,7 +11299,7 @@ dependencies = [ "spin 0.10.1", "symbolic-demangle", "tempfile", - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -11368,7 +11502,7 @@ dependencies = [ "parking_lot 0.12.4", "procfs", "protobuf 3.7.2", - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -11823,6 +11957,7 @@ dependencies = [ "bitflags 2.12.1", "bytes", "common-base", + "common-cache", "common-error", "common-macro", "common-runtime", @@ -11840,6 +11975,7 @@ dependencies = [ "snafu 0.8.6", "tokio", "tokio-util", + "tracing", "uuid", ] @@ -11875,6 +12011,7 @@ dependencies = [ "async-stream", "async-trait", "bytes", + "cache", "catalog", "chrono", "common-base", @@ -11885,6 +12022,7 @@ dependencies = [ "common-macro", "common-meta", "common-plugins", + "common-procedure", "common-query", "common-recordbatch", "common-runtime", @@ -11982,7 +12120,7 @@ dependencies = [ "rustc-hash", "rustls", "socket2 0.5.10", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tracing", "web-time", @@ -12005,7 +12143,7 @@ dependencies = [ "rustls", "rustls-pki-types", "slab", - "thiserror 2.0.17", + "thiserror 2.0.21", "tinyvec", "tracing", "web-time", @@ -12376,6 +12514,28 @@ dependencies = [ "memchr", ] +[[package]] +name = "relational-engine" +version = "1.4.0-alpha.0" +dependencies = [ + "api", + "async-trait", + "common-catalog", + "common-error", + "common-macro", + "common-telemetry", + "common-test-util", + "common-time", + "datatypes", + "object-store", + "serde", + "serde_json", + "snafu 0.8.6", + "store-api", + "table", + "tokio", +] + [[package]] name = "relative-path" version = "1.9.3" @@ -12474,7 +12634,7 @@ dependencies = [ "pem", "percent-encoding", "reqsign-core", - "rsa", + "rsa 0.9.10", "serde", "serde_json", "sha1 0.11.0", @@ -12497,7 +12657,7 @@ dependencies = [ "log", "mea", "percent-encoding", - "rsa", + "rsa 0.9.10", "serde", "serde_json", "sha1 0.11.0", @@ -12528,7 +12688,7 @@ dependencies = [ "percent-encoding", "reqsign-aws-v4", "reqsign-core", - "rsa", + "rsa 0.9.10", "serde", "serde_json", "tokio", @@ -12574,7 +12734,7 @@ dependencies = [ "wasm-bindgen-futures", "wasm-streams 0.4.2", "web-sys", - "webpki-roots 1.0.9", + "webpki-roots 1.0.10", ] [[package]] @@ -12650,7 +12810,7 @@ dependencies = [ "reqwest 0.12.24", "reqwest-middleware", "retry-policies", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "wasmtimer", ] @@ -12769,16 +12929,34 @@ dependencies = [ "num-bigint-dig", "num-integer", "num-traits", - "pkcs1", - "pkcs8", + "pkcs1 0.7.5", + "pkcs8 0.10.2", "rand_core 0.6.4", "sha2 0.10.9", - "signature", - "spki", + "signature 2.2.0", + "spki 0.7.3", "subtle", "zeroize", ] +[[package]] +name = "rsa" +version = "0.10.0-rc.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33405e321e16ef941ca66e254f6904c3aafd3c7ff001fb88bf11c73eb75c2b85" +dependencies = [ + "const-oid 0.10.2", + "crypto-bigint", + "crypto-primes", + "digest 0.11.3", + "pkcs1 0.8.0-rc.5", + "pkcs8 0.11.0", + "rand_core 0.10.0", + "signature 3.0.0", + "spki 0.8.1", + "zeroize", +] + [[package]] name = "rsasl" version = "2.2.1" @@ -12795,7 +12973,7 @@ dependencies = [ "serde_json", "sha2 0.10.9", "stringprep", - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -12815,7 +12993,7 @@ dependencies = [ "rsasl", "rustls", "snap", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tokio-rustls", "tracing", @@ -13016,9 +13194,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.28" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7160e3e10bf4535308537f3c4e1641468cd0e485175d6163087c0393c7d46643" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "aws-lc-rs", "log", @@ -13103,9 +13281,9 @@ checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" [[package]] name = "rustls-webpki" -version = "0.103.13" +version = "0.103.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2" dependencies = [ "aws-lc-rs", "ring", @@ -13292,7 +13470,7 @@ dependencies = [ "proc-macro2", "quote", "syn 2.0.117", - "thiserror 2.0.17", + "thiserror 2.0.21", ] [[package]] @@ -13545,6 +13723,16 @@ dependencies = [ "unsafe-libyaml", ] +[[package]] +name = "serdect" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66cf8fedced2fcf12406bcb34223dffb92eaf34908ede12fed414c82b7f00b3e" +dependencies = [ + "base16ct", + "serde", +] + [[package]] name = "servers" version = "1.4.0-alpha.0" @@ -13711,6 +13899,7 @@ dependencies = [ "prost 0.14.1", "snafu 0.8.6", "sql", + "tokio", "uuid", ] @@ -13819,6 +14008,16 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "signature" +version = "3.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28d567dcbaf0049cb8ac2608a76cd95ff9e4412e1899d389ee400918ca7537f5" +dependencies = [ + "digest 0.11.3", + "rand_core 0.10.0", +] + [[package]] name = "simba" version = "0.9.0" @@ -13840,13 +14039,11 @@ checksum = "d66dc143e6b11c1eddc06d5c423cfc97062865baf299914ab64caa38182078fe" [[package]] name = "simd-json" -version = "0.15.1" +version = "0.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c962f626b54771990066e5435ec8331d1462576cd2d1e62f24076ae014f92112" +checksum = "e8f2e4d523646f7da6dde3983bd18ede0babc21c258ab708b3dd93ea0259719d" dependencies = [ - "getrandom 0.3.4", "halfbrown", - "ref-cast", "serde", "serde_json", "simdutf8", @@ -14047,7 +14244,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" dependencies = [ "base64ct", - "der", + "der 0.7.10", +] + +[[package]] +name = "spki" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ef958a98b9d5da290cfc78946e9f3e61e1e62a18db0d92cac0b83cc161491a9" +dependencies = [ + "base64ct", + "der 0.8.2", ] [[package]] @@ -14174,11 +14381,24 @@ version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fefb893899429669dcdd979aff487bd78f4064e5e7907e4269081e0ef7d97dc" dependencies = [ - "sqlx-core", - "sqlx-macros", - "sqlx-mysql", - "sqlx-postgres", - "sqlx-sqlite", + "sqlx-core 0.8.6", + "sqlx-macros 0.8.6", + "sqlx-mysql 0.8.6", + "sqlx-postgres 0.8.6", + "sqlx-sqlite 0.8.6", +] + +[[package]] +name = "sqlx" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "378620ccc25c62c89d8be1c819e76a88d59bdcc3304733330788948e619bfd71" +dependencies = [ + "sqlx-core 0.9.0", + "sqlx-macros 0.9.0", + "sqlx-mysql 0.9.0", + "sqlx-postgres 0.9.0", + "sqlx-sqlite 0.9.0", ] [[package]] @@ -14189,7 +14409,6 @@ checksum = "ee6798b1838b6a0f69c007c133b8df5866302197e404e8b6ee8ed3e3a5e68dc6" dependencies = [ "base64 0.22.1", "bytes", - "chrono", "crc", "crossbeam-queue", "either", @@ -14199,24 +14418,61 @@ dependencies = [ "futures-io", "futures-util", "hashbrown 0.15.4", - "hashlink", + "hashlink 0.10.0", "indexmap 2.14.2", "log", "memchr", "once_cell", "percent-encoding", + "rustls", + "serde", + "serde_json", + "sha2 0.10.9", + "smallvec", + "thiserror 2.0.21", + "tokio", + "tokio-stream", + "tracing", + "url", + "webpki-roots 0.26.11", +] + +[[package]] +name = "sqlx-core" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05b44e85bf579a8eeb4ceaa77a3a523baf2bf0e9bac7e40f405d537b5d2d5ccb" +dependencies = [ + "base64 0.22.1", + "bytes", + "cfg-if", + "chrono", + "crc", + "crossbeam-queue", + "either", + "event-listener 5.4.0", + "futures-core", + "futures-intrusive", + "futures-io", + "futures-util", + "hashbrown 0.16.1", + "hashlink 0.11.1", + "indexmap 2.14.2", + "log", + "memchr", + "percent-encoding", "rust_decimal", "rustls", "serde", "serde_json", "sha2 0.10.9", "smallvec", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tokio-stream", "tracing", "url", - "webpki-roots 0.26.11", + "webpki-roots 1.0.10", ] [[package]] @@ -14227,8 +14483,21 @@ checksum = "a2d452988ccaacfbf5e0bdbc348fb91d7c8af5bee192173ac3636b5fb6e6715d" dependencies = [ "proc-macro2", "quote", - "sqlx-core", - "sqlx-macros-core", + "sqlx-core 0.8.6", + "sqlx-macros-core 0.8.6", + "syn 2.0.117", +] + +[[package]] +name = "sqlx-macros" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bd2b84f2bc39a5705ef27ec785a11c934a41bbd4a24941e257927cddc26b60bf" +dependencies = [ + "proc-macro2", + "quote", + "sqlx-core 0.9.0", + "sqlx-macros-core 0.9.0", "syn 2.0.117", ] @@ -14248,15 +14517,41 @@ dependencies = [ "serde", "serde_json", "sha2 0.10.9", - "sqlx-core", - "sqlx-mysql", - "sqlx-postgres", - "sqlx-sqlite", + "sqlx-core 0.8.6", + "sqlx-mysql 0.8.6", + "sqlx-postgres 0.8.6", + "sqlx-sqlite 0.8.6", "syn 2.0.117", "tokio", "url", ] +[[package]] +name = "sqlx-macros-core" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb8d96de5fdc85a5c4ec813432b523ec637e80ba98f046555f75f7908ddac7c3" +dependencies = [ + "cfg-if", + "dotenvy", + "either", + "heck 0.5.0", + "hex", + "proc-macro2", + "quote", + "serde", + "serde_json", + "sha2 0.10.9", + "sqlx-core 0.9.0", + "sqlx-mysql 0.9.0", + "sqlx-postgres 0.9.0", + "sqlx-sqlite 0.9.0", + "syn 2.0.117", + "thiserror 2.0.21", + "tokio", + "url", +] + [[package]] name = "sqlx-mysql" version = "0.8.6" @@ -14268,7 +14563,6 @@ dependencies = [ "bitflags 2.12.1", "byteorder", "bytes", - "chrono", "crc", "digest 0.10.7", "dotenvy", @@ -14279,7 +14573,7 @@ dependencies = [ "futures-util", "generic-array", "hex", - "hkdf", + "hkdf 0.12.4", "hmac 0.12.1", "itoa", "log", @@ -14288,19 +14582,48 @@ dependencies = [ "once_cell", "percent-encoding", "rand 0.8.5", - "rsa", - "rust_decimal", + "rsa 0.9.10", "serde", "sha1 0.10.6", "sha2 0.10.9", "smallvec", - "sqlx-core", + "sqlx-core 0.8.6", "stringprep", - "thiserror 2.0.17", + "thiserror 2.0.21", "tracing", "whoami 1.6.0", ] +[[package]] +name = "sqlx-mysql" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90b8020fe17c5f2c245bfa2505d7ef59c5604839527c740266ad2214acebea27" +dependencies = [ + "bitflags 2.12.1", + "byteorder", + "bytes", + "chrono", + "crc", + "digest 0.11.3", + "dotenvy", + "either", + "futures-core", + "futures-util", + "generic-array", + "log", + "percent-encoding", + "rand 0.10.1", + "rsa 0.10.0-rc.19", + "rust_decimal", + "serde", + "sha1 0.11.0", + "sha2 0.11.0", + "sqlx-core 0.9.0", + "thiserror 2.0.21", + "tracing", +] + [[package]] name = "sqlx-postgres" version = "0.8.6" @@ -14311,15 +14634,14 @@ dependencies = [ "base64 0.22.1", "bitflags 2.12.1", "byteorder", - "chrono", "crc", "dotenvy", - "etcetera", + "etcetera 0.8.0", "futures-channel", "futures-core", "futures-util", "hex", - "hkdf", + "hkdf 0.12.4", "hmac 0.12.1", "home", "itoa", @@ -14328,18 +14650,54 @@ dependencies = [ "memchr", "once_cell", "rand 0.8.5", - "rust_decimal", "serde", "serde_json", "sha2 0.10.9", "smallvec", - "sqlx-core", + "sqlx-core 0.8.6", "stringprep", - "thiserror 2.0.17", + "thiserror 2.0.21", "tracing", "whoami 1.6.0", ] +[[package]] +name = "sqlx-postgres" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87a2bdd6e83f6b3ea525ca9fee568030508b58355a43d0b2c1674d5f79dcd65e" +dependencies = [ + "atoi", + "base64 0.22.1", + "bitflags 2.12.1", + "byteorder", + "chrono", + "crc", + "dotenvy", + "etcetera 0.11.0", + "futures-channel", + "futures-core", + "futures-util", + "hex", + "hkdf 0.13.0", + "hmac 0.13.0", + "itoa", + "log", + "md-5 0.11.0", + "memchr", + "rand 0.10.1", + "rust_decimal", + "serde", + "serde_json", + "sha2 0.11.0", + "smallvec", + "sqlx-core 0.9.0", + "stringprep", + "thiserror 2.0.21", + "tracing", + "whoami 2.1.3", +] + [[package]] name = "sqlx-sqlite" version = "0.8.6" @@ -14347,8 +14705,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2d12fe70b2c1b4401038055f90f151b78208de1f9f89a7dbfd41587a10c3eea" dependencies = [ "atoi", - "chrono", - "flume", + "flume 0.11.1", "futures-channel", "futures-core", "futures-executor", @@ -14359,8 +14716,33 @@ dependencies = [ "percent-encoding", "serde", "serde_urlencoded", - "sqlx-core", - "thiserror 2.0.17", + "sqlx-core 0.8.6", + "thiserror 2.0.21", + "tracing", + "url", +] + +[[package]] +name = "sqlx-sqlite" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "488e99c397a62007e4229aec669a179816339afc6d2620ca6fa420dbee2e982c" +dependencies = [ + "atoi", + "chrono", + "flume 0.12.0", + "form_urlencoded", + "futures-channel", + "futures-core", + "futures-executor", + "futures-intrusive", + "futures-util", + "libsqlite3-sys", + "log", + "percent-encoding", + "serde", + "sqlx-core 0.9.0", + "thiserror 2.0.21", "tracing", "url", ] @@ -14390,6 +14772,7 @@ version = "1.4.0-alpha.0" dependencies = [ "api", "async-trait", + "auth", "catalog", "client", "common-base", @@ -14413,6 +14796,7 @@ dependencies = [ "flow", "frontend", "hostname 0.4.1", + "humantime-serde", "log-store", "mito2", "pipeline", @@ -14453,6 +14837,7 @@ dependencies = [ "common-time", "common-wal", "datafusion-common", + "datafusion-execution", "datafusion-expr", "datafusion-physical-plan", "datatypes", @@ -14854,7 +15239,7 @@ dependencies = [ "tantivy-stacker", "tantivy-tokenizer-api", "tempfile", - "thiserror 2.0.17", + "thiserror 2.0.21", "time", "typetag", "uuid", @@ -15084,7 +15469,7 @@ dependencies = [ "snafu 0.8.6", "sql", "sqlparser", - "sqlx", + "sqlx 0.9.0", "store-api", "strum 0.27.1", "tinytemplate", @@ -15101,6 +15486,7 @@ dependencies = [ "auth", "axum 0.8.4", "base64 0.22.1", + "bytes", "cache", "catalog", "chrono", @@ -15170,7 +15556,7 @@ dependencies = [ "session", "similar-asserts", "sql", - "sqlx", + "sqlx 0.9.0", "standalone", "store-api", "substrait 1.4.0-alpha.0", @@ -15202,11 +15588,11 @@ dependencies = [ [[package]] name = "thiserror" -version = "2.0.17" +version = "2.0.21" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f63587ca0f12b72a0600bcba1d40081f830876000bb46dd2337a3051618f4fc8" +checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e" dependencies = [ - "thiserror-impl 2.0.17", + "thiserror-impl 2.0.21", ] [[package]] @@ -15222,13 +15608,13 @@ dependencies = [ [[package]] name = "thiserror-impl" -version = "2.0.17" +version = "2.0.21" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" +checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 3.0.5", ] [[package]] @@ -15860,7 +16246,7 @@ checksum = "050686193eb999b4bb3bc2acfa891a13da00f79734704c4b8b4ef1a10b368a3c" dependencies = [ "crossbeam-channel", "symlink", - "thiserror 2.0.17", + "thiserror 2.0.21", "time", "tracing-subscriber", ] @@ -16030,7 +16416,7 @@ dependencies = [ "serde", "serde_json", "syn 2.0.117", - "thiserror 2.0.17", + "thiserror 2.0.21", "unicode-ident", ] @@ -16074,7 +16460,7 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f0495d3e02c0e9b074e5631de7af419b9f4fdbfec4b1ca1b3180c8b3a926cb5" dependencies = [ - "thiserror 2.0.17", + "thiserror 2.0.21", "vu128", ] @@ -16257,9 +16643,9 @@ checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" [[package]] name = "value-trait" -version = "0.11.0" +version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0508fce11ad19e0aab49ce20b6bec7f8f82902ded31df1c9fc61b90f0eb396b8" +checksum = "545a3d7089f11f1741d12963c057adf9443fc24325110a6a6daf582e825fd51b" dependencies = [ "float-cmp", "halfbrown", @@ -16402,7 +16788,7 @@ dependencies = [ "strum_macros 0.26.4", "syslog_loose", "termcolor", - "thiserror 2.0.17", + "thiserror 2.0.21", "tokio", "tracing", "ua-parser", @@ -16695,14 +17081,14 @@ version = "0.26.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9" dependencies = [ - "webpki-roots 1.0.9", + "webpki-roots 1.0.10", ] [[package]] name = "webpki-roots" -version = "1.0.9" +version = "1.0.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7dcd9d09a39985f5344844e66b0c530a33843579125f23e21e9f0f220850f22a" +checksum = "1c1c8e20c8a38fc7d65abfe74eb7280e4aa667e8eec8bc84996d2874810748d9" dependencies = [ "rustls-pki-types", ] @@ -17359,8 +17745,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1301e935010a701ae5f8655edc0ad17c44bad3ac5ce8c39185f75453b720ae94" dependencies = [ "const-oid 0.9.6", - "der", - "spki", + "der 0.7.10", + "spki 0.7.3", "tls_codec", ] @@ -17373,13 +17759,13 @@ dependencies = [ "bcder", "bytes", "chrono", - "der", + "der 0.7.10", "hex", "pem", "ring", - "signature", - "spki", - "thiserror 2.0.17", + "signature 2.2.0", + "spki 0.7.3", + "thiserror 2.0.21", "zeroize", ] diff --git a/Cargo.toml b/Cargo.toml index f32247088ae..d3a1b6dab43 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -9,6 +9,7 @@ members = [ "src/cmd", "src/common/base", "src/common/batcher", + "src/common/cache", "src/common/catalog", "src/common/config", "src/common/datasource", @@ -63,6 +64,7 @@ members = [ "src/promql", "src/puffin", "src/query", + "src/relational-engine", "src/standalone", "src/servers", "src/session", @@ -133,6 +135,7 @@ dashmap = "6.1" datafusion = "=55.1.0" datafusion-common = "=55.1.0" datafusion-datasource = "=55.1.0" +datafusion-execution = "=55.1.0" datafusion-expr = "=55.1.0" datafusion-expr-common = "=55.1.0" datafusion-functions = "=55.1.0" @@ -140,7 +143,7 @@ datafusion-functions-aggregate-common = "=55.1.0" datafusion-functions-window-common = "=55.1.0" datafusion-optimizer = "=55.1.0" datafusion-orc = "0.10.0" -datafusion-pg-catalog = "0.18.3" +datafusion-pg-catalog = "0.19.0" datafusion-physical-expr = "=55.1.0" datafusion-physical-plan = "=55.1.0" datafusion-proto = "=55.1.0" @@ -161,7 +164,10 @@ fs2 = "0.4" fst = "0.4.7" futures = "0.3" futures-util = "0.3" -greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "f51cb5312bf7feaf1199361ce16edc63a0ad40b2" } +# TODO(narthex-merge): temporary pre-merge pin. Points at the fork branch carrying +# PR GreptimeTeam/greptime-proto#355 (RawSeriesSource on top of main's 5870f5adc). +# Replace with the GreptimeTeam rev once #355 lands. +greptime-proto = { git = "https://github.com/discord9/greptime-proto.git", rev = "b9159cd680ac2b88d155cbc9effd6df9694e8597" } hdfs-native = "0.14.6" hex = "0.4" hostname = "0.4.0" @@ -173,7 +179,7 @@ hyper-util = "0.1" icu_properties = "2.0.1" indicatif = "0.17" itertools = "0.14" -jsonb = { git = "https://github.com/GreptimeTeam/jsonb", rev = "dd1e2341e3df35e9de38959328caa7628e81b6b1", default-features = false, features = [ +jsonb = { git = "https://github.com/GreptimeTeam/jsonb", rev = "5a6472858d860ac034075c17253f41617bdfa956", default-features = false, features = [ "databend", ] } lazy_static = "1.4" @@ -238,19 +244,20 @@ rskafka = { git = "https://github.com/influxdata/rskafka.git", rev = "80f42eb15e rstest = "0.25" rstest_reuse = "0.7" rust_decimal = "1.33" -rustls = { version = "0.23.25", default-features = false } +rustls = { version = "0.23.46", default-features = false } ryu = "1.0" sea-query = "0.32" serde = { version = "1.0", features = ["derive"] } serde_json = { version = "1.0", features = ["float_roundtrip"] } serde_with = "3" serde_yaml_ng = "0.10" -simd-json = "0.15" +sha2 = "0.10" +simd-json = "0.19" similar-asserts = "1.6.0" smallvec = { version = "1", features = ["serde"] } snafu = "0.8" sqlparser = { version = "0.62.0", default-features = false, features = ["std", "visitor", "serde"] } -sqlx = { version = "0.8", default-features = false, features = [ +sqlx = { version = "0.9", default-features = false, features = [ "any", "macros", "json", @@ -301,6 +308,7 @@ client = { path = "src/client" } cmd = { path = "src/cmd", default-features = false } common-base = { path = "src/common/base" } common-batcher = { path = "src/common/batcher" } +common-cache = { path = "src/common/cache" } common-catalog = { path = "src/common/catalog" } common-config = { path = "src/common/config" } common-datasource = { path = "src/common/datasource" } @@ -354,6 +362,7 @@ plugins = { path = "src/plugins" } promql = { path = "src/promql" } puffin = { path = "src/puffin" } query = { path = "src/query" } +relational-engine = { path = "src/relational-engine" } servers = { path = "src/servers" } session = { path = "src/session" } sql = { path = "src/sql" } @@ -368,25 +377,30 @@ rev = "f743c3a5fe8c1f57363eac5866c356b7d91f9b39" [patch.crates-io] substrait = { git = "https://github.com/GreptimeTeam/substrait-rs.git", rev = "91ec978b0649417ad3da8390e7a515baec723b1b" } -datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" } -datafusion-pg-catalog = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "ffb14a52b3d7c7489559812e1ec38096f225ce93" } -arrow-pg = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "ffb14a52b3d7c7489559812e1ec38096f225ce93" } +datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-execution = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion-pg-catalog = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" } +arrow-pg = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" } sqlparser = { git = "https://github.com/GreptimeTeam/sqlparser-rs.git", rev = "9e9019bb1c7040ed956f654e39378dd42ab17884" } +# pprof 0.15.0 initializes framehop inside SIGPROF; pin the unreleased fix +# that initializes it before sampling. Remove once tikv/pprof-rs#282 is released. +pprof = { git = "https://github.com/tikv/pprof-rs", rev = "01cff82dbe6fe110a707bf2b38d8ebb1d14a18f8" } + # Temporary: use the GreptimeTeam fork of tikv-jemalloc-sys embedding # jemalloc 5.3.1 + backport of 54f22c83 ("Initialize TSD tcache before # enabling it"), fixing the TSD/tcache enable-before-init reentrancy diff --git a/README.md b/README.md index 96058cc79b7..7c9bdc01b96 100644 --- a/README.md +++ b/README.md @@ -9,13 +9,11 @@

Metrics, logs, and traces.
One engine, on your infrastructure.

-> A columnar database for metrics, logs, and traces on object storage. -> Apache-2.0 licensed core. -
User Guide  ·  API Docs  ·  +Live Demo  ·  Roadmap 2026  ·  Slack @@ -31,7 +29,7 @@ One engine, on your infrastructure. ## Introduction -**GreptimeDB** is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one [table model](https://docs.greptime.com/user-guide/concepts/data-model/): tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases. +**GreptimeDB** is an open-source [observability database](https://docs.greptime.com/user-guide/concepts/why-greptimedb/#what-an-observability-database-is). Metrics, logs, and traces run on one columnar engine over object storage and share one [table model](https://docs.greptime.com/user-guide/concepts/data-model/): tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases. Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors. @@ -61,76 +59,6 @@ LIMIT 20; Metrics join the same way, on any tag the tables share, such as `service`, `host`, or `pod`. -## Why You Might Use It - -- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three -- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface -- You are hitting Loki's query performance limits as log volume grows -- You need long retention on object storage without a separate analytics stack -- You want to query telemetry with SQL, not only a domain query language -- You are storing GenAI or agent telemetry ([OTel GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)) alongside infrastructure signals - -Learn more in [Why GreptimeDB](https://docs.greptime.com/user-guide/concepts/why-greptimedb). - -## What's Supported - -| | | -|---|---| -| **Ingest** | [OpenTelemetry (OTLP)](https://docs.greptime.com/user-guide/ingest-data/for-observability/opentelemetry/), [Prometheus Remote Write](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/), [Loki Push](https://docs.greptime.com/user-guide/ingest-data/for-observability/loki), [Elasticsearch Bulk](https://docs.greptime.com/user-guide/ingest-data/for-observability/elasticsearch), [InfluxDB line protocol](https://docs.greptime.com/user-guide/protocols/influxdb-line-protocol), [gRPC](https://docs.greptime.com/user-guide/protocols/grpc/) | -| **Query** | [SQL](https://docs.greptime.com/user-guide/query-data/overview/), [PromQL](https://docs.greptime.com/user-guide/query-data/promql/), [Jaeger-compatible trace queries](https://docs.greptime.com/user-guide/query-data/jaeger/), MySQL and PostgreSQL wire protocols | -| **Storage** | S3, GCS, Azure Blob and S3-compatible endpoints as [primary storage](https://docs.greptime.com/user-guide/deployments-administration/configuration/#storage-options), with memory and local-disk caches | -| **Built in** | Retention policies, downsampling, [continuous aggregation](https://docs.greptime.com/user-guide/flow-computation/overview), explicit table partitioning, and [inverted / skipping / fulltext indexes](https://docs.greptime.com/user-guide/manage-data/data-index) | - -Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute. - -

- - GreptimeDB Overview - -

- -## Benchmarks - -* [Agent RCA Bench](https://rca-bench.greptime.com/): LLM agents doing root cause analysis - over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer - input tokens, 45% lower cost - ([write-up](https://greptime.com/blogs/2026-09-08-agent-rca-bench-interface-semantic-layer)) -* [GreptimeDB tops JSONBench's billion-record cold run test](https://greptime.com/blogs/2025-03-18-jsonbench-greptimedb-performance) -* [TSBS Benchmark](https://github.com/GreptimeTeam/greptimedb/tree/main/docs/benchmarks/tsbs) -* [More benchmark reports](https://docs.greptime.com/user-guide/concepts/features-that-you-concern#how-is-greptimedbs-performance-compared-to-other-solutions) - -## Compatibility and Migration - -Compatibility is per protocol, and query-side coverage is narrower than ingestion. - -| | Compatible | Not compatible | -|---|---|---| -| **Prometheus** | Remote Write ingestion; PromQL queries | Gaps are listed in [PromQL compatibility](https://docs.greptime.com/user-guide/query-data/promql/) | -| **Loki** | Push ingestion; dual-write through Grafana Alloy makes the [cutover gradual](https://greptime.com/blogs/2026-07-23-from-loki-to-greptimedb-dual-write-migration) | LogQL and the rest of the Loki query API | -| **Elasticsearch** | `_bulk` ingestion in the open-source core; [QueryDSL](https://docs.greptime.com/enterprise/elasticsearch-compatible/query/) partially, in Enterprise | Most other Elasticsearch APIs | - -## Limitations and Edition Boundary - -Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there. - -Read replicas, workload isolation, and automated repartitioning are **GreptimeDB Enterprise** features, along with enterprise security and governance. The [Enterprise overview](https://docs.greptime.com/enterprise/overview/) has the current list, and [pricing](https://www.greptime.com/pricing#differences) has the edition comparison. - -## Architecture - -GreptimeDB can run in two modes: -* **Standalone** — single binary for development and small deployments. -* **Distributed** — four components, each independently scalable: - - **Frontend** — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally. - - **Datanode** — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic. - - **Metasrv** — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS). - - **Flownode** (optional) — continuous flow computation (streaming and materialized views). - -For deeper coverage, see the [architecture doc](https://docs.greptime.com/contributor-guide/overview/#architecture) or [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb/1-overview). - - - GreptimeDB System Overview - - ## Try GreptimeDB **For AI agents** — paste this prompt into your agent: @@ -150,20 +78,81 @@ docker run -p 127.0.0.1:4000-4003:4000-4003 \ --mysql-addr 0.0.0.0:4002 \ --postgres-addr 0.0.0.0:4003 ``` -Dashboard: [http://localhost:4000/dashboard](http://localhost:4000/dashboard) +Dashboard: [http://localhost:4000/dashboard](http://localhost:4000/dashboard) +Or skip the install and try the [Live Demo](https://live.greptime.cloud/dashboard/?info=eyJ1c2VybmFtZSI6ImdyZXB0aW1lIiwicGFzc3dvcmQiOiJkZW1vc2Vzc2lvbiJ9&query_guide=1&utm_source=github&utm_medium=readme&utm_campaign=greptimedb#/dashboard/explore). -Read more in the [full Install Guide](https://docs.greptime.com/getting-started/installation/overview). +Next: [Quickstart](https://docs.greptime.com/getting-started/quick-start), the [full Install Guide](https://docs.greptime.com/getting-started/installation/overview), [Demo Scenes](https://github.com/GreptimeTeam/demo-scene), and the [FAQ](https://docs.greptime.com/faq-and-others/faq). -**Troubleshooting:** -* Cannot connect to the database? Ensure that ports `4000`, `4001`, `4002`, and `4003` are not blocked by a firewall or used by other services. -* Failed to start? Check the container logs with `docker logs greptime` for further details. +## Why You Might Use It -## Getting Started +- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three +- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface +- You are hitting Loki's query performance limits as log volume grows +- You need long retention on object storage without a separate analytics stack +- You want to query telemetry with SQL, not only a domain query language +- You are storing GenAI or agent telemetry ([OTel GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)) alongside infrastructure signals -- [Quickstart](https://docs.greptime.com/getting-started/quick-start) -- [User Guide](https://docs.greptime.com/user-guide/overview) -- [Demo Scenes](https://github.com/GreptimeTeam/demo-scene) -- [FAQ](https://docs.greptime.com/faq-and-others/faq) +Learn more in [Why GreptimeDB](https://docs.greptime.com/user-guide/concepts/why-greptimedb). + +## What's Supported + +| | | +|---|---| +| **Ingest** | [OpenTelemetry (OTLP)](https://docs.greptime.com/user-guide/ingest-data/for-observability/opentelemetry/), [Prometheus Remote Write](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/) (with [native histograms](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/#native-histograms) over Remote Write 2.0), [Loki Push](https://docs.greptime.com/user-guide/ingest-data/for-observability/loki), [Elasticsearch Bulk](https://docs.greptime.com/user-guide/ingest-data/for-observability/elasticsearch), [InfluxDB line protocol](https://docs.greptime.com/user-guide/protocols/influxdb-line-protocol), [gRPC](https://docs.greptime.com/user-guide/protocols/grpc/) | +| **Query** | [SQL](https://docs.greptime.com/user-guide/query-data/overview/), [PromQL](https://docs.greptime.com/user-guide/query-data/promql/), [Jaeger-compatible trace queries](https://docs.greptime.com/user-guide/query-data/jaeger/), MySQL and PostgreSQL wire protocols | +| **Storage** | S3, GCS, Azure Blob and S3-compatible endpoints as [primary storage](https://docs.greptime.com/user-guide/deployments-administration/configuration/#storage-options), with memory and local-disk caches | +| **Built in** | [Pipelines](https://docs.greptime.com/user-guide/logs/use-custom-pipelines) that parse and transform logs at ingestion, retention policies, downsampling, [continuous aggregation](https://docs.greptime.com/user-guide/flow-computation/overview), explicit table partitioning, and [inverted / skipping / fulltext indexes](https://docs.greptime.com/user-guide/manage-data/data-index) | + +Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute. + +

+ + GreptimeDB Overview + +

+ +## Benchmarks + +* [Agent RCA Bench](https://rca-bench.greptime.com/): LLM agents doing root cause analysis + over GreptimeDB versus Prometheus + Loki + Tempo, 168 investigations per interface across + six models. 40% fewer wrong diagnoses, about 48% fewer input tokens, about 45% lower + estimated cost + ([write-up](https://greptime.com/blogs/2026-09-08-agent-rca-bench-interface-semantic-layer)) +* [JSONBench billion-record cold run: first place as of March 2025](https://greptime.com/blogs/2025-03-18-jsonbench-greptimedb-performance) +* [TSBS Benchmark](https://github.com/GreptimeTeam/greptimedb/tree/main/docs/benchmarks/tsbs) +* [More benchmark reports](https://docs.greptime.com/user-guide/concepts/features-that-you-concern#how-is-greptimedbs-performance-compared-to-other-solutions) + +## Compatibility and Migration + +Compatibility is per protocol, and query-side coverage is narrower than ingestion. + +| | Compatible | Not compatible | +|---|---|---| +| **Prometheus** | Remote Write ingestion; PromQL queries | Gaps are listed in [PromQL compatibility](https://docs.greptime.com/user-guide/query-data/promql/) | +| **Loki** | Push ingestion; dual-write through Grafana Alloy makes the [cutover gradual](https://greptime.com/blogs/2026-07-23-from-loki-to-greptimedb-dual-write-migration) | LogQL and the rest of the Loki query API | +| **Elasticsearch** | `_bulk` ingestion in the open-source core; [QueryDSL](https://docs.greptime.com/enterprise/elasticsearch-compatible/query/) partially, in Enterprise | Most other Elasticsearch APIs | + +## Limitations and Edition Boundary + +Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning and region migration are manual operations there. + +Read replicas, workload isolation, and automated repartitioning are **GreptimeDB Enterprise** features, along with enterprise security and governance. The [Enterprise overview](https://docs.greptime.com/enterprise/overview/) has the current list, and [pricing](https://www.greptime.com/pricing#differences) has the edition comparison. For GreptimeDB Enterprise inquiries, [contact us](https://greptime.com/contactus). + +## Architecture + +GreptimeDB can run in two modes: +* **Standalone** — single binary for development and small deployments. +* **Distributed** — four components, each independently scalable: + - **Frontend** — protocol endpoints and the distributed query engine. Stateless, scales horizontally. + - **Datanode** — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. + - **Metasrv** — metadata, region routing, and cluster coordination (DDL procedures, failover, region migration, repartitioning). Backed by a pluggable KV layer (etcd or RDS). + - **Flownode** (optional) — continuous flow computation (streaming and materialized views). + +For deeper coverage, see the [architecture doc](https://docs.greptime.com/contributor-guide/overview/#architecture) or [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb/1-overview). + + + GreptimeDB System Overview + ## Build From Source @@ -202,12 +191,10 @@ See the [Contribution Guidelines](CONTRIBUTING.md) for the full developer workfl GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — [OceanBase Cloud](https://greptime.com/blogs/2025-07-22-user-case-obcloud-log-management-greptimedb) operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in [case studies](https://greptime.com/blogs/?category=Use%20Case). -Release lines and support windows are in the [version reference](https://docs.greptime.com/nightly/reference/about-greptimedb-version). For where the project is going, read the [v1.0 highlights](https://greptime.com/blogs/2025-11-05-greptimedb-v1-highlights) and the [2026 roadmap](https://greptime.com/blogs/2026-02-11-greptimedb-roadmap-2026). +Releases follow [Semantic Versioning](https://docs.greptime.com/reference/about-greptimedb-version). For where the project is going, read the [v1.0 highlights](https://greptime.com/blogs/2025-11-05-greptimedb-v1-highlights), the [2026 roadmap](https://github.com/GreptimeTeam/greptimedb/issues/7685), and [what we mean by an observability database](https://greptime.com/blogs/2026-08-26-what-is-an-observability-database). ## Community -We invite you to engage and contribute! - - [Slack](https://greptime.com/slack) - [Discussions](https://github.com/GreptimeTeam/greptimedb/discussions) - [Official Website](https://greptime.com/) @@ -218,6 +205,14 @@ We invite you to engage and contribute! If GreptimeDB is useful to you, please star the repo. + + + + + Star History Chart + + + Known Users ## License @@ -230,17 +225,10 @@ A small set of peripheral, enterprise-only features are gated behind the separate [GreptimeDB Enterprise License](LICENSE-ENTERPRISE). Source files under that license carry an explicit Enterprise License header. -## Commercial Support - -Scaling observability on your infrastructure? -[GreptimeDB Enterprise](https://docs.greptime.com/enterprise/overview/) adds the operational, -security, and support layer for production deployments. -[Contact us](https://greptime.com/contactus) for details. - ## Contributing - Read our [Contribution Guidelines](CONTRIBUTING.md). -- Explore [Internal Concepts](https://docs.greptime.com/contributor-guide/overview.html) and [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb). +- Explore [Internal Concepts](https://docs.greptime.com/contributor-guide/overview/) and [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb). - Pick up a [good first issue](https://github.com/GreptimeTeam/greptimedb/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) and join the #contributors [Slack](https://greptime.com/slack) channel. [![Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/workflows/integration.yml/badge.svg)](https://github.com/GreptimeTeam/greptimedb/actions/workflows/integration.yml) diff --git a/config/config.md b/config/config.md index b5aa8488929..03029ae1fdc 100644 --- a/config/config.md +++ b/config/config.md @@ -20,6 +20,8 @@ | `write_bytes_exhausted_policy` | String | `wait` | Policy when write bytes quota is exhausted.
Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail" | | `init_regions_in_background` | Bool | `false` | Initialize all regions in the background during the startup.
By default, it provides services after all regions have been initialized. | | `init_regions_parallelism` | Integer | `16` | Parallelism of initializing regions. | +| `max_concurrent_queries` | Integer | `0` | The maximum concurrent region-read streams. Zero means unlimited.
A single query can consume multiple permits. | +| `concurrent_query_limiter_timeout` | String | `100ms` | Timeout to acquire a permit when `max_concurrent_queries` is reached.
Only effective when the limiter is enabled. | | `enable_telemetry` | Bool | `true` | Enable telemetry to collect anonymous usage data. Enabled by default. | | `runtime` | -- | -- | The runtime options. | | `runtime.global_rt_size` | Integer | Auto | The number of threads to execute the runtime for global read operations.
Defaults to max(num_cpus, 2). | @@ -82,7 +84,7 @@ | `pending_rows_batcher.worker_channel_capacity` | Integer | `65536` | Maximum queued submissions per table worker. | | `pending_rows_batcher.max_inflight_requests` | Integer | `3000` | Maximum admitted original requests awaiting completion. | | `pending_rows_batcher.flow_notification_queue_capacity` | Integer | `1024` | Maximum number of queued table Flow notifications. | -| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics.
Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible.
Enable independently with protocols and a nonzero flush interval.
Omitted fields use independent defaults, not parent settings.
Empty protocols or a zero interval disables logical batching without fallback.
Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. | +| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and OTLP metrics.
Requires prom_store.with_metric_engine. Logs and traces are not eligible.
Enable independently with protocols and a nonzero flush interval.
Omitted fields use independent defaults, not parent settings.
Empty protocols or a zero interval disables logical batching without fallback.
Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. | | `pending_rows_batcher.logical_table.pending_rows_flush_interval` | String | `0s` | Flush interval measured from the first pending submission. Zero disables batching. | | `pending_rows_batcher.logical_table.max_batch_rows` | Integer | `100000` | Flush after a complete submission reaches this row threshold. | | `pending_rows_batcher.logical_table.max_concurrent_flushes` | Integer | `256` | Maximum concurrent flushes shared by Prom and OTLP metrics. | @@ -131,6 +133,7 @@ | `wal.max_unpersisted_bytes` | String | `64MB` | The size of the unpersisted backlog at which new appends stall until an upload completes, in `enqueued` mode.
Nothing is dropped and nothing is rejected; the threshold does not bound what a crash during an outage can lose.
**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. | | `wal.max_unpersisted_age` | String | `8s` | The age of the oldest unpersisted entry at which new appends stall until an upload completes, in `enqueued` mode.
**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. | | `wal.on_corrupted_segment` | String | `skip` | What a read does with a segment that still does not decode after a second fetch, because its checksum does not match or its content disagrees with its footer entry.
- `skip`: the segment is skipped and recorded as a WAL hole of its region, a metric is incremented and a warning is logged; the other regions of the object are unaffected (the default).
- `fail`: the read fails, so the region does not open.
**It's only used when the provider is `experimental_object_store`**. | +| `wal.accept_missing_history` | Bool | `false` | Whether a region opens although the WAL objects under the prefix do not hold its history.
An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.
To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.
When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.
**It's only used when the provider is `experimental_object_store`**. | | `metadata_store` | -- | -- | Metadata storage options. | | `metadata_store.file_size` | String | `64MB` | The size of the metadata store log file. | | `metadata_store.purge_threshold` | String | `256MB` | The threshold of the metadata store size to trigger a purge. | @@ -142,7 +145,7 @@ | `query` | -- | -- | The query engine options. | | `query.experimental_native_promql` | Bool | `false` | Enable experimental native PromQL execution for compatible queries.
Requires the per-request query.experimental_native_promql=true extension.
Default: false | | `query.parallelism` | Integer | `0` | Parallelism of the query engine.
Default to 0, which means the number of CPU cores. | -| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).
Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").
Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
When this limit is reached, queries will fail with ResourceExhausted error.
NOTE: This does NOT limit memory used by table scans. | +| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join,
and internal series merges during table scans).
Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").
Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
When this limit is reached, queries will fail with ResourceExhausted error.
Projected Parquet row groups and streamed scan batches use scan_memory_limit separately. | | `query.experimental_memory_pool_policy` | String | `greedy` | Experimental memory pool allocation policy:
- "greedy" (default): first-come-first-served allocation; preserves current behavior.
- "fair": divides available memory among spillable operators and may spill earlier.
Only effective when `memory_pool_size` is bounded (>0). | | `query.experimental_spill_mode` | String | `default` | Spill mode:
- "default": preserve DataFusion built-in OS temp directory (default).
- "custom": explicitly configure spill path, quota, and compression.
- "disabled": explicitly disable disk spilling.
Set this to "custom" before using the path/quota/compression keys below. | | `query.experimental_spill_path` | String | Unset | Spill directory path. Ignored unless mode is "custom". | @@ -151,6 +154,7 @@ | `storage` | -- | -- | The data storage options. | | `storage.data_home` | String | `./greptimedb_data` | The working home directory. | | `storage.copy_root` | String | `./greptimedb_data/copy` | Root directory for standalone SQL access to local files.
Relative SQL paths are resolved below this directory. Absolute paths are accepted only when
they are inside this directory. Defaults to `/copy`.
Distributed deployments always reject SQL access to local files.
Upgrade note: COPY commands and existing external tables that reference paths outside this
directory will fail. Move those files below the copy root, set this option to a dedicated
directory containing them, or migrate the files to object storage before upgrading. | +| `storage.probe_on_startup` | Bool | `true` | Whether to probe every remote object storage when the datanode starts.
The probe writes, reads, overwrites, lists and deletes an object under
`.greptime_probe/`, and a failure aborts startup. The probe lives at the storage
root, so the credentials must allow writes and deletes below the root. Disable it
when the credentials are scoped to narrower prefixes or when the storage may be
unavailable at startup. | | `storage.type` | String | `File` | The storage type used to store the data.
- `File`: the data is stored in the local file system.
- `S3`: the data is stored in the S3 object storage.
- `Gcs`: the data is stored in the Google Cloud Storage.
- `Azblob`: the data is stored in the Azure Blob Storage.
- `Oss`: the data is stored in the Aliyun OSS.
- `Hdfs`: the data is stored in the Hadoop Distributed File System. | | `storage.bucket` | String | Unset | The S3 bucket name.
**It's only used when the storage type is `S3`, `Oss` and `Gcs`**. | | `storage.root` | String | Unset | The directory or object prefix under which data is stored.
**It's only used when the storage type is `S3`, `Oss`, `Gcs`, `Azblob` and `Hdfs`**. | @@ -174,7 +178,7 @@ | `storage.http_client.timeout` | String | `30s` | The total request timeout, applied from when the request starts connecting until the response body has finished.
Also considered a total deadline. | | `storage.http_client.pool_idle_timeout` | String | `90s` | The timeout for idle sockets being kept-alive. | | `storage.http_client.skip_ssl_validation` | Bool | `false` | To skip the ssl verification
**Security Notice**: Setting `skip_ssl_validation = true` disables certificate verification, making connections vulnerable to man-in-the-middle attacks. Only use this in development or trusted private networks. | -| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.
Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail. | +| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.
Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail.
The default configuration lists `mito` and `file`; `metric` is always registered, so naming
it only supplies its options rather than enabling the engine. | | `region_engine.mito` | -- | -- | The Mito engine options. | | `region_engine.mito.num_workers` | Integer | Auto | Number of region workers. Defaults to half of the available CPU cores, rounded up. | | `region_engine.mito.worker_channel_size` | Integer | `128` | Request channel size of each worker. | @@ -188,10 +192,11 @@ | `region_engine.mito.experimental_series_index_maintenance_interval` | String | `5m` | Interval between series-index maintenance runs. Zero uses the default of 5 min. | | `region_engine.mito.experimental_series_index_bucket_width` | String | `5days` | Requested minimum series-index bucket width (default: 5 days), rounded up to
an exact multiple of each region's compaction time window. | | `region_engine.mito.max_background_flushes` | Integer | Auto | Max number of running background flush jobs (default: 1/2 of cpu cores). | -| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores). | +| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores).
The same limit bounds engine-wide admission of local compaction units and
the merge-output budget of each automatic per-region pick. Outputs sharing
inputs form one unit. Expired SSTs may add one expiration-only unit outside
the merge-output budget; it still consumes a slot in the same admission limit.
Manual compaction keeps using the request's parallelism. | | `region_engine.mito.max_background_purges` | Integer | Auto | Max number of running background purge jobs (default: number of cpu cores). | | `region_engine.mito.experimental_compaction_memory_limit` | String | 0 | Memory budget for compaction tasks.
Supports absolute size (e.g., "2GiB", "512MB") or percentage of system memory (e.g., "50%").
Setting it to 0 or "unlimited" disables the limit. | | `region_engine.mito.experimental_compaction_on_exhausted` | String | wait | Behavior when compaction cannot acquire memory from the budget.
Options: "wait" (default, 10s), "wait()", "fail" | +| `region_engine.mito.experimental_compaction_output_parallelism` | Integer | 1 | Default per-output parallelism for input decoding and SST column encoding.
Each stage is bounded independently; this is not a global thread limit.
Values below 2 disable parallel decoding/encoding, but not the merge pipeline.
Tables can override it with 'compaction.experimental_output_parallelism' (positive integer). | | `region_engine.mito.auto_flush_interval` | String | `10m` | Interval to auto flush a region if it has not flushed yet. | | `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. | | `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size`. | @@ -211,10 +216,11 @@ | `region_engine.mito.enable_refill_cache_on_read` | Bool | `true` | Enable refilling cache on read operations (default: true).
When disabled, cache refilling on read won't happen. | | `region_engine.mito.manifest_cache_size` | String | `256MB` | Capacity for manifest cache (default: 256MB). | | `region_engine.mito.sst_write_buffer_size` | String | `8MB` | Buffer size for SST writing. | -| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently. | +| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently when the memory limit is disabled
or a scan includes extension sources not covered by row-group accounting. | | `region_engine.mito.allow_stale_entries` | Bool | `false` | Whether to allow stale WAL entries read during replay. | -| `region_engine.mito.scan_memory_limit` | String | `unlimited` | Memory limit for table scans across all queries.
Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").
Setting it to 0 or "unlimited" disables the limit. | -| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Controls what happens when a scan cannot get memory immediately.
"fail" (default) fails fast and is the recommended option for most users.
"wait" / "wait()" waits for memory to become available. This is mainly
for advanced tuning in bursty workloads where temporary contention is common and
higher latency is acceptable.
"wait" means "wait(10s)", not unlimited waiting. | +| `region_engine.mito.scan_memory_limit` | String | `25%` | Memory limit for table scans across all queries and regions.
Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available.
Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").
Setting it to 0 or "unlimited" disables the limit and restores file-count checks. | +| `region_engine.mito.scan_memory_mode` | String | `basic` | Memory accounting mode: "basic" (default) counts projected Parquet columns'
uncompressed row-group sizes while decoding, including files from older versions.
"extended" also counts cumulative streamed batches until stream drop.
Limited reservations round up to KB; unlimited accounting is byte-exact.
Internal series merges use query.memory_pool_size in both modes, sharing capacity
with other query operators on the executing node. Neither budget measures total process memory. | +| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Scan memory exhaustion always fails immediately.
Legacy "wait" / "wait()" settings are deprecated. They are accepted
for upgrade compatibility, but startup logs a warning and treats them as "fail".
Change existing wait settings to "fail"; scans no longer wait for quota to free up.
This policy applies only to scan reservations. Series merges use the query pool
and fail immediately when that pool is exhausted. | | `region_engine.mito.min_compaction_interval` | String | `0m` | Minimum time interval between two compactions.
To align with the old behavior, the default value is 0 (no restrictions). | | `region_engine.mito.schedule_compaction_after_edit` | Bool | `true` | Whether to allow to schedule a compaction after a successful region edit.

Setting this to "true" is a necessary but not sufficient condition for scheduling compaction after a region edit.
Other constraints, such as "min_compaction_interval", may still prevent compaction from being scheduled.
Setting this to "false", however, guarantees that compaction will not be scheduled after a region edit. | | `region_engine.mito.default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. | @@ -223,6 +229,7 @@ | `region_engine.mito.index.aux_path` | String | `""` | Auxiliary directory path for the index in filesystem, used to store intermediate files for
creating the index and staging files for searching the index, defaults to `{data_home}/index_intermediate`.
The default name for this directory is `index_intermediate` for backward compatibility.

This path contains two subdirectories:
- `__intm`: for storing intermediate files used during creating index.
- `staging`: for storing staging files used during searching index. | | `region_engine.mito.index.staging_size` | String | `2GB` | The max capacity of the staging directory. | | `region_engine.mito.index.staging_ttl` | String | `7d` | The TTL of the staging directory.
Defaults to 7 days.
Setting it to "0s" to disable TTL. | +| `region_engine.mito.index.build_mode` | String | `sync` | When to build indexes for new SST files.
- `sync`: build indexes while writing SST files during flush and compaction.
- `async`: build indexes in the background after the SST files are written. Queries read
these files without indexes until the build finishes. | | `region_engine.mito.index.metadata_cache_size` | String | `64MiB` | Cache size for inverted index metadata. | | `region_engine.mito.index.content_cache_size` | String | `128MiB` | Cache size for inverted index content. | | `region_engine.mito.index.content_cache_page_size` | String | `64KiB` | Page size for inverted index content cache. | @@ -262,7 +269,7 @@ | `logging.tracing_sample_ratio.default_ratio` | Float | `1.0` | -- | | `slow_query` | -- | -- | The slow query log options. | | `slow_query.enable` | Bool | `true` | Whether to enable slow query log. | -| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.
If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.
If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`. | +| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.
If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.
If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled,
and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty.
Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings. | | `slow_query.threshold` | String | `30s` | The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`. | | `slow_query.sample_ratio` | Float | `1.0` | The sampling ratio of slow query log. The value should be in the range of (0, 1]. For example, `0.1` means 10% of the slow queries will be logged and `1.0` means all slow queries will be logged. | | `slow_query.ttl` | String | `90d` | The TTL of the `slow_queries` system table. Default is `90d` when `record_type` is `system_table`. | @@ -270,6 +277,8 @@ | `tracing.tokio_console_addr` | String | Unset | The tokio console address. | | `pipeline` | -- | -- | The pipeline options. | | `pipeline.cache_ttl` | String | `10s` | Time to live of the local pipeline cache. Default is `10s`. | +| `pipeline.allow_vrl_host_access` | Bool | `false` | Allow VRL to access the server environment, filesystem, hostname, local timezone,
network, and logs. Default is false. Enable only for trusted pipeline authors.
Applies to pipeline creation, execution, and dry-runs. Requires a restart. | +| `pipeline.ignore_errors` | Bool | `false` | Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter.
Explicit request values override this setting. Malformed input may be discarded,
and invalid custom timestamps in identity ingestion fall back to server time.
Pipeline execution errors are controlled separately by `skip_error`.
Default is false. Requires a restart. | | `event_recorder` | -- | -- | Configuration options for the event recorder. | | `event_recorder.ttl` | String | `90d` | TTL for the events table that will be used to store the events. Default is `90d`. | | `event_recorder.event_types` | Array | All | Event types to record. Current available event types: `create_database`,
`alter_database`, `drop_database`, `create_flow`, `drop_flow`,
`create_table`, `create_logical_tables`, `alter_table`, `alter_logical_tables`,
`drop_table`, `undrop_table`, `purge_dropped_table`, `truncate_table`,
`create_view`, `drop_view`, `admin_function`, `reconcile_catalog`,
`reconcile_database`, `reconcile_table`, `reconcile_logical_tables`.
When omitted, all current and future event types are recorded.
Set to an empty array to disable event recording. | @@ -357,7 +366,7 @@ | `pending_rows_batcher.worker_channel_capacity` | Integer | `65536` | Maximum queued submissions per table worker. | | `pending_rows_batcher.max_inflight_requests` | Integer | `3000` | Maximum admitted original requests awaiting completion. | | `pending_rows_batcher.flow_notification_queue_capacity` | Integer | `1024` | Maximum number of queued table Flow notifications. | -| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics.
Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible.
Enable independently with protocols and a nonzero flush interval.
Omitted fields use independent defaults, not parent settings.
Empty protocols or a zero interval disables logical batching without fallback.
Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. | +| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and OTLP metrics.
Requires prom_store.with_metric_engine. Logs and traces are not eligible.
Enable independently with protocols and a nonzero flush interval.
Omitted fields use independent defaults, not parent settings.
Empty protocols or a zero interval disables logical batching without fallback.
Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. | | `pending_rows_batcher.logical_table.pending_rows_flush_interval` | String | `0s` | Flush interval measured from the first pending submission. Zero disables batching. | | `pending_rows_batcher.logical_table.max_batch_rows` | Integer | `100000` | Flush after a complete submission reaches this row threshold. | | `pending_rows_batcher.logical_table.max_concurrent_flushes` | Integer | `256` | Maximum concurrent flushes shared by Prom and OTLP metrics. | @@ -388,7 +397,7 @@ | `query.experimental_native_promql` | Bool | `false` | Enable experimental native PromQL execution for compatible standalone queries.
Requires in-process region transport and the per-request
query.experimental_native_promql=true extension. Default: false | | `query.parallelism` | Integer | `0` | Parallelism of the query engine.
Default to 0, which means the number of CPU cores. | | `query.allow_query_fallback` | Bool | `false` | Whether to allow query fallback when push down optimize fails.
Default to false, meaning when push down optimize failed, return error msg | -| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).
Supports absolute size (e.g., "4GB", "8GB") or percentage of system memory (e.g., "30%").
Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
When this limit is reached, queries will fail with ResourceExhausted error.
NOTE: This does NOT limit memory used by table scans (only applies to datanodes). | +| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).
Supports absolute size (e.g., "4GB", "8GB") or percentage of system memory (e.g., "30%").
Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
When this limit is reached, queries will fail with ResourceExhausted error.
This pool applies to operators on this frontend. Datanodes use their own query
pools for local operators and internal series merges during table scans. | | `query.experimental_memory_pool_policy` | String | `greedy` | Experimental memory pool allocation policy:
- "greedy" (default): first-come-first-served allocation; preserves current behavior.
- "fair": divides available memory among spillable operators and may spill earlier.
Only effective when `memory_pool_size` is bounded (>0). | | `query.experimental_spill_mode` | String | `default` | Spill mode:
- "default": preserve DataFusion built-in OS temp directory (default).
- "custom": explicitly configure spill path, quota, and compression.
- "disabled": explicitly disable disk spilling.
Set this to "custom" before using the path/quota/compression keys below. | | `query.experimental_spill_path` | String | Unset | Spill directory path. Ignored unless mode is "custom". | @@ -422,7 +431,7 @@ | `logging.tracing_sample_ratio.default_ratio` | Float | `1.0` | -- | | `slow_query` | -- | -- | The slow query log options. | | `slow_query.enable` | Bool | `true` | Whether to enable slow query log. | -| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.
If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.
If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`. | +| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.
If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.
If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled,
and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty.
Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings. | | `slow_query.threshold` | String | `30s` | The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`. | | `slow_query.sample_ratio` | Float | `1.0` | The sampling ratio of slow query log. The value should be in the range of (0, 1]. For example, `0.1` means 10% of the slow queries will be logged and `1.0` means all slow queries will be logged. | | `slow_query.ttl` | String | `90d` | The TTL of the `slow_queries` system table. Default is `90d` when `record_type` is `system_table`. | @@ -432,6 +441,8 @@ | `memory.enable_heap_profiling` | Bool | `true` | Whether to enable heap profiling activation during startup.
When enabled, heap profiling will be activated if the `MALLOC_CONF` environment variable
is set to "prof:true,prof_active:false". The official image adds this env variable.
Default is true. | | `pipeline` | -- | -- | The pipeline options. | | `pipeline.cache_ttl` | String | `10s` | Time to live of the frontend-local pipeline cache. A pipeline created or deleted on
another frontend takes effect on this one after at most this duration. | +| `pipeline.allow_vrl_host_access` | Bool | `false` | Allow VRL to access the server environment, filesystem, hostname, local timezone,
network, and logs. Default is false. Enable only for trusted pipeline authors.
Applies to pipeline creation, execution, and dry-runs. Requires a restart. | +| `pipeline.ignore_errors` | Bool | `false` | Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter.
Explicit request values override this setting. Malformed input may be discarded,
and invalid custom timestamps in identity ingestion fall back to server time.
Pipeline execution errors are controlled separately by `skip_error`.
Default is false. Requires a restart. | | `event_recorder` | -- | -- | Configuration options for the event recorder. | | `event_recorder.ttl` | String | `90d` | TTL for the events table that will be used to store the events. Default is `90d`. | | `event_recorder.event_types` | Array | All | Event types to record. Current available event type: `admin_function`.
When omitted, all current and future event types are recorded.
Set to an empty array to disable event recording. | @@ -615,9 +626,10 @@ | `wal.max_unpersisted_bytes` | String | `64MB` | The size of the unpersisted backlog at which new appends stall until an upload completes, in `enqueued` mode.
Nothing is dropped and nothing is rejected; the threshold does not bound what a crash during an outage can lose.
**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. | | `wal.max_unpersisted_age` | String | `8s` | The age of the oldest unpersisted entry at which new appends stall until an upload completes, in `enqueued` mode.
**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. | | `wal.on_corrupted_segment` | String | `skip` | What a read does with a segment that still does not decode after a second fetch, because its checksum does not match or its content disagrees with its footer entry.
- `skip`: the segment is skipped and recorded as a WAL hole of its region, a metric is incremented and a warning is logged; the other regions of the object are unaffected (the default).
- `fail`: the read fails, so the region does not open.
**It's only used when the provider is `experimental_object_store`**. | +| `wal.accept_missing_history` | Bool | `false` | Whether a region opens although the WAL objects under the prefix do not hold its history.
An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.
To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.
When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.
**It's only used when the provider is `experimental_object_store`**. | | `query` | -- | -- | The query engine options. | | `query.parallelism` | Integer | `0` | Parallelism of the query engine.
Default to 0, which means the number of CPU cores. | -| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).
Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").
Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
When this limit is reached, queries will fail with ResourceExhausted error.
NOTE: This does NOT limit memory used by table scans. | +| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join,
and internal series merges during table scans).
Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").
Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
When this limit is reached, queries will fail with ResourceExhausted error.
Projected Parquet row groups and streamed scan batches use scan_memory_limit separately. | | `query.experimental_memory_pool_policy` | String | `greedy` | Experimental memory pool allocation policy:
- "greedy" (default): first-come-first-served allocation; preserves current behavior.
- "fair": divides available memory among spillable operators and may spill earlier.
Only effective when `memory_pool_size` is bounded (>0). | | `query.experimental_spill_mode` | String | `default` | Spill mode:
- "default": preserve DataFusion built-in OS temp directory (default).
- "custom": explicitly configure spill path, quota, and compression.
- "disabled": explicitly disable disk spilling.
Set this to "custom" before using the path/quota/compression keys below. | | `query.experimental_spill_path` | String | Unset | Spill directory path. Ignored unless mode is "custom". | @@ -625,6 +637,7 @@ | `query.experimental_spill_compression` | String | `uncompressed` | Compression for spilled data files: "uncompressed" (default), "lz4_frame", "zstd".
Ignored unless mode is "custom". | | `storage` | -- | -- | The data storage options. | | `storage.data_home` | String | `./greptimedb_data` | The working home directory. | +| `storage.probe_on_startup` | Bool | `true` | Whether to probe every remote object storage when the datanode starts.
The probe writes, reads, overwrites, lists and deletes an object under
`.greptime_probe/`, and a failure aborts startup. The probe lives at the storage
root, so the credentials must allow writes and deletes below the root. Disable it
when the credentials are scoped to narrower prefixes or when the storage may be
unavailable at startup. | | `storage.type` | String | `File` | The storage type used to store the data.
- `File`: the data is stored in the local file system.
- `S3`: the data is stored in the S3 object storage.
- `Gcs`: the data is stored in the Google Cloud Storage.
- `Azblob`: the data is stored in the Azure Blob Storage.
- `Oss`: the data is stored in the Aliyun OSS.
- `Hdfs`: the data is stored in the Hadoop Distributed File System. | | `storage.bucket` | String | Unset | The S3 bucket name.
**It's only used when the storage type is `S3`, `Oss` and `Gcs`**. | | `storage.root` | String | Unset | The directory or object prefix under which data is stored.
**It's only used when the storage type is `S3`, `Oss`, `Gcs`, `Azblob` and `Hdfs`**. | @@ -648,7 +661,7 @@ | `storage.http_client.timeout` | String | `30s` | The total request timeout, applied from when the request starts connecting until the response body has finished.
Also considered a total deadline. | | `storage.http_client.pool_idle_timeout` | String | `90s` | The timeout for idle sockets being kept-alive. | | `storage.http_client.skip_ssl_validation` | Bool | `false` | To skip the ssl verification
**Security Notice**: Setting `skip_ssl_validation = true` disables certificate verification, making connections vulnerable to man-in-the-middle attacks. Only use this in development or trusted private networks. | -| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.
Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail. | +| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.
Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail.
The default configuration lists `mito` and `file`; `metric` is always registered, so naming
it only supplies its options rather than enabling the engine. | | `region_engine.mito` | -- | -- | The Mito engine options. | | `region_engine.mito.num_workers` | Integer | Auto | Number of region workers. Defaults to half of the available CPU cores, rounded up. | | `region_engine.mito.worker_channel_size` | Integer | `128` | Request channel size of each worker. | @@ -664,10 +677,11 @@ | `region_engine.mito.experimental_series_index_maintenance_interval` | String | `5m` | Interval between series-index maintenance runs. Zero uses the default of 5 min. | | `region_engine.mito.experimental_series_index_bucket_width` | String | `5days` | Requested minimum series-index bucket width (default: 5 days), rounded up to
an exact multiple of each region's compaction time window. | | `region_engine.mito.max_background_flushes` | Integer | Auto | Max number of running background flush jobs (default: 1/2 of cpu cores). | -| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores). | +| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores).
The same limit bounds engine-wide admission of local compaction units and
the merge-output budget of each automatic per-region pick. Outputs sharing
inputs form one unit. Expired SSTs may add one expiration-only unit outside
the merge-output budget; it still consumes a slot in the same admission limit.
Manual compaction keeps using the request's parallelism. | | `region_engine.mito.max_background_purges` | Integer | Auto | Max number of running background purge jobs (default: number of cpu cores). | | `region_engine.mito.experimental_compaction_memory_limit` | String | 0 | Memory budget for compaction tasks.
Supports absolute size (e.g., "2GiB", "512MB") or percentage of system memory (e.g., "50%").
Setting it to 0 or "unlimited" disables the limit. | | `region_engine.mito.experimental_compaction_on_exhausted` | String | wait | Behavior when compaction cannot acquire memory from the budget.
Options: "wait" (default, 10s), "wait()", "fail" | +| `region_engine.mito.experimental_compaction_output_parallelism` | Integer | 1 | Default per-output parallelism for input decoding and SST column encoding.
Each stage is bounded independently; this is not a global thread limit.
Values below 2 disable parallel decoding/encoding, but not the merge pipeline.
Tables can override it with 'compaction.experimental_output_parallelism' (positive integer). | | `region_engine.mito.auto_flush_interval` | String | `10m` | Interval to auto flush a region if it has not flushed yet. | | `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. | | `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` | @@ -687,10 +701,11 @@ | `region_engine.mito.enable_refill_cache_on_read` | Bool | `true` | Enable refilling cache on read operations (default: true).
When disabled, cache refilling on read won't happen. | | `region_engine.mito.manifest_cache_size` | String | `256MB` | Capacity for manifest cache (default: 256MB). | | `region_engine.mito.sst_write_buffer_size` | String | `8MB` | Buffer size for SST writing. | -| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently. | +| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently when the memory limit is disabled
or a scan includes extension sources not covered by row-group accounting. | | `region_engine.mito.allow_stale_entries` | Bool | `false` | Whether to allow stale WAL entries read during replay. | -| `region_engine.mito.scan_memory_limit` | String | `unlimited` | Memory limit for table scans across all queries.
Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").
Setting it to 0 or "unlimited" disables the limit. | -| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Controls what happens when a scan cannot get memory immediately.
"fail" (default) fails fast and is the recommended option for most users.
"wait" / "wait()" waits for memory to become available. This is mainly
for advanced tuning in bursty workloads where temporary contention is common and
higher latency is acceptable.
"wait" means "wait(10s)", not unlimited waiting. | +| `region_engine.mito.scan_memory_limit` | String | `25%` | Memory limit for table scans across all queries and regions.
Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available.
Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").
Setting it to 0 or "unlimited" disables the limit and restores file-count checks. | +| `region_engine.mito.scan_memory_mode` | String | `basic` | Memory accounting mode: "basic" (default) counts projected Parquet columns'
uncompressed row-group sizes while decoding, including files from older versions.
"extended" also counts cumulative streamed batches until stream drop.
Limited reservations round up to KB; unlimited accounting is byte-exact.
Internal series merges use query.memory_pool_size in both modes, sharing capacity
with other query operators on the executing node. Neither budget measures total process memory. | +| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Scan memory exhaustion always fails immediately.
Legacy "wait" / "wait()" settings are deprecated. They are accepted
for upgrade compatibility, but startup logs a warning and treats them as "fail".
Change existing wait settings to "fail"; scans no longer wait for quota to free up.
This policy applies only to scan reservations. Series merges use the query pool
and fail immediately when that pool is exhausted. | | `region_engine.mito.min_compaction_interval` | String | `0m` | Minimum time interval between two compactions.
To align with the old behavior, the default value is 0 (no restrictions). | | `region_engine.mito.schedule_compaction_after_edit` | Bool | `true` | Whether to allow to schedule a compaction after a successful region edit.

Setting this to "true" is a necessary but not sufficient condition for scheduling compaction after a region edit.
Other constraints, such as "min_compaction_interval", may still prevent compaction from being scheduled.
Setting this to "false", however, guarantees that compaction will not be scheduled after a region edit. | | `region_engine.mito.default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. | @@ -699,6 +714,7 @@ | `region_engine.mito.index.aux_path` | String | `""` | Auxiliary directory path for the index in filesystem, used to store intermediate files for
creating the index and staging files for searching the index, defaults to `{data_home}/index_intermediate`.
The default name for this directory is `index_intermediate` for backward compatibility.

This path contains two subdirectories:
- `__intm`: for storing intermediate files used during creating index.
- `staging`: for storing staging files used during searching index. | | `region_engine.mito.index.staging_size` | String | `2GB` | The max capacity of the staging directory. | | `region_engine.mito.index.staging_ttl` | String | `7d` | The TTL of the staging directory.
Defaults to 7 days.
Setting it to "0s" to disable TTL. | +| `region_engine.mito.index.build_mode` | String | `sync` | When to build indexes for new SST files.
- `sync`: build indexes while writing SST files during flush and compaction.
- `async`: build indexes in the background after the SST files are written. Queries read
these files without indexes until the build finishes. | | `region_engine.mito.index.metadata_cache_size` | String | `64MiB` | Cache size for inverted index metadata. | | `region_engine.mito.index.content_cache_size` | String | `128MiB` | Cache size for inverted index content. | | `region_engine.mito.index.content_cache_page_size` | String | `64KiB` | Page size for inverted index content cache. | @@ -722,7 +738,7 @@ | `region_engine.mito.gc` | -- | -- | -- | | `region_engine.mito.gc.enable` | Bool | `false` | Whether GC is enabled. Need to be the same with metasrv's `gc.enable` or unexpected behavior will occur | | `region_engine.mito.gc.lingering_time` | String | `1h` | Lingering time before deleting files.
Should be long enough to allow long running queries to finish.
If set to None, then unused files will be deleted immediately. | -| `region_engine.mito.gc.unknown_file_lingering_time` | String | `1d` | Lingering time before deleting unknown files (files with undetermined expel time).
Only applies during full file listing GC.
This uses the object's last-modified timestamp as a heuristic (strict less-than comparison);
do not configure this value too small in production to avoid deleting pre-manifest files
from in-progress compaction or flush.
For active/open regions, an unknown file is deleted only if its object last-modified time exceeds this TTL.
If the object store does not provide a last-modified timestamp, the file is conservatively kept.
For dropped regions, unknown files are deleted immediately. | +| `region_engine.mito.gc.unknown_file_lingering_time` | String | `1d` | Lingering time before deleting unknown files (files with undetermined expel time).
It is also the age threshold of the sweep that reclaims temporary files left behind by
a crashed HDFS writer, and that sweep runs even if `enable` is false. HDFS advances the
modification time of a file under construction only as its blocks complete, so lowering
this value also shrinks the margin between the sweep and files that are still being written.
This uses the object's last-modified timestamp as a heuristic (strict less-than comparison);
do not configure this value too small in production to avoid deleting pre-manifest files
from in-progress compaction or flush.
For active/open regions, an unknown file is deleted only if its object last-modified time exceeds this TTL.
If the object store does not provide a last-modified timestamp, the file is conservatively kept.
For dropped regions, unknown files are deleted immediately. | | `region_engine.file` | -- | -- | Enable the file engine. | | `region_engine.metric` | -- | -- | Metric engine options. | | `logging` | -- | -- | The logging options. | diff --git a/config/datanode.example.toml b/config/datanode.example.toml index 0a01fa31973..698bc0e9600 100644 --- a/config/datanode.example.toml +++ b/config/datanode.example.toml @@ -258,6 +258,13 @@ overwrite_entry_start_id = false ## **It's only used when the provider is `experimental_object_store`**. #+ on_corrupted_segment = "skip" +## Whether a region opens although the WAL objects under the prefix do not hold its history. +## An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted. +## To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes. +## When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries. +## **It's only used when the provider is `experimental_object_store`**. +#+ accept_missing_history = false + # The Kafka SASL configuration. # **It's only used when the provider is `kafka`**. # Available SASL mechanisms: @@ -331,11 +338,12 @@ overwrite_entry_start_id = false ## Default to 0, which means the number of CPU cores. parallelism = 0 -## Memory pool size for query execution operators (aggregation, sorting, join). +## Memory pool size for query execution operators (aggregation, sorting, join, +## and internal series merges during table scans). ## Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%"). ## Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`. ## When this limit is reached, queries will fail with ResourceExhausted error. -## NOTE: This does NOT limit memory used by table scans. +## Projected Parquet row groups and streamed scan batches use scan_memory_limit separately. ## @toml2docs:none-default="0" memory_pool_size = "50%" @@ -368,6 +376,14 @@ memory_pool_size = "50%" ## The working home directory. data_home = "./greptimedb_data" +## Whether to probe every remote object storage when the datanode starts. +## The probe writes, reads, overwrites, lists and deletes an object under +## `.greptime_probe/`, and a failure aborts startup. The probe lives at the storage +## root, so the credentials must allow writes and deletes below the root. Disable it +## when the credentials are scoped to narrower prefixes or when the storage may be +## unavailable at startup. +probe_on_startup = true + ## The storage type used to store the data. ## - `File`: the data is stored in the local file system. ## - `S3`: the data is stored in the S3 object storage. @@ -502,6 +518,8 @@ skip_ssl_validation = false ## The region engine options. You can configure multiple region engines. ## Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail. +## The default configuration lists `mito` and `file`; `metric` is always registered, so naming +## it only supplies its options rather than enabling the engine. [[region_engine]] ## The Mito engine options. @@ -566,6 +584,11 @@ compress_manifest = false #+ max_background_flushes = 4 ## Max number of running background compaction jobs (default: 1/4 of cpu cores). +## The same limit bounds engine-wide admission of local compaction units and +## the merge-output budget of each automatic per-region pick. Outputs sharing +## inputs form one unit. Expired SSTs may add one expiration-only unit outside +## the merge-output budget; it still consumes a slot in the same admission limit. +## Manual compaction keeps using the request's parallelism. ## @toml2docs:none-default="Auto" #+ max_background_compactions = 2 @@ -584,6 +607,13 @@ compress_manifest = false ## @toml2docs:none-default="wait" #+ experimental_compaction_on_exhausted = "wait" +## Default per-output parallelism for input decoding and SST column encoding. +## Each stage is bounded independently; this is not a global thread limit. +## Values below 2 disable parallel decoding/encoding, but not the merge pipeline. +## Tables can override it with 'compaction.experimental_output_parallelism' (positive integer). +## @toml2docs:none-default="1" +#+ experimental_compaction_output_parallelism = 1 + ## Interval to auto flush a region if it has not flushed yet. auto_flush_interval = "10m" @@ -662,22 +692,33 @@ manifest_cache_size = "256MB" ## Buffer size for SST writing. sst_write_buffer_size = "8MB" -## Maximum number of SST files to scan concurrently. +## Maximum number of SST files to scan concurrently when the memory limit is disabled +## or a scan includes extension sources not covered by row-group accounting. max_concurrent_scan_files = 384 ## Whether to allow stale WAL entries read during replay. allow_stale_entries = false -## Memory limit for table scans across all queries. +## Memory limit for table scans across all queries and regions. +## Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available. ## Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%"). -## Setting it to 0 or "unlimited" disables the limit. -scan_memory_limit = "unlimited" -## Controls what happens when a scan cannot get memory immediately. -## "fail" (default) fails fast and is the recommended option for most users. -## "wait" / "wait()" waits for memory to become available. This is mainly -## for advanced tuning in bursty workloads where temporary contention is common and -## higher latency is acceptable. -## "wait" means "wait(10s)", not unlimited waiting. +## Setting it to 0 or "unlimited" disables the limit and restores file-count checks. +scan_memory_limit = "25%" + +## Memory accounting mode: "basic" (default) counts projected Parquet columns' +## uncompressed row-group sizes while decoding, including files from older versions. +## "extended" also counts cumulative streamed batches until stream drop. +## Limited reservations round up to KB; unlimited accounting is byte-exact. +## Internal series merges use query.memory_pool_size in both modes, sharing capacity +## with other query operators on the executing node. Neither budget measures total process memory. +scan_memory_mode = "basic" + +## Scan memory exhaustion always fails immediately. +## Legacy "wait" / "wait()" settings are deprecated. They are accepted +## for upgrade compatibility, but startup logs a warning and treats them as "fail". +## Change existing wait settings to "fail"; scans no longer wait for quota to free up. +## This policy applies only to scan reservations. Series merges use the query pool +## and fail immediately when that pool is exhausted. scan_memory_on_exhausted = "fail" ## Minimum time interval between two compactions. @@ -717,6 +758,12 @@ staging_size = "2GB" ## Setting it to "0s" to disable TTL. staging_ttl = "7d" +## When to build indexes for new SST files. +## - `sync`: build indexes while writing SST files during flush and compaction. +## - `async`: build indexes in the background after the SST files are written. Queries read +## these files without indexes until the build finishes. +build_mode = "sync" + ## Cache size for inverted index metadata. metadata_cache_size = "64MiB" @@ -814,7 +861,10 @@ enable = false lingering_time = "1h" ## Lingering time before deleting unknown files (files with undetermined expel time). -## Only applies during full file listing GC. +## It is also the age threshold of the sweep that reclaims temporary files left behind by +## a crashed HDFS writer, and that sweep runs even if `enable` is false. HDFS advances the +## modification time of a file under construction only as its blocks complete, so lowering +## this value also shrinks the margin between the sweep and files that are still being written. ## This uses the object's last-modified timestamp as a heuristic (strict less-than comparison); ## do not configure this value too small in production to avoid deleting pre-manifest files ## from in-progress compaction or flush. diff --git a/config/frontend.example.toml b/config/frontend.example.toml index d798f67e56e..6fac16b4b6a 100644 --- a/config/frontend.example.toml +++ b/config/frontend.example.toml @@ -274,8 +274,8 @@ max_inflight_requests = 3000 ## Maximum number of queued table Flow notifications. flow_notification_queue_capacity = 1024 -## Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics. -## Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible. +## Metric-engine logical-table batching for Prom remote write and OTLP metrics. +## Requires prom_store.with_metric_engine. Logs and traces are not eligible. ## Enable independently with protocols and a nonzero flush interval. ## Omitted fields use independent defaults, not parent settings. ## Empty protocols or a zero interval disables logical batching without fallback. @@ -375,7 +375,8 @@ allow_query_fallback = false ## Supports absolute size (e.g., "4GB", "8GB") or percentage of system memory (e.g., "30%"). ## Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`. ## When this limit is reached, queries will fail with ResourceExhausted error. -## NOTE: This does NOT limit memory used by table scans (only applies to datanodes). +## This pool applies to operators on this frontend. Datanodes use their own query +## pools for local operators and internal series merges during table scans. ## @toml2docs:none-default="0" memory_pool_size = "50%" @@ -486,7 +487,9 @@ enable = true ## The record type of slow queries. It can be `system_table` or `log`. ## If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`. -## If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`. +## If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled, +## and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty. +## Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings. record_type = "system_table" ## The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`. @@ -517,6 +520,16 @@ enable_heap_profiling = true ## Time to live of the frontend-local pipeline cache. A pipeline created or deleted on ## another frontend takes effect on this one after at most this duration. cache_ttl = "10s" +## Allow VRL to access the server environment, filesystem, hostname, local timezone, +## network, and logs. Default is false. Enable only for trusted pipeline authors. +## Applies to pipeline creation, execution, and dry-runs. Requires a restart. +allow_vrl_host_access = false +## Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter. +## Explicit request values override this setting. Malformed input may be discarded, +## and invalid custom timestamps in identity ingestion fall back to server time. +## Pipeline execution errors are controlled separately by `skip_error`. +## Default is false. Requires a restart. +ignore_errors = false ## Configuration options for the event recorder. [event_recorder] diff --git a/config/standalone.example.toml b/config/standalone.example.toml index ea39a2e5a6b..ef5dff43dd9 100644 --- a/config/standalone.example.toml +++ b/config/standalone.example.toml @@ -43,6 +43,14 @@ init_regions_in_background = false ## Parallelism of initializing regions. init_regions_parallelism = 16 +## The maximum concurrent region-read streams. Zero means unlimited. +## A single query can consume multiple permits. +max_concurrent_queries = 0 + +## Timeout to acquire a permit when `max_concurrent_queries` is reached. +## Only effective when the limiter is enabled. +concurrent_query_limiter_timeout = "100ms" + ## Enable telemetry to collect anonymous usage data. Enabled by default. #+ enable_telemetry = true @@ -246,8 +254,8 @@ max_inflight_requests = 3000 ## Maximum number of queued table Flow notifications. flow_notification_queue_capacity = 1024 -## Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics. -## Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible. +## Metric-engine logical-table batching for Prom remote write and OTLP metrics. +## Requires prom_store.with_metric_engine. Logs and traces are not eligible. ## Enable independently with protocols and a nonzero flush interval. ## Omitted fields use independent defaults, not parent settings. ## Empty protocols or a zero interval disables logical batching without fallback. @@ -449,6 +457,13 @@ overwrite_entry_start_id = false ## **It's only used when the provider is `experimental_object_store`**. #+ on_corrupted_segment = "skip" +## Whether a region opens although the WAL objects under the prefix do not hold its history. +## An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted. +## To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes. +## When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries. +## **It's only used when the provider is `experimental_object_store`**. +#+ accept_missing_history = false + # The Kafka SASL configuration. # **It's only used when the provider is `kafka`**. # Available SASL mechanisms: @@ -546,11 +561,12 @@ max_running_procedures = 128 ## Default to 0, which means the number of CPU cores. parallelism = 0 -## Memory pool size for query execution operators (aggregation, sorting, join). +## Memory pool size for query execution operators (aggregation, sorting, join, +## and internal series merges during table scans). ## Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%"). ## Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`. ## When this limit is reached, queries will fail with ResourceExhausted error. -## NOTE: This does NOT limit memory used by table scans. +## Projected Parquet row groups and streamed scan batches use scan_memory_limit separately. ## @toml2docs:none-default="0" memory_pool_size = "50%" @@ -592,6 +608,14 @@ data_home = "./greptimedb_data" ## directory containing them, or migrate the files to object storage before upgrading. #+ copy_root = "./greptimedb_data/copy" +## Whether to probe every remote object storage when the datanode starts. +## The probe writes, reads, overwrites, lists and deletes an object under +## `.greptime_probe/`, and a failure aborts startup. The probe lives at the storage +## root, so the credentials must allow writes and deletes below the root. Disable it +## when the credentials are scoped to narrower prefixes or when the storage may be +## unavailable at startup. +probe_on_startup = true + ## The storage type used to store the data. ## - `File`: the data is stored in the local file system. ## - `S3`: the data is stored in the S3 object storage. @@ -726,6 +750,8 @@ skip_ssl_validation = false ## The region engine options. You can configure multiple region engines. ## Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail. +## The default configuration lists `mito` and `file`; `metric` is always registered, so naming +## it only supplies its options rather than enabling the engine. [[region_engine]] ## The Mito engine options. @@ -777,6 +803,11 @@ compress_manifest = false #+ max_background_flushes = 4 ## Max number of running background compaction jobs (default: 1/4 of cpu cores). +## The same limit bounds engine-wide admission of local compaction units and +## the merge-output budget of each automatic per-region pick. Outputs sharing +## inputs form one unit. Expired SSTs may add one expiration-only unit outside +## the merge-output budget; it still consumes a slot in the same admission limit. +## Manual compaction keeps using the request's parallelism. ## @toml2docs:none-default="Auto" #+ max_background_compactions = 2 @@ -795,6 +826,13 @@ compress_manifest = false ## @toml2docs:none-default="wait" #+ experimental_compaction_on_exhausted = "wait" +## Default per-output parallelism for input decoding and SST column encoding. +## Each stage is bounded independently; this is not a global thread limit. +## Values below 2 disable parallel decoding/encoding, but not the merge pipeline. +## Tables can override it with 'compaction.experimental_output_parallelism' (positive integer). +## @toml2docs:none-default="1" +#+ experimental_compaction_output_parallelism = 1 + ## Interval to auto flush a region if it has not flushed yet. auto_flush_interval = "10m" @@ -873,22 +911,33 @@ manifest_cache_size = "256MB" ## Buffer size for SST writing. sst_write_buffer_size = "8MB" -## Maximum number of SST files to scan concurrently. +## Maximum number of SST files to scan concurrently when the memory limit is disabled +## or a scan includes extension sources not covered by row-group accounting. max_concurrent_scan_files = 384 ## Whether to allow stale WAL entries read during replay. allow_stale_entries = false -## Memory limit for table scans across all queries. +## Memory limit for table scans across all queries and regions. +## Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available. ## Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%"). -## Setting it to 0 or "unlimited" disables the limit. -scan_memory_limit = "unlimited" -## Controls what happens when a scan cannot get memory immediately. -## "fail" (default) fails fast and is the recommended option for most users. -## "wait" / "wait()" waits for memory to become available. This is mainly -## for advanced tuning in bursty workloads where temporary contention is common and -## higher latency is acceptable. -## "wait" means "wait(10s)", not unlimited waiting. +## Setting it to 0 or "unlimited" disables the limit and restores file-count checks. +scan_memory_limit = "25%" + +## Memory accounting mode: "basic" (default) counts projected Parquet columns' +## uncompressed row-group sizes while decoding, including files from older versions. +## "extended" also counts cumulative streamed batches until stream drop. +## Limited reservations round up to KB; unlimited accounting is byte-exact. +## Internal series merges use query.memory_pool_size in both modes, sharing capacity +## with other query operators on the executing node. Neither budget measures total process memory. +scan_memory_mode = "basic" + +## Scan memory exhaustion always fails immediately. +## Legacy "wait" / "wait()" settings are deprecated. They are accepted +## for upgrade compatibility, but startup logs a warning and treats them as "fail". +## Change existing wait settings to "fail"; scans no longer wait for quota to free up. +## This policy applies only to scan reservations. Series merges use the query pool +## and fail immediately when that pool is exhausted. scan_memory_on_exhausted = "fail" ## Minimum time interval between two compactions. @@ -928,6 +977,12 @@ staging_size = "2GB" ## Setting it to "0s" to disable TTL. staging_ttl = "7d" +## When to build indexes for new SST files. +## - `sync`: build indexes while writing SST files during flush and compaction. +## - `async`: build indexes in the background after the SST files are written. Queries read +## these files without indexes until the build finishes. +build_mode = "sync" + ## Cache size for inverted index metadata. metadata_cache_size = "64MiB" @@ -1081,7 +1136,9 @@ enable = true ## The record type of slow queries. It can be `system_table` or `log`. ## If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`. -## If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`. +## If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled, +## and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty. +## Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings. record_type = "system_table" ## The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`. @@ -1103,6 +1160,16 @@ ttl = "90d" [pipeline] ## Time to live of the local pipeline cache. Default is `10s`. cache_ttl = "10s" +## Allow VRL to access the server environment, filesystem, hostname, local timezone, +## network, and logs. Default is false. Enable only for trusted pipeline authors. +## Applies to pipeline creation, execution, and dry-runs. Requires a restart. +allow_vrl_host_access = false +## Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter. +## Explicit request values override this setting. Malformed input may be discarded, +## and invalid custom timestamps in identity ingestion fall back to server time. +## Pipeline execution errors are controlled separately by `skip_error`. +## Default is false. Requires a restart. +ignore_errors = false ## Configuration options for the event recorder. [event_recorder] diff --git a/docker/query-regression-helpers/Dockerfile b/docker/query-regression-helpers/Dockerfile new file mode 100644 index 00000000000..d1a93171695 --- /dev/null +++ b/docker/query-regression-helpers/Dockerfile @@ -0,0 +1,7 @@ +# Artifact image: binaries run on the query-regression ECS host after extraction. +# Build on the same ECS image as the consumer to preserve the runtime ABI. +FROM scratch +ARG SOURCE_REVISION +LABEL org.opencontainers.image.source="https://github.com/GreptimeTeam/greptimedb" +LABEL org.opencontainers.image.revision=$SOURCE_REVISION +COPY query_perf_fixture query_regression_runner / diff --git a/docs/how-to/metric-snapshot-s3-bench.md b/docs/how-to/metric-snapshot-s3-bench.md index 86468bf3ffd..7bd5fb2efed 100644 --- a/docs/how-to/metric-snapshot-s3-bench.md +++ b/docs/how-to/metric-snapshot-s3-bench.md @@ -1,14 +1,17 @@ # Metric snapshot benchmark on AWS S3 The manual `Metric Snapshot S3 Benchmark` workflow tests packed Metric export and -restore against real S3 in `ap-southeast-1`. It must first exist on the repository's -default branch for GitHub to register `workflow_dispatch`. It was validated on AWS +restore against real S3 in the configured CI bucket region. It must first exist on +the repository's default branch for GitHub to register `workflow_dispatch`. It was validated on AWS by run [38544464731](https://github.com/GreptimeTeam/greptimedb/actions/runs/38544464731); [#9573](https://github.com/GreptimeTeam/greptimedb/pull/9573) records its conditions and results. ## Run Use the repository's existing EC2 runner configuration and CI S3 credentials. +`AWS_CI_TEST_BUCKET_REGION` and `EC2_RUNNER_REGION` must be nonempty and equal; +for example, both can be `us-east-1`. The configured subnet, security group and +AMI (if explicitly supplied) must also belong to the runner region. The workflow passes credentials only through CI; no local AWS credentials are needed. Source, target and CLI use the same immutable image. This image's source SHA and required CLI capabilities have been checked: @@ -20,6 +23,7 @@ gh workflow run metric-snapshot-s3-bench.yml \ -f expected_commit=f690bbb2040865c0d22b0b57d51612b60ae27cd0 \ -f logical_tables_per_group=10000 \ -f benchmark_timeout_minutes=240 \ + -f ec2_hourly_usd=0.9408 \ -f prior_spend_usd=0 ``` @@ -28,9 +32,12 @@ earlier cloud attempts in the campaign, including failed setup and cleanup. Do not reset the estimate to zero when retrying. The workflow accepts 50–10,000 logical tables per group and a 30–240 minute benchmark job timeout. -The runner is fixed at `c6id.4xlarge` (16 vCPU, 32 GiB RAM). At the recorded -Singapore Linux on-demand estimate of $0.9408/hour, the default attempt reserves -$6.4392: four benchmark hours, 45 minutes of provisioning, 10 minutes for queueing, +The runner is fixed at `c6id.4xlarge` (16 vCPU, 32 GiB RAM). Set +`ec2_hourly_usd` to a positive, finite hourly budget estimate appropriate for the +selected region and instance. Its default $0.9408/hour preserves the historical +Singapore estimate; it is not a live regional price or a guaranteed upper bound. +The same estimate is used for preflight reservation and the final cost summary. +At that default rate, the default attempt reserves $6.4392: four benchmark hours, 45 minutes of provisioning, 10 minutes for queueing, 20 minutes of cleanup, and $1.50 for S3, disk and IPv4 costs. Preflight rejects prior spend at or above $15, or an attempt whose reserve would exceed the cumulative $20 budget. These estimates and checks are not an AWS hard spending cap or an AWS bill. diff --git a/docs/how-to/query-options.md b/docs/how-to/query-options.md new file mode 100644 index 00000000000..3fdaa55979d --- /dev/null +++ b/docs/how-to/query-options.md @@ -0,0 +1,18 @@ +# Query options + +Query options adjust selected behavior for a connection or request; they are not server-wide settings or resource-pool controls. + +| Canonical option | SQL aliases | Values | +| --- | --- | --- | +| `query.parallelism` | `query_parallelism`, `query.parallelism` | Integer `1`–`1024`; server default `0` means automatic CPU-based parallelism. This is not a resource-pool limit. | +| `query.allow_query_fallback` | `query_fallback`, `allow_query_fallback`, `query.allow_query_fallback` (legacy `ALLOW_QUERY_FALLBACK` syntax / `query_fallback` hint also work) | `true` or `false` | +| `query.enable_remote_dynamic_filter_pushdown` | canonical name | `true` or `false` | +| `datafusion.optimizer.*` | canonical name | Keys, types, defaults, and validation come from the linked DataFusion version (for example, numeric `max_passes` and Boolean `filter_null_join_keys`). | + +Use `SET query_parallelism = 4` or `SET query.parallelism TO 4`; `SHOW VARIABLES query.parallelism` works in MySQL and PostgreSQL, which also accepts `SHOW query.parallelism`. `SHOW VARIABLES