Merge upstream/main into narthex (pre-merge rehearsal, not for push)

Rehearsal merge of main (f8d8fe1071) into narthex (115d85b669) on a throwaway
local branch. 31 semantic conflicts resolved; workspace and all test targets
compile; promql 1215/1215 and mito2 series_index 185/185 pass.

Pre-merge pins and follow-ups are marked with TODO(narthex-merge):
- greptime-proto is temporarily pinned to the fork branch carrying
  GreptimeTeam/greptime-proto#355; replace with the upstream rev once it lands.
- native payload byte cache is disabled (main removed PageRangeCache); two
  samples tests are #[ignore]d until it gets a dedicated page-cache budget.
- columnar metadata now has its own CacheManager tier (a quarter of
  sst_meta_cache_size); the unified decoded tier was not kept.
- query.experimental_native_promql is registered as a query option, otherwise
  the native route is silently rejected by the option snapshot.

Signed-off-by: discord9 <discord9@163.com>

.agents/skills/greptimedb-release-note/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: greptimedb-release-note
-description: Generate a GreptimeDB release changelog with git cliff (correct range, subtract already-released patch PRs, rebuild contributors, add human-curated highlights), output to a file, and prepare the docs-repo blog PR. Use when asked to write/generate a GreptimeDB release note or changelog.
+description: Generate a GreptimeDB release changelog (minor, patch, or alpha/beta/rc pre-release) with git cliff (correct range, subtract already-released patch PRs, rebuild contributors, add human-curated highlights and upgrade notes), output to a file, and prepare the docs-repo blog PR. Use when asked to write/generate a GreptimeDB release note or changelog.
 ---
 
 # GreptimeDB Release Note / Changelog
@@ -27,41 +27,54 @@ a token it still runs, but may be rate-limited / incomplete.
 
 ## 2. Determine the previous version (skip nightlies)
 
-From `gh release list --repo GreptimeTeam/greptimedb`, ignore `*-nightly-*`, `-rc.*`, `-beta.*`,
-and build-suffixed tags; focus on formal `vX.Y.Z`.
+From `gh release list --repo GreptimeTeam/greptimedb`, ignore `*-nightly-*` and build-suffixed
+tags (e.g. `v1.3.0-alpha.1-3c0e2a8d5-20260928-1790639248`).
 - Patch `vX.Y.Z` (Z>0): previous = `vX.Y.(Z-1)`.
 - New minor `vX.Y.0`: previous = the latest `vX.(Y-1).*` (for `v1.0.0`, the biggest `0.x`).
+- Pre-release `vX.Y.Z-{alpha,beta,rc}.N`: previous = the latest pre-release tag of the same
+  `vX.Y.Z` (e.g. `v1.3.0-beta.1` for `v1.3.0-beta.2`). If there is none (the first alpha),
+  see "First pre-release" in §3: the previous minor's `.0` may not be published yet, and since
+  v1.2 a `.0` tag sits on `release/vX.Y`, not on `main`.
 Double-check with the user.
 
 ## 3. Pick the range and generate
 
-Key topology fact: a **minor tag** (e.g. `v1.0.0`) is an ancestor of `main`; **patch tags**
-(`v1.0.1`, `v1.0.2`) live on the `release/v1.0` branch and are **NOT** ancestors of `main`
-(they are cherry-picks with different SHAs). Verify with
+Key topology fact: **patch tags** (`v1.0.1`, `v1.0.2`) live on `release/vX.Y` and are **NOT**
+ancestors of `main` (they are cherry-picks with different SHAs). Up to v1.1 the `.0` tag was
+the point where `release/vX.Y` left `main`; from v1.2 the branch is cut before the `.0`, and
+pre-releases and the `.0` itself are tagged on the branch with backports. Verify with
 `git merge-base --is-ancestor <tag> <remote>/main`.
 
-### New minor (`X.Y.0`, cut from main)
-Note the two different tags here: the `git cliff` **base is the previous minor `.0` tag**
-(e.g. `v1.0.0`, an ancestor of `main`) — **not** the "previous version" from §2 (the latest
-patch, e.g. `v1.0.2`), which is only used below to decide which patch PRs to subtract.
+### New minor (`X.Y.0`)
+The note compares against the whole previous line: everything new since `release/vX.(Y-1)`
+left `main`, minus what any published `vX.(Y-1).*` release (pre-releases, `.0`, patches)
+already shipped. The line's own pre-releases (`vX.Y.0-beta.*`) are not subtracted; the `.0`
+note covers the whole minor.
 
-Base = the **previous minor tag** (e.g. `v1.0.0`); tip = the release commit (= `release/vX.Y`
-tip, usually `<remote>/main`):
+Base = the **previous minor tag** (e.g. `v1.1.0`); tip = the tagged `vX.Y.0` commit, resolved as
+in "Pre-release" below. Split the range at the point where `release/vX.Y` left `main`:
 ```
-GITHUB_TOKEN=$(gh auth token) git cliff <prev-minor-tag>..<release-commit> --tag vX.Y.0 \
-  -o /path/CHANGELOG-vX.Y.0.md
+bp=$(git merge-base <remote>/release/vX.Y <remote>/main)
 ```
-`cliff.toml`'s `ignore_tags` folds in-range nightly tags into the single section.
+- `<prev-minor-tag>..$bp` is on `main`; generate it with `git cliff`:
+  ```
+  GITHUB_TOKEN=$(gh auth token) git cliff <prev-minor-tag>..$bp --tag vX.Y.0 \
+    -o /path/CHANGELOG-vX.Y.0.md
+  ```
+  `cliff.toml`'s `ignore_tags` folds in-range nightly tags into the single section.
+- `$bp..<tip>` is the release branch: backports, squashed picks, and commits without a PR.
+  `git cliff` fails on such commits (`Filter call 'sort' failed`), so build this part as a
+  pre-release: collect the original PRs with the loop in "Pre-release" and add their bullets.
+  List the commits that carry no PR with
+  `git log --format='%h %s' $bp..<tip> | grep -vE '\(#[0-9]+\)$'`; drop mechanical ones, and
+  for a manual port of a `main` PR (the body says `cherry picked from commit` a SHA that is not
+  on `main`), find that PR by title and cite it.
 
-**Then subtract PRs already shipped in the intermediate patch releases** (`vX.(Y-1).1`,
-`.2`, …) — their main-branch commits are inside the range and would duplicate, and the
-audience cares about what's new vs the latest patch. Collect the PR set from the patch release
-bodies and remove matching lines:
-```
-gh release view vX.(Y-1).Z --repo GreptimeTeam/greptimedb | grep -oE 'pull/[0-9]+'
-```
-Remove every changelog bullet whose `#NNNN` is in that combined set (a small Python script is
-the reliable way — match `pull/<n>)` on lines starting with `*`).
+**Then subtract PRs already shipped in published `vX.(Y-1).*` releases**: their `main` commits
+are inside the range. Build the set as for the first pre-release (below): the PR links in
+every published `vX.(Y-1).*` release body, plus the `(#N)` line endings in the history of the
+newest published `vX.(Y-1).*` tag. Remove every changelog bullet whose `#NNNN` is in that set (a
+small Python script is the reliable way — match `pull/<n>)` on lines starting with `*`).
 
 ### Patch (`X.Y.Z`, Z>0, cherry-picks on the release branch)
 The previous patch tag is an ancestor of the release branch, so a plain range works **when the
@@ -99,11 +112,78 @@ user which they prefer**):
 If the branch history is otherwise messy, the same "identify the picked PRs and keep only those"
 rule applies.
 
+### First pre-release (`X.Y.0-alpha.1`, cut from main)
+The base is the commit where the previous line's release branch left `main`. It exists when the
+alpha is cut, even if `vX.(Y-1).0` does not:
+```
+base=$(git merge-base <remote>/release/vX.(Y-1) <tag>)
+```
+If there is no `release/vX.(Y-1)`, use the newest published release tag that is an ancestor of
+the tag (`git merge-base --is-ancestor <candidate> <tag>`). Run `git cliff <base>..<tag>`, then
+subtract every PR that already shipped in a published `vX.(Y-1).*` release, pre-releases
+included. Take the union of the PR links in those release bodies and the `(#N)` line endings in
+the history of the newest published `vX.(Y-1).*` tag:
+```
+gh release view <published-vX.(Y-1)-tag> --repo GreptimeTeam/greptimedb | grep -oE 'pull/[0-9]+'
+git log <newest-published-vX.(Y-1)-tag> --format=%B | grep -oE '\(#[0-9]+\)$'
+```
+Either source alone misses PRs: a release body can omit a backport that its tag contains, and a
+backport commit can name only the backport PR.
+
+### Pre-release (`X.Y.Z-{alpha,beta,rc}.N` after an earlier pre-release)
+**The tip is the tagged commit, not the release branch tip.** Resolve it from the remote and
+write this SHA in the docs PR body (§8) as what the note was checked against:
+```
+git ls-remote --tags <remote> 'vX.Y.Z-beta.N*'
+```
+Use the `refs/tags/vX.Y.Z-beta.N` line (its `^{}` line if the tag is annotated), not the
+build-suffixed tags it also lists. PRs that land on `release/vX.Y` after the tag
+(`git log --oneline <tag>..<remote>/release/vX.Y`) belong to the next release, even if they are
+already merged when you write the note.
+
+The range `<prev-pre-release>..<tag>` usually mixes two kinds of commits: individual backports
+(`... [Backport release/vX.Y] (#NNNN)`) and squashed pick commits (`chore: backport selected
+changes for ... (#NNNN)`). In both, the original PRs are the `(#NN)` at the end of body lines;
+a backport commit can also carry prerequisite PRs. Collect the PR set, falling back to the
+subject for commits whose body lists none:
+```
+for c in $(git rev-list <prev-pre-release>..<tag>); do
+  prs=$(git log -1 --format=%b "$c" | grep -oE '\(#[0-9]+\)$')
+  [ -n "$prs" ] || prs=$(git log -1 --format=%s "$c" | grep -oE '\(#[0-9]+\)$')
+  echo "$prs"
+done | tr -dc '0-9\n' | sort -un
+```
+List the bullets under the **original** PRs (title and author of the PR merged to `main`), not
+the backport PRs. Build them with patch (a) or (b) above; for (b), fetch
+`gh api repos/GreptimeTeam/greptimedb/pulls/<n> --jq '{number,title,author:.user.login,merged_at}'`,
+group by the `cliff.toml` parsers (first match wins), order by `merged_at`, and put `!` titles
+under Breaking changes (§6).
+
+A squashed pick PR can also contain release-only commits that have no PR. List them:
+```
+gh api repos/GreptimeTeam/greptimedb/pulls/<squash-pr>/commits --paginate \
+  --jq '.[].commit.message | split("\n")[0]' | grep -vE '\(#[0-9]+\)$'
+```
+Mechanical ones (version bump, toolchain or lint adaptations; example:
+`fix: adapt release OptionMap iteration to Rust 1.98`) get no bullet. A release-only
+commit that changes behavior gets a bullet in its group.
+
 > Flaky-network note for (a): `git cliff` fetches the repo's full commit history from the GitHub
 > API for author enrichment (paginates back thousands of commits, even for a tiny range). On a
 > flaky connection it can time out mid-pagination; it caches pages, so just **re-run until it
 > completes** — each retry resumes from the cache. If it keeps failing, fall back to (b).
 
+### Groups (all paths)
+`git cliff` drops PRs whose title isn't a conventional commit (`filter_unconventional`). List
+them and add the user-facing ones by hand:
+```
+git log --format=%s <range> | grep -E '\(#[0-9]+\)$' | grep -vE '^[a-z]+(\([^)]*\))?!?: '
+```
+Use only the groups in `cliff.toml`. `git cliff` prints a `### Build` section for `build:` commits
+because no parser matches them; move a user-facing `build(deps)` fix to Bug Fixes and the rest to
+Miscellaneous Tasks. Keep each bullet list free of blank lines; a blank line turns it into a
+loose list that renders with different spacing.
+
 ## 4. Rebuild contributor sections after subtracting
 
 `cliff` computes `New Contributors` / `All Contributors` over the full range, so after removing
@@ -114,6 +194,21 @@ lines, recompute from the **remaining** commit bullets:
   in-range PR shipped in a patch).
 Do this in the same script that subtracts the PRs.
 
+When the bullets are built by hand (§3 (b), pre-releases), `git cliff` computes no New
+Contributors, and GitHub's generated notes don't either (they only see the backport PRs). An
+author is new if none of their merged PRs shipped before, i.e. none appears in the history of the
+base tag:
+```
+git log <base-tag> --format=%B | grep -oE '\(#[0-9]+\)$' | tr -dc '0-9\n' | sort -u > /tmp/shipped.txt
+gh search prs --repo GreptimeTeam/greptimedb --author <user> --merged --limit 1000 \
+  --json number --jq '.[].number' | sort -u > /tmp/author.txt
+comm -12 /tmp/author.txt /tmp/shipped.txt   # empty => new contributor
+```
+Run it for every non-bot author in the release, and cite their earliest-merged PR in this
+release. "Earliest merged PR is in range" is not the same test: a contributor whose first PR
+merged on `main` but was not picked into any release is still new when a later PR of theirs
+ships.
+
 ## 5. Verify
 
 Cross-check the result against `git log <base>..<tip>`: no subtracted PR remains, and no
@@ -122,17 +217,71 @@ line (ask the user; some prefer to keep it).
 
 ## 6. Human-curated sections (insert after the `Release date:` line)
 
+**Style.** Follow the Google developer documentation style guide: sentence-case headings, active voice, present
+tense for behavior, American spelling, serial comma, code font for identifiers, bold only for UI
+names. Keep it short:
+- Intro: two sentences at most, what the release covers and its main theme.
+- Highlight: one paragraph of up to four sentences, plus an example if it earns one (see below).
+- Upgrade note or breaking change: one bullet, one or two sentences: what changes, who is
+  affected, what to do.
+- Every sentence states a fact the reader acts on or checks. Delete sentences that restate the
+  heading or the previous sentence, transitions, and summaries.
+- No marketing words (`significantly`, `powerful`, `seamless`, `robust`, `enhanced`), no vague
+  verbs (`supports`, `handles`, `improves`; say what happens), no `several`/`many` when the number
+  is known. Keep the source's hedges; don't add new ones.
+
+Example:
+> Bad: This release significantly enhances write reliability by introducing a robust retry
+> mechanism that seamlessly handles transient failures.
+>
+> Good: In distributed clusters, the frontend retries a failed write up to three times with
+> exponential backoff. Set `max_retries = 0` to disable retries.
+
 Mirror past release bodies (`gh release view v1.0.0 --repo GreptimeTeam/greptimedb`):
-- **Short intro**, terse, engineer-written tone (no marketing adjectives).
-- **`### 👍 Highlights`** — *few, deep* highlights, each with a **working example**
-  (SQL / TOML config). To get examples right, **read the highlight PR and the related docs
-  PR/page** in the docs repo (`GreptimeTeam/docs`, often checked out locally) — verify syntax
-  in `docs/reference/sql/*.md`, `config/*.example.toml`, `config/config.md`. Do **not** mention
-  implementation details, tiny features, or unfinished/experimental-but-not-ready features.
-  Always have the user review and edit the highlights; iterate.
+- **Short intro** (see Style).
+- **`### 👍 Highlights`** — *few, deep* highlights, ranked by user impact. Fixes for silently
+  wrong query results, lost or invisible data, and security issues rank with new features and
+  can be highlights.
+  - Write a fix as its symptom and the affected deployments or usage, not the PR title. Example:
+    "with no traffic, `errors / requests > 0.05` was true, so the alert fired".
+  - State whether a feature is on by default. If an example config uses the default values, say
+    so.
+  - Give an example (SQL / TOML config) only where it helps readers act. Don't pick highlights,
+    or the example, by what is easy to demonstrate.
+  - One highlight covers one change or one user-visible problem. Don't merge unrelated PRs into
+    one entry because they touch the same component.
+  - Sources: the PR body's sections on impact, compatibility, and behavior changes, plus the
+    code and config it touches, not the title alone. Verify examples against the related docs
+    PR/page in the docs repo (`GreptimeTeam/docs`, often checked out locally),
+    `docs/reference/sql/*.md`, `config/*.example.toml`, `config/config.md`.
+  Do **not** mention implementation details, tiny features, or unfinished/experimental-but-not-ready
+  features. Always have the user review and edit the highlights; iterate.
 - **Dashboard** subsection — don't just bump the version. Read the dashboard PRs in the bundled
   release (`gh release view <ver> --repo GreptimeTeam/dashboard`; then the linked PRs) and
-  describe the user-facing change.
+  describe the user-facing change. Use the names the UI shows, not the PR titles; check them in
+  the dashboard's locale strings at the bundled tag:
+  ```
+  gh api 'repos/GreptimeTeam/dashboard/contents/src/locale/en-US/menu.ts?ref=<dashboard-tag>' \
+    --jq .content | base64 -d
+  ```
+  Screenshots need absolute `https://` URLs: the body is copied to the GitHub release, where
+  relative and site-root (`/img/...`) paths don't resolve.
+- **`### Upgrade notes`** and **`### Breaking changes`** — check every PR in the release for:
+  `!` titles; `Compatibility`, `Behavior change`, `Breaking` paragraphs in the body (often a
+  plain `Compatibility:` line, not a heading); unchecked "API changes are backward compatible" /
+  "Schema or data changes are backward compatible" boxes; new config options and their defaults;
+  changed defaults; persisted format changes; rolling-upgrade order; new hard limits. These
+  signals set reading order, not a filter: a limit can sit in plain prose with every box
+  checked, so read the body of every user-facing `feat`/`fix` PR. Examples of what is easy to
+  miss: a 512 MiB cap on decompressed request bodies (#9264); a default step for stepless
+  subqueries (#9481).
+  - **Upgrade notes**: steps operators take — upgrade order, index rebuilds, settings to review.
+  - **Breaking changes**: behavior that used to work and now errors or returns different
+    results. Put the `!` PR bullets here too, after these entries (see `v1.3.0-beta.1`).
+- **Benchmark results** (optional) — state the baseline version (often the previous formal
+  release, not the previous beta), data size, cache state, and hardware. Don't attribute results
+  to single PRs. Leave out results the benchmark marks as not for publication, and smoke-level
+  runs.
 
 ## 7. Output
 
@@ -144,7 +293,8 @@ it**. The release runbook deletes it after the release + docs PR are done.
 The release note is also published as a blog post in **`GreptimeTeam/docs`**.
 - **Ask the user for their local `GreptimeTeam/docs` checkout path**; if they have none, offer
   to clone (`git clone git@github.com:GreptimeTeam/docs.git <path>`).
-- File: `blog/release-X-Y-Z.md` (version with dashes; see `blog/release-1-0-0.md`).
+- File: `blog/release-X-Y-Z.md` (version with dashes; see `blog/release-1-0-0.md`). Pre-releases
+  use dashes throughout: `blog/release-1-3-0-beta-2.md`.
 - Content = docs frontmatter (below) followed by the GitHub release body, **keeping the
   `# vX.Y.Z` H1 directly under the frontmatter**. The blog frontmatter has no `title:`, so
   Docusaurus uses that H1 as the page title and sidebar label — omitting it makes the page
@@ -156,11 +306,13 @@ The release note is also published as a blog post in **`GreptimeTeam/docs`**.
   description: GreptimeDB vX.Y.Z Changelog
   date: YYYY-MM-DD
   ---
-
   # vX.Y.Z
 
   Release date: ...
   ```
+  The docs CI check `check-release-note-sync` requires the post minus its frontmatter to equal
+  the GitHub release body byte for byte, so the H1 goes on the line right after the closing
+  `---`, with no blank line.
 - The docs working tree may have unrelated WIP — **don't disturb it**: use a `git worktree` off
   `origin/main`:
   ```
@@ -169,8 +321,19 @@ The release note is also published as a blog post in **`GreptimeTeam/docs`**.
   ```
 - **Read and follow the current docs repo PR template**
   (`.github/pull_request_template.md`) from the checkout used for the release.
-  Fill in every required section and leave reviewer-owned checklist boxes unchecked;
-  do not rely on hard-coded section names from an older template.
+  Fill in every required section; do not rely on hard-coded section names from an older
+  template. The checklist items are the author's to confirm: check the ones you did, and for one
+  that doesn't apply, check it and say why. For pre-releases, state the tagged SHA (§3) under
+  Verification.
+- **Build the site before opening the PR.** A new worktree has no `node_modules`. Install with
+  the pnpm version pinned in `package.json` (`packageManager`), as CI does:
+  ```
+  cd /tmp/docs-release-note
+  pm=$(jq -r .packageManager package.json)   # e.g. pnpm@8.6.0
+  npx -y "$pm" install --frozen-lockfile
+  npx -y "$pm" build
+  ```
+  If the build fails, fix it or stop; don't mark the PR ready for review.
 - Commit with sign-off, push, open a **draft** PR, then remove the worktree:
   ```
   git -C /tmp/docs-release-note add blog/release-X-Y-Z.md

.github/actions/build-linux-artifacts/action.yml

@@ -50,6 +50,11 @@ inputs:
 runs:
   using: composite
   steps:
+    - name: Set up Linux runner
+      uses: ./.github/actions/setup-linux-runner
+      with:
+        tools: make shasum
+
     - name: Run integration test
       if: ${{ inputs.disable-run-tests == 'false' }}
       shell: bash

.github/actions/fuzz-test/action.yaml

@@ -28,7 +28,38 @@ inputs:
 runs:
   using: composite
   steps:
+    - name: Prepare Fuzz Corpus
+      id: corpus
+      shell: bash
+      env:
+        FUZZ_TARGETS: ${{ inputs.targets }}
+      run: bash .github/scripts/prepare-fuzz-corpus.sh
+
+    - name: Restore Fuzz Corpus
+      id: restore-corpus
+      uses: actions/cache/restore@v4
+      continue-on-error: true
+      with:
+        path: ${{ steps.corpus.outputs.paths }}
+        key: ${{ steps.corpus.outputs.key-prefix }}${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
+        restore-keys: ${{ steps.corpus.outputs.key-prefix }}
+
+    - name: Report Fuzz Corpus Cache
+      shell: bash
+      env:
+        CACHE_OUTCOME: ${{ steps.restore-corpus.outcome }}
+        CACHE_MATCHED_KEY: ${{ steps.restore-corpus.outputs.cache-matched-key }}
+      run: |
+        if [[ "${CACHE_OUTCOME}" == failure ]]; then
+          echo '::warning::Fuzz corpus restore failed; running with available inputs'
+        elif [[ -n "${CACHE_MATCHED_KEY}" ]]; then
+          printf 'Restored fuzz corpus: %s\n' "${CACHE_MATCHED_KEY}"
+        else
+          echo 'No cached fuzz corpus found; starting with an empty corpus'
+        fi
+
     - name: Run Fuzz Tests
+      id: run-fuzz
       shell: bash
       env:
         FUZZ_TARGETS: ${{ inputs.targets }}
@@ -39,4 +70,13 @@ runs:
         FUZZ_COLLECT_CLUSTER_ARTIFACTS: ${{ inputs.collect-cluster-artifacts }}
         FUZZ_ARTIFACT_ROOT: ${{ inputs.artifact-root }}
         FUZZ_BIN_DIR: ${{ inputs.prebuilt-binaries-dir }}
+        FUZZ_CORPUS_ROOT: ${{ github.workspace }}/tests-fuzz/corpus
       run: .github/scripts/run-fuzz-targets.sh
+
+    - name: Save Fuzz Corpus
+      if: ${{ !cancelled() && (steps.run-fuzz.outcome == 'success' || steps.run-fuzz.outcome == 'failure') }}
+      uses: actions/cache/save@v4
+      continue-on-error: true
+      with:
+        path: ${{ steps.corpus.outputs.paths }}
+        key: ${{ steps.corpus.outputs.key-prefix }}${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}

.github/actions/github-script-tests/action.yml

@@ -8,7 +8,7 @@ runs:
         node-version: 24
     - name: Run Node GitHub script tests
       shell: bash
-      run: node --test .github/scripts/query-regression-comment.test.cjs .github/scripts/heavy-ci-policy.test.cjs
+      run: node --test .github/scripts/tests/ci-cache-policy.test.cjs .github/scripts/query-regression-comment.test.cjs .github/scripts/heavy-ci-policy.test.cjs .github/scripts/tests/nightly-merge.test.cjs
     - name: Setup uv
       uses: astral-sh/setup-uv@v3
       with:
@@ -44,6 +44,7 @@ runs:
         python3 tests/perf/test_query_regression_runner_otlp_trace_load.py
         python3 tests/perf/test_query_regression_summary_otlp.py
         python3 tests/perf/test_query_regression_case_selection.py
+        python3 tests/perf/test_query_regression_helpers.py
         python3 tests/perf/test_query_regression_nightly_refs.py
         python3 tests/perf/test_query_regression_slash.py
         python3 tests/perf/test_aliyun_ecs_runner_scripts.py

.github/actions/setup-linux-runner/action.yml

@@ -1,19 +1,41 @@
-name: Set up benchmark dependencies
+name: Set up Linux runner dependencies
 description: Reuse installed tools and install missing dependencies on an Ubuntu self-hosted runner
+inputs:
+  tools:
+    description: Space-separated commands to install if missing (Ubuntu package names, except shasum)
+    required: false
+    default: curl
+  expected-architecture:
+    description: Optional uname architecture required by the workload
+    required: false
+    default: ''
 runs:
   using: composite
   steps:
     - name: Check host and install missing tools
       shell: bash
+      env:
+        REQUIRED_TOOLS: ${{ inputs.tools }}
+        EXPECTED_ARCH: ${{ inputs.expected-architecture }}
       run: |
         set -euo pipefail
-        [[ "$(uname -m)" == x86_64 ]]
+        if [[ -n "$EXPECTED_ARCH" ]]; then
+          [[ "$(uname -m)" == "$EXPECTED_ARCH" ]]
+        fi
         command -v git
         missing=()
-        for tool in curl jq python3 rsync; do
-          command -v "$tool" >/dev/null || missing+=("$tool")
+        read -r -a tools <<< "$REQUIRED_TOOLS"
+        for tool in "${tools[@]}"; do
+          if ! command -v "$tool" >/dev/null; then
+            case "$tool" in
+              shasum) missing+=(libdigest-sha-perl) ;;
+              *) missing+=("$tool") ;;
+            esac
+          fi
         done
         if ((${#missing[@]})); then
+          . /etc/os-release
+          [[ "$ID" == ubuntu ]]
           sudo -n apt-get update
           sudo -n env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get install -y --no-install-recommends "${missing[@]}"
         fi
@@ -24,6 +46,8 @@ runs:
         if ! command -v docker >/dev/null; then
           . /etc/os-release
           [[ "$ID" == ubuntu ]]
+          sudo -n apt-get update
+          sudo -n env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get install -y --no-install-recommends curl ca-certificates
           sudo -n install -m 0755 -d /etc/apt/keyrings
           curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo -n tee /etc/apt/keyrings/docker.asc >/dev/null
           sudo -n chmod a+r /etc/apt/keyrings/docker.asc

.github/actions/start-runner/action.yml

@@ -20,6 +20,9 @@ inputs:
   image-id:
     description: The EC2 image id
     required: true
+  ec2-volume-size:
+    description: Root volume size in GiB (uses the AMI default when omitted)
+    required: false
   security-group-id:
     description: The EC2 security group id
     required: true
@@ -61,6 +64,7 @@ runs:
       with:
         mode: start
         ec2-image-id: ${{ inputs.image-id }}
+        ec2-volume-size: ${{ inputs.ec2-volume-size }}
         ec2-instance-type: ${{ steps.get-ec2-instance-type.outputs.instance-type }}
         subnet-id: ${{ inputs.subnet-id }}
         security-group-id: ${{ inputs.security-group-id }}

.github/ci/cache-policy.md

@@ -0,0 +1,45 @@
+# CI cache policy
+
+Checks, Rust CI, Integration CI, and Nightly CI use one explicit Rust cache per
+build job. Disable the implicit cache in `setup-rust-toolchain` when adding an
+explicit cache: otherwise its independent save policy can write PR-scoped
+caches even when the explicit cache only saves on main.
+
+Integration CI's existing weekday schedule runs the fuzz builds and suites,
+including the profile-CI binary and Kafka helper, to populate main caches.
+To warm these caches manually after deployment, run Integration CI on main with
+`fuzz_profile: all`; no internal `ci_command` marker is required.
+
+Rust CI's existing weekday schedule warms the Windows and RISC-V compilation
+caches. Nightly CI's existing schedule warms its Windows runtime test caches.
+A manual Rust CI run on main can warm the compilation caches sooner. These jobs
+keep their separate keys and existing runner types; no extra warmup job is added.
+Coverage caches are not warmed by these schedules.
+PR, merge-group, and non-main manual runs restore caches without saving them.
+Fuzz binaries still flow to downstream jobs through workflow artifacts on every
+run; only the persistent binary cache is main-written. Re-running a PR cannot
+reuse a newly saved PR-specific binary cache under this policy.
+
+Keep helper version variables outside the `CARGO*` and `RUST*` prefixes unless
+they affect the compilation being cached. rust-cache includes those prefixes in
+its environment hash. Fuzz tool versions remain explicitly included in the fuzz
+cache keys. Compiler versions, Cargo manifests, flags, OS, and architecture
+remain part of the existing cache isolation.
+
+This policy reduces duplicate and PR-only writes; it does not guarantee that all
+build caches fit in the repository quota. Renamed environment variables cause
+an initial cold restore until main produces the new keys. Existing caches are
+not deleted by this change. Coverage warming and additional runner capacity are
+not introduced.
+
+Validate the policy with:
+
+```sh
+node --test .github/scripts/tests/ci-cache-policy.test.cjs
+```
+
+After deployment, compare the same commit and runner image across repeated runs.
+Record exact versus fallback restores, archive size, restore/save duration, and
+compilation duration. Verify main writes the keys consumed by PRs and that no
+implicit post-action uploads remain. Local policy tests do not establish a
+runtime speedup or long-term cache hit rate.

.github/ci/scope-rules.toml

@@ -32,8 +32,11 @@ always_run = [
   ".github/ci/**",
   ".github/workflows/integration.yml",
   ".github/actions/**",
-  # These helpers run through composite actions and affect test execution.
+  # Both run through a composite action, so neither appears in the workflow:
+  # `setup-kind` starts the cluster for the distributed and chaos jobs, and
+  # `fuzz-test` seeds the corpus every fuzz job starts from.
   ".github/scripts/kind-with-registry.sh",
+  ".github/scripts/prepare-fuzz-corpus.sh",
   ".github/scripts/run-fuzz-targets.sh",
   ".github/scripts/collect-fuzz-target-artifacts.sh",
   ".github/scripts/wait-kafka-ready.sh",
@@ -100,6 +103,7 @@ server = [
   "src/datanode/**",
   "src/common/base/**",
   "src/common/batcher/**",
+  "src/common/cache/**",
   "src/common/catalog/**",
   "src/common/config/**",
   "src/common/datasource/**",
@@ -137,6 +141,7 @@ engine = [
   "src/mito-codec/**",
   "src/metric-engine/**",
   "src/file-engine/**",
+  "src/relational-engine/**",
   "src/log-store/**",
   "src/common/wal/**",
 ]

.github/ci/tests/fixtures/scope-cases.toml

@@ -150,9 +150,9 @@ paths = [
 suites = ["*"]
 
 [[case]]
-name = "fuzz target orchestration: every suite"
-note = "Fuzz jobs execute targets through the `fuzz-test` action."
+name = "fuzz corpus seeding: every suite"
+note = "Every fuzz job seeds its corpus through the `fuzz-test` action first."
 paths = [
-  ".github/scripts/run-fuzz-targets.sh",
+  ".github/scripts/prepare-fuzz-corpus.sh",
 ]
 suites = ["*"]

.github/heavy-ci.md

@@ -5,8 +5,8 @@ action before expensive builds and tests. A failed, skipped, or cancelled
 script-test job blocks those jobs, including manually dispatched runs. Each
 workflow runs its own preflight so it does not wait on another workflow run.
 
-Rust CI runs Windows and RISC-V compilation checks only on schedule or manual
-dispatch, not on PRs or merge groups. Windows nextest and Windows SQLness run in the existing Nightly CI workflow, which also
+Rust CI keeps the Windows compilation check on PRs and merge groups. Windows
+nextest and Windows SQLness run in the existing Nightly CI workflow, which also
 supports manual dispatch. Its existing schedule is Tuesday through Friday at
 07:00 Asia/Shanghai, not every day. No new Windows test job is added.
 
@@ -26,12 +26,3 @@ for same-repository draft PRs with their existing authorization rules.
 
 Validate job admission with `node --test .github/scripts/heavy-ci-policy.test.cjs`.
 These tests verify triggering decisions, not runner availability or performance.
-
-## Narthex branch
-
-GitHub schedules run only on the default branch. They do not test `narthex`
-automatically. To validate this branch, manually dispatch Rust CI, Integration
-CI (selecting a fuzz profile), or Nightly CI with `narthex` as the ref.
-
-Land these scope rules on `narthex` before updating dependent PR branches:
-the classifier checks out the PR base revision, not the PR head.

.github/nightly-merge.md

@@ -0,0 +1,49 @@
+# Nightly merge
+
+The PR template opts new PRs into the `merge nightly` label. The Nightly Merge
+workflow reconciles the label on PR creation, edits, and reopening. Existing PRs
+without the checkbox can opt in by adding the label manually; label sync leaves
+their labels unchanged. Unchecking removes the label; it does not
+request an immediate merge or cancel an existing queue entry/auto-merge request.
+
+Every day at 02:10, 03:10, 04:10, 05:10, 06:10, 07:10, and 08:10
+Asia/Shanghai (UTC+8), the workflow enqueues up to five eligible PRs targeting
+`main` per run, oldest first. Already queued PRs do not count toward this limit.
+GitHub may delay scheduled runs.
+The label alone controls nightly opt-in; the enqueue job does not check the PR
+body. PRs must be approved and clean, and have all
+review threads resolved. Drafts are skipped.
+If GitHub rejects enqueueing because a PR belongs to a stack, the workflow skips
+it and continues without failing the run or counting it toward the five-PR limit.
+Other API errors still fail the run. Dry runs do not call the enqueue API, so
+they cannot detect this stack rejection.
+Already queued PRs and manually enabled auto-merge requests are left alone;
+Maintainers can use **Merge when ready** to proceed earlier. Unready PRs wait for a later
+scheduled run; this workflow never enables auto-merge on their behalf.
+
+## Repository setup
+
+- Create the `merge nightly` label before enabling the workflow.
+- Require the merge queue, reviews, status checks, and conversation resolution
+  on `main`. Conversation resolution must also be enforced by GitHub to cover
+  new comments arriving between the script's check and enqueueing.
+- Ensure the existing `GH_PERSONAL_ACCESS_TOKEN` automation secret can read PRs
+  and review threads and enqueue PRs in this repository. Do not grant it a
+  branch-rule bypass. This credential is already referenced by other workflows;
+  its actual permissions have not been validated by the local tests.
+- Run **Nightly Merge** manually with the default `dry_run: true` first and
+  inspect its job summary. Setting it to false explicitly enqueues outside the
+  scheduled time as well. Merge queue build concurrency is configured separately
+  in GitHub; this workflow does not change it.
+
+The workflow never checks out or executes PR code. Label reconciliation uses
+`GITHUB_TOKEN`; enqueueing uses the automation credential so downstream CI can
+run. No runner remains allocated while waiting for the nightly schedule.
+
+## Local validation
+
+Run `actionlint .github/workflows/nightly-merge.yml .github/workflows/checks.yml`
+and `node --test .github/scripts/tests/nightly-merge.test.cjs`.
+The tests execute the actual inline workflow scripts with mocked GitHub APIs,
+including one-factor comparisons against an eligible PR. They do not validate
+live GitHub permissions, branch rules, or runner savings.

.github/pull_request_template.md

@@ -25,3 +25,14 @@ Please convert it to a draft if some of the following conditions are not met.
 - [ ] API changes are backward compatible.
 - [ ] Schema or data changes are backward compatible.
 - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`).
+
+## Merge timing
+
+- [x] Merge during the nightly window <!-- nightly-merge -->
+
+<!--
+Eligible PRs enter the merge queue hourly from 02:10 through 08:10 Beijing time (UTC+8), up to five per run.
+Uncheck to opt out and remove the `merge nightly` label; this does not enable immediate merging.
+Maintainers can use **Merge when ready** to merge earlier.
+Changing this checkbox does not cancel an existing queue entry or manually enabled auto-merge.
+-->

.github/runner-scale-sets/query-regression/README.md

@@ -80,8 +80,13 @@ after its TTL, so finish the inspection within that window.
 Trust model on the ECS path: the instance receives only the one-hour runner
 registration token via user data and holds no cloud credentials; the Aliyun
 AK/SK exist only in the control-plane jobs. Instance tags
-(`managed-by=query-regression-ci`, `query-regression-run-id`) feed the janitor
-sweep.
+(`managed-by=query-regression-ci`, `query-regression-run-id`,
+`github-repository`) feed the janitor sweep. The sweep only selects instances
+whose `github-repository` matches the requested repository. Older instances
+without that tag are excluded: verify their ownership before adding the tag,
+or clean them up by explicit instance ID. Per-run teardown remains unchanged.
+Update every repository sharing the account and region to the scoped helper;
+an older janitor can still sweep instances belonging to other repositories.
 
 ### Building and updating the ECS image
 

.github/scripts/agent-observability-summary.py

@@ -133,9 +133,10 @@ def benchmark(root, targets):
     return '\n'.join(lines)
 
 
-def lifecycle(root, targets):
-    manifest = read(root / 'run-manifest.json')
-    corpus = read(root / 'corpus' / 'summary.json')
+def lifecycle(root, targets, data_root=None):
+    data_root = data_root or root
+    manifest = read(data_root / 'run-manifest.json')
+    corpus = read(data_root / 'corpus' / 'summary.json')
     lines = ['# Load / lifecycle summary', '',
              f'Dataset: **{cell(manifest.get("profile", corpus.get("data_profile", {}).get("id")))}**; rows: **{cell(corpus.get("line_count"))}**; '
              f'seed: `{cell(corpus.get("seed"))}`.',
@@ -161,12 +162,13 @@ def lifecycle(root, targets):
     physical_rows = []
     load_rows = []
     details = ['<details>', '<summary>DDL, index configuration and image provenance</summary>', '']
-    raw_bytes = read(root / 'corpus' / 'size.json').get('raw_jsonl_bytes')
+    raw_bytes = read(data_root / 'corpus' / 'size.json').get('raw_jsonl_bytes')
     for t in targets:
         directory = result_dir(root, t)
         physical = read(directory / 'physical-evidence.json')
         storage = physical.get('storage', {})
-        total = storage.get('engine_total_bytes')
+        s3 = read(root / f'{t}-1' / 'storage.json').get('type') == 's3'
+        total = None if s3 else storage.get('engine_total_bytes')
         stats = read(root / f'{t}-1' / 'post-load-docker-stats.json')
         ratio = f'{raw_bytes / total:.2f}×' if (physical.get('passed') is True and
                  isinstance(raw_bytes, (int, float)) and raw_bytes > 0 and
@@ -186,6 +188,7 @@ def lifecycle(root, targets):
     lines += ['## Load and physical evidence', '', table(['DB', 'Write transport', 'Batches', 'Load smoke rows'], load_rows), '',
               f'Raw JSONL size: **{size(raw_bytes)}**. Ratio = raw JSONL / engine bytes at collection time; '
               'not a settled-storage guarantee. CPU/memory are post-load snapshots, not peaks. '
+              'S3 total/compression is omitted: local files exclude remote objects. '
               'Missing artifacts remain blank; pre-cleanup stats are not substituted.', '',
               table(['DB', 'Engine size', 'Size ratio', 'Materialization', 'SSTs / active parts',
                      'Post-load CPU', 'Post-load memory / limit'], physical_rows), '', *details, '</details>', '']

.github/scripts/aliyun-ecs-runner-provision.py

@@ -59,6 +59,7 @@ MANAGED_BY_TAG_KEY = "managed-by"
 MANAGED_BY_TAG_VALUE = "query-regression-ci"
 RUN_TAG_KEY = "query-regression-run-id"
 TTL_TAG_KEY = "runner-ttl-hours"
+REPO_TAG_KEY = "github-repository"
 
 # Runner cache paths on the instance system disk. They are created empty
 # every provision and deleted with the VM.
@@ -359,6 +360,7 @@ def run_instance(client, config: ProvisionConfig, user_data: str) -> str:
                 key=MANAGED_BY_TAG_KEY, value=MANAGED_BY_TAG_VALUE
             ),
             ecs_models.RunInstancesRequestTag(key=RUN_TAG_KEY, value=config.run_id),
+            ecs_models.RunInstancesRequestTag(key=REPO_TAG_KEY, value=config.repo),
         ]
         + (
             [

.github/scripts/aliyun-ecs-runner-teardown.py

@@ -32,8 +32,8 @@ Two modes:
   runner by name. Both steps are idempotent and best-effort; a missing
   instance or runner is not an error.
 - Sweep (scheduled janitor): delete every instance tagged as managed by
-  query-regression CI whose creation time is older than its tagged TTL (or
-  the given fallback TTL for untagged instances), and
+  query-regression CI in the requested repository whose creation time is older
+  than its tagged TTL (or the fallback TTL for instances without a TTL tag), and
   deregister the matching runners. This is the safety net for runs whose
   teardown job never executed.
 """
@@ -162,7 +162,7 @@ def delete_instance(client, instance_id: str, region_id: str | None = None) -> b
 
 
 def list_managed_instances(
-    client, region_id: str
+    client, region_id: str, repo: str
 ) -> list[tuple[str, str, str, str | None]]:
     from alibabacloud_ecs20140526 import models as ecs_models
 
@@ -175,6 +175,9 @@ def list_managed_instances(
                 ecs_models.DescribeInstancesRequestTag(
                     key=provision.MANAGED_BY_TAG_KEY,
                     value=provision.MANAGED_BY_TAG_VALUE,
+                ),
+                ecs_models.DescribeInstancesRequestTag(
+                    key=provision.REPO_TAG_KEY, value=repo
                 )
             ],
             max_results=100,
@@ -201,7 +204,7 @@ def list_managed_instances(
 
 
 def sweep(client, region_id: str, repo: str, github_token: str, ttl: timedelta) -> int:
-    instances = list_managed_instances(client, region_id)
+    instances = list_managed_instances(client, region_id, repo)
     print(f"Found {len(instances)} managed instance(s) in {region_id}", flush=True)
     expired = expired_instance_names(instances, datetime.now(timezone.utc), ttl)
     ok = True
@@ -227,7 +230,7 @@ def main() -> int:
     parser.add_argument(
         "--sweep",
         action="store_true",
-        help="Janitor mode: delete all managed instances older than --sweep-ttl-hours.",
+        help="Janitor mode: delete expired managed instances tagged for --repo.",
     )
     parser.add_argument("--sweep-ttl-hours", type=float, default=SWEEP_TTL.total_seconds() / 3600)
     args = parser.parse_args()

.github/scripts/benchmark-version-comparison.py

@@ -62,11 +62,12 @@ def row(query, metric, old, new, compatible, reason):
                 improvement_percent=delta, status='comparable' if delta is not None else reason)
 
 
-def agent(root):
+def agent(root, storage="file"):
+    candidate, baseline = (root, root / "baseline") if storage == "file" else (root / "s3", root / "baseline/s3")
     def load(r):
         return read(r / 'greptimedb-1/greptimedb-1/measured-summary.json')
-    a, b = load(root / 'baseline'), load(root)
-    finished = [r / 'greptimedb-1/exit-code.txt' for r in (root / 'baseline', root)]
+    a, b = load(baseline), load(candidate)
+    finished = [r / 'greptimedb-1/exit-code.txt' for r in (baseline, candidate)]
     compatible = (all(p.is_file() and p.read_text().strip() == '0' for p in finished)
                   and a.get('passed') is True and b.get('passed') is True
                   and same(a, b, ('contract_sha256', 'data_profile', 'query_window', 'warmup_runs', 'measured_runs')))
@@ -81,8 +82,9 @@ def agent(root):
     return rows
 
 
-def traces(root):
-    a, b = (read(r / 'runs/greptimedb/run.json') for r in (root / 'baseline', root))
+def traces(root, storage="file"):
+    candidate, baseline = (root, root / "baseline") if storage == "file" else (root / "s3", root / "baseline/s3")
+    a, b = (read(r / 'runs/greptimedb/run.json') for r in (baseline, candidate))
     compatible = (a.get('success') is True and b.get('success') is True
                   and same(a, b, ('expected', 'protocol', 'audit_expected', 'corpus'))
                   and same(a.get('execution', {}), b.get('execution', {}),
@@ -110,18 +112,22 @@ def traces(root):
     return rows
 
 
-def long_range(root):
+def long_range(root, storage="file"):
+    names = ("greptimedb-baseline", "greptimedb") if storage == "file" else ("greptimedb-s3-baseline", "greptimedb-s3")
+    def target_root(name):
+        nested = root / "o11ybench" / ("o11ybench-" + name)
+        return nested if nested.exists() else root
     rows = []
-    manifest = read(root / 'run-manifest.json')
+    manifests = {name: read(target_root(name) / 'run-manifest.json') for name in names}
     healthy = all(
-        all(field in (manifest.get(phase, {}).get(target, {}).get('docker', {}) if phase == 'post_load_results'
-                      else manifest.get(phase, {}).get(target, {})).get('state', '').split()
+        all(field in (manifests[target].get(phase, {}).get(target, {}).get('docker', {}) if phase == 'post_load_results'
+                      else manifests[target].get(phase, {}).get(target, {})).get('state', '').split()
             for field in ('Running=true', 'ExitCode=0', 'OOMKilled=false'))
         for phase in ('post_load_results', 'post_query_results')
-        for target in ('greptimedb-baseline', 'greptimedb'))
+        for target in names)
     for mode in ('serial', 'concurrent-duration-c1', 'concurrent-duration-c4'):
-        a, b = (read(root / name / 'results-vmbench-long-range' / (mode + '-summary.json'))
-                for name in ('greptimedb-baseline', 'greptimedb'))
+        a, b = (read(target_root(name) / name / 'results-vmbench-long-range' / (mode + '-summary.json'))
+                for name in names)
         fields = ('workload', 'query_file_sha256', 'start', 'end', 'step', 'profile', 'data_duration',
                   'scrape_interval', 'cache_policy_effective', 'mode', 'connection_mode', 'request_timeout_seconds')
         fields += ('tries', 'warmup_runs') if mode == 'serial' else ('concurrency', 'duration_seconds', 'warmup_seconds')
@@ -178,7 +184,13 @@ def main():
         parser.error('report requires --root and --kind')
     document = {'kind': args.kind, 'baseline_tag': os.environ.get('BASELINE_GREPTIMEDB_TAG', ''),
                 'candidate_tag': os.environ.get('GREPTIMEDB_TAG', ''),
-                'rows': {'agent': agent, 'traces': traces, 'long-range': long_range}[args.kind](args.root)}
+                'rows': []}
+    mode = os.environ.get('GREPTIMEDB_STORAGE', 'file')
+    for storage in (['file', 's3'] if mode == 'both' else [mode]):
+        rows = {'agent': agent, 'traces': traces, 'long-range': long_range}[args.kind](args.root, storage)
+        for item in rows:
+            item['query'] = storage.upper() + '/' + item['query']
+        document['rows'].extend(rows)
     text = render(document)
     # Preserve partial comparisons even if a target failed before creating output.
     args.root.mkdir(parents=True, exist_ok=True)

.github/scripts/build-query-regression-helpers.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+set -euo pipefail
+: "${CARGO_TARGET_DIR:?CARGO_TARGET_DIR must be configured by the runner}"
+revision="$(git rev-parse HEAD)"
+tag="query-regression-helpers:${revision}"
+staging="$(mktemp -d)"
+trap 'rm -rf -- "$staging"' EXIT
+cargo build --profile nightly -p cmd --features dev-tools \
+  --bin query_perf_fixture --bin query_regression_runner
+for binary in query_perf_fixture query_regression_runner; do
+  "$CARGO_TARGET_DIR/nightly/$binary" --help >/dev/null
+  cp "$CARGO_TARGET_DIR/nightly/$binary" "$staging/$binary"
+done
+mkdir -p helpers-image helpers-validation
+docker build --provenance=false --build-arg "SOURCE_REVISION=$revision" \
+  --file docker/query-regression-helpers/Dockerfile --tag "$tag" "$staging"
+container="$(docker create "$tag" /query_perf_fixture --help)"
+trap 'docker rm --force --volumes "$container"; rm -rf -- "$staging"' EXIT
+mkdir -p "$staging/extracted"
+for binary in query_perf_fixture query_regression_runner; do
+  docker cp "$container:/$binary" "$staging/extracted/$binary"
+  chmod +x "$staging/extracted/$binary"
+  "$staging/extracted/$binary" --help >/dev/null
+  cmp "$staging/$binary" "$staging/extracted/$binary"
+done
+# Hold the revision and cases constant: compare source-built versus extracted
+# fixture normalization twice. Keep the raw outputs as workflow artifacts.
+printf '%s\n' "$revision" > helpers-validation/source-revision.txt
+for iteration in 1 2; do
+  while IFS= read -r case; do
+    name="$(basename "$(dirname "$case")")"
+    for mode in source image; do
+      binary="$staging/query_perf_fixture"
+      [[ "$mode" != image ]] || binary="$staging/extracted/query_perf_fixture"
+      "$binary" plan --case "$case" > "helpers-validation/${name}-${iteration}-${mode}.json"
+    done
+    cmp "helpers-validation/${name}-${iteration}-source.json" \
+        "helpers-validation/${name}-${iteration}-image.json"
+  done < <(find tests/perf/query_cases -name case.toml -type f | sort)
+done
+(cd "$staging" && sha256sum query_perf_fixture query_regression_runner) > helpers-image/SHA256SUMS
+printf '%s\n' "$revision" > helpers-image/source-revision.txt
+docker save "$tag" --output helpers-image/image.tar

.github/scripts/ci-runners.md

@@ -13,8 +13,12 @@ AWS uses `aws-ec2-runner-provision.py` and `aws-ec2-runner-teardown.py` directly
 Repository variables supply `EC2_RUNNER_REGION`, `EC2_RUNNER_SUBNET_ID`, and
 `EC2_RUNNER_SECURITY_GROUP_ID`; credentials use `AWS_ACCESS_KEY_ID`,
 `AWS_SECRET_ACCESS_KEY`, and `GH_PERSONAL_ACCESS_TOKEN` secrets.
-The current region is Singapore (`ap-southeast-1`). By default the script selects
-the newest available Canonical Ubuntu 24.04 amd64 server AMI in that region
+The runner region comes from `EC2_RUNNER_REGION` (currently `us-east-1`).
+The subnet and security group must belong to the selected AWS account and region.
+For benchmarks using real S3, `AWS_CI_TEST_BUCKET_REGION` must match the runner
+region; `AWS_CI_TEST_BUCKET` must be accessible with the configured credentials.
+By default the script selects the newest available Canonical Ubuntu 24.04 amd64
+server AMI in that region
 (owner `099720109477`) and logs its ID. Set `BENCHMARK_EC2_IMAGE_ID` to pin a
 prepared Ubuntu AMI. Existing build CI's `EC2_RUNNER_LINUX_AMD64_IMAGE_ID` is
 unaffected. Both workflows default `system_disk_gib` to `auto` (also accepting
@@ -31,9 +35,12 @@ swappiness 10, disables systemd-oomd and background automatic updates, and sets
 `OOMPolicy=continue` on the runner service. Explicit setup package installation
 remains enabled.
 
-Both benchmark jobs call `.github/actions/setup-benchmark` after checkout. It
-reuses existing tools, installs missing curl/jq/python3/rsync, installs Docker CE
-if absent, and checks Docker access. Missing packages require passwordless sudo;
+Benchmark jobs call `.github/actions/setup-linux-runner` after checkout with
+`tools: curl jq python3 rsync` and require x86_64. Linux artifact builds reuse
+the same action with `tools: make shasum` on either amd64 or arm64; the RISC-V
+cross-build calls it on amd64 too. The action reuses installed commands, maps
+`shasum` to Ubuntu package `libdigest-sha-perl`, installs Docker CE if absent,
+and checks Docker access. Missing packages require passwordless sudo;
 Aliyun's prepared image normally provides the tools and Docker already. Existing
 Docker installations are preserved, so exact versions need not match between
 providers. Provisioning must establish Docker permissions before starting the

.github/scripts/metric-snapshot-s3-bench.py

@@ -105,8 +105,9 @@ class Benchmark:
         self.endpoint = os.environ.get("S3_ENDPOINT", "")
         if self.endpoint and not local_smoke:
             raise ValueError("Custom S3 endpoints are restricted to local smoke")
-        if not local_smoke and os.environ["S3_REGION"] != "ap-southeast-1":
-            raise ValueError("Cloud benchmark requires ap-southeast-1")
+        if not local_smoke and (not os.environ["S3_REGION"].strip()
+                                or os.environ["S3_REGION"] != os.environ["RUNNER_REGION"]):
+            raise ValueError("Cloud benchmark requires the same nonempty bucket and runner region")
         self.containers = set()
         self.network_created = False
         self.phases = []
@@ -324,7 +325,7 @@ level = "info"
         lines = ["# Metric packed snapshot benchmark", "", f"Image: `{self.image}`",
                  f"Source commit: `{self.commit}`", "",
                  "Local smoke only; timings are not AWS results." if self.local_smoke else
-                 "AWS S3 in ap-southeast-1; source and target use local File storage.", "",
+                 f"S3 in {os.environ['S3_REGION']}; source and target use local File storage.", "",
                  "| Round | Phase | Seconds | Result |", "| --- | --- | ---: | --- |"]
         for phase in self.phases:
             lines.append(f"| {phase['round']} | {phase['phase']} | {phase['wall_seconds']:.3f} | {phase['outcome']} |")

.github/scripts/metric-snapshot-s3-ops.py

@@ -21,7 +21,7 @@
 """Budget preflight and strictly scoped S3 operations for the metric snapshot benchmark."""
 
 import argparse
-from decimal import Decimal
+from decimal import Decimal, InvalidOperation
 import json
 import os
 from pathlib import Path
@@ -29,8 +29,8 @@ import re
 import time
 import urllib.request
 
-REGION = "ap-southeast-1"
-HOURLY_USD = Decimal("0.9408")
+# Historical budget estimate; dispatch can supply a rate for the configured region.
+DEFAULT_HOURLY_USD = "0.9408"
 OTHER_RESERVE_USD = Decimal("1.50")
 
 
@@ -40,6 +40,16 @@ def owned_prefix(prefix):
     return prefix
 
 
+def hourly_estimate(env):
+    try:
+        estimate = Decimal(env.get("EC2_HOURLY_USD", DEFAULT_HOURLY_USD))
+    except InvalidOperation as error:
+        raise ValueError("EC2_HOURLY_USD must be a positive finite estimate") from error
+    if not estimate.is_finite() or estimate <= 0:
+        raise ValueError("EC2_HOURLY_USD must be a positive finite estimate")
+    return estimate
+
+
 def plan(env):
     image = env["GREPTIMEDB_IMAGE"]
     if not re.fullmatch(r"greptime/greptimedb(?:-nightly)?@sha256:[0-9a-f]{64}", image):
@@ -49,21 +59,22 @@ def plan(env):
     tables, minutes = int(env["LOGICAL_TABLES_PER_GROUP"]), int(env["BENCHMARK_TIMEOUT_MINUTES"])
     if not 50 <= tables <= 10_000 or not 30 <= minutes <= 240:
         raise ValueError("logical tables per group must be 50..10000; timeout must be 30..240 minutes")
-    if env["S3_REGION"] != REGION or env["RUNNER_REGION"] != REGION:
-        raise ValueError("bucket and runner must both use ap-southeast-1")
+    if not env["S3_REGION"].strip() or env["S3_REGION"] != env["RUNNER_REGION"]:
+        raise ValueError("bucket and runner must use the same nonempty region")
     if not env["S3_BUCKET"]:
         raise ValueError("S3 bucket is required")
     prior = Decimal(env["PRIOR_SPEND_USD"])
     # Reserve 45m provision, 10m queue allowance and 20m cleanup beyond the job timeout.
-    reservation = HOURLY_USD * Decimal(minutes + 75) / 60 + OTHER_RESERVE_USD
+    hourly = hourly_estimate(env)
+    reservation = hourly * Decimal(minutes + 75) / 60 + OTHER_RESERVE_USD
     if not prior.is_finite() or prior < 0 or prior >= 15 or prior + reservation > 20:
         raise ValueError("campaign budget exhausted: prior spend must be below $15 and leave the full attempt reserve within $20")
     return {"image": image, "expected_commit": env["EXPECTED_COMMIT"],
             "logical_tables_per_group": tables, "smoke_logical_tables": 500,
             "timeout_minutes": minutes, "prefix": owned_prefix(env["RUN_PREFIX"]),
-            "instance_type": "c6id.4xlarge", "region": REGION,
+            "instance_type": "c6id.4xlarge", "region": env["S3_REGION"],
             "prior_estimated_usd": str(prior), "attempt_reserve_usd": str(reservation),
-            "ec2_hourly_usd": str(HOURLY_USD), "other_reserve_usd": str(OTHER_RESERVE_USD),
+            "ec2_hourly_usd": str(hourly), "other_reserve_usd": str(OTHER_RESERVE_USD),
             "cost_note": "Estimate, not an AWS billing total or a hard spending cap."}
 
 
@@ -71,8 +82,8 @@ def client(region):
     import boto3
     from botocore.config import Config
 
-    if region != REGION:
-        raise ValueError("S3 region must be ap-southeast-1")
+    if not region.strip():
+        raise ValueError("S3 region is required")
     return boto3.client("s3", region_name=region, config=Config(
         connect_timeout=10, read_timeout=30, retries={"mode": "standard", "max_attempts": 3}))
 
@@ -193,8 +204,10 @@ def main():
         if args.started_at is None:
             parser.error("cost requires --started-at")
         seconds = max(0, int(time.time()) - args.started_at)
-        estimate = HOURLY_USD * Decimal(seconds) / 3600
-        result = {"allocation_window_seconds": seconds, "ec2_estimate_usd": str(round(estimate, 4)),
+        hourly = hourly_estimate(os.environ)
+        estimate = hourly * Decimal(seconds) / 3600
+        result = {"allocation_window_seconds": seconds, "ec2_hourly_usd": str(hourly),
+                  "ec2_estimate_usd": str(round(estimate, 4)),
                   "campaign_estimate_with_other_reserve_usd": str(round(
                       Decimal(os.environ["PRIOR_SPEND_USD"]) + estimate + OTHER_RESERVE_USD, 4)),
                   "note": "Allocation wall-time estimate plus $1.50 reserve for S3, disk and IPv4; not an AWS bill."}

.github/scripts/o11ybench-storage.py

@@ -0,0 +1,316 @@
+# Copyright 2023 Greptime Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Select File/S3 benchmark arms without changing workload adapters."""
+import argparse
+import copy
+import importlib.util
+import json
+from pathlib import Path
+import sys
+import os
+import subprocess
+
+
+def validate(env):
+    mode = env.get('GREPTIMEDB_STORAGE', 'file')
+    if mode not in ('file', 's3', 'both'):
+        raise ValueError('GREPTIMEDB_STORAGE must be file, s3 or both')
+    if mode != 'file':
+        if env.get('RUN_GREPTIMEDB') != 'true':
+            raise ValueError('S3 requires Run GreptimeDB')
+        if env.get('PROVIDER') != 'AWS':
+            raise ValueError('S3 comparisons require an AWS runner')
+        if not env.get('S3_BUCKET') or not env.get('S3_REGION'):
+            raise ValueError('AWS_CI_TEST_BUCKET and AWS_CI_TEST_BUCKET_REGION must be configured')
+        if env['S3_REGION'] != env.get('RUNNER_REGION'):
+            raise ValueError('S3 bucket and EC2 runner regions must match')
+    return mode
+
+
+def replace(command, key, value):
+    result = list(command)
+    if key in result:
+        result[result.index(key) + 1] = value
+    else:
+        result += [key, value]
+    return result
+
+
+def commands(command, mode):
+    """Recovery visits each owned root; execution uses explicit workflow steps."""
+    stage = command[command.index('--stage') + 1]
+    if stage != 'cleanup':
+        raise ValueError('only cleanup may expand multiple arms')
+    result = [list(command)]
+    if mode != 'file':
+        root = command[command.index('--root') + 1]
+        owner = command[command.index('--owner') + 1]
+        secondary = replace(command, '--root', root + '/s3')
+        result.append(replace(secondary, '--owner', owner + '-s3'))
+    return result
+
+
+def module(path):
+    sys.path.insert(0, str(path.parent))
+    spec = importlib.util.spec_from_file_location(path.stem.replace('-', '_'), path)
+    result = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(result)
+    return result
+
+
+def summary(kind, root, targets, section, mode):
+    filename = 'agent-observability-summary.py' if kind == 'agent' else 'tracesbench-summary.py'
+    renderer = module(Path(__file__).with_name(filename))
+    primary = [t for t in targets if t != 'greptimedb' or mode != 's3']
+    arms = [(root, primary, 'File')]
+    if mode != 'file' and 'greptimedb' in targets:
+        arms.append((root / 's3', ['greptimedb'], 'S3'))
+    output = []
+    for arm, selected, label in arms:
+        if not selected:
+            continue
+        output.append('## ' + ('GreptimeDB S3' if label == 'S3' else 'Primary targets (GreptimeDB File)'))
+        if kind == 'agent':
+            body = (renderer.benchmark(arm, selected) if section == 'benchmark'
+                    else renderer.lifecycle(arm, selected, data_root=root.parent if root.name == 'baseline' else root))
+        else:
+            report, runs = renderer.load(arm)
+            body = (renderer.benchmark(report, runs, selected) if section == 'benchmark'
+                    else renderer.lifecycle(report, runs, arm, selected))
+        body = body.replace('| greptimedb |', '| GreptimeDB ' + label + ' |')
+        body = body.replace('**greptimedb**', '**GreptimeDB ' + label + '**')
+        output.append(body)
+        if label == 'S3':
+            directories = [arm / 'greptimedb-1'] if kind == 'agent' else [arm / 'runs/greptimedb']
+            for directory in directories:
+                for action in ('probe', 'inventory', 'cleanup'):
+                    path = directory / ('s3-' + action + '.json')
+                    data = json.loads(path.read_text()) if path.exists() else {}
+                    status = 'passed' if data.get('passed') is True else 'failed / unavailable'
+                    count = data.get('object_count', data.get('deleted_objects', '—'))
+                    output.append(f'- S3 {action}: **{status}**; objects: {count}.')
+    return '\n\n'.join(output)
+
+
+def split_metrics(plan):
+    """One executor root per target, with the immutable dataset shared by all."""
+    result = {}
+    for target in plan['targets']:
+        child = copy.deepcopy(plan)
+        name = target['name']
+        dest = str(Path(plan['output_root']) / 'o11ybench' / ('o11ybench-' + name))
+        child['output_root'] = dest
+        child['run_id'] = plan['run_id'] + '-' + name
+        child['targets'] = [copy.deepcopy(target)]
+        child['data']['generate'] = False
+        child['report'].update(root=dest + '/report', index_html=dest + '/report/index.html')
+        child['artifacts'] = {k: dest + '/' + Path(v).name for k, v in plan['artifacts'].items()}
+        result[name] = child
+    return result
+
+
+def metrics(action, args):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--plan', type=Path, required=True)
+    parser.add_argument('--tools', type=Path, required=True)
+    parser.add_argument('--runtime-image', required=True)
+    parser.add_argument('--target')
+    a = parser.parse_args(args)
+    plan = json.loads(a.plan.read_text())
+    plan['runtime_image'] = a.runtime_image
+    executor = module(a.tools / 'scripts/run_benchmark_plan.py')
+    root = Path(plan['output_root'])
+    if action == 'metrics-prepare':
+        executor.require_safe_execute_root(root)
+        root.mkdir(parents=True, exist_ok=False)
+        manifest = executor.build_manifest(plan)
+        executor.run_checked([str(executor.DOWNLOAD_DATA)], manifest,
+                             env=executor.generation_environment(plan))
+        (root / 'generation.json').write_text(json.dumps(manifest, indent=2) + '\n')
+        for name, child in split_metrics(plan).items():
+            (a.plan.parent / (name + '.json')).write_text(json.dumps(child, indent=2) + '\n')
+    elif action == 'metrics-run':
+        path = a.plan.parent / (a.target + '.json')
+        child = json.loads(path.read_text())
+        executor.execute_plan(child, executor.build_manifest(child), replace=False)
+        subprocess.run([sys.executable, str(a.tools / 'scripts/check_run_benchmark_plan.py'),
+                        '--artifacts', child['output_root']], check=True)
+    else:
+        # Validate every selected target together, retaining cross-target gates.
+        manifests = [executor.build_manifest(child) for child in split_metrics(plan).values()]
+        combined = executor.build_manifest(plan)
+        combined['targets'] = [target for m in manifests for target in m['targets']]
+        for command in executor.validation_commands(combined):
+            subprocess.run(command, check=True)
+
+
+def overview(path):
+    """Render the same validated report data into the GitHub job summary."""
+    data = json.loads(path.read_text())
+    renderer = module(Path(__file__).with_name('agent-observability-summary.py'))
+    lines = ['# Storage comparison', '', 'Smoke results are diagnostic; see classification per arm. '
+             'No cold-object-store performance claim. Missing results are not passes.', '']
+    for workload in data['workloads']:
+        labels = {p['id']: p['label'] for p in workload['participants']}
+        for scenario in workload['scenarios']:
+            lines += ['## ' + workload['title'] + ' / ' + scenario['label'], '']
+            rows = []
+            for result in scenario['results']:
+                lines += [labels[result['participant']] + ': ' + str(result.get('classification')), '']
+                for query_id, query in result['queries'].items():
+                    m = query['metrics']
+                    rows.append([query_id, labels[result['participant']], query['status'],
+                                 renderer.duration(m.get('p50_ms', m.get('median_ms'))),
+                                 renderer.duration(m.get('p95_ms')), renderer.duration(m.get('p99_ms'))])
+            lines.append(renderer.table(['Query', 'DB / storage', 'Status', 'P50 / median', 'P95', 'P99'], rows))
+    return '\n'.join(lines)
+
+
+def audit(root, report, targets, mode, baseline=False, allow_trace_query_failures=False):
+    data = json.loads(report.read_text())
+    actual = {(p['target'], p.get('storage', {}).get('type'), p['role'])
+              for w in data['workloads'] for p in w['participants']}
+    expected = {(t, None, 'comparison') for t in targets if t != 'greptimedb'}
+    modes = ['file', 's3'] if mode == 'both' else [mode]
+    if 'greptimedb' in targets:
+        expected |= {('greptimedb', m, role) for m in modes
+                     for role in (['candidate', 'baseline'] if baseline else ['candidate'])}
+    if actual != expected:
+        raise ValueError(f'participant mismatch: expected {sorted(map(str, expected))}, got {sorted(map(str, actual))}')
+    query_failures = []
+    for w in data['workloads']:
+        partial = allow_trace_query_failures and w.get('id') == 'traces'
+        if partial and not w['scenarios']:
+            raise ValueError('missing trace scenarios')
+        for scenario in w['scenarios']:
+            if partial and {r['participant'] for r in scenario['results']} != {p['id'] for p in w['participants']}:
+                raise ValueError('missing trace participant results')
+            for result in scenario['results']:
+                if partial and result.get('health') != 'recorded healthy snapshots':
+                    raise ValueError('missing or unhealthy trace evidence: ' + result['participant'])
+                queries = result['queries']
+                if not any(q['status'] == 'passed' for q in queries.values()):
+                    raise ValueError('no successful measured queries: ' + result['participant'])
+                if any(q['status'] == 'unavailable' or (q['status'] == 'failed' and not partial)
+                       for q in queries.values()):
+                    raise ValueError('failed or unavailable queries: ' + result['participant'])
+                for query_id, query in queries.items():
+                    if query['status'] == 'failed':
+                        query_failures.append(dict(workload=w['id'], scenario=scenario['id'],
+                                                   participant=result['participant'], query=query_id,
+                                                   reason=query.get('reason', '')))
+    for manifest_path in root.glob('o11ybench/*/run-manifest.json'):
+        manifest = json.loads(manifest_path.read_text())
+        if manifest.get('status') != 'passed':
+            raise ValueError('incomplete target manifest: ' + str(manifest_path))
+    probes = list(root.rglob('s3-probe.json'))
+    expected_s3 = sum(m == 's3' for _, m, _ in expected)
+    if len(probes) != expected_s3:
+        raise ValueError('missing or duplicate S3 evidence')
+    for probe in probes:
+        for action in ('probe', 'inventory', 'cleanup'):
+            payload = json.loads(probe.with_name('s3-' + action + '.json').read_text())
+            if payload.get('passed') is not True or (action == 'inventory' and payload.get('object_count', 0) <= 0):
+                raise ValueError('failed S3 ' + action)
+    text = (root / 'ci-summary.md').read_text()
+    for m in modes if 'greptimedb' in targets else []:
+        if m.upper() not in text:
+            raise ValueError('storage mode missing from saved summary: ' + m)
+    result = dict(passed=True, participants=[dict(target=t, storage=m, role=role)
+                  for t, m, role in sorted(expected, key=str)], s3_arms=len(probes), query_failures=query_failures)
+    (root / 'artifact-audit.json').write_text(json.dumps(result, indent=2) + '\n')
+    lines = ['## Artifact validation', '', '- Selected participants: ' + str(len(expected)) + '; all present.',
+             '- Query result coverage and S3 probe/inventory/cleanup: passed.', '']
+    if query_failures:
+        lines += ['- Collection completed with query failures; this is not an all-pass query run.', '']
+        lines += [f"- {f['participant']} / {f['query']}: failed (excluded from comparison)."
+                  for f in query_failures]
+        lines.append('')
+    renderer = module(Path(__file__).with_name('agent-observability-summary.py'))
+    rows = []
+    for probe in probes:
+        inv = json.loads(probe.with_name('s3-inventory.json').read_text())
+        cleanup = json.loads(probe.with_name('s3-cleanup.json').read_text())
+        rows.append([str(probe.parent.relative_to(root)), inv['object_count'], inv['object_bytes'],
+                     cleanup.get('deleted_objects'), 'passed'])
+    if rows:
+        lines.append(renderer.table(['S3 arm', 'Objects', 'Remote bytes', 'Deleted current objects', 'Cleanup'], rows))
+    text = '\n'.join(lines) + '\n'
+    with (root / 'ci-summary.md').open('a') as output:
+        output.write(text)
+    if os.environ.get('GITHUB_STEP_SUMMARY'):
+        with Path(os.environ['GITHUB_STEP_SUMMARY']).open('a') as output:
+            output.write(text)
+    return result
+
+
+def main():
+    if len(sys.argv) > 1 and sys.argv[1] == 'audit':
+        parser = argparse.ArgumentParser()
+        parser.add_argument('--root', type=Path, required=True)
+        parser.add_argument('--report', type=Path, required=True)
+        parser.add_argument('--targets', required=True)
+        parser.add_argument('--allow-trace-query-failures', action='store_true',
+                            help='Retain failed Traces queries after native validation; missing results still fail')
+        args = parser.parse_args(sys.argv[2:])
+        print(json.dumps(audit(args.root, args.report, args.targets.split(','),
+                              os.environ.get('GREPTIMEDB_STORAGE', 'file'),
+                              os.environ.get('COMPARE_GREPTIMEDB') == 'true', args.allow_trace_query_failures)))
+        return
+    if len(sys.argv) > 1 and sys.argv[1] == 'normalize-metrics':
+        path = Path(sys.argv[2])
+        plan = json.loads(path.read_text())
+        for target in plan['targets']:
+            if target.get('storage', {}).get('type') == 's3' and '-s3' not in target['name']:
+                target['name'] = target['name'].replace('-baseline', '-s3-baseline') if target['name'].endswith('-baseline') else target['name'] + '-s3'
+        path.write_text(json.dumps(plan, indent=2) + '\n')
+        return
+    if len(sys.argv) > 1 and sys.argv[1].startswith('metrics-'):
+        metrics(sys.argv[1], sys.argv[2:])
+        return
+    if len(sys.argv) > 1 and sys.argv[1] == 'overview':
+        print(overview(Path(sys.argv[2])))
+        return
+    if len(sys.argv) > 1 and sys.argv[1] == 'summary':
+        parser = argparse.ArgumentParser()
+        parser.add_argument('--kind', choices=['agent', 'traces'], required=True)
+        parser.add_argument('--root', type=Path, required=True)
+        parser.add_argument('--targets', required=True)
+        parser.add_argument('--section', choices=['benchmark', 'lifecycle'], required=True)
+        args = parser.parse_args(sys.argv[2:])
+        print(summary(args.kind, args.root, args.targets.split(','), args.section,
+                      os.environ.get('GREPTIMEDB_STORAGE', 'file')))
+        return
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('action', choices=['validate', 'run'])
+    parser.add_argument('command', nargs=argparse.REMAINDER)
+    args = parser.parse_args()
+    mode = validate(os.environ)
+    if args.action == 'validate':
+        print('GreptimeDB storage: ' + mode)
+        return
+    command = args.command[1:] if args.command[:1] == ['--'] else args.command
+    planned = commands(command, mode)
+    failed = 0
+    for cmd in planned:
+        result = subprocess.run(cmd)
+        failed = failed or result.returncode
+        # Cleanup must try both arms even if one failed. Target failures remain
+        # visible, while the other arm can still produce diagnostic evidence.
+    raise SystemExit(failed)
+
+
+if __name__ == '__main__':
+    main()

.github/scripts/prepare-fuzz-corpus.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+: "${FUZZ_TARGETS:?FUZZ_TARGETS is required}"
+: "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE is required}"
+: "${GITHUB_OUTPUT:?GITHUB_OUTPUT is required}"
+
+targets=()
+while IFS= read -r target; do
+  target="${target%$'\r'}"
+  [[ -z "${target}" ]] && continue
+  if [[ ! "${target}" =~ ^[a-zA-Z0-9_]+$ ]]; then
+    printf 'Invalid fuzz target: %s\n' "${target}" >&2
+    exit 2
+  fi
+  targets+=("${target}")
+done <<<"${FUZZ_TARGETS}"
+
+if [[ "${#targets[@]}" -eq 0 ]]; then
+  printf 'FUZZ_TARGETS contains no targets\n' >&2
+  exit 2
+fi
+
+# Corpus identity depends on the targets, not the source revision or their order.
+target_set="$(printf '%s\n' "${targets[@]}" | LC_ALL=C sort -u)"
+target_hash="$(printf '%s\n' "${target_set}" | sha256sum | cut -d ' ' -f 1)"
+corpus_root="${GITHUB_WORKSPACE}/tests-fuzz/corpus"
+
+{
+  printf 'key-prefix=fuzz-corpus-v1-%s-\n' "${target_hash}"
+  printf 'paths<<FUZZ_CORPUS_PATHS\n'
+  while IFS= read -r target; do
+    mkdir -p "${corpus_root}/${target}"
+    printf '%s\n' "${corpus_root}/${target}"
+  done <<<"${target_set}"
+  printf 'FUZZ_CORPUS_PATHS\n'
+} >>"${GITHUB_OUTPUT}"

.github/scripts/publish-query-regression-helpers.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -euo pipefail
+repository=greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/query-regression-helpers
+revision="$(cat helpers-image/source-revision.txt)"
+[[ "$revision" == "$GITHUB_SHA" ]]
+tag="${repository}:${revision}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+# Match release-dev-builder-images.yaml: publish ACR images with Skopeo from
+# a hosted runner. A failed publish can reuse the completed build artifact.
+mkdir -p helpers-validation
+docker run --rm \
+  -v "${DOCKER_CONFIG:-$HOME/.docker}:/root/.docker:ro" \
+  -v "$PWD/helpers-image:/image:ro" \
+  -v "$PWD/helpers-validation:/result" \
+  -e REGISTRY_AUTH_FILE=/root/.docker/config.json \
+  quay.io/skopeo/stable:latest --command-timeout 20m copy --retry-times 3 \
+  --digestfile /result/digest.txt \
+  docker-archive:/image/image.tar "docker://$tag"
+digest="$(cat helpers-validation/digest.txt)"
+[[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]]
+image="$repository@$digest"
+# The ECS build tested these exact bytes. Verify them again from the registry;
+# hosted runners need not share the ECS runtime library versions.
+python3 .github/scripts/query-regression-helpers.py "$image" helpers-validation/extracted \
+  --checksums helpers-image/SHA256SUMS
+jq -e --arg revision "$revision" '.source_revision == $revision' \
+  helpers-validation/extracted/helpers-image.json >/dev/null
+printf 'image=%s\n' "$image" >> "$GITHUB_OUTPUT"
+printf 'Published helpers: `%s` (source `%s`)\n' "$image" "$revision" >> "$GITHUB_STEP_SUMMARY"

.github/scripts/query-regression-helpers.py

@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+# Copyright 2023 Greptime Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Extract digest-pinned query regression tools from an artifact image."""
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import subprocess
+from pathlib import Path
+
+BINARIES = ("query_perf_fixture", "query_regression_runner")
+
+
+def resolve_image(value):
+    if value == "pinned":
+        value = os.environ.get("QUERY_REGRESSION_HELPERS_IMAGE", "")
+    if not re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9._/:@-]*@sha256:[0-9a-f]{64}", value):
+        raise ValueError("helpers_image requires an image@sha256 digest; set QUERY_REGRESSION_HELPERS_IMAGE by publishing a verified image before using 'pinned'")
+    return value
+
+
+def run(*args):
+    return subprocess.check_output(args, text=True).strip()
+
+
+def extract(image, destination, checksums=None):
+    expected = None
+    if checksums is not None:
+        entries = [line.split() for line in checksums.read_text().splitlines()]
+        expected = {name: digest for digest, name in entries}
+        if set(expected) != set(BINARIES) or len(entries) != len(BINARIES):
+            raise ValueError("Checksums must contain exactly the two helper binaries")
+    image = resolve_image(image)
+    destination.mkdir(parents=True, exist_ok=True)
+    run("docker", "pull", image)
+    metadata = json.loads(run("docker", "image", "inspect", image))[0]
+    revision = (metadata.get("Config", {}).get("Labels") or {}).get("org.opencontainers.image.revision", "")
+    if not re.fullmatch(r"[0-9a-f]{40}", revision):
+        raise ValueError("Helper image is missing its source revision label")
+    container = run("docker", "create", image, "/query_perf_fixture", "--help")
+    try:
+        for binary in BINARIES:
+            target = destination / binary
+            run("docker", "cp", f"{container}:/{binary}", str(target))
+            target.chmod(0o755)
+            if expected is None:
+                run(str(target.resolve()), "--help")
+            else:
+                with target.open("rb") as file:
+                    actual = hashlib.file_digest(file, "sha256").hexdigest()
+                if actual != expected[binary]:
+                    raise ValueError(f"Published helper checksum mismatch: {binary}")
+    finally:
+        run("docker", "rm", "--force", "--volumes", container)
+    provenance = {"image": image, "image_id": metadata["Id"], "source_revision": revision}
+    (destination / "helpers-image.json").write_text(json.dumps(provenance, indent=2) + "\n")
+    print(json.dumps(provenance))
+    return provenance
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("image")
+    parser.add_argument("destination", type=Path)
+    parser.add_argument("--checksums", type=Path, help="Verify build checksums instead of executing on this host")
+    args = parser.parse_args()
+    extract(args.image, args.destination, args.checksums)

.github/scripts/query-regression-run.py

@@ -21,10 +21,13 @@ import argparse
 import json
 import os
 import re
+import shlex
 import signal
 import socket
 import subprocess
+import sys
 import time
+import uuid
 import urllib.request
 from dataclasses import dataclass
 from pathlib import Path
@@ -130,6 +133,7 @@ class RunTarget:
     datanode_http_port: int
     datanode_data_dir: Path
     frontend_config: Path | None = None
+    containerized: bool = False
 
 
 def allocate_ports(n: int) -> list[int]:
@@ -151,6 +155,7 @@ def make_target(
     root: Path,
     ports: list[int],
     frontend_config: Path | None = None,
+    containerized: bool = False,
 ) -> RunTarget:
     work_dir = root / name
     return RunTarget(
@@ -167,6 +172,7 @@ def make_target(
         datanode_http_port=ports[3],
         datanode_data_dir=work_dir / "datanode-0" / "data",
         frontend_config=frontend_config,
+        containerized=containerized,
     )
 
 
@@ -258,7 +264,11 @@ def stop_component(
     procs: dict[tuple[str, str], subprocess.Popen[bytes]],
 ) -> None:
     proc = procs.pop((target.name, name), None)
-    if proc is None or proc.poll() is not None:
+    if proc is None:
+        return
+    if proc.poll() is not None:
+        if target.containerized and proc.returncode != 0:
+            raise RuntimeError(f"Docker target {target.name}/{name} exited with {proc.returncode}")
         return
     try:
         os.killpg(proc.pid, signal.SIGTERM)
@@ -272,6 +282,8 @@ def stop_component(
         except ProcessLookupError:
             return
         proc.wait(timeout=20)
+    if target.containerized and proc.returncode not in (0, 128 + signal.SIGTERM):
+        raise RuntimeError(f"Docker target {target.name}/{name} cleanup failed: {proc.returncode}")
 
 
 def restart_component(
@@ -284,9 +296,15 @@ def restart_component(
 
 
 def stop_all(targets: list[RunTarget], procs: dict[tuple[str, str], subprocess.Popen[bytes]]) -> None:
+    errors = []
     for target in reversed(targets):
         for name in ("frontend", "datanode", "metasrv"):
-            stop_component(target, name, procs)
+            try:
+                stop_component(target, name, procs)
+            except Exception as err:
+                errors.append(err)
+    if errors:
+        raise RuntimeError(f"component cleanup failed: {errors}") from errors[0]
 
 
 def load_plan(fixture_generator: Path, case_path: Path) -> dict[str, Any]:
@@ -313,8 +331,8 @@ def run_direct_case(
 ) -> int:
     ports = allocate_ports(16)
     targets = [
-        make_target("base", base_bin, work_dir, ports[:8]),
-        make_target("candidate", candidate_bin, work_dir, ports[8:]),
+        make_target("base", base_bin, work_dir, ports[:8], containerized=bool(getattr(args, "base_image", ""))),
+        make_target("candidate", candidate_bin, work_dir, ports[8:], containerized=bool(getattr(args, "candidate_image", ""))),
     ]
     for target in targets:
         if target.work_dir.exists() and any(target.work_dir.iterdir()):
@@ -407,6 +425,7 @@ def run_remote_case(
             work_dir,
             ports[:8],
             work_dir / "base" / "frontend-prom-store.toml",
+            containerized=bool(getattr(args, "base_image", "")),
         ),
         make_target(
             "candidate",
@@ -414,6 +433,7 @@ def run_remote_case(
             work_dir,
             ports[8:],
             work_dir / "candidate" / "frontend-prom-store.toml",
+            containerized=bool(getattr(args, "candidate_image", "")),
         ),
     ]
     for target in targets:
@@ -495,8 +515,8 @@ def run_otlp_case(
         raise ValueError("--otelgen-bin is required for otlp_trace_load")
     ports = allocate_ports(16)
     targets = [
-        make_target("base", base_bin, work_dir, ports[:8]),
-        make_target("candidate", candidate_bin, work_dir, ports[8:]),
+        make_target("base", base_bin, work_dir, ports[:8], containerized=bool(getattr(args, "base_image", ""))),
+        make_target("candidate", candidate_bin, work_dir, ports[8:], containerized=bool(getattr(args, "candidate_image", ""))),
     ]
     for target in targets:
         if target.work_dir.exists() and any(target.work_dir.iterdir()):
@@ -634,6 +654,106 @@ def write_failed_report(work_dir: Path, case_path: Path, error: Exception) -> No
     )
 
 
+def docker_target(image: str, root: Path, command: list[str]) -> int:
+    """Expose an image as a binary, including read-bench subcommands.
+
+    The caller terminates a process group. Keep Docker clients in separate
+    sessions so cleanup can stop the container before SST inspection starts.
+    """
+    name = f"query-regression-{uuid.uuid4().hex}"
+
+    def interrupted(signum, _frame):
+        raise SystemExit(128 + signum)
+
+    previous = {sig: signal.signal(sig, interrupted) for sig in (signal.SIGTERM, signal.SIGINT)}
+    try:
+        subprocess.run([
+            "docker", "create", "--name", name, "--network", "host",
+            "--user", f"{os.getuid()}:{os.getgid()}",
+            "--volume", f"{root}:{root}", "--workdir", str(root),
+            "--entrypoint", "/greptime/bin/greptime", image, *command,
+        ], check=True, stdout=subprocess.DEVNULL, start_new_session=True)
+        attached = subprocess.run(
+            ["docker", "start", "--attach", name], check=False, start_new_session=True,
+        )
+        if attached.returncode != 0:
+            return attached.returncode
+        # docker start's status is not the contract for the container's exit code.
+        result = subprocess.run(
+            ["docker", "inspect", "--format", "{{.State.ExitCode}}", name],
+            check=True, capture_output=True, text=True, start_new_session=True,
+        )
+        return int(result.stdout.strip())
+    finally:
+        for sig in previous:
+            signal.signal(sig, signal.SIG_IGN)
+        try:
+            subprocess.run(
+                ["docker", "stop", "--time", "5", name], check=False,
+                stdout=subprocess.DEVNULL, timeout=8, start_new_session=True,
+            )
+        except subprocess.TimeoutExpired:
+            # Force-removal below still waits for the container to stop.
+            pass
+        finally:
+            try:
+                subprocess.run(
+                    ["docker", "rm", "--force", "--volumes", name], check=True,
+                    stdout=subprocess.DEVNULL, timeout=5, start_new_session=True,
+                )
+            finally:
+                for sig, handler in previous.items():
+                    signal.signal(sig, handler)
+
+
+def prepare_targets(args: argparse.Namespace) -> dict[str, Any]:
+    """Resolve images once and create binary-compatible launchers for all cases."""
+    metadata = {}
+    images = {}
+    for name in ("base", "candidate"):
+        image = getattr(args, f"{name}_image")
+        ref = getattr(args, f"{name}_ref")
+        if not image:
+            metadata[name] = {"kind": "source", "ref": os.environ.get(f"VERIFIED_{name.upper()}_SHA") or ref}
+            continue
+        if sys.platform != "linux":
+            raise ValueError("Docker targets require Linux host networking and a local Docker daemon")
+        if image.startswith("-") or any(c.isspace() for c in image):
+            raise ValueError(f"invalid {name} image reference: {image!r}")
+        if image not in images:
+            result = subprocess.run(
+                ["docker", "image", "inspect", image], check=False, capture_output=True, text=True,
+            )
+            if result.returncode != 0:
+                subprocess.run(["docker", "pull", image], check=True)
+                result = subprocess.run(
+                    ["docker", "image", "inspect", image], check=True, capture_output=True, text=True,
+                )
+            images[image] = json.loads(result.stdout)[0]
+        info = images[image]
+        image_id = info["Id"]
+        if not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id):
+            raise ValueError(f"invalid Docker image ID: {image_id!r}")
+        args.work_dir = args.work_dir.resolve()
+        if ":" in str(args.work_dir):
+            raise ValueError("Docker target work directory cannot contain ':'")
+        launchers = args.work_dir / ".targets"
+        launchers.mkdir(parents=True, exist_ok=True)
+        launcher = launchers / name
+        command = [sys.executable, str(Path(__file__).resolve()),
+                   "docker-target", image_id, str(args.work_dir)]
+        launcher.write_text("#!/bin/sh\nexec " + shlex.join(command) + ' "$@"\n', encoding="utf-8")
+        launcher.chmod(0o755)
+        setattr(args, f"{name}_bin", launcher)
+        setattr(args, f"{name}_ref", f"{image} ({image_id})")
+        metadata[name] = {
+            "kind": "docker", "image": image, "image_id": image_id,
+            "repo_digests": info.get("RepoDigests", []),
+            "network": "host", "user": f"{os.getuid()}:{os.getgid()}",
+        }
+    return metadata
+
+
 def main() -> int:
     parser = argparse.ArgumentParser()
     parser.add_argument("--cases", action="append", help="'all', 'heavy', or comma/space separated case paths")
@@ -663,6 +783,8 @@ def main() -> int:
     parser.add_argument("--base-ref", default=os.environ.get("BASE_REF", ""))
     parser.add_argument("--candidate-ref", default=os.environ.get("CANDIDATE_REF", ""))
     parser.add_argument("--github-output", default=os.environ.get("GITHUB_OUTPUT"))
+    parser.add_argument("--base-image", default=os.environ.get("BASE_IMAGE", ""))
+    parser.add_argument("--candidate-image", default=os.environ.get("CANDIDATE_IMAGE", ""))
     args = parser.parse_args()
     try:
         cases = split_cases(args.cases or [os.environ.get("CASE_PATHS", "all")])
@@ -671,6 +793,12 @@ def main() -> int:
         append_github_output(args.github_output, 1)
         return 0
 
+    preparation_error = None
+    metadata = {}
+    try:
+        metadata = prepare_targets(args)
+    except Exception as err:
+        preparation_error = err
     reports: list[Path] = []
     status = 0
     for case in cases:
@@ -678,11 +806,18 @@ def main() -> int:
         work_dir = args.work_dir / case_slug(case_path)
         reports.append(work_dir / "query-regression-report.json")
         try:
+            if preparation_error is not None:
+                raise preparation_error
             case_status = run_case(args, case_path, work_dir)
         except Exception as err:  # noqa: BLE001 - preserve an aggregate report for the summary step
             print(f"error: query regression case {case_path}: {err}", flush=True)
             write_failed_report(work_dir, case_path, err)
             case_status = 1
+        report_path = reports[-1]
+        if (args.base_image or args.candidate_image) and report_path.exists():
+            report = json.loads(report_path.read_text(encoding="utf-8"))
+            report["execution_targets"] = metadata
+            report_path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
         if case_status != 0:
             status = case_status or 1
 
@@ -695,4 +830,6 @@ def main() -> int:
 
 
 if __name__ == "__main__":
+    if len(sys.argv) > 1 and sys.argv[1] == "docker-target":
+        raise SystemExit(docker_target(sys.argv[2], Path(sys.argv[3]), sys.argv[4:]))
     raise SystemExit(main())

.github/scripts/run-fuzz-targets-test.sh

@@ -5,6 +5,7 @@ set -euo pipefail
 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
 RUNNER="${SCRIPT_DIR}/run-fuzz-targets.sh"
 COLLECTOR="${SCRIPT_DIR}/collect-fuzz-target-artifacts.sh"
+PREPARE_CORPUS="${SCRIPT_DIR}/prepare-fuzz-corpus.sh"
 
 fail() {
   printf 'FAIL: %s\n' "$*" >&2
@@ -32,6 +33,13 @@ new_fixture() {
 set -euo pipefail
 printf '%s\t%s\t%s\n' "${GT_FUZZ_DUMP_DIR}" "$*" "${MOCK_CARGO_MARKER:-}" >>"${MOCK_CARGO_LOG}"
 target="$4"
+corpus="$5"
+[[ -d "${corpus}" ]] || exit 18
+printf '%s\t%s\n' "${target}" "${corpus}" >>"${MOCK_CORPUS_LOG}"
+if [[ -f "${corpus}/seed" ]]; then
+  cat "${corpus}/seed" >>"${MOCK_SEED_LOG}"
+fi
+printf '%s\n' "${target}" >"${corpus}/learned"
 case " ${MOCK_FAIL_TARGETS:-} " in
   *" ${target} "*) exit 17 ;;
 esac
@@ -64,12 +72,15 @@ run_fixture() {
     "MOCK_COLLECTOR_LOG=${fixture}/collector.log"
     "MOCK_FAIL_TARGETS=${fail_targets}"
     "MOCK_CARGO_MARKER=fixture"
+    "MOCK_CORPUS_LOG=${fixture}/corpus.log"
+    "MOCK_SEED_LOG=${fixture}/seeds.log"
     "FUZZ_TARGETS=${targets}"
     "FUZZ_GROUP=test-group"
     "FUZZ_MAX_TOTAL_TIME=120"
     "FUZZ_UNSTABLE=${unstable}"
     "FUZZ_COLLECT_CLUSTER_ARTIFACTS=true"
     "FUZZ_ARTIFACT_ROOT=${artifact_root}"
+    "FUZZ_CORPUS_ROOT=${fixture}/corpus with spaces"
     "FUZZ_ARTIFACT_COLLECTOR=${fixture}/collector"
     "GT_FUZZ_BINARY_PATH=${GT_FUZZ_BINARY_PATH:-}"
     "GT_FUZZ_INSTANCE_ROOT_DIR=${GT_FUZZ_INSTANCE_ROOT_DIR:-}"
@@ -81,7 +92,7 @@ run_fixture() {
   fi
 
   set +e
-  env -u FUZZ_FAIL_FAST "${runner_env[@]}" \
+  env -u FUZZ_FAIL_FAST -u FUZZ_BIN_DIR "${runner_env[@]}" \
     "${RUNNER}" >"${fixture}/stdout.log" 2>&1
   fixture_status=$?
   set -e
@@ -89,18 +100,30 @@ run_fixture() {
 
 run_prebuilt_fixture() {
   new_fixture
-  mkdir -p "${fixture}/prebuilt"
+  mkdir -p "${fixture}/prebuilt" "${fixture}/corpus with spaces/fuzz_create_table"
+  printf 'restored prebuilt seed\n' >"${fixture}/corpus with spaces/fuzz_create_table/seed"
   cat >"${fixture}/prebuilt/fuzz_create_table" <<'EOF'
 #!/usr/bin/env bash
 set -euo pipefail
 printf '%s\t%s\t%s\n' "${GT_FUZZ_DUMP_DIR}" "$*" "${MOCK_BINARY_MARKER:-}" >>"${MOCK_BINARY_LOG}"
+[[ -d "$1" ]] || exit 18
+printf '%s\n' "$1" >>"${MOCK_CORPUS_LOG}"
+cat "$1/seed" >>"${MOCK_SEED_LOG}"
+printf 'new prebuilt input\n' >"$1/learned"
+exit "${MOCK_BINARY_EXIT_CODE}"
 EOF
   chmod +x "${fixture}/prebuilt/fuzz_create_table"
 
   set +e
   env \
+    "PATH=${fixture}/bin:${PATH}" \
+    "MOCK_CARGO_LOG=${fixture}/cargo.log" \
     "MOCK_BINARY_LOG=${fixture}/binary.log" \
     "MOCK_BINARY_MARKER=fixture" \
+    "MOCK_BINARY_EXIT_CODE=${1:-0}" \
+    "MOCK_CORPUS_LOG=${fixture}/corpus.log" \
+    "MOCK_SEED_LOG=${fixture}/seeds.log" \
+    "MOCK_COLLECTOR_LOG=${fixture}/collector.log" \
     "FUZZ_TARGETS=fuzz_create_table" \
     "FUZZ_GROUP=test-group" \
     "FUZZ_MAX_TOTAL_TIME=120" \
@@ -108,6 +131,7 @@ EOF
     "FUZZ_FAIL_FAST=true" \
     "FUZZ_COLLECT_CLUSTER_ARTIFACTS=false" \
     "FUZZ_ARTIFACT_ROOT=${fixture}/artifacts" \
+    "FUZZ_CORPUS_ROOT=${fixture}/corpus with spaces" \
     "FUZZ_ARTIFACT_COLLECTOR=${fixture}/collector" \
     "FUZZ_BIN_DIR=${fixture}/prebuilt" \
     "GITHUB_SHA=deadbeef" \
@@ -143,6 +167,7 @@ test_successful_targets_run_in_order() {
     "target order"
   grep -q -- '--features=unstable' "${fixture}/cargo.log" || fail "unstable feature missing"
   grep -q -- '-max_total_time=120' "${fixture}/cargo.log" || fail "fuzz time missing"
+  grep -q -- '-print_final_stats=1' "${fixture}/cargo.log" || fail "fuzz statistics missing"
   grep -q -- '-artifact_prefix=.*/targets/fuzz_create_table/libfuzzer/' "${fixture}/cargo.log" || \
     fail "target-scoped libFuzzer prefix missing"
   assert_eq success "$(jq -r '.[0].status' "${fixture}/artifacts/manifest.json")" "first status"
@@ -167,8 +192,49 @@ test_prebuilt_binary_runs_without_cargo() {
   [[ ! -e "${fixture}/cargo.log" ]] || fail "cargo ran for prebuilt binary"
   assert_eq 1 "$(wc -l <"${fixture}/binary.log" | tr -d ' ')" "prebuilt invocation count"
   grep -q -- '-max_total_time=120' "${fixture}/binary.log" || fail "prebuilt fuzz time missing"
+  grep -q -- '-print_final_stats=1' "${fixture}/binary.log" || fail "prebuilt fuzz statistics missing"
   grep -q -- '-artifact_prefix=.*/targets/fuzz_create_table/libfuzzer/' "${fixture}/binary.log" || \
     fail "prebuilt target-scoped libFuzzer prefix missing"
+  assert_eq "${fixture}/corpus with spaces/fuzz_create_table" "$(cat "${fixture}/corpus.log")" \
+    "prebuilt corpus argument"
+  assert_eq 'restored prebuilt seed' "$(cat "${fixture}/seeds.log")" "prebuilt restored input"
+  assert_file "${fixture}/corpus with spaces/fuzz_create_table/learned"
+  cleanup_fixture
+}
+
+test_cargo_reuses_separate_target_corpora() {
+  new_fixture
+  mkdir -p "${fixture}/corpus with spaces/fuzz_create_table" "${fixture}/corpus with spaces/fuzz_insert"
+  printf 'create seed\n' >"${fixture}/corpus with spaces/fuzz_create_table/seed"
+  printf 'insert seed\n' >"${fixture}/corpus with spaces/fuzz_insert/seed"
+  run_fixture $'fuzz_create_table\nfuzz_insert' false false ""
+
+  assert_eq 0 "${fixture_status}" "restored cargo run status"
+  assert_eq $'create seed\ninsert seed' "$(cat "${fixture}/seeds.log")" "target-scoped restored inputs"
+  assert_eq \
+    "fuzz_create_table"$'\t'"${fixture}/corpus with spaces/fuzz_create_table"$'\n'"fuzz_insert"$'\t'"${fixture}/corpus with spaces/fuzz_insert" \
+    "$(cat "${fixture}/corpus.log")" "cargo corpus arguments"
+  assert_eq fuzz_create_table "$(cat "${fixture}/corpus with spaces/fuzz_create_table/learned")" \
+    "first target learned input"
+  assert_eq fuzz_insert "$(cat "${fixture}/corpus with spaces/fuzz_insert/learned")" \
+    "second target learned input"
+
+  run_fixture fuzz_create_table true false ""
+  assert_eq 0 "${fixture_status}" "repeated cargo run status"
+  assert_file "${fixture}/corpus with spaces/fuzz_create_table/seed"
+  assert_file "${fixture}/corpus with spaces/fuzz_insert/learned"
+  cleanup_fixture
+}
+
+test_prebuilt_failure_preserves_corpus_and_artifacts() {
+  run_prebuilt_fixture 17
+
+  assert_eq 1 "${fixture_status}" "prebuilt failure status"
+  assert_eq 17 "$(jq -r '.[0].exit_code' "${fixture}/artifacts/manifest.json")" \
+    "prebuilt failure exit code"
+  assert_file "${fixture}/corpus with spaces/fuzz_create_table/seed"
+  assert_file "${fixture}/corpus with spaces/fuzz_create_table/learned"
+  assert_file "${fixture}/artifacts/targets/fuzz_create_table/mock-artifacts/state.txt"
   cleanup_fixture
 }
 
@@ -183,6 +249,8 @@ test_fail_fast_stops_after_first_failure() {
   assert_eq skipped_after_failure "$(jq -r '.[2].status' "${fixture}/artifacts/manifest.json")" "skipped status"
   assert_eq true "$(jq -r '.[2].after_prior_failure' "${fixture}/artifacts/manifest.json")" "skipped provenance"
   assert_file "${fixture}/artifacts/targets/fuzz_insert/mock-artifacts/state.txt"
+  assert_file "${fixture}/corpus with spaces/fuzz_insert/learned"
+  [[ ! -e "${fixture}/corpus with spaces/fuzz_alter_table" ]] || fail "skipped target corpus created"
   grep -q 'title=Fuzz target failed.*target=fuzz_insert' "${fixture}/stdout.log" || \
     fail "target failure annotation missing"
   grep -q 'title=Fuzz target skipped.*target=fuzz_alter_table' "${fixture}/stdout.log" || \
@@ -191,6 +259,8 @@ test_fail_fast_stops_after_first_failure() {
     fail "reproduction section missing"
   grep -q 'cargo +nightly fuzz run fuzz_insert' "${fixture}/artifacts/summary.md" || \
     fail "reproduction command missing"
+  grep -q 'fuzz_insert --fuzz-dir tests-fuzz' "${fixture}/artifacts/summary.md" || \
+    fail "reproduction command depends on the CI corpus path"
   cleanup_fixture
 }
 
@@ -241,6 +311,56 @@ test_invalid_configuration_fails_before_cargo() {
   cleanup_fixture
 }
 
+prepare_corpus_fixture() {
+  local targets="$1"
+  local output="$2"
+  set +e
+  FUZZ_TARGETS="${targets}" \
+    GITHUB_WORKSPACE="${fixture}/workspace with spaces" \
+    GITHUB_OUTPUT="${output}" \
+    bash "${PREPARE_CORPUS}" >"${fixture}/prepare.log" 2>&1
+  prepare_status=$?
+  set -e
+}
+
+test_corpus_cache_identity_and_paths() {
+  new_fixture
+  prepare_corpus_fixture $'fuzz_insert\r\n\nfuzz_create_table\r\nfuzz_insert\r' "${fixture}/first-output"
+  assert_eq 0 "${prepare_status}" "corpus preparation status"
+  prepare_corpus_fixture $'fuzz_create_table\nfuzz_insert' "${fixture}/second-output"
+  assert_eq 0 "${prepare_status}" "reordered corpus preparation status"
+  cmp -s "${fixture}/first-output" "${fixture}/second-output" || \
+    fail "cache identity depends on target order, duplicates, or line endings"
+  assert_eq \
+    "${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_create_table"$'\n'"${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_insert" \
+    "$(sed '1,2d; $d' "${fixture}/first-output")" "selected corpus cache paths"
+  [[ -d "${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_insert" ]] || \
+    fail "corpus directory not created"
+  [[ ! -e "${fixture}/workspace with spaces/tests-fuzz/corpus/fuzz_alter_table" ]] || \
+    fail "unselected corpus directory created"
+
+  GITHUB_SHA=another-revision prepare_corpus_fixture $'fuzz_create_table\nfuzz_insert' "${fixture}/new-revision-output"
+  cmp -s "${fixture}/first-output" "${fixture}/new-revision-output" || \
+    fail "source revision changed cache identity"
+  prepare_corpus_fixture fuzz_insert "${fixture}/subset-output"
+  assert_eq 0 "${prepare_status}" "subset corpus preparation status"
+  [[ "$(head -n 1 "${fixture}/first-output")" != "$(head -n 1 "${fixture}/subset-output")" ]] || \
+    fail "different target sets share cache identity"
+  cleanup_fixture
+}
+
+test_corpus_preparation_rejects_invalid_targets_before_writing() {
+  local targets
+  for targets in $'fuzz_insert\n../escape' 'fuzz target' $'\n\r\n'; do
+    new_fixture
+    prepare_corpus_fixture "${targets}" "${fixture}/output"
+    assert_eq 2 "${prepare_status}" "invalid corpus target status"
+    [[ ! -e "${fixture}/output" ]] || fail "invalid targets wrote cache outputs"
+    [[ ! -e "${fixture}/workspace with spaces" ]] || fail "invalid targets created directories"
+    cleanup_fixture
+  done
+}
+
 test_collector_keeps_target_scopes_separate() {
   new_fixture
   service_log="${fixture}/greptime.log"
@@ -373,10 +493,14 @@ EOF
 
 test_successful_targets_run_in_order
 test_prebuilt_binary_runs_without_cargo
+test_cargo_reuses_separate_target_corpora
+test_prebuilt_failure_preserves_corpus_and_artifacts
 test_fail_fast_stops_after_first_failure
 test_fail_fast_defaults_to_true
 test_non_fail_fast_marks_later_targets
 test_invalid_configuration_fails_before_cargo
+test_corpus_cache_identity_and_paths
+test_corpus_preparation_rejects_invalid_targets_before_writing
 test_collector_keeps_target_scopes_separate
 test_cluster_collector_honors_target_scope_and_namespace
 test_setup_failure_writes_artifact_contract

.github/scripts/run-fuzz-targets.sh

@@ -5,6 +5,7 @@ set -euo pipefail
 FUZZ_ARTIFACT_ROOT="${FUZZ_ARTIFACT_ROOT:-/tmp/greptime-fuzz-artifacts}"
 FUZZ_ARTIFACT_COLLECTOR="${FUZZ_ARTIFACT_COLLECTOR:-.github/scripts/collect-fuzz-target-artifacts.sh}"
 FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS="${FUZZ_ARTIFACT_COLLECTION_TIMEOUT_SECS:-180}"
+FUZZ_CORPUS_ROOT="${FUZZ_CORPUS_ROOT:-tests-fuzz/corpus}"
 
 log() {
   printf '[run-fuzz-targets] %s\n' "$*"
@@ -187,19 +188,22 @@ for ((index = 0; index < ${#targets[@]}; index++)); do
   started_epoch="$(date +%s)"
   echo "::group::Fuzz target: ${target}"
   log "run target ${target}"
+  mkdir -p "${FUZZ_CORPUS_ROOT}/${target}"
+  corpus_dir="$(cd -- "${FUZZ_CORPUS_ROOT}/${target}" && pwd)"
+  log "corpus directory: ${corpus_dir}"
   if [[ -n "${FUZZ_BIN_DIR:-}" ]]; then
     fuzz_binary="${FUZZ_BIN_DIR}/${target}"
     if [[ ! -x "${fuzz_binary}" ]]; then
       log "prebuilt fuzz binary does not exist or is not executable: ${fuzz_binary}"
       exit 2
     fi
-    run_args=("${fuzz_binary}" "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-artifact_prefix=${target_dir}/libfuzzer/")
+    run_args=("${fuzz_binary}" "${corpus_dir}" "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-print_final_stats=1" "-artifact_prefix=${target_dir}/libfuzzer/")
   else
-    run_args=(cargo +"${RUSTUP_TOOLCHAIN:-nightly}" fuzz run "${target}" --fuzz-dir tests-fuzz -D -s none)
+    run_args=(cargo +"${RUSTUP_TOOLCHAIN:-nightly}" fuzz run "${target}" "${corpus_dir}" --fuzz-dir tests-fuzz -D -s none)
     if [[ "${fuzz_unstable}" == true ]]; then
       run_args+=(--features=unstable)
     fi
-    run_args+=(-- "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-artifact_prefix=${target_dir}/libfuzzer/")
+    run_args+=(-- "-max_total_time=${FUZZ_MAX_TOTAL_TIME}" "-print_final_stats=1" "-artifact_prefix=${target_dir}/libfuzzer/")
   fi
   if GT_FUZZ_DUMP_DIR="${target_dir}/csv" \
     "${run_args[@]}" 2>&1 | tee "${target_dir}/fuzz.log"; then

.github/scripts/tests/ci-cache-policy.test.cjs

@@ -0,0 +1,155 @@
+const assert = require('node:assert/strict');
+const { readFileSync } = require('node:fs');
+const { resolve } = require('node:path');
+const { test } = require('node:test');
+
+const directory = resolve(__dirname, '../../workflows');
+const workflows = ['checks', 'rust', 'integration', 'nightly-ci'].map(name => ({
+  name,
+  text: readFileSync(resolve(directory, name + '.yml'), 'utf8'),
+}));
+function jobs(text) {
+  return [...text.matchAll(/^  ([\w-]+):\n([\s\S]*?)(?=^  [\w-]+:\n|(?![\s\S]))/gm)];
+}
+function steps(body) {
+  return body.split(/^      - /m).slice(1);
+}
+
+test('build caches have one controller and lightweight checks do not cache targets', () => {
+  let checked = 0;
+  for (const { name, text } of workflows) {
+    for (const [, job, body] of jobs(text)) {
+      if (!body.includes('Swatinem/rust-cache@') && !['fmt', 'check-udeps', 'toml'].includes(job)) continue;
+      for (const step of steps(body).filter(s => s.includes('uses: actions-rust-lang/setup-rust-toolchain@'))) {
+        assert.match(step, /^          cache: false$/m, name + '/' + job);
+        checked++;
+      }
+    }
+  }
+  assert.ok(checked >= 10, 'expected build and lightweight jobs to be covered');
+});
+
+test('only main writes caches, and the nightly read-only cache stays read-only', () => {
+  let checked = 0;
+  for (const { name, text } of workflows) {
+    for (const [, job, body] of jobs(text)) {
+      for (const step of steps(body)) {
+        if (!step.includes('uses: Swatinem/rust-cache@') && !step.includes('uses: actions/cache/save@')) continue;
+        const condition = step.includes('uses: Swatinem/rust-cache@')
+          ? step.match(/^          save-if: (.+)$/m)?.[1]
+          : step.match(/^        if: (.+)$/m)?.[1];
+        assert.ok(condition, name + '/' + job);
+        const expression = condition.replace(/^\$\{\{\s*|\s*\}\}$/g, '')
+          .replaceAll('steps.fuzz-binaries.outputs.cache-hit', 'hit');
+        const enabled = new Function('github', 'hit', 'return (' + expression + ');');
+        for (const [event, ref] of [
+          ['pull_request', 'refs/pull/123/merge'],
+          ['merge_group', 'refs/heads/gh-readonly-queue/main/pr-123'],
+          ['workflow_dispatch', 'refs/heads/feature'],
+        ]) assert.equal(enabled({ event_name: event, ref }, 'false'), false, name + '/' + job);
+        assert.equal(enabled({ event_name: 'schedule', ref: 'refs/heads/main' }, 'false'),
+          condition !== 'false', name + '/' + job);
+        if (step.includes('uses: actions/cache/save@')) {
+          assert.equal(enabled({ ref: 'refs/heads/main' }, 'true'), false);
+        }
+        checked++;
+      }
+    }
+  }
+  assert.ok(checked >= 10);
+});
+
+test('helper tool versions do not fragment unrelated Rust caches', () => {
+  for (const { text } of workflows) {
+    assert.doesNotMatch(text, /CARGO_(SHEAR|FUZZ)_VERSION/);
+  }
+  const integration = workflows.find(w => w.name === 'integration').text;
+  assert.match(integration, /prefix-key: "v1-fuzz-\$\{\{ env.FUZZ_VERSION \}\}"/);
+  assert.match(integration, /cargo install cargo-fuzz --version "\$\{FUZZ_VERSION\}"/);
+});
+
+// Evaluate the actual job guard as well as the cache's save condition. A valid
+// save-if alone cannot prove that a scheduled cache producer is reachable.
+function warmupJob(workflow, job, {
+  event = 'schedule', ref = 'refs/heads/main',
+  profile = event === 'workflow_dispatch' ? 'all' : undefined,
+  outputs = {}, failed = '', cancelled = false, draft = false,
+} = {}) {
+  const text = workflows.find(w => w.name === workflow).text;
+  assert.match(text, /^  schedule:$/m);
+  const body = jobs(text).find(([, name]) => name === job)?.[2];
+  assert.ok(body, workflow + '/' + job);
+  const github = { repository: 'GreptimeTeam/greptimedb', event_name: event, ref,
+    event: { pull_request: { draft } }, run_attempt: 1 };
+  const needs = { scope: { outputs }, ...Object.fromEntries(
+    ['github-script-tests', 'build', 'prepare-fuzz-targets', 'build-greptime-ci'].map(name =>
+      [name, { result: name === failed ? 'failure' : 'success' }])) };
+  const evaluate = expression => new Function('github', 'inputs', 'needs', 'cancelled',
+    'return (' + expression.replace(/^\$\{\{\s*|\s*\}\}$/g, '') + ');')(
+    github, { fuzz_profile: profile, ci_command: false }, needs, () => cancelled);
+  const admitted = evaluate(body.match(/^    if: (.+)$/m)[1]);
+  const saves = [...body.matchAll(/^          save-if: (.+)$/gm)]
+    .map(match => admitted && evaluate(match[1]));
+  return { admitted, saves };
+}
+
+const fuzzJobs = ['prepare-fuzz-targets', 'build-greptime-ci', 'fuzztest',
+  'unstable-fuzztest', 'distributed-fuzztest', 'distributed-fuzztest-with-chaos'];
+
+test('main schedules can execute cache producers and their fuzz consumers', () => {
+  for (const job of fuzzJobs) {
+    // Scheduled events do not supply workflow_dispatch inputs.
+    const result = warmupJob('integration', job, { profile: undefined });
+    assert.equal(result.admitted, true, job);
+    assert.ok(result.saves.every(Boolean), job);
+  }
+  for (const [workflow, job] of [
+    ['integration', 'build'], ['integration', 'export-import-v2-e2e'],
+    ['integration', 'prepare-fuzz-targets'], ['integration', 'build-greptime-ci'],
+    ['integration', 'distributed-fuzztest'],
+    ['rust', 'check-windows'], ['rust', 'check-riscv64'],
+    ['nightly-ci', 'sqlness-windows'], ['nightly-ci', 'test-on-windows'],
+  ]) {
+    for (const event of ['schedule', 'workflow_dispatch']) {
+      assert.deepEqual(warmupJob(workflow, job, { event }),
+        { admitted: true, saves: [true] }, workflow + '/' + job + '/' + event);
+    }
+  }
+});
+
+test('manual warming respects selected profiles without an internal command marker', () => {
+  const expected = {
+    none: [],
+    standalone: fuzzJobs.slice(0, 4),
+    distributed: [fuzzJobs[0], fuzzJobs[1], fuzzJobs[4]],
+    chaos: [fuzzJobs[0], fuzzJobs[1], fuzzJobs[5]],
+    all: fuzzJobs,
+  };
+  for (const [profile, wanted] of Object.entries(expected)) {
+    assert.deepEqual(fuzzJobs.filter(job => warmupJob('integration', job,
+      { event: 'workflow_dispatch', profile }).admitted), wanted, profile);
+  }
+  for (const job of fuzzJobs) {
+    const result = warmupJob('integration', job,
+      { event: 'workflow_dispatch', ref: 'refs/heads/feature' });
+    assert.ok(result.saves.every(value => value === false), job);
+    assert.equal(warmupJob('integration', job,
+      { event: 'pull_request', ref: 'refs/pull/1/merge', draft: true }).admitted, false, job);
+  }
+});
+
+test('warmup keeps scope exclusions, dependency failures and cancellation gates', () => {
+  const gates = ['any_fuzz', 'build_ci_needed', 'fuzz_standalone',
+    'fuzz_unstable', 'fuzz_distributed', 'fuzz_chaos'];
+  for (const [index, job] of fuzzJobs.entries()) {
+    assert.equal(warmupJob('integration', job, { failed: 'github-script-tests' }).admitted, false, job);
+    assert.equal(warmupJob('integration', job, { cancelled: true }).admitted, false, job);
+    assert.equal(warmupJob('integration', job,
+      { outputs: { [gates[index]]: 'false' } }).admitted, false, job);
+  }
+  for (const job of fuzzJobs.slice(2)) {
+    for (const failed of ['github-script-tests', 'prepare-fuzz-targets', job === 'fuzztest' ? 'build' : 'build-greptime-ci']) {
+      assert.equal(warmupJob('integration', job, { failed }).admitted, false, job);
+    }
+  }
+});

.github/scripts/tests/nightly-merge.test.cjs

@@ -0,0 +1,153 @@
+// Copyright 2023 Greptime Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Run with: node --test .github/scripts/tests/nightly-merge.test.cjs
+const { readFileSync } = require('node:fs');
+const { resolve } = require('node:path');
+const assert = require('node:assert/strict');
+const { test } = require('node:test');
+const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor;
+const workflow = readFileSync(resolve(__dirname, '../../workflows/nightly-merge.yml'), 'utf8');
+const scripts = [...workflow.matchAll(/          script: \|\n((?:            .*\n|\n)+)/g)]
+  .map(match => new AsyncFunction('github', 'context', 'core', 'process', match[1].replace(/^            /gm, '')));
+assert.equal(scripts.length, 2);
+const checked = '- [x] Merge during the nightly window <!-- nightly-merge -->';
+
+async function run({ sync = false, body = checked, labels = ['merge nightly'],
+  detail = {}, current = {}, threads = [[true]], count = 1, dryRun = false,
+  failFirst = false, enqueueError = Error('head changed') } = {}) {
+  const calls = [];
+  let page = 0;
+  const github = {
+    rest: {
+      pulls: { get: async () => ({ data: { state: 'open', draft: false, base: { ref: 'main' },
+        body, labels: labels.map(name => ({ name })), ...current } }) },
+      issues: {
+        listForRepo: () => {},
+        addLabels: async args => calls.push(['add', args.labels]),
+        removeLabel: async args => calls.push(['remove', args.name])
+      }
+    },
+    paginate: async () => Array.from({ length: count }, (_, i) => ({ number: i + 1, pull_request: {} })),
+    graphql: async (query, args) => {
+      if (query.includes('mutation')) {
+        calls.push(['enqueue', args]);
+        if (failFirst && calls.filter(call => call[0] === 'enqueue').length === 1) throw enqueueError;
+        return {};
+      }
+      if (query.includes('reviewThreads')) {
+        const index = page++ % threads.length;
+        calls.push(['threads', args.cursor]);
+        return { repository: { pullRequest: { reviewThreads: {
+          nodes: threads[index].map(isResolved => ({ isResolved })),
+          pageInfo: { hasNextPage: index + 1 < threads.length, endCursor: `page-${index}` }
+        } } } };
+      }
+      return { repository: { pullRequest: { id: `PR_${args.number}`, headRefOid: 'abc',
+        reviewDecision: 'APPROVED', mergeStateStatus: 'CLEAN', ...detail } } };
+    }
+  };
+  const summary = { addHeading() { return this; }, addTable(rows) { calls.push(['summary', rows]); return this; }, async write() {} };
+  await scripts[sync ? 0 : 1](github,
+    { repo: { owner: 'GreptimeTeam', repo: 'greptimedb' }, payload: { pull_request: { number: 1 } } },
+    { summary, info() {}, warning() {}, setFailed: message => calls.push(['failed', message]) },
+    { env: { DRY_RUN: String(dryRun) } });
+  return calls;
+}
+
+test('label sync grants PR label write permission without checking out PR code', () => {
+  const syncJob = workflow.split('  sync-label:\n')[1].split('  enqueue:\n')[0];
+  assert.match(syncJob, /^      pull-requests: write$/m);
+  assert.doesNotMatch(syncJob, /uses: actions\/checkout/);
+});
+
+test('template opts in and checkbox edits reconcile only the nightly label', async () => {
+  assert.ok(readFileSync(resolve(__dirname, '../../pull_request_template.md'), 'utf8').includes(checked));
+  assert.deepEqual(await run({ sync: true, labels: ['bug'] }), [['add', ['merge nightly']]]);
+  assert.deepEqual(await run({ sync: true }), []);
+  for (const body of [checked.replace('[x]', '[ ]')]) {
+    assert.deepEqual(await run({ sync: true, body }), [['remove', 'merge nightly']]);
+  }
+  assert.deepEqual(await run({ sync: true, body: checked.replace('[x]', '[X]'), labels: [] }), [['add', ['merge nightly']]]);
+});
+
+test('eligible PR is enqueued with its checked head; dry run performs no mutation', async () => {
+  assert.deepEqual((await run()).filter(c => c[0] === 'enqueue'), [['enqueue', { id: 'PR_1', sha: 'abc' }]]);
+  assert.equal((await run({ dryRun: true })).some(c => c[0] === 'enqueue'), false);
+});
+
+test('each eligibility gate independently prevents enqueueing', async () => {
+  for (const options of [
+    { labels: [] },
+    { current: { draft: true } }, { current: { state: 'closed' } }, { current: { base: { ref: 'release/v1.3' } } },
+    { detail: { reviewDecision: 'REVIEW_REQUIRED' } }, { detail: { reviewDecision: 'CHANGES_REQUESTED' } },
+    { detail: { mergeStateStatus: 'BLOCKED' } }, { detail: { mergeStateStatus: 'UNKNOWN' } },
+    { detail: { mergeQueueEntry: { id: 'entry' } } }, { detail: { autoMergeRequest: { enabledAt: 'now' } } },
+    { threads: [[false]] }, { threads: [[true], [false]] }
+  ]) {
+    assert.equal((await run(options)).some(c => c[0] === 'enqueue'), false, JSON.stringify(options));
+  }
+});
+
+test('all thread pages are checked before enqueueing', async () => {
+  const calls = await run({ threads: [[true], [true]] });
+  assert.deepEqual(calls.filter(c => c[0] === 'threads'), [['threads', null], ['threads', 'page-0']]);
+  assert.equal(calls.filter(c => c[0] === 'enqueue').length, 1);
+});
+
+test('per-run cap and individual failures do not prevent later candidates', async () => {
+  assert.equal((await run({ count: 8 })).filter(c => c[0] === 'enqueue').length, 5);
+  const calls = await run({ count: 2, failFirst: true });
+  assert.equal(calls.filter(c => c[0] === 'enqueue').length, 2);
+  assert.equal(calls.filter(c => c[0] === 'failed').length, 1);
+});
+
+test('label-only opt-in survives PRs without a template and does not require a checkbox', async () => {
+  for (const body of ['', null, 'Existing PR description', `> ${checked}`]) {
+    assert.deepEqual(await run({ sync: true, body }), []);
+    assert.deepEqual(await run({ sync: true, body, labels: [] }), []);
+    assert.equal((await run({ body })).filter(call => call[0] === 'enqueue').length, 1);
+    assert.equal((await run({ body, labels: [] })).some(call => call[0] === 'enqueue'), false);
+  }
+  // The label is authoritative for admission, even before checkbox synchronization.
+  assert.equal((await run({ body: checked.replace('[x]', '[ ]') })).some(call => call[0] === 'enqueue'), true);
+});
+
+const stackMessage = 'This pull request is part of a stack and must be enqueued using the asynchronous merge REST API. For more information, see https://docs.github.com/rest/pulls/pulls#merge-a-pull-request-asynchronously';
+
+test('stacked PR rejection is skipped without consuming the successful enqueue cap', async () => {
+  const calls = await run({ count: 7, failFirst: true,
+    enqueueError: Object.assign(Error('GraphQL request failed'), { errors: [{ message: stackMessage }] }) });
+  assert.equal(calls.some(c => c[0] === 'failed'), false);
+  assert.deepEqual(calls.find(c => c[0] === 'summary')[1], [
+    ['PR', 'Result'], ['1', 'Skipped: stacked PR'],
+    ...[2, 3, 4, 5, 6].map(number => [String(number), 'Enqueued'])
+  ]);
+  assert.deepEqual(calls.filter(c => c[0] === 'enqueue').map(c => c[1].id),
+    ['PR_1', 'PR_2', 'PR_3', 'PR_4', 'PR_5', 'PR_6']);
+});
+
+test('mixed or unstructured errors are not mistaken for a stacked PR rejection', async () => {
+  for (const enqueueError of [
+    Error(stackMessage),
+    Object.assign(Error('GraphQL request failed'), { errors: [] }),
+    Object.assign(Error('GraphQL request failed'), { errors: [{ message: stackMessage }, { message: 'Forbidden' }] })
+  ]) {
+    const calls = await run({ count: 2, failFirst: true, enqueueError });
+    assert.equal(calls.filter(c => c[0] === 'failed').length, 1);
+    assert.deepEqual(calls.find(c => c[0] === 'summary')[1], [
+      ['PR', 'Result'], ['1', 'Failed: see workflow logs'], ['2', 'Enqueued']
+    ]);
+  }
+});

.github/scripts/tests/test_metric_snapshot_s3_bench.py

@@ -28,6 +28,25 @@ SPEC.loader.exec_module(bench)
 
 
 class PhaseTests(unittest.TestCase):
+    def test_cloud_region_configuration(self):
+        for bucket, runner, accepted in (("us-east-1", "us-east-1", True),
+                                          ("ap-southeast-1", "ap-southeast-1", True),
+                                          ("us-east-1", "ap-southeast-1", False),
+                                          ("", "", False)):
+            with self.subTest(bucket=bucket, runner=runner), tempfile.TemporaryDirectory() as directory:
+                env = dict(GREPTIMEDB_IMAGE="greptime/greptimedb@sha256:" + "a" * 64,
+                           EXPECTED_COMMIT="b" * 40, RUN_PREFIX="metric-snapshot-bench/123-1/",
+                           RESULTS=str(Path(directory) / "results"), DATA_ROOT=str(Path(directory) / "data"),
+                           S3_REGION=bucket, RUNNER_REGION=runner, S3_ENDPOINT="")
+                with patch.dict(bench.os.environ, env):
+                    if accepted:
+                        benchmark = bench.Benchmark(local_smoke=False)
+                        benchmark.summary()
+                        self.assertIn(f"S3 in {bucket};", (benchmark.results / "summary.md").read_text())
+                    else:
+                        with self.assertRaisesRegex(ValueError, "same nonempty"):
+                            bench.Benchmark(local_smoke=False)
+
     def test_docker_nanosecond_timestamps(self):
         start = bench.docker_timestamp("2026-10-14T06:06:44.72000818Z")
         end = bench.docker_timestamp("2026-10-14T06:06:46.123456789Z")

.github/scripts/tests/test_metric_snapshot_s3_ops.py

@@ -20,6 +20,7 @@ import io
 import json
 from pathlib import Path
 import tempfile
+import types
 import unittest
 from unittest import mock
 
@@ -82,6 +83,45 @@ class OperationsTest(unittest.TestCase):
             with self.subTest(changes=changes), self.assertRaises(ValueError):
                 ops.plan(self.env(**changes))
 
+    def test_regions_follow_configuration_and_reject_mismatch_or_empty(self):
+        for region in ("us-east-1", "ap-southeast-1"):
+            with self.subTest(region=region):
+                self.assertEqual(ops.plan(self.env(S3_REGION=region, RUNNER_REGION=region))["region"], region)
+        for bucket, runner in (("us-east-1", "ap-southeast-1"), ("", ""), (" ", " ")):
+            with self.subTest(bucket=bucket, runner=runner), self.assertRaises(ValueError):
+                ops.plan(self.env(S3_REGION=bucket, RUNNER_REGION=runner))
+
+    def test_client_uses_configured_region(self):
+        boto = types.ModuleType("boto3")
+        boto.client = mock.Mock()
+        config = types.ModuleType("botocore.config")
+        config.Config = mock.Mock()
+        with mock.patch.dict("sys.modules", {"boto3": boto, "botocore.config": config}):
+            for region in ("us-east-1", "ap-southeast-1"):
+                ops.client(region)
+                self.assertEqual(boto.client.call_args.kwargs["region_name"], region)
+            with self.assertRaises(ValueError):
+                ops.client("")
+
+    def test_hourly_estimate_controls_reservation_and_cost(self):
+        env = self.env(EC2_HOURLY_USD="2", BENCHMARK_TIMEOUT_MINUTES="30")
+        plan = ops.plan(env)
+        self.assertEqual(plan["ec2_hourly_usd"], "2")
+        self.assertEqual(plan["attempt_reserve_usd"], "5.00")
+        with mock.patch.dict(ops.os.environ, env), mock.patch.object(ops.time, "time", return_value=4600), \
+                mock.patch("sys.argv", ["ops", "cost", "--started-at", "1000"]), \
+                mock.patch("sys.stdout", new_callable=io.StringIO) as output:
+            ops.main()
+        cost = json.loads(output.getvalue())
+        self.assertEqual(cost["ec2_hourly_usd"], "2")
+        self.assertEqual(cost["ec2_estimate_usd"], "2.0000")
+        self.assertEqual(cost["campaign_estimate_with_other_reserve_usd"], "3.5000")
+        for rate in ("", "invalid", "0", "-1", "NaN", "Infinity", "-Infinity"):
+            with self.subTest(rate=rate), self.assertRaises(ValueError):
+                ops.plan(self.env(EC2_HOURLY_USD=rate))
+        with self.assertRaises(ValueError):
+            ops.plan(self.env(EC2_HOURLY_USD="4"))
+
     def watcher_env(self):
         return self.env(GITHUB_REPOSITORY="GreptimeTeam/greptimedb", GITHUB_RUN_ID="123",
                         GITHUB_RUN_ATTEMPT="2", GH_TOKEN="test-token", BENCHMARK_TIMEOUT_MINUTES="30")

.github/scripts/update-query-regression-helpers-image.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+set -euo pipefail
+: "${HELPERS_IMAGE:?Verified image digest is required}"
+: "${GITHUB_REPOSITORY:?Repository is required}"
+repository=greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/query-regression-helpers
+[[ "$HELPERS_IMAGE" == "$repository@sha256:"* ]]
+[[ "${HELPERS_IMAGE#"$repository@"}" =~ ^sha256:[0-9a-f]{64}$ ]]
+# Like QUERY_REGRESSION_ECS_IMAGE_ID, this is CI configuration, not a source
+# change. Create on first publication, then update only after image verification.
+gh variable set QUERY_REGRESSION_HELPERS_IMAGE \
+  --repo "$GITHUB_REPOSITORY" --body "$HELPERS_IMAGE"
+gh api "repos/$GITHUB_REPOSITORY/actions/variables/QUERY_REGRESSION_HELPERS_IMAGE" \
+  | jq -e --arg expected "$HELPERS_IMAGE" '.value == $expected' >/dev/null
+printf 'QUERY_REGRESSION_HELPERS_IMAGE updated to `%s`\n' "$HELPERS_IMAGE"

.github/workflows/agent-observability.yml

@@ -2,6 +2,11 @@ name: Agent Observability Benchmark
 on:
   workflow_dispatch:
     inputs:
+      greptimedb_storage:
+        description: 'GreptimeDB data storage (S3 requires AWS in the bucket region; WAL stays local)'
+        type: choice
+        options: [file, s3, both]
+        default: file
       run_greptimedb:
         description: Run GreptimeDB
         type: boolean
@@ -56,7 +61,7 @@ on:
         type: string
         required: false
       runtime_image:
-        description: 'Runtime image: auto = AWS ECR Public or Aliyun ACR by provider; explicit tag/digest overrides'
+        description: 'Runtime image: auto = pinned S3 test runtime on ACR; explicit tag/digest overrides'
         type: string
         default: auto
         required: true
@@ -99,6 +104,11 @@ on:
 permissions:
   contents: read
 env:
+  GREPTIMEDB_STORAGE: ${{ inputs.greptimedb_storage || 'file' }}
+  S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }}
+  S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
+  RUNNER_REGION: ${{ vars.EC2_RUNNER_REGION }}
+  RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }}
   COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }}
   BASELINE_GREPTIMEDB_TAG: ${{ inputs.baseline_greptimedb_tag }}
   PROFILE: ${{ inputs.dataset }}
@@ -106,7 +116,7 @@ env:
   CLICKHOUSE_TAG: ${{ inputs.clickhouse_tag }}
   VICTORIALOGS_TAG: ${{ inputs.victorialogs_tag }}
   O11YBENCH_REF: ${{ inputs.o11ybench_ref }}
-  RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || inputs.provider == 'AWS' && 'public.ecr.aws/i8k6a5e1/greptime/o11ybenchruntime:2f40fb3fdbdc@sha256:8940782216bf5d066308fd71f6e346d950073c41cfb5756c5f3dc8aacf069cbc' || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:2f40fb3fdbdc@sha256:8940782216bf5d066308fd71f6e346d950073c41cfb5756c5f3dc8aacf069cbc' }}
+  RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:s3-test-b0cf5204d2fc@sha256:c2bf622db6056a2029842f4dc1cc1f37b82773bd45b134a0a83ffe2a8cd62214' }}
   PROVIDER: ${{ inputs.provider }}
   REQUESTED_INSTANCE_TYPE: ${{ inputs.instance_type }}
   SYSTEM_DISK_GIB: ${{ inputs.system_disk_gib }}
@@ -130,6 +140,10 @@ jobs:
         with:
           path: greptimedb
           persist-credentials: false
+      - name: Validate GreptimeDB storage
+        run: |
+          python3 greptimedb/tests/perf/test_o11ybench_storage.py
+          python3 greptimedb/.github/scripts/o11ybench-storage.py validate
       - name: Validate version comparison
         env:
           RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }}
@@ -144,6 +158,18 @@ jobs:
           python3 greptimedb/tests/perf/test_aliyun_ecs_runner_scripts.py
           python3 greptimedb/tests/perf/test_ci_runner_scripts.py
           python3 greptimedb/tests/perf/test_agent_observability_summary.py
+      - name: Configure AWS credentials for ECR Public
+        if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+          aws-region: us-east-1
+      - name: Login to Amazon ECR Public
+        if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+        uses: aws-actions/amazon-ecr-login@v2
+        with:
+          registry-type: public
       - name: Check runtime supports shared version-comparison data
         if: inputs.compare_greptimedb
         run: |
@@ -275,7 +301,10 @@ jobs:
           path: greptimedb
           persist-credentials: false
       - name: Set up benchmark dependencies
-        uses: ./greptimedb/.github/actions/setup-benchmark
+        uses: ./greptimedb/.github/actions/setup-linux-runner
+        with:
+          tools: curl jq python3 rsync
+          expected-architecture: x86_64
       - name: Check Docker on ephemeral Linux host
         shell: bash
         run: |
@@ -285,6 +314,18 @@ jobs:
           docker info
           # The workspace (and Docker state) must not use a small /tmp filesystem.
           df -h "$GITHUB_WORKSPACE" /var/lib/docker
+      - name: Configure AWS credentials for ECR Public
+        if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+          aws-region: us-east-1
+      - name: Login to Amazon ECR Public
+        if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+        uses: aws-actions/amazon-ecr-login@v2
+        with:
+          registry-type: public
       - name: Resolve image tags and verify runtime revision
         shell: bash
         run: |
@@ -343,24 +384,48 @@ jobs:
             --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" \
             --root "$ARTIFACT_ROOT" --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY"
           cp "$MANIFEST_PATH" "$ARTIFACT_ROOT/run-manifest.json"
-      - name: Load and benchmark baseline GreptimeDB
-        id: baseline_greptimedb
-        if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.compare_greptimedb }}
+      - name: Load and benchmark baseline GreptimeDB (File)
+        id: baseline_greptimedb_file
+        if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' && inputs.compare_greptimedb }}
         run: |-
-          GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --execute \
+          GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage file --execute \
             --owner "$OWNER-base" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT/baseline" --data-root "$ARTIFACT_ROOT" \
             --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \
             --target greptimedb --image "$BASELINE_GREPTIMEDB_IMAGE" \
             --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT"
-      - name: Load and benchmark greptimedb
-        id: greptimedb
-        if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' }}
+      - name: Load and benchmark baseline GreptimeDB (S3)
+        env:
+          AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+          AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
+        id: baseline_greptimedb_s3
+        if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' && inputs.compare_greptimedb }}
         run: |-
-          GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --execute \
+          GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage s3 --execute \
+            --owner "$OWNER-base-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT/baseline/s3" --data-root "$ARTIFACT_ROOT" \
+            --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \
+            --target greptimedb --image "$BASELINE_GREPTIMEDB_IMAGE" \
+            --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT"
+      - name: Load and benchmark greptimedb (File)
+        id: greptimedb_file
+        if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' }}
+        run: |-
+          GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage file --execute \
             --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT" \
             --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \
             --target greptimedb --image "$GREPTIMEDB_IMAGE" \
             --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT"
+      - name: Load and benchmark greptimedb (S3)
+        env:
+          AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+          AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
+        id: greptimedb_s3
+        if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' }}
+        run: |-
+          GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage target --greptimedb-storage s3 --execute \
+            --owner "$OWNER-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --root "$ARTIFACT_ROOT/s3" --data-root "$ARTIFACT_ROOT" \
+            --profile "$PROFILE" --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" \
+            --target greptimedb --image "$GREPTIMEDB_IMAGE" \
+            --greptimedb-http-port "$GREPTIMEDB_HTTP_PORT" --greptimedb-grpc-port "$GREPTIMEDB_GRPC_PORT"
       - name: Load and benchmark clickhouse
         id: clickhouse
         if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.clickhouse == 'true' }}
@@ -380,7 +445,7 @@ jobs:
             --target victorialogs --image "$VICTORIALOGS_IMAGE" \
             --victorialogs-http-port "$VICTORIALOGS_HTTP_PORT"
       - name: Compare all three targets
-        if: ${{ !cancelled() && steps.greptimedb.outcome == 'success' && steps.clickhouse.outcome == 'success' && steps.victorialogs.outcome
+        if: ${{ !cancelled() && steps.greptimedb_file.outcome == 'success' && steps.clickhouse.outcome == 'success' && steps.victorialogs.outcome
           == 'success' }}
         run: |-
           docker run --rm --user "$(id -u):$(id -g)" --label "o11ybench.owner=$OWNER" \
@@ -391,45 +456,61 @@ jobs:
             --victorialogs-summary /results/victorialogs-1/victorialogs-1/measured-summary.json \
             --output /results/comparison.json
       - name: Clean up owned containers
+        env:
+          AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+          AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
         if: always()
         run: |-
           if [[ -f o11ybench/scripts/run-agent-observability.sh ]]; then
-            GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute \
+            GITHUB_STEP_SUMMARY= python3 greptimedb/.github/scripts/o11ybench-storage.py run -- bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute --runtime-image "$RESOLVED_RUNTIME_IMAGE" \
               --owner "$OWNER" --root "$ARTIFACT_ROOT"
           fi
       - name: Clean up baseline containers
+        env:
+          AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+          AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
         if: always() && inputs.compare_greptimedb
         run: |-
           if [[ -f o11ybench/scripts/run-agent-observability.sh ]]; then
-            GITHUB_STEP_SUMMARY= bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute \
+            GITHUB_STEP_SUMMARY= python3 greptimedb/.github/scripts/o11ybench-storage.py run -- bash o11ybench/scripts/run-agent-observability.sh --stage cleanup --execute --runtime-image "$RESOLVED_RUNTIME_IMAGE" \
               --owner "$OWNER-base" --root "$ARTIFACT_ROOT/baseline"
           fi
       - name: Benchmark summary
         if: always()
         run: |
-          python3 greptimedb/.github/scripts/agent-observability-summary.py \
-            --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section benchmark >> "$GITHUB_STEP_SUMMARY"
+          python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind agent \
+            --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section benchmark | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
       - name: Load / lifecycle summary
         if: always()
         run: |
-          python3 greptimedb/.github/scripts/agent-observability-summary.py \
-            --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section lifecycle >> "$GITHUB_STEP_SUMMARY"
+          python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind agent \
+            --root "$ARTIFACT_ROOT" --targets "$TARGETS" --section lifecycle | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
       - name: Baseline load / lifecycle summary
         if: always() && inputs.compare_greptimedb
         run: |
-          echo '## Baseline GreptimeDB' >> "$GITHUB_STEP_SUMMARY"
-          python3 greptimedb/.github/scripts/agent-observability-summary.py \
-            --root "$ARTIFACT_ROOT/baseline" --targets greptimedb --section lifecycle >> "$GITHUB_STEP_SUMMARY"
+          echo '## Baseline GreptimeDB' | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+          python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind agent \
+            --root "$ARTIFACT_ROOT/baseline" --targets greptimedb --section lifecycle | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
       - name: GreptimeDB version comparison
         if: always() && inputs.compare_greptimedb
         run: |
-          python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind agent --root "$ARTIFACT_ROOT" >> "$GITHUB_STEP_SUMMARY"
+          python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind agent --root "$ARTIFACT_ROOT" | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+      - name: Render storage comparison overview
+        if: always() && steps.generate.outcome == 'success'
+        run: |
+          python3 o11ybench/scripts/build_query_performance_report.py --agent "$ARTIFACT_ROOT" \
+            --output "$ARTIFACT_ROOT-query-report.html" --json-output "$ARTIFACT_ROOT-query-report.json"
+          python3 greptimedb/.github/scripts/o11ybench-storage.py overview "$ARTIFACT_ROOT-query-report.json" \
+            | tee -a "$ARTIFACT_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+          python3 greptimedb/.github/scripts/o11ybench-storage.py audit --root "$ARTIFACT_ROOT" \
+            --report "$ARTIFACT_ROOT-query-report.json" --targets "$TARGETS"
       - name: Upload results (no dataset)
         if: always()
         uses: actions/upload-artifact@v4
         with:
           name: agent-observability-${{ github.run_id }}-${{ github.run_attempt }}
           path: |-
+            ${{ env.ARTIFACT_ROOT }}-query-report.*
             ${{ env.MANIFEST_PATH }}
             ${{ env.ARTIFACT_ROOT }}
             !${{ env.ARTIFACT_ROOT }}/corpus/agent_observations.jsonl

.github/workflows/checks.yml

@@ -107,6 +107,8 @@ jobs:
         with:
           tool: protoc@3.23.4
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
         with:
@@ -130,6 +132,8 @@ jobs:
         with:
           persist-credentials: false
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Install taplo
         uses: taiki-e/install-action@v2
         with:

.github/workflows/dev-build.yml

@@ -4,6 +4,11 @@ name: GreptimeDB Development Build
 on:
   workflow_dispatch: # Allows you to run this workflow manually.
     inputs:
+      ec2_volume_size:
+        description: Root volume size in GiB for each EC2 build runner
+        type: number
+        required: false
+        default: 500
       large-page-size:
         description: Build GreptimeDB with large page size (65536).
         type: boolean
@@ -133,6 +138,7 @@ jobs:
           aws-region: ${{ vars.EC2_RUNNER_REGION }}
           github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
           image-id: ${{ vars.EC2_RUNNER_LINUX_AMD64_IMAGE_ID }}
+          ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }}
           security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }}
           subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }}
 
@@ -147,6 +153,7 @@ jobs:
           aws-region: ${{ vars.EC2_RUNNER_REGION }}
           github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
           image-id: ${{ vars.EC2_RUNNER_LINUX_ARM64_IMAGE_ID }}
+          ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }}
           security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }}
           subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }}
 
@@ -185,8 +192,8 @@ jobs:
           large-page-size: ${{ inputs.large-page-size }}
           aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }}
           aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
-          aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 
   build-linux-arm64-artifacts:
     name: Build linux-arm64 artifacts
@@ -223,8 +230,8 @@ jobs:
           large-page-size: ${{ inputs.large-page-size }}
           aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }}
           aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
-          aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 
   release-images-to-dockerhub:
     name: Build and push images to DockerHub

.github/workflows/docs.yml

@@ -74,5 +74,10 @@ jobs:
           - coverage
           - Sqlness Test (Flat format)
           - Compatibility Test (recent releases)
+          - GitHub Script Tests (Rust CI)
+          - GitHub Script Tests (Integration CI)
+          - Check for conflict
+          - Check Unused Dependencies
+          - Build GreptimeDB binaries (ubuntu-latest)
     steps:
       - run: 'echo "No action required"'

.github/workflows/integration.yml

@@ -30,7 +30,7 @@ on:
 name: Integration CI
 
 env:
-  CARGO_FUZZ_VERSION: "0.13.2"
+  FUZZ_VERSION: "0.13.2"
   # Fuzz targets require a nightly toolchain (cargo-fuzz needs
   # `-Zsanitizer=fuzzer`). This pin decouples the fuzz workflow from the
   # workspace default in rust-toolchain.toml, so it keeps working when the
@@ -121,6 +121,8 @@ jobs:
         with:
           tool: protoc@3.23.4
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Build Binaries Cache
         id: build-binaries-cache
         uses: Swatinem/rust-cache@v2
@@ -292,6 +294,8 @@ jobs:
         with:
           tool: protoc@3.23.4
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
         with:
@@ -436,18 +440,18 @@ jobs:
         id: fuzz-dependencies-cache
         uses: Swatinem/rust-cache@v2
         with:
-          prefix-key: "v1-fuzz-${{ env.CARGO_FUZZ_VERSION }}"
+          prefix-key: "v1-fuzz-${{ env.FUZZ_VERSION }}"
           shared-key: "fuzz-dependencies"
           workspaces: ". -> target"
           cache-targets: "true"
           cache-all-crates: "true"
-          save-if: ${{ github.event_name == 'pull_request' }}
+          save-if: ${{ github.ref == 'refs/heads/main' }}
       - name: Look up prebuilt fuzz binaries
         id: fuzz-binaries
         uses: actions/cache/restore@v4
         with:
           path: ${{ runner.temp }}/greptime-fuzz-binaries
-          key: v1-fuzz-binaries-${{ env.CARGO_FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }}
+          key: v1-fuzz-binaries-${{ env.FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }}
       - name: Report fuzz cache lookup
         shell: bash
         run: |
@@ -462,7 +466,7 @@ jobs:
         env:
           RUSTUP_TOOLCHAIN: ${{ env.FUZZ_RUST_TOOLCHAIN }}
         run: |
-          cargo install cargo-fuzz --version "${CARGO_FUZZ_VERSION}" --locked
+          cargo install cargo-fuzz --version "${FUZZ_VERSION}" --locked
       - name: Build fuzz target binaries
         if: steps.fuzz-binaries.outputs.cache-hit != 'true'
         shell: bash
@@ -511,11 +515,11 @@ jobs:
           build_targets false "${FUZZ_TARGETS}"
           build_targets true "${FUZZ_UNSTABLE_TARGETS}"
       - name: Save prebuilt fuzz binaries
-        if: steps.fuzz-binaries.outputs.cache-hit != 'true'
+        if: github.ref == 'refs/heads/main' && steps.fuzz-binaries.outputs.cache-hit != 'true'
         uses: actions/cache/save@v4
         with:
           path: ${{ runner.temp }}/greptime-fuzz-binaries
-          key: v1-fuzz-binaries-${{ env.CARGO_FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }}
+          key: v1-fuzz-binaries-${{ env.FUZZ_VERSION }}-${{ env.FUZZ_RUST_TOOLCHAIN }}-all-${{ hashFiles('Cargo.toml', 'Cargo.lock', '.cargo/**', 'tests-fuzz/**', 'src/**') }}
       - name: Pack prebuilt fuzz binaries
         shell: bash
         run: |
@@ -688,6 +692,8 @@ jobs:
         with:
           tool: protoc@3.23.4
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Build Profile-CI Cache
         id: build-greptime-ci-cache
         uses: Swatinem/rust-cache@v2
@@ -696,7 +702,7 @@ jobs:
           shared-key: "build-greptime-ci"
           cache-all-crates: "true"
           cache-workspace-crates: "true"
-          save-if: ${{ github.event_name == 'pull_request' }}
+          save-if: ${{ github.ref == 'refs/heads/main' }}
       - name: Report profile-CI cache lookup
         shell: bash
         run: |
@@ -819,7 +825,7 @@ jobs:
           workspaces: ". -> target"
           cache-targets: "true"
           cache-all-crates: "true"
-          save-if: ${{ github.event_name == 'pull_request' }}
+          save-if: ${{ github.ref == 'refs/heads/main' }}
       - if: matrix.mode.kafkaWalHelper
         name: Report Kafka WAL helper cache lookup
         shell: bash

.github/workflows/metric-snapshot-s3-bench.yml

@@ -18,6 +18,11 @@ on:
         description: 'Setup, 500-table smoke and full run timeout, 30..240 minutes'
         type: number
         default: 240
+      ec2_hourly_usd:
+        description: 'Positive EC2 hourly USD budget estimate; historical default, not live regional pricing'
+        required: true
+        type: string
+        default: '0.9408'
       prior_spend_usd:
         description: 'Cumulative estimated cloud spend for all earlier attempts in this $20 campaign'
         required: true
@@ -34,6 +39,7 @@ env:
   LOGICAL_TABLES_PER_GROUP: ${{ inputs.logical_tables_per_group }}
   BENCHMARK_TIMEOUT_MINUTES: ${{ inputs.benchmark_timeout_minutes }}
   PRIOR_SPEND_USD: ${{ inputs.prior_spend_usd }}
+  EC2_HOURLY_USD: ${{ inputs.ec2_hourly_usd }}
   RUN_PREFIX: metric-snapshot-bench/${{ github.run_id }}-${{ github.run_attempt }}/
   S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }}
   S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
@@ -66,8 +72,8 @@ jobs:
     - uses: astral-sh/setup-uv@v6
     - name: Check S3 access before allocating EC2
       env:
-        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
       run: uv run greptimedb/.github/scripts/metric-snapshot-s3-ops.py probe
     - name: Resolve the fixed benchmark runner
       id: runner
@@ -125,7 +131,10 @@ jobs:
         path: greptimedb
         persist-credentials: false
     - name: Set up benchmark dependencies
-      uses: ./greptimedb/.github/actions/setup-benchmark
+      uses: ./greptimedb/.github/actions/setup-linux-runner
+      with:
+        tools: curl jq python3 rsync
+        expected-architecture: x86_64
     - uses: astral-sh/setup-uv@v6
     - name: Mount the temporary local File data disk
       run: |
@@ -148,20 +157,20 @@ jobs:
         findmnt -no SOURCE,FSTYPE -T "$DISK_ROOT"
     - name: Probe the real same-region S3 bucket
       env:
-        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
       run: uv run greptimedb/.github/scripts/metric-snapshot-s3-ops.py probe
     - name: Run the 500-table smoke, then one full packed export and restore
       env:
-        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
       run: python3 greptimedb/.github/scripts/metric-snapshot-s3-bench.py run
     - name: Check results for credentials
       id: scrub
       if: always()
       env:
-        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
       run: python3 greptimedb/.github/scripts/metric-snapshot-s3-ops.py scrub
     - name: Publish the readable result
       if: ${{ always() && steps.scrub.outcome == 'success' }}
@@ -220,8 +229,8 @@ jobs:
     - name: Remove and verify only this run's S3 prefix
       if: always()
       env:
-        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
       run: uv run greptimedb/.github/scripts/metric-snapshot-s3-ops.py cleanup
     - name: Estimate the cumulative cloud cost
       if: ${{ always() && needs.provision.outputs.started_at != '' }}

.github/workflows/nightly-build.yml

@@ -9,6 +9,11 @@ on:
     - cron: '0 0 * * 1-5'
   workflow_dispatch: # Allows you to run this workflow manually.
     inputs:
+      ec2_volume_size:
+        description: Root volume size in GiB for each EC2 build runner
+        type: number
+        required: false
+        default: 500
       linux_amd64_runner:
         type: choice
         description: The runner uses to build linux-amd64 artifacts
@@ -113,6 +118,7 @@ jobs:
           aws-region: ${{ vars.EC2_RUNNER_REGION }}
           github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
           image-id: ${{ vars.EC2_RUNNER_LINUX_AMD64_IMAGE_ID }}
+          ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }}
           security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }}
           subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }}
 
@@ -127,6 +133,7 @@ jobs:
           aws-region: ${{ vars.EC2_RUNNER_REGION }}
           github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
           image-id: ${{ vars.EC2_RUNNER_LINUX_ARM64_IMAGE_ID }}
+          ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }}
           security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }}
           subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }}
 
@@ -153,8 +160,8 @@ jobs:
           image-namespace: ${{ vars.ECR_IMAGE_NAMESPACE }}
           aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }}
           aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
-          aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 
   build-linux-arm64-artifacts:
     name: Build linux-arm64 artifacts
@@ -179,8 +186,8 @@ jobs:
           image-namespace: ${{ vars.ECR_IMAGE_NAMESPACE }}
           aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }}
           aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
-          aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 
   release-images-to-dockerhub:
     name: Build and push images to DockerHub

.github/workflows/nightly-ci.yml

@@ -30,8 +30,8 @@ jobs:
           data-root: sqlness-test
           aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }}
           aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
-          aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
       - name: Upload sqlness logs
         if: failure()
         uses: actions/upload-artifact@v4
@@ -57,8 +57,12 @@ jobs:
         with:
           tool: protoc@3.23.4
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
       - name: Run sqlness
         run: make sqlness-test
         env:
@@ -109,8 +113,8 @@ jobs:
           RUST_MIN_STACK: 8388608 # 8MB
           CARGO_INCREMENTAL: 0
           GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }}
-          GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          GT_S3_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
           GT_ETCD_TLS_ENDPOINTS: https://127.0.0.1:2378
           GT_ETCD_ENDPOINTS: http://127.0.0.1:2379
@@ -143,9 +147,12 @@ jobs:
       - name: Install Rust toolchain
         uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
+          cache: false
           components: llvm-tools-preview
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
       - name: Install Cargo Nextest
         uses: taiki-e/install-action@nextest
       - name: Install WSL distribution
@@ -160,8 +167,8 @@ jobs:
           RUST_MIN_STACK: 8388608 # 8 MiB, aligned with Rust CI and release tests.
           CARGO_INCREMENTAL: 0
           GT_S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }}
-          GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          GT_S3_ACCESS_KEY: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          GT_S3_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          GT_S3_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           GT_S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
           UNITTEST_LOG_DIR: "__unittest_logs"
 

.github/workflows/nightly-jsonbench.yaml

@@ -112,6 +112,8 @@ jobs:
         id: start-linux-runner
         with:
           runner: ${{ vars.LARGE_IOPS_RUNNER }}
+          # Database data, WAL, and compaction files use the root volume.
+          ec2-volume-size: "500"
           aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           aws-region: ${{ vars.EC2_RUNNER_REGION }}

.github/workflows/nightly-merge.yml

@@ -0,0 +1,153 @@
+# Setup and behavior: .github/nightly-merge.md
+name: Nightly Merge
+
+on:
+  pull_request_target:
+    types: [opened, edited, reopened]
+  schedule:
+    # Hourly from 02:10 through 08:10 Asia/Shanghai (UTC+8), inclusive.
+    # GitHub may delay scheduled runs.
+    - cron: '10 0,18-23 * * *'
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: List eligible PRs without adding them to the merge queue
+        type: boolean
+        default: true
+
+permissions: {}
+
+jobs:
+  sync-label:
+    if: github.repository == 'GreptimeTeam/greptimedb' && github.event_name == 'pull_request_target'
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      # PR label mutations require pull-request write permission.
+      pull-requests: write
+      issues: write
+    concurrency:
+      group: nightly-merge-label-${{ github.event.pull_request.number }}
+      cancel-in-progress: false
+    steps:
+      # No checkout: PR contents are untrusted data, never executable code.
+      - uses: actions/github-script@v7
+        with:
+          script: |
+            const { owner, repo } = context.repo;
+            const issue_number = context.payload.pull_request.number;
+            const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: issue_number });
+            const checkbox = /^- \[([ xX])\] Merge during the nightly window <!-- nightly-merge -->\s*$/m.exec(pr.body || '');
+            // PRs without the template manage their opt-in through the label.
+            if (!checkbox) return;
+            const selected = checkbox[1].toLowerCase() === 'x';
+            const label = 'merge nightly';
+            const labeled = pr.labels.some(item => item.name === label);
+            if (selected && !labeled) {
+              await github.rest.issues.addLabels({ owner, repo, issue_number, labels: [label] });
+            } else if (!selected && labeled) {
+              await github.rest.issues.removeLabel({ owner, repo, issue_number, name: label });
+            }
+
+  enqueue:
+    if: github.repository == 'GreptimeTeam/greptimedb' && github.event_name != 'pull_request_target'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    concurrency:
+      group: nightly-merge-enqueue
+      cancel-in-progress: false
+    steps:
+      # Reuses the automation credential used by backport and compat updates.
+      # A PAT (or App token) lets subsequent merge_group workflows run.
+      - uses: actions/github-script@v7
+        env:
+          DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
+        with:
+          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
+          script: |
+            const { owner, repo } = context.repo;
+            const dryRun = process.env.DRY_RUN === 'true';
+            const candidates = await github.paginate(github.rest.issues.listForRepo, {
+              owner, repo, state: 'open', labels: 'merge nightly',
+              sort: 'created', direction: 'asc', per_page: 100
+            });
+            const rows = [];
+            let enqueued = 0;
+            for (const candidate of candidates) {
+              if (!candidate.pull_request || enqueued >= 5) continue;
+              const number = candidate.number;
+              try {
+                // Fetch current labels and PR state, not the earlier list snapshot.
+                const { data: current } = await github.rest.pulls.get({ owner, repo, pull_number: number });
+                if (current.state !== 'open' || current.draft || current.base.ref !== 'main' ||
+                    !current.labels.some(label => label.name === 'merge nightly')) {
+                  rows.push([String(number), 'Skipped: not opted in, draft, closed, or not targeting main']);
+                  continue;
+                }
+                const { repository: { pullRequest: pr } } = await github.graphql(`
+                  query($owner: String!, $repo: String!, $number: Int!) {
+                    repository(owner: $owner, name: $repo) {
+                      pullRequest(number: $number) {
+                        id headRefOid reviewDecision mergeStateStatus
+                        mergeQueueEntry { id }
+                        autoMergeRequest { enabledAt }
+                      }
+                    }
+                  }`, { owner, repo, number });
+                if (pr.mergeQueueEntry || pr.autoMergeRequest ||
+                    pr.reviewDecision !== 'APPROVED' || pr.mergeStateStatus !== 'CLEAN') {
+                  rows.push([String(number), 'Skipped: already queued/auto-merge enabled, or not approved and clean']);
+                  continue;
+                }
+                let cursor = null;
+                let unresolved = false;
+                do {
+                  const { repository: { pullRequest: { reviewThreads } } } = await github.graphql(`
+                    query($owner: String!, $repo: String!, $number: Int!, $cursor: String) {
+                      repository(owner: $owner, name: $repo) {
+                        pullRequest(number: $number) {
+                          reviewThreads(first: 100, after: $cursor) {
+                            nodes { isResolved }
+                            pageInfo { hasNextPage endCursor }
+                          }
+                        }
+                      }
+                    }`, { owner, repo, number, cursor });
+                  unresolved = reviewThreads.nodes.some(thread => !thread.isResolved);
+                  cursor = reviewThreads.pageInfo.hasNextPage ? reviewThreads.pageInfo.endCursor : null;
+                } while (cursor && !unresolved);
+                if (unresolved) {
+                  rows.push([String(number), 'Skipped: unresolved review threads']);
+                  continue;
+                }
+                if (!dryRun) {
+                  // GitHub enforces branch rules; never bypass or enable auto-merge.
+                  try {
+                    await github.graphql(`
+                    mutation($id: ID!, $sha: GitObjectID!) {
+                      enqueuePullRequest(input: {pullRequestId: $id, expectedHeadOid: $sha}) {
+                        mergeQueueEntry { id }
+                      }
+                    }`, { id: pr.id, sha: pr.headRefOid });
+                  } catch (error) {
+                    // Stacked PRs require a different API and are not handled by this workflow.
+                    const stackError = 'This pull request is part of a stack and must be enqueued using the asynchronous merge REST API.';
+                    if (error.errors?.length && error.errors.every(item => item.message?.startsWith(stackError))) {
+                      core.warning(`PR #${number}: skipped stacked PR`);
+                      rows.push([String(number), 'Skipped: stacked PR']);
+                      continue;
+                    }
+                    throw error;
+                  }
+                }
+                enqueued++;
+                rows.push([String(number), dryRun ? 'Would enqueue' : 'Enqueued']);
+              } catch (error) {
+                core.warning(`PR #${number}: ${error.message}`);
+                rows.push([String(number), 'Failed: see workflow logs']);
+                core.setFailed('Some PRs could not be processed; see workflow logs.');
+              }
+            }
+            for (const [number, result] of rows) core.info(`PR #${number}: ${result}`);
+            await core.summary.addHeading(dryRun ? 'Nightly merge (dry run)' : 'Nightly merge')
+              .addTable([['PR', 'Result'], ...rows]).write();

.github/workflows/oswal-s3-bench.yml

@@ -117,6 +117,18 @@ jobs:
           --timeout-minutes "$BENCHMARK_TIMEOUT_MINUTES" --ttl-hours "$JANITOR_TTL_HOURS"
         python3 greptimedb/.github/scripts/oswal-s3-bench.py image --tag "$GREPTIMEDB_TAG" --image "$GREPTIMEDB_IMAGE_INPUT" \
           --images "$OSWAL_IMAGES"
+    - name: Configure AWS credentials for ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/configure-aws-credentials@v4
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+    - name: Login to Amazon ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/amazon-ecr-login@v2
+      with:
+        registry-type: public
     - name: Pull every GreptimeDB image and record its version
       id: images
       env:
@@ -193,13 +205,28 @@ jobs:
         path: greptimedb
         persist-credentials: false
     - name: Set up benchmark dependencies
-      uses: ./greptimedb/.github/actions/setup-benchmark
+      uses: ./greptimedb/.github/actions/setup-linux-runner
+      with:
+        tools: curl jq python3 rsync
+        expected-architecture: x86_64
     - uses: astral-sh/setup-uv@v6
     - name: Start the MinIO fixture
       if: inputs.smoke
       run: |
         bash greptimedb/.github/scripts/pull-benchmark-image.sh greptime/minio:2024
         docker compose -f greptimedb/tests-integration/fixtures/docker-compose.yml up -d --wait minio
+    - name: Configure AWS credentials for ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/configure-aws-credentials@v4
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+    - name: Login to Amazon ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/amazon-ecr-login@v2
+      with:
+        registry-type: public
     - name: Pull and check images, write the plan
       run: |
         bash greptimedb/.github/scripts/oswal-s3-bench-run.sh pull
@@ -211,8 +238,8 @@ jobs:
           --runtime-image "$RUNTIME_IMAGE" --instance-type "$INSTANCE_TYPE" --region "$S3_REGION" --prefix "$RUN_PREFIX"
     - name: Probe the bucket
       env:
-        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }}
       run: uv run greptimedb/.github/scripts/oswal-s3-bench.py probe --bucket "$S3_BUCKET" --region "$S3_REGION" --endpoint "$S3_ENDPOINT" --prefix "$RUN_PREFIX"
     - name: Put the Raft Engine WAL on the instance store
       run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh wal-disk "$WAL_ROOT"
@@ -220,8 +247,8 @@ jobs:
       run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh generate
     - name: Load every cell
       env:
-        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }}
       run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh cells
     - name: Summary
       if: always()
@@ -229,14 +256,14 @@ jobs:
     - name: Delete the run's S3 prefix
       if: always()
       env:
-        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }}
       run: uv run greptimedb/.github/scripts/oswal-s3-bench.py cleanup --bucket "$S3_BUCKET" --region "$S3_REGION" --endpoint "$S3_ENDPOINT" --prefix "$RUN_PREFIX"
     - name: Check the results hold no credentials
       if: always()
       env:
-        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+        AWS_ACCESS_KEY_ID: ${{ inputs.smoke && 'superpower_ci_user' || secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.smoke && 'superpower_password' || secrets.AWS_SECRET_ACCESS_KEY }}
       run: bash greptimedb/.github/scripts/oswal-s3-bench-run.sh scrub
     - name: Upload results (no corpus)
       if: always()

.github/workflows/publish-query-regression-helpers.yml

@@ -0,0 +1,166 @@
+name: Publish query regression helpers
+
+on:
+  workflow_dispatch:
+  workflow_call:
+  push:
+    branches: [main]
+    paths:
+      - src/cmd/src/bin/query_perf_fixture.rs
+      - src/cmd/src/bin/query_perf_fixture/**
+      - src/cmd/src/bin/query_regression_runner.rs
+      - src/cmd/src/bin/query_regression_runner/**
+      - docker/query-regression-helpers/**
+      - .github/workflows/publish-query-regression-helpers.yml
+      - .github/scripts/publish-query-regression-helpers.sh
+      - .github/scripts/build-query-regression-helpers.sh
+      - .github/scripts/update-query-regression-helpers-image.sh
+      - Cargo.lock
+
+permissions:
+  contents: read
+
+# Serialize publication and updates of the repository image variable.
+concurrency:
+  group: publish-query-regression-helpers
+  cancel-in-progress: false
+
+jobs:
+  provision:
+    if: ${{ github.repository == 'GreptimeTeam/greptimedb' }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    outputs:
+      label: ${{ steps.provision.outputs.label }}
+      instance_id: ${{ steps.provision.outputs.instance_id }}
+      runner_name: ${{ steps.provision.outputs.runner_name }}
+    steps:
+      - name: Checkout trusted provisioning scripts
+        uses: actions/checkout@v4
+        with:
+          repository: ${{ github.repository }}
+          ref: ${{ github.sha }}
+          persist-credentials: false
+
+      - name: Provision ECS runner
+        id: provision
+        uses: ./.github/actions/aliyun-ecs-create
+        with:
+          access-key-id: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
+          access-key-secret: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
+          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
+          region-id: ${{ vars.ALIYUN_ECS_REGION_ID }}
+          vswitch-id: ${{ vars.ALIYUN_ECS_VSWITCH_ID }}
+          security-group-id: ${{ vars.ALIYUN_ECS_SECURITY_GROUP_ID }}
+          instance-type: ${{ vars.ALIYUN_ECS_INSTANCE_TYPE }}
+          # Preserve query regression's original resource and janitor budgets.
+          system-disk-gib: '80'
+          ttl-hours: '4'
+          enable-docker: 'true'
+          resource-group-id: ${{ vars.ALIYUN_ECS_RESOURCE_GROUP_ID }}
+          image-id: ${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }}
+          runner-uid: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }}
+          runner-gid: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }}
+
+  build:
+    needs: provision
+    runs-on: ${{ needs.provision.outputs.label }}
+    timeout-minutes: 150
+    env:
+      CARGO_HOME: /home/runner/.cargo
+      CARGO_TARGET_DIR: /home/runner/query-regression-target
+      RUSTUP_HOME: /opt/rustup
+      RUSTUP_TOOLCHAIN: nightly-2026-03-21
+      RUSTUP_AUTO_INSTALL: "0"
+      CARGO_INCREMENTAL: "0"
+      RUSTFLAGS: -D warnings -C link-arg=-fuse-ld=mold
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - name: Build and validate helpers
+        run: bash .github/scripts/build-query-regression-helpers.sh
+      - name: Upload reusable helper image
+        uses: actions/upload-artifact@v4
+        with:
+          name: helpers-image
+          path: helpers-image/
+          compression-level: 1
+          retention-days: 7
+          if-no-files-found: error
+      - name: Upload build validation evidence
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: helpers-build-validation
+          path: helpers-validation/
+          if-no-files-found: warn
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    outputs:
+      image: ${{ steps.publish.outputs.image }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: actions/download-artifact@v4
+        with:
+          name: helpers-image
+          path: helpers-image/
+      - name: Login to registry
+        uses: docker/login-action@v3
+        with:
+          registry: greptime-registry.cn-hangzhou.cr.aliyuncs.com
+          username: ${{ secrets.ALICLOUD_USERNAME }}
+          password: ${{ secrets.ALICLOUD_PASSWORD }}
+      - name: Publish and verify helpers
+        id: publish
+        run: bash .github/scripts/publish-query-regression-helpers.sh
+      - name: Upload publication validation evidence
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: helpers-publish-validation
+          path: |
+            helpers-validation/digest.txt
+            helpers-validation/extracted/helpers-image.json
+          if-no-files-found: warn
+
+  update-variable:
+    needs: publish
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - name: Update helpers image repository variable
+        env:
+          GH_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
+          HELPERS_IMAGE: ${{ needs.publish.outputs.image }}
+        run: bash .github/scripts/update-query-regression-helpers-image.sh
+
+  teardown:
+    if: ${{ always() && needs.provision.outputs.instance_id != '' }}
+    needs: [provision, build]
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Checkout trusted teardown scripts
+        uses: actions/checkout@v4
+        with:
+          repository: ${{ github.repository }}
+          ref: ${{ github.sha }}
+          persist-credentials: false
+
+      - name: Teardown ECS runner
+        uses: ./.github/actions/aliyun-ecs-delete
+        with:
+          access-key-id: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
+          access-key-secret: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
+          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
+          region-id: ${{ vars.ALIYUN_ECS_REGION_ID }}
+          instance-id: ${{ needs.provision.outputs.instance_id }}
+          runner-name: ${{ needs.provision.outputs.runner_name }}

.github/workflows/query-regression.yml

@@ -26,6 +26,21 @@ on:
         description: Candidate ref/sha to build; reusable release calls must provide an immutable full SHA
         required: true
         type: string
+      base_image:
+        description: Base Docker image tag/digest (empty builds base_ref from source)
+        required: false
+        type: string
+        default: ""
+      candidate_image:
+        description: Candidate Docker image tag/digest (empty builds candidate_ref from source)
+        required: false
+        type: string
+        default: ""
+      helpers_image:
+        description: Helpers image@sha256 or pinned from repository variable (empty builds from source)
+        required: false
+        type: string
+        default: ""
       cargo_profile:
         description: Cargo profile to build base and candidate binaries with
         required: false
@@ -80,6 +95,21 @@ on:
         description: Runner HTTP timeout seconds
         required: true
         default: "300"
+      base_image:
+        description: Base Docker image tag/digest (empty builds base_ref from source)
+        required: false
+        type: string
+        default: ""
+      candidate_image:
+        description: Candidate Docker image tag/digest (empty builds candidate_ref from source)
+        required: false
+        type: string
+        default: ""
+      helpers_image:
+        description: Helpers image@sha256 or pinned from repository variable (empty builds from source)
+        required: false
+        type: string
+        default: ""
       cargo_profile:
         description: Cargo profile to build base and candidate binaries with
         required: true
@@ -132,6 +162,7 @@ jobs:
           python3 tests/perf/test_query_regression_runner_otlp_trace_load.py
           python3 tests/perf/test_query_regression_summary_otlp.py
           python3 tests/perf/test_query_regression_case_selection.py
+          python3 tests/perf/test_query_regression_helpers.py
           python3 tests/perf/test_query_regression_nightly_refs.py
           python3 tests/perf/test_query_regression_slash.py
           python3 tests/perf/test_aliyun_ecs_runner_scripts.py
@@ -171,6 +202,7 @@ jobs:
           # Preserve query regression's original resource and janitor budgets.
           system-disk-gib: '80'
           ttl-hours: '4'
+          enable-docker: ${{ inputs.base_image != '' || inputs.candidate_image != '' || inputs.helpers_image != '' }}
           resource-group-id: ${{ vars.ALIYUN_ECS_RESOURCE_GROUP_ID }}
           image-id: ${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }}
           runner-uid: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }}
@@ -214,6 +246,10 @@ jobs:
       PR_NUMBER: ${{ inputs.pr_number }}
       HEAD_REPO: ${{ inputs.head_repo }}
       BASE_REPO: ${{ inputs.base_repo }}
+      HELPERS_IMAGE: ${{ inputs.helpers_image }}
+      QUERY_REGRESSION_HELPERS_IMAGE: ${{ vars.QUERY_REGRESSION_HELPERS_IMAGE }}
+      BASE_IMAGE: ${{ inputs.base_image }}
+      CANDIDATE_IMAGE: ${{ inputs.candidate_image }}
       BASE_REF: ${{ inputs.base_ref }}
       CANDIDATE_REF: ${{ inputs.candidate_ref || github.ref }}
       EVENT_MERGE_SHA: ${{ inputs.pr_number != '' && inputs.candidate_ref || '' }}
@@ -224,6 +260,16 @@ jobs:
       EXPECTED_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }}
       EXPECTED_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }}
     steps:
+      - name: Validate image target admission
+        run: |
+          if [[ "${PR_ADMISSION}" == "true" && ( -n "${BASE_IMAGE}" || -n "${CANDIDATE_IMAGE}" || -n "${HELPERS_IMAGE}" ) ]]; then
+            echo "PR admission requires source targets for verified commit attribution." >&2
+            exit 1
+          fi
+          if [[ -n "${BASE_IMAGE}" || -n "${CANDIDATE_IMAGE}" || -n "${HELPERS_IMAGE}" ]]; then
+            docker info >/dev/null
+          fi
+
       - name: Report provisioned ECS runner
         if: ${{ needs.provision.outputs.instance_id != '' }}
         shell: bash
@@ -418,6 +464,7 @@ jobs:
             > "${GITHUB_WORKSPACE}/query-regression-trusted-scripts/query-regression-summary.py"
 
       - name: Verify runner image tools
+        if: ${{ inputs.base_image == '' || inputs.candidate_image == '' || inputs.helpers_image == '' }}
         shell: bash
         run: |
           set -euo pipefail
@@ -534,6 +581,7 @@ jobs:
           fi
 
       - name: Prepare query regression cache
+        if: ${{ inputs.base_image == '' || inputs.candidate_image == '' || inputs.helpers_image == '' }}
         shell: bash
         working-directory: src
         run: |
@@ -549,7 +597,7 @@ jobs:
           readonly EXPECTED_SCCACHE_DIR="/home/runner/.cache/sccache"
           readonly EXPECTED_RUSTC_WRAPPER="/usr/local/bin/sccache"
           readonly RUNNER_IMAGE_DIGEST="sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11"
-          readonly RUNNER_IMAGE_EPOCH="7"
+          readonly RUNNER_IMAGE_EPOCH="8"
 
           require_expected_root() {
             local name="$1"
@@ -694,6 +742,7 @@ jobs:
           sccache --show-stats
 
       - name: Build base greptime
+        if: ${{ inputs.base_image == '' }}
         working-directory: src
         run: |
           set -euo pipefail
@@ -728,14 +777,27 @@ jobs:
           git reset --hard "${VERIFIED_CANDIDATE_SHA}"
           git clean -ffdx
 
+      - name: Extract precompiled query regression helpers
+        if: ${{ inputs.helpers_image != '' }}
+        working-directory: src
+        run: |
+          python3 .github/scripts/query-regression-helpers.py "$HELPERS_IMAGE" \
+            "$GITHUB_WORKSPACE/query-regression-bins/candidate"
+          cat "$GITHUB_WORKSPACE/query-regression-bins/candidate/helpers-image.json" >> "$GITHUB_STEP_SUMMARY"
+
       - name: Build candidate greptime and query regression helpers
+        if: ${{ inputs.candidate_image == '' || inputs.helpers_image == '' }}
         working-directory: src
         run: |
           set -euo pipefail
           SECONDS=0
-          cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime
-          cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools
-          cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_regression_runner --features dev-tools
+          if [[ -z "${CANDIDATE_IMAGE}" ]]; then
+            cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime
+          fi
+          if [[ -z "${HELPERS_IMAGE}" ]]; then
+            cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools
+            cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_regression_runner --features dev-tools
+          fi
           candidate_build_elapsed="${SECONDS}"
           printf 'Candidate greptime and query regression helper cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}"
           printf -- '- Candidate greptime and query regression helper cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}"
@@ -745,12 +807,16 @@ jobs:
             target_dir="debug"
           fi
           mkdir -p "${GITHUB_WORKSPACE}/query-regression-bins/candidate"
-          cp "${CARGO_TARGET_DIR}/${target_dir}/greptime" \
-            "${GITHUB_WORKSPACE}/query-regression-bins/candidate/greptime"
-          cp "${CARGO_TARGET_DIR}/${target_dir}/query_perf_fixture" \
-            "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_perf_fixture"
-          cp "${CARGO_TARGET_DIR}/${target_dir}/query_regression_runner" \
-            "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_regression_runner"
+          if [[ -z "${CANDIDATE_IMAGE}" ]]; then
+            cp "${CARGO_TARGET_DIR}/${target_dir}/greptime" \
+              "${GITHUB_WORKSPACE}/query-regression-bins/candidate/greptime"
+          fi
+          if [[ -z "${HELPERS_IMAGE}" ]]; then
+            cp "${CARGO_TARGET_DIR}/${target_dir}/query_perf_fixture" \
+              "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_perf_fixture"
+            cp "${CARGO_TARGET_DIR}/${target_dir}/query_regression_runner" \
+              "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_regression_runner"
+          fi
 
       - name: Run query regression
         id: run
@@ -792,6 +858,7 @@ jobs:
             query-regression-work/**/logs/**
             query-regression-work/**/otelgen/**
             query-regression-summary.md
+            query-regression-bins/candidate/helpers-image.json
             machine-telemetry.log
           if-no-files-found: warn
           retention-days: 7

.github/workflows/release-dev-builder-images.yaml

@@ -35,6 +35,11 @@ on:
         description: Rebuild the query-regression ECS runner image (and update QUERY_REGRESSION_ECS_IMAGE_ID + RUNNER_IMAGE_EPOCH)
         required: false
         default: false
+      release_query_regression_helpers_image:
+        type: boolean
+        description: Publish precompiled query regression helpers and update the image repository variable
+        required: false
+        default: false
       update_dev_builder_image_tag:
         type: boolean
         description: Update the DEV_BUILDER_IMAGE_TAG in Makefile and create a PR
@@ -451,3 +456,8 @@ jobs:
           echo "## Lockstep updates" >> "$GITHUB_STEP_SUMMARY"
           echo "- QUERY_REGRESSION_ECS_IMAGE_ID → \`${image_id}\`" >> "$GITHUB_STEP_SUMMARY"
           echo "- RUNNER_IMAGE_EPOCH → \`${new_epoch}\` (PR on \`${BRANCH}\`, pending review/merge)" >> "$GITHUB_STEP_SUMMARY"
+
+  release-query-regression-helpers:
+    if: ${{ github.repository == 'GreptimeTeam/greptimedb' && inputs.release_query_regression_helpers_image }}
+    uses: ./.github/workflows/publish-query-regression-helpers.yml
+    secrets: inherit

.github/workflows/release.yml

@@ -20,6 +20,11 @@ on:
     - cron: '0 0 * * 1-5'
   workflow_dispatch: # Allows you to run this workflow manually.
     inputs:
+      ec2_volume_size:
+        description: Root volume size in GiB for each EC2 build runner
+        type: number
+        required: false
+        default: 500
       linux_amd64_runner:
         type: choice
         description: The runner uses to build linux-amd64 artifacts
@@ -215,6 +220,7 @@ jobs:
           aws-region: ${{ vars.EC2_RUNNER_REGION }}
           github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
           image-id: ${{ vars.EC2_RUNNER_LINUX_AMD64_IMAGE_ID }}
+          ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }}
           security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }}
           subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }}
 
@@ -229,6 +235,7 @@ jobs:
           aws-region: ${{ vars.EC2_RUNNER_REGION }}
           github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
           image-id: ${{ vars.EC2_RUNNER_LINUX_ARM64_IMAGE_ID }}
+          ec2-volume-size: ${{ inputs.ec2_volume_size || 500 }}
           security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }}
           subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }}
 
@@ -255,8 +262,8 @@ jobs:
           image-namespace: ${{ vars.ECR_IMAGE_NAMESPACE }}
           aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }}
           aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
-          aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 
   build-linux-arm64-artifacts:
     name: Build linux-arm64 artifacts
@@ -282,8 +289,8 @@ jobs:
           large-page-size: true
           aws-ci-test-bucket: ${{ vars.AWS_CI_TEST_BUCKET }}
           aws-region: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
-          aws-access-key-id: ${{ secrets.AWS_CI_TEST_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_CI_TEST_SECRET_ACCESS_KEY }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 
   build-linux-riscv64-artifacts:
     name: Build linux-riscv64 artifacts
@@ -298,6 +305,11 @@ jobs:
           fetch-depth: 0
           persist-credentials: false
 
+      - name: Set up Linux runner
+        uses: ./.github/actions/setup-linux-runner
+        with:
+          tools: make shasum
+
       - name: Prepare riscv64 dev-builder image
         shell: bash
         env:

.github/workflows/rust.yml

@@ -24,7 +24,7 @@ on:
 name: Rust CI
 
 env:
-  CARGO_SHEAR_VERSION: "1.13.4"
+  SHEAR_VERSION: "1.13.4"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
@@ -56,6 +56,7 @@ jobs:
           tool: protoc@3.23.4
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
+          cache: false
           components: rustfmt
       - name: Check format
         run: make fmt-check
@@ -75,6 +76,7 @@ jobs:
           tool: protoc@3.23.4
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
+          cache: false
           components: clippy
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
@@ -97,10 +99,12 @@ jobs:
         with:
           persist-credentials: false
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Install cargo-shear
         uses: taiki-e/install-action@v2
         with:
-          tool: cargo-shear@${{ env.CARGO_SHEAR_VERSION }}
+          tool: cargo-shear@${{ env.SHEAR_VERSION }}
           fallback: none
       - name: Check unused dependencies
         run: make check-udeps
@@ -122,6 +126,8 @@ jobs:
       - name: Install riscv64 cross toolchain
         run: sudo apt-get update && sudo apt-get install -y gcc-riscv64-linux-gnu g++-riscv64-linux-gnu
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Add riscv64 target
         run: rustup target add riscv64gc-unknown-linux-gnu
       - name: Rust Cache
@@ -153,6 +159,8 @@ jobs:
         with:
           version: "14.0"
       - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: false
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
         with:

.github/workflows/tracesbench.yml

@@ -2,6 +2,11 @@ name: Tracesbench
 on:
   workflow_dispatch:
     inputs:
+      greptimedb_storage:
+        description: 'GreptimeDB data storage (S3 requires AWS in the bucket region; WAL stays local)'
+        type: choice
+        options: [file, s3, both]
+        default: file
       run_greptimedb:
         description: Run GreptimeDB
         type: boolean
@@ -39,7 +44,7 @@ on:
         default: sanity
         required: true
       runtime_image:
-        description: 'Runtime image: auto = AWS ECR Public or Aliyun ACR by provider; explicit tag/digest overrides'
+        description: 'Runtime image: auto = pinned S3 test runtime on ACR; explicit tag/digest overrides'
         type: string
         default: auto
         required: true
@@ -107,6 +112,11 @@ on:
 permissions:
   contents: read
 env:
+  GREPTIMEDB_STORAGE: ${{ inputs.greptimedb_storage || 'file' }}
+  S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }}
+  S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
+  RUNNER_REGION: ${{ vars.EC2_RUNNER_REGION }}
+  RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }}
   COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }}
   BASELINE_GREPTIMEDB_TAG: ${{ inputs.baseline_greptimedb_tag }}
   DATASET: ${{ inputs.dataset }}
@@ -114,7 +124,7 @@ env:
   RUNTIME_CPUS: ${{ inputs.runtime_cpus }}
   RUNTIME_MEMORY: ${{ inputs.runtime_memory }}
   BENCHMARK_ROOT: ${{ inputs.benchmark_root }}
-  RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || inputs.provider == 'AWS' && 'public.ecr.aws/i8k6a5e1/greptime/o11ybenchruntime:77a84e1c8136@sha256:c482993ec177b026235d6cd0c61d4fb6d6f1088bc78bdc0e5964a5398de6dba0' || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:77a84e1c8136@sha256:c482993ec177b026235d6cd0c61d4fb6d6f1088bc78bdc0e5964a5398de6dba0' }}
+  RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:s3-test-b0cf5204d2fc@sha256:c2bf622db6056a2029842f4dc1cc1f37b82773bd45b134a0a83ffe2a8cd62214' }}
   PROVIDER: ${{ inputs.provider }}
   REQUESTED_INSTANCE_TYPE: ${{ inputs.instance_type }}
   SYSTEM_DISK_GIB: ${{ inputs.system_disk_gib }}
@@ -162,6 +172,10 @@ jobs:
           [[ "$GREPTIMEDB_TAG" =~ ^[a-zA-Z0-9_][a-zA-Z0-9_.-]{0,127}$ ]] || { echo 'Invalid GreptimeDB image tag' >&2; exit 1; }
         fi
         printf 'targets=%s\n' "$selected" >> "$GITHUB_OUTPUT"
+    - name: Validate GreptimeDB storage
+      run: |
+        python3 greptimedb/tests/perf/test_o11ybench_storage.py
+        python3 greptimedb/.github/scripts/o11ybench-storage.py validate
     - name: Validate version comparison
       env:
         RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }}
@@ -180,6 +194,18 @@ jobs:
           esac
         fi
         python3 greptimedb/.github/scripts/ci-runner-config.py --provider "$PROVIDER" --instance-type "$instance_type"
+    - name: Configure AWS credentials for ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/configure-aws-credentials@v4
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+    - name: Login to Amazon ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/amazon-ecr-login@v2
+      with:
+        registry-type: public
     - name: Validate budgets and shared runtime
       id: runtime
       run: |
@@ -305,7 +331,10 @@ jobs:
         path: greptimedb
         persist-credentials: false
     - name: Set up benchmark dependencies
-      uses: ./greptimedb/.github/actions/setup-benchmark
+      uses: ./greptimedb/.github/actions/setup-linux-runner
+      with:
+        tools: curl jq python3 rsync
+        expected-architecture: x86_64
     - name: Checkout matching benchmark tools
       uses: actions/checkout@v4
       with:
@@ -314,6 +343,18 @@ jobs:
         ref: ${{ env.RESOLVED_O11YBENCH_REF }}
         path: o11ybench
         persist-credentials: false
+    - name: Configure AWS credentials for ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/configure-aws-credentials@v4
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+    - name: Login to Amazon ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/amazon-ecr-login@v2
+      with:
+        registry-type: public
     - name: Check runner and runtime identity
       id: runtime
       run: |
@@ -362,22 +403,44 @@ jobs:
         python3 o11ybench/scripts/run_tracesbench_docker.py --stage generate \
           --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
           --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
-    - name: Baseline GreptimeDB load, correctness and benchmark
-      if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.compare_greptimedb }}
+    - name: Baseline GreptimeDB load, correctness and benchmark (File)
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' && inputs.compare_greptimedb }}
       run: |
-        python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --target greptimedb --port 14000 --greptimedb-image "$BASELINE_GREPTIMEDB_IMAGE" \
+        python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage file --target greptimedb --port 14000 --greptimedb-image "$BASELINE_GREPTIMEDB_IMAGE" \
           --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \
           --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
           --root "$RUN_ROOT/baseline" --data-root "$RUN_ROOT" --owner "$OWNER-base" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
         python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/baseline/runs/greptimedb"
-    - name: GreptimeDB load, correctness and benchmark
-      if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' }}
+    - name: Baseline GreptimeDB load, correctness and benchmark (S3)
+      env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' && inputs.compare_greptimedb }}
       run: |
-        python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --target greptimedb --port 14000 --greptimedb-image "$GREPTIMEDB_IMAGE" \
+        python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage s3 --target greptimedb --port 14000 --greptimedb-image "$BASELINE_GREPTIMEDB_IMAGE" \
+          --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \
+          --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
+          --root "$RUN_ROOT/baseline/s3" --data-root "$RUN_ROOT" --owner "$OWNER-base-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+        python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/baseline/s3/runs/greptimedb"
+    - name: GreptimeDB load, correctness and benchmark (File)
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 's3' }}
+      run: |
+        python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage file --target greptimedb --port 14000 --greptimedb-image "$GREPTIMEDB_IMAGE" \
           --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \
           --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
           --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
         python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/runs/greptimedb"
+    - name: GreptimeDB load, correctness and benchmark (S3)
+      env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.greptimedb == 'true' && inputs.greptimedb_storage != 'file' }}
+      run: |
+        python3 o11ybench/scripts/run_tracesbench_docker.py --stage target --greptimedb-storage s3 --target greptimedb --port 14000 --greptimedb-image "$GREPTIMEDB_IMAGE" \
+          --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY" --query-timeout-seconds "$QUERY_TIMEOUT_SECONDS" --load-workers "$LOAD_WORKERS" \
+          --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
+          --root "$RUN_ROOT/s3" --data-root "$RUN_ROOT" --owner "$OWNER-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+        python3 o11ybench/scripts/check_tracesbench_query_plans.py "$RUN_ROOT/s3/runs/greptimedb"
     - name: VictoriaTraces load, correctness and benchmark
       if: ${{ !cancelled() && steps.generate.outcome == 'success' && needs.validate.outputs.victoriatraces == 'true' }}
       run: |
@@ -395,49 +458,80 @@ jobs:
     - name: GreptimeDB version comparison
       if: always() && inputs.compare_greptimedb
       run: |
-        python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind traces --root "$RUN_ROOT" >> "$GITHUB_STEP_SUMMARY"
-    - name: Validate and render report
+        python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind traces --root "$RUN_ROOT" | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+    - name: Validate and render primary report
+      if: always() && steps.generate.outcome == 'success'
       env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
         COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }}
       run: |
+        if [[ "$GREPTIMEDB_STORAGE" == s3 ]]; then
+          TARGETS=$(tr "," "\n" <<< "$TARGETS" | sed "/^greptimedb$/d" | paste -sd, -)
+          [[ -n "$TARGETS" ]] || exit 0
+        fi
         comparison_args=()
-        if [[ "$COMPARE_GREPTIMEDB" == true ]]; then
+        if [[ "$COMPARE_GREPTIMEDB" == true && "$GREPTIMEDB_STORAGE" != s3 ]]; then
           comparison_args+=(--baseline-run "$RUN_ROOT/baseline/runs/greptimedb/run.json")
         fi
-        python3 o11ybench/scripts/run_tracesbench_docker.py --stage report --targets "$TARGETS" "${comparison_args[@]}" \
+        python3 o11ybench/scripts/run_tracesbench_docker.py --stage report --targets "${TARGETS}" "${comparison_args[@]}" \
           --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
           --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Validate and render S3 report
+      if: always() && steps.generate.outcome == 'success' && inputs.greptimedb_storage != 'file'
+      run: |
+        python3 o11ybench/scripts/run_tracesbench_docker.py --stage report --targets greptimedb \
+          --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
+          --root "$RUN_ROOT/s3" --data-root "$RUN_ROOT" --owner "$OWNER-s3" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
     - name: Clean up owned containers
+      env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
       if: always() && steps.runtime.outcome == 'success'
       run: |
-        python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \
+        python3 greptimedb/.github/scripts/o11ybench-storage.py run -- python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \
           --dataset "$DATASET" --runtime-cpus "$RUNTIME_CPUS" --runtime-memory "$RUNTIME_MEMORY" \
           --root "$RUN_ROOT" --owner "$OWNER" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
     - name: Clean up baseline containers
+      env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
       if: always() && steps.runtime.outcome == 'success' && inputs.compare_greptimedb
       run: |
-        python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \
+        python3 greptimedb/.github/scripts/o11ybench-storage.py run -- python3 o11ybench/scripts/run_tracesbench_docker.py --stage cleanup \
           --root "$RUN_ROOT/baseline" --owner "$OWNER-base" --runtime-image "$RESOLVED_RUNTIME_IMAGE"
     - name: Benchmark summary
       if: always()
-      run: python3 greptimedb/.github/scripts/tracesbench-summary.py --root "$RUN_ROOT" --targets "$TARGETS" --section benchmark >> "$GITHUB_STEP_SUMMARY"
+      run: python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind traces --root "$RUN_ROOT" --targets "$TARGETS" --section benchmark | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
     - name: Load / lifecycle summary
       if: always()
-      run: python3 greptimedb/.github/scripts/tracesbench-summary.py --root "$RUN_ROOT" --targets "$TARGETS" --section lifecycle >> "$GITHUB_STEP_SUMMARY"
+      run: python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind traces --root "$RUN_ROOT" --targets "$TARGETS" --section lifecycle | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
     - name: Baseline load / lifecycle summary
       if: always() && inputs.compare_greptimedb
       run: |
-        echo '## Baseline GreptimeDB' >> "$GITHUB_STEP_SUMMARY"
-        python3 greptimedb/.github/scripts/tracesbench-summary.py \
-          --root "$RUN_ROOT/baseline" --targets greptimedb --section lifecycle >> "$GITHUB_STEP_SUMMARY"
+        echo '## Baseline GreptimeDB' | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+        python3 greptimedb/.github/scripts/o11ybench-storage.py summary --kind traces \
+          --root "$RUN_ROOT/baseline" --targets greptimedb --section lifecycle | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+    - name: Render storage comparison overview
+      if: always() && steps.generate.outcome == 'success'
+      run: |
+        mkdir -p "$RUN_ROOT/metadata"
+        cp "$METADATA_ROOT"/*-image.json "$RUN_ROOT/metadata/"
+        python3 o11ybench/scripts/build_query_performance_report.py --traces "$RUN_ROOT" \
+          --output "$RUN_ROOT-query-report.html" --json-output "$RUN_ROOT-query-report.json"
+        python3 greptimedb/.github/scripts/o11ybench-storage.py overview "$RUN_ROOT-query-report.json" \
+          | tee -a "$RUN_ROOT/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+        python3 greptimedb/.github/scripts/o11ybench-storage.py audit --root "$RUN_ROOT" \
+          --report "$RUN_ROOT-query-report.json" --targets "$TARGETS" --allow-trace-query-failures
     - name: Stage benchmark artifacts
       if: always()
       run: |
         mkdir -p "$RUNNER_TEMP/tracesbench-artifacts"
+        cp "$RUN_ROOT"-query-report.* "$RUNNER_TEMP/tracesbench-artifacts/" 2>/dev/null || true
         if [[ -d "$METADATA_ROOT" ]]; then
           rsync -a --no-owner --no-group "$METADATA_ROOT" "$RUNNER_TEMP/tracesbench-artifacts/"
         fi
-        for entry in runs report report.json report.md version-comparison.json version-comparison.md; do
+        for entry in artifact-audit.json metadata runs report report.json report.md ci-summary.md version-comparison.json version-comparison.md; do
           if [[ -e "$RUN_ROOT/$entry" ]]; then
             rsync -a --no-owner --no-group "$RUN_ROOT/$entry" "$RUNNER_TEMP/tracesbench-artifacts/"
           fi
@@ -446,6 +540,12 @@ jobs:
           mkdir -p "$RUNNER_TEMP/tracesbench-artifacts/baseline"
           rsync -a --no-owner --no-group "$RUN_ROOT/baseline/runs" "$RUNNER_TEMP/tracesbench-artifacts/baseline/"
         fi
+        for arm in s3 baseline/s3; do
+          if [[ -d "$RUN_ROOT/$arm" ]]; then
+            mkdir -p "$RUNNER_TEMP/tracesbench-artifacts/$arm"
+            rsync -a --no-owner --no-group --exclude scratch --exclude corpora "$RUN_ROOT/$arm/" "$RUNNER_TEMP/tracesbench-artifacts/$arm/"
+          fi
+        done
     - name: Upload HTML and results (no corpus)
       if: always()
       uses: actions/upload-artifact@v4

.github/workflows/vmbench-long-range.yml

@@ -2,6 +2,11 @@ name: VMBench Long Range
 on:
   workflow_dispatch:
     inputs:
+      greptimedb_storage:
+        description: 'GreptimeDB data storage (S3 requires AWS in the bucket region; WAL stays local)'
+        type: choice
+        options: [file, s3, both]
+        default: file
       run_greptimedb:
         description: Run GreptimeDB
         type: boolean
@@ -12,6 +17,11 @@ on:
         type: boolean
         default: false
         required: false
+      run_prometheus:
+        description: Run Prometheus (requires a runtime image with managed Prometheus support)
+        type: boolean
+        default: false
+        required: false
       range_preset:
         description: Long-range query window (fixed dataset per preset)
         type: choice
@@ -40,8 +50,13 @@ on:
         type: string
         default: v1.114.0@sha256:227ad53fd57a4b4430624898a1be0cac39431e1235fa0dc92f567cce38969344
         required: true
+      prometheus_tag:
+        description: Docker Hub prom/prometheus tag or tag@sha256 digest
+        type: string
+        default: v3.2.1@sha256:6927e0919a144aa7616fd0137d4816816d42f6b816de3af269ab065250859a62
+        required: true
       runtime_image:
-        description: 'Runtime image: auto = AWS ECR Public or Aliyun ACR by provider; explicit tag/digest overrides'
+        description: 'Runtime image: auto = pinned Prometheus/File/S3 runtime on ACR; explicit tag/digest overrides'
         type: string
         default: auto
         required: true
@@ -89,13 +104,19 @@ on:
 permissions:
   contents: read
 env:
+  GREPTIMEDB_STORAGE: ${{ inputs.greptimedb_storage || 'file' }}
+  S3_BUCKET: ${{ vars.AWS_CI_TEST_BUCKET }}
+  S3_REGION: ${{ vars.AWS_CI_TEST_BUCKET_REGION }}
+  RUNNER_REGION: ${{ vars.EC2_RUNNER_REGION }}
+  RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }}
   COMPARE_GREPTIMEDB: ${{ inputs.compare_greptimedb }}
   BASELINE_GREPTIMEDB_TAG: ${{ inputs.baseline_greptimedb_tag }}
   BENCHMARK_ROOT: ${{ inputs.benchmark_root }}
   RANGE_PRESET: ${{ inputs.range_preset }}
   GREPTIMEDB_TAG: ${{ inputs.greptimedb_tag }}
   VICTORIAMETRICS_TAG: ${{ inputs.victoriametrics_tag }}
-  RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || inputs.provider == 'AWS' && 'public.ecr.aws/i8k6a5e1/greptime/o11ybenchruntime:b43785b56f42@sha256:e38811eb68caf3682ba56f5591a3e8e27e78e7983a923abb29fa2d7cd8fdbd46' || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:b43785b56f42@sha256:e38811eb68caf3682ba56f5591a3e8e27e78e7983a923abb29fa2d7cd8fdbd46' }}
+  PROMETHEUS_TAG: ${{ inputs.prometheus_tag }}
+  RUNTIME_IMAGE: ${{ inputs.runtime_image != 'auto' && inputs.runtime_image || 'greptime-registry.cn-hangzhou.cr.aliyuncs.com/tools/o11ybench-runtime:345b6f7c-prometheus-s3@sha256:39d207f4a9c479b8614b8283e2d049fc8f430fcbd2b3362ce3035cfbf294edf9' }}
   PROVIDER: ${{ inputs.provider }}
   REQUESTED_INSTANCE_TYPE: ${{ inputs.instance_type }}
   SYSTEM_DISK_GIB: ${{ inputs.system_disk_gib }}
@@ -119,6 +140,10 @@ jobs:
       with:
         path: greptimedb
         persist-credentials: false
+    - name: Validate GreptimeDB storage
+      run: |
+        python3 greptimedb/tests/perf/test_o11ybench_storage.py
+        python3 greptimedb/.github/scripts/o11ybench-storage.py validate
     - name: Validate version comparison
       env:
         RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }}
@@ -132,11 +157,24 @@ jobs:
       run: |
         python3 greptimedb/tests/perf/test_aliyun_ecs_runner_scripts.py
         python3 greptimedb/tests/perf/test_ci_runner_scripts.py
+    - name: Configure AWS credentials for ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/configure-aws-credentials@v4
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+    - name: Login to Amazon ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/amazon-ecr-login@v2
+      with:
+        registry-type: public
     - name: Validate budgets and build warm plan
       id: plan
       env:
         RUN_GREPTIMEDB: ${{ inputs.run_greptimedb }}
         RUN_VICTORIAMETRICS: ${{ inputs.run_victoriametrics }}
+        RUN_PROMETHEUS: ${{ inputs.run_prometheus }}
       run: |
         set -euo pipefail
         [[ "$BENCHMARK_ROOT" == /* && "$BENCHMARK_ROOT" != / && "$BENCHMARK_ROOT" != */ ]]
@@ -161,17 +199,18 @@ jobs:
         [[ "$revision" =~ ^[0-9a-f]{40}$ ]]
         printf 'runtime_image=%s\no11ybench_ref=%s\nsystem_disk_gib=%s\n' "$resolved" "$revision" "$SYSTEM_DISK_GIB" >> "$GITHUB_OUTPUT"
         printf 'run_id=long-range-%s-%s\nplan_artifact=vmbench-plan-%s-%s\n' "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT" "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT" >> "$GITHUB_OUTPUT"
-        args=(--workload vmbench-long-range --range-preset "$RANGE_PRESET"
+        args=(--workload vmbench-long-range --range-preset "$RANGE_PRESET" --greptimedb-storage "$GREPTIMEDB_STORAGE"
           --db-cpus "$DB_CPUS" --db-memory "$DB_MEMORY"
           --run-classification controlled_server_benchmark --no-open-browser)
         if [[ "$COMPARE_GREPTIMEDB" == true ]]; then
           args+=(--target "greptimedb:name=greptimedb-baseline,image=greptime/greptimedb:$BASELINE_GREPTIMEDB_TAG,url=http://127.0.0.1:14000/v1/prometheus/api/v1/query_range")
         fi
         # Tags become target-spec fields; reject separators before passing them to the planner.
-        for target in greptimedb victoriametrics; do
+        for target in greptimedb victoriametrics prometheus; do
           case "$target" in
             greptimedb) enabled=$RUN_GREPTIMEDB; tag=$GREPTIMEDB_TAG; repo=greptime/greptimedb; url=http://127.0.0.1:14000/v1/prometheus/api/v1/query_range ;;
             victoriametrics) enabled=$RUN_VICTORIAMETRICS; tag=$VICTORIAMETRICS_TAG; repo=victoriametrics/victoria-metrics; url=http://127.0.0.1:18428/api/v1/query_range ;;
+            prometheus) enabled=$RUN_PROMETHEUS; tag=$PROMETHEUS_TAG; repo=prom/prometheus; url=http://127.0.0.1:19090/api/v1/query_range ;;
           esac
           [[ "$enabled" == true ]] || continue
           [[ "$tag" =~ ^[a-zA-Z0-9_][a-zA-Z0-9_.-]{0,127}(@sha256:[0-9a-f]{64})?$ ]]
@@ -180,10 +219,11 @@ jobs:
         # The packaged planner owns profile, target and resource-limit validation.
         mkdir -p "$RUNNER_TEMP/vmbench-plans"
         # Keep the original warm defaults: serial 5/1 and c1/c4 30s/5s.
-        docker run --rm "$resolved" scripts/o11ybench_plan.py "${args[@]}" \
+        docker run --rm -e S3_BUCKET -e S3_REGION "$resolved" scripts/o11ybench_plan.py "${args[@]}" \
           --run-id "long-range-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT-warm" \
           --output-root "$BENCHMARK_ROOT/o11ybench/o11ybench-long-range-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT-warm" \
           --cache-policy warm > "$RUNNER_TEMP/vmbench-plans/warm.json"
+        python3 greptimedb/.github/scripts/o11ybench-storage.py normalize-metrics "$RUNNER_TEMP/vmbench-plans/warm.json"
     - uses: actions/upload-artifact@v4
       with:
         name: vmbench-plan-${{ github.run_id }}-${{ github.run_attempt }}
@@ -266,7 +306,10 @@ jobs:
         path: greptimedb
         persist-credentials: false
     - name: Set up benchmark dependencies
-      uses: ./greptimedb/.github/actions/setup-benchmark
+      uses: ./greptimedb/.github/actions/setup-linux-runner
+      with:
+        tools: curl jq python3 rsync
+        expected-architecture: x86_64
     - name: Checkout matching benchmark tools
       uses: actions/checkout@v4
       with:
@@ -279,6 +322,18 @@ jobs:
       with:
         name: ${{ needs.validate.outputs.plan_artifact }}
         path: ${{ env.METADATA_ROOT }}
+    - name: Configure AWS credentials for ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/configure-aws-credentials@v4
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ECR_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_ECR_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+    - name: Login to Amazon ECR Public
+      if: startsWith(env.RUNTIME_IMAGE, 'public.ecr.aws/')
+      uses: aws-actions/amazon-ecr-login@v2
+      with:
+        registry-type: public
     - name: Check runner and resolve database images
       run: |
         set -euo pipefail
@@ -313,25 +368,92 @@ jobs:
           --plan "$METADATA_ROOT/warm.json" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --dry-run \
           > "$METADATA_ROOT/warm-execution-plan.json"
         cat "$METADATA_ROOT/warm-execution-plan.json"
-    - name: Warm serial and concurrent benchmarks
+    - name: Generate shared dataset and split target plans
+      id: generate
       run: |
-        set -euo pipefail
-        python3 -u o11ybench/scripts/run_benchmark_plan.py \
-          --plan "$METADATA_ROOT/warm.json" --runtime-image "$RESOLVED_RUNTIME_IMAGE" --execute \
-          2>&1 | tee "$METADATA_ROOT/warm-execution.log"
-        python3 o11ybench/scripts/check_run_benchmark_plan.py --artifacts "$ARTIFACT_ROOT-warm"
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-prepare \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Load and benchmark Baseline GreptimeDB File
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.compare_greptimedb && inputs.greptimedb_storage != 's3' }}
+      run: |
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb-baseline \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Load and benchmark Baseline GreptimeDB S3
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.compare_greptimedb && inputs.greptimedb_storage != 'file' }}
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      run: |
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb-s3-baseline \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Load and benchmark GreptimeDB File
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_greptimedb && inputs.greptimedb_storage != 's3' }}
+      run: |
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Load and benchmark GreptimeDB S3
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_greptimedb && inputs.greptimedb_storage != 'file' }}
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      run: |
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target greptimedb-s3 \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Load and benchmark VictoriaMetrics
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_victoriametrics }}
+      run: |
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target victoriametrics \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Load and benchmark Prometheus
+      if: ${{ !cancelled() && steps.generate.outcome == 'success' && inputs.run_prometheus }}
+      run: |
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-run --target prometheus \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
+    - name: Validate all selected target results
+      if: always() && steps.generate.outcome == 'success'
+      run: |
+        python3 greptimedb/.github/scripts/o11ybench-storage.py metrics-validate \
+          --plan "$METADATA_ROOT/warm.json" --tools o11ybench --runtime-image "$RESOLVED_RUNTIME_IMAGE"
     - name: Clean up owned containers
+      env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_ACCESS_KEY_ID || '' }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.greptimedb_storage != 'file' && secrets.AWS_SECRET_ACCESS_KEY || '' }}
       if: always()
       run: |
         docker ps -aq --filter "label=o11ybench.run-id=$RUN_ID-warm" | xargs -r docker rm -fv
+        for name in greptimedb greptimedb-s3 greptimedb-baseline greptimedb-s3-baseline victoriametrics prometheus; do
+          plan="$METADATA_ROOT/$name.json"
+          [[ -f "$plan" ]] || continue
+          run_id=$(jq -r '.run_id // empty' "$plan")
+          [[ -n "$run_id" ]] || continue
+          docker ps -aq --filter "label=o11ybench.run-id=$run_id" | xargs -r docker rm -fv
+        done
+        if [[ -d "$ARTIFACT_ROOT-warm" ]]; then
+          for probe in "$ARTIFACT_ROOT-warm"/o11ybench/*/*/s3-probe.json; do
+            [[ -f "$probe" ]] || continue
+            python3 o11ybench/scripts/greptimedb_storage.py recover --runtime-image "$RESOLVED_RUNTIME_IMAGE" --output "${probe%/*}"
+          done
+        fi
     - name: GreptimeDB version comparison
       if: always() && inputs.compare_greptimedb
       run: |
-        python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind long-range --root "$ARTIFACT_ROOT-warm" >> "$GITHUB_STEP_SUMMARY"
+        python3 greptimedb/.github/scripts/benchmark-version-comparison.py report --kind long-range --root "$ARTIFACT_ROOT-warm" | tee -a "$ARTIFACT_ROOT-warm/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+    - name: Render storage comparison overview
+      if: always() && steps.generate.outcome == 'success'
+      run: |
+        mkdir -p "$ARTIFACT_ROOT-warm/metadata"
+        cp "$METADATA_ROOT"/*-image.json "$ARTIFACT_ROOT-warm/metadata/"
+        python3 o11ybench/scripts/build_query_performance_report.py --metrics "$ARTIFACT_ROOT-warm" \
+          --output "$ARTIFACT_ROOT-warm-query-report.html" --json-output "$ARTIFACT_ROOT-warm-query-report.json"
+        python3 greptimedb/.github/scripts/o11ybench-storage.py overview "$ARTIFACT_ROOT-warm-query-report.json" \
+          | tee -a "$ARTIFACT_ROOT-warm/ci-summary.md" >> "$GITHUB_STEP_SUMMARY"
+        python3 greptimedb/.github/scripts/o11ybench-storage.py audit --root "$ARTIFACT_ROOT-warm" \
+          --report "$ARTIFACT_ROOT-warm-query-report.json" --targets "$(jq -r '[.targets[].kind] | unique | join(",")' "$METADATA_ROOT/warm.json")"
     - name: Stage benchmark artifacts
       if: always()
       run: |
         mkdir -p "$RUNNER_TEMP/vmbench-artifacts"
+        cp "$ARTIFACT_ROOT-warm"-query-report.* "$RUNNER_TEMP/vmbench-artifacts/" 2>/dev/null || true
         if [[ -d "$METADATA_ROOT" ]]; then
           rsync -a --no-owner --no-group "$METADATA_ROOT" "$RUNNER_TEMP/vmbench-artifacts/"
         fi

.gitignore

@@ -79,3 +79,5 @@ docs/specs/
 
 # Claude Code personal/local overrides (keep shared .claude/ content tracked)
 .claude/*.local.* 
+
+output/

Cargo.toml

@@ -9,6 +9,7 @@ members = [
     "src/cmd",
     "src/common/base",
     "src/common/batcher",
+    "src/common/cache",
     "src/common/catalog",
     "src/common/config",
     "src/common/datasource",
@@ -63,6 +64,7 @@ members = [
     "src/promql",
     "src/puffin",
     "src/query",
+    "src/relational-engine",
     "src/standalone",
     "src/servers",
     "src/session",
@@ -133,6 +135,7 @@ dashmap = "6.1"
 datafusion = "=55.1.0"
 datafusion-common = "=55.1.0"
 datafusion-datasource = "=55.1.0"
+datafusion-execution = "=55.1.0"
 datafusion-expr = "=55.1.0"
 datafusion-expr-common = "=55.1.0"
 datafusion-functions = "=55.1.0"
@@ -140,7 +143,7 @@ datafusion-functions-aggregate-common = "=55.1.0"
 datafusion-functions-window-common = "=55.1.0"
 datafusion-optimizer = "=55.1.0"
 datafusion-orc = "0.10.0"
-datafusion-pg-catalog = "0.18.3"
+datafusion-pg-catalog = "0.19.0"
 datafusion-physical-expr = "=55.1.0"
 datafusion-physical-plan = "=55.1.0"
 datafusion-proto = "=55.1.0"
@@ -161,7 +164,10 @@ fs2 = "0.4"
 fst = "0.4.7"
 futures = "0.3"
 futures-util = "0.3"
-greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "f51cb5312bf7feaf1199361ce16edc63a0ad40b2" }
+# TODO(narthex-merge): temporary pre-merge pin. Points at the fork branch carrying
+# PR GreptimeTeam/greptime-proto#355 (RawSeriesSource on top of main's 5870f5adc).
+# Replace with the GreptimeTeam rev once #355 lands.
+greptime-proto = { git = "https://github.com/discord9/greptime-proto.git", rev = "b9159cd680ac2b88d155cbc9effd6df9694e8597" }
 hdfs-native = "0.14.6"
 hex = "0.4"
 hostname = "0.4.0"
@@ -173,7 +179,7 @@ hyper-util = "0.1"
 icu_properties = "2.0.1"
 indicatif = "0.17"
 itertools = "0.14"
-jsonb = { git = "https://github.com/GreptimeTeam/jsonb", rev = "dd1e2341e3df35e9de38959328caa7628e81b6b1", default-features = false, features = [
+jsonb = { git = "https://github.com/GreptimeTeam/jsonb", rev = "5a6472858d860ac034075c17253f41617bdfa956", default-features = false, features = [
     "databend",
 ] }
 lazy_static = "1.4"
@@ -238,19 +244,20 @@ rskafka = { git = "https://github.com/influxdata/rskafka.git", rev = "80f42eb15e
 rstest = "0.25"
 rstest_reuse = "0.7"
 rust_decimal = "1.33"
-rustls = { version = "0.23.25", default-features = false }
+rustls = { version = "0.23.46", default-features = false }
 ryu = "1.0"
 sea-query = "0.32"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = { version = "1.0", features = ["float_roundtrip"] }
 serde_with = "3"
 serde_yaml_ng = "0.10"
-simd-json = "0.15"
+sha2 = "0.10"
+simd-json = "0.19"
 similar-asserts = "1.6.0"
 smallvec = { version = "1", features = ["serde"] }
 snafu = "0.8"
 sqlparser = { version = "0.62.0", default-features = false, features = ["std", "visitor", "serde"] }
-sqlx = { version = "0.8", default-features = false, features = [
+sqlx = { version = "0.9", default-features = false, features = [
     "any",
     "macros",
     "json",
@@ -301,6 +308,7 @@ client = { path = "src/client" }
 cmd = { path = "src/cmd", default-features = false }
 common-base = { path = "src/common/base" }
 common-batcher = { path = "src/common/batcher" }
+common-cache = { path = "src/common/cache" }
 common-catalog = { path = "src/common/catalog" }
 common-config = { path = "src/common/config" }
 common-datasource = { path = "src/common/datasource" }
@@ -354,6 +362,7 @@ plugins = { path = "src/plugins" }
 promql = { path = "src/promql" }
 puffin = { path = "src/puffin" }
 query = { path = "src/query" }
+relational-engine = { path = "src/relational-engine" }
 servers = { path = "src/servers" }
 session = { path = "src/session" }
 sql = { path = "src/sql" }
@@ -368,25 +377,30 @@ rev = "f743c3a5fe8c1f57363eac5866c356b7d91f9b39"
 
 [patch.crates-io]
 substrait = { git = "https://github.com/GreptimeTeam/substrait-rs.git", rev = "91ec978b0649417ad3da8390e7a515baec723b1b" }
-datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "e3221b8f74b15c7192621576e5d1e354a6cf097a" }
-datafusion-pg-catalog = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "ffb14a52b3d7c7489559812e1ec38096f225ce93" }
-arrow-pg = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "ffb14a52b3d7c7489559812e1ec38096f225ce93" }
+datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-execution = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" }
+datafusion-pg-catalog = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" }
+arrow-pg = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" }
 sqlparser = { git = "https://github.com/GreptimeTeam/sqlparser-rs.git", rev = "9e9019bb1c7040ed956f654e39378dd42ab17884" }
 
+# pprof 0.15.0 initializes framehop inside SIGPROF; pin the unreleased fix
+# that initializes it before sampling. Remove once tikv/pprof-rs#282 is released.
+pprof = { git = "https://github.com/tikv/pprof-rs", rev = "01cff82dbe6fe110a707bf2b38d8ebb1d14a18f8" }
+
 # Temporary: use the GreptimeTeam fork of tikv-jemalloc-sys embedding
 # jemalloc 5.3.1 + backport of 54f22c83 ("Initialize TSD tcache before
 # enabling it"), fixing the TSD/tcache enable-before-init reentrancy

README.md

@@ -9,13 +9,11 @@
 <h2 align="center">Metrics, logs, and traces.<br/>
 One engine, on your infrastructure.</h2>
 
-> A columnar database for metrics, logs, and traces on object storage.
-> Apache-2.0 licensed core.
-
 <div align="center">
 
 <a href="https://docs.greptime.com/user-guide/overview/">User Guide</a> &nbsp;·&nbsp;
 <a href="https://greptimedb.rs/">API Docs</a> &nbsp;·&nbsp;
+<a href="https://live.greptime.cloud/dashboard/?info=eyJ1c2VybmFtZSI6ImdyZXB0aW1lIiwicGFzc3dvcmQiOiJkZW1vc2Vzc2lvbiJ9&query_guide=1&utm_source=github&utm_medium=readme&utm_campaign=greptimedb#/dashboard/explore">Live Demo</a> &nbsp;·&nbsp;
 <a href="https://github.com/GreptimeTeam/greptimedb/issues/7685">Roadmap 2026</a> &nbsp;·&nbsp;
 <a href="https://greptime.com/slack">Slack</a>
 
@@ -31,7 +29,7 @@ One engine, on your infrastructure.</h2>
 
 ## Introduction
 
-**GreptimeDB** is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one [table model](https://docs.greptime.com/user-guide/concepts/data-model/): tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
+**GreptimeDB** is an open-source [observability database](https://docs.greptime.com/user-guide/concepts/why-greptimedb/#what-an-observability-database-is). Metrics, logs, and traces run on one columnar engine over object storage and share one [table model](https://docs.greptime.com/user-guide/concepts/data-model/): tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
 
 Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
 
@@ -61,76 +59,6 @@ LIMIT 20;
 Metrics join the same way, on any tag the tables share, such as `service`,
 `host`, or `pod`.
 
-## Why You Might Use It
-
-- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
-- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
-- You are hitting Loki's query performance limits as log volume grows
-- You need long retention on object storage without a separate analytics stack
-- You want to query telemetry with SQL, not only a domain query language
-- You are storing GenAI or agent telemetry ([OTel GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)) alongside infrastructure signals
-
-Learn more in [Why GreptimeDB](https://docs.greptime.com/user-guide/concepts/why-greptimedb).
-
-## What's Supported
-
-| | |
-|---|---|
-| **Ingest** | [OpenTelemetry (OTLP)](https://docs.greptime.com/user-guide/ingest-data/for-observability/opentelemetry/), [Prometheus Remote Write](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/), [Loki Push](https://docs.greptime.com/user-guide/ingest-data/for-observability/loki), [Elasticsearch Bulk](https://docs.greptime.com/user-guide/ingest-data/for-observability/elasticsearch), [InfluxDB line protocol](https://docs.greptime.com/user-guide/protocols/influxdb-line-protocol), [gRPC](https://docs.greptime.com/user-guide/protocols/grpc/) |
-| **Query** | [SQL](https://docs.greptime.com/user-guide/query-data/overview/), [PromQL](https://docs.greptime.com/user-guide/query-data/promql/), [Jaeger-compatible trace queries](https://docs.greptime.com/user-guide/query-data/jaeger/), MySQL and PostgreSQL wire protocols |
-| **Storage** | S3, GCS, Azure Blob and S3-compatible endpoints as [primary storage](https://docs.greptime.com/user-guide/deployments-administration/configuration/#storage-options), with memory and local-disk caches |
-| **Built in** | Retention policies, downsampling, [continuous aggregation](https://docs.greptime.com/user-guide/flow-computation/overview), explicit table partitioning, and [inverted / skipping / fulltext indexes](https://docs.greptime.com/user-guide/manage-data/data-index) |
-
-Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
-
-<p align="center">
-  <a href="https://github.com/GreptimeTeam/greptimedb/raw/main/docs/overview.png" target="_blank" rel="noopener">
-    <img alt="GreptimeDB Overview" src="docs/overview.png" width="900px">
-  </a>
-</p>
-
-## Benchmarks
-
-* [Agent RCA Bench](https://rca-bench.greptime.com/): LLM agents doing root cause analysis
-  over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer
-  input tokens, 45% lower cost
-  ([write-up](https://greptime.com/blogs/2026-09-08-agent-rca-bench-interface-semantic-layer))
-* [GreptimeDB tops JSONBench's billion-record cold run test](https://greptime.com/blogs/2025-03-18-jsonbench-greptimedb-performance)
-* [TSBS Benchmark](https://github.com/GreptimeTeam/greptimedb/tree/main/docs/benchmarks/tsbs)
-* [More benchmark reports](https://docs.greptime.com/user-guide/concepts/features-that-you-concern#how-is-greptimedbs-performance-compared-to-other-solutions)
-
-## Compatibility and Migration
-
-Compatibility is per protocol, and query-side coverage is narrower than ingestion.
-
-| | Compatible | Not compatible |
-|---|---|---|
-| **Prometheus** | Remote Write ingestion; PromQL queries | Gaps are listed in [PromQL compatibility](https://docs.greptime.com/user-guide/query-data/promql/) |
-| **Loki** | Push ingestion; dual-write through Grafana Alloy makes the [cutover gradual](https://greptime.com/blogs/2026-07-23-from-loki-to-greptimedb-dual-write-migration) | LogQL and the rest of the Loki query API |
-| **Elasticsearch** | `_bulk` ingestion in the open-source core; [QueryDSL](https://docs.greptime.com/enterprise/elasticsearch-compatible/query/) partially, in Enterprise | Most other Elasticsearch APIs |
-
-## Limitations and Edition Boundary
-
-Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
-
-Read replicas, workload isolation, and automated repartitioning are **GreptimeDB Enterprise** features, along with enterprise security and governance. The [Enterprise overview](https://docs.greptime.com/enterprise/overview/) has the current list, and [pricing](https://www.greptime.com/pricing#differences) has the edition comparison.
-
-## Architecture
-
-GreptimeDB can run in two modes:
-* **Standalone** — single binary for development and small deployments.
-* **Distributed** — four components, each independently scalable:
-  - **Frontend** — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
-  - **Datanode** — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
-  - **Metasrv** — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
-  - **Flownode** (optional) — continuous flow computation (streaming and materialized views).
-
-For deeper coverage, see the [architecture doc](https://docs.greptime.com/contributor-guide/overview/#architecture) or [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb/1-overview).
-
-<a href="https://github.com/GreptimeTeam/greptimedb/raw/main/docs/architecture.png" target="_blank" rel="noopener">
-  <img alt="GreptimeDB System Overview" src="https://cdn.jsdelivr.net/gh/GreptimeTeam/greptimedb@main/docs/architecture.png">
-</a>
-
 ## Try GreptimeDB
 
 **For AI agents** — paste this prompt into your agent:
@@ -150,20 +78,81 @@ docker run -p 127.0.0.1:4000-4003:4000-4003 \
   --mysql-addr 0.0.0.0:4002 \
   --postgres-addr 0.0.0.0:4003
 ```
-Dashboard: [http://localhost:4000/dashboard](http://localhost:4000/dashboard)
+Dashboard: [http://localhost:4000/dashboard](http://localhost:4000/dashboard)  
+Or skip the install and try the [Live Demo](https://live.greptime.cloud/dashboard/?info=eyJ1c2VybmFtZSI6ImdyZXB0aW1lIiwicGFzc3dvcmQiOiJkZW1vc2Vzc2lvbiJ9&query_guide=1&utm_source=github&utm_medium=readme&utm_campaign=greptimedb#/dashboard/explore).
 
-Read more in the [full Install Guide](https://docs.greptime.com/getting-started/installation/overview).
+Next: [Quickstart](https://docs.greptime.com/getting-started/quick-start), the [full Install Guide](https://docs.greptime.com/getting-started/installation/overview), [Demo Scenes](https://github.com/GreptimeTeam/demo-scene), and the [FAQ](https://docs.greptime.com/faq-and-others/faq).
 
-**Troubleshooting:**
-* Cannot connect to the database? Ensure that ports `4000`, `4001`, `4002`, and `4003` are not blocked by a firewall or used by other services.
-* Failed to start? Check the container logs with `docker logs greptime` for further details.
+## Why You Might Use It
 
-## Getting Started
+- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
+- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
+- You are hitting Loki's query performance limits as log volume grows
+- You need long retention on object storage without a separate analytics stack
+- You want to query telemetry with SQL, not only a domain query language
+- You are storing GenAI or agent telemetry ([OTel GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)) alongside infrastructure signals
 
-- [Quickstart](https://docs.greptime.com/getting-started/quick-start)
-- [User Guide](https://docs.greptime.com/user-guide/overview)
-- [Demo Scenes](https://github.com/GreptimeTeam/demo-scene)
-- [FAQ](https://docs.greptime.com/faq-and-others/faq)
+Learn more in [Why GreptimeDB](https://docs.greptime.com/user-guide/concepts/why-greptimedb).
+
+## What's Supported
+
+| | |
+|---|---|
+| **Ingest** | [OpenTelemetry (OTLP)](https://docs.greptime.com/user-guide/ingest-data/for-observability/opentelemetry/), [Prometheus Remote Write](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/) (with [native histograms](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/#native-histograms) over Remote Write 2.0), [Loki Push](https://docs.greptime.com/user-guide/ingest-data/for-observability/loki), [Elasticsearch Bulk](https://docs.greptime.com/user-guide/ingest-data/for-observability/elasticsearch), [InfluxDB line protocol](https://docs.greptime.com/user-guide/protocols/influxdb-line-protocol), [gRPC](https://docs.greptime.com/user-guide/protocols/grpc/) |
+| **Query** | [SQL](https://docs.greptime.com/user-guide/query-data/overview/), [PromQL](https://docs.greptime.com/user-guide/query-data/promql/), [Jaeger-compatible trace queries](https://docs.greptime.com/user-guide/query-data/jaeger/), MySQL and PostgreSQL wire protocols |
+| **Storage** | S3, GCS, Azure Blob and S3-compatible endpoints as [primary storage](https://docs.greptime.com/user-guide/deployments-administration/configuration/#storage-options), with memory and local-disk caches |
+| **Built in** | [Pipelines](https://docs.greptime.com/user-guide/logs/use-custom-pipelines) that parse and transform logs at ingestion, retention policies, downsampling, [continuous aggregation](https://docs.greptime.com/user-guide/flow-computation/overview), explicit table partitioning, and [inverted / skipping / fulltext indexes](https://docs.greptime.com/user-guide/manage-data/data-index) |
+
+Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
+
+<p align="center">
+  <a href="https://github.com/GreptimeTeam/greptimedb/raw/main/docs/overview.png" target="_blank" rel="noopener">
+    <img alt="GreptimeDB Overview" src="docs/overview.png" width="900px">
+  </a>
+</p>
+
+## Benchmarks
+
+* [Agent RCA Bench](https://rca-bench.greptime.com/): LLM agents doing root cause analysis
+  over GreptimeDB versus Prometheus + Loki + Tempo, 168 investigations per interface across
+  six models. 40% fewer wrong diagnoses, about 48% fewer input tokens, about 45% lower
+  estimated cost
+  ([write-up](https://greptime.com/blogs/2026-09-08-agent-rca-bench-interface-semantic-layer))
+* [JSONBench billion-record cold run: first place as of March 2025](https://greptime.com/blogs/2025-03-18-jsonbench-greptimedb-performance)
+* [TSBS Benchmark](https://github.com/GreptimeTeam/greptimedb/tree/main/docs/benchmarks/tsbs)
+* [More benchmark reports](https://docs.greptime.com/user-guide/concepts/features-that-you-concern#how-is-greptimedbs-performance-compared-to-other-solutions)
+
+## Compatibility and Migration
+
+Compatibility is per protocol, and query-side coverage is narrower than ingestion.
+
+| | Compatible | Not compatible |
+|---|---|---|
+| **Prometheus** | Remote Write ingestion; PromQL queries | Gaps are listed in [PromQL compatibility](https://docs.greptime.com/user-guide/query-data/promql/) |
+| **Loki** | Push ingestion; dual-write through Grafana Alloy makes the [cutover gradual](https://greptime.com/blogs/2026-07-23-from-loki-to-greptimedb-dual-write-migration) | LogQL and the rest of the Loki query API |
+| **Elasticsearch** | `_bulk` ingestion in the open-source core; [QueryDSL](https://docs.greptime.com/enterprise/elasticsearch-compatible/query/) partially, in Enterprise | Most other Elasticsearch APIs |
+
+## Limitations and Edition Boundary
+
+Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning and region migration are manual operations there.
+
+Read replicas, workload isolation, and automated repartitioning are **GreptimeDB Enterprise** features, along with enterprise security and governance. The [Enterprise overview](https://docs.greptime.com/enterprise/overview/) has the current list, and [pricing](https://www.greptime.com/pricing#differences) has the edition comparison. For GreptimeDB Enterprise inquiries, [contact us](https://greptime.com/contactus).
+
+## Architecture
+
+GreptimeDB can run in two modes:
+* **Standalone** — single binary for development and small deployments.
+* **Distributed** — four components, each independently scalable:
+  - **Frontend** — protocol endpoints and the distributed query engine. Stateless, scales horizontally.
+  - **Datanode** — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage.
+  - **Metasrv** — metadata, region routing, and cluster coordination (DDL procedures, failover, region migration, repartitioning). Backed by a pluggable KV layer (etcd or RDS).
+  - **Flownode** (optional) — continuous flow computation (streaming and materialized views).
+
+For deeper coverage, see the [architecture doc](https://docs.greptime.com/contributor-guide/overview/#architecture) or [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb/1-overview).
+
+<a href="https://github.com/GreptimeTeam/greptimedb/raw/main/docs/architecture.png" target="_blank" rel="noopener">
+  <img alt="GreptimeDB System Overview" src="https://cdn.jsdelivr.net/gh/GreptimeTeam/greptimedb@main/docs/architecture.png">
+</a>
 
 ## Build From Source
 
@@ -202,12 +191,10 @@ See the [Contribution Guidelines](CONTRIBUTING.md) for the full developer workfl
 
 GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — [OceanBase Cloud](https://greptime.com/blogs/2025-07-22-user-case-obcloud-log-management-greptimedb) operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in [case studies](https://greptime.com/blogs/?category=Use%20Case).
 
-Release lines and support windows are in the [version reference](https://docs.greptime.com/nightly/reference/about-greptimedb-version). For where the project is going, read the [v1.0 highlights](https://greptime.com/blogs/2025-11-05-greptimedb-v1-highlights) and the [2026 roadmap](https://greptime.com/blogs/2026-02-11-greptimedb-roadmap-2026).
+Releases follow [Semantic Versioning](https://docs.greptime.com/reference/about-greptimedb-version). For where the project is going, read the [v1.0 highlights](https://greptime.com/blogs/2025-11-05-greptimedb-v1-highlights), the [2026 roadmap](https://github.com/GreptimeTeam/greptimedb/issues/7685), and [what we mean by an observability database](https://greptime.com/blogs/2026-08-26-what-is-an-observability-database).
 
 ## Community
 
-We invite you to engage and contribute!
-
 - [Slack](https://greptime.com/slack)
 - [Discussions](https://github.com/GreptimeTeam/greptimedb/discussions)
 - [Official Website](https://greptime.com/)
@@ -218,6 +205,14 @@ We invite you to engage and contribute!
 
 If GreptimeDB is useful to you, please star the repo.
 
+<a href="https://www.star-history.com/?repos=GreptimeTeam%2FGreptimeDB&type=date&legend=top-left">
+ <picture>
+   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=GreptimeTeam/GreptimeDB&type=date&theme=dark&legend=top-left" />
+   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=GreptimeTeam/GreptimeDB&type=date&legend=top-left" />
+   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=GreptimeTeam/GreptimeDB&type=date&legend=top-left" />
+ </picture>
+</a>
+
 <img alt="Known Users" src="https://greptime.com/logo/img/users.png"/>
 
 ## License
@@ -230,17 +225,10 @@ A small set of peripheral, enterprise-only features are gated behind the
 separate [GreptimeDB Enterprise License](LICENSE-ENTERPRISE). Source files under
 that license carry an explicit Enterprise License header.
 
-## Commercial Support
-
-Scaling observability on your infrastructure?
-[GreptimeDB Enterprise](https://docs.greptime.com/enterprise/overview/) adds the operational,
-security, and support layer for production deployments.
-[Contact us](https://greptime.com/contactus) for details.
-
 ## Contributing
 
 - Read our [Contribution Guidelines](CONTRIBUTING.md).
-- Explore [Internal Concepts](https://docs.greptime.com/contributor-guide/overview.html) and [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb).
+- Explore [Internal Concepts](https://docs.greptime.com/contributor-guide/overview/) and [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb).
 - Pick up a [good first issue](https://github.com/GreptimeTeam/greptimedb/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) and join the #contributors [Slack](https://greptime.com/slack) channel.
 
 [![Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/workflows/integration.yml/badge.svg)](https://github.com/GreptimeTeam/greptimedb/actions/workflows/integration.yml)

config/config.md

@@ -20,6 +20,8 @@
 | `write_bytes_exhausted_policy` | String | `wait` | Policy when write bytes quota is exhausted.<br/>Options: "wait" (default, 10s timeout), "wait(<duration>)" (e.g., "wait(30s)"), "fail" |
 | `init_regions_in_background` | Bool | `false` | Initialize all regions in the background during the startup.<br/>By default, it provides services after all regions have been initialized. |
 | `init_regions_parallelism` | Integer | `16` | Parallelism of initializing regions. |
+| `max_concurrent_queries` | Integer | `0` | The maximum concurrent region-read streams. Zero means unlimited.<br/>A single query can consume multiple permits. |
+| `concurrent_query_limiter_timeout` | String | `100ms` | Timeout to acquire a permit when `max_concurrent_queries` is reached.<br/>Only effective when the limiter is enabled. |
 | `enable_telemetry` | Bool | `true` | Enable telemetry to collect anonymous usage data. Enabled by default. |
 | `runtime` | -- | -- | The runtime options. |
 | `runtime.global_rt_size` | Integer | Auto | The number of threads to execute the runtime for global read operations.<br/>Defaults to max(num_cpus, 2). |
@@ -82,7 +84,7 @@
 | `pending_rows_batcher.worker_channel_capacity` | Integer | `65536` | Maximum queued submissions per table worker. |
 | `pending_rows_batcher.max_inflight_requests` | Integer | `3000` | Maximum admitted original requests awaiting completion. |
 | `pending_rows_batcher.flow_notification_queue_capacity` | Integer | `1024` | Maximum number of queued table Flow notifications. |
-| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics.<br/>Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible.<br/>Enable independently with protocols and a nonzero flush interval.<br/>Omitted fields use independent defaults, not parent settings.<br/>Empty protocols or a zero interval disables logical batching without fallback.<br/>Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. |
+| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and OTLP metrics.<br/>Requires prom_store.with_metric_engine. Logs and traces are not eligible.<br/>Enable independently with protocols and a nonzero flush interval.<br/>Omitted fields use independent defaults, not parent settings.<br/>Empty protocols or a zero interval disables logical batching without fallback.<br/>Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. |
 | `pending_rows_batcher.logical_table.pending_rows_flush_interval` | String | `0s` | Flush interval measured from the first pending submission. Zero disables batching. |
 | `pending_rows_batcher.logical_table.max_batch_rows` | Integer | `100000` | Flush after a complete submission reaches this row threshold. |
 | `pending_rows_batcher.logical_table.max_concurrent_flushes` | Integer | `256` | Maximum concurrent flushes shared by Prom and OTLP metrics. |
@@ -131,6 +133,7 @@
 | `wal.max_unpersisted_bytes` | String | `64MB` | The size of the unpersisted backlog at which new appends stall until an upload completes, in `enqueued` mode.<br/>Nothing is dropped and nothing is rejected; the threshold does not bound what a crash during an outage can lose.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.max_unpersisted_age` | String | `8s` | The age of the oldest unpersisted entry at which new appends stall until an upload completes, in `enqueued` mode.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.on_corrupted_segment` | String | `skip` | What a read does with a segment that still does not decode after a second fetch, because its checksum does not match or its content disagrees with its footer entry.<br/>- `skip`: the segment is skipped and recorded as a WAL hole of its region, a metric is incremented and a warning is logged; the other regions of the object are unaffected (the default).<br/>- `fail`: the read fails, so the region does not open.<br/>**It's only used when the provider is `experimental_object_store`**. |
+| `wal.accept_missing_history` | Bool | `false` | Whether a region opens although the WAL objects under the prefix do not hold its history.<br/>An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.<br/>To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.<br/>When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.<br/>**It's only used when the provider is `experimental_object_store`**. |
 | `metadata_store` | -- | -- | Metadata storage options. |
 | `metadata_store.file_size` | String | `64MB` | The size of the metadata store log file. |
 | `metadata_store.purge_threshold` | String | `256MB` | The threshold of the metadata store size to trigger a purge. |
@@ -142,7 +145,7 @@
 | `query` | -- | -- | The query engine options. |
 | `query.experimental_native_promql` | Bool | `false` | Enable experimental native PromQL execution for compatible queries.<br/>Requires the per-request query.experimental_native_promql=true extension.<br/>Default: false |
 | `query.parallelism` | Integer | `0` | Parallelism of the query engine.<br/>Default to 0, which means the number of CPU cores. |
-| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).<br/>Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.<br/>When this limit is reached, queries will fail with ResourceExhausted error.<br/>NOTE: This does NOT limit memory used by table scans. |
+| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join,<br/>and internal series merges during table scans).<br/>Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.<br/>When this limit is reached, queries will fail with ResourceExhausted error.<br/>Projected Parquet row groups and streamed scan batches use scan_memory_limit separately. |
 | `query.experimental_memory_pool_policy` | String | `greedy` | Experimental memory pool allocation policy:<br/>- "greedy" (default): first-come-first-served allocation; preserves current behavior.<br/>- "fair": divides available memory among spillable operators and may spill earlier.<br/>Only effective when `memory_pool_size` is bounded (>0). |
 | `query.experimental_spill_mode` | String | `default` | Spill mode:<br/>- "default": preserve DataFusion built-in OS temp directory (default).<br/>- "custom": explicitly configure spill path, quota, and compression.<br/>- "disabled": explicitly disable disk spilling.<br/>Set this to "custom" before using the path/quota/compression keys below. |
 | `query.experimental_spill_path` | String | Unset | Spill directory path. Ignored unless mode is "custom". |
@@ -151,6 +154,7 @@
 | `storage` | -- | -- | The data storage options. |
 | `storage.data_home` | String | `./greptimedb_data` | The working home directory. |
 | `storage.copy_root` | String | `./greptimedb_data/copy` | Root directory for standalone SQL access to local files.<br/>Relative SQL paths are resolved below this directory. Absolute paths are accepted only when<br/>they are inside this directory. Defaults to `<data_home>/copy`.<br/>Distributed deployments always reject SQL access to local files.<br/>Upgrade note: COPY commands and existing external tables that reference paths outside this<br/>directory will fail. Move those files below the copy root, set this option to a dedicated<br/>directory containing them, or migrate the files to object storage before upgrading. |
+| `storage.probe_on_startup` | Bool | `true` | Whether to probe every remote object storage when the datanode starts.<br/>The probe writes, reads, overwrites, lists and deletes an object under<br/>`.greptime_probe/`, and a failure aborts startup. The probe lives at the storage<br/>root, so the credentials must allow writes and deletes below the root. Disable it<br/>when the credentials are scoped to narrower prefixes or when the storage may be<br/>unavailable at startup. |
 | `storage.type` | String | `File` | The storage type used to store the data.<br/>- `File`: the data is stored in the local file system.<br/>- `S3`: the data is stored in the S3 object storage.<br/>- `Gcs`: the data is stored in the Google Cloud Storage.<br/>- `Azblob`: the data is stored in the Azure Blob Storage.<br/>- `Oss`: the data is stored in the Aliyun OSS.<br/>- `Hdfs`: the data is stored in the Hadoop Distributed File System. |
 | `storage.bucket` | String | Unset | The S3 bucket name.<br/>**It's only used when the storage type is `S3`, `Oss` and `Gcs`**. |
 | `storage.root` | String | Unset | The directory or object prefix under which data is stored.<br/>**It's only used when the storage type is `S3`, `Oss`, `Gcs`, `Azblob` and `Hdfs`**. |
@@ -174,7 +178,7 @@
 | `storage.http_client.timeout` | String | `30s` | The total request timeout, applied from when the request starts connecting until the response body has finished.<br/>Also considered a total deadline. |
 | `storage.http_client.pool_idle_timeout` | String | `90s` | The timeout for idle sockets being kept-alive. |
 | `storage.http_client.skip_ssl_validation` | Bool | `false` | To skip the ssl verification<br/>**Security Notice**: Setting `skip_ssl_validation = true` disables certificate verification, making connections vulnerable to man-in-the-middle attacks. Only use this in development or trusted private networks. |
-| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.<br/>Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail. |
+| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.<br/>Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail.<br/>The default configuration lists `mito` and `file`; `metric` is always registered, so naming<br/>it only supplies its options rather than enabling the engine. |
 | `region_engine.mito` | -- | -- | The Mito engine options. |
 | `region_engine.mito.num_workers` | Integer | Auto | Number of region workers. Defaults to half of the available CPU cores, rounded up. |
 | `region_engine.mito.worker_channel_size` | Integer | `128` | Request channel size of each worker. |
@@ -188,10 +192,11 @@
 | `region_engine.mito.experimental_series_index_maintenance_interval` | String | `5m` | Interval between series-index maintenance runs. Zero uses the default of 5 min. |
 | `region_engine.mito.experimental_series_index_bucket_width` | String | `5days` | Requested minimum series-index bucket width (default: 5 days), rounded up to<br/>an exact multiple of each region's compaction time window. |
 | `region_engine.mito.max_background_flushes` | Integer | Auto | Max number of running background flush jobs (default: 1/2 of cpu cores). |
-| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores). |
+| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores).<br/>The same limit bounds engine-wide admission of local compaction units and<br/>the merge-output budget of each automatic per-region pick. Outputs sharing<br/>inputs form one unit. Expired SSTs may add one expiration-only unit outside<br/>the merge-output budget; it still consumes a slot in the same admission limit.<br/>Manual compaction keeps using the request's parallelism. |
 | `region_engine.mito.max_background_purges` | Integer | Auto | Max number of running background purge jobs (default: number of cpu cores). |
 | `region_engine.mito.experimental_compaction_memory_limit` | String | 0 | Memory budget for compaction tasks.<br/>Supports absolute size (e.g., "2GiB", "512MB") or percentage of system memory (e.g., "50%").<br/>Setting it to 0 or "unlimited" disables the limit. |
 | `region_engine.mito.experimental_compaction_on_exhausted` | String | wait | Behavior when compaction cannot acquire memory from the budget.<br/>Options: "wait" (default, 10s), "wait(<duration>)", "fail" |
+| `region_engine.mito.experimental_compaction_output_parallelism` | Integer | 1 | Default per-output parallelism for input decoding and SST column encoding.<br/>Each stage is bounded independently; this is not a global thread limit.<br/>Values below 2 disable parallel decoding/encoding, but not the merge pipeline.<br/>Tables can override it with 'compaction.experimental_output_parallelism' (positive integer). |
 | `region_engine.mito.auto_flush_interval` | String | `10m` | Interval to auto flush a region if it has not flushed yet. |
 | `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
 | `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size`. |
@@ -211,10 +216,11 @@
 | `region_engine.mito.enable_refill_cache_on_read` | Bool | `true` | Enable refilling cache on read operations (default: true).<br/>When disabled, cache refilling on read won't happen. |
 | `region_engine.mito.manifest_cache_size` | String | `256MB` | Capacity for manifest cache (default: 256MB). |
 | `region_engine.mito.sst_write_buffer_size` | String | `8MB` | Buffer size for SST writing. |
-| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently. |
+| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently when the memory limit is disabled<br/>or a scan includes extension sources not covered by row-group accounting. |
 | `region_engine.mito.allow_stale_entries` | Bool | `false` | Whether to allow stale WAL entries read during replay. |
-| `region_engine.mito.scan_memory_limit` | String | `unlimited` | Memory limit for table scans across all queries.<br/>Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 or "unlimited" disables the limit. |
-| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Controls what happens when a scan cannot get memory immediately.<br/>"fail" (default) fails fast and is the recommended option for most users.<br/>"wait" / "wait(<duration>)" waits for memory to become available. This is mainly<br/>for advanced tuning in bursty workloads where temporary contention is common and<br/>higher latency is acceptable.<br/>"wait" means "wait(10s)", not unlimited waiting. |
+| `region_engine.mito.scan_memory_limit` | String | `25%` | Memory limit for table scans across all queries and regions.<br/>Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available.<br/>Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 or "unlimited" disables the limit and restores file-count checks. |
+| `region_engine.mito.scan_memory_mode` | String | `basic` | Memory accounting mode: "basic" (default) counts projected Parquet columns'<br/>uncompressed row-group sizes while decoding, including files from older versions.<br/>"extended" also counts cumulative streamed batches until stream drop.<br/>Limited reservations round up to KB; unlimited accounting is byte-exact.<br/>Internal series merges use query.memory_pool_size in both modes, sharing capacity<br/>with other query operators on the executing node. Neither budget measures total process memory. |
+| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Scan memory exhaustion always fails immediately.<br/>Legacy "wait" / "wait(<duration>)" settings are deprecated. They are accepted<br/>for upgrade compatibility, but startup logs a warning and treats them as "fail".<br/>Change existing wait settings to "fail"; scans no longer wait for quota to free up.<br/>This policy applies only to scan reservations. Series merges use the query pool<br/>and fail immediately when that pool is exhausted. |
 | `region_engine.mito.min_compaction_interval` | String | `0m` | Minimum time interval between two compactions.<br/>To align with the old behavior, the default value is 0 (no restrictions). |
 | `region_engine.mito.schedule_compaction_after_edit` | Bool | `true` | Whether to allow to schedule a compaction after a successful region edit.<br/><br/>Setting this to "true" is a necessary but not sufficient condition for scheduling compaction after a region edit.<br/>Other constraints, such as "min_compaction_interval", may still prevent compaction from being scheduled.<br/>Setting this to "false", however, guarantees that compaction will not be scheduled after a region edit. |
 | `region_engine.mito.default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. |
@@ -223,6 +229,7 @@
 | `region_engine.mito.index.aux_path` | String | `""` | Auxiliary directory path for the index in filesystem, used to store intermediate files for<br/>creating the index and staging files for searching the index, defaults to `{data_home}/index_intermediate`.<br/>The default name for this directory is `index_intermediate` for backward compatibility.<br/><br/>This path contains two subdirectories:<br/>- `__intm`: for storing intermediate files used during creating index.<br/>- `staging`: for storing staging files used during searching index. |
 | `region_engine.mito.index.staging_size` | String | `2GB` | The max capacity of the staging directory. |
 | `region_engine.mito.index.staging_ttl` | String | `7d` | The TTL of the staging directory.<br/>Defaults to 7 days.<br/>Setting it to "0s" to disable TTL. |
+| `region_engine.mito.index.build_mode` | String | `sync` | When to build indexes for new SST files.<br/>- `sync`: build indexes while writing SST files during flush and compaction.<br/>- `async`: build indexes in the background after the SST files are written. Queries read<br/>  these files without indexes until the build finishes. |
 | `region_engine.mito.index.metadata_cache_size` | String | `64MiB` | Cache size for inverted index metadata. |
 | `region_engine.mito.index.content_cache_size` | String | `128MiB` | Cache size for inverted index content. |
 | `region_engine.mito.index.content_cache_page_size` | String | `64KiB` | Page size for inverted index content cache. |
@@ -262,7 +269,7 @@
 | `logging.tracing_sample_ratio.default_ratio` | Float | `1.0` | -- |
 | `slow_query` | -- | -- | The slow query log options. |
 | `slow_query.enable` | Bool | `true` | Whether to enable slow query log. |
-| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.<br/>If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.<br/>If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`. |
+| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.<br/>If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.<br/>If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled,<br/>and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty.<br/>Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings. |
 | `slow_query.threshold` | String | `30s` | The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`. |
 | `slow_query.sample_ratio` | Float | `1.0` | The sampling ratio of slow query log. The value should be in the range of (0, 1]. For example, `0.1` means 10% of the slow queries will be logged and `1.0` means all slow queries will be logged. |
 | `slow_query.ttl` | String | `90d` | The TTL of the `slow_queries` system table. Default is `90d` when `record_type` is `system_table`. |
@@ -270,6 +277,8 @@
 | `tracing.tokio_console_addr` | String | Unset | The tokio console address. |
 | `pipeline` | -- | -- | The pipeline options. |
 | `pipeline.cache_ttl` | String | `10s` | Time to live of the local pipeline cache. Default is `10s`. |
+| `pipeline.allow_vrl_host_access` | Bool | `false` | Allow VRL to access the server environment, filesystem, hostname, local timezone,<br/>network, and logs. Default is false. Enable only for trusted pipeline authors.<br/>Applies to pipeline creation, execution, and dry-runs. Requires a restart. |
+| `pipeline.ignore_errors` | Bool | `false` | Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter.<br/>Explicit request values override this setting. Malformed input may be discarded,<br/>and invalid custom timestamps in identity ingestion fall back to server time.<br/>Pipeline execution errors are controlled separately by `skip_error`.<br/>Default is false. Requires a restart. |
 | `event_recorder` | -- | -- | Configuration options for the event recorder. |
 | `event_recorder.ttl` | String | `90d` | TTL for the events table that will be used to store the events. Default is `90d`. |
 | `event_recorder.event_types` | Array | All | Event types to record. Current available event types: `create_database`,<br/>`alter_database`, `drop_database`, `create_flow`, `drop_flow`,<br/>`create_table`, `create_logical_tables`, `alter_table`, `alter_logical_tables`,<br/>`drop_table`, `undrop_table`, `purge_dropped_table`, `truncate_table`,<br/>`create_view`, `drop_view`, `admin_function`, `reconcile_catalog`,<br/>`reconcile_database`, `reconcile_table`, `reconcile_logical_tables`.<br/>When omitted, all current and future event types are recorded.<br/>Set to an empty array to disable event recording. |
@@ -357,7 +366,7 @@
 | `pending_rows_batcher.worker_channel_capacity` | Integer | `65536` | Maximum queued submissions per table worker. |
 | `pending_rows_batcher.max_inflight_requests` | Integer | `3000` | Maximum admitted original requests awaiting completion. |
 | `pending_rows_batcher.flow_notification_queue_capacity` | Integer | `1024` | Maximum number of queued table Flow notifications. |
-| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics.<br/>Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible.<br/>Enable independently with protocols and a nonzero flush interval.<br/>Omitted fields use independent defaults, not parent settings.<br/>Empty protocols or a zero interval disables logical batching without fallback.<br/>Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. |
+| `pending_rows_batcher.logical_table` | -- | -- | Metric-engine logical-table batching for Prom remote write and OTLP metrics.<br/>Requires prom_store.with_metric_engine. Logs and traces are not eligible.<br/>Enable independently with protocols and a nonzero flush interval.<br/>Omitted fields use independent defaults, not parent settings.<br/>Empty protocols or a zero interval disables logical batching without fallback.<br/>Omitting this entire section preserves legacy Prom batching; it does not enable OTLP batching. |
 | `pending_rows_batcher.logical_table.pending_rows_flush_interval` | String | `0s` | Flush interval measured from the first pending submission. Zero disables batching. |
 | `pending_rows_batcher.logical_table.max_batch_rows` | Integer | `100000` | Flush after a complete submission reaches this row threshold. |
 | `pending_rows_batcher.logical_table.max_concurrent_flushes` | Integer | `256` | Maximum concurrent flushes shared by Prom and OTLP metrics. |
@@ -388,7 +397,7 @@
 | `query.experimental_native_promql` | Bool | `false` | Enable experimental native PromQL execution for compatible standalone queries.<br/>Requires in-process region transport and the per-request<br/>query.experimental_native_promql=true extension. Default: false |
 | `query.parallelism` | Integer | `0` | Parallelism of the query engine.<br/>Default to 0, which means the number of CPU cores. |
 | `query.allow_query_fallback` | Bool | `false` | Whether to allow query fallback when push down optimize fails.<br/>Default to false, meaning when push down optimize failed, return error msg |
-| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).<br/>Supports absolute size (e.g., "4GB", "8GB") or percentage of system memory (e.g., "30%").<br/>Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.<br/>When this limit is reached, queries will fail with ResourceExhausted error.<br/>NOTE: This does NOT limit memory used by table scans (only applies to datanodes). |
+| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).<br/>Supports absolute size (e.g., "4GB", "8GB") or percentage of system memory (e.g., "30%").<br/>Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.<br/>When this limit is reached, queries will fail with ResourceExhausted error.<br/>This pool applies to operators on this frontend. Datanodes use their own query<br/>pools for local operators and internal series merges during table scans. |
 | `query.experimental_memory_pool_policy` | String | `greedy` | Experimental memory pool allocation policy:<br/>- "greedy" (default): first-come-first-served allocation; preserves current behavior.<br/>- "fair": divides available memory among spillable operators and may spill earlier.<br/>Only effective when `memory_pool_size` is bounded (>0). |
 | `query.experimental_spill_mode` | String | `default` | Spill mode:<br/>- "default": preserve DataFusion built-in OS temp directory (default).<br/>- "custom": explicitly configure spill path, quota, and compression.<br/>- "disabled": explicitly disable disk spilling.<br/>Set this to "custom" before using the path/quota/compression keys below. |
 | `query.experimental_spill_path` | String | Unset | Spill directory path. Ignored unless mode is "custom". |
@@ -422,7 +431,7 @@
 | `logging.tracing_sample_ratio.default_ratio` | Float | `1.0` | -- |
 | `slow_query` | -- | -- | The slow query log options. |
 | `slow_query.enable` | Bool | `true` | Whether to enable slow query log. |
-| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.<br/>If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.<br/>If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`. |
+| `slow_query.record_type` | String | `system_table` | The record type of slow queries. It can be `system_table` or `log`.<br/>If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.<br/>If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled,<br/>and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty.<br/>Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings. |
 | `slow_query.threshold` | String | `30s` | The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`. |
 | `slow_query.sample_ratio` | Float | `1.0` | The sampling ratio of slow query log. The value should be in the range of (0, 1]. For example, `0.1` means 10% of the slow queries will be logged and `1.0` means all slow queries will be logged. |
 | `slow_query.ttl` | String | `90d` | The TTL of the `slow_queries` system table. Default is `90d` when `record_type` is `system_table`. |
@@ -432,6 +441,8 @@
 | `memory.enable_heap_profiling` | Bool | `true` | Whether to enable heap profiling activation during startup.<br/>When enabled, heap profiling will be activated if the `MALLOC_CONF` environment variable<br/>is set to "prof:true,prof_active:false". The official image adds this env variable.<br/>Default is true. |
 | `pipeline` | -- | -- | The pipeline options. |
 | `pipeline.cache_ttl` | String | `10s` | Time to live of the frontend-local pipeline cache. A pipeline created or deleted on<br/>another frontend takes effect on this one after at most this duration. |
+| `pipeline.allow_vrl_host_access` | Bool | `false` | Allow VRL to access the server environment, filesystem, hostname, local timezone,<br/>network, and logs. Default is false. Enable only for trusted pipeline authors.<br/>Applies to pipeline creation, execution, and dry-runs. Requires a restart. |
+| `pipeline.ignore_errors` | Bool | `false` | Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter.<br/>Explicit request values override this setting. Malformed input may be discarded,<br/>and invalid custom timestamps in identity ingestion fall back to server time.<br/>Pipeline execution errors are controlled separately by `skip_error`.<br/>Default is false. Requires a restart. |
 | `event_recorder` | -- | -- | Configuration options for the event recorder. |
 | `event_recorder.ttl` | String | `90d` | TTL for the events table that will be used to store the events. Default is `90d`. |
 | `event_recorder.event_types` | Array | All | Event types to record. Current available event type: `admin_function`.<br/>When omitted, all current and future event types are recorded.<br/>Set to an empty array to disable event recording. |
@@ -615,9 +626,10 @@
 | `wal.max_unpersisted_bytes` | String | `64MB` | The size of the unpersisted backlog at which new appends stall until an upload completes, in `enqueued` mode.<br/>Nothing is dropped and nothing is rejected; the threshold does not bound what a crash during an outage can lose.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.max_unpersisted_age` | String | `8s` | The age of the oldest unpersisted entry at which new appends stall until an upload completes, in `enqueued` mode.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.on_corrupted_segment` | String | `skip` | What a read does with a segment that still does not decode after a second fetch, because its checksum does not match or its content disagrees with its footer entry.<br/>- `skip`: the segment is skipped and recorded as a WAL hole of its region, a metric is incremented and a warning is logged; the other regions of the object are unaffected (the default).<br/>- `fail`: the read fails, so the region does not open.<br/>**It's only used when the provider is `experimental_object_store`**. |
+| `wal.accept_missing_history` | Bool | `false` | Whether a region opens although the WAL objects under the prefix do not hold its history.<br/>An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.<br/>To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.<br/>When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.<br/>**It's only used when the provider is `experimental_object_store`**. |
 | `query` | -- | -- | The query engine options. |
 | `query.parallelism` | Integer | `0` | Parallelism of the query engine.<br/>Default to 0, which means the number of CPU cores. |
-| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).<br/>Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.<br/>When this limit is reached, queries will fail with ResourceExhausted error.<br/>NOTE: This does NOT limit memory used by table scans. |
+| `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join,<br/>and internal series merges during table scans).<br/>Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.<br/>When this limit is reached, queries will fail with ResourceExhausted error.<br/>Projected Parquet row groups and streamed scan batches use scan_memory_limit separately. |
 | `query.experimental_memory_pool_policy` | String | `greedy` | Experimental memory pool allocation policy:<br/>- "greedy" (default): first-come-first-served allocation; preserves current behavior.<br/>- "fair": divides available memory among spillable operators and may spill earlier.<br/>Only effective when `memory_pool_size` is bounded (>0). |
 | `query.experimental_spill_mode` | String | `default` | Spill mode:<br/>- "default": preserve DataFusion built-in OS temp directory (default).<br/>- "custom": explicitly configure spill path, quota, and compression.<br/>- "disabled": explicitly disable disk spilling.<br/>Set this to "custom" before using the path/quota/compression keys below. |
 | `query.experimental_spill_path` | String | Unset | Spill directory path. Ignored unless mode is "custom". |
@@ -625,6 +637,7 @@
 | `query.experimental_spill_compression` | String | `uncompressed` | Compression for spilled data files: "uncompressed" (default), "lz4_frame", "zstd".<br/>Ignored unless mode is "custom". |
 | `storage` | -- | -- | The data storage options. |
 | `storage.data_home` | String | `./greptimedb_data` | The working home directory. |
+| `storage.probe_on_startup` | Bool | `true` | Whether to probe every remote object storage when the datanode starts.<br/>The probe writes, reads, overwrites, lists and deletes an object under<br/>`.greptime_probe/`, and a failure aborts startup. The probe lives at the storage<br/>root, so the credentials must allow writes and deletes below the root. Disable it<br/>when the credentials are scoped to narrower prefixes or when the storage may be<br/>unavailable at startup. |
 | `storage.type` | String | `File` | The storage type used to store the data.<br/>- `File`: the data is stored in the local file system.<br/>- `S3`: the data is stored in the S3 object storage.<br/>- `Gcs`: the data is stored in the Google Cloud Storage.<br/>- `Azblob`: the data is stored in the Azure Blob Storage.<br/>- `Oss`: the data is stored in the Aliyun OSS.<br/>- `Hdfs`: the data is stored in the Hadoop Distributed File System. |
 | `storage.bucket` | String | Unset | The S3 bucket name.<br/>**It's only used when the storage type is `S3`, `Oss` and `Gcs`**. |
 | `storage.root` | String | Unset | The directory or object prefix under which data is stored.<br/>**It's only used when the storage type is `S3`, `Oss`, `Gcs`, `Azblob` and `Hdfs`**. |
@@ -648,7 +661,7 @@
 | `storage.http_client.timeout` | String | `30s` | The total request timeout, applied from when the request starts connecting until the response body has finished.<br/>Also considered a total deadline. |
 | `storage.http_client.pool_idle_timeout` | String | `90s` | The timeout for idle sockets being kept-alive. |
 | `storage.http_client.skip_ssl_validation` | Bool | `false` | To skip the ssl verification<br/>**Security Notice**: Setting `skip_ssl_validation = true` disables certificate verification, making connections vulnerable to man-in-the-middle attacks. Only use this in development or trusted private networks. |
-| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.<br/>Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail. |
+| `[[region_engine]]` | -- | -- | The region engine options. You can configure multiple region engines.<br/>Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail.<br/>The default configuration lists `mito` and `file`; `metric` is always registered, so naming<br/>it only supplies its options rather than enabling the engine. |
 | `region_engine.mito` | -- | -- | The Mito engine options. |
 | `region_engine.mito.num_workers` | Integer | Auto | Number of region workers. Defaults to half of the available CPU cores, rounded up. |
 | `region_engine.mito.worker_channel_size` | Integer | `128` | Request channel size of each worker. |
@@ -664,10 +677,11 @@
 | `region_engine.mito.experimental_series_index_maintenance_interval` | String | `5m` | Interval between series-index maintenance runs. Zero uses the default of 5 min. |
 | `region_engine.mito.experimental_series_index_bucket_width` | String | `5days` | Requested minimum series-index bucket width (default: 5 days), rounded up to<br/>an exact multiple of each region's compaction time window. |
 | `region_engine.mito.max_background_flushes` | Integer | Auto | Max number of running background flush jobs (default: 1/2 of cpu cores). |
-| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores). |
+| `region_engine.mito.max_background_compactions` | Integer | Auto | Max number of running background compaction jobs (default: 1/4 of cpu cores).<br/>The same limit bounds engine-wide admission of local compaction units and<br/>the merge-output budget of each automatic per-region pick. Outputs sharing<br/>inputs form one unit. Expired SSTs may add one expiration-only unit outside<br/>the merge-output budget; it still consumes a slot in the same admission limit.<br/>Manual compaction keeps using the request's parallelism. |
 | `region_engine.mito.max_background_purges` | Integer | Auto | Max number of running background purge jobs (default: number of cpu cores). |
 | `region_engine.mito.experimental_compaction_memory_limit` | String | 0 | Memory budget for compaction tasks.<br/>Supports absolute size (e.g., "2GiB", "512MB") or percentage of system memory (e.g., "50%").<br/>Setting it to 0 or "unlimited" disables the limit. |
 | `region_engine.mito.experimental_compaction_on_exhausted` | String | wait | Behavior when compaction cannot acquire memory from the budget.<br/>Options: "wait" (default, 10s), "wait(<duration>)", "fail" |
+| `region_engine.mito.experimental_compaction_output_parallelism` | Integer | 1 | Default per-output parallelism for input decoding and SST column encoding.<br/>Each stage is bounded independently; this is not a global thread limit.<br/>Values below 2 disable parallel decoding/encoding, but not the merge pipeline.<br/>Tables can override it with 'compaction.experimental_output_parallelism' (positive integer). |
 | `region_engine.mito.auto_flush_interval` | String | `10m` | Interval to auto flush a region if it has not flushed yet. |
 | `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
 | `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` |
@@ -687,10 +701,11 @@
 | `region_engine.mito.enable_refill_cache_on_read` | Bool | `true` | Enable refilling cache on read operations (default: true).<br/>When disabled, cache refilling on read won't happen. |
 | `region_engine.mito.manifest_cache_size` | String | `256MB` | Capacity for manifest cache (default: 256MB). |
 | `region_engine.mito.sst_write_buffer_size` | String | `8MB` | Buffer size for SST writing. |
-| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently. |
+| `region_engine.mito.max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently when the memory limit is disabled<br/>or a scan includes extension sources not covered by row-group accounting. |
 | `region_engine.mito.allow_stale_entries` | Bool | `false` | Whether to allow stale WAL entries read during replay. |
-| `region_engine.mito.scan_memory_limit` | String | `unlimited` | Memory limit for table scans across all queries.<br/>Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 or "unlimited" disables the limit. |
-| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Controls what happens when a scan cannot get memory immediately.<br/>"fail" (default) fails fast and is the recommended option for most users.<br/>"wait" / "wait(<duration>)" waits for memory to become available. This is mainly<br/>for advanced tuning in bursty workloads where temporary contention is common and<br/>higher latency is acceptable.<br/>"wait" means "wait(10s)", not unlimited waiting. |
+| `region_engine.mito.scan_memory_limit` | String | `25%` | Memory limit for table scans across all queries and regions.<br/>Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available.<br/>Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 or "unlimited" disables the limit and restores file-count checks. |
+| `region_engine.mito.scan_memory_mode` | String | `basic` | Memory accounting mode: "basic" (default) counts projected Parquet columns'<br/>uncompressed row-group sizes while decoding, including files from older versions.<br/>"extended" also counts cumulative streamed batches until stream drop.<br/>Limited reservations round up to KB; unlimited accounting is byte-exact.<br/>Internal series merges use query.memory_pool_size in both modes, sharing capacity<br/>with other query operators on the executing node. Neither budget measures total process memory. |
+| `region_engine.mito.scan_memory_on_exhausted` | String | `fail` | Scan memory exhaustion always fails immediately.<br/>Legacy "wait" / "wait(<duration>)" settings are deprecated. They are accepted<br/>for upgrade compatibility, but startup logs a warning and treats them as "fail".<br/>Change existing wait settings to "fail"; scans no longer wait for quota to free up.<br/>This policy applies only to scan reservations. Series merges use the query pool<br/>and fail immediately when that pool is exhausted. |
 | `region_engine.mito.min_compaction_interval` | String | `0m` | Minimum time interval between two compactions.<br/>To align with the old behavior, the default value is 0 (no restrictions). |
 | `region_engine.mito.schedule_compaction_after_edit` | Bool | `true` | Whether to allow to schedule a compaction after a successful region edit.<br/><br/>Setting this to "true" is a necessary but not sufficient condition for scheduling compaction after a region edit.<br/>Other constraints, such as "min_compaction_interval", may still prevent compaction from being scheduled.<br/>Setting this to "false", however, guarantees that compaction will not be scheduled after a region edit. |
 | `region_engine.mito.default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. |
@@ -699,6 +714,7 @@
 | `region_engine.mito.index.aux_path` | String | `""` | Auxiliary directory path for the index in filesystem, used to store intermediate files for<br/>creating the index and staging files for searching the index, defaults to `{data_home}/index_intermediate`.<br/>The default name for this directory is `index_intermediate` for backward compatibility.<br/><br/>This path contains two subdirectories:<br/>- `__intm`: for storing intermediate files used during creating index.<br/>- `staging`: for storing staging files used during searching index. |
 | `region_engine.mito.index.staging_size` | String | `2GB` | The max capacity of the staging directory. |
 | `region_engine.mito.index.staging_ttl` | String | `7d` | The TTL of the staging directory.<br/>Defaults to 7 days.<br/>Setting it to "0s" to disable TTL. |
+| `region_engine.mito.index.build_mode` | String | `sync` | When to build indexes for new SST files.<br/>- `sync`: build indexes while writing SST files during flush and compaction.<br/>- `async`: build indexes in the background after the SST files are written. Queries read<br/>  these files without indexes until the build finishes. |
 | `region_engine.mito.index.metadata_cache_size` | String | `64MiB` | Cache size for inverted index metadata. |
 | `region_engine.mito.index.content_cache_size` | String | `128MiB` | Cache size for inverted index content. |
 | `region_engine.mito.index.content_cache_page_size` | String | `64KiB` | Page size for inverted index content cache. |
@@ -722,7 +738,7 @@
 | `region_engine.mito.gc` | -- | -- | -- |
 | `region_engine.mito.gc.enable` | Bool | `false` | Whether GC is enabled. Need to be the same with metasrv's `gc.enable` or unexpected behavior will occur |
 | `region_engine.mito.gc.lingering_time` | String | `1h` | Lingering time before deleting files.<br/>Should be long enough to allow long running queries to finish.<br/>If set to None, then unused files will be deleted immediately. |
-| `region_engine.mito.gc.unknown_file_lingering_time` | String | `1d` | Lingering time before deleting unknown files (files with undetermined expel time).<br/>Only applies during full file listing GC.<br/>This uses the object's last-modified timestamp as a heuristic (strict less-than comparison);<br/>do not configure this value too small in production to avoid deleting pre-manifest files<br/>from in-progress compaction or flush.<br/>For active/open regions, an unknown file is deleted only if its object last-modified time exceeds this TTL.<br/>If the object store does not provide a last-modified timestamp, the file is conservatively kept.<br/>For dropped regions, unknown files are deleted immediately. |
+| `region_engine.mito.gc.unknown_file_lingering_time` | String | `1d` | Lingering time before deleting unknown files (files with undetermined expel time).<br/>It is also the age threshold of the sweep that reclaims temporary files left behind by<br/>a crashed HDFS writer, and that sweep runs even if `enable` is false. HDFS advances the<br/>modification time of a file under construction only as its blocks complete, so lowering<br/>this value also shrinks the margin between the sweep and files that are still being written.<br/>This uses the object's last-modified timestamp as a heuristic (strict less-than comparison);<br/>do not configure this value too small in production to avoid deleting pre-manifest files<br/>from in-progress compaction or flush.<br/>For active/open regions, an unknown file is deleted only if its object last-modified time exceeds this TTL.<br/>If the object store does not provide a last-modified timestamp, the file is conservatively kept.<br/>For dropped regions, unknown files are deleted immediately. |
 | `region_engine.file` | -- | -- | Enable the file engine. |
 | `region_engine.metric` | -- | -- | Metric engine options. |
 | `logging` | -- | -- | The logging options. |

config/datanode.example.toml

@@ -258,6 +258,13 @@ overwrite_entry_start_id = false
 ## **It's only used when the provider is `experimental_object_store`**.
 #+ on_corrupted_segment = "skip"
 
+## Whether a region opens although the WAL objects under the prefix do not hold its history.
+## An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.
+## To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.
+## When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.
+## **It's only used when the provider is `experimental_object_store`**.
+#+ accept_missing_history = false
+
 # The Kafka SASL configuration.
 # **It's only used when the provider is `kafka`**.
 # Available SASL mechanisms:
@@ -331,11 +338,12 @@ overwrite_entry_start_id = false
 ## Default to 0, which means the number of CPU cores.
 parallelism = 0
 
-## Memory pool size for query execution operators (aggregation, sorting, join).
+## Memory pool size for query execution operators (aggregation, sorting, join,
+## and internal series merges during table scans).
 ## Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").
 ## Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
 ## When this limit is reached, queries will fail with ResourceExhausted error.
-## NOTE: This does NOT limit memory used by table scans.
+## Projected Parquet row groups and streamed scan batches use scan_memory_limit separately.
 ## @toml2docs:none-default="0"
 memory_pool_size = "50%"
 
@@ -368,6 +376,14 @@ memory_pool_size = "50%"
 ## The working home directory.
 data_home = "./greptimedb_data"
 
+## Whether to probe every remote object storage when the datanode starts.
+## The probe writes, reads, overwrites, lists and deletes an object under
+## `.greptime_probe/`, and a failure aborts startup. The probe lives at the storage
+## root, so the credentials must allow writes and deletes below the root. Disable it
+## when the credentials are scoped to narrower prefixes or when the storage may be
+## unavailable at startup.
+probe_on_startup = true
+
 ## The storage type used to store the data.
 ## - `File`: the data is stored in the local file system.
 ## - `S3`: the data is stored in the S3 object storage.
@@ -502,6 +518,8 @@ skip_ssl_validation = false
 
 ## The region engine options. You can configure multiple region engines.
 ## Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail.
+## The default configuration lists `mito` and `file`; `metric` is always registered, so naming
+## it only supplies its options rather than enabling the engine.
 [[region_engine]]
 
 ## The Mito engine options.
@@ -566,6 +584,11 @@ compress_manifest = false
 #+ max_background_flushes = 4
 
 ## Max number of running background compaction jobs (default: 1/4 of cpu cores).
+## The same limit bounds engine-wide admission of local compaction units and
+## the merge-output budget of each automatic per-region pick. Outputs sharing
+## inputs form one unit. Expired SSTs may add one expiration-only unit outside
+## the merge-output budget; it still consumes a slot in the same admission limit.
+## Manual compaction keeps using the request's parallelism.
 ## @toml2docs:none-default="Auto"
 #+ max_background_compactions = 2
 
@@ -584,6 +607,13 @@ compress_manifest = false
 ## @toml2docs:none-default="wait"
 #+ experimental_compaction_on_exhausted = "wait"
 
+## Default per-output parallelism for input decoding and SST column encoding.
+## Each stage is bounded independently; this is not a global thread limit.
+## Values below 2 disable parallel decoding/encoding, but not the merge pipeline.
+## Tables can override it with 'compaction.experimental_output_parallelism' (positive integer).
+## @toml2docs:none-default="1"
+#+ experimental_compaction_output_parallelism = 1
+
 ## Interval to auto flush a region if it has not flushed yet.
 auto_flush_interval = "10m"
 
@@ -662,22 +692,33 @@ manifest_cache_size = "256MB"
 ## Buffer size for SST writing.
 sst_write_buffer_size = "8MB"
 
-## Maximum number of SST files to scan concurrently.
+## Maximum number of SST files to scan concurrently when the memory limit is disabled
+## or a scan includes extension sources not covered by row-group accounting.
 max_concurrent_scan_files = 384
 
 ## Whether to allow stale WAL entries read during replay.
 allow_stale_entries = false
 
-## Memory limit for table scans across all queries.
+## Memory limit for table scans across all queries and regions.
+## Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available.
 ## Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").
-## Setting it to 0 or "unlimited" disables the limit.
-scan_memory_limit = "unlimited"
-## Controls what happens when a scan cannot get memory immediately.
-## "fail" (default) fails fast and is the recommended option for most users.
-## "wait" / "wait(<duration>)" waits for memory to become available. This is mainly
-## for advanced tuning in bursty workloads where temporary contention is common and
-## higher latency is acceptable.
-## "wait" means "wait(10s)", not unlimited waiting.
+## Setting it to 0 or "unlimited" disables the limit and restores file-count checks.
+scan_memory_limit = "25%"
+
+## Memory accounting mode: "basic" (default) counts projected Parquet columns'
+## uncompressed row-group sizes while decoding, including files from older versions.
+## "extended" also counts cumulative streamed batches until stream drop.
+## Limited reservations round up to KB; unlimited accounting is byte-exact.
+## Internal series merges use query.memory_pool_size in both modes, sharing capacity
+## with other query operators on the executing node. Neither budget measures total process memory.
+scan_memory_mode = "basic"
+
+## Scan memory exhaustion always fails immediately.
+## Legacy "wait" / "wait(<duration>)" settings are deprecated. They are accepted
+## for upgrade compatibility, but startup logs a warning and treats them as "fail".
+## Change existing wait settings to "fail"; scans no longer wait for quota to free up.
+## This policy applies only to scan reservations. Series merges use the query pool
+## and fail immediately when that pool is exhausted.
 scan_memory_on_exhausted = "fail"
 
 ## Minimum time interval between two compactions.
@@ -717,6 +758,12 @@ staging_size = "2GB"
 ## Setting it to "0s" to disable TTL.
 staging_ttl = "7d"
 
+## When to build indexes for new SST files.
+## - `sync`: build indexes while writing SST files during flush and compaction.
+## - `async`: build indexes in the background after the SST files are written. Queries read
+##   these files without indexes until the build finishes.
+build_mode = "sync"
+
 ## Cache size for inverted index metadata.
 metadata_cache_size = "64MiB"
 
@@ -814,7 +861,10 @@ enable = false
 lingering_time = "1h"
 
 ## Lingering time before deleting unknown files (files with undetermined expel time).
-## Only applies during full file listing GC.
+## It is also the age threshold of the sweep that reclaims temporary files left behind by
+## a crashed HDFS writer, and that sweep runs even if `enable` is false. HDFS advances the
+## modification time of a file under construction only as its blocks complete, so lowering
+## this value also shrinks the margin between the sweep and files that are still being written.
 ## This uses the object's last-modified timestamp as a heuristic (strict less-than comparison);
 ## do not configure this value too small in production to avoid deleting pre-manifest files
 ## from in-progress compaction or flush.

config/frontend.example.toml

@@ -274,8 +274,8 @@ max_inflight_requests = 3000
 ## Maximum number of queued table Flow notifications.
 flow_notification_queue_capacity = 1024
 
-## Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics.
-## Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible.
+## Metric-engine logical-table batching for Prom remote write and OTLP metrics.
+## Requires prom_store.with_metric_engine. Logs and traces are not eligible.
 ## Enable independently with protocols and a nonzero flush interval.
 ## Omitted fields use independent defaults, not parent settings.
 ## Empty protocols or a zero interval disables logical batching without fallback.
@@ -375,7 +375,8 @@ allow_query_fallback = false
 ## Supports absolute size (e.g., "4GB", "8GB") or percentage of system memory (e.g., "30%").
 ## Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
 ## When this limit is reached, queries will fail with ResourceExhausted error.
-## NOTE: This does NOT limit memory used by table scans (only applies to datanodes).
+## This pool applies to operators on this frontend. Datanodes use their own query
+## pools for local operators and internal series merges during table scans.
 ## @toml2docs:none-default="0"
 memory_pool_size = "50%"
 
@@ -486,7 +487,9 @@ enable = true
 
 ## The record type of slow queries. It can be `system_table` or `log`.
 ## If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.
-## If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`.
+## If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled,
+## and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty.
+## Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings.
 record_type = "system_table"
 
 ## The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`.
@@ -517,6 +520,16 @@ enable_heap_profiling = true
 ## Time to live of the frontend-local pipeline cache. A pipeline created or deleted on
 ## another frontend takes effect on this one after at most this duration.
 cache_ttl = "10s"
+## Allow VRL to access the server environment, filesystem, hostname, local timezone,
+## network, and logs. Default is false. Enable only for trusted pipeline authors.
+## Applies to pipeline creation, execution, and dry-runs. Requires a restart.
+allow_vrl_host_access = false
+## Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter.
+## Explicit request values override this setting. Malformed input may be discarded,
+## and invalid custom timestamps in identity ingestion fall back to server time.
+## Pipeline execution errors are controlled separately by `skip_error`.
+## Default is false. Requires a restart.
+ignore_errors = false
 
 ## Configuration options for the event recorder.
 [event_recorder]

config/standalone.example.toml

@@ -43,6 +43,14 @@ init_regions_in_background = false
 ## Parallelism of initializing regions.
 init_regions_parallelism = 16
 
+## The maximum concurrent region-read streams. Zero means unlimited.
+## A single query can consume multiple permits.
+max_concurrent_queries = 0
+
+## Timeout to acquire a permit when `max_concurrent_queries` is reached.
+## Only effective when the limiter is enabled.
+concurrent_query_limiter_timeout = "100ms"
+
 ## Enable telemetry to collect anonymous usage data. Enabled by default.
 #+ enable_telemetry = true
 
@@ -246,8 +254,8 @@ max_inflight_requests = 3000
 ## Maximum number of queued table Flow notifications.
 flow_notification_queue_capacity = 1024
 
-## Metric-engine logical-table batching for Prom remote write and non-legacy OTLP metrics.
-## Requires prom_store.with_metric_engine. Logs, traces and legacy metrics are not eligible.
+## Metric-engine logical-table batching for Prom remote write and OTLP metrics.
+## Requires prom_store.with_metric_engine. Logs and traces are not eligible.
 ## Enable independently with protocols and a nonzero flush interval.
 ## Omitted fields use independent defaults, not parent settings.
 ## Empty protocols or a zero interval disables logical batching without fallback.
@@ -449,6 +457,13 @@ overwrite_entry_start_id = false
 ## **It's only used when the provider is `experimental_object_store`**.
 #+ on_corrupted_segment = "skip"
 
+## Whether a region opens although the WAL objects under the prefix do not hold its history.
+## An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.
+## To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.
+## When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.
+## **It's only used when the provider is `experimental_object_store`**.
+#+ accept_missing_history = false
+
 # The Kafka SASL configuration.
 # **It's only used when the provider is `kafka`**.
 # Available SASL mechanisms:
@@ -546,11 +561,12 @@ max_running_procedures = 128
 ## Default to 0, which means the number of CPU cores.
 parallelism = 0
 
-## Memory pool size for query execution operators (aggregation, sorting, join).
+## Memory pool size for query execution operators (aggregation, sorting, join,
+## and internal series merges during table scans).
 ## Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").
 ## Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.
 ## When this limit is reached, queries will fail with ResourceExhausted error.
-## NOTE: This does NOT limit memory used by table scans.
+## Projected Parquet row groups and streamed scan batches use scan_memory_limit separately.
 ## @toml2docs:none-default="0"
 memory_pool_size = "50%"
 
@@ -592,6 +608,14 @@ data_home = "./greptimedb_data"
 ## directory containing them, or migrate the files to object storage before upgrading.
 #+ copy_root = "./greptimedb_data/copy"
 
+## Whether to probe every remote object storage when the datanode starts.
+## The probe writes, reads, overwrites, lists and deletes an object under
+## `.greptime_probe/`, and a failure aborts startup. The probe lives at the storage
+## root, so the credentials must allow writes and deletes below the root. Disable it
+## when the credentials are scoped to narrower prefixes or when the storage may be
+## unavailable at startup.
+probe_on_startup = true
+
 ## The storage type used to store the data.
 ## - `File`: the data is stored in the local file system.
 ## - `S3`: the data is stored in the S3 object storage.
@@ -726,6 +750,8 @@ skip_ssl_validation = false
 
 ## The region engine options. You can configure multiple region engines.
 ## Each engine type (mito, file, metric) may appear only once; duplicates cause startup to fail.
+## The default configuration lists `mito` and `file`; `metric` is always registered, so naming
+## it only supplies its options rather than enabling the engine.
 [[region_engine]]
 
 ## The Mito engine options.
@@ -777,6 +803,11 @@ compress_manifest = false
 #+ max_background_flushes = 4
 
 ## Max number of running background compaction jobs (default: 1/4 of cpu cores).
+## The same limit bounds engine-wide admission of local compaction units and
+## the merge-output budget of each automatic per-region pick. Outputs sharing
+## inputs form one unit. Expired SSTs may add one expiration-only unit outside
+## the merge-output budget; it still consumes a slot in the same admission limit.
+## Manual compaction keeps using the request's parallelism.
 ## @toml2docs:none-default="Auto"
 #+ max_background_compactions = 2
 
@@ -795,6 +826,13 @@ compress_manifest = false
 ## @toml2docs:none-default="wait"
 #+ experimental_compaction_on_exhausted = "wait"
 
+## Default per-output parallelism for input decoding and SST column encoding.
+## Each stage is bounded independently; this is not a global thread limit.
+## Values below 2 disable parallel decoding/encoding, but not the merge pipeline.
+## Tables can override it with 'compaction.experimental_output_parallelism' (positive integer).
+## @toml2docs:none-default="1"
+#+ experimental_compaction_output_parallelism = 1
+
 ## Interval to auto flush a region if it has not flushed yet.
 auto_flush_interval = "10m"
 
@@ -873,22 +911,33 @@ manifest_cache_size = "256MB"
 ## Buffer size for SST writing.
 sst_write_buffer_size = "8MB"
 
-## Maximum number of SST files to scan concurrently.
+## Maximum number of SST files to scan concurrently when the memory limit is disabled
+## or a scan includes extension sources not covered by row-group accounting.
 max_concurrent_scan_files = 384
 
 ## Whether to allow stale WAL entries read during replay.
 allow_stale_entries = false
 
-## Memory limit for table scans across all queries.
+## Memory limit for table scans across all queries and regions.
+## Defaults to 25% of cgroup memory, or host memory when no cgroup limit is available.
 ## Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%").
-## Setting it to 0 or "unlimited" disables the limit.
-scan_memory_limit = "unlimited"
-## Controls what happens when a scan cannot get memory immediately.
-## "fail" (default) fails fast and is the recommended option for most users.
-## "wait" / "wait(<duration>)" waits for memory to become available. This is mainly
-## for advanced tuning in bursty workloads where temporary contention is common and
-## higher latency is acceptable.
-## "wait" means "wait(10s)", not unlimited waiting.
+## Setting it to 0 or "unlimited" disables the limit and restores file-count checks.
+scan_memory_limit = "25%"
+
+## Memory accounting mode: "basic" (default) counts projected Parquet columns'
+## uncompressed row-group sizes while decoding, including files from older versions.
+## "extended" also counts cumulative streamed batches until stream drop.
+## Limited reservations round up to KB; unlimited accounting is byte-exact.
+## Internal series merges use query.memory_pool_size in both modes, sharing capacity
+## with other query operators on the executing node. Neither budget measures total process memory.
+scan_memory_mode = "basic"
+
+## Scan memory exhaustion always fails immediately.
+## Legacy "wait" / "wait(<duration>)" settings are deprecated. They are accepted
+## for upgrade compatibility, but startup logs a warning and treats them as "fail".
+## Change existing wait settings to "fail"; scans no longer wait for quota to free up.
+## This policy applies only to scan reservations. Series merges use the query pool
+## and fail immediately when that pool is exhausted.
 scan_memory_on_exhausted = "fail"
 
 ## Minimum time interval between two compactions.
@@ -928,6 +977,12 @@ staging_size = "2GB"
 ## Setting it to "0s" to disable TTL.
 staging_ttl = "7d"
 
+## When to build indexes for new SST files.
+## - `sync`: build indexes while writing SST files during flush and compaction.
+## - `async`: build indexes in the background after the SST files are written. Queries read
+##   these files without indexes until the build finishes.
+build_mode = "sync"
+
 ## Cache size for inverted index metadata.
 metadata_cache_size = "64MiB"
 
@@ -1081,7 +1136,9 @@ enable = true
 
 ## The record type of slow queries. It can be `system_table` or `log`.
 ## If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.
-## If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`.
+## If `log` is selected, slow queries are logged to stdout when `logging.append_stdout` is enabled,
+## and to `greptimedb-slow-queries.*` when `logging.enable_file_logging` is enabled and `logging.dir` is nonempty.
+## Slow query logging is independent of `logging.level` and uses `logging.log_format` and the logging retention settings.
 record_type = "system_table"
 
 ## The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`.
@@ -1103,6 +1160,16 @@ ttl = "90d"
 [pipeline]
 ## Time to live of the local pipeline cache. Default is `10s`.
 cache_ttl = "10s"
+## Allow VRL to access the server environment, filesystem, hostname, local timezone,
+## network, and logs. Default is false. Enable only for trusted pipeline authors.
+## Applies to pipeline creation, execution, and dry-runs. Requires a restart.
+allow_vrl_host_access = false
+## Default for the HTTP log ingestion and pipeline dry-run `ignore_errors` parameter.
+## Explicit request values override this setting. Malformed input may be discarded,
+## and invalid custom timestamps in identity ingestion fall back to server time.
+## Pipeline execution errors are controlled separately by `skip_error`.
+## Default is false. Requires a restart.
+ignore_errors = false
 
 ## Configuration options for the event recorder.
 [event_recorder]

docker/query-regression-helpers/Dockerfile

@@ -0,0 +1,7 @@
+# Artifact image: binaries run on the query-regression ECS host after extraction.
+# Build on the same ECS image as the consumer to preserve the runtime ABI.
+FROM scratch
+ARG SOURCE_REVISION
+LABEL org.opencontainers.image.source="https://github.com/GreptimeTeam/greptimedb"
+LABEL org.opencontainers.image.revision=$SOURCE_REVISION
+COPY query_perf_fixture query_regression_runner /

docs/how-to/metric-snapshot-s3-bench.md

@@ -1,14 +1,17 @@
 # Metric snapshot benchmark on AWS S3
 
 The manual `Metric Snapshot S3 Benchmark` workflow tests packed Metric export and
-restore against real S3 in `ap-southeast-1`. It must first exist on the repository's
-default branch for GitHub to register `workflow_dispatch`. It was validated on AWS
+restore against real S3 in the configured CI bucket region. It must first exist on
+the repository's default branch for GitHub to register `workflow_dispatch`. It was validated on AWS
 by run [38544464731](https://github.com/GreptimeTeam/greptimedb/actions/runs/38544464731);
 [#9573](https://github.com/GreptimeTeam/greptimedb/pull/9573) records its conditions and results.
 
 ## Run
 
 Use the repository's existing EC2 runner configuration and CI S3 credentials.
+`AWS_CI_TEST_BUCKET_REGION` and `EC2_RUNNER_REGION` must be nonempty and equal;
+for example, both can be `us-east-1`. The configured subnet, security group and
+AMI (if explicitly supplied) must also belong to the runner region.
 The workflow passes credentials only through CI; no local AWS credentials are
 needed. Source, target and CLI use the same immutable image. This image's source
 SHA and required CLI capabilities have been checked:
@@ -20,6 +23,7 @@ gh workflow run metric-snapshot-s3-bench.yml \
   -f expected_commit=f690bbb2040865c0d22b0b57d51612b60ae27cd0 \
   -f logical_tables_per_group=10000 \
   -f benchmark_timeout_minutes=240 \
+  -f ec2_hourly_usd=0.9408 \
   -f prior_spend_usd=0
 ```
 
@@ -28,9 +32,12 @@ earlier cloud attempts in the campaign, including failed setup and cleanup.
 Do not reset the estimate to zero when retrying. The workflow accepts 50–10,000
 logical tables per group and a 30–240 minute benchmark job timeout.
 
-The runner is fixed at `c6id.4xlarge` (16 vCPU, 32 GiB RAM). At the recorded
-Singapore Linux on-demand estimate of $0.9408/hour, the default attempt reserves
-$6.4392: four benchmark hours, 45 minutes of provisioning, 10 minutes for queueing,
+The runner is fixed at `c6id.4xlarge` (16 vCPU, 32 GiB RAM). Set
+`ec2_hourly_usd` to a positive, finite hourly budget estimate appropriate for the
+selected region and instance. Its default $0.9408/hour preserves the historical
+Singapore estimate; it is not a live regional price or a guaranteed upper bound.
+The same estimate is used for preflight reservation and the final cost summary.
+At that default rate, the default attempt reserves $6.4392: four benchmark hours, 45 minutes of provisioning, 10 minutes for queueing,
 20 minutes of cleanup, and $1.50 for S3, disk and IPv4 costs. Preflight rejects prior spend at or above
 $15, or an attempt whose reserve would exceed the cumulative $20 budget.
 These estimates and checks are not an AWS hard spending cap or an AWS bill.

docs/how-to/query-options.md

@@ -0,0 +1,18 @@
+# Query options
+
+Query options adjust selected behavior for a connection or request; they are not server-wide settings or resource-pool controls.
+
+| Canonical option | SQL aliases | Values |
+| --- | --- | --- |
+| `query.parallelism` | `query_parallelism`, `query.parallelism` | Integer `1`–`1024`; server default `0` means automatic CPU-based parallelism. This is not a resource-pool limit. |
+| `query.allow_query_fallback` | `query_fallback`, `allow_query_fallback`, `query.allow_query_fallback` (legacy `ALLOW_QUERY_FALLBACK` syntax / `query_fallback` hint also work) | `true` or `false` |
+| `query.enable_remote_dynamic_filter_pushdown` | canonical name | `true` or `false` |
+| `datafusion.optimizer.*` | canonical name | Keys, types, defaults, and validation come from the linked DataFusion version (for example, numeric `max_passes` and Boolean `filter_null_join_keys`). |
+
+Use `SET query_parallelism = 4` or `SET query.parallelism TO 4`; `SHOW VARIABLES query.parallelism` works in MySQL and PostgreSQL, which also accepts `SHOW query.parallelism`. `SHOW VARIABLES <option>` reports the effective value. Unspecified options inherit engine settings; defaults are automatic parallelism, fallback `false`, and remote dynamic-filter pushdown `true`. Invalid values and unsupported names are errors; unknown third-party MySQL/PostgreSQL initialization settings retain existing tolerance.
+
+HTTP `X-Greptime-Hints` (for example, `query.parallelism=4`) and gRPC `x-greptime-hints` request metadata set request options; gRPC hints are not a protobuf `Hints` field. The Rust Flight client offers `Database::sql_with_hint` and `flight_request().with_hints`. Hints override session `SET`, including within a multi-statement HTTP request. SQL `SET` lasts for its connection only; `SET LOCAL` transaction-scoped restoration is not implemented.
+
+`query.enable_remote_dynamic_filter_pushdown` controls propagation across frontend-to-datanode remote queries; it is distinct from DataFusion's native `datafusion.optimizer.enable_dynamic_filter_pushdown`.
+
+Internal flow/query-reserved keys are not public query options. HTTP response timeout `x-greptime-timeout` defaults to `0` (no deadline); this is a response deadline, not a per-statement or stream/body-processing limit.

docs/rfcs/2026-05-28-table-semantic-layer.md

@@ -100,24 +100,19 @@ Two design decisions worth pinning down up front, because they constrain everyth
 - **Conflict.** Some table-level keys (`trace.conventions` lifted from `schema_url`, `metric.temporality`, ...) cannot represent the truth when a long-lived table sees rows from multiple sources. v1 records `mixed` or `unknown` rather than a fictitious single value. Downstream consumers must treat any single-valued semantic key as best-effort, not strong evidence.
 - **Update.** Semantic options are stamped at table creation. v1 does not specify an update path; promoting `metadata_quality` from `inferred` to `declared`, refreshing `resource.attributes_preserved`, or revising `trace.conventions` on later writes is deferred. If real usage shows update is needed, it lands as a separate RFC.
 
-OTLP delta sums and explicit histograms store raw interval values without
-accumulating across timestamps. Delta-to-cumulative conversion and its
-per-series ingestion state are intentionally out of scope. Explicit histogram
-bucket counts are prefix-summed only within each point to produce the classic
-histogram representation. Delta exponential histograms are currently rejected;
-[planned raw-delta support](2026-08-04-native-histograms.md#planned-raw-delta-exponential-histograms)
-will store their interval values in the canonical native-histogram Struct using
-the same temporality tag and a gauge reset hint. The tag will select histogram
-summation for `increase()` and summation divided by range duration for `rate()`,
-without cumulative reset correction or extrapolation.
+OTLP delta sums, explicit histograms and exponential histograms store raw interval
+values without accumulation across timestamps or requests. Explicit histogram
+buckets are prefix-summed within each point. Exponential histograms retain their
+canonical native-histogram Struct and use a gauge reset hint for delta samples.
+All use the fixed String tag `otlp_aggregation_temporality="delta"`, independent
+of `default_column_prefix`, as part of series identity.
+
+For tagged float series, the marker selects existing delta `rate()`/`increase()`
+behavior. Native histogram counter functions are unchanged: query raw delta
+histograms with `sum_over_time(metric[3m])`, or divide that histogram by `180`
+for a per-second result. See [raw-delta exponential histograms](2026-08-04-native-histograms.md#raw-delta-exponential-histograms).
+Table options never select per-row query behavior.
 
-OTLP delta sums and explicit histograms additionally store the query-visible
-String tag `otlp_aggregation_temporality="delta"` on each generated row. Its
-name is fixed and does not follow `default_column_prefix`. The tag is part of
-series identity and is authoritative for per-series float `rate()` and
-`increase()` behavior; the table option is never used as a row-level
-discriminator. Native histograms currently retain their native algorithms;
-the planned delta extension will change only tagged delta histogram series.
 Prometheus metadata reports `unknown` for counter, histogram, and up/down-counter tables
 whose catalog temporality is `delta` or `mixed`. A same-request conflict can
 create `mixed`; a later write does not update an existing table option, so the
@@ -143,7 +138,7 @@ schema lacks it; unrelated nullable labels retain their existing matching
 behavior. Ignoring the marker adds no marker-related matching work.
 
 Use `ignoring(otlp_aggregation_temporality)` when temporality should not
-participate in matching. Apply `rate()` or `increase()` before an aggregation
+participate in matching. For float series, apply `rate()` or `increase()` before an aggregation
 or subquery that drops the marker. `irate()` and `resets()` are not
 temporality-aware in v1; `delta()`, `idelta()`, and `changes()` retain their
 existing raw-sample semantics.

docs/rfcs/2026-08-04-native-histograms.md

@@ -16,8 +16,8 @@ Struct before persistence and are queried only as native histograms.
 Prometheus Remote Write 2.0 native histograms and cumulative OTLP/HTTP
 exponential histograms are enabled by default.
 Remote Write 1.0 histogram payloads are rejected instead of being acknowledged and dropped.
-Native-histogram Remote Read is deferred; the existing Remote Read path
-continues to return scalar samples only.
+Remote Read returns scalar samples and native histograms through `SAMPLES`,
+including exact stored integer counts. Streamed responses remain deferred.
 
 # Goals
 
@@ -25,13 +25,15 @@ continues to return scalar samples only.
 2. Keep one stable Prometheus-compatible persisted representation across
    Prometheus and OTLP ingestion.
 3. Match Prometheus query behavior where it is observable and practical.
-4. State intentional limitations explicitly so incomplete behavior is not
+4. Return stored native histograms through sampled Remote Read without rounding
+   integer counts or losing supported histogram fields.
+5. State intentional limitations explicitly so incomplete behavior is not
    mistaken for support.
 
 # Non-Goals
 
 - Supporting native histograms in Remote Write 1.0.
-- Returning native histograms through Prometheus Remote Read.
+- Streaming Remote Read responses (`STREAMED_XOR_CHUNKS`).
 - Persisting Remote Write metric metadata.
 - Persisting exemplars.
 - Converting OTLP delta values to cumulative values during ingestion. Supported
@@ -107,39 +109,72 @@ beyond it. The overflow index is `(1024 << schema) + 1` for non-negative schemas
 and `(1024 >> -schema) + 1` for negative schemas. Inputs beyond that index are
 rejected rather than clamped or merged into infinity.
 
+Ingestion and sampled Remote Read share histogram layout validation. An initial
+exponential span offset of `i32::MIN` is rejected on either side, including for
+empty spans; other negative initial offsets remain supported. For custom
+histograms with `N` bounds, bucket index `N` is the overflow bucket, and empty
+spans may end at `N + 1`. Span extents beyond that boundary are rejected.
+
+This tightens ingestion acceptance: these layouts were previously accepted on
+write but rejected by Remote Read. A validation failure rejects the entire
+Remote Write 2.0 request with HTTP 400 and all written-count headers set to zero,
+before any samples or histograms are persisted. Healthy samples in the same
+request are also rejected; senders that do not retry HTTP 400 can drop them.
+
 Exemplars and declared metric metadata are accepted on the wire but are not
 persisted by this feature.
 
 ## Remote Read
 
-Native-histogram Remote Read is deferred. GreptimeDB's existing Remote Read path
-continues to negotiate `SAMPLES` responses and serialize scalar samples only.
+`/v1/prometheus/read` returns scalar samples and integer or float native
+histograms through the existing Snappy-compressed `SAMPLES` protobuf response.
+Omitting accepted types, requesting `SAMPLES`, or requesting
+`[STREAMED_XOR_CHUNKS, SAMPLES]` selects this format. Streaming-only requests
+remain unsupported; the response is still buffered as a whole.
 
-A follow-up can map the stored Struct into `TimeSeries.histograms`. It must
-preserve integer counts without passing through `f64`, retain the stored
-histogram fields, and decide sampled versus streamed response coverage. The
-sampled protobuf has no start-timestamp field, so that omission must remain
-explicit if sampled responses are implemented.
+One request can query both scalar and histogram metrics. A histogram series
+can contain integer and float histogram samples, but the one-sample-kind-per-
+metric-table rule remains unchanged. The selected field must be `Float64` or
+the canonical native-histogram Struct, regardless of its stored column name.
+
+Integer counts remain exact within the existing nonnegative `i64` storage
+range, including above `2^53`. Stored absolute integer buckets become signed
+wire deltas, retaining their sequence across span boundaries. Float bucket
+counts remain absolute. Schema, spans, custom bounds, zero threshold, sum
+(including stale-NaN bits), reset hint, and millisecond timestamps are retained.
+The sampled protobuf has no `start_timestamp` field, so that value remains in
+storage but is omitted from the response. Raw-delta OTLP histograms retain
+their temporality labels and gauge hints; Remote Read does not accumulate them
+or reconstruct the original OTLP point.
+
+Malformed or wire-incompatible non-null stored histograms fail the entire
+`ReadRequest` with HTTP 500, including healthy queries in that request. Read
+validation remains necessary for older rows and typed inserts that bypass the
+histogram ingestion encoder. Incompatible stored rows are not repaired by the
+stricter ingestion checks. Null histograms remain absent samples.
+
+Custom-bucket interoperability is exercised against Prometheus revision
+`e471c7708c6b0b13e2050222bbd5a907b36f4113`, including `AtHistogram` and
+`AtFloatHistogram` on integer custom buckets. This is a tested revision, not
+a minimum-release claim; older readers can lose custom bounds during integer-
+to-float conversion. Integer payloads are not downgraded for those readers.
+The [wire fixture and iterator check](../../tests/compatibility/wire/native_histogram_remote_read/README.md)
+record the independent expectations and reproduction commands.
 
 ## OTLP
 
 OTLP/HTTP exponential histograms are accepted by default.
 OTel Arrow exponential histograms are rejected because
 the current Arrow wire format omits `zero_threshold`; accepting them would
-silently change the distribution. Currently, cumulative temporality is required;
-delta and unspecified exponential histograms are rejected before their points are
-converted. Explicit OTLP histograms keep their existing `_bucket`, `_sum`, and
-`_count` representation. OTLP delta sums and explicit histograms are stored as
-raw interval values with `otlp_aggregation_temporality="delta"`. Explicit
-histogram bucket counts are prefix-summed within each point for the classic
-histogram representation, but values are never accumulated across timestamps.
-For these tagged float series, PromQL `increase()` sums the interval values
-in the query range, and `rate()` divides that sum by the range duration.
-Delta-to-cumulative conversion is intentionally out of scope, not deferred work.
-Raw-delta exponential-histogram support is planned below; delta exponential
-points remain rejected until it is implemented. See
-[Table Semantic Layer](2026-05-28-table-semantic-layer.md#conflict-and-update-semantics)
-for the per-series temporality contract.
+silently change the distribution. Cumulative and delta temporality are accepted;
+unspecified and unrecognized temporality are rejected before processing points.
+Explicit OTLP histograms keep their existing `_bucket`, `_sum`, and `_count`
+representation. All delta metrics store raw interval values with
+`otlp_aggregation_temporality="delta"`; ingestion never accumulates across points
+or requests. Explicit histogram bucket counts are prefix-summed within a point.
+Tagged float series retain GreptimeDB's automatic delta `rate()`/`increase()`
+extension. Native histograms use the explicit queries below instead.
+Delta-to-cumulative conversion is outside this feature.
 
 OTLP scales `-4` through `8` map directly to Prometheus schemas. Higher scales
 are downscaled to schema `8`: dense counts that collide are merged before the
@@ -148,11 +183,14 @@ This preserves count mass but irreversibly loses distinctions between source
 buckets that merge. Lower scales are rejected. OTLP counts always populate the
 integer histogram family. The transmitted non-negative finite zero threshold
 and zero count are preserved; the Struct start timestamp is stored in
-milliseconds and the reset hint is unknown. Point timestamps and attributes
-retain their existing mode-specific conversion rules. Legacy mode retains its
-normalized name, attribute rules, and nanosecond row timestamp. Non-legacy mode
-retains Prometheus-compatible translation and its millisecond row timestamp.
-Both modes write the same canonical Struct.
+milliseconds. The reset hint is gauge for delta and unknown for cumulative samples.
+OTLP metrics always use Prometheus-compatible name and attribute conversion;
+`otlp_metric_compat` remains accepted as table metadata but does not select a
+conversion mode. Metric-engine conversion retains nanoseconds until alignment
+to the destination time-index unit: seconds, milliseconds, microseconds, or nanoseconds.
+New logical tables inherit their backing table's unit; auto-created physical
+tables default to milliseconds. Ordinary insertion uses milliseconds.
+Both storage paths write the same canonical Struct.
 
 An absent sum is stored as an ordinary quiet NaN, so the sample remains
 selectable while `histogram_sum` is unknown. Any NaN sum on a point without
@@ -172,33 +210,71 @@ produced an accepted row.
 Zero-count buckets are omitted after downscaling, splitting nonzero runs into
 sparse spans without changing integer counts. Rejected data points increment
 `greptime_servers_otlp_exponential_histogram_rejected_data_points_total`, labeled
-by `reason`: `delta_temporality`, `unspecified_temporality`, or
-`invalid_data_point`. Each rejected point is counted once, including when
+by `reason`: `unspecified_temporality` or `invalid_data_point`. Attribute and
+schema rejections use `invalid_data_point`; other metric families do not
+increment this histogram-specific counter. Each rejected point is counted once, including when
 resource descriptors are enabled. Minimum, maximum, and exemplars are not persisted.
 
-### Planned raw-delta exponential histograms
+### Raw-delta exponential histograms
 
-Accept OTLP delta exponential histograms and store each point's interval counts
-and sum in the existing canonical native-histogram Struct, without accumulating
-across timestamps. Reuse the integer-count encoding, scale conversion,
-zero-threshold preservation, stale-marker handling, and partial-success path.
-OTel Arrow support remains blocked independently by its missing `zero_threshold`.
+Each stored delta sample retains its interval count, sum, zero count and buckets.
+The temporality tag separates delta and cumulative series under the same table
+name. Query raw deltas explicitly:
 
-Store `otlp_aggregation_temporality="delta"` as part of series identity, as for
-delta sums and explicit histograms. Use the gauge reset hint for delta histogram
-samples so decreasing interval counts are not interpreted as counter resets.
-Prometheus uses the same gauge hint for
-[raw-delta exponential-histogram ingestion](https://github.com/prometheus/prometheus/blob/9f159f6df198280b757163ec555ffde0c8bfe8e0/storage/remote/otlptranslator/prometheusremotewrite/histograms.go#L125).
-The temporality tag, rather than the gauge hint alone or table metadata,
-determines raw-delta query behavior.
+```promql
+sum_over_time(request_duration{otlp_aggregation_temporality="delta"}[3m])
+sum_over_time(request_duration{otlp_aggregation_temporality="delta"}[3m]) / 180
+histogram_count(sum_over_time(request_duration{otlp_aggregation_temporality="delta"}[3m]))
+```
 
-Extend the per-series `increase()` and `rate()` selection to native histogram
-inputs: sum interval histograms over the query range, then divide by the range
-duration for `rate()`. Do not apply cumulative reset correction, synthetic-zero
-insertion, or counter extrapolation to these delta samples. Preserve cumulative
-histogram behavior and mixed float/histogram range warnings. Acceptance coverage
-must include multiple intervals whose counts decrease, exact integer ingestion,
-stale points, partial rejection, and pure-native and mixed-range query paths.
+Samples are selected by their endpoint timestamp in the existing left-open,
+right-closed range. Each selected sample contributes its entire interval,
+including intervals crossing the left boundary. A single sample contributes its
+full value; an empty window emits no sample. Counts `10,20,15` at seconds
+`60,120,180` give count `45` at second `180`, or `0.25` after division by `180`.
+Missing intervals do not shorten the divisor. There is no prorating, reset
+correction, extrapolation, synthetic zero insertion or inferred coverage.
+
+Native-histogram `rate()` and `increase()` retain counter behavior, regardless
+of the tag; they are not supported queries for raw deltas. Converting a histogram
+to a float can enter the existing scalar-delta extension, so the explicit
+expressions above are the supported examples. Existing sum/division UDFs and
+serialization are reused without a new wire format.
+
+Under default storage options, identical table/tags/stored timestamps overwrite
+with the last point, without adding buckets. Timestamp alignment can collapse
+endpoints at coarser units; interval starts do not form part of the key. Preserve
+source order through default resolution and row merging. Existing storage
+options continue to apply.
+
+Metrics recovery is point-based across gauges, sums, explicit histograms,
+summaries and exponential histograms. Point-local validation precedes
+existing-schema validation, which precedes simultaneous detection of fresh
+logical/physical conflicts. Reject the union of contributors to all conflicts;
+remove every output of a rejected point. Defaults distinguish omitted columns
+from explicit nulls. Auto-DDL configuration and hints remain authoritative;
+metric-engine timestamp alignment remains permitted with auto-DDL disabled.
+Only surviving points contribute metadata and descriptor windows. Metadata
+describes actual outputs, excluding absent sums and empty quantile outputs.
+
+Authentication, authorization, admission, catalog and primary-write failures
+retain their failure paths. Synchronous acceptance waits for writes; eligible
+asynchronous batching acknowledges enqueue, and later flush errors retain their
+existing metrics. Descriptor failures after primary acceptance remain warnings,
+including a partial-success warning with zero rejected points. No write is
+retried in smaller subsets after an ambiguous result.
+
+Prometheus compatibility follows raw-delta ingestion with a gauge reset hint
+and explicit summation. GreptimeDB retains its temporality label, transmitted
+zero threshold, NaN for absent sums, canonical stale representation, integer
+bounds, sparse compaction and existing arithmetic. Equivalent distributions
+need not have identical layouts or floating-point results. Start timestamps
+remain stored but explicit summation does not use them for counter correction.
+Public metadata is `unknown` for freshly created delta/mixed-temporality tables;
+existing creation-time options are not refreshed by later writes.
+
+Upgrade receiving ingestion frontends before enabling delta senders. Mixed-version
+query execution is unverified; this feature introduces no new query UDF or format.
 
 # PromQL Compatibility Decisions
 
@@ -304,16 +380,20 @@ partial-success behavior, kind exclusivity,
 stale-marker selector semantics, synthetic-zero rates, incompatible empty
 layouts, overflow-safe averages, layout-sensitive equality, infinite custom
 midpoints, and exponential overflow indices for schemas `-4`, `0`, and `8`.
-Behavioral coverage also verifies frontend PromQL warning and info responses.
+Behavioral coverage also verifies frontend PromQL warning and info responses,
+sampled Remote Read before and after flush in standalone and distributed
+setups, exact large integers, stale-marker bits, timestamp units, and OTLP
+normalization. Frozen upstream protobuf fixtures guard the wire encoder.
 
 This work does not change GreptimeDB's persisted Struct or protobuf
-dependencies. It adds the disabled-by-default public OTLP gate; existing
-native-histogram data remains readable.
+dependencies. Remote Read uses the existing endpoint and negotiation without
+a new configuration option; existing native-histogram data remains readable.
 
 # Future Work
 
-- Native-histogram Remote Read, including exact integer round-trips and streamed
-  chunks with start timestamps.
+- TODO: Add native histogram and float histogram chunks to
+  `STREAMED_XOR_CHUNKS`, with checksummed framing and bounded buffering. Decide
+  separately whether to include start-timestamp-bearing chunk encodings.
 - Raw-delta OTLP exponential-histogram storage and temporality-aware native
   histogram queries, as described above; no delta-to-cumulative conversion.
 - Persistent Remote Write metadata and accurate help/unit updates.

docs/rfcs/2026-09-06-object-store-wal.md

@@ -31,7 +31,7 @@ Object storage is already provisioned, already durable across zones, and already
 # Non-goals
 
 - Distributed mode. The provider is rejected when a datanode is controlled by a metasrv; the metasrv side has no allocation logic for it yet.
-- Persistence of per-region obsolete watermarks. Mito re-establishes every watermark when it opens a region, which is what garbage collection reasons from; persisting them is a follow-up.
+- Persistence of the obsolete watermarks that flushes and truncates move. Mito re-establishes each of them when it opens a region, which is what garbage collection reasons from. Only the release of a region that is never opened again, after a drop or an offline cleanup, is persisted, as described under *Entry ids*.
 - Matching Raft Engine's write latency. The provider trades latency for the absence of local durable WAL state.
 - Making standalone metadata independent of local disk. The metadata store is unchanged by this RFC.
 
@@ -50,7 +50,7 @@ The provider is added to the four parallel WAL enums so that every resolution pa
 
 Two choices are deliberate:
 
-- The persisted `WalOptions` carry the derived node prefix `<prefix>/datanodes/<node_id>/epochs/<generation>` described under *Object format*, not the configured root. On reopen the log store compares the persisted prefix with the one it runs under and fails fast on a mismatch, so an operator who changes the prefix or the node id cannot silently point a region at an empty WAL. Changing the prefix of an existing region requires an explicit migration, which is out of scope here.
+- The persisted `WalOptions` carry the derived node prefix `<prefix>/datanodes/<node_id>/epochs/<generation>` described under *Object format*, not the configured root. On reopen the log store compares the persisted prefix with the one it runs under and fails fast on a mismatch, so an operator who changes the prefix or the node id cannot silently point a region at an empty WAL. Changing the prefix of an existing region requires an explicit migration, which is out of scope here. The prefix does not name the object store that holds it, so changing `storage_provider`, or the bucket or root of the provider it names, keeps the prefix; the history check under *Log store* catches such a change when a region opens.
 - `Provider::ObjectStore` is classified as a remote WAL. Mito's remote-WAL branches (initial flushed entry id from `latest_entry_id`, `topic_latest_entry_id` after replay, region-scoped reads, no topic grouping on batch open) apply unchanged. This is a classification of replay semantics, not of physical location.
 
 A datanode that is configured with a metasrv client rejects the provider before any store is built: `DatanodeBuilder` returns an error stating that the object store WAL is standalone-only. Separately, `DatanodeWalConfig::ObjectStore` cannot be converted into `MetasrvWalConfig`, so a metasrv cannot be configured with it either. The standalone bootstrap matches the datanode WAL configuration before that conversion and constructs `WalProvider::ObjectStore` directly.
@@ -65,11 +65,12 @@ header | segment(region 1) | ... | segment(region N) | footer | trailer
 
 - The header carries the magic `GTWALOBJ`, format version 1, the object sequence number, the epoch of the store instance that wrote the object, a link to the object this one extends, as that object's sequence number and epoch or none for the first object of a chain, the reclaim boundary described under *Entry ids*, and a CRC32 of the preceding header bytes; the header is 54 bytes. A reclaim boundary above the object's own sequence fails the header check. A sequence number and an epoch together identify an object: an instance never writes two objects under one sequence, and instances have distinct epochs, as described under *Log store*. Recovery selects the history it replays by the links and the epochs, as described under *Log store*.
 - An object may hold no segment. A store writes such an object when it opens, as described under *Log store*; it is a chain object like any other, and its segments are trivially all flushed.
+- An object may carry release records, which a store writes when it releases a region, as described under *Entry ids*; each names a region and the entry id the region is released through. They form the single entry of a segment of the reserved region id `u64::MAX`, whose entry id is at position 1 of the object; segments are ordered by region id, so it is the last segment of the object. Its payload is their count, then per record, ordered by region id and at most one per region, the region id and the entry id. The object keeps format version 1, so a reader that predates release records reads it as any other object. An object written for a release alone holds only that segment.
 - Each segment holds the entries of exactly one region, ordered by entry id; segments are ordered by region id. Ordering makes the encoding deterministic, which the identical-retry rule below depends on.
 - The footer records, per segment, the region id, the entry id range, the entry count, the byte range and a CRC32 of the segment.
 - The fixed-size trailer carries the magic `GTWALTRL`, the footer location, a CRC32 of the footer and a CRC32 of the whole object. A reader locates the footer by reading the trailer at the end of the object and can decode a single region's segment without touching the others.
 
-Corruption is handled at two levels; nothing else is checked in production. Structural damage fails recovery: recovery reads only the header, the trailer and the footer of every object, as described under *Log store*, and verifies the header and trailer magic, the format version, the header CRC, the header sequence against the key, the footer CRC and that the segment byte ranges tile the object body exactly; the first object that fails any of these checks fails construction of the store, since nothing can be trusted about such an object. A segment is verified by the read that decodes it, and a segment that does not decode, because its CRC does not match or its content disagrees with its footer entry, is a data-level failure confined to one region. The read fetches the segment once more, since the download rather than the object may be damaged, and decodes it again; if it still does not decode, the behaviour follows `on_corrupted_segment`. `skip`, the default, skips the segment, records a WAL hole for the region with the object key, the object sequence and the entry id range the footer holds for the segment, increments the counter `greptime_logstore_object_store_wal_skipped_segments_total` and logs a warning with those fields, and the read continues with the next object; every other region in the object is unaffected. The holes are kept in memory per region and can be listed from the store; nothing is persisted, so a store opened later on the same prefix records a hole again when a read meets the segment. `fail` fails the read with the decode error, so a region whose replay meets the segment does not open. Skipping loses only the unflushed entries of that region in that object: nothing already flushed is affected, and the loss is reported rather than hidden. A failed fetch is not corruption and fails the read in both modes, unless garbage collection deleted the object, as described under *Entry ids*, in which case the object is skipped; an object the catalog still holds once a delete of it has settled fails the read whether it is missing or unreadable, and a segment of such an object that does not decode follows `on_corrupted_segment` as it would without any collection. The whole-object CRC is not a recovery requirement; it stays in the trailer as a fixture-level check. Because recovery never checks it, the header carries its own CRC, so a damaged link fails recovery instead of silently cutting acknowledged objects off the chain. The format is frozen at version 1.
+Corruption is handled at two levels; nothing else is checked in production. Structural damage fails recovery: recovery reads only the header, the trailer and the footer of every object, as described under *Log store*, and verifies the header and trailer magic, the format version, the header CRC, the header sequence against the key, the footer CRC and that the segment byte ranges tile the object body exactly; the first object that fails any of these checks fails construction of the store, since nothing can be trusted about such an object. The segment of release records of an indexed object, which recovery reads too, is held to the same rule rather than to `on_corrupted_segment`: one that cannot be fetched or does not decode fails construction, since the watermarks it seeds would otherwise be lost. A segment is verified by the read that decodes it, and a segment that does not decode, because its CRC does not match or its content disagrees with its footer entry, is a data-level failure confined to one region. The read fetches the segment once more, since the download rather than the object may be damaged, and decodes it again; if it still does not decode, the behaviour follows `on_corrupted_segment`. `skip`, the default, skips the segment, records a WAL hole for the region with the object key, the object sequence and the entry id range the footer holds for the segment, increments the counter `greptime_logstore_object_store_wal_skipped_segments_total` and logs a warning with those fields, and the read continues with the next object; every other region in the object is unaffected. The holes are kept in memory per region and can be listed from the store; nothing is persisted, so a store opened later on the same prefix records a hole again when a read meets the segment. `fail` fails the read with the decode error, so a region whose replay meets the segment does not open. Skipping loses only the unflushed entries of that region in that object: nothing already flushed is affected, and the loss is reported rather than hidden. A failed fetch is not corruption and fails the read in both modes, unless garbage collection deleted the object, as described under *Entry ids*, in which case the object is skipped; an object the catalog still holds once a delete of it has settled fails the read whether it is missing or unreadable, and a segment of such an object that does not decode follows `on_corrupted_segment` as it would without any collection. The whole-object CRC is not a recovery requirement; it stays in the trailer as a fixture-level check. Because recovery never checks it, the header carries its own CRC, so a damaged link fails recovery instead of silently cutting acknowledged objects off the chain.
 
 The operator configures only the prefix root. The store runs under a node prefix derived from it, `<prefix>/datanodes/<node_id>/epochs/<generation>`, and object keys are `<node prefix>/objects/<sequence>.wal`, where the sequence is zero-padded to 20 decimal digits. Standalone uses its datanode id, zero unless configured, and generation zero, so with prefix `wal` the first object is `wal/datanodes/0/epochs/0/objects/00000000000000000000.wal`. Every place that names the prefix, whether to open the store or to allocate the WAL options of a region, derives it from the same function, so the prefix a region persists matches the one its store runs under. Listing accepts only well-formed keys under `<node prefix>/objects/` and returns them in sequence order. This is the standalone form of the key layout under *Alignment with cluster mode*; the cluster assignment of node id and generation is proposed there.
 
@@ -101,21 +102,23 @@ Positions start at one, so entry id zero is never assigned: Mito treats zero as
 
 `latest_entry_id(provider)` is region-scoped: the highest entry id durable under the prefix for the provider's region, or zero when the region has none. A freshly created region therefore starts at zero even if other regions under the prefix have entries. This differs from Kafka, whose high watermark is per topic; it is sufficient here because ids are per region, and Mito consumes the value only for the region that asked.
 
-Two rules keep every new id of a region above every id it already has, on a prefix that already holds objects and across the watermark Mito hands the store. On open, the next object sequence is the larger of the largest listed sequence plus one, orphans included, and, over every region, the object named by the region's largest existing entry id plus one, that is `(max_entry_id >> 20) + 1`, so that the catalog's strictly increasing ranges hold; sequence gaps are tolerated, and a sequence at or above `2^44` is rejected as exhausted. Objects written under an earlier contiguous scheme are not rewritten: they remain readable, their ids carry no object information, and the rule places them by their high bits just the same.
+Two rules keep every new id of a region above every id it already has, on a prefix that already holds objects and across the watermark Mito hands the store. On open, the next object sequence is the larger of the largest listed sequence plus one, orphans included, and, over every region, the object named by the region's largest existing entry id plus one, that is `(max_entry_id >> 20) + 1`, so that the catalog's strictly increasing ranges hold; sequence gaps are tolerated, and a sequence at or above `2^44` is rejected as exhausted. Ids have been object-sequence-major since the first version of the provider; a contiguous scheme existed only in the reference implementation before release, so no deployed prefix holds ids without object information, and the history check under *Log store* relies on that.
 
-The second rule is the sequence floor of `obsolete`. Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, and the actor applies the watermark and a floor derived from that entry id together: the next object sequence is raised to `(entry_id >> 20) + 1` unless it is there already, while zero, the watermark of a region without entries, names no object and needs no floor, so that a watermark assigned under another prefix, or one whose object this prefix no longer holds, is never passed by a new id. The floor moves only the next sequence, never the chain: the next batch still links to its predecessor, so a skipped sequence leaves no gap in the history recovery replays. While the open batch holds ids under the next sequence, the call is refused with a retryable error and records neither the watermark nor the floor, because those ids would fall below the floor. In the `enqueued` mode an id the store itself handed out needs no floor, even before it is durable: it is never handed out again while the store runs, so every later id of the region is greater. In the current single-prefix layout the floor never needs to move, since every watermark Mito hands the store names an object below the next sequence; the rule exists for the cross-prefix cases under *Alignment with cluster mode*. No additional persisted state is introduced.
+The second rule is the sequence floor of `obsolete`. Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, and the actor applies the watermark and a floor derived from that entry id together: the next object sequence is raised to `(entry_id >> 20) + 1` unless it is there already, while zero, the watermark of a region without entries, names no object and needs no floor, so that a watermark assigned under another prefix, or one whose object this prefix no longer holds, is never passed by a new id. The floor moves only the next sequence, never the chain: the next batch still links to its predecessor, so a skipped sequence leaves no gap in the history recovery replays. While the open batch holds ids under the next sequence, the call is refused with a retryable error and records neither the watermark nor the floor, because those ids would fall below the floor. In the `enqueued` mode an id the store itself handed out needs no floor, even before it is durable: it is never handed out again while the store runs, so every later id of the region is greater. In the current single-prefix layout the floor never needs to move for a prefix that holds the region's history, since every watermark Mito hands the store then names an object below the next sequence. A watermark above the newest object holding entries that the store found means the prefix does not hold that history: the history check under *Log store* refuses such an open before Mito calls `obsolete`, and the floor moves only when `accept_missing_history` accepted the loss. Otherwise the rule exists for the cross-prefix cases under *Alignment with cluster mode*. No additional persisted state is introduced.
 
-Garbage collection deletes the objects below a reclaim boundary that the store persists in every header, so that recovery can tell a collected predecessor from one that never landed. The boundary is computed from the catalog and the watermarks the store already keeps: each region with a recorded watermark contributes the sequence of its first indexed object holding a segment whose largest entry id is above the watermark; a region without a recorded watermark contributes its first indexed object; a region whose watermark is the maximum entry id, after `obsolete_all`, contributes nothing; the boundary is the smallest contribution, or the last indexed sequence plus one when nothing contributes, and it never decreases. Whether a segment is flushed is decided per segment from the footer, on the full entry id, so a watermark that stops in the middle of an object keeps the object. The boundary is recomputed after every successful `obsolete`, after `obsolete_all` and after every object is indexed; Mito calls `obsolete` after a flush, after a truncate, when it drops a region and when it opens one, so a restart re-establishes every watermark and the boundary advances on the same rule. Every sealed batch carries the boundary current at sealing and the start object carries the boundary recovery found, so a boundary is always at or below the sequence of the object carrying it. The boundary that permits deletion is the one carried by the last indexed object; the boundary of a batch whose create failed or is in flight permits nothing.
+Garbage collection deletes the objects below a reclaim boundary that the store persists in every header, so that recovery can tell a collected predecessor from one that never landed. The boundary is computed from the catalog and the watermarks the store already keeps: each region with a recorded watermark contributes the sequence of its first indexed object holding a segment whose largest entry id is above the watermark; a region without a recorded watermark contributes its first indexed object; a region whose watermark is the maximum entry id, after a release in full, contributes nothing; the boundary is the smallest contribution, or the last indexed sequence plus one when nothing contributes, and it never decreases. Whether a segment is flushed is decided per segment from the footer, on the full entry id, so a watermark that stops in the middle of an object keeps the object. The boundary is recomputed after every successful `obsolete`, after every release and after every object is indexed; Mito calls `obsolete` after a flush, after a truncate, when it drops a region and when it opens one, so a restart re-establishes the watermark of every region Mito opens again, and the boundary advances on the same rule. A region that is never opened again needs the release described below. Every sealed batch carries the boundary current at sealing and the start object carries the boundary recovery found, or its own sequence when the recovered chain holds no segment, so a boundary is always at or below the sequence of the object carrying it. The boundary that permits deletion is the one carried by the last indexed object; the boundary of a batch whose create failed or is in flight permits nothing.
 
-Every present object with a sequence below the durable boundary is deletable, on the chain or not: the objects the catalog indexes, the objects recovery listed but did not index, the objects of an earlier epoch that the start object moved past, and the sequences of batches that failed in this run, whose objects may or may not exist. A collection starts after a successful `obsolete`, after `obsolete_all` and after an indexing that raised the durable boundary. Deletes run in the background, at most four at a time and in no particular order, and never hold up admission, sealing, uploads or acknowledgements; an indexed object is unindexed only after its delete succeeded, and the WAL holes recorded for it are dropped with it. Every deleted object is counted in `greptime_logstore_object_store_wal_deleted_objects_total`; a failed delete is logged, counted in `greptime_logstore_object_store_wal_failed_deletes_total` and retried at the next collection, and the object stays indexed meanwhile. An object deleted but not yet unindexed when the process crashes is simply absent at the next recovery, which lists the prefix. Nothing is collected once `stop` has begun or while the store is poisoned; a delete in flight when `stop` begins runs to completion before `stop` returns, and the actor settles every attempt it abandons when it exits.
+Every present object with a sequence below the durable boundary is deletable, on the chain or not: the objects the catalog indexes, the objects recovery listed but did not index, the objects of an earlier epoch that the start object moved past, and the sequences of batches that failed in this run, whose objects may or may not exist. A collection starts after a successful `obsolete`, after a release and after an indexing that raised the durable boundary. Deletes run in the background, at most four at a time and in no particular order, and never hold up admission, sealing, uploads or acknowledgements; an indexed object is unindexed only after its delete succeeded, and the WAL holes recorded for it are dropped with it. Every deleted object is counted in `greptime_logstore_object_store_wal_deleted_objects_total`; a failed delete is logged, counted in `greptime_logstore_object_store_wal_failed_deletes_total` and retried at the next collection, and the object stays indexed meanwhile. An object deleted but not yet unindexed when the process crashes is simply absent at the next recovery, which lists the prefix. Nothing is collected once `stop` has begun or while the store is poisoned; a delete in flight when `stop` begins runs to completion before `stop` returns, and the actor settles every attempt it abandons when it exits.
 
-A read never needs a deleted object, since entries at or below a region's watermark are hidden anyway, but a read that listed an object before it was collected skips the object when it gets there. A read that meets an error on such an object makes two observations, in the reverse of the order the actor writes them: it looks at the set of objects whose delete is in flight, and then at the catalog. The actor registers an object in that set before its delete starts, unindexes it if the delete succeeded, and only then takes it out, so an object absent from the set was either never collected, in which case the catalog still holds it, or is already unindexed by the time the catalog is read. Observing the catalog first would let a collection complete between the two observations and leave the read with an object that still looked indexed and no longer looked deleting. An object that is still in the set has an undecided delete, so the read waits for that one delete, a single object store request, rather than assume it succeeds: a delete that fails leaves the object present and indexed, and the error the read met is then reported as any other, `on_corrupted_segment` included. What the read waits for is the delete attempt its observation found, taken together with the lookup, not the object: a retry of the same object is a separate attempt, so the attempt the read holds releases it even while that retry is in flight, and a read is never handed from one attempt to the next. A read stream outlives the store it came from, so the actor settles every attempt it abandons when it exits, whether the store was stopped or dropped; the objects of those attempts stay present and indexed, so the reads waiting for them report the error they met. A collected object is skipped silently: it is not recorded as a hole and does not count as a skipped segment. An object the catalog still holds after those observations fails the read like any other fetch failure, `on_corrupted_segment` included. Within one run the catalog keeps the largest entry id of a region after the objects holding it were collected, so `latest_entry_id`, the durability wait and the `enqueued` clamp keep their reference point; a store opened later learns only what is listed, so after a restart `latest_entry_id` of a region whose later objects were collected is the largest id among the objects left, which only the pruning hint of Mito consumes. No state is persisted outside the headers and no configuration option is added.
+A region that Mito drops is never opened again, so no restart re-establishes its watermark, and a region without a watermark holds the boundary at its first indexed object. The store therefore persists the release of such a region in the WAL. `obsolete_all`, which Mito calls when it cleans a region up offline, releases the region in full, through the maximum entry id. `release_region`, a `LogStore` method that Mito calls after `obsolete` when it drops a region, releases it through the largest entry id the store has handed out for the region, so entries of a region that reuses the id later stay above the release. A release moves the region's watermark up in memory, recomputes the boundary, starts a collection and seals the open batch at once, empty or not, with a release record of the region. The call returns once that object is durable and indexed, in both acknowledgement modes, and fails whenever its batch fails, as an append in the `durable` mode does. In the `enqueued` mode the watermark of a release is not clamped to the durable entry id: a released region is never replayed, and its record becomes durable only after every object sealed before it. The object of a release carries the boundary its watermark allows, so the record and that boundary become durable together. Recovery seeds the watermarks from the release records of the indexed objects before it computes any boundary, and a later `obsolete` never moves a watermark down. A release record needs no bookkeeping of its own. It is sealed once every entry it covers was admitted, so it lies in the object holding the last segment it covers or in a later one; collection deletes, in any order, only objects below the durable boundary; and recovery indexes only objects at or above the boundary of the tip, which every start object carries forward. So recovery indexes a release record whenever it indexes a segment the record covers, and once the record is below the boundary, so is every segment it covers, which recovery never indexes again, whether collection deleted it yet or not. A release in full also covers the entries a region would be assigned afterwards, so it is made only for a region that writes nothing more. When the batch of a release fails, the call fails, and the store carries the record again in the next batch it seals; while such a record waits, a timer tick seals it even when no entry is admitted, and a stop in the `durable` mode gives it one more create and reports the record lost when that create fails, so the release becomes durable once a create succeeds again. While the store is failing, as described under *Log store*, a release fails at once with the retryable error rejected appends carry, and so does a release whose batch is still pending when the store becomes failing; neither poisons the store. The record of the first is kept like that of a failed batch, but no tick seals it while the store is failing, so repeated releases add no batch to the backlog; the first tick after a create succeeds seals it, and a `stop` seals it with the backlog for its last attempt. The record of the second stays in the backlog. Both become durable once a create succeeds. A crash before that loses the record. Two windows therefore remain in which a dropped region is never released: a crash between the deletion of the table metadata and a durable release record, which includes a crash during the drop itself, and a crash after a failed release before any later create succeeds. Nothing retries the drop of such a region. The first window is a gap of the drop procedure rather than of the WAL, issue #9577 (https://github.com/GreptimeTeam/greptimedb/issues/9577): a crash during the drop leaves the region's data behind whatever the WAL provider. In the second window the drop completed except for its release, and a standalone region drop that fails is not retried. An operator sees `greptime_logstore_object_store_wal_indexed_objects` only grow after the restart while `deleted_objects_total` stays put, because the region's segments hold the reclaim boundary. Its remedy belongs in the drop procedure, for example an offline cleanup of a region that a resumed region drop finds not open, whose `obsolete_all` writes a durable release. Reclamation that consults the metadata for live regions, as Kafka's pruning by the metasrv does, would be a second safety net for both windows; it is left to garbage collection across nodes, see *Alignment with cluster mode*. Neither is part of this change. The object of the failed batch is seeded only when nothing was acknowledged after it, as the entries of a failed append are replayed. Every release record in an object the store indexes is counted in `greptime_logstore_object_store_wal_release_records_total`, records carried over from a failed batch included.
+
+A read never needs a deleted object, since entries at or below a region's watermark are hidden anyway, but a read that listed an object before it was collected skips the object when it gets there. A read that meets an error on such an object makes two observations, in the reverse of the order the actor writes them: it looks at the set of objects whose delete is in flight, and then at the catalog. The actor registers an object in that set before its delete starts, unindexes it if the delete succeeded, and only then takes it out, so an object absent from the set was either never collected, in which case the catalog still holds it, or is already unindexed by the time the catalog is read. Observing the catalog first would let a collection complete between the two observations and leave the read with an object that still looked indexed and no longer looked deleting. An object that is still in the set has an undecided delete, so the read waits for that one delete, a single object store request, rather than assume it succeeds: a delete that fails leaves the object present and indexed, and the error the read met is then reported as any other, `on_corrupted_segment` included. What the read waits for is the delete attempt its observation found, taken together with the lookup, not the object: a retry of the same object is a separate attempt, so the attempt the read holds releases it even while that retry is in flight, and a read is never handed from one attempt to the next. A read stream outlives the store it came from, so the actor settles every attempt it abandons when it exits, whether the store was stopped or dropped; the objects of those attempts stay present and indexed, so the reads waiting for them report the error they met. A collected object is skipped silently: it is not recorded as a hole and does not count as a skipped segment. An object the catalog still holds after those observations fails the read like any other fetch failure, `on_corrupted_segment` included. Within one run the catalog keeps the largest entry id of a region after the objects holding it were collected, so `latest_entry_id`, the durability wait and the `enqueued` clamp keep their reference point; a store opened later learns only what is listed, so after a restart `latest_entry_id` of a region whose later objects were collected is the largest id among the objects left, which only the pruning hint of Mito consumes. Reclaim boundaries are persisted in the headers and release records in the segments of the reserved region, no other state is persisted, and no configuration option is added.
 
 ## Log store
 
 `ObjectStoreLogStore` implements the `LogStore` trait for `Provider::ObjectStore`.
 
-**Construction and recovery.** The prefix, flush interval, batch size and backlog thresholds are validated, then the store lists the prefix and fetches, for up to eight objects at a time, the header and a window at the end of each object with range reads; a short object is read whole, and the footer is read separately only when it starts before the window. Each header and footer is verified as described under *Object format*; the first corrupted object fails construction; a fetch that finds the object gone is judged once the history is selected, as described below, and any other failed fetch abandons the rest. Segments are not read, so recovery costs a few small reads per object however large the objects are.
+**Construction and recovery.** The prefix, flush interval, batch size and backlog thresholds are validated, then the store lists the prefix and fetches, for up to eight objects at a time, the header and a window at the end of each object with range reads; a short object is read whole, and the footer is read separately only when it starts before the window. Each header and footer is verified as described under *Object format*; the first corrupted object fails construction; a fetch that finds the object gone is judged once the history is selected, as described below, and any other failed fetch abandons the rest. Segments are not read, except the segment of release records of each indexed object that holds one, with one range read, so recovery costs a few small reads per object however large the objects are.
 
 The store then selects the history it replays from the headers. An object's link holds when the object has no predecessor, when its predecessor is present and carries the epoch the link records, or when the predecessor's sequence is below the reclaim boundary the object carries, in which case the chain ends there whether or not the predecessor is present; above that boundary a missing predecessor breaks the link. An object is complete when every link on its chain holds, each judged with the boundary of the object carrying the link. The tip is the complete object with the largest epoch and, among those, the largest sequence. Only the objects on the tip's chain are indexed and replayed, in sequence order whatever the order the fetches completed in, so the catalog checks the entry ranges of every object against its predecessors like a sequential replay, and a conflicting object fails construction. Every other present object is an orphan: it is neither indexed nor read, but its sequence counts, so the next sequence is above every listed object, and the objects below the tip's boundary are collected. A listed object whose header or footer is gone by the time it is fetched is ignored when it lies below the tip's boundary and fails construction when it lies at or above it, since collection never deletes one there. Construction also fails when a listed object carries an epoch above the next sequence, which no instance can have produced, since an epoch is a start sequence plus one. When objects are present but none is complete, construction fails rather than start an empty history. Construction then records the largest durable entry id per region.
 
@@ -142,28 +145,36 @@ The mode also needs a barrier that the durable mode gets for free. An entry id t
 | create succeeds, or identical retry | advances | acknowledged in sequence order | healthy |
 | transient I/O error, `durable` mode | consumed and never reused, nor are the sequences of the other failed batches | this and every batch that is not indexed fail, including those whose objects were created; a create still in flight completes as an orphan | healthy: the next batch links to the last indexed object |
 | an object of an earlier epoch holds the sequence, `durable` mode | consumed and never reused | as for a transient I/O error | healthy |
-| transient I/O error, `enqueued` mode | unchanged | already acknowledged | healthy: the create is repeated after a short delay |
-| I/O error that requires intervention, `enqueued` mode | unchanged | already acknowledged; later appends are rejected with a retryable error until a create succeeds | failing: the lowest pending create is repeated alone with backoff; the first create that succeeds ends it |
-| any I/O error, `enqueued` mode after `stop` began | unchanged | already acknowledged | the backlog is dropped and `stop` reports the loss |
+| transient I/O error, `enqueued` mode | unchanged | already acknowledged, except a release, which waits for the repeat | healthy: the create is repeated after a short delay |
+| I/O error that requires intervention, `enqueued` mode | unchanged | already acknowledged, except a release, which fails with a retryable error while its record stays in the backlog; later appends and releases are rejected with that error until a create succeeds | failing: the lowest pending create is repeated alone with backoff; the first create that succeeds ends it |
+| any I/O error, `enqueued` mode after `stop` began | unchanged | already acknowledged, except a release, which fails with the stopped error | the backlog is dropped and `stop` reports the loss |
 | an object of the same or a later epoch holds the sequence, or any conflicting object in the `enqueued` mode | unchanged | every unacknowledged waiter fails, including those of batches whose objects were created but not yet indexed; an `enqueued` acknowledgement already returned cannot be revoked | poisoned |
 | encoding or catalog error | unchanged | every unacknowledged waiter fails, including those of batches whose objects were created but not yet indexed; an `enqueued` acknowledgement already returned cannot be revoked | poisoned |
 | create succeeds at the last representable sequence | cannot advance | acknowledged | poisoned: no later batch can be allocated a sequence, every later append fails |
 | the start object meets an object of an earlier epoch | moves to the next sequence | none admitted yet | construction continues |
 | the start object meets an object of the same or a later epoch, or its create has an unknown outcome | unchanged | none admitted yet | construction fails; an unknown outcome is retryable |
 
-A store cannot be constructed on a prefix whose largest object already carries the maximum sequence. A poisoned store fails every `LogStore` operation except `stop` with the same terminal error; in particular `obsolete` can no longer move a watermark. In the `durable` mode transient errors are not retried inside the store, but a failed create is reconciled once: the store immediately reads the key back, accepts the batch if the object exists with identical bytes, treats it as a conflict if the bytes differ, and reports the transient error only when the key is absent or that read fails too. So a lost response alone does not fail the append. When the append does fail, neither the store nor Mito keeps the batch: it surfaces through Mito to the caller of the write, and a retry is a new write by the caller under a new sequence. If the create had in fact succeeded and both its response and the read-back were lost, or its request lands only later, the object is an orphan as soon as a later object is acknowledged. Retrying the batch itself inside the store in the `durable` mode would only hide the latency and would hold writes up for as long as an outage lasts; the `enqueued` mode repeats the create with the same bytes, so a create that succeeded without a response is accepted as an identical retry.
+A release is a waiter of the batch that carries its record in both modes: it returns when the batch is indexed and fails whenever the batch fails. A store cannot be constructed on a prefix whose largest object already carries the maximum sequence. A poisoned store fails every `LogStore` operation except `stop` with the same terminal error; in particular `obsolete` can no longer move a watermark. In the `durable` mode transient errors are not retried inside the store, but a failed create is reconciled once: the store immediately reads the key back, accepts the batch if the object exists with identical bytes, treats it as a conflict if the bytes differ, and reports the transient error only when the key is absent or that read fails too. So a lost response alone does not fail the append. When the append does fail, neither the store nor Mito keeps the batch: it surfaces through Mito to the caller of the write, and a retry is a new write by the caller under a new sequence. If the create had in fact succeeded and both its response and the read-back were lost, or its request lands only later, the object is an orphan as soon as a later object is acknowledged. Retrying the batch itself inside the store in the `durable` mode would only hide the latency and would hold writes up for as long as an outage lasts; the `enqueued` mode repeats the create with the same bytes, so a create that succeeded without a response is accepted as an identical retry.
 
-**Stopping.** `stop` is idempotent and awaits the actor. Once stop has begun, queued appends are not admitted, no timer or size triggered seal starts, and appends after stop, including empty ones, fail with the stopped error. In the `durable` mode the open batch and every sealed batch whose create has not started are dropped: their waiters receive the stopped error. A create that is already in flight runs to completion; if it succeeds while its batch is still pending, the batch is indexed in order and acknowledged. A transient failure after stop began fails every batch that is not indexed, as it does while the store runs, and those batches receive the stopped error; a create of such a batch that completes later is an orphan and acknowledges nothing. A create that finds a conflicting object of the same or a later epoch after stop began poisons the store as it would while the store runs, but the waiters of the batches that are not indexed receive the stopped error, the conflict is kept only as the terminal error, and `stop` returns successfully once the creates in flight have completed. In the `enqueued` mode the held-back appends receive the stopped error, the open batch is sealed and the whole backlog is uploaded before `stop` returns, by a failing store too; any object store failure in that phase drops the backlog and `stop` returns the error, as described under *Acknowledgement*. A durability wait still pending when the actor exits receives the stopped error, and a delete already in flight runs to completion before `stop` returns.
+**Stopping.** `stop` is idempotent and awaits the actor. Once stop has begun, queued appends are not admitted, no timer or size triggered seal starts, and appends after stop, including empty ones, and releases fail with the stopped error. In the `durable` mode the release records of failed batches and of the batches dropped here are sealed into one more object, which is created before `stop` returns; `stop` reports the error when that create fails too. In the `durable` mode the open batch and every sealed batch whose create has not started are dropped: their waiters receive the stopped error. A create that is already in flight runs to completion; if it succeeds while its batch is still pending, the batch is indexed in order and acknowledged. A transient failure after stop began fails every batch that is not indexed, as it does while the store runs, and those batches receive the stopped error; a create of such a batch that completes later is an orphan and acknowledges nothing. A create that finds a conflicting object of the same or a later epoch after stop began poisons the store as it would while the store runs, but the waiters of the batches that are not indexed receive the stopped error, the conflict is kept only as the terminal error, and `stop` returns successfully once the creates in flight have completed, unless the poisoning drops release records, which `stop` reports by returning the conflict. In the `enqueued` mode the held-back appends receive the stopped error, the open batch is sealed and the whole backlog is uploaded before `stop` returns, by a failing store too; any object store failure in that phase drops the backlog and `stop` returns the error, as described under *Acknowledgement*. A durability wait still pending when the actor exits receives the stopped error, and a delete already in flight runs to completion before `stop` returns.
 
-**Reading.** `read(provider, entry_id)` returns the region's entries with id at least `entry_id`, in order, fetching only that region's segments with range reads and decoding nothing else, and hides entries at or below the region's obsolete watermark. `obsolete` records the watermark in memory, clamped to the durable entry id in the `enqueued` mode; the watermark is re-established after a restart because Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, so no WAL-owned watermark state is persisted. `obsolete_all` hides the whole region. `latest_entry_id` is not affected by the watermark. Every method taking a provider requires `Provider::ObjectStore` with the store's prefix and the caller's region.
+**Reading.** `read(provider, entry_id)` returns the region's entries with id at least `entry_id`, in order, fetching only that region's segments with range reads and decoding nothing else, and hides entries at or below the region's obsolete watermark. `obsolete` records the watermark in memory, clamped to the durable entry id in the `enqueued` mode; the watermark is re-established after a restart because Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, so the store persists no watermark of a region Mito opens again. The release of a region, which `obsolete_all` and `release_region` make, is persisted as described under *Entry ids*. `obsolete_all` hides the whole region. `latest_entry_id` is not affected by the watermark. Every method taking a provider requires `Provider::ObjectStore` with the store's prefix and the caller's region.
 
-**Metrics.** The store exports metrics under `greptime_logstore_object_store_wal_`. The write path reports `seal_to_durable_seconds`, the time from sealing a batch until its object is durable, `append_ack_seconds`, the time an append waits for its acknowledgement in the `durable` mode, `object_bytes` and `object_entries`, the size and the entry count of every object sealed, and `created_objects_total`, the creates that succeeded, an identical retry included; a create is counted and timed whether or not its batch is still there to be indexed, since an object a poisoned store gave up on is as durable as any other. `indexed_objects` and `indexed_bytes` are the objects and the bytes the catalog holds, set by a recovery that succeeded and moved by every object indexed or unindexed afterwards, so a collection takes the object it deleted off both, and a recovery that is abandoned publishes neither. A recovery that succeeds reports `recovery_seconds` and the objects it read footers from in `recovered_objects_total`, both of the same recoveries, so one that is abandoned at a fetch or rejected by the catalog contributes to neither; a read stream reports `read_seconds` once it has been read to its end, and one that fails or is dropped part way reports nothing. The failure paths are counted: `create_conflicts_total` for a create that found a conflicting object, `create_failures_total` for one the object store did not confirm, `poisoned_total` for a store that recorded a terminal error, `skipped_segments_total` for a segment a read skipped, and `stalled_appends_total` with `stalled_append_seconds` for an append held back because the sealed batches were full, in either mode, or by the `enqueued` backlog thresholds, timed until it is admitted or refused, a stop, a terminal error, a failing store or the teardown of the actor releasing it included. The gauge `failing` counts the stores whose backlog the object store refuses with an error that requires intervention, and `rejected_appends_total` the appends such a store rejected; the attempts of a failing store are counted in `create_failures_total`. The delete counters of a collection are described under *Entry ids*. The histogram buckets are chosen for the default flush interval of 100 ms and the default `max_batch_bytes` of 8 MiB.
+**History check.** A region persists its prefix, not the object store that holds the prefix, so pointing `storage_provider`, or the bucket or root of the provider it names, at another location, or deleting the objects under the prefix, would let every region open against a prefix without its history, replay nothing and lose its acknowledged writes that were not flushed. Two invariants of a prefix expose such a location without any persisted state of their own. A prefix that a store opened is never empty afterwards, unless the store withdrew its start object as described below: the store writes its start object before it admits an append, and collection deletes only objects below the durable boundary, the boundary of the last indexed object, which is at or below that object's own sequence, since a larger one fails the header check. And a watermark a region persisted names an object at or below the newest object holding entries that any later recovery of the prefix indexes: the entry it names was durable and indexed when the watermark was persisted, as described under *Acknowledgement*; no durable boundary passes the newest object holding entries, since a batch carries at most the sequence after the last indexed object and a start object carries the boundary recovery found, so collection keeps that object and recovery indexes it on the chain of the tip. The segment of release records counts as entries: it never holds the reclaim boundary, so an object holding only release records can be the one that collection keeps while it deletes every older object. Objects without segments, start objects and the like, do not move it. `LogStore::check_region_history(provider, entry_id)` therefore reports the history of the provider's region as missing when recovery found no object under the prefix, or when `entry_id` names an object above the newest object holding entries that recovery indexed, unless the catalog holds the region's entries through `entry_id`, as it does for a region written since the store opened, or the region's history starts in the store: Mito calls `LogStore::start_region_history` when it creates a region, before it persists the region's manifest, and an open that accepted a missing history records the region the same way, in memory, so the region reopens on the same store without any entry and is counted once. Both methods default to accepting and to doing nothing, so Raft Engine and Kafka are unchanged. Mito calls it before it replays an existing region from its persisted state, on open, on catch-up and when a create finds the region already present, with the larger of the manifest's `flushed_entry_id` and `truncated_entry_id`; a region whose options skip the WAL keeps no history there and is not checked, nor is a region with the noop provider. A missing history fails the open, and such a create, which does not fall back to creating the region over its manifest, with an `InvalidArguments` error that names the region, the prefix, the configured storage provider with its scheme, bucket and root, and the two ways forward below; in standalone a region that fails to open stops startup. The check runs before Mito calls `obsolete` with the watermark, so the sequence floor never hides a missing history.
+
+A refused open must not make the next start pass. When the store found its prefix empty and refuses a region, it gives up the start object it wrote: the actor agrees only while nothing was ever handed an id, no batch, append, release or create waits, no failed release is kept for a later object, recovery listed no other object, the catalog holds only the start object and no region's history started in the store, and from then on the store records a terminal error and accepts nothing, since a later object would link to the given-up one. A release seals its own batch without handing an id, so once its object is indexed the catalog shows it, alongside the start object or alone after collection deleted the start object; the actor deletes the object like a collected one, so `stop` waits for that delete and a cancelled caller does not abandon it, and every refusal returns its error only once the delete settled. A region registers its history and the actor decides the withdrawal under one lock, so a region created at that moment either prevents the withdrawal or fails to register, and then fails its create before it leaves a manifest that a later start would refuse. Another instance may open the prefix while the delete is in flight, as when two processes of the same node overlap: its recovery finds a chain without any segment, so its start object reclaims every object below it, and the delete, which may land after that start object, leaves its chain complete. The terminal error is not counted in `poisoned_total`. In standalone, regions open before the instance serves, so at that point nothing else can have written to the prefix, and the next start, an operator retry or an orchestrator restart, finds the prefix empty again and refuses the same regions. A refused start on a prefix that held objects writes a start object, which holds no entry and so does not move the newest object holding entries: later starts refuse the same watermarks.
+
+To move the WAL to another location, stop the instance, copy every object under the prefix to the same keys in the new location, and then change the configuration: recovery finds the same objects and every region opens with all of its acknowledged writes. When the old objects are lost, `accept_missing_history = true` turns the failure into an accepted loss for that start: the region opens without its writes that were not flushed, a warning names the region, the prefix, the location and the watermark above which nothing was replayed, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts the region. The option is meant to be turned off again once the instance has written to the WAL after that start: until a write creates an object holding entries at or above the sequence the accepted watermark names, which the floor raised by the accepted open makes the next write do, a later start reports such a region again.
+
+The check does not catch every change of location. A location whose prefix already holds objects passes a region that never flushed, and a region whose watermark names an object at or below the newest object holding entries there, whatever history those objects hold. A refused start on an empty prefix leaves its start object behind when the process ends before the delete completes or when the delete fails, which is logged; the next start then finds a prefix with only start objects and passes a region that never flushed, while it still refuses every region whose watermark names an object. The same holds when the store created a region before the refusal. When it admitted an append or a release before the refusal, as it can when regions open while the instance already serves, the store withdraws nothing and the prefix keeps an object holding entries, which may already have collected the start object; the next start then also passes every region whose watermark names an object at or below that one.
+
+**Metrics.** The store exports metrics under `greptime_logstore_object_store_wal_`. The write path reports `seal_to_durable_seconds`, the time from sealing a batch until its object is durable, `append_ack_seconds`, the time an append waits for its acknowledgement in the `durable` mode, `object_bytes` and `object_entries`, the size and the entry count of every object sealed, and `created_objects_total`, the creates that succeeded, an identical retry included; a create is counted and timed whether or not its batch is still there to be indexed, since an object a poisoned store gave up on is as durable as any other. `indexed_objects` and `indexed_bytes` are the objects and the bytes the catalog holds, set by a recovery that succeeded and moved by every object indexed or unindexed afterwards, so a collection takes the object it deleted off both, and a recovery that is abandoned publishes neither. A recovery that succeeds reports `recovery_seconds` and the objects it read footers from in `recovered_objects_total`, both of the same recoveries, so one that is abandoned at a fetch or rejected by the catalog contributes to neither; a read stream reports `read_seconds` once it has been read to its end, and one that fails or is dropped part way reports nothing. The failure paths are counted: `create_conflicts_total` for a create that found a conflicting object, `create_failures_total` for one the object store did not confirm, `poisoned_total` for a store that recorded a terminal error, `skipped_segments_total` for a segment a read skipped, `missing_history_accepted_total` for a region that `accept_missing_history` let open without its history, and `stalled_appends_total` with `stalled_append_seconds` for an append held back because the sealed batches were full, in either mode, or by the `enqueued` backlog thresholds, timed until it is admitted or refused, a stop, a terminal error, a failing store or the teardown of the actor releasing it included. The gauge `failing` counts the stores whose backlog the object store refuses with an error that requires intervention, and `rejected_appends_total` the appends and releases such a store rejected; the attempts of a failing store are counted in `create_failures_total`. The delete counters of a collection and the release record counter are described under *Entry ids*. The histogram buckets are chosen for the default flush interval of 100 ms and the default `max_batch_bytes` of 8 MiB.
 
 A `testing` cargo feature exposes hooks to wait for admitted appends, to seal the open batch, to hold and release creates and to make creates fail, so engine tests can flush deterministically and observe entries that are admitted but not durable.
 
 ## Mito integration
 
-Mito maps `WalOptions::ObjectStore { prefix }` to `Provider::object_store_provider(region_id, prefix)`; resolving object store options against a Raft Engine or Kafka log store reports the existing incompatible-provider error. Reads go through the region-scoped reader used for Kafka, so a namespace shared by many regions never leaks entries across regions even if a log store returned them. Object store regions are opened individually rather than grouped by topic. After replay, `topic_latest_entry_id` follows the existing remote-WAL rule: it is taken from the store's latest entry id only when the region's memtables are empty, otherwise it stays at the replay start, because that value feeds pruning and must never advance past unflushed data. Before a flush, a full truncate or a discard of unflushed data publishes a manifest watermark, Mito waits on the durability barrier described under *Acknowledgement*; with the `durable` mode and with every other provider the wait returns at once.
+Mito maps `WalOptions::ObjectStore { prefix }` to `Provider::object_store_provider(region_id, prefix)`; resolving object store options against a Raft Engine or Kafka log store reports the existing incompatible-provider error. Reads go through the region-scoped reader used for Kafka, so a namespace shared by many regions never leaks entries across regions even if a log store returned them. Object store regions are opened individually rather than grouped by topic. After replay, `topic_latest_entry_id` follows the existing remote-WAL rule: it is taken from the store's latest entry id only when the region's memtables are empty, otherwise it stays at the replay start, because that value feeds pruning and must never advance past unflushed data. Before a flush, a full truncate or a discard of unflushed data publishes a manifest watermark, Mito waits on the durability barrier described under *Acknowledgement*; with the `durable` mode and with every other provider the wait returns at once. When Mito drops a region it calls `release_region` after `obsolete`, as described under *Entry ids*; the default of that method does nothing, so the other providers handle a dropped region as before. Neither a release nor an `obsolete` of an object store region that fails, the latter for instance on a store that the release of another region poisoned, fails the drop, since nothing sends the drop of a removed region again and a caller that drops several regions, such as the metric engine with its data and metadata regions, would leave the others undone: the region's runtime state is cleaned up and its files are scheduled for deletion as for any drop, and the failure is logged; an `obsolete` that fails for another provider fails the drop as before. A store that is neither stopped nor poisoned keeps the record and writes it with its next object; a stopped or poisoned store keeps none, and the region stays unreleased after a restart. The store carries the record in a later object, as described under *Entry ids*; a crash before any later create succeeds loses it, and the dropped region's segments then hold the reclaim boundary after the restart. Before Mito replays an existing region on open, on catch-up or for a create that finds the region present, it runs the history check described under *Log store*.
 
 ## Datanode wiring and lifecycle
 
@@ -208,6 +219,10 @@ max_unpersisted_age = "8s"
 # regions of the object are unaffected (the default). "fail": the read
 # fails, so the region does not open.
 on_corrupted_segment = "skip"
+# Whether a region opens although the prefix does not hold its WAL
+# history, losing its writes that were not flushed; see the history
+# check under *Log store*. Meant for one start after the old WAL is lost.
+accept_missing_history = false
 ```
 
 `config/config.md` is generated from the example files.
@@ -215,11 +230,13 @@ on_corrupted_segment = "skip"
 ## Compatibility
 
 - `WalOptions::ObjectStore` is a new persisted variant. Existing `raft_engine`, `kafka` and `noop` options decode unchanged; a region's options never change provider implicitly.
-- The object format is version 1 and frozen once the provider is released; a version bump after that is a new decoder, not an in-place change. The version 1 header layout was revised before release to carry the epoch, the link, the reclaim boundary and the header CRC. No object in an earlier layout exists, since the provider has not been enabled anywhere.
+- The object format is version 1 and frozen once the provider is released; a version bump after that is a new decoder, not an in-place change. The version 1 header layout was revised before release to carry the epoch, the link, the reclaim boundary and the header CRC. No object in an earlier layout exists, since the provider has not been enabled anywhere. Release records add no format version: they are the segment of a reserved region in a version 1 object, so a build that predates them still opens the prefix. Such a build indexes that segment as a region it never opens, which holds its reclaim boundary at the first release until the release records are understood again. A frozen object with release records under `tests/compatibility/wal/object_store` is checked byte for byte by the format tests; the SQL compatibility runner owns `wal.provider` and does not cover this provider.
 - The configuration tag carries `experimental_`; the persisted tag does not, so promoting the provider to stable is a configuration rename with an alias, not a metadata migration.
 - `LogStore::wait_durable` has a default implementation that returns at once, so Raft Engine, Kafka and the noop provider keep their own acknowledgement and persistence semantics; the durability barrier binds only the object store provider.
-- The object-sequence-major id scheme needs no migration of existing objects. A prefix written under the contiguous scheme keeps its objects, and the sequence allocation rule under *Entry ids* keeps new ids above the old ones. Garbage collection reasons from footers and from the reclaim boundary, not from the id encoding, so it applies to objects of either scheme.
+- `LogStore::release_region` has a default implementation that does nothing, so Raft Engine, Kafka and the noop provider keep their handling of a dropped region.
+- The object-sequence-major id scheme is the only one the provider has written, so no prefix needs a migration; the contiguous scheme of the pre-release reference implementation has no deployed prefixes, and the history check relies on every id naming its object.
 - `on_corrupted_segment` is a new configuration option with a default, so existing configurations decode unchanged; it changes what a read does with a damaged segment, not what is written.
+- `accept_missing_history` is a new configuration option with a default of `false`, so existing configurations decode unchanged. `LogStore::check_region_history` has a default implementation that accepts and `LogStore::start_region_history` one that does nothing, so only the object store provider checks a region's history, and the check reads only the recovered catalog: no object format or region metadata changes. A region that previously opened against a prefix without its history, replaying nothing, now fails to open until its objects are restored or the loss is accepted.
 
 ## Evidence
 
@@ -236,14 +253,15 @@ The reference implementation was validated on the following evidence; the tests
 | A manifest watermark never names an entry that is not durable | Durability barrier tests: a flush held until its entry is durable; a crash in `enqueued` mode after an acknowledgement and a flush attempt but before the object exists, followed by a restart, new writes and a second crash; a drop cancelling a flush that waits; a truncate that waits; a flush after a backlog was lost. | Every durable entry replays; the flush after the lost backlog publishes no frontier; `obsolete` never passes the durable id. |
 | A region replays only its own entries | The two-region recovery test above. | Per-region row-level scan equality with the other region unflushed. |
 | Recovery never silently loses data | Unit coverage of every corruption class at recovery and on read; footer-only recovery compared with a full decode; a footer longer than the tail window. | Structural damage fails construction of the store; a corrupted segment is skipped, recorded as a hole of its region and counted, or fails the read under `fail`; the footer-only catalog equals the full decode. |
-| A region cannot be silently pointed at another WAL | Reopen on a mismatched prefix; reopen on an empty prefix. | The mismatch is rejected with both prefixes named and the region is not opened; the empty prefix opens cleanly with the expected row counts. |
+| A region cannot be silently pointed at another WAL | Reopen on a mismatched prefix; open and catch-up against a location whose prefix is empty, and open against one whose prefix holds objects only below a flushed or a truncated watermark; a copy of every object on a new location; a restart after collection left only the newest object; the option accepting the loss; store tests of the same cases. | The mismatch is rejected with both prefixes named; a missing history fails the open and the catch-up with the region, the prefix and the location named, while a region that skips the WAL, a region whose history the location holds and a new region still open; the copy and the restart open with every row; the option opens the region without its unflushed rows and counts it once. |
 | Garbage collection deletes everything below the persisted reclaim boundary, and recovery stays correct while objects disappear | Unit tests of the boundary computation (a region without a watermark holds it at its first object, a watermark inside an object keeps the object, `obsolete_all` releases it, last indexed plus one when everything is flushed), of stamping and durability (headers carry the boundary current at sealing, a failed create's boundary permits nothing, the start object carries the recovered boundary), of deletion (chain objects, listed orphans and failed batches below the durable boundary, nothing at or above it, a failed delete retried, `stop` waiting for deletes in flight, holes dropped), of recovery (two restarts with a collection between them, a late object of an earlier epoch below the boundary collected, a missing predecessor above the boundary still failing) and of a read that listed an object before it was collected; the Mito flush-then-collect test in both acknowledgement modes; the standalone integration test's assertion after a flush. | Objects below the durable boundary are gone after a flush, the start object included; a restart replays everything above the boundary; exactly one WAL object remains after a flush and one more insert in the integration test. |
-| Every row of the *Failure matrix* holds, and every metric moves as documented | A table-driven fault matrix, `test_store_fault_matrix`, that drives every failure the object store access can produce through the store and asserts, per case, which waiters fail and with what error, whether the store is poisoned, whether the sequence is consumed, the indexed gauges at the failure itself, the exact movement of every metric, and what a read of the store sees afterwards; every timer is observed twice around two forced waits whose intervals cannot overlap. | 36 cases: 19 for the eleven rows of the matrix and 17 for the failures listed under *Failures driven through the store*. The test fails if a row of the matrix is left uncovered. |
+| A dropped region holds no object after a restart that never opens it | Unit tests of the release record an object carries and the boundary it allows; of recovery seeding the watermark from the release record before any boundary is computed, with collection proceeding once the remaining region is flushed; of a failed release carried by a timer tick on an idle prefix and by a `durable` stop, which reports the record lost when its create fails too; of the release of a region without segments, and of a region id used again after its release; of a release record that outlives a restart together with the segment it covers and goes once both are below the boundary, deleted out of order; the fault-matrix cases of a release, which return only once its object is durable and indexed, fail with its create, and fail at once while the store is failing; the Mito test that drops one of two regions, restarts gracefully, abruptly, or after a release that failed, with nothing written in between, then flushes the other region and writes once more; the metric engine test of a physical region whose data and metadata regions wrote to the WAL, dropped while the release of the data region fails, while both releases fail, and while the release of the data region meets a conflicting object that poisons the store, which still drops both regions and forgets the physical region, and, in the first two cases, after which a store opened later keeps no segment of either; the frozen object with release records. | Recovery seeds the release, and after the flush and the next write every object written before the restart is deleted. |
+| Every row of the *Failure matrix* holds, and every metric moves as documented | A table-driven fault matrix, `test_store_fault_matrix`, that drives every failure the object store access can produce through the store and asserts, per case, which waiters fail and with what error, whether the store is poisoned, whether the sequence is consumed, the indexed gauges at the failure itself, the exact movement of every metric, and what a read of the store sees afterwards; every timer is observed twice around two forced waits whose intervals cannot overlap. | 42 cases: 24 for the eleven rows of the matrix, five of them for the create of a release record, and 18 for the failures listed under *Failures driven through the store*. The test fails if a row of the matrix is left uncovered. |
 | Recovery cost does not grow with object size | Footer-only recovery over 2,000 objects; recovery concurrency bounded and ordered by sequence. | Construction of the store fell from 3.6 s to 0.9 s. |
 
 ### Failures driven through the store
 
-- Conditional create outcomes: success, identical retry, conflicting content from an earlier epoch and from the same or a later one, lost response, transient failure in `durable` mode with later objects already created and with creates still in flight (every batch that is not indexed fails, no sequence is reused, late completions are orphans), transient failure in `enqueued` mode and after `stop` began, a failure that requires intervention in `enqueued` mode and a `stop` while it lasts, completion after a conflict had already poisoned the store, a create at the last representable sequence.
+- Conditional create outcomes: success, identical retry, conflicting content from an earlier epoch and from the same or a later one, lost response, transient failure in `durable` mode with later objects already created and with creates still in flight (every batch that is not indexed fails, no sequence is reused, late completions are orphans), transient failure in `enqueued` mode and after `stop` began, a failure that requires intervention in `enqueued` mode and a `stop` and releases while it lasts, completion after a conflict had already poisoned the store, a create at the last representable sequence.
 - Object store access: an unwritable store; a listing, a trailer fetch, a footer fetch, a segment fetch and a delete that fail; a delete that fails and is retried by the next collection; an object collected between the listing and its fetch, below and at the tip's reclaim boundary; an object store that answers slowly.
 - Store state: an object the catalog rejects, per-region entry id exhaustion, the catalog refusing a next sequence past the maximum, every stop path with several creates in flight that succeed, fail transiently or conflict after stop began, a stop with a delete in flight, an append held back by full sealed batches, and the appends the backlog thresholds were holding back released by admission, by a stop, by a terminal error and by the teardown of the actor.
 - Pipelined uploader: concurrent creates acknowledged in sequence order, every sealed batch linked to its pipeline predecessor, the open batch sealing at the threshold while creates are in flight, a transient failure failing every batch that is not indexed, created ones included, a permanent failure poisoning the store.
@@ -322,7 +340,9 @@ Simpler catalog, but every region then produces its own timer-driven object, so
 
 - Write latency under the `durable` mode is the time until the batch seals (up to `flush_interval`, 100 ms by default), plus waiting for every earlier object, plus one PUT. A serial writer waits for the next seal, so its extra latency over Raft Engine is the interval minus the base insert cost modulo the interval, and concurrency does not close the gap: measured within one session on a laptop it was about 23% of throughput at concurrency 1 and about 28% at concurrency 8, and on S3 against Raft Engine on a local NVMe store the `durable` mode reached 31% of Raft Engine's throughput at concurrency 1 and 54% at concurrency 8, a serial writer being bound to one batch per interval, while the `enqueued` mode matched it, see *Write path measurements* and *Cloud write path measurements*.
 - Every object costs a request. Timer-triggered sealing creates at most one object per interval per node, ten per second at the default; the `enqueued` mode can seal earlier when the unpersisted backlog reaches a threshold, and the batch seals on size only when the admitted bytes reach `max_batch_bytes` before the next tick. Under sustained load on S3 the timer ceiling was reached, about 600 creates per minute per node, about $0.18 per node-hour at the S3 standard list price.
-- Garbage collection keeps every object at or above the reclaim boundary, so a region that is written but rarely flushed, or never flushed, holds the boundary at its first object and keeps every later object the prefix shares with other regions; recovery time grows with the number of retained objects. A prefix that receives no appends cannot persist a higher boundary, so an idle prefix is collected only after its next object.
+- Garbage collection keeps every object at or above the reclaim boundary, so a region that is written but rarely flushed, or never flushed, holds the boundary at its first object and keeps every later object the prefix shares with other regions; recovery time grows with the number of retained objects. A prefix that receives no appends cannot persist a higher boundary, so an idle prefix is collected only after its next object; a release seals that object itself.
+- A dropped region stays unreleased, and holds the reclaim boundary for good, after a crash between the deletion of its table's metadata and a durable release record, or after a crash that follows a failed release before any later create succeeds; `indexed_objects` then only grows. The first window is a gap of the drop procedure for every WAL provider, issue #9577. See *Entry ids*.
+- A release costs one object of its own and waits for every object sealed before it, in the `enqueued` mode as well, so dropping a region waits for the uploads ahead of its release record; when they fail, or the store is stopped or poisoned, the drop succeeds all the same and logs the failure; a stopped or poisoned store leaves the region unreleased.
 - Every open of the store writes one start object without segments, one extra PUT per open, and every object carries a 54-byte header. Orphans, the objects of failed batches that were stored anyway, hold storage together with the rows Mito wrote again after those failures until garbage collection deletes them.
 - Object-sequence-major entry ids consume 20 bits per object for positions, which caps a region at about a million entries per object; the batch seals when the cap is reached. Sequence numbers are 44 bits wide as a result, enough for one object per second for half a million years.
 
@@ -338,18 +358,19 @@ Distributed mode is out of scope, but several decisions were taken so that the c
 
 Sealing may only exclude objects whose writes were never acknowledged, which takes two fences. Admission is fenced by the lease mechanism Mito already has: when a region's lease expires the datanode demotes it to a follower, which refuses writes, and the metasrv waits for that deadline before it upgrades the candidate. Completion is fenced by the store: every conditional create runs under an upload deadline, a create that has not completed by its deadline is reported failed to its waiters and is never acknowledged afterwards even if the object turns out to exist, and when a region's lease expires the store fails every waiter of that region that is not yet acknowledged, so that a create still in flight at that moment cannot be acknowledged later. The metasrv seals the segment only after the lease deadline plus the upload deadline have both passed, so an object that appears after sealing belongs to a create the old writer had already reported as failed. Under the durable acknowledgement mode this means a sealed set never excludes an acknowledged write; under the `enqueued` mode the entries in the old writer's unpersisted backlog at that moment are lost, which is that mode's stated recovery point objective. The late-completing create is part of the acceptance criteria for takeover, stated in terms of the sealing instant: an object that was absent at sealing, whether it fills a hole later or lies above the cutover, is excluded on every open of the region, and so is a visible object outside the selected chain; an object on the sealed chain replays on every open even if its write was never acknowledged, which is the same outcome as a crash between an object's creation and its acknowledgement; and in the durable mode every caller whose create had not completed when the lease expired must have been failed, so no acknowledged write is among the excluded objects. The generation in the key is an additional guarantee that such a late object lands in a directory the new writer never reads beyond the sealed set. Object-sequence-major entry ids are what make this cheap: `entry_id >> 20` locates the object under the right prefix without the source node's in-memory catalog.
 
-**Entry id floor across prefixes.** A region that arrives under a new prefix, by migration or by failover, brings a watermark and replayed ids that were assigned under another prefix's sequence. Before the target assigns the region its first new id, the target store raises its next object sequence above `max_id >> 20`, where `max_id` is the largest of the region's inherited watermark and every id it replayed, so that every new id of that region is greater than every id it already has; sequence gaps are tolerated and the sequence space is wide enough for this. The rule is applied on every open from the region's persisted state, in the same place Mito already hands the store the manifest watermark, so it needs no persisted state of its own and survives a crash before the target's first flush: the restart re-derives the floor and the region's acknowledged-but-unflushed entries under the target prefix are above the inherited watermark and replay.
+**Entry id floor across prefixes.** A region that arrives under a new prefix, by migration or by failover, brings a watermark and replayed ids that were assigned under another prefix's sequence. Before the target assigns the region its first new id, the target store raises its next object sequence above `max_id >> 20`, where `max_id` is the largest of the region's inherited watermark and every id it replayed, so that every new id of that region is greater than every id it already has; sequence gaps are tolerated and the sequence space is wide enough for this. The rule is applied on every open from the region's persisted state, in the same place Mito already hands the store the manifest watermark, so it needs no persisted state of its own and survives a crash before the target's first flush: the restart re-derives the floor and the region's acknowledged-but-unflushed entries under the target prefix are above the inherited watermark and replay. The history check under *Log store* stays compatible with this: it judges a watermark against the prefix a region persists, and a region taken over from another prefix carries that prefix as an earlier segment of its `WalOptions`, so its inherited watermark is judged against the segment that assigned it rather than against the target prefix, and the floor, not the check, applies to it on the target.
 
 **Acknowledgement modes.** Both are implemented. Cluster mode is expected to run `enqueued` for latency-sensitive workloads; the object layout, the flush durability barrier and the recovery path are identical in both modes.
 
 **Corruption.** Segment-level skipping with the hole recorded per region is implemented, see *Object format*. *Proposed*: cluster mode routes the WAL hole event to the metasrv so that a follower or a takeover knows the region needs a fresh replica rather than a replay.
 
-**Garbage collection across nodes.** Within one prefix, garbage collection is implemented as described under *Entry ids*: watermark from each region's manifest, per-segment footer comparison, a reclaim boundary persisted in every header. A source prefix that still holds segments of regions which have moved away cannot be trimmed by the source alone, because it no longer learns those regions' watermarks. Either the target reports to the metasrv, after each flush, the source sequence below which it no longer needs the region, and the metasrv drives deletion, or a cluster-level janitor collects the watermarks; which of the two is the last of the *Unresolved questions*.
+**Garbage collection across nodes.** Within one prefix, garbage collection is implemented as described under *Entry ids*: watermark from each region's manifest, per-segment footer comparison, a reclaim boundary persisted in every header. A source prefix that still holds segments of regions which have moved away cannot be trimmed by the source alone, because it no longer learns those regions' watermarks. Either the target reports to the metasrv, after each flush, the source sequence below which it no longer needs the region, and the metasrv drives deletion, or a cluster-level janitor collects the watermarks; which of the two is the last of the *Unresolved questions*. Either way reclamation consults the metadata for the regions that still exist, as Kafka's pruning by the metasrv does, which also lets go of a dropped region whose release never became durable, see *Entry ids*.
 
 # Future work
 
 1. Probe the next sequence with a GET from the last known sequence instead of listing the whole prefix at recovery; on a prefix with many objects the probe is cheaper than the listing.
 2. Distributed mode, which needs metasrv-side allocation of node ids and generations, the takeover chain in `WalOptions`, read-only replay of a foreign prefix, and cross-node garbage collection, as sketched under *Alignment with cluster mode*.
+3. Close the crash window of the drop procedure, issue #9577 (https://github.com/GreptimeTeam/greptimedb/issues/9577), which also leaves the region's WAL unreleased, see *Entry ids*.
 
 # Unresolved questions
 

grafana/dashboards/metrics/cluster/dashboard.json

@@ -9112,6 +9112,132 @@
           "title": "Read Stage P99 and Avg per Instance",
           "type": "timeseries"
         },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${metrics}"
+          },
+          "description": "Estimated maximum concurrent files by instance and type (query or compaction). Query samples are region scan limit evaluations, including memtable sources counted by the limit, rejected scans, and scans using memory budgeting. Compaction samples count all input files when building a compaction reader. A SQL query can produce multiple region scan evaluations. P95/P99 are histogram estimates, not exact per-query maxima or live open-file counts.",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisBorderShow": false,
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "barWidthFactor": 0.6,
+                "drawStyle": "line",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green"
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short",
+              "min": 0
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 8,
+            "x": 8,
+            "y": 154
+          },
+          "id": 565,
+          "options": {
+            "legend": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "displayMode": "table",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "hideZeros": false,
+              "mode": "multi",
+              "sort": "desc"
+            }
+          },
+          "pluginVersion": "11.6.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "histogram_quantile(0.95, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket{instance=~\"$datanode\"}[$__rate_interval])))",
+              "instant": false,
+              "legendFormat": "[{{instance}}]-[{{pod}}]-{{type}}-p95",
+              "range": true,
+              "refId": "A"
+            },
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "histogram_quantile(0.99, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket{instance=~\"$datanode\"}[$__rate_interval])))",
+              "instant": false,
+              "legendFormat": "[{{instance}}]-[{{pod}}]-{{type}}-p99",
+              "range": true,
+              "refId": "B"
+            },
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_sum{instance=~\"$datanode\"}[$__rate_interval])) / sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_count{instance=~\"$datanode\"}[$__rate_interval]))",
+              "instant": false,
+              "legendFormat": "[{{instance}}]-[{{pod}}]-{{type}}-avg",
+              "range": true,
+              "refId": "C"
+            }
+          ],
+          "title": "Scan Max Concurrent Files per Instance",
+          "type": "timeseries"
+        },
         {
           "datasource": {
             "type": "prometheus",
@@ -9176,7 +9302,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 8,
+            "x": 16,
             "y": 154
           },
           "id": 490,
@@ -9292,8 +9418,8 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 16,
-            "y": 154
+            "x": 0,
+            "y": 162
           },
           "id": 495,
           "options": {
@@ -9391,7 +9517,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 0,
+            "x": 8,
             "y": 162
           },
           "id": 501,
@@ -9505,7 +9631,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 8,
+            "x": 16,
             "y": 162
           },
           "id": 502,
@@ -9619,8 +9745,8 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 16,
-            "y": 162
+            "x": 0,
+            "y": 170
           },
           "id": 503,
           "options": {
@@ -9736,7 +9862,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 0,
+            "x": 8,
             "y": 170
           },
           "id": 504,
@@ -13836,7 +13962,7 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (result) (\n  changes(greptime_region_balancer_actions_total[$__rate_interval])\n)",
+              "expr": "sum by (result) (\n  increase(greptime_region_balancer_actions_total[$__rate_interval])\n)",
               "hide": false,
               "instant": false,
               "legendFormat": "{{result}}",
@@ -13938,7 +14064,7 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (gate, reason) (changes(greptime_region_balancer_gate_stop_total[$__rate_interval]))",
+              "expr": "sum by (gate, reason) (increase(greptime_region_balancer_gate_stop_total[$__rate_interval]))",
               "hide": false,
               "instant": false,
               "legendFormat": "{{gate}} / {{reason}}",
@@ -14273,7 +14399,7 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (result) (changes(greptime_auto_repartition_actions_total[$__rate_interval]))",
+              "expr": "sum by (result) (increase(greptime_auto_repartition_actions_total[$__rate_interval]))",
               "hide": false,
               "instant": false,
               "legendFormat": "{{result}}",
@@ -14289,7 +14415,7 @@
             "type": "prometheus",
             "uid": "${metrics}"
           },
-          "description": "Auto repartition gate stop count by gate and reason",
+          "description": "Repartition gate stop increments. Baseline failures show detailed reasons when available; otherwise the aggregate invalid_baseline series is retained. Fallback is matched per source before aggregation to avoid double counting across mixed versions.",
           "fieldConfig": {
             "defaults": {
               "color": {
@@ -14375,12 +14501,25 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (gate, reason) (changes(greptime_auto_repartition_gate_stop_total[$__rate_interval]))",
+              "expr": "sum by (gate, reason) (\n  increase(greptime_auto_repartition_gate_stop_total{reason!=\"invalid_baseline\"}[$__rate_interval])\n  or\n  (increase(greptime_auto_repartition_gate_stop_total{reason=\"invalid_baseline\"}[$__rate_interval])\n    unless ignoring (gate, reason)\n    increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))\n)",
               "hide": false,
               "instant": false,
               "legendFormat": "{{gate}} / {{reason}}",
               "range": true,
               "refId": "A"
+            },
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "sum by (reason) (increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))",
+              "hide": false,
+              "instant": false,
+              "legendFormat": "baseline / {{reason}}",
+              "range": true,
+              "refId": "B"
             }
           ],
           "title": "Auto Repartition Gate Stops",

grafana/dashboards/metrics/cluster/dashboard.md

@@ -104,6 +104,7 @@
 | Write Rows per Instance | `sum by (instance, pod) (rate(greptime_mito_write_rows_total{instance=~"$datanode"}[$__rate_interval]))` | `timeseries` | Ingestion size by row counts. | `prometheus` | `rowsps` | `[{{instance}}]-[{{pod}}]` |
 | Read Stage OPS per Instance | `sum by(instance, pod) (rate(greptime_mito_read_stage_elapsed_count{instance=~"$datanode", stage="total"}[$__rate_interval]))` | `timeseries` | Read Stage OPS per Instance. | `prometheus` | `ops` | `[{{instance}}]-[{{pod}}]` |
 | Read Stage P99 and Avg per Instance | `histogram_quantile(0.99, sum by(instance, pod, le, stage) (rate(greptime_mito_read_stage_elapsed_bucket{instance=~"$datanode"}[$__rate_interval])))`<br/>`sum by(instance, pod, stage) (rate(greptime_mito_read_stage_elapsed_sum{instance=~"$datanode"}[$__rate_interval])) / sum by(instance, pod, stage) (rate(greptime_mito_read_stage_elapsed_count{instance=~"$datanode"}[$__rate_interval]))` | `timeseries` | Read Stage P99 and average per Instance. | `prometheus` | `s` | `[{{instance}}]-[{{pod}}]-[{{stage}}]` |
+| Scan Max Concurrent Files per Instance | `histogram_quantile(0.95, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket{instance=~"$datanode"}[$__rate_interval])))`<br/>`histogram_quantile(0.99, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket{instance=~"$datanode"}[$__rate_interval])))`<br/>`sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_sum{instance=~"$datanode"}[$__rate_interval])) / sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_count{instance=~"$datanode"}[$__rate_interval]))` | `timeseries` | Estimated maximum concurrent files by instance and type (query or compaction). Query samples are region scan limit evaluations, including memtable sources counted by the limit, rejected scans, and scans using memory budgeting. Compaction samples count all input files when building a compaction reader. A SQL query can produce multiple region scan evaluations. P95/P99 are histogram estimates, not exact per-query maxima or live open-file counts. | `prometheus` | `short` | `[{{instance}}]-[{{pod}}]-{{type}}-p95` |
 | Write Stage P99 and Avg per Instance | `histogram_quantile(0.99, sum by(instance, pod, le, stage) (rate(greptime_mito_write_stage_elapsed_bucket{instance=~"$datanode"}[$__rate_interval])))`<br/>`sum by(instance, pod, stage) (rate(greptime_mito_write_stage_elapsed_sum{instance=~"$datanode"}[$__rate_interval])) / sum by(instance, pod, stage) (rate(greptime_mito_write_stage_elapsed_count{instance=~"$datanode"}[$__rate_interval]))` | `timeseries` | Write Stage P99 and average per Instance. | `prometheus` | `s` | `[{{instance}}]-[{{pod}}]-[{{stage}}]` |
 | Cached Bytes per Instance | `greptime_mito_cache_bytes{instance=~"$datanode"}` | `timeseries` | Cached Bytes per Instance. | `prometheus` | `decbytes` | `[{{instance}}]-[{{pod}}]-[{{type}}]` |
 | Region Worker Handle Bulk Insert Requests | `histogram_quantile(0.95, sum by(le,instance, stage, pod) (rate(greptime_region_worker_handle_write_bucket[$__rate_interval])))`<br/>`sum by(instance, stage, pod) (rate(greptime_region_worker_handle_write_sum[$__rate_interval]))/sum by(instance, stage, pod) (rate(greptime_region_worker_handle_write_count[$__rate_interval]))` | `timeseries` | Per-stage elapsed time for region worker to handle bulk insert region requests. | `prometheus` | `s` | `[{{instance}}]-[{{pod}}]-[{{stage}}]-P95` |
@@ -240,14 +241,20 @@ ORDER BY data_size DESC;` | `piechart` | Distribution of leader regions and data
 | Title | Query | Type | Description | Datasource | Unit | Legend Format |
 | --- | --- | --- | --- | --- | --- | --- |
 | Region Balancer Actions | `sum by (result) (
-  changes(greptime_region_balancer_actions_total[$__rate_interval])
+  increase(greptime_region_balancer_actions_total[$__rate_interval])
 )` | `timeseries` | Region balancer action count | `prometheus` | `short` | `{{result}}` |
-| Region Balancer Gate Stops | `sum by (gate, reason) (changes(greptime_region_balancer_gate_stop_total[$__rate_interval]))` | `timeseries` | Region balancer gate stop count by gate and reason | `prometheus` | `short` | `{{gate}} / {{reason}}` |
+| Region Balancer Gate Stops | `sum by (gate, reason) (increase(greptime_region_balancer_gate_stop_total[$__rate_interval]))` | `timeseries` | Region balancer gate stop count by gate and reason | `prometheus` | `short` | `{{gate}} / {{reason}}` |
 | Region Balancer Datanodes | `sum by (state) (greptime_region_balancer_datanodes)` | `stat` | Region balancer datanode count by state | `prometheus` | `short` | `{{state}}` |
 | Region Balancer Regions | `sum by (state) (greptime_region_balancer_regions)` | `stat` | Region balancer region count by state | `prometheus` | `short` | `{{state}}` |
 | Region Balancer Datanode Stability | `sum by (state) (greptime_region_balancer_datanode_stability)` | `stat` | Region balancer datanode stability statistics by state | `prometheus` | `binBps` | `{{state}}` |
-| Auto Repartition Actions | `sum by (result) (changes(greptime_auto_repartition_actions_total[$__rate_interval]))` | `timeseries` | Auto repartition action count by result | `prometheus` | `short` | `{{result}}` |
-| Auto Repartition Gate Stops | `sum by (gate, reason) (changes(greptime_auto_repartition_gate_stop_total[$__rate_interval]))` | `timeseries` | Auto repartition gate stop count by gate and reason | `prometheus` | `short` | `{{gate}} / {{reason}}` |
+| Auto Repartition Actions | `sum by (result) (increase(greptime_auto_repartition_actions_total[$__rate_interval]))` | `timeseries` | Auto repartition action count by result | `prometheus` | `short` | `{{result}}` |
+| Auto Repartition Gate Stops | `sum by (gate, reason) (
+  increase(greptime_auto_repartition_gate_stop_total{reason!="invalid_baseline"}[$__rate_interval])
+  or
+  (increase(greptime_auto_repartition_gate_stop_total{reason="invalid_baseline"}[$__rate_interval])
+    unless ignoring (gate, reason)
+    increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))
+)`<br/>`sum by (reason) (increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))` | `timeseries` | Repartition gate stop increments. Baseline failures show detailed reasons when available; otherwise the aggregate invalid_baseline series is retained. Fallback is matched per source before aggregation to avoid double counting across mixed versions. | `prometheus` | `short` | `{{gate}} / {{reason}}` |
 | Auto Repartition Sampling P99 | `histogram_quantile(0.99, sum by (le, stage) (rate(greptime_auto_repartition_sampling_elapsed_bucket[$__rate_interval])))` | `timeseries` | Auto repartition sampling elapsed time by stage | `prometheus` | `s` | `{{stage}}` |
 | Auto Repartition Executor P99 | `histogram_quantile(0.99, sum by (le, stage) (rate(greptime_auto_repartition_executor_elapsed_bucket[$__rate_interval])))` | `timeseries` | Auto repartition executor elapsed time by stage | `prometheus` | `s` | `{{stage}}` |
 # Object Store

grafana/dashboards/metrics/cluster/dashboard.yaml

@@ -1279,6 +1279,26 @@ groups:
                 type: prometheus
                 uid: ${metrics}
               legendFormat: '[{{instance}}]-[{{pod}}]-[{{stage}}]-avg'
+        - title: Scan Max Concurrent Files per Instance
+          type: timeseries
+          description: Estimated maximum concurrent files by instance and type (query or compaction). Query samples are region scan limit evaluations, including memtable sources counted by the limit, rejected scans, and scans using memory budgeting. Compaction samples count all input files when building a compaction reader. A SQL query can produce multiple region scan evaluations. P95/P99 are histogram estimates, not exact per-query maxima or live open-file counts.
+          unit: short
+          queries:
+            - expr: histogram_quantile(0.95, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket{instance=~"$datanode"}[$__rate_interval])))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: '[{{instance}}]-[{{pod}}]-{{type}}-p95'
+            - expr: histogram_quantile(0.99, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket{instance=~"$datanode"}[$__rate_interval])))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: '[{{instance}}]-[{{pod}}]-{{type}}-p99'
+            - expr: sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_sum{instance=~"$datanode"}[$__rate_interval])) / sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_count{instance=~"$datanode"}[$__rate_interval]))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: '[{{instance}}]-[{{pod}}]-{{type}}-avg'
         - title: Write Stage P99 and Avg per Instance
           type: timeseries
           description: Write Stage P99 and average per Instance.
@@ -1884,7 +1904,7 @@ groups:
           queries:
             - expr: |-
                 sum by (result) (
-                  changes(greptime_region_balancer_actions_total[$__rate_interval])
+                  increase(greptime_region_balancer_actions_total[$__rate_interval])
                 )
               datasource:
                 type: prometheus
@@ -1895,7 +1915,7 @@ groups:
           description: Region balancer gate stop count by gate and reason
           unit: short
           queries:
-            - expr: sum by (gate, reason) (changes(greptime_region_balancer_gate_stop_total[$__rate_interval]))
+            - expr: sum by (gate, reason) (increase(greptime_region_balancer_gate_stop_total[$__rate_interval]))
               datasource:
                 type: prometheus
                 uid: ${metrics}
@@ -1935,21 +1955,33 @@ groups:
           description: Auto repartition action count by result
           unit: short
           queries:
-            - expr: sum by (result) (changes(greptime_auto_repartition_actions_total[$__rate_interval]))
+            - expr: sum by (result) (increase(greptime_auto_repartition_actions_total[$__rate_interval]))
               datasource:
                 type: prometheus
                 uid: ${metrics}
               legendFormat: '{{result}}'
         - title: Auto Repartition Gate Stops
           type: timeseries
-          description: Auto repartition gate stop count by gate and reason
+          description: Repartition gate stop increments. Baseline failures show detailed reasons when available; otherwise the aggregate invalid_baseline series is retained. Fallback is matched per source before aggregation to avoid double counting across mixed versions.
           unit: short
           queries:
-            - expr: sum by (gate, reason) (changes(greptime_auto_repartition_gate_stop_total[$__rate_interval]))
+            - expr: |-
+                sum by (gate, reason) (
+                  increase(greptime_auto_repartition_gate_stop_total{reason!="invalid_baseline"}[$__rate_interval])
+                  or
+                  (increase(greptime_auto_repartition_gate_stop_total{reason="invalid_baseline"}[$__rate_interval])
+                    unless ignoring (gate, reason)
+                    increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))
+                )
               datasource:
                 type: prometheus
                 uid: ${metrics}
               legendFormat: '{{gate}} / {{reason}}'
+            - expr: sum by (reason) (increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: baseline / {{reason}}
         - title: Auto Repartition Sampling P99
           type: timeseries
           description: Auto repartition sampling elapsed time by stage

grafana/dashboards/metrics/standalone/dashboard.json

@@ -9112,6 +9112,132 @@
           "title": "Read Stage P99 and Avg per Instance",
           "type": "timeseries"
         },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${metrics}"
+          },
+          "description": "Estimated maximum concurrent files by instance and type (query or compaction). Query samples are region scan limit evaluations, including memtable sources counted by the limit, rejected scans, and scans using memory budgeting. Compaction samples count all input files when building a compaction reader. A SQL query can produce multiple region scan evaluations. P95/P99 are histogram estimates, not exact per-query maxima or live open-file counts.",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisBorderShow": false,
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "barWidthFactor": 0.6,
+                "drawStyle": "line",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green"
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short",
+              "min": 0
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 8,
+            "x": 8,
+            "y": 154
+          },
+          "id": 565,
+          "options": {
+            "legend": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "displayMode": "table",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "hideZeros": false,
+              "mode": "multi",
+              "sort": "desc"
+            }
+          },
+          "pluginVersion": "11.6.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "histogram_quantile(0.95, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket[$__rate_interval])))",
+              "instant": false,
+              "legendFormat": "[{{instance}}]-[{{pod}}]-{{type}}-p95",
+              "range": true,
+              "refId": "A"
+            },
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "histogram_quantile(0.99, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket[$__rate_interval])))",
+              "instant": false,
+              "legendFormat": "[{{instance}}]-[{{pod}}]-{{type}}-p99",
+              "range": true,
+              "refId": "B"
+            },
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_sum[$__rate_interval])) / sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_count[$__rate_interval]))",
+              "instant": false,
+              "legendFormat": "[{{instance}}]-[{{pod}}]-{{type}}-avg",
+              "range": true,
+              "refId": "C"
+            }
+          ],
+          "title": "Scan Max Concurrent Files per Instance",
+          "type": "timeseries"
+        },
         {
           "datasource": {
             "type": "prometheus",
@@ -9176,7 +9302,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 8,
+            "x": 16,
             "y": 154
           },
           "id": 490,
@@ -9292,8 +9418,8 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 16,
-            "y": 154
+            "x": 0,
+            "y": 162
           },
           "id": 495,
           "options": {
@@ -9391,7 +9517,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 0,
+            "x": 8,
             "y": 162
           },
           "id": 501,
@@ -9505,7 +9631,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 8,
+            "x": 16,
             "y": 162
           },
           "id": 502,
@@ -9619,8 +9745,8 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 16,
-            "y": 162
+            "x": 0,
+            "y": 170
           },
           "id": 503,
           "options": {
@@ -9736,7 +9862,7 @@
           "gridPos": {
             "h": 8,
             "w": 8,
-            "x": 0,
+            "x": 8,
             "y": 170
           },
           "id": 504,
@@ -13836,7 +13962,7 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (result) (\n  changes(greptime_region_balancer_actions_total[$__rate_interval])\n)",
+              "expr": "sum by (result) (\n  increase(greptime_region_balancer_actions_total[$__rate_interval])\n)",
               "hide": false,
               "instant": false,
               "legendFormat": "{{result}}",
@@ -13938,7 +14064,7 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (gate, reason) (changes(greptime_region_balancer_gate_stop_total[$__rate_interval]))",
+              "expr": "sum by (gate, reason) (increase(greptime_region_balancer_gate_stop_total[$__rate_interval]))",
               "hide": false,
               "instant": false,
               "legendFormat": "{{gate}} / {{reason}}",
@@ -14273,7 +14399,7 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (result) (changes(greptime_auto_repartition_actions_total[$__rate_interval]))",
+              "expr": "sum by (result) (increase(greptime_auto_repartition_actions_total[$__rate_interval]))",
               "hide": false,
               "instant": false,
               "legendFormat": "{{result}}",
@@ -14289,7 +14415,7 @@
             "type": "prometheus",
             "uid": "${metrics}"
           },
-          "description": "Auto repartition gate stop count by gate and reason",
+          "description": "Repartition gate stop increments. Baseline failures show detailed reasons when available; otherwise the aggregate invalid_baseline series is retained. Fallback is matched per source before aggregation to avoid double counting across mixed versions.",
           "fieldConfig": {
             "defaults": {
               "color": {
@@ -14375,12 +14501,25 @@
                 "uid": "${metrics}"
               },
               "editorMode": "code",
-              "expr": "sum by (gate, reason) (changes(greptime_auto_repartition_gate_stop_total[$__rate_interval]))",
+              "expr": "sum by (gate, reason) (\n  increase(greptime_auto_repartition_gate_stop_total{reason!=\"invalid_baseline\"}[$__rate_interval])\n  or\n  (increase(greptime_auto_repartition_gate_stop_total{reason=\"invalid_baseline\"}[$__rate_interval])\n    unless ignoring (gate, reason)\n    increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))\n)",
               "hide": false,
               "instant": false,
               "legendFormat": "{{gate}} / {{reason}}",
               "range": true,
               "refId": "A"
+            },
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${metrics}"
+              },
+              "editorMode": "code",
+              "expr": "sum by (reason) (increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))",
+              "hide": false,
+              "instant": false,
+              "legendFormat": "baseline / {{reason}}",
+              "range": true,
+              "refId": "B"
             }
           ],
           "title": "Auto Repartition Gate Stops",

grafana/dashboards/metrics/standalone/dashboard.md

@@ -104,6 +104,7 @@
 | Write Rows per Instance | `sum by (instance, pod) (rate(greptime_mito_write_rows_total[$__rate_interval]))` | `timeseries` | Ingestion size by row counts. | `prometheus` | `rowsps` | `[{{instance}}]-[{{pod}}]` |
 | Read Stage OPS per Instance | `sum by(instance, pod) (rate(greptime_mito_read_stage_elapsed_count{stage="total"}[$__rate_interval]))` | `timeseries` | Read Stage OPS per Instance. | `prometheus` | `ops` | `[{{instance}}]-[{{pod}}]` |
 | Read Stage P99 and Avg per Instance | `histogram_quantile(0.99, sum by(instance, pod, le, stage) (rate(greptime_mito_read_stage_elapsed_bucket[$__rate_interval])))`<br/>`sum by(instance, pod, stage) (rate(greptime_mito_read_stage_elapsed_sum[$__rate_interval])) / sum by(instance, pod, stage) (rate(greptime_mito_read_stage_elapsed_count[$__rate_interval]))` | `timeseries` | Read Stage P99 and average per Instance. | `prometheus` | `s` | `[{{instance}}]-[{{pod}}]-[{{stage}}]` |
+| Scan Max Concurrent Files per Instance | `histogram_quantile(0.95, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket[$__rate_interval])))`<br/>`histogram_quantile(0.99, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket[$__rate_interval])))`<br/>`sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_sum[$__rate_interval])) / sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_count[$__rate_interval]))` | `timeseries` | Estimated maximum concurrent files by instance and type (query or compaction). Query samples are region scan limit evaluations, including memtable sources counted by the limit, rejected scans, and scans using memory budgeting. Compaction samples count all input files when building a compaction reader. A SQL query can produce multiple region scan evaluations. P95/P99 are histogram estimates, not exact per-query maxima or live open-file counts. | `prometheus` | `short` | `[{{instance}}]-[{{pod}}]-{{type}}-p95` |
 | Write Stage P99 and Avg per Instance | `histogram_quantile(0.99, sum by(instance, pod, le, stage) (rate(greptime_mito_write_stage_elapsed_bucket[$__rate_interval])))`<br/>`sum by(instance, pod, stage) (rate(greptime_mito_write_stage_elapsed_sum[$__rate_interval])) / sum by(instance, pod, stage) (rate(greptime_mito_write_stage_elapsed_count[$__rate_interval]))` | `timeseries` | Write Stage P99 and average per Instance. | `prometheus` | `s` | `[{{instance}}]-[{{pod}}]-[{{stage}}]` |
 | Cached Bytes per Instance | `greptime_mito_cache_bytes` | `timeseries` | Cached Bytes per Instance. | `prometheus` | `decbytes` | `[{{instance}}]-[{{pod}}]-[{{type}}]` |
 | Region Worker Handle Bulk Insert Requests | `histogram_quantile(0.95, sum by(le,instance, stage, pod) (rate(greptime_region_worker_handle_write_bucket[$__rate_interval])))`<br/>`sum by(instance, stage, pod) (rate(greptime_region_worker_handle_write_sum[$__rate_interval]))/sum by(instance, stage, pod) (rate(greptime_region_worker_handle_write_count[$__rate_interval]))` | `timeseries` | Per-stage elapsed time for region worker to handle bulk insert region requests. | `prometheus` | `s` | `[{{instance}}]-[{{pod}}]-[{{stage}}]-P95` |
@@ -240,14 +241,20 @@ ORDER BY data_size DESC;` | `piechart` | Distribution of leader regions and data
 | Title | Query | Type | Description | Datasource | Unit | Legend Format |
 | --- | --- | --- | --- | --- | --- | --- |
 | Region Balancer Actions | `sum by (result) (
-  changes(greptime_region_balancer_actions_total[$__rate_interval])
+  increase(greptime_region_balancer_actions_total[$__rate_interval])
 )` | `timeseries` | Region balancer action count | `prometheus` | `short` | `{{result}}` |
-| Region Balancer Gate Stops | `sum by (gate, reason) (changes(greptime_region_balancer_gate_stop_total[$__rate_interval]))` | `timeseries` | Region balancer gate stop count by gate and reason | `prometheus` | `short` | `{{gate}} / {{reason}}` |
+| Region Balancer Gate Stops | `sum by (gate, reason) (increase(greptime_region_balancer_gate_stop_total[$__rate_interval]))` | `timeseries` | Region balancer gate stop count by gate and reason | `prometheus` | `short` | `{{gate}} / {{reason}}` |
 | Region Balancer Datanodes | `sum by (state) (greptime_region_balancer_datanodes)` | `stat` | Region balancer datanode count by state | `prometheus` | `short` | `{{state}}` |
 | Region Balancer Regions | `sum by (state) (greptime_region_balancer_regions)` | `stat` | Region balancer region count by state | `prometheus` | `short` | `{{state}}` |
 | Region Balancer Datanode Stability | `sum by (state) (greptime_region_balancer_datanode_stability)` | `stat` | Region balancer datanode stability statistics by state | `prometheus` | `binBps` | `{{state}}` |
-| Auto Repartition Actions | `sum by (result) (changes(greptime_auto_repartition_actions_total[$__rate_interval]))` | `timeseries` | Auto repartition action count by result | `prometheus` | `short` | `{{result}}` |
-| Auto Repartition Gate Stops | `sum by (gate, reason) (changes(greptime_auto_repartition_gate_stop_total[$__rate_interval]))` | `timeseries` | Auto repartition gate stop count by gate and reason | `prometheus` | `short` | `{{gate}} / {{reason}}` |
+| Auto Repartition Actions | `sum by (result) (increase(greptime_auto_repartition_actions_total[$__rate_interval]))` | `timeseries` | Auto repartition action count by result | `prometheus` | `short` | `{{result}}` |
+| Auto Repartition Gate Stops | `sum by (gate, reason) (
+  increase(greptime_auto_repartition_gate_stop_total{reason!="invalid_baseline"}[$__rate_interval])
+  or
+  (increase(greptime_auto_repartition_gate_stop_total{reason="invalid_baseline"}[$__rate_interval])
+    unless ignoring (gate, reason)
+    increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))
+)`<br/>`sum by (reason) (increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))` | `timeseries` | Repartition gate stop increments. Baseline failures show detailed reasons when available; otherwise the aggregate invalid_baseline series is retained. Fallback is matched per source before aggregation to avoid double counting across mixed versions. | `prometheus` | `short` | `{{gate}} / {{reason}}` |
 | Auto Repartition Sampling P99 | `histogram_quantile(0.99, sum by (le, stage) (rate(greptime_auto_repartition_sampling_elapsed_bucket[$__rate_interval])))` | `timeseries` | Auto repartition sampling elapsed time by stage | `prometheus` | `s` | `{{stage}}` |
 | Auto Repartition Executor P99 | `histogram_quantile(0.99, sum by (le, stage) (rate(greptime_auto_repartition_executor_elapsed_bucket[$__rate_interval])))` | `timeseries` | Auto repartition executor elapsed time by stage | `prometheus` | `s` | `{{stage}}` |
 # Object Store

grafana/dashboards/metrics/standalone/dashboard.yaml

@@ -1279,6 +1279,26 @@ groups:
                 type: prometheus
                 uid: ${metrics}
               legendFormat: '[{{instance}}]-[{{pod}}]-[{{stage}}]-avg'
+        - title: Scan Max Concurrent Files per Instance
+          type: timeseries
+          description: Estimated maximum concurrent files by instance and type (query or compaction). Query samples are region scan limit evaluations, including memtable sources counted by the limit, rejected scans, and scans using memory budgeting. Compaction samples count all input files when building a compaction reader. A SQL query can produce multiple region scan evaluations. P95/P99 are histogram estimates, not exact per-query maxima or live open-file counts.
+          unit: short
+          queries:
+            - expr: histogram_quantile(0.95, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket[$__rate_interval])))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: '[{{instance}}]-[{{pod}}]-{{type}}-p95'
+            - expr: histogram_quantile(0.99, sum by(instance, pod, type, le) (rate(greptime_mito_scan_max_concurrent_files_bucket[$__rate_interval])))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: '[{{instance}}]-[{{pod}}]-{{type}}-p99'
+            - expr: sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_sum[$__rate_interval])) / sum by(instance, pod, type) (rate(greptime_mito_scan_max_concurrent_files_count[$__rate_interval]))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: '[{{instance}}]-[{{pod}}]-{{type}}-avg'
         - title: Write Stage P99 and Avg per Instance
           type: timeseries
           description: Write Stage P99 and average per Instance.
@@ -1884,7 +1904,7 @@ groups:
           queries:
             - expr: |-
                 sum by (result) (
-                  changes(greptime_region_balancer_actions_total[$__rate_interval])
+                  increase(greptime_region_balancer_actions_total[$__rate_interval])
                 )
               datasource:
                 type: prometheus
@@ -1895,7 +1915,7 @@ groups:
           description: Region balancer gate stop count by gate and reason
           unit: short
           queries:
-            - expr: sum by (gate, reason) (changes(greptime_region_balancer_gate_stop_total[$__rate_interval]))
+            - expr: sum by (gate, reason) (increase(greptime_region_balancer_gate_stop_total[$__rate_interval]))
               datasource:
                 type: prometheus
                 uid: ${metrics}
@@ -1935,21 +1955,33 @@ groups:
           description: Auto repartition action count by result
           unit: short
           queries:
-            - expr: sum by (result) (changes(greptime_auto_repartition_actions_total[$__rate_interval]))
+            - expr: sum by (result) (increase(greptime_auto_repartition_actions_total[$__rate_interval]))
               datasource:
                 type: prometheus
                 uid: ${metrics}
               legendFormat: '{{result}}'
         - title: Auto Repartition Gate Stops
           type: timeseries
-          description: Auto repartition gate stop count by gate and reason
+          description: Repartition gate stop increments. Baseline failures show detailed reasons when available; otherwise the aggregate invalid_baseline series is retained. Fallback is matched per source before aggregation to avoid double counting across mixed versions.
           unit: short
           queries:
-            - expr: sum by (gate, reason) (changes(greptime_auto_repartition_gate_stop_total[$__rate_interval]))
+            - expr: |-
+                sum by (gate, reason) (
+                  increase(greptime_auto_repartition_gate_stop_total{reason!="invalid_baseline"}[$__rate_interval])
+                  or
+                  (increase(greptime_auto_repartition_gate_stop_total{reason="invalid_baseline"}[$__rate_interval])
+                    unless ignoring (gate, reason)
+                    increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))
+                )
               datasource:
                 type: prometheus
                 uid: ${metrics}
               legendFormat: '{{gate}} / {{reason}}'
+            - expr: sum by (reason) (increase(greptime_auto_repartition_invalid_baseline_total[$__rate_interval]))
+              datasource:
+                type: prometheus
+                uid: ${metrics}
+              legendFormat: baseline / {{reason}}
         - title: Auto Repartition Sampling P99
           type: timeseries
           description: Auto repartition sampling elapsed time by stage

licenserc-enterprise.toml

@@ -15,6 +15,7 @@ headerPath = "licenses/enterprise-header.txt"
 # with the "# enterprise" block in `licenserc.toml`'s `excludes` — both are
 # enforced by `make check-enterprise-license`.
 includes = [
+    "src/datatypes/src/schema/sketch.rs",
     "src/common/meta/src/rpc/ddl/trigger.rs",
     "src/operator/src/expr_helper/trigger.rs",
     "src/sql/src/statements/alter/trigger.rs",
@@ -25,6 +26,16 @@ includes = [
     "src/sql/src/parsers/create_parser/trigger.rs",
     "src/sql/src/parsers/show_parser/trigger.rs",
     "src/mito2/src/extension.rs",
+    "src/mito2/src/engine/table_compaction.rs",
+    "src/mito2/src/engine/table_compaction/output.rs",
+    "src/mito2/src/engine/table_compaction/tests.rs",
+    "src/mito2/src/worker/handle_table_compaction.rs",
+    "src/mito2/src/compaction/multi_region.rs",
+    "src/mito2/src/compaction/multi_region/source.rs",
+    "src/mito2/src/compaction/multi_region/memory.rs",
+    "src/mito2/src/compaction/multi_region/worker.rs",
+    "src/mito2/src/compaction/multi_region/writer.rs",
+    "src/mito2/src/compaction/multi_region/tests.rs",
     "src/common/function/src/admin/purge_table.rs",
     "src/catalog/src/system_schema/information_schema/recycle_bin.rs",
     "src/common/meta/src/ddl/purge_dropped_table.rs",

licenserc.toml

@@ -31,6 +31,7 @@ excludes = [
     # Keep this list in sync with that file's `includes`; enforced by
     # `make check-enterprise-license`.
     # enterprise:start
+    "src/datatypes/src/schema/sketch.rs",
     "src/common/meta/src/rpc/ddl/trigger.rs",
     "src/operator/src/expr_helper/trigger.rs",
     "src/sql/src/statements/alter/trigger.rs",
@@ -41,6 +42,16 @@ excludes = [
     "src/sql/src/parsers/create_parser/trigger.rs",
     "src/sql/src/parsers/show_parser/trigger.rs",
     "src/mito2/src/extension.rs",
+    "src/mito2/src/engine/table_compaction.rs",
+    "src/mito2/src/engine/table_compaction/output.rs",
+    "src/mito2/src/engine/table_compaction/tests.rs",
+    "src/mito2/src/worker/handle_table_compaction.rs",
+    "src/mito2/src/compaction/multi_region.rs",
+    "src/mito2/src/compaction/multi_region/source.rs",
+    "src/mito2/src/compaction/multi_region/memory.rs",
+    "src/mito2/src/compaction/multi_region/worker.rs",
+    "src/mito2/src/compaction/multi_region/writer.rs",
+    "src/mito2/src/compaction/multi_region/tests.rs",
     "src/common/function/src/admin/purge_table.rs",
     "src/catalog/src/system_schema/information_schema/recycle_bin.rs",
     "src/common/meta/src/ddl/purge_dropped_table.rs",

src/api/src/region.rs

@@ -15,7 +15,19 @@
 use std::collections::HashMap;
 
 use common_base::AffectedRows;
-use greptime_proto::v1::region::RegionResponse as RegionResponseV1;
+use greptime_proto::v1::region::{RegionRequestHeader, RegionResponse as RegionResponseV1};
+
+/// Request extension advertising support for per-entry write failures.
+pub const ACCEPT_PARTIAL_RESULT: &str = "accept_partial_result";
+
+/// Advertises support for per-entry write failures in the request header.
+pub fn set_accept_partial_result(header: &mut RegionRequestHeader) {
+    header
+        .query_context
+        .get_or_insert_with(Default::default)
+        .extensions
+        .insert(ACCEPT_PARTIAL_RESULT.to_string(), "true".to_string());
+}
 
 /// This result struct is derived from [RegionResponseV1]
 #[derive(Debug)]
@@ -25,6 +37,21 @@ pub struct RegionResponse {
     pub metadata: Vec<u8>,
 }
 
+/// Successful entries and failed-entry errors indexed by input order.
+#[derive(Debug)]
+pub struct BatchRegionResponse<E> {
+    pub affected_rows: AffectedRows,
+    pub extensions: HashMap<String, Vec<u8>>,
+    pub failures: Vec<(usize, E)>,
+}
+
+impl<E> BatchRegionResponse<E> {
+    /// Returns the failure with the smallest original request index.
+    pub fn first_failure(&self) -> Option<&(usize, E)> {
+        self.failures.iter().min_by_key(|(index, _)| *index)
+    }
+}
+
 impl RegionResponse {
     pub fn from_region_response(region_response: RegionResponseV1) -> Self {
         Self {

src/api/src/v1/column_def.rs

@@ -16,8 +16,8 @@ use std::collections::HashMap;
 
 use arrow_schema::extension::{EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY};
 use datatypes::schema::{
-    COMMENT_KEY, COMPRESSION_KEY, ColumnDefaultConstraint, ColumnSchema, FULLTEXT_KEY,
-    FulltextAnalyzer, FulltextBackend, FulltextOptions, INVERTED_INDEX_KEY, Metadata,
+    COMMENT_KEY, COMPRESSION_KEY, ColumnDefaultConstraint, ColumnSchema, CompressionOptions,
+    FULLTEXT_KEY, FulltextAnalyzer, FulltextBackend, FulltextOptions, INVERTED_INDEX_KEY, Metadata,
     SKIPPING_INDEX_KEY, SkippingIndexOptions, SkippingIndexType,
 };
 use greptime_proto::v1::{
@@ -240,6 +240,14 @@ pub fn options_from_skipping(skipping: &SkippingIndexOptions) -> Result<Option<C
     Ok((!options.options.is_empty()).then_some(options))
 }
 
+/// Tries to construct a `ColumnOptions` from the given `CompressionOptions`.
+pub fn options_from_compression(compression: &CompressionOptions) -> Result<ColumnOptions> {
+    let mut options = ColumnOptions::default();
+    let v = serde_json::to_string(compression).context(error::SerializeJsonSnafu)?;
+    options.options.insert(COMPRESSION_GRPC_KEY.to_string(), v);
+    Ok(options)
+}
+
 /// Tries to construct a `ColumnOptions` for inverted index.
 pub fn options_from_inverted() -> ColumnOptions {
     let mut options = ColumnOptions::default();
@@ -272,9 +280,71 @@ pub fn as_skipping_index_type(skipping_index_type: PbSkippingIndexType) -> Skipp
     }
 }
 
+/// Converts fulltext options for a JSON2 type hint to the wire representation.
+pub fn json_fulltext_options_to_proto(
+    options: FulltextOptions,
+) -> crate::v1::JsonFulltextIndexOptions {
+    crate::v1::JsonFulltextIndexOptions {
+        analyzer: match options.analyzer {
+            FulltextAnalyzer::English => Analyzer::English.into(),
+            FulltextAnalyzer::Chinese => Analyzer::Chinese.into(),
+        },
+        case_sensitive: options.case_sensitive,
+        backend: match options.backend {
+            FulltextBackend::Bloom => PbFulltextBackend::Bloom.into(),
+            FulltextBackend::Tantivy => PbFulltextBackend::Tantivy.into(),
+        },
+        granularity: options.granularity,
+        false_positive_rate: options.false_positive_rate(),
+    }
+}
+
+/// Validates fulltext options received for a JSON2 type hint.
+/// The presence of these options enables the fulltext index.
+pub fn json_fulltext_options_from_proto(
+    options: crate::v1::JsonFulltextIndexOptions,
+) -> datatypes::error::Result<FulltextOptions> {
+    let analyzer = Analyzer::try_from(options.analyzer).map_err(|_| {
+        datatypes::error::InvalidFulltextOptionSnafu {
+            msg: format!("Unknown fulltext analyzer: {}", options.analyzer),
+        }
+        .build()
+    })?;
+    let backend = PbFulltextBackend::try_from(options.backend).map_err(|_| {
+        datatypes::error::InvalidFulltextOptionSnafu {
+            msg: format!("Unknown fulltext backend: {}", options.backend),
+        }
+        .build()
+    })?;
+    FulltextOptions::new(
+        true,
+        as_fulltext_option_analyzer(analyzer),
+        options.case_sensitive,
+        as_fulltext_option_backend(backend),
+        options.granularity,
+        options.false_positive_rate,
+    )
+}
+
+/// Validates skipping options received for a JSON2 type hint.
+pub fn json_skipping_options_from_proto(
+    options: crate::v1::JsonSkippingIndexOptions,
+) -> datatypes::error::Result<SkippingIndexOptions> {
+    let index_type = PbSkippingIndexType::try_from(options.index_type).map_err(|_| {
+        datatypes::error::InvalidSkippingIndexOptionSnafu {
+            msg: format!("Unknown skipping index type: {}", options.index_type),
+        }
+        .build()
+    })?;
+    SkippingIndexOptions::new(
+        options.granularity,
+        options.false_positive_rate,
+        as_skipping_index_type(index_type),
+    )
+}
+
 #[cfg(test)]
 mod tests {
-
     use common_base::readable_size::ReadableSize;
     use datatypes::data_type::ConcreteDataType;
     use datatypes::schema::{CompressionOptions, FulltextAnalyzer, FulltextBackend};
@@ -282,6 +352,26 @@ mod tests {
     use super::*;
     use crate::v1::ColumnDataType;
 
+    #[test]
+    fn json_fulltext_options_roundtrip() {
+        for backend in [FulltextBackend::Bloom, FulltextBackend::Tantivy] {
+            let options =
+                FulltextOptions::new(true, FulltextAnalyzer::Chinese, true, backend, 8, 0.05)
+                    .unwrap();
+            let wire = json_fulltext_options_to_proto(options.clone());
+            assert_eq!(options, json_fulltext_options_from_proto(wire).unwrap());
+            let mut invalid = wire;
+            invalid.backend = 99;
+            assert!(json_fulltext_options_from_proto(invalid).is_err());
+            let mut invalid = wire;
+            invalid.analyzer = 99;
+            assert!(json_fulltext_options_from_proto(invalid).is_err());
+            let mut invalid = wire;
+            invalid.granularity = 0;
+            assert!(json_fulltext_options_from_proto(invalid).is_err());
+        }
+    }
+
     #[test]
     fn test_try_as_column_schema() {
         let column_def = ColumnDef {

src/auth/Cargo.toml

@@ -24,6 +24,7 @@ hex.workspace = true
 hmac = "0.12"
 pbkdf2 = "0.12"
 rand.workspace = true
+serde.workspace = true
 sha1 = "0.10"
 sha2 = "0.10"
 snafu.workspace = true

src/auth/src/common.rs

@@ -14,10 +14,11 @@
 
 use std::sync::{Arc, LazyLock};
 
-use common_base::secrets::SecretString;
+use common_base::secrets::{ExposeSecret, SecretString};
 use digest::Digest;
 use hmac::{Hmac, Mac};
 use pbkdf2::pbkdf2_hmac;
+use serde::Serializer;
 use sha1::Sha1;
 use sha2::Sha256;
 use snafu::{OptionExt, ensure};
@@ -39,6 +40,9 @@ pub const MAX_PBKDF2_SHA256_ITERATIONS: u32 = 1_000_000;
 pub const MAX_PBKDF2_SHA256_SALT_LEN: usize = 1024;
 pub const PG_SCRAM_SHA256_KEY_LEN: usize = 32;
 
+/// Placeholder used when rendering potentially secret user-provider options.
+pub(crate) const REDACTED_USER_PROVIDER_OPTION: &str = "[REDACTED]";
+
 type HmacSha256 = Hmac<Sha256>;
 
 /// Process-wide secret used to derive a stable mock salt for unknown users, so
@@ -52,8 +56,8 @@ pub fn userinfo_by_name(username: Option<String>) -> UserInfoRef {
 }
 
 pub fn user_provider_from_option(opt: &str) -> Result<UserProviderRef> {
-    let (name, content) = opt.split_once(':').with_context(|| InvalidConfigSnafu {
-        value: opt.to_string(),
+    let (name, content) = opt.split_once(':').context(InvalidConfigSnafu {
+        value: REDACTED_USER_PROVIDER_OPTION,
         msg: "UserProviderOption must be in format `<option>:<value>`",
     })?;
     match name {
@@ -66,16 +70,57 @@ pub fn user_provider_from_option(opt: &str) -> Result<UserProviderRef> {
             WatchFileUserProvider::new(content).map(|p| Arc::new(p) as UserProviderRef)
         }
         _ => InvalidConfigSnafu {
-            value: name.to_string(),
+            value: REDACTED_USER_PROVIDER_OPTION,
             msg: "Invalid UserProviderOption",
         }
         .fail(),
     }
 }
 
+/// Serializes a `user_provider` option, exposing only known file-based forms.
+///
+/// The option is held as a [`SecretString`] so `Debug` and the default
+/// serializer never print it. This serializer exposes the value only to keep
+/// provider names and file paths visible for those forms in configuration dumps
+/// such as the `/config` endpoint.
+pub fn serialize_user_provider_option<S>(
+    value: &Option<SecretString>,
+    serializer: S,
+) -> std::result::Result<S::Ok, S::Error>
+where
+    S: Serializer,
+{
+    match value {
+        Some(value) => {
+            serializer.serialize_some(&redact_user_provider_option(value.expose_secret()))
+        }
+        None => serializer.serialize_none(),
+    }
+}
+
+/// Redacts user-provider options except known file-based forms with nonempty paths.
+///
+/// Only `static_user_provider:file:<path>` and `watch_file_user_provider:<path>`
+/// are returned unchanged. Inline credentials, unknown providers and malformed
+/// options are replaced entirely with a placeholder.
+pub fn redact_user_provider_option(value: &str) -> String {
+    let is_file_provider = match value.split_once(':') {
+        Some((STATIC_USER_PROVIDER, content)) => content
+            .strip_prefix("file:")
+            .is_some_and(|path| !path.is_empty()),
+        Some((WATCH_FILE_USER_PROVIDER, path)) => !path.is_empty(),
+        _ => false,
+    };
+    if is_file_provider {
+        value.to_string()
+    } else {
+        REDACTED_USER_PROVIDER_OPTION.to_string()
+    }
+}
+
 pub fn static_user_provider_from_option(opt: &str) -> Result<StaticUserProvider> {
-    let (name, content) = opt.split_once(':').with_context(|| InvalidConfigSnafu {
-        value: opt.to_string(),
+    let (name, content) = opt.split_once(':').context(InvalidConfigSnafu {
+        value: REDACTED_USER_PROVIDER_OPTION,
         msg: "UserProviderOption must be in format `<option>:<value>`",
     })?;
     match name {
@@ -84,7 +129,7 @@ pub fn static_user_provider_from_option(opt: &str) -> Result<StaticUserProvider>
             Ok(provider)
         }
         _ => InvalidConfigSnafu {
-            value: name.to_string(),
+            value: REDACTED_USER_PROVIDER_OPTION,
             msg: format!("Invalid UserProviderOption, expect only {STATIC_USER_PROVIDER}"),
         }
         .fail(),
@@ -361,6 +406,18 @@ impl PgScramSha256Verifier {
         &self.salt
     }
 
+    /// `SHA256(ClientKey)`, used to verify the client proof in SCRAM.
+    pub fn stored_key(&self) -> &[u8] {
+        &self.stored_key
+    }
+
+    /// `HMAC(SaltedPassword, "Server Key")`, used to compute the SCRAM
+    /// server signature. This is secret material and must not be exposed to
+    /// clients.
+    pub fn server_key(&self) -> &[u8] {
+        &self.server_key
+    }
+
     pub fn verify_plain_password(&self, password: &[u8]) -> Result<bool> {
         let salted_password =
             pg_scram_sha256_salted_password(password, &self.salt, self.iterations)?;
@@ -515,6 +572,7 @@ fn double_sha1(data: &[u8]) -> Vec<u8> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::error::Error;
 
     fn mysql_native_password_auth_data(password: &[u8], salt: &[u8]) -> Vec<u8> {
         let hash_stage_1 = sha1_one(password);
@@ -639,6 +697,75 @@ mod tests {
         }
     }
 
+    #[test]
+    fn test_redact_user_provider_option() {
+        for option in [
+            "static_user_provider:file:/etc/greptimedb/users",
+            "watch_file_user_provider:/etc/greptimedb/users",
+            "static_user_provider:file:users.txt",
+            "watch_file_user_provider:users.txt",
+        ] {
+            assert_eq!(redact_user_provider_option(option), option);
+        }
+
+        for option in [
+            "",
+            "secret",
+            "alice=secret",
+            "static_user_provider",
+            "static_user_provider:",
+            "static_user_provider:file",
+            "static_user_provider:file:",
+            "static_user_provider:filex:/etc/greptimedb/users",
+            "static_user_provider:cmd:alice=secret,bob=pbkdf2_sha256:4096:aa:bb",
+            "static_user_provider:cmd:alice",
+            "static_user_provider:cmd:alice:secret",
+            "static_user_provider:cmd:alice=plain:prefix,secret-tail",
+            "static_user_provider:cmdx:alice=secret",
+            "watch_file_user_provider",
+            "watch_file_user_provider:",
+            "ldap_user_provider:/etc/greptimedb/ldap.toml",
+            "unknown_user_provider:cmd:alice=secret",
+        ] {
+            assert_eq!(redact_user_provider_option(option), "[REDACTED]");
+        }
+    }
+
+    #[test]
+    fn test_user_provider_errors_redact_credentials() {
+        for option in [
+            "secret",
+            "alice=secret",
+            "alice=secret,bob=other_secret",
+            "alice=secret:cmd:alice=secret",
+            "static_user_provider:alice=secret",
+            "static_user_provider:alice=secret:cmd",
+            "static_user_provider:cmdx:alice=secret",
+            "unknown_user_provider:cmd:alice=secret",
+            "static_user_provider:cmd:alice:invalid=secret",
+            "static_user_provider:cmd:alice=secret=tail",
+            "static_user_provider:cmd:alice=pbkdf2_sha256:secret",
+            "static_user_provider:cmd:alice=mysql_native_password:secret",
+            "static_user_provider:cmd:alice=pg_scram_sha256:secret",
+            "static_user_provider:cmd:bob=secret,alice=pbkdf2_sha256:secret",
+            "static_user_provider:cmd:alice:secret",
+            "static_user_provider:cmd:alice=plain:prefix,secret-tail",
+            "static_user_provider:cmd:watch_file_user_provider:secret",
+        ] {
+            for error in [
+                user_provider_from_option(option).err().unwrap(),
+                static_user_provider_from_option(option).err().unwrap(),
+            ] {
+                assert!(!error.to_string().contains("secret"));
+                assert!(!format!("{error:?}").contains("secret"));
+                let Error::InvalidConfig { value, .. } = error else {
+                    panic!("Expected an invalid config error");
+                };
+                assert_eq!(value, "[REDACTED]");
+            }
+        }
+    }
+
     #[test]
     fn test_mock_verifier_is_stable_per_username() {
         let alice = PgScramSha256Verifier::mock_for_unknown_user(b"alice");

src/auth/src/lib.rs

@@ -27,8 +27,9 @@ pub use common::{
     PG_SCRAM_SHA256_KEY_LEN, Password, PgScramSha256Verifier, auth_mysql, auth_mysql_with_verifier,
     format_mysql_native_password_verifier, format_pbkdf2_sha256_password_verifier,
     format_pg_scram_sha256_password_verifier, mysql_native_password_hash,
-    parse_pg_scram_sha256_password_verifier, static_user_provider_from_option,
-    user_provider_from_option, userinfo_by_name, validate_mysql_native_password_verifier,
+    parse_pg_scram_sha256_password_verifier, redact_user_provider_option,
+    serialize_user_provider_option, static_user_provider_from_option, user_provider_from_option,
+    userinfo_by_name, validate_mysql_native_password_verifier,
 };
 pub use permission::{
     ALL_ACTIONS, AccessMode, DASHBOARD_DELETE, DASHBOARD_QUERY, DASHBOARD_SAVE,
@@ -39,7 +40,7 @@ pub use permission::{
 };
 pub use user_info::UserInfo;
 pub use user_provider::static_user_provider::StaticUserProvider;
-pub use user_provider::{BEARER_TOKEN_USER, MysqlAuthMethod, PgAuthInfo, UserProvider};
+pub use user_provider::{AuthMethod, BEARER_TOKEN_USER, MysqlAuthMethod, PgAuthInfo, UserProvider};
 
 /// pub type alias
 pub type UserInfoRef = std::sync::Arc<dyn UserInfo>;

src/auth/src/permission.rs

@@ -34,6 +34,9 @@ pub struct PermissionTableTarget {
     pub catalog: String,
     pub schema: String,
     pub table: String,
+    /// Set from catalog metadata, never from a client-supplied table name.
+    /// Physical metric tables contain data belonging to multiple logical tables.
+    pub is_physical_metric_table: bool,
 }
 
 impl PermissionTableTarget {
@@ -47,8 +50,15 @@ impl PermissionTableTarget {
             catalog: catalog.into(),
             schema: schema.into(),
             table: table.into(),
+            is_physical_metric_table: false,
         }
     }
+
+    /// Sets whether catalog metadata identifies this target as a physical metric table.
+    pub fn with_physical_metric_table(mut self, is_physical_metric_table: bool) -> Self {
+        self.is_physical_metric_table = is_physical_metric_table;
+        self
+    }
 }
 
 /// The result of resolving every table target in a permission request.
@@ -183,6 +193,11 @@ pub enum PermissionReq<'a> {
     GrpcRequest(&'a Request),
     SqlStatement(&'a Statement),
     Action(PermissionAction),
+    /// Enumerates a schema's metadata without accessing any table data.
+    SchemaRead {
+        catalog: &'a str,
+        schema: &'a str,
+    },
     BulkInsert {
         catalog: &'a str,
         schema: &'a str,
@@ -199,6 +214,7 @@ impl<'a> PermissionReq<'a> {
             }
             PermissionReq::SqlStatement(stmt) => stmt.is_readonly(),
             PermissionReq::Action(action) => action.access_mode() == AccessMode::Read,
+            PermissionReq::SchemaRead { .. } => true,
 
             PermissionReq::GrpcRequest(_) | PermissionReq::BulkInsert { .. } => false,
         }
@@ -241,6 +257,17 @@ pub trait PermissionChecker: Send + Sync {
         true
     }
 
+    /// Extracts user-visible gRPC targets before the frontend classifies them
+    /// using catalog metadata. Unrecognized requests must fail closed.
+    fn grpc_table_targets(
+        &self,
+        _request: &Request,
+        _catalog: &str,
+        _schema: &str,
+    ) -> PermissionTableTargets {
+        PermissionTableTargets::Unresolved
+    }
+
     /// Checks the operation privilege and its resolved user-visible table targets.
     ///
     /// [`PermissionTableTargets::Resolved`] with an empty vector still requires
@@ -296,6 +323,18 @@ impl PermissionChecker for Option<&PermissionCheckerRef> {
         }
     }
 
+    fn grpc_table_targets(
+        &self,
+        request: &Request,
+        catalog: &str,
+        schema: &str,
+    ) -> PermissionTableTargets {
+        match self {
+            Some(checker) => checker.grpc_table_targets(request, catalog, schema),
+            None => PermissionTableTargets::Unresolved,
+        }
+    }
+
     fn check_permission_with_table_targets(
         &self,
         user_info: UserInfoRef,
@@ -386,6 +425,20 @@ mod tests {
             Ok(PermissionResp::Reject)
         }
 
+        fn grpc_table_targets(
+            &self,
+            request: &Request,
+            catalog: &str,
+            schema: &str,
+        ) -> PermissionTableTargets {
+            match request {
+                Request::RowInserts(requests) => {
+                    PermissionTableTargets::from_row_insert_requests(catalog, schema, requests)
+                }
+                _ => PermissionTableTargets::Unresolved,
+            }
+        }
+
         fn check_permission_with_table_targets(
             &self,
             _user_info: UserInfoRef,
@@ -600,8 +653,8 @@ mod tests {
 
     #[test]
     fn test_table_target_permission_forwarding() {
-        let checker: PermissionCheckerRef = Arc::new(TargetAwarePermissionChecker);
-        let checker = Some(&checker);
+        let inner_checker: PermissionCheckerRef = Arc::new(TargetAwarePermissionChecker);
+        let checker = Some(&inner_checker);
         assert!(checker.uses_table_targets());
 
         let allowed = checker
@@ -658,6 +711,25 @@ mod tests {
 
         let no_checker: Option<&PermissionCheckerRef> = None;
         assert!(!no_checker.uses_table_targets());
+        for request in [
+            Request::RowInserts(RowInsertRequests {
+                inserts: vec![api::v1::RowInsertRequest {
+                    table_name: "allowed".to_string(),
+                    ..Default::default()
+                }],
+            }),
+            Request::RowInserts(RowInsertRequests::default()),
+            Request::Query(api::v1::QueryRequest::default()),
+        ] {
+            assert_eq!(
+                inner_checker.grpc_table_targets(&request, "catalog", "schema"),
+                checker.grpc_table_targets(&request, "catalog", "schema"),
+            );
+            assert_eq!(
+                PermissionTableTargets::Unresolved,
+                no_checker.grpc_table_targets(&request, "catalog", "schema"),
+            );
+        }
         let allowed = no_checker
             .check_permission_with_table_targets(
                 crate::userinfo_by_name(None),

src/auth/src/user_provider.rs

@@ -24,6 +24,7 @@ use std::{fmt, io};
 use common_base::secrets::ExposeSecret;
 use common_telemetry::warn;
 use pbkdf2::pbkdf2_hmac;
+use serde::Serialize;
 use sha2::Sha256;
 use snafu::{OptionExt, ResultExt, ensure};
 use subtle::ConstantTimeEq;
@@ -49,6 +50,27 @@ use crate::{UserInfoRef, auth_mysql};
 /// deployment choice.
 pub const BEARER_TOKEN_USER: &str = "*";
 
+/// A login method advertised by the HTTP authentication configuration endpoint.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
+#[serde(tag = "type", rename_all = "snake_case")]
+pub enum AuthMethod {
+    /// Username and password authentication.
+    Password,
+    /// OpenID Connect authentication for browser clients.
+    Oidc {
+        /// Stable identifier for this identity provider.
+        id: String,
+        /// Human-readable identity provider name.
+        display_name: String,
+        /// OpenID Connect issuer URL.
+        issuer: String,
+        /// Public client identifier.
+        client_id: String,
+        /// OAuth scopes requested during login.
+        scopes: Vec<String>,
+    },
+}
+
 #[async_trait::async_trait]
 pub trait UserProvider: Send + Sync {
     fn name(&self) -> &str;
@@ -107,6 +129,11 @@ pub trait UserProvider: Send + Sync {
         Ok(user_info)
     }
 
+    /// Returns the login methods that browser clients can use.
+    fn auth_methods(&self) -> Vec<AuthMethod> {
+        vec![AuthMethod::Password]
+    }
+
     fn mysql_auth_method(&self) -> MysqlAuthMethod {
         if self.external() {
             MysqlAuthMethod::ClearPassword

src/auth/src/user_provider/static_user_provider.rs

@@ -15,6 +15,7 @@
 use async_trait::async_trait;
 use snafu::OptionExt;
 
+use crate::common::REDACTED_USER_PROVIDER_OPTION;
 use crate::error::{InvalidConfigSnafu, Result};
 use crate::user_provider::{
     PgAuthInfo, UserInfoMap, authenticate_with_credential, load_credential_from_file,
@@ -31,7 +32,7 @@ pub struct StaticUserProvider {
 impl StaticUserProvider {
     pub(crate) fn new(value: &str) -> Result<Self> {
         let (mode, content) = value.split_once(':').context(InvalidConfigSnafu {
-            value: value.to_string(),
+            value: REDACTED_USER_PROVIDER_OPTION,
             msg: "StaticUserProviderOption must be in format `<option>:<value>`",
         })?;
         match mode {
@@ -43,7 +44,7 @@ impl StaticUserProvider {
                 .split(',')
                 .map(|kv| {
                    parse_credential_line(kv).context(InvalidConfigSnafu {
-                        value: kv.to_string(),
+                        value: REDACTED_USER_PROVIDER_OPTION,
                         msg: "StaticUserProviderOption cmd values must be in format `user=pwd[,user=pwd]`",
                     })
                 })
@@ -53,7 +54,7 @@ impl StaticUserProvider {
                     StaticUserProvider { users }
                 }),
             _ => InvalidConfigSnafu {
-                value: mode.to_string(),
+                value: REDACTED_USER_PROVIDER_OPTION,
                 msg: "StaticUserProviderOption must be in format `file:<path>` or `cmd:<values>`",
             }
                 .fail(),

src/catalog/src/error.rs

@@ -291,8 +291,9 @@ pub enum Error {
         location: Location,
     },
 
-    #[snafu(display("Failed to handle query"))]
+    #[snafu(display("Failed to handle query, addr: {}", addr))]
     HandleQuery {
+        addr: String,
         source: common_meta::error::Error,
         #[snafu(implicit)]
         location: Location,

src/catalog/src/information_extension.rs

@@ -25,11 +25,15 @@ use common_meta::rpc::procedure;
 use common_procedure::{ProcedureInfo, ProcedureState};
 use common_query::request::QueryRequest;
 use common_recordbatch::adapter::{AsyncRecordBatchStreamAdapter, DfRecordBatchStreamAdapter};
+use common_recordbatch::filter::MetadataFilter;
 use common_recordbatch::util::{ChainedRecordBatchStream, LimitedRecordBatchStream};
-use common_recordbatch::{DfSendableRecordBatchStream, SendableRecordBatchStream};
-use datafusion::common::Result;
+use common_recordbatch::{
+    DfSendableRecordBatchStream, EmptyRecordBatchStream, SendableRecordBatchStream,
+};
 use datafusion::common::tree_node::TreeNodeRecursion;
+use datafusion::common::{Result, ScalarValue};
 use datafusion::execution::TaskContext;
+use datafusion::logical_expr::LogicalPlan;
 use datafusion::physical_expr::{EquivalenceProperties, Partitioning, PhysicalExpr};
 use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
 use datafusion::physical_plan::{DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties};
@@ -38,10 +42,13 @@ use datatypes::schema::SchemaRef;
 use futures_util::future::try_join_all;
 use meta_client::MetaClientRef;
 use snafu::ResultExt;
+use store_api::compaction_task;
 use store_api::storage::RegionId;
 
 use crate::error;
-use crate::information_schema::{DatanodeInspectRequest, InformationExtension};
+use crate::information_schema::{
+    DatanodeInspectKind, DatanodeInspectRequest, InformationExtension,
+};
 
 pub struct DistributedInformationExtension {
     meta_client: MetaClientRef,
@@ -60,18 +67,53 @@ impl DistributedInformationExtension {
         meta_client: MetaClientRef,
         datanode_manager: DatanodeManagerRef,
         request: DatanodeInspectRequest,
-    ) -> std::result::Result<SendableRecordBatchStream, crate::error::Error> {
+    ) -> crate::error::Result<SendableRecordBatchStream> {
         let nodes = meta_client
             .list_nodes(Some(Role::Datanode))
             .await
             .map_err(BoxedError::new)
             .context(crate::error::ListNodesSnafu)?;
+        Self::inspect_nodes(datanode_manager, request, nodes).await
+    }
 
+    /// Executes inspection against discovered nodes after applying safe node pruning.
+    async fn inspect_nodes(
+        datanode_manager: DatanodeManagerRef,
+        request: DatanodeInspectRequest,
+        nodes: Vec<NodeInfo>,
+    ) -> crate::error::Result<SendableRecordBatchStream> {
+        let nodes = prune_compaction_nodes(&request, nodes);
         let limit = request.scan.limit;
         let plan = request
             .build_plan()
             .context(crate::error::DatafusionSnafu)?;
+        let stream = Self::inspect_query_nodes(datanode_manager, plan, nodes).await?;
+        match limit {
+            Some(limit) => Ok(Box::pin(LimitedRecordBatchStream::new(stream, limit))),
+            None => Ok(stream),
+        }
+    }
 
+    async fn inspect_query_stream(
+        meta_client: MetaClientRef,
+        datanode_manager: DatanodeManagerRef,
+        plan: LogicalPlan,
+    ) -> crate::error::Result<SendableRecordBatchStream> {
+        let nodes = meta_client
+            .list_nodes(Some(Role::Datanode))
+            .await
+            .map_err(BoxedError::new)
+            .context(crate::error::ListNodesSnafu)?;
+        Self::inspect_query_nodes(datanode_manager, plan, nodes).await
+    }
+
+    /// Shares query dispatch and empty-stream schema handling for scans and query plans.
+    async fn inspect_query_nodes(
+        datanode_manager: DatanodeManagerRef,
+        plan: LogicalPlan,
+        nodes: Vec<NodeInfo>,
+    ) -> crate::error::Result<SendableRecordBatchStream> {
+        let schema = inspection_query_schema(&plan)?;
         let streams = try_join_all(nodes.into_iter().map(|node| {
             let datanode_manager = datanode_manager.clone();
             let plan = plan.clone();
@@ -84,25 +126,62 @@ impl DistributedInformationExtension {
                         header: None,
                     })
                     .await
-                    .context(crate::error::HandleQuerySnafu)
+                    .context(crate::error::HandleQuerySnafu {
+                        addr: node.peer.addr,
+                    })
             }
         }))
         .await?;
 
-        let chained =
-            ChainedRecordBatchStream::new(streams).context(crate::error::CreateRecordBatchSnafu)?;
-        match limit {
-            Some(limit) => Ok(Box::pin(LimitedRecordBatchStream::new(
-                Box::pin(chained),
-                limit,
-            ))),
-            None => Ok(Box::pin(chained)),
-        }
+        chain_inspection_streams(streams, schema)
     }
 }
 
+fn inspection_query_schema(plan: &LogicalPlan) -> crate::error::Result<SchemaRef> {
+    datatypes::schema::Schema::try_from(plan.schema().clone())
+        .map(Arc::new)
+        .map_err(|error| datafusion::common::DataFusionError::External(Box::new(error)))
+        .context(crate::error::DatafusionSnafu)
+}
+
+fn chain_inspection_streams(
+    streams: Vec<SendableRecordBatchStream>,
+    schema: SchemaRef,
+) -> crate::error::Result<SendableRecordBatchStream> {
+    if streams.is_empty() {
+        return Ok(Box::pin(EmptyRecordBatchStream::new(schema)));
+    }
+    Ok(Box::pin(
+        ChainedRecordBatchStream::new(streams).context(crate::error::CreateRecordBatchSnafu)?,
+    ))
+}
+
+#[derive(Debug, Clone)]
+enum InspectPlan {
+    Scan(DatanodeInspectRequest),
+    Query(LogicalPlan),
+}
+
+/// Prunes reporting nodes only for live compaction inspection. Region routes cannot
+/// locate tasks left on a former owner, so only known node IDs can exclude nodes.
+fn prune_compaction_nodes(request: &DatanodeInspectRequest, nodes: Vec<NodeInfo>) -> Vec<NodeInfo> {
+    if request.kind != DatanodeInspectKind::CompactionTasks {
+        return nodes;
+    }
+    let filter = MetadataFilter::new(&request.scan.filters);
+    nodes
+        .into_iter()
+        .filter(|node| {
+            filter.matches(|name| {
+                (name == compaction_task::NODE_ID)
+                    .then_some(ScalarValue::UInt64(Some(node.peer.id)))
+            })
+        })
+        .collect()
+}
+
 struct DistributedInspectExec {
-    request: DatanodeInspectRequest,
+    plan: InspectPlan,
     meta_client: MetaClientRef,
     datanode_manager: DatanodeManagerRef,
     schema: SchemaRef,
@@ -113,7 +192,7 @@ struct DistributedInspectExec {
 impl std::fmt::Debug for DistributedInspectExec {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         f.debug_struct("DistributedInspectExec")
-            .field("request", &self.request)
+            .field("plan", &self.plan)
             .field("schema", &self.schema)
             .finish_non_exhaustive()
     }
@@ -125,6 +204,20 @@ impl DistributedInspectExec {
         schema: SchemaRef,
         meta_client: MetaClientRef,
         datanode_manager: DatanodeManagerRef,
+    ) -> Self {
+        Self::with_plan(
+            InspectPlan::Scan(request),
+            schema,
+            meta_client,
+            datanode_manager,
+        )
+    }
+
+    fn with_plan(
+        plan: InspectPlan,
+        schema: SchemaRef,
+        meta_client: MetaClientRef,
+        datanode_manager: DatanodeManagerRef,
     ) -> Self {
         let arrow_schema = schema.arrow_schema().clone();
         let properties = Arc::new(PlanProperties::new(
@@ -134,7 +227,7 @@ impl DistributedInspectExec {
             Boundedness::Bounded,
         ));
         Self {
-            request,
+            plan,
             meta_client,
             datanode_manager,
             schema,
@@ -146,11 +239,19 @@ impl DistributedInspectExec {
 
 impl DisplayAs for DistributedInspectExec {
     fn fmt_as(&self, _t: DisplayFormatType, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        write!(
-            f,
-            "DistributedInspectExec: kind={:?}, scan={}, schema={:?}",
-            self.request.kind, self.request.scan, self.arrow_schema
-        )
+        match &self.plan {
+            InspectPlan::Scan(request) => write!(
+                f,
+                "DistributedInspectExec: kind={:?}, scan={}, schema={:?}",
+                request.kind, request.scan, self.arrow_schema
+            ),
+            InspectPlan::Query(plan) => write!(
+                f,
+                "DistributedInspectExec: query={}, schema={:?}",
+                plan.display_indent(),
+                self.arrow_schema
+            ),
+        }
     }
 }
 
@@ -205,15 +306,27 @@ impl ExecutionPlan for DistributedInspectExec {
 
         let meta_client = self.meta_client.clone();
         let datanode_manager = self.datanode_manager.clone();
-        let request = self.request.clone();
+        let plan = self.plan.clone();
         let schema = self.schema.clone();
         let future = async move {
-            DistributedInformationExtension::inspect_datanode_stream(
-                meta_client,
-                datanode_manager,
-                request,
-            )
-            .await
+            match plan {
+                InspectPlan::Scan(request) => {
+                    DistributedInformationExtension::inspect_datanode_stream(
+                        meta_client,
+                        datanode_manager,
+                        request,
+                    )
+                    .await
+                }
+                InspectPlan::Query(plan) => {
+                    DistributedInformationExtension::inspect_query_stream(
+                        meta_client,
+                        datanode_manager,
+                        plan,
+                    )
+                    .await
+                }
+            }
             .map_err(BoxedError::new)
             .context(common_recordbatch::error::ExternalSnafu)
         };
@@ -297,6 +410,23 @@ impl InformationExtension for DistributedInformationExtension {
         .await
     }
 
+    fn supports_inspect_query_plan(&self) -> bool {
+        true
+    }
+
+    fn inspect_datanode_query_plan(
+        &self,
+        plan: LogicalPlan,
+    ) -> crate::error::Result<Option<Arc<dyn ExecutionPlan>>> {
+        let schema = inspection_query_schema(&plan)?;
+        Ok(Some(Arc::new(DistributedInspectExec::with_plan(
+            InspectPlan::Query(plan),
+            schema,
+            self.meta_client.clone(),
+            self.datanode_manager.clone(),
+        ))))
+    }
+
     fn inspect_datanode_plan(
         &self,
         request: DatanodeInspectRequest,
@@ -310,3 +440,351 @@ impl InformationExtension for DistributedInformationExtension {
         ))))
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use common_meta::node_manager::{DatanodeManager, DatanodeRef};
+    use common_meta::peer::Peer;
+    use datafusion::common::tree_node::TreeNode;
+    use datafusion::datasource::empty::EmptyTable;
+    use datafusion::logical_expr::{Expr, LogicalPlan};
+    use datafusion::physical_plan::sorts::sort::SortExec;
+    use datafusion::prelude::SessionContext;
+    use datatypes::arrow::datatypes::DataType;
+    use meta_client::client::MetaClientBuilder;
+    use store_api::sst_entry::ManifestSstEntry;
+    use store_api::storage::ScanRequest;
+
+    use store_api::compaction_task::CompactionTaskEntry;
+
+    use super::*;
+
+    struct UnusedDatanodeManager;
+
+    #[async_trait::async_trait]
+    impl DatanodeManager for UnusedDatanodeManager {
+        async fn datanode(&self, _node: &Peer) -> DatanodeRef {
+            panic!("this test must not contact datanodes")
+        }
+    }
+
+    fn inspection_nodes() -> Vec<NodeInfo> {
+        [1, 2, 3]
+            .into_iter()
+            .map(|id| NodeInfo {
+                peer: Peer::new(id, format!("datanode-{id}:3001")),
+                last_activity_ts: 0,
+                status: common_meta::cluster::NodeStatus::Standalone,
+                version: String::new(),
+                git_commit: String::new(),
+                start_time_ms: 0,
+                total_cpu_millicores: 0,
+                total_memory_bytes: 0,
+                cpu_usage_millicores: 0,
+                memory_usage_bytes: 0,
+                hostname: String::new(),
+                env_vars: Default::default(),
+                schedulable: None,
+            })
+            .collect()
+    }
+
+    /// Uses the default optimizer to produce the predicates seen by inspection scans.
+    async fn compaction_sql_filters(predicate: Option<&str>) -> Vec<Expr> {
+        let ctx = SessionContext::new();
+        ctx.register_table(
+            "compaction_tasks",
+            Arc::new(EmptyTable::new(
+                CompactionTaskEntry::schema().arrow_schema().clone(),
+            )),
+        )
+        .unwrap();
+        let mut sql = "SELECT * FROM compaction_tasks".to_string();
+        if let Some(predicate) = predicate {
+            sql.push_str(" WHERE ");
+            sql.push_str(predicate);
+        }
+        let plan = ctx.sql(&sql).await.unwrap().into_optimized_plan().unwrap();
+        let mut filters = Vec::new();
+        plan.apply(|plan| {
+            match plan {
+                LogicalPlan::Filter(filter) => filters.push(filter.predicate.clone()),
+                LogicalPlan::TableScan(scan) => filters.extend(scan.filters.iter().cloned()),
+                _ => {}
+            }
+            Ok(TreeNodeRecursion::Continue)
+        })
+        .unwrap();
+        filters
+    }
+
+    #[tokio::test]
+    async fn test_compaction_node_pruning() {
+        for (predicate, expected, expect_cast) in [
+            (Some("node_id = 2"), vec![2], false),
+            (Some("node_id != 2"), vec![1, 3], false),
+            (Some("node_id > 2"), vec![3], false),
+            (Some("node_id >= 2"), vec![2, 3], false),
+            (Some("node_id < 2"), vec![1], false),
+            (Some("node_id <= 2"), vec![1, 2], false),
+            (Some("2 = node_id"), vec![2], false),
+            (Some("2 < node_id"), vec![3], false),
+            (None, vec![1, 2, 3], false),
+            (Some("state = 'running'"), vec![1, 2, 3], false),
+            (Some("node_id = 2 OR table_id = 42"), vec![1, 2, 3], false),
+            (Some("node_id = 2 AND state = 'running'"), vec![2], false),
+        ] {
+            let filters = compaction_sql_filters(predicate).await;
+            let mut has_cast = false;
+            for filter in &filters {
+                filter
+                    .apply(|expr| {
+                        has_cast |= matches!(expr, Expr::Cast(cast) if cast.field.data_type() == &DataType::Decimal128(20, 0)
+                            && matches!(cast.expr.as_ref(), Expr::Column(_)));
+                        Ok(TreeNodeRecursion::Continue)
+                    })
+                    .unwrap();
+            }
+            assert_eq!(has_cast, expect_cast, "optimized predicate: {predicate:?}");
+            let request = DatanodeInspectRequest {
+                kind: DatanodeInspectKind::CompactionTasks,
+                scan: ScanRequest {
+                    filters,
+                    ..Default::default()
+                },
+            };
+            let actual: Vec<_> = prune_compaction_nodes(&request, inspection_nodes())
+                .into_iter()
+                .map(|node| node.peer.id)
+                .collect();
+            assert_eq!(
+                actual, expected,
+                "{predicate:?}: {:?}",
+                request.scan.filters
+            );
+        }
+        let filters = compaction_sql_filters(Some("node_id = 99")).await;
+        for kind in [
+            DatanodeInspectKind::SstManifest,
+            DatanodeInspectKind::SstStorage,
+            DatanodeInspectKind::SstIndexMeta,
+            DatanodeInspectKind::RegionInfo,
+        ] {
+            let request = DatanodeInspectRequest {
+                kind,
+                scan: ScanRequest {
+                    filters: filters.clone(),
+                    ..Default::default()
+                },
+            };
+            let actual: Vec<_> = prune_compaction_nodes(&request, inspection_nodes())
+                .into_iter()
+                .map(|node| node.peer.id)
+                .collect();
+            assert_eq!(actual, vec![1, 2, 3]);
+        }
+    }
+
+    #[tokio::test]
+    async fn test_compaction_empty_nodes_preserve_projected_schema() {
+        let filters = compaction_sql_filters(Some("node_id = 99")).await;
+        for nodes in [vec![], inspection_nodes()] {
+            for (projection, expected) in [
+                (
+                    vec![3, 0, 2],
+                    vec![
+                        ("node_id", DataType::UInt64, true),
+                        ("task_id", DataType::Utf8, false),
+                        ("table_id", DataType::UInt32, false),
+                    ],
+                ),
+                (vec![], vec![]),
+            ] {
+                let request = DatanodeInspectRequest {
+                    kind: DatanodeInspectKind::CompactionTasks,
+                    scan: ScanRequest {
+                        projection: Some(projection),
+                        filters: filters.clone(),
+                        ..Default::default()
+                    },
+                };
+                // Panics if the production dispatch path contacts any excluded node.
+                let stream = DistributedInformationExtension::inspect_nodes(
+                    Arc::new(UnusedDatanodeManager),
+                    request,
+                    nodes.clone(),
+                )
+                .await
+                .unwrap();
+                let batches = common_recordbatch::RecordBatches::try_collect(stream)
+                    .await
+                    .unwrap();
+                assert_eq!(
+                    batches.iter().map(|batch| batch.num_rows()).sum::<usize>(),
+                    0
+                );
+                let schema = batches.schema();
+                let actual: Vec<_> = schema
+                    .arrow_schema()
+                    .fields()
+                    .iter()
+                    .map(|field| {
+                        (
+                            field.name().as_str(),
+                            field.data_type().clone(),
+                            field.is_nullable(),
+                        )
+                    })
+                    .collect();
+                assert_eq!(actual, expected);
+            }
+        }
+    }
+
+    #[derive(Debug)]
+    struct ManifestScanProvider;
+
+    #[async_trait::async_trait]
+    impl datafusion::catalog::TableProvider for ManifestScanProvider {
+        fn schema(&self) -> ArrowSchemaRef {
+            ManifestSstEntry::schema().arrow_schema().clone()
+        }
+
+        fn table_type(&self) -> datafusion::logical_expr::TableType {
+            datafusion::logical_expr::TableType::View
+        }
+
+        fn supports_filters_pushdown(
+            &self,
+            filters: &[&datafusion::logical_expr::Expr],
+        ) -> Result<Vec<datafusion::logical_expr::TableProviderFilterPushDown>> {
+            Ok(vec![datafusion::logical_expr::TableProviderFilterPushDown::Inexact; filters.len()])
+        }
+
+        async fn scan(
+            &self,
+            _state: &dyn datafusion::catalog::Session,
+            projection: Option<&Vec<usize>>,
+            filters: &[datafusion::logical_expr::Expr],
+            limit: Option<usize>,
+        ) -> Result<Arc<dyn ExecutionPlan>> {
+            let schema = ManifestSstEntry::schema();
+            let schema = match projection {
+                Some(projection) => Arc::new(schema.try_project(projection).unwrap()),
+                None => schema,
+            };
+            Ok(Arc::new(DistributedInspectExec::new(
+                DatanodeInspectRequest {
+                    kind: DatanodeInspectKind::SstManifest,
+                    scan: ScanRequest {
+                        projection: projection.cloned(),
+                        filters: filters.to_vec(),
+                        limit,
+                        ..Default::default()
+                    },
+                },
+                schema,
+                Arc::new(MetaClientBuilder::new(0, api::v1::meta::Role::Frontend).build()),
+                Arc::new(UnusedDatanodeManager),
+            )))
+        }
+    }
+
+    #[tokio::test]
+    async fn test_inspection_query_plan_is_lazy_and_preserves_schema() {
+        let ctx = datafusion::prelude::SessionContext::new();
+        let remote = ctx
+            .sql("SELECT count(*), sum(value) FROM (VALUES (1), (2)) AS t(value)")
+            .await
+            .unwrap()
+            .into_unoptimized_plan();
+        let extension = DistributedInformationExtension::new(
+            Arc::new(MetaClientBuilder::new(0, api::v1::meta::Role::Frontend).build()),
+            Arc::new(UnusedDatanodeManager),
+        );
+        assert!(extension.supports_inspect_query_plan());
+        let exec = extension
+            .inspect_datanode_query_plan(remote.clone())
+            .unwrap()
+            .unwrap();
+        assert_eq!(exec.schema(), remote.schema().inner().clone());
+        let inspect = exec.downcast_ref::<DistributedInspectExec>().unwrap();
+        let InspectPlan::Query(plan) = &inspect.plan else {
+            panic!("expected inspection query")
+        };
+        assert_eq!(plan, &remote);
+        assert!(exec.children().is_empty());
+        // Creating or dropping an unpolled execution must not contact metadata or datanodes.
+        for _ in 0..2 {
+            let stream = exec.execute(0, ctx.task_ctx()).unwrap();
+            assert_eq!(stream.schema(), exec.schema());
+            drop(stream);
+        }
+        assert!(exec.execute(1, ctx.task_ctx()).is_err());
+    }
+
+    #[tokio::test]
+    async fn test_empty_inspection_query_stream_keeps_output_schema() {
+        use futures_util::StreamExt;
+
+        let ctx = datafusion::prelude::SessionContext::new();
+        let remote = ctx
+            .sql("SELECT count(*), sum(value) FROM (VALUES (1)) AS t(value)")
+            .await
+            .unwrap()
+            .into_unoptimized_plan();
+        let schema = inspection_query_schema(&remote).unwrap();
+        let mut stream = chain_inspection_streams(vec![], schema.clone()).unwrap();
+        assert_eq!(stream.schema(), schema);
+        assert!(stream.next().await.is_none());
+    }
+
+    #[tokio::test]
+    async fn test_sql_manifest_sort_stays_on_frontend() {
+        let ctx = datafusion::prelude::SessionContext::new();
+        ctx.register_table("ssts_manifest", Arc::new(ManifestScanProvider))
+            .unwrap();
+        for sql in [
+            "SELECT region_id, file_id, min_ts, max_ts, file_size FROM ssts_manifest WHERE region_id = 42 ORDER BY min_ts LIMIT 100",
+            "SELECT file_id FROM ssts_manifest WHERE region_id = 42 ORDER BY max_ts DESC LIMIT 10 OFFSET 3",
+            "SELECT min_ts AS ts, file_id FROM ssts_manifest WHERE region_id = 42 ORDER BY ts DESC NULLS FIRST LIMIT 10",
+            "SELECT file_id FROM ssts_manifest WHERE region_id = 42 ORDER BY min_ts, max_ts DESC LIMIT 10",
+        ] {
+            let plan = ctx
+                .sql(sql)
+                .await
+                .unwrap()
+                .create_physical_plan()
+                .await
+                .unwrap();
+            let mut found_scan = false;
+            let mut found_sort = false;
+            plan.apply(|plan| {
+                found_sort |= plan.is::<SortExec>();
+                if let Some(scan) = plan.downcast_ref::<DistributedInspectExec>() {
+                    found_scan = true;
+                    // All matching rows must reach the frontend global sort.
+                    let InspectPlan::Scan(request) = &scan.plan else {
+                        panic!("expected scan plan")
+                    };
+                    assert_eq!(request.scan.limit, None);
+                    let remote = request.clone().build_plan().unwrap();
+                    remote.apply(|node| {
+                        assert!(!matches!(
+                            node,
+                            LogicalPlan::Sort(_) | LogicalPlan::Limit(_)
+                        ));
+                        Ok(TreeNodeRecursion::Continue)
+                    })?;
+                }
+                Ok(TreeNodeRecursion::Continue)
+            })
+            .unwrap();
+            assert!(
+                found_scan && found_sort,
+                "{}",
+                datafusion::physical_plan::displayable(plan.as_ref()).indent(true)
+            );
+        }
+    }
+}

src/catalog/src/process_manager.rs

@@ -25,7 +25,7 @@ use common_event_recorder::EventRecorderRef;
 use common_frontend::selector::{FrontendSelector, MetaClientSelector};
 use common_frontend::slow_query_event::SlowQueryEvent;
 use common_telemetry::logging::SlowQueriesRecordType;
-use common_telemetry::{debug, info, slow, warn};
+use common_telemetry::{debug, info, slow};
 use common_time::util::current_time_millis;
 use meta_client::MetaClientRef;
 use promql_parser::parser::EvalStmt;
@@ -41,8 +41,14 @@ pub type ProcessManagerRef = Arc<ProcessManager>;
 
 /// Query process manager.
 pub struct ProcessManager {
-    /// Local frontend server address,
-    server_addr: String,
+    /// Address this frontend advertises to the cluster.
+    ///
+    /// It is the heartbeat peer address, which is also the only address other
+    /// frontends use to reach this node's process management service. The same
+    /// value identifies processes owned by this frontend and routes kill
+    /// requests, so discovery addresses and process addresses share one
+    /// namespace.
+    peer_addr: String,
     /// Next process id for local queries.
     next_id: AtomicU32,
     /// Running process per catalog.
@@ -79,11 +85,15 @@ impl Display for QueryStatement {
 }
 
 impl ProcessManager {
-    /// Create a [ProcessManager] instance with server address and kv client.
-    pub fn new(server_addr: String, meta_client: Option<MetaClientRef>) -> Self {
+    /// Creates a [ProcessManager] for the frontend identified by `peer_addr`.
+    ///
+    /// `peer_addr` must be the address the frontend advertises to metasrv, not
+    /// its public gRPC address: discovered peers are matched by that address to
+    /// exclude this node from remote listings and to route kill requests.
+    pub fn new(peer_addr: String, meta_client: Option<MetaClientRef>) -> Self {
         let frontend_selector = meta_client.map(MetaClientSelector::new);
         Self {
-            server_addr,
+            peer_addr,
             next_id: Default::default(),
             catalogs: Default::default(),
             frontend_selector,
@@ -111,7 +121,7 @@ impl ProcessManager {
             query,
             start_timestamp: current_time_millis(),
             client,
-            frontend: self.server_addr.clone(),
+            frontend: self.peer_addr.clone(),
         };
         let cancellation_handle = Arc::new(CancellationHandle::default());
         let cancellable_process = CancellableProcess::new(cancellation_handle.clone(), process);
@@ -166,6 +176,13 @@ impl ProcessManager {
         Ok(result)
     }
 
+    /// Lists running processes from every selected frontend and the local registry.
+    ///
+    /// The local frontend is excluded from the remote selection by its advertised
+    /// peer address, which is what [`MetaClientSelector`] discovers. Its processes
+    /// are appended from the local registry exactly once.
+    ///
+    /// Returns an error if a remote frontend cannot provide its process list.
     pub async fn list_all_processes(
         &self,
         catalog: Option<&str>,
@@ -173,24 +190,17 @@ impl ProcessManager {
         let mut processes = vec![];
         if let Some(remote_frontend_selector) = self.frontend_selector.as_ref() {
             let frontends = remote_frontend_selector
-                .select(|peer| peer.addr != self.server_addr)
+                .select(|peer| peer.addr != self.peer_addr)
                 .await
                 .context(error::InvokeFrontendSnafu)?;
             for mut f in frontends {
-                let result = f
+                let resp = f
                     .list_process(ListProcessRequest {
                         catalog: catalog.unwrap_or_default().to_string(),
                     })
                     .await
-                    .context(error::InvokeFrontendSnafu);
-                match result {
-                    Ok(resp) => {
-                        processes.extend(resp.processes);
-                    }
-                    Err(e) => {
-                        warn!(e; "Skipping failing node: {:?}", f)
-                    }
-                }
+                    .context(error::InvokeFrontendSnafu)?;
+                processes.extend(resp.processes);
             }
         }
         processes.extend(self.local_processes(catalog)?);
@@ -198,29 +208,32 @@ impl ProcessManager {
     }
 
     /// Kills query with provided catalog and id.
+    ///
+    /// `peer_addr` is the frontend address from the process id, which is the
+    /// advertised peer address of the frontend that owns the process.
     pub async fn kill_process(
         &self,
-        server_addr: String,
+        peer_addr: String,
         catalog: String,
         id: ProcessId,
     ) -> error::Result<bool> {
-        if server_addr == self.server_addr {
+        if peer_addr == self.peer_addr {
             self.kill_local_process(catalog, id).await
         } else {
             let mut nodes = self
                 .frontend_selector
                 .as_ref()
                 .context(error::MetaClientMissingSnafu)?
-                .select(|peer| peer.addr == server_addr)
+                .select(|peer| peer.addr == peer_addr)
                 .await
                 .context(error::InvokeFrontendSnafu)?;
             ensure!(
                 !nodes.is_empty(),
-                error::FrontendNotFoundSnafu { addr: server_addr }
+                error::FrontendNotFoundSnafu { addr: peer_addr }
             );
 
             let request = KillProcessRequest {
-                server_addr,
+                server_addr: peer_addr,
                 catalog,
                 process_id: id,
             };
@@ -417,7 +430,7 @@ impl SlowQueryTimer {
             SlowQueriesRecordType::SystemTable => {
                 self.recorder.record(Box::new(slow_query_event));
             }
-            // Record the slow query in a specific logs file.
+            // Record the slow query through its dedicated logger.
             SlowQueriesRecordType::Log => {
                 slow!(
                     cost = slow_query_event.cost,

src/catalog/src/system_schema/information_schema.rs

@@ -14,6 +14,7 @@
 
 mod cluster_info;
 pub mod columns;
+mod compaction_tasks;
 pub mod flow_statistics;
 pub mod flows;
 mod information_memory_table;
@@ -58,6 +59,7 @@ use lazy_static::lazy_static;
 use paste::paste;
 use process_list::InformationSchemaProcessList;
 use region_info::InformationSchemaRegionInfo;
+use store_api::compaction_task::CompactionTaskEntry;
 use store_api::metric_engine_consts::{
     MEMTABLE_PARTITION_TREE_PRIMARY_KEY_ENCODING, PRIMARY_KEY_ENCODING,
 };
@@ -74,6 +76,7 @@ use crate::CatalogManager;
 use crate::error::Error;
 use crate::process_manager::ProcessManagerRef;
 use crate::system_schema::information_schema::cluster_info::InformationSchemaClusterInfo;
+use crate::system_schema::information_schema::compaction_tasks::InformationSchemaCompactionTasks;
 use crate::system_schema::information_schema::flow_statistics::InformationSchemaFlowStatistics;
 use crate::system_schema::information_schema::flows::InformationSchemaFlows;
 use crate::system_schema::information_schema::information_memory_table::get_schema_columns;
@@ -139,7 +142,7 @@ lazy_static! {
 }
 
 macro_rules! setup_memory_table {
-    ($name: expr) => {
+    ($name: expr, $scope: ident) => {
         paste! {
             {
                 let (schema, columns) = get_schema_columns($name);
@@ -147,7 +150,8 @@ macro_rules! setup_memory_table {
                     consts::[<INFORMATION_SCHEMA_ $name  _TABLE_ID>],
                     $name,
                     schema,
-                    columns
+                    columns,
+                    InformationSchemaTableScope::$scope
                 )) as _)
             }
         }
@@ -165,7 +169,9 @@ pub struct MakeInformationTableRequest {
 pub enum InformationSchemaTableScope {
     /// Every query is filtered to the catalog supplied when constructing the provider.
     Catalog,
-    /// The table contains catalog-independent or shared data, including compatibility stubs.
+    /// Catalog-independent data that is safe for authenticated non-admin users to read.
+    PublicGlobal,
+    /// Shared data without a non-admin visibility guarantee, including compatibility stubs.
     Global,
 }
 
@@ -173,6 +179,7 @@ impl InformationSchemaTableScope {
     pub const fn as_str(self) -> &'static str {
         match self {
             Self::Catalog => "catalog",
+            Self::PublicGlobal => "public_global",
             Self::Global => "global",
         }
     }
@@ -228,30 +235,30 @@ impl InformationSchemaProvider {
                 self.catalog_name.clone(),
                 self.catalog_manager.clone(),
             )) as _),
-            ENGINES => setup_memory_table!(ENGINES),
-            COLUMN_PRIVILEGES => setup_memory_table!(COLUMN_PRIVILEGES),
-            COLUMN_STATISTICS => setup_memory_table!(COLUMN_STATISTICS),
-            BUILD_INFO => setup_memory_table!(BUILD_INFO),
-            CHARACTER_SETS => setup_memory_table!(CHARACTER_SETS),
-            COLLATIONS => setup_memory_table!(COLLATIONS),
+            ENGINES => setup_memory_table!(ENGINES, PublicGlobal),
+            COLUMN_PRIVILEGES => setup_memory_table!(COLUMN_PRIVILEGES, Global),
+            COLUMN_STATISTICS => setup_memory_table!(COLUMN_STATISTICS, Global),
+            BUILD_INFO => setup_memory_table!(BUILD_INFO, PublicGlobal),
+            CHARACTER_SETS => setup_memory_table!(CHARACTER_SETS, PublicGlobal),
+            COLLATIONS => setup_memory_table!(COLLATIONS, PublicGlobal),
             COLLATION_CHARACTER_SET_APPLICABILITY => {
-                setup_memory_table!(COLLATION_CHARACTER_SET_APPLICABILITY)
+                setup_memory_table!(COLLATION_CHARACTER_SET_APPLICABILITY, PublicGlobal)
             }
-            CHECK_CONSTRAINTS => setup_memory_table!(CHECK_CONSTRAINTS),
-            EVENTS => setup_memory_table!(EVENTS),
-            FILES => setup_memory_table!(FILES),
-            OPTIMIZER_TRACE => setup_memory_table!(OPTIMIZER_TRACE),
-            PARAMETERS => setup_memory_table!(PARAMETERS),
-            PROFILING => setup_memory_table!(PROFILING),
-            REFERENTIAL_CONSTRAINTS => setup_memory_table!(REFERENTIAL_CONSTRAINTS),
-            ROUTINES => setup_memory_table!(ROUTINES),
-            SCHEMA_PRIVILEGES => setup_memory_table!(SCHEMA_PRIVILEGES),
-            TABLE_PRIVILEGES => setup_memory_table!(TABLE_PRIVILEGES),
-            GLOBAL_STATUS => setup_memory_table!(GLOBAL_STATUS),
-            SESSION_STATUS => setup_memory_table!(SESSION_STATUS),
-            PLUGINS => setup_memory_table!(PLUGINS),
-            USER_PRIVILEGES => setup_memory_table!(USER_PRIVILEGES),
-            PROCESSLIST => setup_memory_table!(PROCESSLIST),
+            CHECK_CONSTRAINTS => setup_memory_table!(CHECK_CONSTRAINTS, Global),
+            EVENTS => setup_memory_table!(EVENTS, Global),
+            FILES => setup_memory_table!(FILES, Global),
+            OPTIMIZER_TRACE => setup_memory_table!(OPTIMIZER_TRACE, Global),
+            PARAMETERS => setup_memory_table!(PARAMETERS, Global),
+            PROFILING => setup_memory_table!(PROFILING, Global),
+            REFERENTIAL_CONSTRAINTS => setup_memory_table!(REFERENTIAL_CONSTRAINTS, Global),
+            ROUTINES => setup_memory_table!(ROUTINES, Global),
+            SCHEMA_PRIVILEGES => setup_memory_table!(SCHEMA_PRIVILEGES, Global),
+            TABLE_PRIVILEGES => setup_memory_table!(TABLE_PRIVILEGES, Global),
+            GLOBAL_STATUS => setup_memory_table!(GLOBAL_STATUS, Global),
+            SESSION_STATUS => setup_memory_table!(SESSION_STATUS, Global),
+            PLUGINS => setup_memory_table!(PLUGINS, Global),
+            USER_PRIVILEGES => setup_memory_table!(USER_PRIVILEGES, Global),
+            PROCESSLIST => setup_memory_table!(PROCESSLIST, Global),
             KEY_COLUMN_USAGE => Some(Arc::new(InformationSchemaKeyColumnUsage::new(
                 self.catalog_name.clone(),
                 self.catalog_manager.clone(),
@@ -308,6 +315,9 @@ impl InformationSchemaProvider {
                     self.catalog_manager.clone(),
                 ),
             ) as _),
+            COMPACTION_TASKS => Some(Arc::new(InformationSchemaCompactionTasks::new(
+                self.catalog_manager.clone(),
+            )) as _),
             REGION_INFO => Some(Arc::new(InformationSchemaRegionInfo::new(
                 self.catalog_manager.clone(),
             )) as _),
@@ -374,11 +384,11 @@ impl InformationSchemaProvider {
         // Only put these tables under `greptime` catalog to prevent info leak.
         if self.catalog_name == DEFAULT_CATALOG_NAME {
             table_names.extend([
-                BUILD_INFO,
                 REGION_PEERS,
                 CLUSTER_INFO,
                 PROCEDURE_INFO,
                 REGION_STATISTICS,
+                COMPACTION_TASKS,
                 REGION_INFO,
                 SSTS_MANIFEST,
                 SSTS_STORAGE,
@@ -387,6 +397,7 @@ impl InformationSchemaProvider {
         }
 
         table_names.extend([
+            BUILD_INFO,
             TABLES,
             VIEWS,
             SCHEMATA,
@@ -462,6 +473,21 @@ pub trait InformationExtension {
         request: DatanodeInspectRequest,
     ) -> std::result::Result<SendableRecordBatchStream, Self::Error>;
 
+    /// Whether this extension can execute an inspection query on every datanode.
+    fn supports_inspect_query_plan(&self) -> bool {
+        false
+    }
+
+    /// Builds a plan that executes an inspection query independently on each datanode.
+    /// The caller supplies a plan over reserved inspection tables and retains any
+    /// required global aggregation. The returned schema is the query output schema.
+    fn inspect_datanode_query_plan(
+        &self,
+        _plan: LogicalPlan,
+    ) -> std::result::Result<Option<Arc<dyn ExecutionPlan>>, Self::Error> {
+        Ok(None)
+    }
+
     /// Builds a physical plan for datanode inspect if the extension can expose
     /// the distributed fan-in semantics to DataFusion.
     fn inspect_datanode_plan(
@@ -487,6 +513,8 @@ pub struct DatanodeInspectRequest {
 /// The kind of the datanode inspect request.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum DatanodeInspectKind {
+    /// Live compaction cycles.
+    CompactionTasks,
     /// List SST entries recorded in manifest
     SstManifest,
     /// List SST entries discovered in storage layer
@@ -501,6 +529,7 @@ impl DatanodeInspectRequest {
     /// Builds a logical plan for the datanode inspect request.
     pub fn build_plan(self) -> std::result::Result<LogicalPlan, DataFusionError> {
         match self.kind {
+            DatanodeInspectKind::CompactionTasks => CompactionTaskEntry::build_plan(self.scan),
             DatanodeInspectKind::SstManifest => ManifestSstEntry::build_plan(self.scan),
             DatanodeInspectKind::SstStorage => StorageSstEntry::build_plan(self.scan),
             DatanodeInspectKind::SstIndexMeta => PuffinIndexMetaEntry::build_plan(self.scan),
@@ -605,14 +634,15 @@ mod tests {
 
     #[test]
     fn every_registered_information_schema_table_has_scope() {
-        use InformationSchemaTableScope::{Catalog, Global};
+        use InformationSchemaTableScope::{Catalog, Global, PublicGlobal};
 
         let manager: Arc<dyn CatalogManager> = MemoryCatalogManager::new();
         let backend = Arc::new(MemoryKvBackend::default());
         let factories: HashMap<_, InformationSchemaTableFactoryRef> = [
             ("future_catalog_table", Catalog),
             ("future_global_table", Global),
-            (ENGINES, Catalog),
+            ("future_public_table", PublicGlobal),
+            (BUILD_INFO, Global),
         ]
         .into_iter()
         .map(|(name, scope)| {
@@ -638,7 +668,13 @@ mod tests {
             #[cfg(feature = "enterprise")]
             RECYCLE_BIN,
             "future_catalog_table",
+        ];
+        let public_global_tables = [
             ENGINES,
+            CHARACTER_SETS,
+            COLLATIONS,
+            COLLATION_CHARACTER_SET_APPLICABILITY,
+            "future_public_table",
         ];
 
         for (catalog, with_process_manager) in [
@@ -654,22 +690,31 @@ mod tests {
                 with_process_manager.then(|| Arc::new(ProcessManager::new(String::new(), None))),
                 backend.clone(),
             );
-            assert_eq!(
-                provider
-                    .table(ENGINES)
-                    .unwrap()
-                    .table_info()
-                    .meta
-                    .options
-                    .extra_options
-                    .get(SEMANTIC_TABLE_SCOPE)
-                    .map(String::as_str),
-                Some(Global.as_str()),
-            );
+            for name in [
+                BUILD_INFO,
+                ENGINES,
+                CHARACTER_SETS,
+                COLLATIONS,
+                COLLATION_CHARACTER_SET_APPLICABILITY,
+            ] {
+                assert_eq!(
+                    provider
+                        .table(name)
+                        .unwrap()
+                        .table_info()
+                        .meta
+                        .options
+                        .extra_options
+                        .get(SEMANTIC_TABLE_SCOPE)
+                        .map(String::as_str),
+                    Some(PublicGlobal.as_str()),
+                    "{catalog}.{name}"
+                );
+            }
             // An extension replacing a built-in supplies the replacement table's scope.
             let provider = provider.with_extra_table_factories(factories.clone());
             let expected_count =
-                43 + usize::from(with_process_manager) + usize::from(cfg!(feature = "enterprise"))
+                45 + usize::from(with_process_manager) + usize::from(cfg!(feature = "enterprise"))
                     - if catalog == DEFAULT_CATALOG_NAME {
                         0
                     } else {
@@ -679,6 +724,8 @@ mod tests {
             for (name, table) in provider.tables() {
                 let scope = if catalog_tables.contains(&name.as_str()) {
                     Catalog
+                } else if public_global_tables.contains(&name.as_str()) {
+                    PublicGlobal
                 } else {
                     Global
                 };
@@ -787,6 +834,29 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_compaction_tasks_build_plan() {
+        let request = DatanodeInspectRequest {
+            kind: DatanodeInspectKind::CompactionTasks,
+            scan: ScanRequest {
+                projection: Some(vec![0]),
+                filters: vec![
+                    datafusion::logical_expr::col("state")
+                        .eq(datafusion::logical_expr::lit("running")),
+                ],
+                limit: Some(1),
+                ..Default::default()
+            },
+        };
+        let plan = request.build_plan().unwrap();
+        assert_eq!(plan.schema().fields().len(), 1);
+        assert!(
+            plan.display_indent()
+                .to_string()
+                .contains("__compaction_tasks")
+        );
+    }
+
     #[test]
     fn test_datanode_inspect_region_info_build_plan() {
         let plan = DatanodeInspectRequest {

src/catalog/src/system_schema/information_schema/compaction_tasks.rs

@@ -0,0 +1,97 @@
+// Copyright 2023 Greptime Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use std::sync::{Arc, Weak};
+
+use common_catalog::consts::INFORMATION_SCHEMA_COMPACTION_TASKS_TABLE_ID;
+use common_error::ext::BoxedError;
+use common_recordbatch::SendableRecordBatchStream;
+use common_recordbatch::adapter::AsyncRecordBatchStreamAdapter;
+use datafusion::physical_plan::ExecutionPlan;
+use datatypes::schema::SchemaRef;
+use snafu::ResultExt;
+use store_api::compaction_task::CompactionTaskEntry;
+use store_api::storage::{ScanRequest, TableId};
+
+use crate::CatalogManager;
+use crate::error::Result;
+use crate::information_schema::{
+    COMPACTION_TASKS, DatanodeInspectKind, InformationSchemaTableScope, InformationTable,
+};
+use crate::system_schema::SystemTable;
+use crate::system_schema::utils;
+
+/// Information schema table for live compaction tasks.
+pub struct InformationSchemaCompactionTasks {
+    schema: SchemaRef,
+    catalog_manager: Weak<dyn CatalogManager>,
+}
+
+impl InformationSchemaCompactionTasks {
+    pub(crate) fn new(catalog_manager: Weak<dyn CatalogManager>) -> Self {
+        Self {
+            schema: CompactionTaskEntry::schema(),
+            catalog_manager,
+        }
+    }
+}
+
+impl InformationTable for InformationSchemaCompactionTasks {
+    fn scope(&self) -> InformationSchemaTableScope {
+        InformationSchemaTableScope::Global
+    }
+}
+
+impl SystemTable for InformationSchemaCompactionTasks {
+    fn table_id(&self) -> TableId {
+        INFORMATION_SCHEMA_COMPACTION_TASKS_TABLE_ID
+    }
+
+    fn table_name(&self) -> &'static str {
+        COMPACTION_TASKS
+    }
+
+    fn schema(&self) -> SchemaRef {
+        self.schema.clone()
+    }
+
+    fn to_stream(&self, request: ScanRequest) -> Result<SendableRecordBatchStream> {
+        let schema = utils::projected_schema(&self.schema, &request)?;
+
+        let info_ext = utils::information_extension(&self.catalog_manager)?;
+        let req =
+            utils::inspect_request(DatanodeInspectKind::CompactionTasks, &self.schema, request)?;
+
+        let future = async move {
+            info_ext
+                .inspect_datanode(req)
+                .await
+                .map_err(BoxedError::new)
+                .context(common_recordbatch::error::ExternalSnafu)
+        };
+        Ok(Box::pin(AsyncRecordBatchStreamAdapter::new(
+            schema,
+            Box::pin(future),
+        )))
+    }
+
+    fn scan_plan(&self, request: ScanRequest) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+        utils::inspect_plan(
+            &self.catalog_manager,
+            DatanodeInspectKind::CompactionTasks,
+            &self.schema,
+            request,
+        )
+    }
+}

src/catalog/src/system_schema/information_schema/region_statistics.rs

@@ -75,7 +75,12 @@ const INIT_CAPACITY: usize = 42;
 /// - `query_scanned_bytes`: The total bytes scanned by queries since region opened.
 /// - `memtable_size`: The memtable size in bytes.
 /// - `disk_size`: The approximate disk size in bytes.
-/// - `manifest_size`: The manifest size in bytes.
+/// - `manifest_size`: Stored bytes of the latest observed checkpoint and subsequent
+///   deltas (compressed when enabled), excluding staging and `_last_checkpoint`.
+///   Older files retained on disk, including failed deletions, are not counted.
+///   Followers update this estimate on sync; metadata read failures can leave it stale.
+///   Promotion does not refresh it, so an incomplete estimate can persist without
+///   further syncs or checkpoints.
 /// - `sst_size`: The sst data files size in bytes.
 /// - `index_size`: The sst index files size in bytes.
 /// - `engine`: The engine type.

src/catalog/src/system_schema/information_schema/ssts.rs

@@ -28,38 +28,14 @@ use store_api::sst_entry::{ManifestSstEntry, PuffinIndexMetaEntry, StorageSstEnt
 use store_api::storage::{ScanRequest, TableId};
 
 use crate::CatalogManager;
-use crate::error::{ProjectSchemaSnafu, Result};
+use crate::error::Result;
 use crate::information_schema::{
-    DatanodeInspectKind, DatanodeInspectRequest, InformationSchemaTableScope, InformationTable,
-    SSTS_INDEX_META, SSTS_MANIFEST, SSTS_STORAGE,
+    DatanodeInspectKind, InformationSchemaTableScope, InformationTable, SSTS_INDEX_META,
+    SSTS_MANIFEST, SSTS_STORAGE,
 };
 use crate::system_schema::SystemTable;
 use crate::system_schema::utils;
 
-fn projected_schema(schema: &SchemaRef, request: &ScanRequest) -> Result<SchemaRef> {
-    if let Some(p) = request.projection.as_deref() {
-        Ok(Arc::new(schema.try_project(p).context(ProjectSchemaSnafu)?))
-    } else {
-        Ok(schema.clone())
-    }
-}
-
-fn inspect_plan(
-    catalog_manager: &Weak<dyn CatalogManager>,
-    kind: DatanodeInspectKind,
-    schema: &SchemaRef,
-    request: ScanRequest,
-) -> Result<Option<Arc<dyn ExecutionPlan>>> {
-    let projected_schema = projected_schema(schema, &request)?;
-    let info_ext = utils::information_extension(catalog_manager)?;
-    let req = DatanodeInspectRequest {
-        kind,
-        scan: request,
-    };
-
-    info_ext.inspect_datanode_plan(req, projected_schema)
-}
-
 /// Information schema table for sst manifest.
 pub struct InformationSchemaSstsManifest {
     schema: SchemaRef,
@@ -95,12 +71,9 @@ impl SystemTable for InformationSchemaSstsManifest {
     }
 
     fn to_stream(&self, request: ScanRequest) -> Result<SendableRecordBatchStream> {
-        let schema = projected_schema(&self.schema, &request)?;
+        let schema = utils::projected_schema(&self.schema, &request)?;
         let info_ext = utils::information_extension(&self.catalog_manager)?;
-        let req = DatanodeInspectRequest {
-            kind: DatanodeInspectKind::SstManifest,
-            scan: request,
-        };
+        let req = utils::inspect_request(DatanodeInspectKind::SstManifest, &self.schema, request)?;
 
         let future = async move {
             info_ext
@@ -116,7 +89,7 @@ impl SystemTable for InformationSchemaSstsManifest {
     }
 
     fn scan_plan(&self, request: ScanRequest) -> Result<Option<Arc<dyn ExecutionPlan>>> {
-        inspect_plan(
+        utils::inspect_plan(
             &self.catalog_manager,
             DatanodeInspectKind::SstManifest,
             &self.schema,
@@ -160,13 +133,10 @@ impl SystemTable for InformationSchemaSstsStorage {
     }
 
     fn to_stream(&self, request: ScanRequest) -> Result<SendableRecordBatchStream> {
-        let schema = projected_schema(&self.schema, &request)?;
+        let schema = utils::projected_schema(&self.schema, &request)?;
 
         let info_ext = utils::information_extension(&self.catalog_manager)?;
-        let req = DatanodeInspectRequest {
-            kind: DatanodeInspectKind::SstStorage,
-            scan: request,
-        };
+        let req = utils::inspect_request(DatanodeInspectKind::SstStorage, &self.schema, request)?;
 
         let future = async move {
             info_ext
@@ -182,7 +152,7 @@ impl SystemTable for InformationSchemaSstsStorage {
     }
 
     fn scan_plan(&self, request: ScanRequest) -> Result<Option<Arc<dyn ExecutionPlan>>> {
-        inspect_plan(
+        utils::inspect_plan(
             &self.catalog_manager,
             DatanodeInspectKind::SstStorage,
             &self.schema,
@@ -226,13 +196,10 @@ impl SystemTable for InformationSchemaSstsIndexMeta {
     }
 
     fn to_stream(&self, request: ScanRequest) -> Result<SendableRecordBatchStream> {
-        let schema = projected_schema(&self.schema, &request)?;
+        let schema = utils::projected_schema(&self.schema, &request)?;
 
         let info_ext = utils::information_extension(&self.catalog_manager)?;
-        let req = DatanodeInspectRequest {
-            kind: DatanodeInspectKind::SstIndexMeta,
-            scan: request,
-        };
+        let req = utils::inspect_request(DatanodeInspectKind::SstIndexMeta, &self.schema, request)?;
 
         let future = async move {
             info_ext
@@ -248,7 +215,7 @@ impl SystemTable for InformationSchemaSstsIndexMeta {
     }
 
     fn scan_plan(&self, request: ScanRequest) -> Result<Option<Arc<dyn ExecutionPlan>>> {
-        inspect_plan(
+        utils::inspect_plan(
             &self.catalog_manager,
             DatanodeInspectKind::SstIndexMeta,
             &self.schema,
@@ -259,6 +226,7 @@ impl SystemTable for InformationSchemaSstsIndexMeta {
 
 #[cfg(test)]
 mod tests {
+    use crate::information_schema::DatanodeInspectRequest;
     use cache::{build_fundamental_cache_registry, with_default_composite_cache_registry};
     use common_catalog::consts::{DEFAULT_CATALOG_NAME, INFORMATION_SCHEMA_NAME};
     use common_meta::cache::{CacheRegistryBuilder, LayeredCacheRegistryBuilder};
@@ -317,6 +285,63 @@ mod tests {
         }
     }
 
+    #[test]
+    fn test_inspect_request_keeps_unsafe_filters_on_frontend() {
+        use datafusion::arrow::datatypes::{DataType, TimeUnit};
+        use datafusion::logical_expr::expr::Cast;
+        use datafusion::logical_expr::{Expr, col, lit};
+
+        for (kind, schema) in [
+            (
+                DatanodeInspectKind::CompactionTasks,
+                store_api::compaction_task::CompactionTaskEntry::schema(),
+            ),
+            (DatanodeInspectKind::SstManifest, ManifestSstEntry::schema()),
+            (DatanodeInspectKind::SstStorage, StorageSstEntry::schema()),
+            (
+                DatanodeInspectKind::SstIndexMeta,
+                PuffinIndexMetaEntry::schema(),
+            ),
+        ] {
+            let safe = col(&schema.column_schemas()[0].name).is_not_null();
+            let scan = ScanRequest {
+                projection: Some(vec![0]),
+                filters: vec![safe.clone()],
+                limit: Some(1),
+                ..Default::default()
+            };
+            assert_eq!(
+                utils::inspect_request(kind, &schema, scan.clone()).unwrap(),
+                DatanodeInspectRequest {
+                    kind,
+                    scan: scan.clone()
+                }
+            );
+            for unsafe_filter in [
+                Expr::Cast(Cast::new(
+                    Box::new(lit(0_i64)),
+                    DataType::Timestamp(TimeUnit::Nanosecond, Some("+08:00".into())),
+                ))
+                .is_not_null(),
+                Expr::Cast(Cast::new(Box::new(lit(1_i64)), DataType::Decimal32(9, 0)))
+                    .is_not_null(),
+                col("missing_column").is_null(),
+            ] {
+                let mut input = scan.clone();
+                input.filters.push(unsafe_filter);
+                let mut expected = scan.clone();
+                expected.limit = None;
+                assert_eq!(
+                    utils::inspect_request(kind, &schema, input).unwrap(),
+                    DatanodeInspectRequest {
+                        kind,
+                        scan: expected
+                    }
+                );
+            }
+        }
+    }
+
     #[tokio::test]
     async fn ssts_manifest_uses_information_extension_scan_plan() {
         let backend = Arc::new(MemoryKvBackend::default());
@@ -349,6 +374,13 @@ mod tests {
             .unwrap()
             .unwrap();
 
+        let filter =
+            datafusion::logical_expr::col("region_id").eq(datafusion::logical_expr::lit(42_u64));
+        assert_eq!(
+            table.supports_filters_pushdown(&[&filter]).unwrap(),
+            vec![table::metadata::FilterPushDownType::Inexact],
+        );
+
         let request = ScanRequest {
             projection: Some(vec![0]),
             ..Default::default()

src/catalog/src/system_schema/information_schema/table_names.rs

@@ -60,3 +60,6 @@ pub const USER_PRIVILEGES: &str = "user_privileges";
 /// MySQL's session list. GreptimeDB reports its own sessions through [`PROCESS_LIST`];
 /// this table only exists so MySQL tooling finds the name and column shape it expects.
 pub const PROCESSLIST: &str = "processlist";
+
+/// Live compaction tasks.
+pub const COMPACTION_TASKS: &str = "compaction_tasks";

src/catalog/src/system_schema/memory_table.rs

@@ -40,6 +40,7 @@ pub(crate) struct MemoryTable {
     pub(crate) table_name: &'static str,
     pub(crate) schema: SchemaRef,
     pub(crate) columns: Vec<VectorRef>,
+    scope: InformationSchemaTableScope,
 }
 
 impl MemoryTable {
@@ -49,12 +50,14 @@ impl MemoryTable {
         table_name: &'static str,
         schema: SchemaRef,
         columns: Vec<VectorRef>,
+        scope: InformationSchemaTableScope,
     ) -> Self {
         Self {
             table_id,
             table_name,
             schema,
             columns,
+            scope,
         }
     }
 
@@ -107,7 +110,7 @@ impl DfPartitionStream for MemoryTable {
 
 impl InformationTable for MemoryTable {
     fn scope(&self) -> InformationSchemaTableScope {
-        InformationSchemaTableScope::Global
+        self.scope
     }
 }
 
@@ -172,6 +175,7 @@ mod tests {
                 Arc::new(StringVector::from(vec!["a1", "a2"])),
                 Arc::new(StringVector::from(vec!["b1", "b2"])),
             ],
+            InformationSchemaTableScope::Global,
         );
 
         assert_eq!(42, table.table_id());
@@ -201,7 +205,13 @@ mod tests {
             ColumnSchema::new("b", ConcreteDataType::string_datatype(), false),
         ]));
 
-        let table = MemoryTable::new(42, "test", schema.clone(), vec![]);
+        let table = MemoryTable::new(
+            42,
+            "test",
+            schema.clone(),
+            vec![],
+            InformationSchemaTableScope::Global,
+        );
 
         assert_eq!(42, table.table_id());
         assert_eq!("test", table.table_name());

src/catalog/src/system_schema/utils.rs

@@ -12,18 +12,82 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use std::sync::Weak;
+pub mod tables;
+
+use std::sync::{Arc, Weak};
 
 use common_meta::key::TableMetadataManagerRef;
-use snafu::OptionExt;
+use common_query::logical_plan::is_lossless_substrait_type;
+use datafusion::common::DFSchema;
+use datafusion::common::tree_node::{TreeNode, TreeNodeRecursion};
+use datafusion::logical_expr::ExprSchemable;
+use datafusion::physical_plan::ExecutionPlan;
+use datatypes::schema::SchemaRef;
+use snafu::{OptionExt, ResultExt};
+use store_api::storage::ScanRequest;
 
 use crate::CatalogManager;
-use crate::error::{GetInformationExtensionSnafu, Result, UpgradeWeakCatalogManagerRefSnafu};
-use crate::information_schema::InformationExtensionRef;
+use crate::error::{
+    GetInformationExtensionSnafu, ProjectSchemaSnafu, Result, UpgradeWeakCatalogManagerRefSnafu,
+};
+use crate::information_schema::{
+    DatanodeInspectKind, DatanodeInspectRequest, InformationExtensionRef,
+};
 use crate::kvbackend::KvBackendCatalogManager;
 use crate::system_schema::semantic_graph::EntityGraphProviderRef;
 
-pub mod tables;
+/// Projects an inspection table schema according to the scan request.
+pub(crate) fn projected_schema(schema: &SchemaRef, request: &ScanRequest) -> Result<SchemaRef> {
+    if let Some(p) = request.projection.as_deref() {
+        Ok(Arc::new(schema.try_project(p).context(ProjectSchemaSnafu)?))
+    } else {
+        Ok(schema.clone())
+    }
+}
+
+/// Builds a datanode inspection plan with the requested output schema.
+pub(crate) fn inspect_plan(
+    catalog_manager: &Weak<dyn CatalogManager>,
+    kind: DatanodeInspectKind,
+    schema: &SchemaRef,
+    request: ScanRequest,
+) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+    let projected_schema = projected_schema(schema, &request)?;
+    let info_ext = information_extension(catalog_manager)?;
+    let req = inspect_request(kind, schema, request)?;
+
+    info_ext.inspect_datanode_plan(req, projected_schema)
+}
+
+/// System table predicates are inexact: the frontend keeps a residual filter. Dropping
+/// an unsafe remote filter preserves correctness only if no remote limit truncates it.
+pub(crate) fn inspect_request(
+    kind: DatanodeInspectKind,
+    schema: &SchemaRef,
+    mut scan: ScanRequest,
+) -> Result<DatanodeInspectRequest> {
+    let schema =
+        DFSchema::try_from(schema.arrow_schema().clone()).context(crate::error::DatafusionSnafu)?;
+    let filter_count = scan.filters.len();
+    scan.filters.retain(|filter| {
+        let mut supported = true;
+        let visited = filter.apply(|expr| {
+            if !expr
+                .get_type(&schema)
+                .is_ok_and(|data_type| is_lossless_substrait_type(&data_type))
+            {
+                supported = false;
+                return Ok(TreeNodeRecursion::Stop);
+            }
+            Ok(TreeNodeRecursion::Continue)
+        });
+        visited.is_ok() && supported
+    });
+    if scan.filters.len() != filter_count {
+        scan.limit = None;
+    }
+    Ok(DatanodeInspectRequest { kind, scan })
+}
 
 /// Try to get the entity-graph provider from a `[CatalogManager]` weak reference.
 /// Returns `None` when the manager is not the kv-backed one or the provider has

src/client/src/error.rs

@@ -18,7 +18,7 @@ use common_error::define_from_tonic_status;
 use common_error::ext::{BoxedError, ErrorExt, RetryHint};
 use common_error::status_code::StatusCode;
 use common_macro::stack_trace_debug;
-use snafu::{Location, Snafu, location};
+use snafu::{Location, OptionExt, Snafu, location};
 use tonic::Code;
 use tonic::metadata::errors::InvalidMetadataValue;
 
@@ -26,6 +26,15 @@ use tonic::metadata::errors::InvalidMetadataValue;
 #[snafu(visibility(pub))]
 #[stack_trace_debug]
 pub enum Error {
+    #[snafu(display(
+        "Region request has {} failed entries; {}",
+        response.failures.len(),
+        response.first_failure().map(|(index, error)| format!("entry {index}: {}", error.output_msg())).unwrap_or_default()
+    ))]
+    PartialRegion {
+        response: Box<api::region::BatchRegionResponse<Error>>,
+    },
+
     #[snafu(display("Illegal Flight messages, reason: {}", reason))]
     IllegalFlightMessages {
         reason: String,
@@ -33,7 +42,7 @@ pub enum Error {
         location: Location,
     },
 
-    #[snafu(display("Failed to do Flight get, code: {}", tonic_code))]
+    #[snafu(display("Failed to do Flight get, addr: {}, code: {}", addr, tonic_code))]
     FlightGet {
         addr: String,
         tonic_code: Code,
@@ -109,11 +118,12 @@ pub enum Error {
         location: Location,
     },
 
-    // Server error carried in Tonic Status's metadata.
+    // Server error carried in a protobuf response status.
     #[snafu(display("{}", msg))]
     Server {
         code: StatusCode,
         msg: String,
+        retry_hint: RetryHint,
         #[snafu(implicit)]
         location: Location,
     },
@@ -163,6 +173,9 @@ pub type Result<T> = std::result::Result<T, Error>;
 impl ErrorExt for Error {
     fn status_code(&self) -> StatusCode {
         match self {
+            Error::PartialRegion { response } => response
+                .first_failure()
+                .map_or(StatusCode::Internal, |(_, error)| error.status_code()),
             Error::IllegalFlightMessages { .. }
             | Error::MissingField { .. }
             | Error::IllegalDatabaseResponse { .. } => StatusCode::Internal,
@@ -188,7 +201,8 @@ impl ErrorExt for Error {
 
     fn retry_hint(&self) -> RetryHint {
         match self {
-            Error::Tonic { retry_hint, .. } => *retry_hint,
+            Error::PartialRegion { .. } => RetryHint::Retryable,
+            Error::Tonic { retry_hint, .. } | Error::Server { retry_hint, .. } => *retry_hint,
             Error::FlightGet { source, .. }
             | Error::FlightStream { source, .. }
             | Error::FlowServer { source, .. }
@@ -220,9 +234,22 @@ define_from_tonic_status!(Error, Tonic);
 
 impl From<Error> for common_meta::error::Error {
     fn from(error: Error) -> Self {
-        Self::External {
-            source: BoxedError::new(error),
-            location: location!(),
+        match error {
+            Error::PartialRegion { response } => Self::PartialRegion {
+                response: Box::new(api::region::BatchRegionResponse {
+                    affected_rows: response.affected_rows,
+                    extensions: response.extensions,
+                    failures: response
+                        .failures
+                        .into_iter()
+                        .map(|(index, error)| (index, error.into()))
+                        .collect(),
+                }),
+            },
+            error => Self::External {
+                source: BoxedError::new(error),
+                location: location!(),
+            },
         }
     }
 }
@@ -258,3 +285,20 @@ impl Error {
             )
     }
 }
+
+impl Error {
+    /// Decodes an application error carried in a protobuf response.
+    pub(crate) fn try_from_response_status(status: &api::v1::Status) -> Result<Self> {
+        let code = StatusCode::from_u32(status.status_code).with_context(|| {
+            IllegalDatabaseResponseSnafu {
+                err_msg: format!("Unknown server status: {}", status.status_code),
+            }
+        })?;
+        Ok(ServerSnafu {
+            code,
+            msg: status.err_msg.clone(),
+            retry_hint: status.retry_hint.parse().unwrap_or(RetryHint::NonRetryable),
+        }
+        .build())
+    }
+}

src/client/src/lib.rs

@@ -35,7 +35,7 @@ use snafu::OptionExt;
 pub use self::client::{Client, ClientOptions};
 pub use self::database::{Database, OutputMetrics, OutputWithMetrics};
 pub use self::error::{Error, Result};
-use crate::error::{IllegalDatabaseResponseSnafu, ServerSnafu};
+use crate::error::IllegalDatabaseResponseSnafu;
 
 pub fn from_grpc_response(response: GreptimeResponse) -> Result<u32> {
     let header = response.header.context(IllegalDatabaseResponseSnafu {
@@ -53,14 +53,6 @@ pub fn from_grpc_response(response: GreptimeResponse) -> Result<u32> {
             Response::AffectedRows(AffectedRows { value }) => Ok(value),
         }
     } else {
-        let status_code =
-            StatusCode::from_u32(status.status_code).context(IllegalDatabaseResponseSnafu {
-                err_msg: format!("invalid status: {:?}", status),
-            })?;
-        ServerSnafu {
-            code: status_code,
-            msg: status.err_msg,
-        }
-        .fail()
+        Err(Error::try_from_response_status(&status)?)
     }
 }

src/client/src/region.rs

@@ -18,8 +18,9 @@ use std::borrow::Borrow;
 use std::sync::Arc;
 use std::time::Duration;
 
-use api::region::RegionResponse;
+use api::region::{BatchRegionResponse, RegionResponse};
 use api::v1::ResponseHeader;
+use api::v1::region::RegionResponse as PbRegionResponse;
 use api::v1::region::{
     RegionRequest, RegionRequestHeader, RemoteDynFilterRequest, RemoteDynFilterUnregister,
     RemoteDynFilterUpdate, region_request, remote_dyn_filter_request,
@@ -49,7 +50,7 @@ use tonic::codec::CompressionEncoding;
 
 use crate::error::{
     self, FlightGetSnafu, IllegalDatabaseResponseSnafu, IllegalFlightMessagesSnafu,
-    MissingFieldSnafu, Result, ServerSnafu,
+    MissingFieldSnafu, Result,
 };
 use crate::flight::{FlightMessageReader, decode_flight_data};
 use crate::region::codec::RegionCodec;
@@ -141,7 +142,8 @@ impl RegionRequester {
     ///
     /// Accepts both [`RegionRequest`] and [`Arc<RegionRequest>`]. Callers can retain a
     /// shared request for another attempt; this method does not retry or determine
-    /// whether replaying the request is safe.
+    /// whether replaying the request is safe. Callers accepting partial results in the
+    /// request header must inspect failed entries before retrying.
     pub async fn handle<R>(&self, request: R) -> Result<RegionResponse>
     where
         R: Borrow<RegionRequest> + Send + Sync + 'static,
@@ -180,6 +182,11 @@ impl RegionRequester {
             .body
             .as_ref()
             .context(MissingFieldSnafu { field: "body" })?;
+        let request_count = match request_body {
+            region_request::Body::Inserts(requests) => Some(requests.requests.len()),
+            region_request::Body::Deletes(requests) => Some(requests.requests.len()),
+            _ => None,
+        };
         let is_insert = matches!(
             request_body,
             region_request::Body::Inserts(_) | region_request::Body::BulkInsert(_)
@@ -199,6 +206,13 @@ impl RegionRequester {
             }
         }
 
+        let accept_partial_result = request
+            .borrow()
+            .header
+            .as_ref()
+            .and_then(|header| header.query_context.as_ref())
+            .and_then(|context| context.extensions.get(api::region::ACCEPT_PARTIAL_RESULT))
+            .is_some_and(|value| value == "true");
         let response = async {
             client
                 .ready()
@@ -223,9 +237,10 @@ impl RegionRequester {
         .map_err(|status| map_region_error(status, addr))?
         .into_inner();
 
-        check_response_header(&response.header)?;
-
-        Ok(RegionResponse::from_region_response(response))
+        match request_count {
+            Some(count) if accept_partial_result => decode_write_response(response, count),
+            _ => decode_region_response(response),
+        }
     }
 }
 
@@ -385,6 +400,7 @@ fn build_remote_dyn_filter_request(
     }
 }
 
+/// Checks that the response contains a successful application status.
 pub fn check_response_header(header: &Option<ResponseHeader>) -> Result<()> {
     let status = header
         .as_ref()
@@ -396,35 +412,107 @@ pub fn check_response_header(header: &Option<ResponseHeader>) -> Result<()> {
     if StatusCode::is_success(status.status_code) {
         Ok(())
     } else {
-        let code =
-            StatusCode::from_u32(status.status_code).context(IllegalDatabaseResponseSnafu {
-                err_msg: format!("unknown server status: {:?}", status),
-            })?;
-        ServerSnafu {
-            code,
-            msg: status.err_msg.clone(),
-        }
-        .fail()
+        Err(error::Error::try_from_response_status(status)?)
     }
 }
 
+/// Decodes an ordinary region response without per-entry write failures.
+pub fn decode_region_response(response: PbRegionResponse) -> Result<RegionResponse> {
+    if !response.failures.is_empty() {
+        return IllegalDatabaseResponseSnafu {
+            err_msg: "Failure details are only valid for inserts/deletes",
+        }
+        .fail();
+    }
+    check_response_header(&response.header)?;
+    Ok(RegionResponse::from_region_response(response))
+}
+
+/// Decodes an inserts/deletes response, validating failures against the original entry count.
+/// A failed single-entry request returns its underlying error.
+fn decode_write_response(response: PbRegionResponse, entry_count: usize) -> Result<RegionResponse> {
+    if response.failures.is_empty() {
+        if response
+            .header
+            .as_ref()
+            .and_then(|header| header.status.as_ref())
+            .is_some_and(|status| status.status_code == StatusCode::PartialFailure as u32)
+        {
+            return IllegalDatabaseResponseSnafu {
+                err_msg: "Partial failure has no failed entries",
+            }
+            .fail();
+        }
+        return decode_region_response(response);
+    }
+    let status = response
+        .header
+        .as_ref()
+        .and_then(|header| header.status.as_ref())
+        .context(IllegalDatabaseResponseSnafu {
+            err_msg: "Missing region response status",
+        })?;
+    if status.status_code != StatusCode::PartialFailure as u32 {
+        return IllegalDatabaseResponseSnafu {
+            err_msg: "Failure details require a partial-failure status",
+        }
+        .fail();
+    }
+    let mut seen = std::collections::HashSet::with_capacity(response.failures.len());
+    let mut failures = Vec::with_capacity(response.failures.len());
+    for error in response.failures {
+        let index = error.request_index as usize;
+        if index >= entry_count || !seen.insert(index) {
+            return IllegalDatabaseResponseSnafu {
+                err_msg: "Invalid or duplicate batch error index",
+            }
+            .fail();
+        }
+        let status = error.status.context(IllegalDatabaseResponseSnafu {
+            err_msg: "Missing batch error status",
+        })?;
+        if StatusCode::is_success(status.status_code) {
+            return IllegalDatabaseResponseSnafu {
+                err_msg: "Batch error contains a success status",
+            }
+            .fail();
+        }
+        failures.push((index, error::Error::try_from_response_status(&status)?));
+    }
+    // A single failed entry has no successful peers to preserve.
+    if entry_count == 1
+        && let Some((_, error)) = failures.pop()
+    {
+        return Err(error);
+    }
+    Err(error::Error::PartialRegion {
+        response: Box::new(BatchRegionResponse {
+            affected_rows: response.affected_rows as _,
+            extensions: response.extensions,
+            failures,
+        }),
+    })
+}
+
 #[cfg(test)]
 #[allow(deprecated)]
 mod tests {
     use std::collections::VecDeque;
     use std::sync::Mutex;
 
+    use api::v1::ResponseHeader;
     use api::v1::region::region_server::{Region, RegionServer};
     use api::v1::region::{
-        BulkInsertRequest, InsertRequest, InsertRequests, RegionResponse as PbRegionResponse,
-        RemoteDynFilterUnregister, RemoteDynFilterUpdate, region_request,
-        remote_dyn_filter_request,
+        BulkInsertRequest, InsertRequest, InsertRequests, RegionRequestFailure,
+        RegionResponse as PbRegionResponse, RemoteDynFilterUnregister, RemoteDynFilterUpdate,
+        region_request, remote_dyn_filter_request,
     };
     use api::v1::value::ValueData;
     use api::v1::{
         ColumnDataType, ColumnSchema as PbColumnSchema, Row, Rows, Status as PbStatus, Value,
     };
     use common_error::ext::RetryHint;
+    use common_error::status_code::StatusCode;
     use common_grpc::channel_manager::{ChannelConfig, ChannelManager};
     use common_recordbatch::adapter::RecordBatchMetrics;
     use datatypes::arrow::array::Int32Array;
@@ -538,6 +626,44 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_cancel_compaction_through_handle() {
+        let server = RegionTestServer::start().await;
+        let requester = server.create_requester(ChannelConfig::default(), false, false);
+        let request = RegionRequest {
+            header: Some(RegionRequestHeader::default()),
+            body: Some(region_request::Body::CancelCompaction(
+                api::v1::region::CancelCompactionRequest {
+                    region_id: 42,
+                    task_id: "cycle-id".into(),
+                },
+            )),
+        };
+        for affected_rows in [0, 1] {
+            let mut response = build_response(StatusCode::Success);
+            response.affected_rows = affected_rows;
+            server
+                .service
+                .responses
+                .lock()
+                .unwrap()
+                .push_back(Ok(response));
+            assert_eq!(
+                requester
+                    .handle(request.clone())
+                    .await
+                    .unwrap()
+                    .affected_rows,
+                affected_rows as usize
+            );
+        }
+        let requests = server.service.requests.lock().unwrap();
+        assert_eq!(requests.len(), 2);
+        for (actual, _, _) in requests.iter() {
+            assert_eq!(actual.body, request.body);
+        }
+    }
+
     fn build_insert_request() -> RegionRequest {
         RegionRequest {
             header: Some(RegionRequestHeader {
@@ -579,6 +705,20 @@ mod tests {
         }
     }
 
+    fn with_partial_result(mut request: RegionRequest) -> RegionRequest {
+        request
+            .header
+            .get_or_insert_with(Default::default)
+            .query_context
+            .get_or_insert_with(Default::default)
+            .extensions
+            .insert(
+                api::region::ACCEPT_PARTIAL_RESULT.to_string(),
+                "true".to_string(),
+            );
+        request
+    }
+
     #[tokio::test]
     async fn test_owned_and_shared_requests_preserve_payload_and_compression() {
         let server = RegionTestServer::start().await;
@@ -636,6 +776,54 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_partial_result_requires_explicit_acceptance() {
+        let server = RegionTestServer::start().await;
+        let requester = server.create_requester(ChannelConfig::default(), false, false);
+        for accepted in [false, true] {
+            let request = if accepted {
+                with_partial_result(build_insert_request())
+            } else {
+                build_insert_request()
+            };
+            let shared = Arc::new(request);
+            server
+                .service
+                .responses
+                .lock()
+                .unwrap()
+                .push_back(Ok(PbRegionResponse {
+                    header: Some(ResponseHeader {
+                        status: Some(PbStatus {
+                            status_code: StatusCode::PartialFailure as u32,
+                            ..Default::default()
+                        }),
+                    }),
+                    failures: vec![RegionRequestFailure {
+                        request_index: 0,
+                        status: Some(PbStatus {
+                            status_code: StatusCode::RegionNotFound as u32,
+                            retry_hint: RetryHint::RetryAfterRouteRefresh.as_str().to_string(),
+                            ..Default::default()
+                        }),
+                    }],
+                    ..Default::default()
+                }));
+            let error = requester.handle(Arc::clone(&shared)).await.unwrap_err();
+            if accepted {
+                assert_eq!(error.status_code(), StatusCode::RegionNotFound);
+                assert_eq!(error.retry_hint(), RetryHint::RetryAfterRouteRefresh);
+            } else {
+                assert!(matches!(error, Error::IllegalDatabaseResponse { .. }));
+            }
+            assert_eq!(
+                &server.service.requests.lock().unwrap().last().unwrap().0,
+                shared.as_ref()
+            );
+            assert_eq!(Arc::strong_count(&shared), 1);
+        }
+    }
+
     #[tokio::test]
     async fn test_shared_request_can_be_resent_after_transport_error() {
         let server = RegionTestServer::start().await;
@@ -762,7 +950,10 @@ mod tests {
                 assert_eq!(error.tonic_code(), baseline.tonic_code());
             }
             let datanode: &dyn Datanode = &requester;
-            let error = datanode.handle(build_insert_request()).await.unwrap_err();
+            let error = datanode
+                .handle_request(Arc::new(build_insert_request()))
+                .await
+                .unwrap_err();
             assert_eq!(error.status_code(), StatusCode::RegionNotFound);
             assert_eq!(error.retry_hint(), hint);
             assert!(matches!(error, meta_error::Error::External { .. }));
@@ -832,10 +1023,16 @@ mod tests {
                     body: Some(body),
                 };
                 let shared = Arc::new(request);
-                let shared_error = datanode.handle(Arc::clone(&shared)).await.unwrap_err();
+                let shared_error = datanode
+                    .handle_request(Arc::clone(&shared))
+                    .await
+                    .unwrap_err();
                 assert_eq!(Arc::strong_count(&shared), 1);
                 let request = Arc::try_unwrap(shared).unwrap();
-                let owned_error = datanode.handle(request).await.unwrap_err();
+                let owned_error = datanode
+                    .handle_request(Arc::new(request))
+                    .await
+                    .unwrap_err();
                 for error in [owned_error, shared_error] {
                     assert_eq!(
                         error.retry_hint(),
@@ -1053,8 +1250,281 @@ mod tests {
         .unwrap()
     }
 
+    #[tokio::test]
+    async fn test_partial_response_validation_and_compression() {
+        let server = RegionTestServer::start().await;
+        let requester = server.create_requester(ChannelConfig::default(), true, true);
+        let mut request = build_insert_request();
+        if let Some(region_request::Body::Inserts(inserts)) = &mut request.body {
+            inserts.requests.push(inserts.requests[0].clone());
+        }
+        let request = Arc::new(with_partial_result(request));
+        let failure = RegionRequestFailure {
+            request_index: 0,
+            status: Some(PbStatus {
+                status_code: StatusCode::RegionNotFound as u32,
+                err_msg: "moved".to_string(),
+                retry_hint: "retry_after_route_refresh".to_string(),
+            }),
+        };
+        for (failures, valid) in [
+            (vec![], true),
+            (vec![failure.clone()], true),
+            (vec![failure.clone(), failure.clone()], false),
+            (
+                vec![RegionRequestFailure {
+                    request_index: u32::MAX,
+                    ..failure.clone()
+                }],
+                false,
+            ),
+            (
+                vec![RegionRequestFailure {
+                    request_index: 0,
+                    status: None,
+                }],
+                false,
+            ),
+            (
+                vec![RegionRequestFailure {
+                    request_index: 0,
+                    status: Some(PbStatus::default()),
+                }],
+                false,
+            ),
+            (
+                vec![RegionRequestFailure {
+                    request_index: 0,
+                    status: Some(PbStatus {
+                        status_code: u32::MAX,
+                        ..Default::default()
+                    }),
+                }],
+                false,
+            ),
+        ] {
+            let has_error = !failures.is_empty();
+            server
+                .service
+                .responses
+                .lock()
+                .unwrap()
+                .push_back(Ok(PbRegionResponse {
+                    header: Some(ResponseHeader {
+                        status: Some(PbStatus {
+                            status_code: if has_error {
+                                StatusCode::PartialFailure
+                            } else {
+                                StatusCode::Success
+                            } as u32,
+                            ..Default::default()
+                        }),
+                    }),
+                    affected_rows: 3,
+                    extensions: std::collections::HashMap::from([("test".to_string(), vec![1, 2])]),
+                    failures,
+                    ..Default::default()
+                }));
+            let response = requester.handle_inner(Arc::clone(&request)).await;
+            if valid {
+                if has_error {
+                    let error = response.unwrap_err();
+                    assert!(error.output_msg().contains("entry 0"));
+                    assert!(error.output_msg().contains("moved"));
+                    let error: common_meta::error::Error = error.into();
+                    assert!(error.output_msg().contains("moved"));
+                    let common_meta::error::Error::PartialRegion { response } = error else {
+                        panic!("Expected partial region failure");
+                    };
+                    assert_eq!(response.affected_rows, 3);
+                    assert_eq!(response.extensions["test"], vec![1, 2]);
+                    assert_eq!(
+                        response.failures[0].1.retry_hint(),
+                        RetryHint::RetryAfterRouteRefresh
+                    );
+                } else {
+                    let response = response.unwrap();
+                    assert_eq!(response.affected_rows, 3);
+                    assert_eq!(response.extensions["test"], vec![1, 2]);
+                }
+            } else {
+                assert!(matches!(
+                    response.unwrap_err(),
+                    Error::IllegalDatabaseResponse { .. }
+                ));
+            }
+        }
+        for (received, send, accept) in server.service.requests.lock().unwrap().iter() {
+            assert_eq!(received, request.as_ref());
+            assert!(*send && *accept);
+        }
+        assert_eq!(Arc::strong_count(&request), 1);
+    }
+
+    #[test]
+    fn test_partial_response_envelope() {
+        let failure = RegionRequestFailure {
+            request_index: 0,
+            status: Some(PbStatus {
+                status_code: StatusCode::RegionNotFound as u32,
+                retry_hint: RetryHint::RetryAfterRouteRefresh.as_str().to_string(),
+                ..Default::default()
+            }),
+        };
+        for (status, failures, count) in [
+            (Some(StatusCode::Success), vec![failure.clone()], Some(2)),
+            (Some(StatusCode::PartialFailure), vec![], Some(2)),
+            (
+                Some(StatusCode::PartialFailure),
+                vec![failure.clone()],
+                None,
+            ),
+            (None, vec![failure.clone()], Some(2)),
+        ] {
+            let response = PbRegionResponse {
+                header: status.map(|code| ResponseHeader {
+                    status: Some(PbStatus {
+                        status_code: code as u32,
+                        ..Default::default()
+                    }),
+                }),
+                failures,
+                ..Default::default()
+            };
+            assert!(matches!(
+                match count {
+                    Some(count) => decode_write_response(response, count),
+                    None => decode_region_response(response),
+                },
+                Err(Error::IllegalDatabaseResponse { .. })
+            ));
+        }
+        let response = PbRegionResponse {
+            header: Some(ResponseHeader {
+                status: Some(PbStatus {
+                    status_code: StatusCode::PartialFailure as u32,
+                    ..Default::default()
+                }),
+            }),
+            failures: vec![failure],
+            ..Default::default()
+        };
+        // A partial envelope must not be interpreted as success by the header-only decoder.
+        assert!(check_response_header(&response.header).is_err());
+        let error = decode_write_response(response, 1).unwrap_err();
+        assert_eq!(error.status_code(), StatusCode::RegionNotFound);
+        assert_eq!(error.retry_hint(), RetryHint::RetryAfterRouteRefresh);
+    }
+
+    #[test]
+    fn test_partial_error_preserves_status_and_retry_hint() {
+        let expected_message = "Region request has 2 failed entries; entry 0: first input failed";
+        for code in [StatusCode::StorageUnavailable, StatusCode::InvalidArguments] {
+            let response = PbRegionResponse {
+                header: Some(ResponseHeader {
+                    status: Some(PbStatus {
+                        status_code: StatusCode::PartialFailure as u32,
+                        ..Default::default()
+                    }),
+                }),
+                failures: vec![
+                    RegionRequestFailure {
+                        request_index: 1,
+                        status: Some(PbStatus {
+                            status_code: StatusCode::Internal as u32,
+                            err_msg: "later input failed".to_string(),
+                            ..Default::default()
+                        }),
+                    },
+                    RegionRequestFailure {
+                        request_index: 0,
+                        status: Some(PbStatus {
+                            status_code: code as u32,
+                            err_msg: "first input failed".to_string(),
+                            retry_hint: RetryHint::NonRetryable.as_str().to_string(),
+                        }),
+                    },
+                ],
+                ..Default::default()
+            };
+            let error = decode_write_response(response, 2).unwrap_err();
+            assert_eq!(code, error.status_code());
+            assert_eq!(expected_message, error.to_string());
+            assert_eq!(RetryHint::Retryable, error.retry_hint());
+            let error: common_meta::error::Error = error.into();
+            assert_eq!(code, error.status_code());
+            assert_eq!(expected_message, error.to_string());
+            assert_eq!(RetryHint::Retryable, error.retry_hint());
+            let common_meta::error::Error::PartialRegion { response } = error else {
+                panic!("expected partial result")
+            };
+            assert!(
+                response
+                    .failures
+                    .iter()
+                    .all(|(_, error)| error.retry_hint() == RetryHint::NonRetryable)
+            );
+        }
+    }
+
+    #[tokio::test]
+    async fn test_region_rpc_errors() {
+        let server = RegionTestServer::start().await;
+        let requester = server.create_requester(ChannelConfig::default(), false, false);
+        let request = Arc::new(build_insert_request());
+        let mut business_error = Status::unimplemented("unsupported write");
+        business_error.metadata_mut().insert(
+            common_error::GREPTIME_DB_HEADER_ERROR_CODE,
+            (StatusCode::Unsupported as u32)
+                .to_string()
+                .parse()
+                .unwrap(),
+        );
+        for status in [
+            Status::unimplemented("unknown method"),
+            business_error,
+            Status::unavailable("lost response"),
+            Status::deadline_exceeded("timeout"),
+        ] {
+            let code = status.code();
+            server
+                .service
+                .responses
+                .lock()
+                .unwrap()
+                .push_back(Err(status));
+            let error = requester
+                .handle_inner(Arc::clone(&request))
+                .await
+                .unwrap_err();
+            assert!(matches!(error, Error::RegionServer { code: actual, .. } if actual == code));
+        }
+        assert_eq!(server.service.requests.lock().unwrap().len(), 4);
+    }
+
     #[test]
     fn test_check_response_header() {
+        for (hint, expected) in [
+            ("retryable", RetryHint::Retryable),
+            (
+                "retry_after_route_refresh",
+                RetryHint::RetryAfterRouteRefresh,
+            ),
+            ("non_retryable", RetryHint::NonRetryable),
+            ("", RetryHint::NonRetryable),
+            ("future_hint", RetryHint::NonRetryable),
+        ] {
+            let error = check_response_header(&Some(ResponseHeader {
+                status: Some(PbStatus {
+                    status_code: StatusCode::RegionNotFound as u32,
+                    err_msg: "region moved".to_string(),
+                    retry_hint: hint.to_string(),
+                }),
+            }))
+            .unwrap_err();
+            assert_eq!(error.retry_hint(), expected, "{hint}");
+        }
+
         let result = check_response_header(&None);
         assert!(matches!(
             result.unwrap_err(),
@@ -1071,6 +1541,7 @@ mod tests {
             status: Some(PbStatus {
                 status_code: StatusCode::Success as u32,
                 err_msg: String::default(),
+                ..Default::default()
             }),
         }));
         assert!(result.is_ok());
@@ -1079,6 +1550,7 @@ mod tests {
             status: Some(PbStatus {
                 status_code: u32::MAX,
                 err_msg: String::default(),
+                ..Default::default()
             }),
         }));
         assert!(matches!(
@@ -1090,6 +1562,7 @@ mod tests {
             status: Some(PbStatus {
                 status_code: StatusCode::Internal as u32,
                 err_msg: "blabla".to_string(),
+                ..Default::default()
             }),
         }));
         let Error::Server { code, msg, .. } = result.unwrap_err() else {

src/cmd/src/bin/query_perf_fixture/direct_sst.rs

@@ -470,6 +470,7 @@ pub(super) async fn run_direct_sst(args: DirectArgs) {
             row_group_size: scenario.layout.row_group_size,
             max_file_size: None,
             float_field_encoding: Default::default(),
+            ..Default::default()
         };
         let infos = match format {
             FormatType::Flat => writer.write_all_flat(source, Some(sequence), &opts).await,
@@ -511,6 +512,8 @@ pub(super) async fn run_direct_sst(args: DirectArgs) {
         compaction_time_window: None,
         sst_format: format,
         append_mode: table.append_mode,
+        pending_tails: Default::default(),
+        handoff_floors: Default::default(),
     };
     let checkpoint = RegionCheckpoint {
         last_version: args.checkpoint_version,

src/cmd/src/datanode/objbench.rs

@@ -27,7 +27,7 @@ use mito2::config::{FulltextIndexConfig, MitoConfig, Mode};
 use mito2::read::FlatSource;
 use mito2::sst::FormatType;
 use mito2::sst::file::{FileHandle, FileMeta};
-use mito2::sst::file_purger::{FilePurger, FilePurgerRef};
+use mito2::sst::file_purger::{FilePurgerRef, NoopFilePurger};
 use mito2::sst::index::intermediate::IntermediateManager;
 use mito2::sst::index::puffin_manager::PuffinManagerFactory;
 use mito2::sst::parquet::WriteOptions;
@@ -203,6 +203,7 @@ impl ObjbenchCommand {
             max_sequence: None,
             sst_write_format: FormatType::PrimaryKey,
             index_options: Default::default(),
+            index_build_inputs: None,
             index_config: mito_engine_config.index.clone(),
             inverted_index_config: MitoConfig::default().inverted_index,
             fulltext_index_config,
@@ -508,12 +509,7 @@ async fn build_cache_manager(
 }
 
 fn new_noop_file_purger() -> FilePurgerRef {
-    #[derive(Debug)]
-    struct Noop;
-    impl FilePurger for Noop {
-        fn remove_file(&self, _file_meta: FileMeta, _is_delete: bool, _index_outdated: bool) {}
-    }
-    Arc::new(Noop)
+    Arc::new(NoopFilePurger)
 }
 
 async fn load_parquet_metadata(

src/cmd/src/datanode/parquetbench.rs

@@ -713,7 +713,6 @@ async fn run_direct_iteration(
             file_path.clone(),
             object_store.clone(),
             CacheStrategy::Disabled,
-            row_group_idx,
             None,
         );
         let mut stream = build_sst_parquet_record_batch_stream(

src/cmd/src/datanode/scanbench.rs

@@ -1016,6 +1016,7 @@ impl ScanbenchCommand {
             let num_partitions = scanner.properties().partitions.len();
             let ctx = QueryScanContext {
                 explain_verbose: collect_scanner_explain,
+                ..Default::default()
             };
             let metrics_set = ExecutionPlanMetricsSet::new();
 

src/cmd/src/datanode/sst_replace.rs

@@ -208,6 +208,7 @@ impl SstReplaceCommand {
             flushed_entry_id: None,
             flushed_sequence: None,
             committed_sequence: None,
+            wal_recovery: Default::default(),
         };
         let version = manifest_manager
             .update(

src/cmd/src/error.rs

@@ -85,6 +85,13 @@ pub enum Error {
         source: datanode::error::Error,
     },
 
+    #[snafu(display("Failed to close the metadata kv backend"))]
+    CloseMetadataKvBackend {
+        #[snafu(implicit)]
+        location: Location,
+        source: common_meta::error::Error,
+    },
+
     #[snafu(display("Failed to start flownode"))]
     StartFlownode {
         #[snafu(implicit)]
@@ -333,6 +340,7 @@ impl ErrorExt for Error {
             Error::StartDatanode { source, .. } => source.status_code(),
             Error::StartFrontend { source, .. } => source.status_code(),
             Error::ShutdownDatanode { source, .. } => source.status_code(),
+            Error::CloseMetadataKvBackend { source, .. } => source.status_code(),
             Error::ShutdownFrontend { source, .. } => source.status_code(),
             Error::StartMetaServer { source, .. } => source.status_code(),
             Error::ShutdownMetaServer { source, .. } => source.status_code(),