rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-25 21:32:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: merge latest main

Browse Source 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
This commit is contained in:
jeremyhi
2026-08-31 13:01:27 +08:00
parent 5af1aa0104 d32cd77505
commit 865a29584b
622 changed files with 71033 additions and 11579 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/README.md
+10 -3
View File
@@ -11,16 +11,23 @@ Shared agent skills live in `.agents/skills`.
- Add or update shared skills under `.agents/skills/<skill-name>/SKILL.md`.
- If a new skill does not appear, restart the agent or start a new thread.
## Per-crate guides
## Per-directory guides
Hot crates carry an `AGENTS.md` next to their code as a navigation aid (module
map, read/write paths, change-coupling points, test commands, gotchas):
Read every `AGENTS.md` from the repository root down to the path being changed.
The root guide applies everywhere; nested guides add rules and navigation for
their subtree.
- [`src/common/meta/AGENTS.md`](../src/common/meta/AGENTS.md) — metadata keys, KV backends, DDL procedures, and caches
- [`src/query/AGENTS.md`](../src/query/AGENTS.md) — query planning, optimization, and distributed execution
- [`src/servers/AGENTS.md`](../src/servers/AGENTS.md) — wire protocols and network servers
- [`src/operator/AGENTS.md`](../src/operator/AGENTS.md) — statement, DDL/DML, and write orchestration
- [`src/mito2/AGENTS.md`](../src/mito2/AGENTS.md) — primary time-series storage engine
- [`src/metric-engine/AGENTS.md`](../src/metric-engine/AGENTS.md) — metrics engine (logical/physical regions)
- [`src/flow/AGENTS.md`](../src/flow/AGENTS.md) — stream processing / continuous aggregation
- [`src/frontend/AGENTS.md`](../src/frontend/AGENTS.md) — request entry point and orchestration
- [`src/meta-srv/AGENTS.md`](../src/meta-srv/AGENTS.md) — metadata and cluster coordination
- [`tests/compatibility/AGENTS.md`](../tests/compatibility/AGENTS.md) — persisted/wire compatibility cases
- [`tests/perf/AGENTS.md`](../tests/perf/AGENTS.md) — query regression harness and case DSL
## Architecture invariants
.agents/architecture-invariants.md
+25 -26
View File
@@ -20,15 +20,18 @@ metadata KV values (`common-meta` keys, metric-engine metadata), and gRPC
messages. A node running an old version may read data written by a new one and
vice versa.
- Add fields, don't repurpose or reorder them. For serde types use
`#[serde(default)]` / `#[serde(alias)]`; never change the meaning of an existing
field or the discriminant of an existing enum variant.
- Monotonic version counters (e.g. the mito2 manifest version) only ever move
forward — never reset or skip.
- Preserve the contract of the actual encoding: Protobuf field numbers/types,
serde field and variant names, positional field order, and explicitly encoded
enum discriminants. Do not treat Rust declaration order as a wire contract
for name-based formats such as JSON.
- Use `#[serde(default)]` for newly optional data and `#[serde(alias)]` for a
compatible rename. Do not reset or reuse versions within one persisted
history.
- When a change touches a persisted or wire format, add a case to the
compatibility test suite. See `docs/rfcs/2025-07-04-compatibility-test-framework.md`
— compatibility has been broken on releases before (v0.14.1, v0.15.1) precisely
because this step was skipped.
compatibility test suite. Follow [`tests/compatibility/README.md`](../tests/compatibility/README.md)
and [`tests/compatibility/AGENTS.md`](../tests/compatibility/AGENTS.md). The
original compatibility RFC describes a superseded case format and is not the
current runbook.
- Wire types are generated from the external `greptime-proto` crate; change the
format there first, then bump the dependency (see invariant 6's pattern).
@@ -43,8 +46,8 @@ The workspace is layered; dependencies point downward only.
`file-engine`) implement it; `datanode` drives engines **through the trait**,
not through engine internals.
- `frontend` reaches storage through `operator` / `query` / `catalog`, not by
depending on `datanode` internals. Standalone mode is the one bridge, via a
`RegionServer` wrapper (`src/frontend/src/instance/standalone.rs`).
depending on `datanode` internals. Standalone mode is the one bridge, via the
`RegionServer` adapter in `src/standalone/src/datanode_manager.rs`.
- Do not introduce circular dependencies. New deps go through
`[workspace.dependencies]` in the root `Cargo.toml`, not per-crate version
literals.
@@ -54,9 +57,9 @@ The workspace is layered; dependencies point downward only.
Runtimes are partitioned by workload so one workload can't starve another. They
live in `common-runtime` (`src/common/runtime/`).
- Use the categorized spawns — `spawn_global`, `spawn_query`, `spawn_ingest`,
`spawn_compact`, `spawn_hb` — instead of constructing your own tokio runtime,
and pick the category that matches the work.
- Runtime construction belongs in process bootstrap or test/benchmark harnesses.
Product components use `spawn_global`, `spawn_query`, `spawn_ingest`,
`spawn_compact`, or `spawn_hb` instead of a component-local Tokio runtime.
- Run CPU-bound or synchronous-blocking work via `spawn_blocking_*`; never do
heavy CPU or blocking syscalls directly inside an async task.
- Never call `block_on*` from inside an async context or an engine worker — it
@@ -85,23 +88,19 @@ unfinished work merge without freezing it into the stable config surface.
When you stabilize such a feature, drop the prefix and document the migration.
## 6. DataFusion is a pinned fork — two sections, two forms
## 6. DataFusion is a pinned fork — workspace dependencies plus patches
GreptimeDB uses a fork at `GreptimeTeam/datafusion`, wired up in the root
`Cargo.toml` through **two sections that hold different things**:
`Cargo.toml` through two sections:
- `[workspace.dependencies]` pins each DataFusion sub-crate to an **exact
crates.io version** (e.g. `datafusion = "=53.1.0"`).
- `[patch.crates-io]` redirects those same crates to the **fork at a git rev**
(e.g. `datafusion = { git = ".../GreptimeTeam/datafusion.git", rev = "..." }`).
- `[workspace.dependencies]` pins directly referenced sub-crates to an exact
crates.io version.
- `[patch.crates-io]` redirects resolution to the fork. It may also contain
transitive-only sub-crates with no workspace dependency.
So:
- Adding a new DataFusion sub-crate dependency means adding it to **both**
sections — the `=<version>` entry under `[workspace.dependencies]` and the
matching git-rev patch under `[patch.crates-io]`.
- Upgrading DataFusion means bumping the version in `[workspace.dependencies]`
**and** the rev in `[patch.crates-io]` together, for all of them.
Add a direct dependency to `[workspace.dependencies]` and patch it if needed; a
transitive-only override needs only the patch. Upgrade all exact pins and fork
revisions together.
## 7. Enterprise-gated code is licensed differently from the rest
.agents/generated-files.md
+11 -3
View File
@@ -35,7 +35,14 @@ Edit the example TOMLs and/or the template, then regenerate. Do not edit
## Grafana dashboards
Generated artifacts; regenerate rather than editing by hand.
For the metrics dashboards, edit
`grafana/dashboards/metrics/cluster/dashboard.json`. The generator derives the
standalone `dashboard.json` by removing instance filters, then produces
`dashboard.yaml` and `dashboard.md` for both cluster and standalone variants.
Do not hand-edit those five derived files.
`grafana/dashboards/events/dashboard.json` and
`grafana/dashboards/logs/dashboard.json` are not outputs of this generator.
```bash
make dashboards
@@ -56,5 +63,6 @@ dependency.
## License headers
Managed by the license-header tooling and the pre-commit hooks; let the tooling
add/fix them rather than editing headers by hand.
License headers are checked by `korandoru/hawkeye@v5` in
`.github/workflows/checks.yml`. For enterprise-gated files, also update both
license lists and run `make check-enterprise-license`.
.agents/skills/greptimedb-release-note/SKILL.md
+4 -3
View File
@@ -167,9 +167,10 @@ The release note is also published as a blog post in **`GreptimeTeam/docs`**.
git -C <docs> fetch origin main
git -C <docs> worktree add -b chore/X.Y.Z-release-note /tmp/docs-release-note origin/main
```
- **The PR body must follow the docs repo's template** (`.github/pull_request_template.md` —
"What's Changed in this PR" + a Checklist). Fill in the description; leave checklist boxes
for the reviewer.
- **Read and follow the current docs repo PR template**
(`.github/pull_request_template.md`) from the checkout used for the release.
Fill in every required section and leave reviewer-owned checklist boxes unchecked;
do not rely on hard-coded section names from an older template.
- Commit with sign-off, push, open a **draft** PR, then remove the worktree:
```
git -C /tmp/docs-release-note add blog/release-X-Y-Z.md
.cargo/config.toml
+3
View File
@@ -1,6 +1,9 @@
[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
[target.riscv64gc-unknown-linux-gnu]
linker = "riscv64-linux-gnu-gcc"
[alias]
sqlness = "run --bin sqlness-runner --target-dir target/sqlness --"
.github/CODEOWNERS
+5 -5
View File
@@ -6,8 +6,8 @@
## [Module] Database Engine
/src/index @evenyag @discord9 @WenyXu
/src/mito2 @evenyag @v0y4g3r @waynexia
/src/query @evenyag @waynexia @discord9
/src/mito2 @evenyag @v0y4g3r
/src/query @evenyag @discord9
## [Module] Distributed
/src/common/meta @MichaelScofield @WenyXu
@@ -20,8 +20,8 @@
/src/store-api @v0y4g3r @evenyag
## [Module] Metrics Engine
/src/metric-engine @waynexia @WenyXu
/src/promql @waynexia @evenyag @discord9
/src/metric-engine @WenyXu @v0y4g3r
/src/promql @evenyag @discord9
## [Module] Flow
/src/flow @discord9 @waynexia
/src/flow @discord9 @WenyXu
.github/actions/build-dev-builder-images/action.yml
+19
View File
@@ -31,6 +31,10 @@ inputs:
description: Build dev-builder-android image
required: false
default: "true"
build-dev-builder-riscv64:
description: Build dev-builder-riscv64 image
required: false
default: "true"
runs:
using: composite
steps:
@@ -70,6 +74,21 @@ runs:
IMAGE_NAMESPACE=${{ inputs.dockerhub-image-namespace }} \
DEV_BUILDER_IMAGE_TAG=${{ inputs.version }}
- name: Build and push dev-builder-riscv64 image # Only build image for amd64 platform (it cross-compiles riscv64 binaries).
shell: bash
if: ${{ inputs.build-dev-builder-riscv64 == 'true' }}
run: |
make dev-builder-riscv64 \
IMAGE_REGISTRY=${{ inputs.dockerhub-image-registry }} \
IMAGE_NAMESPACE=${{ inputs.dockerhub-image-namespace }} \
DEV_BUILDER_RISCV64_IMAGE_TAG=${{ inputs.version }}
docker push ${{ inputs.dockerhub-image-registry }}/${{ inputs.dockerhub-image-namespace }}/dev-builder-riscv64:${{ inputs.version }}
if [[ "${{ github.event_name }}" == "push" ]]; then
docker tag ${{ inputs.dockerhub-image-registry }}/${{ inputs.dockerhub-image-namespace }}/dev-builder-riscv64:${{ inputs.version }} \
${{ inputs.dockerhub-image-registry }}/${{ inputs.dockerhub-image-namespace }}/dev-builder-riscv64:latest
docker push ${{ inputs.dockerhub-image-registry }}/${{ inputs.dockerhub-image-namespace }}/dev-builder-riscv64:latest
fi
- name: Build and push dev-builder-android image # Only build image for amd64 platform.
shell: bash
if: ${{ inputs.build-dev-builder-android == 'true' }}
.github/runner-scale-sets/query-regression/README.md
+158 -295
View File
@@ -1,321 +1,184 @@
# Query regression self-hosted runners
The `Query Regression` workflow uses the dedicated ARC runner scale set
`perf-regression-8-cores`. ARC runner Pods run in the target Kubernetes cluster
and connect outbound to GitHub. The live scale set is currently **paused**:
`minRunners=0`, `maxRunners=0`, and no runner Pods. Do not resume it without
explicit approval.
The `Query Regression` workflow runs on **Aliyun ECS ephemeral runners**
(`aliyun-ecs`, the default and only automated path): a `provision` job on
`ubuntu-latest` creates one pay-as-you-go ECS instance per run, and the
instance registers itself as an ephemeral GitHub runner with a per-run
label. A `teardown` job (`if: always()`) deletes the instance;
`query-regression-janitor.yml` sweeps tagged leftovers older than 4 hours
daily.
## Prerequisites and trust admission
Build caches live on the instance's system disk, so **every run compiles
cold**; only the within-run reuse (base build warms the candidate build
through the shared target dir and sccache) applies. There is no retained
data disk.
Install the ARC scale set controller if it is not already installed:
Dispatching with any other `runner` value uses it as a literal self-hosted
runner label, which is how a manually prepared host (see
`ecs-image/bootstrap-runner-host.sh`) runs the workflow. For PR labels, set
the repository variable `QUERY_REGRESSION_PR_RUNNER` to such a label to
redirect PR runs away from ECS.
The office ARC scale set `perf-regression-8-cores` that previously ran this
workflow is retired; see git history for its values files and pause/deploy
procedures. Tearing down the office cluster (helm release, runner namespace,
and the `query-regression-build-cache` PVC) is a manual operator action
outside this repository. This directory keeps its historical
`runner-scale-sets` name for path stability.
## Nightly vs previous nightly
`query-regression-nightly.yml` runs after a successful `GreptimeDB Nightly
Build` (`workflow_run`). It resolves that run's `head_sha` as the candidate
and the previous successful nightly (same branch, typically Friday when
Monday's nightly fires) as the base, then calls `query-regression.yml` with
those immutable SHAs. Both binaries are still compiled on the ECS runner;
this is not an artifact-download path. `workflow_dispatch` can pass explicit
`base_ref` / `candidate_ref` or a nightly run id. If there is no previous
nightly, or both nightlies built the same commit, the comparison is skipped.
## Aliyun ECS path
Configuration lives in repository variables/secrets:
| Kind | Name | Purpose |
| --- | --- | --- |
| secret | `ALICLOUD_ECS_ACCESS_KEY_ID` / `ALICLOUD_ECS_ACCESS_KEY_SECRET` | RAM user scoped to ECS RunInstances/DeleteInstances/Describe*/CreateImage/RunCommand. Used only by provision/teardown jobs on `ubuntu-latest`; never reaches the ECS instance. |
| secret | `GH_PERSONAL_ACCESS_TOKEN` | Creates the short-lived runner registration token (shared with the jsonbench EC2 path). |
| vars | `ALIYUN_ECS_REGION_ID` / `ALIYUN_ECS_VSWITCH_ID` / `ALIYUN_ECS_SECURITY_GROUP_ID` | Network placement. The security group should allow egress only; no inbound rules are needed. The vSwitch pins the zone. |
| vars | `ALIYUN_ECS_INSTANCE_TYPE` | Dedicated (non-burstable, non-shared) instance family. Prefer 32 GiB (e.g. `ecs.g8i.2xlarge`); `ecs.c9i.2xlarge` is 8c16g and nightly thin-LTO of greptime peaks above that. Both base and candidate clusters run on the same machine, so noisy neighbors break thresholds. |
| vars | `QUERY_REGRESSION_ECS_IMAGE_ID` | Custom image built by `ecs-image/build-ecs-image.py`. |
The system disk is 40 GiB, which covers the image, a 16 GiB swapfile, the
checkout, and cold build caches (target dir, cargo registry, sccache). ENOSPC
stops the runner itself from writing logs, which GitHub reports as `The
operation was canceled` with no telemetry, indistinguishable from a
platform-side cancellation.
cloud-init masks `systemd-oomd`, disables `unattended-upgrades` /
`apt-daily-upgrade`, creates `/swapfile`, and sets `OOMPolicy=continue`
on the runner unit. Ubuntu 24.04 defaults to `DefaultOOMPolicy=stop`,
which SIGTERM-s the whole unit when rustc is OOM-killed and GitHub
reports `The operation was canceled` with no telemetry. Unattended
upgrades can do the same via `systemctl restart` of the runner after a
library update; GitHub then records `UserCancelled` even though the job
was still valid. Swap is a safety net for 16 GiB types, not a substitute
for 32 GiB; linking on swap is slow.
When a run dies with an unexplained `The operation was canceled` (no
"Canceled by" banner, healthy machine), re-dispatch with `keep_instance`
checked: the teardown job is skipped and the instance survives for
post-mortem inspection. The security group is egress-only, so inspect via
Cloud Assistant (`RunCommand`) or VNC: the runner's `_diag` logs under the
runner home record reconnects and worker crashes, `journalctl -u
ephemeral-github-runner.service` mirrors the console stream, `dmesg -T`
shows kernel OOM kills, and `machine-telemetry.log` in the job workspace
has the 30 s sampler history. The janitor sweep still deletes the instance
after its TTL, so finish the inspection within that window.
Trust model on the ECS path: the instance receives only the one-hour runner
registration token via user data and holds no cloud credentials; the Aliyun
AK/SK exist only in the control-plane jobs. Instance tags
(`managed-by=query-regression-ci`, `query-regression-run-id`) feed the janitor
sweep.
### Building and updating the ECS image
The runner `Dockerfile` in the parent directory stays the single source of the
tool contract. Build a new ECS image from it:
```bash
helm upgrade --install arc \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
--namespace arc-systems \
--create-namespace \
--version 0.14.2
ALIBABA_CLOUD_ACCESS_KEY_ID=... ALIBABA_CLOUD_ACCESS_KEY_SECRET=... \
uv run .github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py \
--region-id <region> --vswitch-id <vsw-...> --security-group-id <sg-...> \
--base-image-id <ubuntu-24.04-image-id>
```
Create the GitHub App secret in the runner namespace. Prefer an App limited to
`GreptimeTeam/greptimedb`:
The script boots a temporary builder instance, `docker build`s the runner
image, materializes `/opt/rustup`, `/opt/cargo`, `/usr/local/bin` tools, and
`/home/runner` (actions-runner) onto the host, installs the ephemeral-runner
systemd unit from `ecs-image/`, and snapshots a custom image. It prints the
image id; set it as `QUERY_REGRESSION_ECS_IMAGE_ID`, and bump
`RUNNER_IMAGE_EPOCH` in `query-regression.yml` at the same time so the target
cache invalidates. Builder sentinel polling requires the Cloud Assistant
agent, which Aliyun public Ubuntu images include.
```bash
kubectl -n arc-runners create secret generic greptimedb-arc-github-app \
--from-literal=github_app_id=<app-id> \
--from-literal=github_app_installation_id=<installation-id> \
--from-file=github_app_private_key=<private-key.pem>
```
The values files here reference that secret by name.
## Trust admission for PR runs
A maintainer applying the `query-regression` or `heavy-regression` label is
**trust admission for that exact PR revision**. `query-regression` runs the six
routine default cases; `heavy-regression` runs only the high-cardinality
`prom_remote_write_7913` remote-write case. The admitted job may use this scale
set's dedicated, writable persistent cache. `pull_request: labeled` is the only
PR trigger: the label event snapshots its merge, head, and base SHAs. A queued
job fetches that immutable event merge SHA directly, verifies it is a two-parent
merge whose parents include the snapshotted head exactly once, and uses its
other parent as the actual base build revision. The snapshotted event base is
retained for audit only, so a difference from the merge's non-head parent is
not a failure. The job never follows a newer mutable PR merge ref. An
unavailable event merge, or one that does not contain exactly one snapshotted
head parent, fails closed. A later PR head change does not retarget an already
queued run: it may execute only its previously trusted event revision if that
revision remains fetchable. To run the new revision, the maintainer must review
it, remove the label, and re-add the desired regression label; cancel the old
run if it is no longer wanted. An existing label does not automatically rerun
the benchmark.
**trust admission for that exact PR revision**. `query-regression` runs the
six routine default cases; `heavy-regression` runs only the high-cardinality
`prom_remote_write_7913` remote-write case. `pull_request: labeled` is the only PR
trigger: the label event snapshots its merge, head, and base SHAs. A queued
job fetches that immutable event merge SHA directly, verifies it is a
two-parent merge whose parents include the snapshotted head exactly once, and
uses its other parent as the actual base build revision. The snapshotted
event base is retained for audit only, so a difference from the merge's
non-head parent is not a failure. The job never follows a newer mutable PR
merge ref. An unavailable event merge, or one that does not contain exactly
one snapshotted head parent, fails closed. A later PR head change does not
retarget an already queued run: it may execute only its previously trusted
event revision if that revision remains fetchable. To run the new revision,
the maintainer must review it, remove the label, and re-add the desired
regression label; cancel the old run if it is no longer wanted. An existing
label does not automatically rerun the benchmark.
Admission does not relax runner hardening or GitHub permissions. Keep
service-account token mounting disabled; do not mount host paths, the Docker
socket, kubeconfig, or long-lived credentials. The runner and cache initializer
use UID/GID 1001, disallow privilege escalation, drop all capabilities, and use
the RuntimeDefault seccomp profile. Keep GitHub tokens least-privilege and
review workflow changes before admission. Where the CNI supports it, restrict
egress to required GitHub Actions, artifact/cache, Rust/crate/toolchain, DNS,
and image-registry endpoints; block unrelated cluster services, private ranges,
and metadata endpoints unless a case requires them.
### Network routing prerequisite
Required split routing is an **external environment-specific prerequisite**. The
responsible network operator must route GitHub Actions, GitHub content,
artifact/cache, crates.io, Rust toolchain, and image-registry traffic through
the approved path rather than the VPN where applicable. Neither this repository
nor Kubernetes configures that route. Verify it with the responsible network
operator before any canary.
Admission does not relax runner hardening or GitHub permissions. Keep the ECS
instance free of cloud credentials and long-lived tokens, keep the security
group egress-only, keep GitHub tokens least-privilege, and review workflow
changes before admission.
## Runner image and workflow tools
Build and push the derived runner image; it preserves the official
`/home/runner/run.sh` entrypoint and supplies CI tools needed at runtime. The
image builds `otelgen` from
The runner `Dockerfile` builds `otelgen` from
[`WenyXu/otelgen`](https://github.com/WenyXu/otelgen) commit
[`863a3f395d062c7322cc1de08a38774b7fdaa6c8`](https://github.com/WenyXu/otelgen/commit/863a3f395d062c7322cc1de08a38774b7fdaa6c8)
so trace cases do not download or compile tools during a benchmark run:
so trace cases do not download or compile tools during a benchmark run. It is
built only as a toolchain factory: `build-ecs-image.py` and
`bootstrap-runner-host.sh` both materialize its contents onto a host, and the
benchmark itself runs host-native.
```bash
docker build \
--platform linux/amd64 \
-f .github/runner-scale-sets/query-regression/Dockerfile \
-t greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest \
.github/runner-scale-sets/query-regression
docker push greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest
```
Deploy by digest, not mutable tag, by updating both image references in
`values-8-cores.yaml` after a rebuild. Update `RUNNER_IMAGE_DIGEST` and bump
`RUNNER_IMAGE_EPOCH` in `query-regression.yml` at the same time. If the registry
is private, use a dedicated read-only pull secret only as `imagePullSecrets`;
never expose registry credentials to runner containers.
Both digest-pinned init and runner containers use `IfNotPresent`: the immutable
digest makes a cached image safe and avoids adding a registry dependency to every
runner startup.
The runner optionally imports only the non-sensitive `HTTP_PROXY`,
`HTTPS_PROXY`, and `NO_PROXY` variables from the
`query-regression-runner-local-env` ConfigMap. Manage that ConfigMap locally in
the target namespace; private endpoint configuration must not be committed, and
credentials or secrets must never be placed in a ConfigMap.
Before builds, the workflow asserts UID/GID 1001 and exact image tool versions:
`libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.30.0`, `Python 3.12.3`, `sccache
0.16.0`, `otelgen` commit `863a3f395d062c7322cc1de08a38774b7fdaa6c8`,
root-owned `rustup 1.29.0`, and the image-baked
`nightly-2026-03-21` Rust toolchain. Rustup, Cargo, and Rustc must resolve from
`/opt/cargo/bin`; the runner cannot write `/opt/rustup` or `/opt/cargo/bin`.
Protobuf well-known includes, including `google/protobuf/any.proto` and
`google/protobuf/empty.proto`, are an image contract and must compile with
`protoc`.
Before builds, the workflow asserts the runner UID/GID (1001 in the ECS image;
overridable via `QUERY_REGRESSION_RUNNER_UID`/`QUERY_REGRESSION_RUNNER_GID`)
and exact tool versions: `libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.40.4`,
`Python 3.14.4`, `sccache 0.16.0`, `otelgen` commit
`863a3f395d062c7322cc1de08a38774b7fdaa6c8`, root-owned `rustup 1.29.0`, and
the image-baked `nightly-2026-03-21` Rust toolchain. `mold` and `python3`
come from apt at image-build time (not Ubuntu 24.04's default 3.12); bump
the Verify pins together with `QUERY_REGRESSION_ECS_IMAGE_ID` when the
image is rebuilt. Rustup, Cargo, and Rustc
must resolve from `/opt/cargo/bin`; the runner cannot write `/opt/rustup` or
`/opt/cargo/bin`. Protobuf well-known includes, including
`google/protobuf/any.proto` and `google/protobuf/empty.proto`, are an image
contract and must compile with `protoc`.
`actions-rust-lang/setup-rust-toolchain@v1` is intentionally removed. The
workflow sets its warning-denying mold `RUSTFLAGS` directly, disables automatic
Rustup installation, and performs no runtime toolchain downloads.
workflow sets its warning-denying mold `RUSTFLAGS` directly, disables
automatic Rustup installation, and performs no runtime toolchain downloads.
The workflow no longer uses GitHub `rust-cache`, `setup-protoc`, `setup-uv`, or
runtime Rust setup: the image establishes immutable executable state and the PVC
supplies only reusable Cargo data. Do not reintroduce those actions unless the
corresponding cache or image contract changes.
The workflow no longer uses GitHub `rust-cache`, `setup-protoc`, `setup-uv`,
or runtime Rust setup: the image establishes immutable executable state and
each instance's system disk holds only that run's Cargo data. Do not
reintroduce those actions unless the image contract changes.
## Capacity and persistent cache
## Capacity
`values-8-cores.yaml` is normal operation: `minRunners=0`, `maxRunners=1`.
`values-paused.yaml` is the mandatory pause overlay: `minRunners=0`,
`maxRunners=0`. The job uses group `query-regression-persistent-cache-v1`,
`queue: max`, and `cancel-in-progress: false`; admitted jobs queue rather than
replacing older pending jobs. During maintenance, cancel admitted queued runs as
well as pausing ARC. Runner Pods have `activeDeadlineSeconds=12600`.
The runner requests 6 CPU and limits at 8 CPU to preserve `minipc-3`
allocatable-capacity scheduling headroom; do not reset the request to 8 CPU
without revalidating scheduling capacity.
Each run provisions its own ECS instance, so overlapping dispatches proceed
in parallel. There is no workflow `concurrency` group. All Cargo state
(`CARGO_HOME` including registry/git, `CARGO_TARGET_DIR`, sccache, cache
metadata) lives on the 40 GiB system disk and is discarded with the VM.
`RUSTUP_HOME=/opt/rustup` and `/opt/cargo/bin` are image-owned. The runner
sets `RUSTC_WRAPPER=/usr/local/bin/sccache`,
`SCCACHE_DIR=/home/runner/.cache/sccache`, `SCCACHE_CACHE_SIZE=10G`, and
`CARGO_INCREMENTAL=0`. sccache uses its local disk backend and self-evicts
at 10G. Base and candidate builds share the target on that disk; Cargo
fingerprints invalidate source and dependency changes.
The cache claim `query-regression-build-cache` is a nominal 600Gi `local-path`
PVC in `arc-runners`. It is `ReadWriteOnce`; `local-path` uses
WaitForFirstConsumer binding and Delete reclaim behavior, produces a
node-affine local PV, is non-expandable, and the 600Gi request is not a hard
storage quota. The runner's `minipc-3` selector is its only consumer candidate.
The initializer mounts the PVC root at `/cache`, creates and write-tests these
versioned subpaths as non-root UID/GID 1001, and the runner mounts them as:
| Persistent state | PVC subpath | Runner mount |
| --- | --- | --- |
| Ephemeral Cargo home | `emptyDir` | `/home/runner/.cargo` |
| Cargo registry data | `cargo-registry-v1` | `/home/runner/.cargo/registry` |
| Cargo Git data | `cargo-git-v1` | `/home/runner/.cargo/git` |
| Cargo target | `query-regression-target-v1` | `/home/runner/query-regression-target` |
| Cache metadata | `meta-v1` | `/home/runner/query-regression-cache-meta` |
| sccache local disk cache | `sccache-v1` | `/home/runner/.cache/sccache` |
| Immutable Rust toolchain | image-owned | `/opt/rustup`, `/opt/cargo/bin` |
The Pod security context uses UID/GID and `fsGroup` 1001 with
`fsGroupChangePolicy: OnRootMismatch`; no privileged `chown` or raw `hostPath`
is used. `CARGO_HOME` is a per-Pod `emptyDir`; only its nested `registry` and
`git` mounts are persistent. `RUSTUP_HOME=/opt/rustup` and `/opt/cargo/bin` are
image-owned immutable paths, while `CARGO_TARGET_DIR`, cache metadata, and
`SCCACHE_DIR` are persistent absolute paths. The runner sets
`RUSTC_WRAPPER=/usr/local/bin/sccache`,
`SCCACHE_DIR=/home/runner/.cache/sccache`, `SCCACHE_CACHE_SIZE=40G`, and
`CARGO_INCREMENTAL=0`. sccache uses its local PVC disk backend and self-evicts
at 40G; do not add runtime downloads, object storage, or a shared backend.
The repository's `.cargo/config.toml` remains a trusted per-revision build input.
In contrast, `$CARGO_HOME/config*`, credentials, installed bins, and Cargo
metadata outside the persistent `registry` and `git` data mounts are ephemeral
and cannot survive to another Pod.
The local disk backend has a one-server constraint. `maxRunners=1` and the
unchanged `query-regression-persistent-cache-v1` workflow concurrency group
serialize runs; do not increase runner capacity or relax that serialization
while this backend is in use. Base and candidate builds share the target; Cargo
fingerprints invalidate source and dependency changes. The workflow records the
sccache version and relevant environment in the target ABI marker, starts and
zeros sccache after cache and toolchain checks, shows initial/base/candidate
statistics, and resets statistics between base and candidate builds.
### Disk preflight and cleanup contract
Before applying or unpausing, verify the backing filesystem on `minipc-3` has
at least 900GiB free. The current local-path provisioner source is
`/opt/local-path-provisioner`; measure the filesystem containing it:
```bash
df -PB1G /opt/local-path-provisioner
```
The workflow reports `du`, `df -P`, human-readable free space, and inode
availability before builds and in an always-run report. Its cleanup is narrow
and non-destructive:
- warn at target size 400GiB; at 450GiB clear only the complete target root;
- warn at Cargo registry-plus-Git data size 60GiB; at 80GiB remove only
`registry/src` and `git/checkouts`, then abort if that persistent data remains
at least 80GiB;
- below 300GiB backing free space, clear the complete target root first,
remeasure, then remove only those Cargo extracted trees and checkouts; abort
if free space is still below 300GiB;
- never automatically remove Cargo registry cache/index, Git database, the
image-owned Cargo bin or Rustup toolchain, cache metadata, the self-evicting
sccache directory, or the PVC.
The target clear uses fixed absolute roots and removes all entries, including
dotfiles. After migration validation, remove obsolete `cargo-home-v1` and
`rustup-home-v1` only in explicit maintenance while ARC is 0/0 and no runner Pod
exists; they are not mounted by the current configuration.
## Deploy and pause safely
First verify the configured external network route and the disk preflight. Apply the PVC; while
the scale set is paused, it is expected to remain `Pending` because
WaitForFirstConsumer has no scheduled runner:
```bash
kubectl apply --dry-run=server \
-f .github/runner-scale-sets/query-regression/cache-pvc.yaml
kubectl apply -f .github/runner-scale-sets/query-regression/cache-pvc.yaml
```
Render normal and paused configurations. Normal values are always first; the
pause overlay is always last:
```bash
helm template perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --version 0.14.2 \
--set controllerServiceAccount.name=arc-gha-rs-controller \
--set controllerServiceAccount.namespace=arc-systems \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml
helm template perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --version 0.14.2 \
--set controllerServiceAccount.name=arc-gha-rs-controller \
--set controllerServiceAccount.namespace=arc-systems \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml \
-f .github/runner-scale-sets/query-regression/values-paused.yaml
```
The **first post-merge Helm deployment must reconcile the release in paused
mode**. Keep the pause overlay last:
```bash
# First post-merge deployment and every return to paused mode: 0/0.
helm upgrade --install perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --create-namespace --version 0.14.2 \
--reset-values --wait \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml \
-f .github/runner-scale-sets/query-regression/values-paused.yaml
# Expect 0/0 and no runner resources before considering normal mode.
kubectl -n arc-runners get autoscalingrunnerset perf-regression-8-cores \
-o jsonpath='{.spec.minRunners}{"/"}{.spec.maxRunners}{"\n"}'
kubectl -n arc-runners get ephemeralrunners,pods \
-l actions.github.com/scale-set-name=perf-regression-8-cores
```
Only after that verification and separate explicit approval, apply normal 0/1
operation without the pause overlay:
```bash
helm upgrade --install perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --create-namespace --version 0.14.2 \
--reset-values --wait \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml
```
Do not use bare `helm rollback`, `--atomic`, or `--reuse-values`: a stored
revision can restore nonzero runner capacity. Inspect rendered manifests for
capacity, the `minipc-3` selector, cache claim and mounts, initializer,
security context, and resources. After approved normal mode receives its first
canary, the PVC binds to `minipc-3`.
For local-PV node loss, cache recovery is intentionally disposable: return to
0/0, recreate the PVC on a healthy node, cold-fill it, and run a new canary.
## Canary and rollback
With explicit approval, run two identical `workflow_dispatch` canaries on
`perf-regression-8-cores`, using immutable full base and candidate commit SHAs
and `cargo_profile=nightly`. The first is the cold fill; the second verifies warm
reuse. Record the workflow's base/candidate build elapsed logs and cache
ABI-marker output, initial/base/candidate sccache statistics, and cache report.
Confirm the image tool contract (root-owned Rustup/Cargo paths, baked nightly,
and non-writable `/opt` roots) and that the ephemeral Cargo home contains only
the mounted registry/Git data before Cargo creates per-Pod state.
Obtain dependency and tool network byte counters from the configured
environment counter source, filtered to `minipc-3` and the dependency/tool
destinations.
Accept the canary only when all of the following hold:
- exactly one runner Pod runs on `minipc-3`, and both jobs use the same bound PV;
- UID/GID 1001 cache mounts are writable; the warm run does not invalidate the
target or bulk-redownload crates or toolchains; sccache reports separate base
and candidate build statistics without server or cache-path errors; immutable
Rustup/Cargo roots remain non-writable and only registry/Git data persists;
- warm base build time is at most 50% of cold base build time;
- warm dependency/tool network bytes are at most 10% of cold fill bytes;
- cache sizes remain below soft watermarks, node free space remains at least
300GiB, and the benchmark is correct without TLS EOFs or timeouts;
- the configured environment counter source confirms this traffic is outside VPN
accounting.
Immediately return to 0/0 after either canary unless ongoing normal operation
has been explicitly approved; return immediately on any traffic, cache, disk,
TLS, or correctness failure. To roll back, use the paused Helm upgrade above,
or another explicit `helm upgrade --install` with known-good values followed by
`values-paused.yaml`, `--reset-values`, and `--wait`. Do not delete the PVC
automatically; preserve it for diagnosis unless intentionally discarding cache.
The workflow reports `du` / `df` in telemetry. It does not try to reclaim
space across runs: a cold 40 GiB disk that fills up fails the build.
## Future optional phases
The current phase uses a digest-pinned image with sccache 0.16.0 and no shared
cache service. Optional follow-ups are an image additionally seeded with the
exact Rust toolchain and `cargo fetch --locked`; or an internal read/write
sccache backend or Cargo/Git mirror. Evaluate them only if persistent PVC reuse
is insufficient.
The current phase uses a materialized runner toolchain with sccache 0.16.0 and
no shared cache service. Optional follow-ups are an image additionally seeded
with `cargo fetch --locked` results, or an internal read/write sccache
backend or Cargo/Git mirror. Evaluate them only if cold compile time on the
system disk becomes the bottleneck.
.github/runner-scale-sets/query-regression/cache-pvc.yaml
-15
View File
@@ -1,15 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: query-regression-build-cache
namespace: arc-runners
labels:
app.kubernetes.io/part-of: perf-regression-8-cores
spec:
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: 600Gi
.github/runner-scale-sets/query-regression/ecs-image/bootstrap-runner-host.sh
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Bootstrap an Ubuntu 24.04 host as a query-regression self-hosted runner.
#
# This is the host-native equivalent of the ECS image contract: it
# installs docker-ce from Docker's official repository, builds the runner
# image from the sibling Dockerfile (the single source of the tool contract),
# materializes the toolchain onto the host, and optionally registers the
# runner and installs its systemd service.
#
# Large state lives under --data-root (a data disk): docker's data-root, the
# runner home, and the rustup/cargo toolchain roots, symlinked back to the
# hard-coded contract paths (/home/runner, /opt/rustup, /opt/cargo).
#
# The script is idempotent: re-running refreshes the toolchain in place, and
# the runner registration (.runner/.credentials) and _work survive. A live
# runner service is stopped first and restarted at the end.
#
# Usage:
# sudo bash bootstrap-runner-host.sh --repo-dir /path/to/greptimedb
# sudo RUNNER_TOKEN=<token> bash bootstrap-runner-host.sh --repo-dir . \
# --register --repo-url https://github.com/<owner>/<repo>
#
# Options:
# --repo-dir PATH greptimedb checkout containing the runner Dockerfile (required)
# --data-root PATH data disk mount point (default: /data)
# --uid / --gid N runner user id (default: 3141; the ECS image contract is 1001)
# --register also register the runner and install the systemd service
# --repo-url URL repository URL for registration (required with --register)
# --runner-name N runner name (default: qreg-host)
# --labels L runner labels (default: perf-regression-8-cores)
#
# With --register, provide a fresh registration token via RUNNER_TOKEN.
set -euo pipefail
REPO_DIR=""
DATA_ROOT="/data"
RUNNER_UID="3141"
RUNNER_GID="3141"
REGISTER="false"
REPO_URL=""
RUNNER_NAME="qreg-host"
RUNNER_LABELS="perf-regression-8-cores"
IMAGE_TAG="qreg-runner:manual"
while [[ $# -gt 0 ]]; do
case "$1" in
--repo-dir) REPO_DIR="$2"; shift 2 ;;
--data-root) DATA_ROOT="$2"; shift 2 ;;
--uid) RUNNER_UID="$2"; shift 2 ;;
--gid) RUNNER_GID="$2"; shift 2 ;;
--register) REGISTER="true"; shift ;;
--repo-url) REPO_URL="$2"; shift 2 ;;
--runner-name) RUNNER_NAME="$2"; shift 2 ;;
--labels) RUNNER_LABELS="$2"; shift 2 ;;
-h | --help) sed -n '17,40p' "$0"; exit 0 ;;
*) echo "Unknown argument: $1" >&2; exit 2 ;;
esac
done
[[ $EUID -eq 0 ]] || { echo "Run as root, e.g. sudo bash $0 ..." >&2; exit 1; }
[[ -n "${REPO_DIR}" ]] || { echo "--repo-dir is required" >&2; exit 2; }
DOCKERFILE="${REPO_DIR}/.github/runner-scale-sets/query-regression/Dockerfile"
[[ -f "${DOCKERFILE}" ]] || { echo "Dockerfile not found at ${DOCKERFILE}" >&2; exit 1; }
if [[ "${REGISTER}" == "true" ]]; then
[[ -n "${REPO_URL}" ]] || { echo "--repo-url is required with --register" >&2; exit 2; }
[[ -n "${RUNNER_TOKEN:-}" ]] || { echo "RUNNER_TOKEN env is required with --register" >&2; exit 2; }
fi
step() { printf '\n==> %s\n' "$*"; }
# A re-run refreshes files under the (possibly live) runner service, so stop
# it first and restart it at the end (unless --register re-creates it).
RUNNER_SERVICE="$(systemctl list-units --type=service --all --no-legend 'actions.runner.*' 2>/dev/null | awk '{print $1}' | head -n 1 || true)"
RUNNER_WAS_ACTIVE="false"
if [[ -n "${RUNNER_SERVICE}" ]] && systemctl is-active --quiet "${RUNNER_SERVICE}"; then
step "Stop running runner service ${RUNNER_SERVICE}"
systemctl stop "${RUNNER_SERVICE}"
RUNNER_WAS_ACTIVE="true"
fi
step "Install docker-ce from Docker's official repository"
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do
apt-get remove -y "${pkg}" 2>/dev/null || true
done
apt-get update
apt-get install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
# shellcheck disable=SC1091
. /etc/os-release
tee /etc/apt/sources.list.d/docker.sources > /dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: ${UBUNTU_CODENAME:-$VERSION_CODENAME}
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin
step "Move docker data-root to ${DATA_ROOT}/docker"
systemctl stop docker containerd
mkdir -p "${DATA_ROOT}/docker" "${DATA_ROOT}/containerd"
if [[ -f /etc/docker/daemon.json ]] && ! grep -q '"data-root"' /etc/docker/daemon.json; then
echo "ERROR: /etc/docker/daemon.json exists without data-root; merge manually." >&2
exit 1
fi
echo "{ \"data-root\": \"${DATA_ROOT}/docker\" }" > /etc/docker/daemon.json
# containerd keeps its own content store (where build-time image data lands);
# move it off the system disk too.
mkdir -p /etc/containerd
containerd config default | sed "s|root = \"/var/lib/containerd\"|root = \"${DATA_ROOT}/containerd\"|" > /etc/containerd/config.toml
systemctl start containerd docker
docker info | grep "Docker Root Dir"
step "Build the runner image"
docker build --platform linux/amd64 -f "${DOCKERFILE}" -t "${IMAGE_TAG}" "${REPO_DIR}"
step "Materialize the toolchain onto the host"
mkdir -p "${DATA_ROOT}/opt" "${DATA_ROOT}/runner" /opt /home
container="$(docker create "${IMAGE_TAG}")"
trap 'docker rm -f "${container}" >/dev/null 2>&1 || true' EXIT
# The `/.` suffix copies directory *contents*, so re-running over an existing
# materialization refreshes it in place instead of nesting runner/runner.
# Files absent from the image (runner registration, _work) are preserved.
docker cp "${container}:/home/runner/." "${DATA_ROOT}/runner"
docker cp "${container}:/opt/rustup/." "${DATA_ROOT}/opt/rustup"
docker cp "${container}:/opt/cargo/." "${DATA_ROOT}/opt/cargo"
for tool in uv uvx otelgen sccache; do
docker cp "${container}:/usr/local/bin/${tool}" "/usr/local/bin/${tool}"
done
docker rm "${container}" > /dev/null
trap - EXIT
ln -sfn "${DATA_ROOT}/runner" /home/runner
ln -sfn "${DATA_ROOT}/opt/rustup" /opt/rustup
ln -sfn "${DATA_ROOT}/opt/cargo" /opt/cargo
docker image rm "${IMAGE_TAG}" > /dev/null
step "Install system packages"
apt-get install -y --no-install-recommends \
build-essential clang cmake git gzip jq libprotobuf-dev libssl-dev mold \
openssh-client pkg-config protobuf-compiler python3 sudo tar unzip wget xz-utils zip zstd
step "Create runner user (${RUNNER_UID}:${RUNNER_GID}) and environment"
getent group "${RUNNER_GID}" > /dev/null || groupadd -g "${RUNNER_GID}" runner
id -u runner > /dev/null 2>&1 || useradd -u "${RUNNER_UID}" -g "${RUNNER_GID}" -d /home/runner -s /bin/bash runner
chown -R "${RUNNER_UID}:${RUNNER_GID}" "${DATA_ROOT}/runner"
echo 'PATH=/opt/cargo/bin:/usr/local/bin:/usr/bin:/bin' > "${DATA_ROOT}/runner/.env"
chown "${RUNNER_UID}:${RUNNER_GID}" "${DATA_ROOT}/runner/.env"
if [[ "${REGISTER}" == "true" ]]; then
step "Register runner ${RUNNER_NAME} (labels: ${RUNNER_LABELS})"
if [[ -n "${RUNNER_SERVICE}" ]]; then
(cd /home/runner && ./svc.sh uninstall) || true
fi
runuser -u runner -- bash -c "cd /home/runner && HOME=/home/runner ./config.sh \
--url '${REPO_URL}' --token '${RUNNER_TOKEN}' --name '${RUNNER_NAME}' \
--labels '${RUNNER_LABELS}' --unattended --replace --disableupdate"
step "Install and start the runner service"
cd /home/runner
./svc.sh install runner
./svc.sh start
./svc.sh status
else
step "Skipping registration (pass --register --repo-url ... with RUNNER_TOKEN to enable)"
if [[ "${RUNNER_WAS_ACTIVE}" == "true" ]]; then
step "Restart runner service ${RUNNER_SERVICE}"
systemctl start "${RUNNER_SERVICE}"
systemctl --no-pager status "${RUNNER_SERVICE}" || true
fi
fi
step "Done"
echo "Runner home: /home/runner -> ${DATA_ROOT}/runner"
echo "Before each workflow run on this persistent host, clean transient cargo state:"
echo " sudo rm -f /home/runner/.cargo/.package-cache"
echo " sudo rm -rf /home/runner/.cargo/.global-cache"
.github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py
+392
View File
@@ -0,0 +1,392 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# PEP 723 inline metadata (see .github/scripts/aliyun-ecs-runner-provision.py
# for the convention).
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "alibabacloud_ecs20140526>=4.1.0,<6",
# "alibabacloud_tea_openapi>=0.3.12,<1",
# ]
# ///
"""Build the query-regression ECS custom image (manual ops tool).
Boots a temporary pay-as-you-go ECS instance from a public Ubuntu 24.04 image,
builds the existing runner container image (the Dockerfile in the parent
directory remains the single source of the tool contract), materializes the
tool directories onto the host filesystem so the workflow's "Verify runner
image tools" step holds unchanged, installs the ephemeral-runner systemd unit,
and snapshots the result as a custom image. The temporary instance is deleted
afterwards.
Sentinel polling reads the instance's serial console output
(GetInstanceConsoleOutput) and looks for marker lines the user data writes to
/dev/console. This has no in-guest agent dependency.
Usage:
uv run .github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py \
--region-id cn-hangzhou --vswitch-id vsw-... --security-group-id sg-... \
--base-image-id ubuntu_24_04_x64_20G_alibase_*.vhd
"""
from __future__ import annotations
import argparse
import base64
import os
import time
from pathlib import Path
ASSETS_DIR = Path(__file__).resolve().parent
DONE_MARKER = "QREG_IMAGE_BUILD_DONE"
FAILED_MARKER = "QREG_IMAGE_BUILD_FAILED"
POLL_INTERVAL_SECONDS = 15
CONSOLE_POLL_INTERVAL_SECONDS = 30
BUILD_TIMEOUT_SECONDS = 60 * 60
# Same apt package contract as the runner Dockerfile; the base
# actions-runner image is Ubuntu 24.04, so an Ubuntu 24.04 host resolves the
# same tool versions (protoc 3.21.12, mold 2.40.4, Python 3.14.4).
# Docker itself comes from Docker's official repository (docker-ce), not the
# distribution-packaged docker.io.
APT_PACKAGES = [
"build-essential",
"ca-certificates",
"clang",
"cmake",
"curl",
"git",
"gpg",
"gzip",
"jq",
"libprotobuf-dev",
"libssl-dev",
"mold",
"openssh-client",
"pkg-config",
"protobuf-compiler",
"python3",
"sudo",
"tar",
"unzip",
"wget",
"xz-utils",
"zip",
"zstd",
]
DOCKER_CE_PACKAGES = "docker-ce docker-ce-cli containerd.io docker-buildx-plugin"
def render_user_data(dockerfile: str, start_runner: str, unit: str) -> str:
dockerfile_b64 = base64.b64encode(dockerfile.encode()).decode()
start_runner_b64 = base64.b64encode(start_runner.encode()).decode()
unit_b64 = base64.b64encode(unit.encode()).decode()
packages = " ".join(APT_PACKAGES)
return f"""#!/bin/bash
set -euo pipefail
trap 'echo "{FAILED_MARKER} at line $LINENO" > /dev/console' ERR
# Stream the full build log to the serial console so the poller (and anyone
# watching GetInstanceConsoleOutput) sees real progress, not a silent login
# prompt for the whole build.
exec > >(tee -a /dev/console) 2>&1
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends {packages}
# Docker from the official repository, not the distribution-packaged docker.io.
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
. /etc/os-release
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu ${{VERSION_CODENAME}} stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends {DOCKER_CE_PACKAGES}
base64 -d > /tmp/Dockerfile <<'EOF'
{dockerfile_b64}
EOF
mkdir -p /tmp/image-context
docker build --platform linux/amd64 -f /tmp/Dockerfile -t qreg-runner:local /tmp/image-context
# Materialize the tool contract onto the host filesystem.
container="$(docker create qreg-runner:local)"
trap 'docker rm -f "${{container}}" >/dev/null 2>&1 || true' EXIT
docker cp "${{container}}:/opt/rustup" /opt/rustup
docker cp "${{container}}:/opt/cargo" /opt/cargo
for tool in uv uvx otelgen sccache; do
docker cp "${{container}}:/usr/local/bin/${{tool}}" "/usr/local/bin/${{tool}}"
done
docker cp "${{container}}:/home/runner" /home/runner
# Runner identity mirrors the container image (UID/GID 1001).
groupadd --gid 1001 runner 2>/dev/null || true
useradd --uid 1001 --gid 1001 --home-dir /home/runner --shell /bin/bash runner 2>/dev/null || true
chown -R 1001:1001 /home/runner
mkdir -p /opt/ephemeral-github-runner
base64 -d > /opt/ephemeral-github-runner/start-runner.sh <<'EOF'
{start_runner_b64}
EOF
chmod 0755 /opt/ephemeral-github-runner/start-runner.sh
base64 -d > /etc/systemd/system/ephemeral-github-runner.service <<'EOF'
{unit_b64}
EOF
systemctl daemon-reload
systemctl enable ephemeral-github-runner.service
# Keep the image free of the build-time docker state (also shrinks the
# snapshot: buildkit cache is several GB).
docker rm -f "${{container}}" >/dev/null
docker system prune -af >/dev/null
trap - EXIT
echo "{DONE_MARKER}" > /dev/console
"""
def make_ecs_client(region_id: str):
from alibabacloud_ecs20140526.client import Client as EcsClient
from alibabacloud_tea_openapi.models import Config as OpenApiConfig
return EcsClient(
OpenApiConfig(
access_key_id=os.environ["ALIBABA_CLOUD_ACCESS_KEY_ID"],
access_key_secret=os.environ["ALIBABA_CLOUD_ACCESS_KEY_SECRET"],
region_id=region_id,
endpoint=f"ecs.{region_id}.aliyuncs.com",
)
)
def wait_for_instance_status(client, region_id: str, instance_id: str, wanted: str, deadline: float) -> None:
from alibabacloud_ecs20140526 import models as ecs_models
import json
while time.monotonic() < deadline:
response = call_api_with_retry(
lambda: client.describe_instances(
ecs_models.DescribeInstancesRequest(
region_id=region_id, instance_ids=json.dumps([instance_id])
)
),
"DescribeInstances",
)
instances = response.body.instances.instance
if instances and instances[0].status == wanted:
return
time.sleep(POLL_INTERVAL_SECONDS)
raise TimeoutError(f"Instance {instance_id} did not reach status {wanted} in time")
# Aliyun error codes worth retrying: throttling and server-side faults. Client
# errors (4xx: permissions, bad parameters) are configuration problems and must
# fail fast instead of being retried.
TRANSIENT_ERROR_CODES = {"Throttling", "Throttling.User", "InternalError", "ServiceUnavailable"}
def is_transient(error: Exception) -> bool:
# UnretryableException comes from the darabonba network layer: connection
# reset, read timeout, DNS blip.
if type(error).__name__ == "UnretryableException":
return True
code = getattr(error, "code", "") or ""
status = getattr(error, "statusCode", None)
return code in TRANSIENT_ERROR_CODES or (isinstance(status, int) and status >= 500)
def call_api_with_retry(fn, description: str, attempts: int = 5):
"""Retry transient API/network failures; the build is too long to die on a blip.
Only call this with idempotent or safely-repeatable requests (reads,
StopInstance, CreateImage). Never with RunInstances: if the request
succeeded but the response was lost, a retry double-creates instances.
"""
for attempt in range(1, attempts + 1):
try:
return fn()
except Exception as error: # noqa: BLE001
if attempt == attempts or not is_transient(error):
raise
print(f"{description} failed (attempt {attempt}/{attempts}): {error}", flush=True)
time.sleep(POLL_INTERVAL_SECONDS)
raise RuntimeError("unreachable: retry loop exited without returning")
def read_console_output(client, region_id: str, instance_id: str) -> str:
"""Fetch the instance's serial console output; no in-guest agent needed."""
from alibabacloud_ecs20140526 import models as ecs_models
response = call_api_with_retry(
lambda: client.get_instance_console_output(
ecs_models.GetInstanceConsoleOutputRequest(region_id=region_id, instance_id=instance_id)
),
"GetInstanceConsoleOutput",
)
return base64.b64decode(response.body.console_output or "").decode("utf-8", "replace")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID"))
parser.add_argument("--vswitch-id", default=os.environ.get("ALIYUN_ECS_VSWITCH_ID"))
parser.add_argument("--security-group-id", default=os.environ.get("ALIYUN_ECS_SECURITY_GROUP_ID"))
parser.add_argument("--base-image-id", default=os.environ.get("ALIYUN_ECS_BASE_IMAGE_ID"))
parser.add_argument("--resource-group-id", default=os.environ.get("ALIYUN_ECS_RESOURCE_GROUP_ID"))
parser.add_argument("--instance-type", default="ecs.g7.xlarge")
parser.add_argument("--image-name", default=None, help="Defaults to a timestamped name.")
args = parser.parse_args()
for name in ("region_id", "vswitch_id", "security_group_id", "base_image_id"):
if not getattr(args, name):
raise SystemExit(f"Missing required configuration: --{name.replace('_', '-')}")
from alibabacloud_ecs20140526 import models as ecs_models
client = make_ecs_client(args.region_id)
image_name = args.image_name or time.strftime(
"greptimedb-query-regression-runner-%Y%m%d%H%M%S", time.gmtime()
)
user_data = base64.b64encode(
render_user_data(
(ASSETS_DIR.parent / "Dockerfile").read_text(),
(ASSETS_DIR / "start-runner.sh").read_text(),
(ASSETS_DIR / "ephemeral-github-runner.service").read_text(),
).encode()
).decode()
instance_id = None
try:
response = client.run_instances(
ecs_models.RunInstancesRequest(
region_id=args.region_id,
image_id=args.base_image_id,
resource_group_id=args.resource_group_id,
instance_type=args.instance_type,
v_switch_id=args.vswitch_id,
security_group_id=args.security_group_id,
instance_name=f"build-{image_name}",
description="Temporary builder for the query-regression ECS image",
amount=1,
instance_charge_type="PostPaid",
spot_strategy="NoSpot",
internet_charge_type="PayByTraffic",
internet_max_bandwidth_out=100,
# Peak usage is ~18G (OS+apt, docker image, and the materialized
# toolchain coexist briefly): deliberately tight, and a smaller
# disk makes the image snapshot faster. Instances created from
# the image get a larger system disk from the provision side.
system_disk=ecs_models.RunInstancesRequestSystemDisk(
category="cloud_essd", size="20"
),
user_data=user_data,
tag=[
ecs_models.RunInstancesRequestTag(key="managed-by", value="query-regression-ci"),
ecs_models.RunInstancesRequestTag(key="role", value="image-builder"),
],
)
)
instance_id = response.body.instance_id_sets.instance_id_set[0]
print(f"Builder instance: {instance_id}", flush=True)
wait_for_instance_status(client, args.region_id, instance_id, "Running", time.monotonic() + 10 * 60)
deadline = time.monotonic() + BUILD_TIMEOUT_SECONDS
while time.monotonic() < deadline:
console = read_console_output(client, args.region_id, instance_id)
if DONE_MARKER in console:
break
if FAILED_MARKER in console:
tail = "\n".join(console.splitlines()[-20:])
raise RuntimeError(f"Image build failed on the builder; console tail:\n{tail}")
lines = console.splitlines()
print(f"Build in progress; last console line: {lines[-1] if lines else '(none yet)'}", flush=True)
time.sleep(CONSOLE_POLL_INTERVAL_SECONDS)
else:
raise TimeoutError("Image build did not finish in time")
print("Stopping builder before image creation", flush=True)
try:
call_api_with_retry(
lambda: client.stop_instance(
ecs_models.StopInstanceRequest(
instance_id=instance_id,
# The instance is deleted right after the snapshot, but
# there is no reason to keep billing vCPU during the stop.
stopped_mode="StopCharging",
)
),
"StopInstance",
)
except Exception as error: # noqa: BLE001
# Instance families with local disks do not support StopCharging.
print(f"StopCharging unavailable ({error}); stopping with default mode", flush=True)
call_api_with_retry(
lambda: client.stop_instance(ecs_models.StopInstanceRequest(instance_id=instance_id)),
"StopInstance",
)
wait_for_instance_status(client, args.region_id, instance_id, "Stopped", time.monotonic() + 10 * 60)
image = call_api_with_retry(
lambda: client.create_image(
ecs_models.CreateImageRequest(
region_id=args.region_id, instance_id=instance_id, image_name=image_name
)
),
"CreateImage",
)
image_id = image.body.image_id
deadline = time.monotonic() + 60 * 60
while time.monotonic() < deadline:
description = call_api_with_retry(
lambda: client.describe_images(
ecs_models.DescribeImagesRequest(region_id=args.region_id, image_id=image_id)
),
"DescribeImages",
)
images = description.body.images.image
if images:
status = images[0].status
print(f"Image {image_id} status: {status} (progress {images[0].progress})", flush=True)
if status == "Available":
break
time.sleep(POLL_INTERVAL_SECONDS)
else:
raise TimeoutError(
f"Image {image_id} did not become Available in time. The snapshot "
f"continues server-side: check its status in the console and reuse it "
f"once Available instead of rebuilding."
)
print(f"Custom image ready: {image_id} ({image_name})", flush=True)
print(f"Set the repo variable QUERY_REGRESSION_ECS_IMAGE_ID={image_id}", flush=True)
return 0
finally:
if instance_id:
try:
client.delete_instance(
ecs_models.DeleteInstanceRequest(instance_id=instance_id, force=True)
)
print(f"Deleted builder instance {instance_id}", flush=True)
except Exception as error: # noqa: BLE001
print(f"Failed to delete builder instance {instance_id}: {error}", flush=True)
if __name__ == "__main__":
raise SystemExit(main())
.github/runner-scale-sets/query-regression/ecs-image/ephemeral-github-runner.service
+45
View File
@@ -0,0 +1,45 @@
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Description=Ephemeral GitHub Actions runner
After=network-online.target
Wants=network-online.target
# cloud-final runs the provision user data, which writes
# /etc/ephemeral-github-runner.env and the drop-ins below. Ordering after it
# makes the runner come up configured on first boot, no restart needed.
# (cloud-init's `systemctl restart` in user data remains as the bridge for
# images built before this ordering existed.)
After=cloud-final.service
[Service]
Type=simple
# The toolchain lives in the image at /opt/cargo/bin (the Dockerfile's ENV
# PATH does not survive materialization onto the host). Publish it here so
# every runner job inherits it. /etc/ephemeral-github-runner.env currently
# carries the same PATH line as a bridge for images built before this
# directive existed; EnvironmentFile is applied after Environment=, and the
# two values are kept identical, so there is no conflict.
Environment=PATH=/opt/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
EnvironmentFile=/etc/ephemeral-github-runner.env
ExecStart=/opt/ephemeral-github-runner/start-runner.sh
Restart=no
# If the kernel OOM killer kills a job child (rustc/cargo), keep the runner
# service alive so the job is reported as a normal failure with logs instead
# of the runner vanishing and GitHub reporting a cancellation. cloud-init's
# oom.conf drop-in carries the same directive for images built before this.
OOMPolicy=continue
[Install]
WantedBy=multi-user.target
.github/runner-scale-sets/query-regression/ecs-image/start-runner.sh
+37
View File
@@ -0,0 +1,37 @@
#!/bin/bash
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configure and start the ephemeral GitHub Actions runner on an ECS instance.
# Baked into the custom image at /opt/ephemeral-github-runner/ by
# build-ecs-image.py; the per-run environment is written by cloud-init to
# /etc/ephemeral-github-runner.env.
set -euo pipefail
# shellcheck disable=SC1091
source /etc/ephemeral-github-runner.env
export HOME=/home/runner
cd /home/runner
if [[ ! -f /home/runner/.runner ]]; then
runuser -u runner -- ./config.sh \
--url "${REPO_URL}" \
--token "${RUNNER_TOKEN}" \
--name "${RUNNER_NAME}" \
--labels "${RUNNER_LABELS}" \
--ephemeral --unattended --replace --disableupdate
fi
exec runuser -u runner -- ./run.sh
.github/runner-scale-sets/query-regression/values-8-cores.yaml
-146
View File
@@ -1,146 +0,0 @@
githubConfigUrl: "https://github.com/GreptimeTeam/greptimedb"
githubConfigSecret: greptimedb-arc-github-app
runnerScaleSetName: "perf-regression-8-cores"
minRunners: 0
maxRunners: 1
template:
spec:
automountServiceAccountToken: false
activeDeadlineSeconds: 12600
nodeSelector:
kubernetes.io/hostname: minipc-3
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
volumes:
- name: cargo-home
emptyDir: {}
- name: build-cache
persistentVolumeClaim:
claimName: query-regression-build-cache
initContainers:
- name: initialize-build-cache
image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -ec
- |
umask 0002
for directory in cargo-registry-v1 cargo-git-v1 query-regression-target-v1 meta-v1 sccache-v1; do
cache_directory="/cache/${directory}"
mkdir -p "${cache_directory}"
test -w "${cache_directory}"
probe_file="${cache_directory}/.write-probe"
: > "${probe_file}"
test -f "${probe_file}"
rm "${probe_file}"
done
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
resources:
requests:
cpu: 100m
memory: 128Mi
ephemeral-storage: 1Gi
limits:
cpu: 500m
memory: 512Mi
ephemeral-storage: 1Gi
volumeMounts:
- name: build-cache
mountPath: /cache
containers:
- name: runner
image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11
imagePullPolicy: IfNotPresent
command: ["/home/runner/run.sh"]
env:
- name: CARGO_HOME
value: /home/runner/.cargo
- name: HTTP_PROXY
valueFrom:
configMapKeyRef:
name: query-regression-runner-local-env
key: HTTP_PROXY
optional: true
- name: HTTPS_PROXY
valueFrom:
configMapKeyRef:
name: query-regression-runner-local-env
key: HTTPS_PROXY
optional: true
- name: NO_PROXY
valueFrom:
configMapKeyRef:
name: query-regression-runner-local-env
key: NO_PROXY
optional: true
- name: UV_CACHE_DIR
value: /home/runner/.cargo/uv-cache
- name: RUSTUP_HOME
value: /opt/rustup
- name: RUSTUP_TOOLCHAIN
value: nightly-2026-03-21
- name: RUSTUP_AUTO_INSTALL
value: "0"
- name: CARGO_TARGET_DIR
value: /home/runner/query-regression-target
- name: QUERY_REGRESSION_CACHE_META
value: /home/runner/query-regression-cache-meta
- name: RUSTC_WRAPPER
value: /usr/local/bin/sccache
- name: SCCACHE_DIR
value: /home/runner/.cache/sccache
- name: SCCACHE_CACHE_SIZE
value: 40G
- name: CARGO_INCREMENTAL
value: "0"
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
resources:
requests:
cpu: "6"
memory: 24Gi
ephemeral-storage: 80Gi
limits:
cpu: "8"
memory: 24Gi
ephemeral-storage: 80Gi
volumeMounts:
- name: cargo-home
mountPath: /home/runner/.cargo
- name: build-cache
mountPath: /home/runner/.cargo/registry
subPath: cargo-registry-v1
- name: build-cache
mountPath: /home/runner/.cargo/git
subPath: cargo-git-v1
- name: build-cache
mountPath: /home/runner/query-regression-target
subPath: query-regression-target-v1
- name: build-cache
mountPath: /home/runner/query-regression-cache-meta
subPath: meta-v1
- name: build-cache
mountPath: /home/runner/.cache/sccache
subPath: sccache-v1
.github/runner-scale-sets/query-regression/values-paused.yaml
-2
View File
@@ -1,2 +0,0 @@
minRunners: 0
maxRunners: 0
.github/scripts/aliyun-ecs-runner-provision.py
+485
View File
@@ -0,0 +1,485 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is the first `.github/scripts/` user of PEP 723 inline script metadata:
# `uv run` installs the pinned Aliyun SDK before executing. The SDK import stays
# lazy (inside `make_ecs_client`) so unit tests can import this module with a
# plain stdlib interpreter.
# /// script
# requires-python = ">=3.10"
# dependencies = [
# # Pin exactly after the first live run against the Aliyun API.
# "alibabacloud_ecs20140526>=4.1.0,<6",
# "alibabacloud_tea_openapi>=0.3.12,<1",
# ]
# ///
"""Provision an ephemeral Aliyun ECS query-regression runner.
Creates one pay-as-you-go ECS instance from the query-regression custom image
and waits until the instance registers itself as an ephemeral GitHub Actions
runner with a per-run label. Build caches live on the instance system disk
and disappear with the VM. The instance id is written to job outputs as soon
as the VM exists so the teardown job is the single DeleteInstance caller (a
timeout here used to delete and then teardown deleted again). On runner
online timeout the console is dumped and the job fails; teardown releases
the instance.
The Aliyun credentials are used only in this control-plane job (ubuntu-latest)
and are never passed to the ECS instance; the instance receives only a
short-lived GitHub runner registration token via user data.
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
RUNNER_NAME_PREFIX = "qreg-ecs"
RUNNER_LABEL_PREFIX = "query-regression-ecs"
MANAGED_BY_TAG_KEY = "managed-by"
MANAGED_BY_TAG_VALUE = "query-regression-ci"
RUN_TAG_KEY = "query-regression-run-id"
# Runner cache paths on the instance system disk. They are created empty
# every provision and deleted with the VM.
CACHE_PATHS = (
"/home/runner/.cargo/registry",
"/home/runner/.cargo/git",
"/home/runner/query-regression-target",
"/home/runner/query-regression-cache-meta",
"/home/runner/.cache/sccache",
)
RUNNER_ONLINE_TIMEOUT_SECONDS = 10 * 60
POLL_INTERVAL_SECONDS = 5
# Nightly thin-LTO of greptime peaks above ecs.c9i.2xlarge's 16 GiB. A swap
# file plus masking systemd-oomd lets the kernel reclaim rustc pages instead
# of SIGTERM-ing the runner service cgroup (GitHub then reports the step as
# "The operation was canceled"). Sized to match that 16 GiB instance; a
# 32 GiB type still benefits as a safety net. Lives on the system disk next
# to the cold build caches.
SWAP_FILE = "/swapfile"
SWAP_SIZE_GIB = 16
@dataclass(frozen=True)
class ProvisionConfig:
region_id: str
vswitch_id: str
security_group_id: str
image_id: str
instance_type: str
repo: str
run_id: str
github_token: str
# Optional resource group; required when the RAM grant is scoped to one.
resource_group_id: str | None = None
# Runner identity inside the image; the workflow asserts the same values.
runner_uid: str = "1001"
runner_gid: str = "1001"
def runner_name_for_run(run_id: str) -> str:
return f"{RUNNER_NAME_PREFIX}-{run_id}"
def runner_label_for_run(run_id: str) -> str:
return f"{RUNNER_LABEL_PREFIX}-{run_id}"
def render_user_data(
runner_name: str,
runner_label: str,
runner_token: str,
repo: str,
runner_uid: str = "1001",
runner_gid: str = "1001",
) -> str:
"""Render the cloud-init shell script for the runner instance."""
destinations = " ".join(f'"{dst}"' for dst in CACHE_PATHS)
cache_setup = f"""# Caches live on the system disk, are deleted with the instance, and every
# run compiles cold. Within-run reuse (base warming candidate via the shared
# target dir and sccache) still applies.
mkdir -p {destinations}
chown -R {runner_uid}:{runner_gid} /home/runner"""
swap_setup = f"""# Mask systemd-oomd before enabling swap: Ubuntu 24.04 kills the whole
# service cgroup on PSI pressure, and swap thrashing looks like pressure.
systemctl disable --now systemd-oomd.socket systemd-oomd.service || true
systemctl mask systemd-oomd.socket systemd-oomd.service || true
# apt-daily-upgrade / unattended-upgrades can `systemctl restart` services
# whose libraries were updated. That SIGTERM-s the runner mid-job; GitHub
# records UserCancelled (job still valid, child exit 143). These VMs live
# ~1h; security updates belong in the image, not at job time.
systemctl disable --now unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer apt-daily.service apt-daily-upgrade.service || true
systemctl mask unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer apt-daily.service apt-daily-upgrade.service || true
systemctl stop unattended-upgrades.service || true
killall -9 unattended-upgr || true
cat > /etc/apt/apt.conf.d/99disable-auto-updates <<'APTEOF'
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APTEOF
if [[ ! -f "{SWAP_FILE}" ]]; then
fallocate --length {SWAP_SIZE_GIB}G "{SWAP_FILE}"
chmod 600 "{SWAP_FILE}"
mkswap "{SWAP_FILE}"
fi
swapon "{SWAP_FILE}"
sysctl --write vm.swappiness=10
swapon --show
free --human"""
return f"""#!/bin/bash
set -euo pipefail
{cache_setup}
{swap_setup}
cat > /etc/ephemeral-github-runner.env <<'ENVEOF'
RUNNER_NAME={runner_name}
RUNNER_LABELS={runner_label}
RUNNER_TOKEN={runner_token}
REPO_URL=https://github.com/{repo}
# The container image baked /opt/cargo/bin into PATH via Dockerfile ENV; on
# the host nothing inherits that, so publish it through the unit's
# EnvironmentFile: runner job processes inherit the runner's environment.
PATH=/opt/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENVEOF
# Stream the runner service's own logs (config output, job lifecycle, crash
# messages) to the serial console, next to the kernel's OOM-killer records.
# Teardown dumps the console tail before deleting the instance, so the
# witness of what killed the runner survives the machine.
mkdir -p /etc/systemd/system/ephemeral-github-runner.service.d
cat > /etc/systemd/system/ephemeral-github-runner.service.d/console.conf <<'CONFEOF'
[Service]
StandardOutput=journal+console
StandardError=journal+console
CONFEOF
# systemd's DefaultOOMPolicy=stop SIGTERM-s the whole unit when *any*
# process in the cgroup is OOM-killed (typically rustc during thin LTO).
# That is the "Session terminated, killing shell..." / job Canceled path:
# runuser dies, GitHub deletes the ephemeral registration, always() steps
# never run. continue keeps the runner up so cargo can report signal 9.
cat > /etc/systemd/system/ephemeral-github-runner.service.d/oom.conf <<'CONFEOF'
[Service]
OOMPolicy=continue
CONFEOF
systemctl daemon-reload
# restart (not start): the image enables this unit, so it may already
# have come up without the drop-ins if it raced cloud-init. restart
# applies OOMPolicy=continue to the running cgroup. --no-block matters:
# newer image units carry After=cloud-final.service, and this script IS
# cloud-final — a blocking restart would deadlock until TimeoutStartSec.
systemctl restart --no-block ephemeral-github-runner.service
"""
def encode_user_data(script: str) -> str:
return base64.b64encode(script.encode("utf-8")).decode("ascii")
def github_api(token: str, method: str, path: str, body: dict | None = None) -> dict:
data = json.dumps(body).encode("utf-8") if body is not None else None
request = urllib.request.Request(
f"https://api.github.com{path}",
data=data,
method=method,
headers={
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {token}",
"X-GitHub-Api-Version": "2022-11-28",
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
# GitHub's error body says exactly why (e.g. "Must have admin rights to
# Repository" for a PAT without the required scope); surface it instead
# of a bare "HTTP Error 403".
body = error.read().decode("utf-8", "replace")
raise SystemExit(
f"GitHub API {method} {path} failed: HTTP {error.code}: {body}\n"
"The token comes from the GH_PERSONAL_ACCESS_TOKEN secret; it needs "
"'repo' scope (classic PAT) or 'Administration: write' on the "
"repository (fine-grained PAT)."
) from error
def create_registration_token(github_token: str, repo: str) -> str:
response = github_api(
github_token,
"POST",
f"/repos/{repo}/actions/runners/registration-token",
body={},
)
return response["token"]
def find_runner_by_name(github_token: str, repo: str, name: str) -> dict | None:
page = 1
while True:
response = github_api(
github_token,
"GET",
f"/repos/{repo}/actions/runners?per_page=100&page={page}",
)
runners = response.get("runners", [])
for runner in runners:
if runner.get("name") == name:
return runner
if len(runners) < 100:
return None
page += 1
def make_ecs_client(config: ProvisionConfig):
from alibabacloud_ecs20140526.client import Client as EcsClient
from alibabacloud_tea_openapi.models import Config as OpenApiConfig
access_key_id = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID", "")
access_key_secret = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET", "")
if not access_key_id or not access_key_secret:
raise SystemExit(
"ALIBABA_CLOUD_ACCESS_KEY_ID/SECRET are empty; in CI they come from the "
"ALICLOUD_ECS_ACCESS_KEY_ID/SECRET repository secrets (a missing or "
"misnamed secret expands to an empty string)."
)
return EcsClient(
OpenApiConfig(
access_key_id=access_key_id,
access_key_secret=access_key_secret,
region_id=config.region_id,
endpoint=f"ecs.{config.region_id}.aliyuncs.com",
)
)
def wait_for_instance_status(client, region_id: str, instance_id: str, wanted: str, deadline: float) -> None:
from alibabacloud_ecs20140526 import models as ecs_models
while time.monotonic() < deadline:
response = client.describe_instances(
ecs_models.DescribeInstancesRequest(
region_id=region_id, instance_ids=json.dumps([instance_id])
)
)
instances = response.body.instances.instance
if instances and instances[0].status == wanted:
return
time.sleep(POLL_INTERVAL_SECONDS)
raise TimeoutError(f"Instance {instance_id} did not reach status {wanted} in time")
def fetch_console_output(client, region_id: str, instance_id: str) -> str:
from alibabacloud_ecs20140526 import models as ecs_models
response = client.get_instance_console_output(
ecs_models.GetInstanceConsoleOutputRequest(region_id=region_id, instance_id=instance_id)
)
return base64.b64decode(response.body.console_output or "").decode("utf-8", "replace")
class ConsoleTailer:
"""Incrementally prints new serial-console lines so cloud-init progress is
visible in the CI log while waiting, not only after a failure dump."""
def __init__(self, client, region_id: str, instance_id: str) -> None:
self.client = client
self.region_id = region_id
self.instance_id = instance_id
self.printed_lines = 0
def poll(self) -> None:
try:
lines = fetch_console_output(self.client, self.region_id, self.instance_id).splitlines()
except Exception as error: # noqa: BLE001
print(f"[console] unable to fetch console output: {error}", flush=True)
return
# The API returns a bounded tail; if it ever shrinks, restart from the
# beginning of the new buffer rather than skipping lines.
if len(lines) < self.printed_lines:
self.printed_lines = 0
for line in lines[self.printed_lines :]:
print(f"[console] {line}", flush=True)
self.printed_lines = len(lines)
def dump_console_output(client, region_id: str, instance_id: str) -> None:
try:
output = fetch_console_output(client, region_id, instance_id)
except Exception as error: # noqa: BLE001
print(f"Unable to fetch console output for {instance_id}: {error}", flush=True)
return
tail = "\n".join(output.splitlines()[-80:])
print(f"::group::Console output tail for {instance_id}\n{tail}\n::endgroup::", flush=True)
summary_path = os.environ.get("GITHUB_STEP_SUMMARY")
if summary_path:
with open(summary_path, "a", encoding="utf-8") as summary:
summary.write(f"<details><summary>ECS console output ({instance_id})</summary>\n\n```\n")
summary.write(tail)
summary.write("\n```\n</details>\n")
def append_github_output(name: str, value: str) -> None:
output_path = os.environ.get("GITHUB_OUTPUT")
if output_path:
with open(output_path, "a", encoding="utf-8") as output:
output.write(f"{name}={value}\n")
def run_instance(client, config: ProvisionConfig, user_data: str) -> str:
from alibabacloud_ecs20140526 import models as ecs_models
request = ecs_models.RunInstancesRequest(
region_id=config.region_id,
image_id=config.image_id,
resource_group_id=config.resource_group_id,
instance_type=config.instance_type,
v_switch_id=config.vswitch_id,
security_group_id=config.security_group_id,
instance_name=runner_name_for_run(config.run_id),
host_name=runner_name_for_run(config.run_id),
description=f"Ephemeral query-regression runner for run {config.run_id}",
amount=1,
instance_charge_type="PostPaid",
spot_strategy="NoSpot",
internet_charge_type="PayByTraffic",
internet_max_bandwidth_out=100,
# Holds the image (~12G), the 16G swapfile, the source checkout, the
# base+candidate cluster data homes, and cold build caches. Deleted
# with the instance.
system_disk=ecs_models.RunInstancesRequestSystemDisk(category="cloud_essd", size="40"),
user_data=user_data,
tag=[
ecs_models.RunInstancesRequestTag(key=MANAGED_BY_TAG_KEY, value=MANAGED_BY_TAG_VALUE),
ecs_models.RunInstancesRequestTag(key=RUN_TAG_KEY, value=config.run_id),
],
)
response = client.run_instances(request)
instance_id = response.body.instance_id_sets.instance_id_set[0]
return instance_id
def provision(config: ProvisionConfig) -> int:
client = make_ecs_client(config)
runner_name = runner_name_for_run(config.run_id)
runner_label = runner_label_for_run(config.run_id)
registration_token = create_registration_token(config.github_token, config.repo)
user_data = encode_user_data(
render_user_data(
runner_name,
runner_label,
registration_token,
config.repo,
config.runner_uid,
config.runner_gid,
)
)
print(f"::group::Run instance {runner_name}", flush=True)
instance_id = run_instance(client, config, user_data)
print(f"Instance id: {instance_id}", flush=True)
# Teardown is the only DeleteInstance caller. Write outputs immediately
# so a later status/online failure still releases this VM.
append_github_output("label", runner_label)
append_github_output("instance_id", instance_id)
append_github_output("runner_name", runner_name)
wait_for_instance_status(client, config.region_id, instance_id, "Running", time.monotonic() + 5 * 60)
print("::endgroup::", flush=True)
print("::group::Wait for runner online", flush=True)
deadline = time.monotonic() + RUNNER_ONLINE_TIMEOUT_SECONDS
tailer = ConsoleTailer(client, config.region_id, instance_id)
last_console_poll = 0.0
while time.monotonic() < deadline:
runner = find_runner_by_name(config.github_token, config.repo, runner_name)
if runner is not None:
status = runner.get("status")
print(f"Runner {runner_name} status: {status}", flush=True)
if status == "online":
print("::endgroup::", flush=True)
print(f"Runner {runner_name} is online with label {runner_label}", flush=True)
return 0
# The serial console lags the guest by a minute or so; 30s polling is
# enough to follow cloud-init without tripping API throttling.
if time.monotonic() - last_console_poll >= 30:
tailer.poll()
last_console_poll = time.monotonic()
time.sleep(POLL_INTERVAL_SECONDS)
print("::endgroup::", flush=True)
print(
f"Runner {runner_name} did not come online in time; "
"leaving instance for the teardown job to delete",
flush=True,
)
dump_console_output(client, config.region_id, instance_id)
return 1
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID"))
parser.add_argument("--vswitch-id", default=os.environ.get("ALIYUN_ECS_VSWITCH_ID"))
parser.add_argument("--security-group-id", default=os.environ.get("ALIYUN_ECS_SECURITY_GROUP_ID"))
parser.add_argument("--image-id", default=os.environ.get("QUERY_REGRESSION_ECS_IMAGE_ID"))
parser.add_argument("--instance-type", default=os.environ.get("ALIYUN_ECS_INSTANCE_TYPE"))
parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY"))
parser.add_argument("--run-id", default=os.environ.get("GITHUB_RUN_ID"))
parser.add_argument("--github-token", default=os.environ.get("GH_PERSONAL_ACCESS_TOKEN"))
parser.add_argument("--runner-uid", default=os.environ.get("QUERY_REGRESSION_RUNNER_UID", "1001"))
parser.add_argument("--runner-gid", default=os.environ.get("QUERY_REGRESSION_RUNNER_GID", "1001"))
parser.add_argument("--resource-group-id", default=os.environ.get("ALIYUN_ECS_RESOURCE_GROUP_ID"))
args = parser.parse_args()
missing = [
name
for name, value in vars(args).items()
if name not in ("resource_group_id",)
and (value is None or (isinstance(value, str) and not value))
]
if missing:
flags = ", ".join(f"--{name.replace('_', '-')}" for name in missing)
raise SystemExit(f"Missing required configuration: {flags} (or their env defaults)")
config = ProvisionConfig(
region_id=args.region_id,
vswitch_id=args.vswitch_id,
security_group_id=args.security_group_id,
image_id=args.image_id,
instance_type=args.instance_type,
repo=args.repo,
run_id=args.run_id,
github_token=args.github_token,
resource_group_id=args.resource_group_id or None,
runner_uid=args.runner_uid,
runner_gid=args.runner_gid,
)
return provision(config)
if __name__ == "__main__":
raise SystemExit(main())
.github/scripts/aliyun-ecs-runner-teardown.py
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# PEP 723 inline metadata (see aliyun-ecs-runner-provision.py for the
# convention). The SDK import stays lazy so unit tests run on a plain
# stdlib interpreter.
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "alibabacloud_ecs20140526>=4.1.0,<6",
# "alibabacloud_tea_openapi>=0.3.12,<1",
# ]
# ///
"""Tear down ephemeral Aliyun ECS query-regression runners.
Two modes:
- Targeted (per workflow run): delete one instance by id and deregister its
runner by name. Both steps are idempotent and best-effort; a missing
instance or runner is not an error.
- Sweep (scheduled janitor): delete every instance tagged as managed by
query-regression CI whose creation time is older than the given TTL, and
deregister the matching runners. This is the safety net for runs whose
teardown job never executed.
"""
from __future__ import annotations
import argparse
import importlib.util
import os
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
# Reuse the GitHub API client, runner lookup, and tag constants from the
# provision script.
_PROVISION_SPEC = importlib.util.spec_from_file_location(
"aliyun_ecs_runner_provision",
Path(__file__).resolve().parent / "aliyun-ecs-runner-provision.py",
)
assert _PROVISION_SPEC is not None and _PROVISION_SPEC.loader is not None
provision = importlib.util.module_from_spec(_PROVISION_SPEC)
sys.modules[_PROVISION_SPEC.name] = provision
_PROVISION_SPEC.loader.exec_module(provision)
SWEEP_TTL = timedelta(hours=4)
# ECS DescribeInstances creation_time is ISO8601 UTC, e.g. "2026-08-17T02:13Z".
CREATION_TIME_FORMATS = ("%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%dT%H:%MZ")
def parse_creation_time(value: str) -> datetime:
for fmt in CREATION_TIME_FORMATS:
try:
return datetime.strptime(value, fmt).replace(tzinfo=timezone.utc)
except ValueError:
continue
raise ValueError(f"Unparsable ECS creation time: {value}")
def expired_instance_names(
instances: list[tuple[str, str, str]], now: datetime, ttl: timedelta
) -> list[tuple[str, str]]:
"""Pick (instance_id, instance_name) pairs whose creation is older than ttl.
`instances` items are (instance_id, instance_name, creation_time).
"""
expired = []
for instance_id, instance_name, creation_time in instances:
age = now - parse_creation_time(creation_time)
if age >= ttl:
expired.append((instance_id, instance_name))
return expired
def make_ecs_client(region_id: str):
from alibabacloud_ecs20140526.client import Client as EcsClient
from alibabacloud_tea_openapi.models import Config as OpenApiConfig
access_key_id = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID", "")
access_key_secret = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET", "")
if not access_key_id or not access_key_secret:
raise SystemExit(
"ALIBABA_CLOUD_ACCESS_KEY_ID/SECRET are empty; in CI they come from the "
"ALICLOUD_ECS_ACCESS_KEY_ID/SECRET repository secrets (a missing or "
"misnamed secret expands to an empty string)."
)
return EcsClient(
OpenApiConfig(
access_key_id=access_key_id,
access_key_secret=access_key_secret,
region_id=region_id,
endpoint=f"ecs.{region_id}.aliyuncs.com",
)
)
def delete_instance(client, instance_id: str, region_id: str | None = None) -> bool:
import time
from alibabacloud_ecs20140526 import models as ecs_models
# Export the serial console before destroying the evidence: a cancelled
# run still runs this teardown job, and the console tail (cloud-init,
# runner service logs, kernel OOM records) is the only witness of what
# happened on the machine.
if region_id is not None:
provision.dump_console_output(client, region_id, instance_id)
# DeleteInstance rejects instances that are still Initializing (or in
# another transitional status). Teardown runs right after a cancelled
# run, when the instance may be only seconds old, so retry the transient
# status errors for a few minutes instead of leaking the instance.
deadline = time.monotonic() + 5 * 60
while True:
try:
client.delete_instance(ecs_models.DeleteInstanceRequest(instance_id=instance_id, force=True))
print(f"Deleted instance {instance_id}", flush=True)
return True
except Exception as error: # noqa: BLE001
message = str(error)
if "InvalidInstanceId.NotFound" in message:
print(f"Instance {instance_id} already gone", flush=True)
return True
if "IncorrectInstanceStatus" in message and time.monotonic() < deadline:
print(f"Instance {instance_id} is in a transitional status; retrying delete", flush=True)
time.sleep(15)
continue
print(f"Failed to delete instance {instance_id}: {error}", flush=True)
return False
def deregister_runner(token: str, repo: str, runner_name: str) -> bool:
runner = provision.find_runner_by_name(token, repo, runner_name)
if runner is None:
print(f"Runner {runner_name} is not registered", flush=True)
return True
try:
provision.github_api(token, "DELETE", f"/repos/{repo}/actions/runners/{runner['id']}")
print(f"Deregistered runner {runner_name} (id {runner['id']})", flush=True)
return True
except Exception as error: # noqa: BLE001
print(f"Failed to deregister runner {runner_name}: {error}", flush=True)
return False
def list_managed_instances(client, region_id: str) -> list[tuple[str, str, str]]:
from alibabacloud_ecs20140526 import models as ecs_models
result: list[tuple[str, str, str]] = []
next_token = None
while True:
request = ecs_models.DescribeInstancesRequest(
region_id=region_id,
tag=[
ecs_models.DescribeInstancesRequestTag(
key=provision.MANAGED_BY_TAG_KEY, value=provision.MANAGED_BY_TAG_VALUE
)
],
max_results=100,
next_token=next_token,
)
response = client.describe_instances(request)
for instance in response.body.instances.instance:
result.append((instance.instance_id, instance.instance_name, instance.creation_time))
next_token = response.body.next_token
if not next_token:
return result
def sweep(client, region_id: str, repo: str, github_token: str, ttl: timedelta) -> int:
instances = list_managed_instances(client, region_id)
print(f"Found {len(instances)} managed instance(s) in {region_id}", flush=True)
expired = expired_instance_names(instances, datetime.now(timezone.utc), ttl)
ok = True
for instance_id, instance_name in expired:
print(f"Instance {instance_id} ({instance_name}) exceeds TTL {ttl}; deleting", flush=True)
# Runner names mirror instance names by construction in the provision
# script (both are qreg-ecs-<run_id>).
ok &= delete_instance(client, instance_id, region_id)
ok &= deregister_runner(github_token, repo, instance_name)
return 0 if ok else 1
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID"))
parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY"))
parser.add_argument("--github-token", default=os.environ.get("GH_PERSONAL_ACCESS_TOKEN"))
parser.add_argument("--instance-id", default=os.environ.get("QUERY_REGRESSION_ECS_INSTANCE_ID"))
parser.add_argument("--runner-name", default=os.environ.get("QUERY_REGRESSION_ECS_RUNNER_NAME"))
parser.add_argument(
"--sweep",
action="store_true",
help="Janitor mode: delete all managed instances older than --sweep-ttl-hours.",
)
parser.add_argument("--sweep-ttl-hours", type=float, default=SWEEP_TTL.total_seconds() / 3600)
args = parser.parse_args()
for name in ("region_id", "repo", "github_token"):
if not getattr(args, name):
raise SystemExit(f"Missing required configuration: --{name.replace('_', '-')}")
client = make_ecs_client(args.region_id)
if args.sweep:
return sweep(
client, args.region_id, args.repo, args.github_token, timedelta(hours=args.sweep_ttl_hours)
)
ok = True
if args.instance_id:
ok &= delete_instance(client, args.instance_id, args.region_id)
else:
print("No instance id given; skipping instance deletion", flush=True)
if args.runner_name:
ok &= deregister_runner(args.github_token, args.repo, args.runner_name)
else:
print("No runner name given; skipping runner deregistration", flush=True)
return 0 if ok else 1
if __name__ == "__main__":
raise SystemExit(main())
.github/scripts/pr-open-limit.js
+144
View File
@@ -0,0 +1,144 @@
// Warns when a repository member has too many open pull requests.
//
// Review capacity is the bottleneck, so authors are expected to land or close
// existing work before opening more. Drafts are not counted and never warned
// about: they are not asking for review yet.
// This is advisory only — it never closes a pull request or fails the job.
import { appendFileSync } from "node:fs";
const MARKER = "<!-- pr-open-limit -->";
// GITHUB_TOKEN can only edit comments authored by the Actions bot itself.
const COMMENT_AUTHOR = "github-actions[bot]";
// Repository permissions that mark someone as part of the team.
const TEAM_PERMISSIONS = ["admin", "maintain", "write"];
// Keep the comment readable for authors who are far over the limit.
const MAX_LISTED_PRS = 10;
function summary(text) {
if (process.env.GITHUB_STEP_SUMMARY) {
appendFileSync(process.env.GITHUB_STEP_SUMMARY, `${text}\n`);
}
console.log(text);
}
// `author_association` cannot be used here: it is computed from what the caller
// is allowed to see, so GITHUB_TOKEN reports a private organization member as
// CONTRIBUTOR. Repository permission is viewer-independent.
//
// On error this returns true rather than false. A token that cannot read
// permissions would otherwise make the whole check silently pass everyone.
async function isTeamMember(octokit, owner, repo, username) {
try {
const { data } = await octokit.repos.getCollaboratorPermissionLevel({
owner,
repo,
username,
});
// Never log the level itself: job logs are public.
return TEAM_PERMISSIONS.includes(data.permission);
} catch (error) {
console.log(
`Cannot read repository permission for \`${username}\` (HTTP ${error.status}); applying the limit anyway.`
);
return true;
}
}
function buildComment(author, total, limit, openPrs) {
const listed = openPrs
.slice(0, MAX_LISTED_PRS)
.map((pr) => `- #${pr.number} ${pr.title}`);
const hidden = openPrs.length - listed.length;
const list = hidden > 0 ? `${listed.join("\n")}\n- ...and ${hidden} more` : listed.join("\n");
return `${MARKER}
> [!WARNING]
> @${author} has **${total}** open pull requests in this repository, over the limit of **${limit}**.
Review is the scarcest resource here. Please land or close some of these before
pushing this one forward:
${list}
This check is advisory for now and blocks nothing.`;
}
async function upsertComment(octokit, params, body) {
const comments = await octokit.paginate(octokit.issues.listComments, {
...params,
per_page: 100,
});
const existing = comments.find(
(comment) =>
comment.user?.login === COMMENT_AUTHOR && comment.body?.includes(MARKER)
);
if (existing) {
await octokit.issues.updateComment({
owner: params.owner,
repo: params.repo,
comment_id: existing.id,
body,
});
} else {
await octokit.issues.createComment({ ...params, body });
}
}
(async () => {
const { Octokit } = await import("@octokit/rest");
const [owner, repo] = process.env.GITHUB_REPOSITORY.split("/");
const prNumber = Number(process.env.PR_NUMBER);
const author = process.env.PR_AUTHOR;
const rawLimit = Number(process.env.MAX_OPEN_PRS);
const limit = Number.isInteger(rawLimit) && rawLimit >= 0 ? rawLimit : 5;
if (author.endsWith("[bot]")) {
summary(`Skipping bot author \`${author}\`.`);
return;
}
if (process.env.PR_IS_DRAFT === "true") {
summary(`Skipping #${prNumber}: draft.`);
return;
}
const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
if (!(await isTeamMember(octokit, owner, repo, author))) {
summary(`Skipping \`${author}\`: not a team member.`);
return;
}
const allOpen = await octokit.paginate(octokit.pulls.list, {
owner,
repo,
state: "open",
sort: "created",
direction: "asc",
per_page: 100,
});
// The pull request that triggered this run is already open, so exclude it and
// report the total separately.
const others = allOpen.filter(
(pr) => pr.user?.login === author && pr.number !== prNumber && !pr.draft
);
const total = others.length + 1;
if (others.length < limit) {
summary(`\`${author}\` has ${total} open pull requests (limit ${limit}). OK.`);
return;
}
summary(`\`${author}\` has ${total} open pull requests, over the limit of ${limit}.`);
await upsertComment(
octokit,
{ owner, repo, issue_number: prNumber },
buildComment(author, total, limit, others)
);
})().catch((error) => {
console.error(error);
process.exit(1);
});
.github/scripts/query-regression-comment.cjs
+48 -5
View File
@@ -92,7 +92,7 @@ function thresholdStatus(thresholds, query) {
const hits = (Array.isArray(thresholds) ? thresholds : [])
.filter(item => query === undefined || (hasScopedQuery(item) && String(item.query) === query))
.map(formatThreshold);
return hits.length > 0 ? hits.join(', ') : 'N/A';
return hits.length > 0 ? hits.join('; ') : 'N/A';
}
function hasScopedQuery(threshold) {
@@ -115,7 +115,7 @@ function formatThreshold(threshold) {
return `${name}${scope.length > 0 ? ` [${scope.join(', ')}]` : ''}: ${status}${reason}`;
}
function syntheticThresholdStatus(thresholds, measurementNames) {
function classifyThresholds(thresholds, measurementNames) {
const unscoped = [];
const unmatched = new Map();
for (const threshold of Array.isArray(thresholds) ? thresholds : []) {
@@ -130,7 +130,11 @@ function syntheticThresholdStatus(thresholds, measurementNames) {
unmatched.set(query, entries);
}
}
return { unscoped, unmatched };
}
function syntheticThresholdStatus(thresholds, measurementNames) {
const { unscoped, unmatched } = classifyThresholds(thresholds, measurementNames);
const parts = [];
if (unscoped.length > 0) {
parts.push(`case/storage threshold: ${thresholdStatus(unscoped)}`);
@@ -141,6 +145,21 @@ function syntheticThresholdStatus(thresholds, measurementNames) {
return parts.length > 0 ? parts.join('; ') : 'N/A';
}
// Case-level (query-unscoped) thresholds formatted for the collapsible details
// block below the summary table: individual items, `; `-separated, without the
// `case/storage threshold:` label (the section heading already says that).
function syntheticThresholdDetail(thresholds, measurementNames) {
const { unscoped, unmatched } = classifyThresholds(thresholds, measurementNames);
const items = [];
if (unscoped.length > 0) {
items.push(thresholdStatus(unscoped));
}
for (const query of Array.from(unmatched.keys()).sort()) {
items.push(`unmatched query ${query}: ${thresholdStatus(unmatched.get(query))}`);
}
return items.length > 0 ? items.join('; ') : 'N/A';
}
function joinDetails(...details) {
const present = details.filter(detail => detail && detail !== 'N/A');
return present.length > 0 ? present.join('; ') : 'N/A';
@@ -229,7 +248,7 @@ function collectReportRows(report, reportPath) {
),
};
});
const syntheticThresholds = syntheticThresholdStatus(thresholds, measurementNames);
const syntheticThresholds = syntheticThresholdDetail(thresholds, measurementNames);
if (syntheticThresholds !== 'N/A') {
rows.push({
caseName: name,
@@ -239,22 +258,46 @@ function collectReportRows(report, reportPath) {
candidateMedian: 'N/A',
regression: 'N/A',
threshold: syntheticThresholds,
kind: 'case-thresholds',
});
}
return rows;
}
function renderThresholdDetails(rows) {
const entries = rows.filter(row => row.kind === 'case-thresholds');
if (entries.length === 0) {
return '';
}
// The <details>/<summary> tags are emitted literally; only the entry content
// goes through text() (so `[ ] ( ) |` etc. stay escaped while the tags render).
const lines = [
'<details><summary>Case / storage thresholds</summary>',
'',
];
for (const row of entries) {
lines.push(`- ${text(row.caseName)}: ${text(row.threshold)}`);
}
lines.push('</details>');
return lines.join('\n');
}
function renderSummaryTable(rows) {
const lines = [
'| Case | Query | Case status | Base median ms | Candidate median ms | Regression | Threshold |',
'| --- | --- | --- | ---: | ---: | ---: | --- |',
];
for (const row of rows) {
if (row.kind === 'case-thresholds') {
continue;
}
lines.push(
`| ${text(row.caseName)} | ${text(row.query)} | ${statusEmoji(row.status)} \`${text(row.status)}\` | ${text(row.baseMedian)} | ${text(row.candidateMedian)} | ${text(row.regression)} | ${text(row.threshold)} |`
`| ${text(row.caseName)} | ${text(row.query)} | ${statusEmoji(row.status)} ${text(row.status)} | ${text(row.baseMedian)} | ${text(row.candidateMedian)} | ${text(row.regression)} | ${text(row.threshold)} |`
);
}
return lines.join('\n');
const details = renderThresholdDetails(rows);
const table = lines.join('\n');
return details === '' ? table : `${table}\n\n${details}`;
}
module.exports = async function validateQueryRegressionComment({ github, context, core }) {
.github/scripts/query-regression-comment.test.cjs
+43 -4
View File
@@ -187,11 +187,11 @@ test('sorts the base and candidate query union and aggregates scoped thresholds'
assert.equal(rows[0].threshold, 'base measurement missing');
assert.equal(
rows[1].threshold,
'candidate measurement missing; p95 [target=base]: warn, absolute [target=candidate, encoding=plain]: pass'
'candidate measurement missing; p95 [target=base]: warn; absolute [target=candidate, encoding=plain]: pass'
);
});
test('preserves unscoped and unmatched thresholds in a synthetic N/A row', () => {
test('keeps unscoped and unmatched thresholds in a case-thresholds detail row', () => {
const rows = collectReportRows(report('thresholds', {
base: [{ name: 'measured', latency_ms_median: 10 }],
candidate: [{ name: 'measured', latency_ms_median: 11 }],
@@ -214,9 +214,10 @@ test('preserves unscoped and unmatched thresholds in a synthetic N/A row', () =>
assert.equal(rows[0].query, 'measured');
assert.equal(rows[0].threshold, 'query limit [target=base]: passed');
assert.equal(rows[1].query, 'N/A');
assert.equal(rows[1].kind, 'case-thresholds');
assert.equal(
rows[1].threshold,
'case/storage threshold: min_files [target=base]: passed, min_files [target=candidate]: failed, encoding limit [target=candidate, encoding=plain]: failed; unmatched query not-measured: orphaned limit [target=base, encoding=json]: failed (reason: measurement unavailable)'
'min_files [target=base]: passed; min_files [target=candidate]: failed; encoding limit [target=candidate, encoding=plain]: failed; unmatched query not-measured: orphaned limit [target=base, encoding=json]: failed (reason: measurement unavailable)'
);
});
@@ -239,12 +240,50 @@ test('keeps unscoped thresholds out of undefined and null query rows', () => {
assert.equal(byQuery.get('undefined').threshold, 'N/A');
assert.equal(byQuery.get('null').threshold, 'N/A');
assert.equal(rows.filter(row => row.query === 'N/A').length, 1);
assert.equal(byQuery.get('N/A').kind, 'case-thresholds');
assert.equal(
byQuery.get('N/A').threshold,
'case/storage threshold: min_files [target=base]: passed, min_files [target=candidate]: failed'
'min_files [target=base]: passed; min_files [target=candidate]: failed'
);
});
test('renders case/storage thresholds in a collapsible details block below the table', () => {
const rows = [
...collectReportRows(report('first', {
base: [{ name: 'q1', latency_ms_median: 10 }],
candidate: [{ name: 'q1', latency_ms_median: 11 }],
}, [
{ query: 'q1', threshold: 'query limit', target: 'base', status: 'passed' },
{ threshold: 'min_files', target: 'base', status: 'passed' },
{ threshold: 'min_files', target: 'candidate', status: 'failed' },
]), '/reports/first/query-regression-report.json'),
...collectReportRows({
case: { name: 'broken' },
status: 'failed',
error: 'connection refused',
}, '/reports/broken/query-regression-report.json'),
];
const rendered = renderSummaryTable(rows);
const detailsIndex = rendered.indexOf('<details>');
const table = rendered.slice(0, detailsIndex);
const details = rendered.slice(detailsIndex);
// Query rows and abnormal N/A rows stay in the main table.
assert.match(table, /\| first \| q1 \| ✅ ok \|/);
assert.match(table, /\| broken \| N\/A \| ❌ failed \|/);
// Storage thresholds are not main-table rows.
assert.doesNotMatch(table, /^\| first \| N\/A \|/m);
assert.doesNotMatch(table, /min_files/);
// Status column is plain text without code-span backticks.
assert.doesNotMatch(table, /`/);
// Details block keeps its tags unescaped and its items '; '-separated.
assert.ok(details.startsWith('<details><summary>Case / storage thresholds</summary>'));
assert.ok(details.includes('- first: min_files \\[target=base\\]: passed; min_files \\[target=candidate\\]: failed'));
assert.ok(details.endsWith('</details>'));
assert.ok(!rendered.includes('&lt;details&gt;'));
});
test('escapes Markdown table content, including bare carriage returns', () => {
const table = renderSummaryTable([{
caseName: 'safe\r| injected |\n<!-- hidden -->@user <tag>',
.github/scripts/query-regression-nightly-refs.py
+270
View File
@@ -0,0 +1,270 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Resolve consecutive successful Nightly Build SHAs for query-regression.
Query regression builds both binaries from git; this script only picks the
head SHAs of two Nightly Build workflow runs (today vs the previous success).
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from typing import Any
NIGHTLY_WORKFLOW = "nightly-build.yml"
NIGHTLY_EVENTS = frozenset({"schedule", "workflow_dispatch"})
@dataclass(frozen=True)
class WorkflowRun:
id: int
head_sha: str
head_branch: str
html_url: str
created_at: str
conclusion: str
event: str
@dataclass(frozen=True)
class RefPair:
base: WorkflowRun | None
candidate: WorkflowRun | None
skip: bool
reason: str
def parse_run(payload: dict[str, Any]) -> WorkflowRun:
return WorkflowRun(
id=int(payload["id"]),
head_sha=str(payload["head_sha"]),
head_branch=str(payload.get("head_branch") or ""),
html_url=str(payload.get("html_url") or ""),
created_at=str(payload.get("created_at") or ""),
conclusion=str(payload.get("conclusion") or ""),
event=str(payload.get("event") or ""),
)
def github_get(token: str, path: str, query: dict[str, str] | None = None) -> dict[str, Any]:
encoded = f"?{urllib.parse.urlencode(query)}" if query else ""
request = urllib.request.Request(
f"https://api.github.com{path}{encoded}",
method="GET",
headers={
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {token}",
"X-GitHub-Api-Version": "2022-11-28",
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", "replace")
raise SystemExit(
f"GitHub API GET {path} failed: HTTP {error.code}: {body}"
) from error
def list_successful_nightly_runs(
token: str,
repo: str,
*,
branch: str | None = None,
per_page: int = 30,
) -> list[WorkflowRun]:
query = {
"status": "success",
"per_page": str(per_page),
}
if branch:
query["branch"] = branch
payload = github_get(
token,
f"/repos/{repo}/actions/workflows/{NIGHTLY_WORKFLOW}/runs",
query,
)
runs = [parse_run(item) for item in payload.get("workflow_runs") or []]
return [
run
for run in runs
if run.conclusion == "success" and run.event in NIGHTLY_EVENTS and run.head_sha
]
def fetch_run(token: str, repo: str, run_id: int) -> WorkflowRun:
payload = github_get(token, f"/repos/{repo}/actions/runs/{run_id}")
return parse_run(payload)
def select_base_and_candidate(
runs: list[WorkflowRun],
*,
candidate_run_id: int | None = None,
candidate: WorkflowRun | None = None,
) -> RefPair:
"""Pick candidate (newest / requested) and the previous successful run.
`runs` must be newest-first. Same-SHA consecutive nightlies are skipped
(no commits that day).
"""
if candidate is None:
if candidate_run_id is not None:
candidate = next((run for run in runs if run.id == candidate_run_id), None)
if candidate is None:
return RefPair(
None,
None,
True,
f"candidate nightly run {candidate_run_id} was not a successful "
f"{NIGHTLY_WORKFLOW} run",
)
elif runs:
candidate = runs[0]
else:
return RefPair(None, None, True, "no successful Nightly Build runs")
branch = candidate.head_branch
older = [
run
for run in runs
if run.id != candidate.id
and (not branch or not run.head_branch or run.head_branch == branch)
and (not candidate.created_at or run.created_at <= candidate.created_at)
]
if not older:
return RefPair(
None,
candidate,
True,
"no previous successful Nightly Build to compare against",
)
base = older[0]
if base.head_sha.lower() == candidate.head_sha.lower():
return RefPair(
base,
candidate,
True,
f"previous nightly SHA {base.head_sha} matches candidate; nothing to compare",
)
return RefPair(base, candidate, False, "")
def _override_run(sha: str) -> WorkflowRun:
return WorkflowRun(
id=0,
head_sha=sha,
head_branch="",
html_url="",
created_at="",
conclusion="success",
event="workflow_dispatch",
)
def override_pair(base_ref: str, candidate_ref: str) -> RefPair:
return RefPair(_override_run(base_ref), _override_run(candidate_ref), False, "explicit refs")
def write_outputs(pair: RefPair) -> None:
values = {
"skip": "true" if pair.skip else "false",
"reason": pair.reason,
"base_sha": pair.base.head_sha if pair.base else "",
"candidate_sha": pair.candidate.head_sha if pair.candidate else "",
"base_run_url": pair.base.html_url if pair.base else "",
"candidate_run_url": pair.candidate.html_url if pair.candidate else "",
"base_run_id": str(pair.base.id) if pair.base and pair.base.id else "",
"candidate_run_id": str(pair.candidate.id) if pair.candidate and pair.candidate.id else "",
}
output_path = os.environ.get("GITHUB_OUTPUT")
if output_path:
with open(output_path, "a", encoding="utf-8") as handle:
for key, value in values.items():
handle.write(f"{key}={value}\n")
for key, value in values.items():
print(f"{key}={value}")
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY"))
parser.add_argument(
"--token",
default=os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN"),
)
parser.add_argument("--branch", default=os.environ.get("NIGHTLY_BRANCH") or "")
parser.add_argument(
"--candidate-run-id",
default=os.environ.get("CANDIDATE_RUN_ID") or "",
)
parser.add_argument("--base-ref", default=os.environ.get("BASE_REF") or "")
parser.add_argument("--candidate-ref", default=os.environ.get("CANDIDATE_REF") or "")
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
base_ref = args.base_ref.strip()
candidate_ref = args.candidate_ref.strip()
if base_ref and candidate_ref:
write_outputs(override_pair(base_ref, candidate_ref))
return 0
if bool(base_ref) != bool(candidate_ref):
print("Provide both --base-ref and --candidate-ref, or neither.", file=sys.stderr)
return 2
if not args.repo or not args.token:
print("--repo and --token (or GITHUB_REPOSITORY / GITHUB_TOKEN) are required.", file=sys.stderr)
return 2
candidate_run_id = int(args.candidate_run_id) if str(args.candidate_run_id).strip() else None
candidate: WorkflowRun | None = None
branch = args.branch.strip() or None
if candidate_run_id is not None:
candidate = fetch_run(args.token, args.repo, candidate_run_id)
if candidate.conclusion != "success":
write_outputs(
RefPair(
None,
candidate,
True,
f"nightly run {candidate_run_id} conclusion is {candidate.conclusion}",
)
)
return 0
branch = branch or candidate.head_branch or None
runs = list_successful_nightly_runs(args.token, args.repo, branch=branch)
pair = select_base_and_candidate(
runs,
candidate_run_id=candidate_run_id if candidate is None else None,
candidate=candidate,
)
write_outputs(pair)
return 0
if __name__ == "__main__":
raise SystemExit(main())
.github/scripts/query-regression-run.py
+501 -26
View File
@@ -18,10 +18,17 @@
from __future__ import annotations
import argparse
import json
import os
import re
import signal
import socket
import subprocess
import time
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from typing import Any
DEFAULT_CASES = [
@@ -100,38 +107,479 @@ def append_step_summary(summary: Path) -> None:
out.write(summary.read_text())
@dataclass(frozen=True)
class RunTarget:
name: str
binary: Path
work_dir: Path
http_port: int
grpc_port: int
mysql_port: int
postgres_port: int
metasrv_rpc_port: int
metasrv_http_port: int
datanode_rpc_port: int
datanode_http_port: int
datanode_data_dir: Path
frontend_config: Path | None = None
def allocate_ports(n: int) -> list[int]:
socks = []
try:
for _ in range(n):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("127.0.0.1", 0))
socks.append(sock)
return [sock.getsockname()[1] for sock in socks]
finally:
for sock in socks:
sock.close()
def make_target(
name: str,
binary: Path,
root: Path,
ports: list[int],
frontend_config: Path | None = None,
) -> RunTarget:
work_dir = root / name
return RunTarget(
name=name,
binary=binary,
work_dir=work_dir,
http_port=ports[4],
grpc_port=ports[5],
mysql_port=ports[6],
postgres_port=ports[7],
metasrv_rpc_port=ports[0],
metasrv_http_port=ports[1],
datanode_rpc_port=ports[2],
datanode_http_port=ports[3],
datanode_data_dir=work_dir / "datanode-0" / "data",
frontend_config=frontend_config,
)
def wait_health(port: int, timeout_s: float = 60.0) -> None:
deadline = time.monotonic() + timeout_s
last: Exception | None = None
while time.monotonic() < deadline:
try:
with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=2) as response:
if response.status < 500:
return
except Exception as err: # noqa: BLE001 - retain the last health diagnostic
last = err
time.sleep(0.5)
raise TimeoutError(f"health check timed out on port {port}: {last}")
def component_log_dir(target: RunTarget, name: str) -> Path:
return target.work_dir / "logs" / ("datanode-0" if name == "datanode" else name)
def component_command(target: RunTarget, name: str) -> list[str]:
log_dir = component_log_dir(target, name)
if name == "metasrv":
return [
str(target.binary), "metasrv", "start",
"--grpc-bind-addr", f"127.0.0.1:{target.metasrv_rpc_port}",
"--grpc-server-addr", f"127.0.0.1:{target.metasrv_rpc_port}",
"--http-addr", f"127.0.0.1:{target.metasrv_http_port}",
"--backend", "memory-store", "--enable-region-failover", "false",
"--log-dir", str(log_dir),
]
if name == "datanode":
return [
str(target.binary), "datanode", "start",
"--grpc-bind-addr", f"127.0.0.1:{target.datanode_rpc_port}",
"--grpc-server-addr", f"127.0.0.1:{target.datanode_rpc_port}",
"--http-addr", f"127.0.0.1:{target.datanode_http_port}",
"--data-home", str(target.datanode_data_dir), "--log-dir", str(log_dir),
"--node-id", "0", "--metasrv-addrs", f"127.0.0.1:{target.metasrv_rpc_port}",
]
if name == "frontend":
command = [str(target.binary), "frontend", "start"]
if target.frontend_config is not None:
command.extend(["--config-file", str(target.frontend_config)])
command.extend([
"--metasrv-addrs", f"127.0.0.1:{target.metasrv_rpc_port}",
"--http-addr", f"127.0.0.1:{target.http_port}",
"--grpc-bind-addr", f"127.0.0.1:{target.grpc_port}",
"--grpc-server-addr", f"127.0.0.1:{target.grpc_port}",
"--mysql-addr", f"127.0.0.1:{target.mysql_port}",
"--postgres-addr", f"127.0.0.1:{target.postgres_port}",
"--log-dir", str(log_dir),
])
return command
raise ValueError(f"unknown component: {name}")
def start_component(
target: RunTarget,
name: str,
procs: dict[tuple[str, str], subprocess.Popen[bytes]],
) -> None:
if name != "metasrv":
metasrv = procs.get((target.name, "metasrv"))
if metasrv is None or metasrv.poll() is not None:
raise RuntimeError("metasrv exited; memory-store metadata is no longer valid")
if name == "datanode":
target.datanode_data_dir.mkdir(parents=True, exist_ok=True)
logs = component_log_dir(target, name)
logs.mkdir(parents=True, exist_ok=True)
with (logs / "stdout.log").open("ab") as out, (logs / "stderr.log").open("ab") as err:
procs[(target.name, name)] = subprocess.Popen(
component_command(target, name),
stdout=out,
stderr=err,
start_new_session=True,
)
wait_health({
"metasrv": target.metasrv_http_port,
"datanode": target.datanode_http_port,
"frontend": target.http_port,
}[name])
def stop_component(
target: RunTarget,
name: str,
procs: dict[tuple[str, str], subprocess.Popen[bytes]],
) -> None:
proc = procs.pop((target.name, name), None)
if proc is None or proc.poll() is not None:
return
try:
os.killpg(proc.pid, signal.SIGTERM)
except ProcessLookupError:
return
try:
proc.wait(timeout=20)
except subprocess.TimeoutExpired:
try:
os.killpg(proc.pid, signal.SIGKILL)
except ProcessLookupError:
return
proc.wait(timeout=20)
def restart_component(
target: RunTarget,
name: str,
procs: dict[tuple[str, str], subprocess.Popen[bytes]],
) -> None:
stop_component(target, name, procs)
start_component(target, name, procs)
def stop_all(targets: list[RunTarget], procs: dict[tuple[str, str], subprocess.Popen[bytes]]) -> None:
for target in reversed(targets):
for name in ("frontend", "datanode", "metasrv"):
stop_component(target, name, procs)
def load_plan(fixture_generator: Path, case_path: Path) -> dict[str, Any]:
result = subprocess.run(
[str(fixture_generator), "plan", "--case", str(case_path)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
check=False,
)
if result.returncode != 0:
raise RuntimeError(f"query_perf_fixture plan failed: {result.stderr[:2000]}")
return json.loads(result.stdout)
def run_direct_case(
args: argparse.Namespace,
case_path: Path,
work_dir: Path,
base_bin: Path,
candidate_bin: Path,
fixture_generator: Path,
runner: Path,
) -> int:
ports = allocate_ports(16)
targets = [
make_target("base", base_bin, work_dir, ports[:8]),
make_target("candidate", candidate_bin, work_dir, ports[8:]),
]
for target in targets:
if target.work_dir.exists() and any(target.work_dir.iterdir()):
raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}")
target.work_dir.mkdir(parents=True, exist_ok=True)
procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {}
try:
for target in targets:
start_component(target, "metasrv", procs)
start_component(target, "datanode", procs)
start_component(target, "frontend", procs)
prepare = [
str(runner), "prepare-direct", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--fixture-dir", str(work_dir / "fixture"),
"--output", str(work_dir / "prepare-direct.json"),
"--http-timeout", str(args.http_timeout),
]
if parse_bool(args.allow_large_fixture):
prepare.append("--allow-large-fixture")
prepare_status = subprocess.run(prepare, check=False).returncode
if prepare_status != 0:
return prepare_status
for target in targets:
stop_component(target, "datanode", procs)
prepared = json.loads((work_dir / "prepare-direct.json").read_text(encoding="utf-8"))
fixtures = prepared.get("fixtures")
if not isinstance(fixtures, list):
raise RuntimeError("prepare-direct report has no fixtures array")
for target in targets:
destination = target.work_dir / "materialize-destination.toml"
destination.write_text(
f"data_home = {json.dumps(str(target.datanode_data_dir))}\n"
"object_store = { type = \"File\" }\n",
encoding="utf-8",
)
for fixture in fixtures:
fixture_dir = fixture.get("fixture_dir") if isinstance(fixture, dict) else None
if not isinstance(fixture_dir, str):
raise RuntimeError("prepare-direct fixture record has no fixture_dir")
materialize = [
str(runner), "materialize", "--fixture-dir", fixture_dir,
"--destination", str(destination),
]
materialize_status = subprocess.run(materialize, check=False).returncode
if materialize_status != 0:
return materialize_status
for target in targets:
start_component(target, "datanode", procs)
measure = [
str(runner), "measure", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--output", str(work_dir / "query-regression-report.json"),
"--http-timeout", str(args.http_timeout),
]
status = subprocess.run(measure, check=False).returncode
if status != 0:
for target in targets:
restart_component(target, "frontend", procs)
status = subprocess.run(measure, check=False).returncode
return status
finally:
stop_all(targets, procs)
def run_remote_case(
args: argparse.Namespace,
case_path: Path,
work_dir: Path,
base_bin: Path,
candidate_bin: Path,
fixture_generator: Path,
runner: Path,
) -> int:
ports = allocate_ports(16)
targets = [
make_target(
"base",
base_bin,
work_dir,
ports[:8],
work_dir / "base" / "frontend-prom-store.toml",
),
make_target(
"candidate",
candidate_bin,
work_dir,
ports[8:],
work_dir / "candidate" / "frontend-prom-store.toml",
),
]
for target in targets:
if target.work_dir.exists() and any(target.work_dir.iterdir()):
raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}")
target.work_dir.mkdir(parents=True, exist_ok=True)
if target.frontend_config is None:
raise RuntimeError(f"remote target has no frontend config path: {target.name}")
render = [
str(runner), "render-remote-config", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--output", str(target.frontend_config),
]
render_status = subprocess.run(render, check=False).returncode
if render_status != 0:
return render_status
procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {}
report = work_dir / "query-regression-report.json"
try:
for target in targets:
start_component(target, "metasrv", procs)
start_component(target, "datanode", procs)
start_component(target, "frontend", procs)
prepare = [
str(runner), "prepare-remote", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--output", str(work_dir / "prepare-remote.json"),
"--http-timeout", str(args.http_timeout),
]
prepare_status = subprocess.run(prepare, check=False).returncode
if prepare_status != 0:
return prepare_status
measure = [
str(runner), "measure", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--output", str(report), "--http-timeout", str(args.http_timeout),
]
measure_status = subprocess.run(measure, check=False).returncode
if measure_status != 0 and not report.exists():
return measure_status
for target in targets:
stop_component(target, "datanode", procs)
finalize = [
str(runner), "finalize-remote", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--candidate-bin", str(candidate_bin),
"--base-data-home", str(targets[0].datanode_data_dir),
"--candidate-data-home", str(targets[1].datanode_data_dir),
"--report", str(report),
]
finalize_status = subprocess.run(finalize, check=False).returncode
if finalize_status != 0:
return finalize_status
final_report = json.loads(report.read_text(encoding="utf-8"))
return 1 if measure_status != 0 or final_report.get("status") == "failed" else 0
finally:
stop_all(targets, procs)
def run_otlp_case(
args: argparse.Namespace,
case_path: Path,
work_dir: Path,
base_bin: Path,
candidate_bin: Path,
fixture_generator: Path,
runner: Path,
) -> int:
if args.otelgen_bin is None:
raise ValueError("--otelgen-bin is required for otlp_trace_load")
ports = allocate_ports(16)
targets = [
make_target("base", base_bin, work_dir, ports[:8]),
make_target("candidate", candidate_bin, work_dir, ports[8:]),
]
for target in targets:
if target.work_dir.exists() and any(target.work_dir.iterdir()):
raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}")
target.work_dir.mkdir(parents=True, exist_ok=True)
procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {}
try:
for target in targets:
try:
start_component(target, "metasrv", procs)
start_component(target, "datanode", procs)
start_component(target, "frontend", procs)
command = [
str(runner), "run-otlp-target", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--otelgen-bin", str(args.otelgen_bin),
"--http-port", str(target.http_port),
"--target-name", target.name,
"--work-dir", str(target.work_dir),
"--output", str(target.work_dir / "report.json"),
"--http-timeout", str(args.http_timeout),
]
status = subprocess.run(command, check=False).returncode
finally:
stop_all([target], procs)
if status != 0:
return status
output = work_dir / "query-regression-report.json"
finalize = [
str(runner), "finalize-otlp", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-result", str(targets[0].work_dir / "report.json"),
"--candidate-result", str(targets[1].work_dir / "report.json"),
"--output", str(output),
]
status = subprocess.run(finalize, check=False).returncode
if status != 0:
return status
report = json.loads(output.read_text(encoding="utf-8"))
return 1 if report.get("status") == "failed" else 0
finally:
stop_all(targets, procs)
def run_case(args: argparse.Namespace, case_path: Path, work_dir: Path) -> int:
target_dir = profile_dir(args.cargo_profile)
base_bin = args.base_bin or args.base_src / "target" / target_dir / "greptime"
candidate_bin = args.candidate_bin or args.candidate_src / "target" / target_dir / "greptime"
fixture_generator = args.fixture_generator or args.candidate_src / "target" / target_dir / "query_perf_fixture"
cmd = [
"uv",
"run",
"--no-project",
"python",
str(args.candidate_src / "tests/perf/query_regression_runner.py"),
"--case",
str(case_path),
"--base-bin",
str(base_bin),
"--candidate-bin",
str(candidate_bin),
"--fixture-generator",
str(fixture_generator),
"--work-dir",
str(work_dir),
"--http-timeout",
str(args.http_timeout),
]
if parse_bool(args.allow_large_fixture):
cmd.append("--allow-large-fixture")
if args.otelgen_bin is not None:
cmd.extend(["--otelgen-bin", str(args.otelgen_bin)])
runner = args.runner or args.candidate_src / "target" / target_dir / "query_regression_runner"
plan = load_plan(fixture_generator, case_path)
scenario = plan.get("scenario")
kind = scenario.get("kind") if isinstance(scenario, dict) else None
print(f"::group::Query regression case: {case_path}", flush=True)
try:
return subprocess.run(cmd, check=False).returncode
if kind == "direct_readable_sst":
return run_direct_case(
args,
case_path,
work_dir,
base_bin,
candidate_bin,
fixture_generator,
runner,
)
if kind == "prom_remote_write_then_query":
return run_remote_case(
args,
case_path,
work_dir,
base_bin,
candidate_bin,
fixture_generator,
runner,
)
if kind == "otlp_trace_load":
return run_otlp_case(
args,
case_path,
work_dir,
base_bin,
candidate_bin,
fixture_generator,
runner,
)
raise ValueError(
f"unsupported scenario kind {kind!r}; supported: "
"'direct_readable_sst', 'prom_remote_write_then_query', 'otlp_trace_load'"
)
finally:
print("::endgroup::", flush=True)
@@ -157,6 +605,27 @@ def write_summary(args: argparse.Namespace, reports: list[Path]) -> int:
return subprocess.run(cmd, check=False).returncode
def write_failed_report(work_dir: Path, case_path: Path, error: Exception) -> None:
work_dir.mkdir(parents=True, exist_ok=True)
path = work_dir / "query-regression-report.json"
try:
report = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(report, dict) or not isinstance(report.get("targets"), list) or not isinstance(report.get("thresholds"), list):
raise ValueError("existing report is malformed")
except (OSError, ValueError, json.JSONDecodeError):
report = {
"case_path": str(case_path),
"targets": [],
"thresholds": [],
}
report["status"] = "failed"
report["error"] = repr(error)
path.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--cases", action="append", help="'all', 'heavy', or comma/space separated case paths")
@@ -170,6 +639,7 @@ def main() -> int:
default=configured_path(os.environ.get("FIXTURE_GENERATOR")),
)
parser.add_argument("--otelgen-bin", type=Path, default=configured_path(os.environ.get("OTELGEN_BIN")))
parser.add_argument("--runner", type=Path, default=configured_path(os.environ.get("QUERY_REGRESSION_RUNNER")))
parser.add_argument("--cargo-profile", default=os.environ.get("CARGO_PROFILE", "nightly"))
parser.add_argument("--work-dir", default=Path("query-regression-work"), type=Path)
parser.add_argument("--http-timeout", default=os.environ.get("HTTP_TIMEOUT", "300"))
@@ -199,7 +669,12 @@ def main() -> int:
case_path = resolve_case_path(args.candidate_src, case)
work_dir = args.work_dir / case_slug(case_path)
reports.append(work_dir / "query-regression-report.json")
case_status = run_case(args, case_path, work_dir)
try:
case_status = run_case(args, case_path, work_dir)
except Exception as err: # noqa: BLE001 - preserve an aggregate report for the summary step
print(f"error: query regression case {case_path}: {err}", flush=True)
write_failed_report(work_dir, case_path, err)
case_status = 1
if case_status != 0:
status = case_status or 1
.github/scripts/run-compat.py
+30 -1
View File
@@ -62,6 +62,10 @@ def parse_args() -> argparse.Namespace:
action="store_true",
help="Pass --preserve-state to the real compat run for artifact upload.",
)
parser.add_argument(
"--from-versions-extra",
help="Comma-separated extra from-versions appended to the ci.toml window (for nightly runs).",
)
return parser.parse_args()
@@ -108,6 +112,28 @@ def load_versions(
return validated
def load_extra_versions(raw: str | None, window: list[str]) -> list[str]:
"""Parse --from-versions-extra, skipping versions already in the window.
Each entry must match VERSION_RE; like load_versions, no version may appear
twice in the effective list, so window entries and duplicates are skipped.
Order is preserved.
"""
if not raw:
return []
extra: list[str] = []
seen: set[str] = set()
for item in raw.split(","):
version = item.strip()
if VERSION_RE.fullmatch(version) is None:
raise SystemExit(f"Invalid compat from-versions-extra version: {version!r}")
if version in window or version in seen:
continue
seen.add(version)
extra.append(version)
return extra
def check_inputs(runner: Path, to_bins_dir: Path) -> None:
if not runner.is_file():
raise SystemExit(f"sqlness-runner binary not found: {runner}")
@@ -253,6 +279,7 @@ def main() -> int:
check_inputs(runner, to_bins_dir)
from_versions = load_versions(config_path, "from_versions")
extra_versions = load_extra_versions(args.from_versions_extra, from_versions)
downgrade_to_versions = load_versions(
config_path, "downgrade_to_versions", required=False
)
@@ -260,8 +287,10 @@ def main() -> int:
print("Compatibility from-version window:", flush=True)
for version in from_versions:
print(f" - {version}", flush=True)
for version in extra_versions:
print(f" - {version} (nightly extra)", flush=True)
for from_version in from_versions:
for from_version in [*from_versions, *extra_versions]:
run_for_version(
runner=runner,
to_bins_dir=to_bins_dir,
.github/scripts/tests/test_update_compat_versions.py
+332
View File
@@ -0,0 +1,332 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import contextlib
import importlib.util
import io
import json
import tempfile
import unittest
import urllib.error
from pathlib import Path
from unittest import mock
SCRIPT = Path(__file__).resolve().parents[1] / "update-compat-versions.py"
SPEC = importlib.util.spec_from_file_location("update_compat_versions", SCRIPT)
assert SPEC is not None and SPEC.loader is not None
UPDATER = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(UPDATER)
class UpdateCompatVersionsTest(unittest.TestCase):
def write_case(self, root: Path, name: str, from_range: str) -> None:
case_dir = root / "tests" / "compatibility" / "cases" / name
case_dir.mkdir(parents=True)
(case_dir / "case.toml").write_text(
f'name = "{name}"\nfrom_range = {from_range}\n', encoding="utf-8"
)
def write_config(self, root: Path, versions: str, trailer: str = "") -> Path:
config = root / "tests" / "compatibility" / "ci.toml"
config.parent.mkdir(parents=True, exist_ok=True)
config.write_text(
"# Keep these comments when the window is regenerated.\n"
f"from_versions = {versions}\n{trailer}",
encoding="utf-8",
)
return config
def test_stable_tags_filter_prereleases_and_nightlies(self) -> None:
self.assertEqual(
["v1.0.0", "v1.0.2", "v1.1.0"],
UPDATER.stable_tags(
["v1.1.0-nightly-20260101", "v1.0.2", "v1.0.0", "v1.1.0", "v1.0.0-rc.1"]
),
)
def test_sliding_versions_roll_over_to_two_newest_minor_lines(self) -> None:
self.assertEqual(
["v1.9.4", "v2.0.1"],
UPDATER.sliding_versions(["v1.8.9", "v1.9.0", "v1.9.4", "v2.0.0", "v2.0.1"]),
)
def test_sliding_versions_supports_one_minor_line(self) -> None:
self.assertEqual(["v1.1.3"], UPDATER.sliding_versions(["v1.1.0", "v1.1.3"]))
def test_sliding_versions_rejects_no_stable_tags(self) -> None:
with self.assertRaisesRegex(UPDATER.CompatVersionError, "No stable release tags"):
UPDATER.sliding_versions(["v1.1.0-rc.1", "v1.1.0-nightly-20260101"])
def test_effective_versions_returns_only_the_sliding_window(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["=v1.0.2", ">=v1.1.0"]')
versions = UPDATER.effective_versions(["v1.0.2", "v1.1.0", "v1.1.1", "v1.1.3"])
self.assertEqual(["v1.0.2", "v1.1.3"], versions)
def test_nightly_versions_includes_anchors_and_sliding(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["=v1.0.2", ">=v1.1.0"]')
self.write_case(root, "more_history", '["=v1.1.0", "=v1.1.1"]')
versions = UPDATER.nightly_versions(
["v1.0.2", "v1.1.0", "v1.1.1", "v1.1.3"],
root / "tests" / "compatibility" / "cases",
)
self.assertEqual(["v1.0.2", "v1.1.0", "v1.1.1", "v1.1.3"], versions)
def test_exact_anchor_syntaxes_are_normalized(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(
root,
"historical",
'[" = v1.0.2 ", " == v1.1.0 ", " v1.1.1 ", "1.1.2", "v01.01.003-rc.1"]',
)
anchors = UPDATER.extract_pinned_anchors(root / "tests" / "compatibility" / "cases")
self.assertEqual(
{"v1.0.2", "v1.1.0", "v1.1.1", "v1.1.2", "v1.1.3"}, anchors
)
def test_ordered_ranges_and_wildcard_are_not_anchors(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(
root,
"non_exact",
'[">= v1.0.0", "<=v1.1.0", ">v1.0.2", "< v2.0.0", "*"]',
)
anchors = UPDATER.extract_pinned_anchors(root / "tests" / "compatibility" / "cases")
self.assertEqual(set(), anchors)
def test_update_is_sorted_and_idempotent(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
config = self.write_config(
root,
'[\n "v1.1.0",\n "v1.0.2",\n]',
"# This trailing comment must remain.\nother_setting = true\n",
)
self.write_case(root, "historical", '["=v1.0.2"]')
tags = ["v1.0.2", "v1.1.0", "v1.1.1", "v1.1.3"]
with mock.patch.object(UPDATER, "git_tags", return_value=tags):
self.assertEqual(0, UPDATER.main(["--repo-root", str(root), "--update"]))
first_update = config.read_text(encoding="utf-8")
self.assertEqual(0, UPDATER.main(["--repo-root", str(root), "--update"]))
self.assertEqual(first_update, config.read_text(encoding="utf-8"))
self.assertIn('from_versions = ["v1.0.2", "v1.1.3"]', first_update)
self.assertIn("# Keep these comments when the window is regenerated.", first_update)
self.assertIn("# This trailing comment must remain.\nother_setting = true\n", first_update)
def test_check_reports_a_unified_diff_for_stale_config(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_config(root, '["v1.0.2"]')
self.write_case(root, "historical", '["=v1.0.2"]')
stderr = io.StringIO()
with mock.patch.object(UPDATER, "git_tags", return_value=["v1.0.2", "v1.1.3"]):
with contextlib.redirect_stderr(stderr):
result = UPDATER.main(["--repo-root", str(root), "--check"])
self.assertEqual(1, result)
self.assertIn("--- a/tests/compatibility/ci.toml", stderr.getvalue())
self.assertIn("+++ b/tests/compatibility/ci.toml", stderr.getvalue())
def test_rejects_duplicate_config_versions(self) -> None:
with tempfile.TemporaryDirectory() as directory:
config = self.write_config(Path(directory), '["v1.0.2", "v1.0.2"]')
with self.assertRaisesRegex(UPDATER.CompatVersionError, "Duplicate from_versions"):
UPDATER.load_from_versions(config)
def test_rejects_unavailable_pinned_anchor(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["=v1.0.2"]')
with self.assertRaisesRegex(UPDATER.CompatVersionError, "unavailable"):
UPDATER.nightly_versions(
["v1.1.0"], root / "tests" / "compatibility" / "cases"
)
def test_rejects_malformed_exact_constraint(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["=v1.0"]')
with self.assertRaisesRegex(UPDATER.CompatVersionError, "Malformed version"):
UPDATER.extract_pinned_anchors(root / "tests" / "compatibility" / "cases")
def test_rejects_malformed_bare_exact_constraint(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["v1.0"]')
with self.assertRaisesRegex(UPDATER.CompatVersionError, "Malformed version"):
UPDATER.extract_pinned_anchors(root / "tests" / "compatibility" / "cases")
def test_rejects_malformed_ordered_constraint(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '[">=v1.0"]')
with self.assertRaisesRegex(UPDATER.CompatVersionError, "Malformed version"):
UPDATER.extract_pinned_anchors(root / "tests" / "compatibility" / "cases")
def test_published_only_excludes_failed_release_tags(self) -> None:
self.assertEqual(
["v1.0.2", "v1.1.3"],
UPDATER.effective_versions(
["v1.0.2", "v1.1.3", "v1.1.4"], published={"v1.0.2", "v1.1.3"}
),
)
def test_published_only_empty_intersection_raises(self) -> None:
with self.assertRaisesRegex(UPDATER.CompatVersionError, "published release assets"):
UPDATER.effective_versions(["v1.0.2"], published={"v1.1.4"})
def test_required_asset_names(self) -> None:
self.assertEqual(
{
"greptime-linux-amd64-v1.1.4.tar.gz",
"greptime-linux-amd64-v1.1.4.sha256sum",
},
UPDATER.required_asset_names("v1.1.4"),
)
class _FakeResponse:
def __init__(self, payload) -> None:
self._payload = payload
def __enter__(self):
return self
def __exit__(self, *exc) -> None:
return None
def read(self) -> bytes:
return json.dumps(self._payload).encode("utf-8")
def _release(self, tag: str, *, draft=False, prerelease=False, assets=()) -> dict:
return {
"tag_name": tag,
"draft": draft,
"prerelease": prerelease,
"assets": [{"name": name} for name in assets],
}
def test_published_release_tags_parses_and_filters(self) -> None:
required = {
"greptime-linux-amd64-v1.1.4.tar.gz",
"greptime-linux-amd64-v1.1.4.sha256sum",
}
page_one = [
self._release(f"preview-{i}", assets=required) for i in range(96)
] + [
self._release("v1.1.4", assets=required),
self._release("v1.1.3", draft=True, assets=required),
self._release("v1.1.2", prerelease=True, assets=UPDATER.required_asset_names("v1.1.2")),
self._release("v1.1.1", assets={"greptime-linux-amd64-v1.1.1.tar.gz"}),
]
self.assertEqual(100, len(page_one))
page_two = [
self._release("v1.0.2", assets=UPDATER.required_asset_names("v1.0.2")),
self._release("v1.0.1-beta", assets=required),
]
calls: list[str] = []
def fake_urlopen(request, *args, **kwargs):
calls.append(request.full_url)
payload = page_two if "page=2" in request.full_url else page_one
return self._FakeResponse(payload)
with mock.patch.object(UPDATER.urllib.request, "urlopen", side_effect=fake_urlopen):
published = UPDATER.published_release_tags("GreptimeTeam", "greptimedb", None)
# Draft releases are excluded; prerelease releases with the required
# assets are included; releases missing assets or with non-stable tags
# are excluded.
self.assertEqual({"v1.1.4", "v1.1.2", "v1.0.2"}, published)
self.assertEqual(2, len(calls))
self.assertTrue(all("per_page=100" in url for url in calls))
def test_published_release_tags_http_error_raises(self) -> None:
with mock.patch.object(
UPDATER.urllib.request,
"urlopen",
side_effect=urllib.error.HTTPError(
"https://api.github.com/repos/GreptimeTeam/greptimedb/releases",
403,
"Forbidden",
None,
None,
),
):
with self.assertRaisesRegex(
UPDATER.CompatVersionError, "Unable to fetch published releases"
):
UPDATER.published_release_tags("GreptimeTeam", "greptimedb", None)
def test_check_anchors_rejects_unpublished_anchor(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["=v1.0.2", "=v1.1.0"]')
with mock.patch.object(UPDATER, "git_tags", return_value=["v1.0.2", "v1.1.0"]):
with mock.patch.object(
UPDATER, "published_release_tags", return_value={"v1.0.2"}
):
result = UPDATER.main(["--check-anchors", "--repo-root", str(root)])
self.assertEqual(2, result)
def test_check_anchors_passes(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["=v1.0.2", "=v1.1.0"]')
with mock.patch.object(UPDATER, "git_tags", return_value=["v1.0.2", "v1.1.0"]):
with mock.patch.object(
UPDATER, "published_release_tags", return_value={"v1.0.2", "v1.1.0"}
):
result = UPDATER.main(["--check-anchors", "--repo-root", str(root)])
self.assertEqual(0, result)
def test_nightly_window_prints_csv(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
self.write_case(root, "historical", '["=v1.0.2"]')
self.write_case(root, "more_history", '["=v1.1.0", "=v1.1.1"]')
stdout = io.StringIO()
with mock.patch.object(
UPDATER, "git_tags", return_value=["v1.0.2", "v1.1.0", "v1.1.1", "v1.1.3"]
):
with contextlib.redirect_stdout(stdout):
result = UPDATER.main(["--nightly-window", "--repo-root", str(root)])
self.assertEqual(0, result)
self.assertEqual("v1.0.2,v1.1.0,v1.1.1,v1.1.3\n", stdout.getvalue())
def test_update_with_published_only(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
config = self.write_config(root, '["v1.0.2"]')
self.write_case(root, "historical", '["=v1.0.2"]')
with mock.patch.object(
UPDATER, "git_tags", return_value=["v1.0.2", "v1.1.3", "v1.1.4"]
):
with mock.patch.object(
UPDATER, "published_release_tags", return_value={"v1.0.2", "v1.1.3"}
):
result = UPDATER.main(
["--update", "--published-only", "--repo-root", str(root)]
)
updated = config.read_text(encoding="utf-8")
self.assertEqual(0, result)
self.assertIn('from_versions = ["v1.0.2", "v1.1.3"]', updated)
if __name__ == "__main__":
unittest.main()
.github/scripts/update-compat-versions.py
+468
View File
@@ -0,0 +1,468 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Update the compatibility CI version window from local release tags and cases.
The PR window keeps only the sliding window over stable release tags whose
GitHub releases carry the sqlness compat artifacts (greptime-linux-amd64
tar.gz and sha256sum). Exact =vX.Y.Z `from_range` anchors in case.toml are not
retained in the PR window: they are validated with --check-anchors and
exercised by nightly runs via --nightly-window.
"""
from __future__ import annotations
import argparse
import ast
import difflib
import json
import os
import re
import subprocess
import sys
import urllib.error
import urllib.request
from pathlib import Path
from typing import Iterable
VERSION_RE = re.compile(r"^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$")
FROM_VERSIONS_RE = re.compile(
r"(?ms)^(?P<indent>[ \t]*)from_versions[ \t]*=[ \t]*(?P<array>\[(?:[^\]]|\n)*?\])"
)
FROM_VERSIONS_ASSIGNMENT_RE = re.compile(r"(?m)^[ \t]*from_versions[ \t]*=")
FROM_RANGE_RE = re.compile(
r"(?ms)^[ \t]*from_range[ \t]*=[ \t]*(?P<array>\[(?:[^\]]|\n)*?\])"
)
FROM_RANGE_ASSIGNMENT_RE = re.compile(r"(?m)^[ \t]*from_range[ \t]*=")
class CompatVersionError(ValueError):
"""A compatibility version source does not satisfy the updater policy."""
def version_key(version: str) -> tuple[int, int, int]:
"""Return a sortable semantic-version key for a stable version tag."""
match = VERSION_RE.fullmatch(version)
if match is None:
raise CompatVersionError(f"Not a stable release tag: {version!r}")
major, minor, patch = (int(part) for part in match.groups())
return major, minor, patch
def stable_tags(tags: Iterable[str]) -> list[str]:
"""Keep only exact stable release tags, excluding prereleases and nightlies."""
return sorted({tag for tag in tags if VERSION_RE.fullmatch(tag)}, key=version_key)
def sliding_versions(tags: Iterable[str]) -> list[str]:
"""Return the latest patch in each of the two newest stable minor lines."""
stable = stable_tags(tags)
if not stable:
raise CompatVersionError("No stable release tags found (expected exact v<major>.<minor>.<patch> tags)")
latest_by_minor: dict[tuple[int, int], str] = {}
for tag in stable:
major, minor, _ = version_key(tag)
minor_line = (major, minor)
if minor_line not in latest_by_minor or version_key(tag) > version_key(
latest_by_minor[minor_line]
):
latest_by_minor[minor_line] = tag
newest_minor_lines = sorted(latest_by_minor, reverse=True)[:2]
return sorted((latest_by_minor[line] for line in newest_minor_lines), key=version_key)
def _parse_string_array(array: str, context: str) -> list[str]:
without_comments = "\n".join(line.split("#", 1)[0] for line in array.splitlines())
try:
values = ast.literal_eval(without_comments)
except (SyntaxError, ValueError) as err:
raise CompatVersionError(f"Malformed string array in {context}: {err}") from err
if not isinstance(values, list) or any(not isinstance(value, str) for value in values):
raise CompatVersionError(f"{context} must be a TOML string array")
return values
def _from_versions_match(content: str, config_path: Path) -> re.Match[str]:
matches = list(FROM_VERSIONS_RE.finditer(content))
assignments = list(FROM_VERSIONS_ASSIGNMENT_RE.finditer(content))
if len(assignments) != 1 or len(matches) != 1:
raise CompatVersionError(f"{config_path} must define exactly one from_versions array")
match = matches[0]
remainder = content[match.end() :].split("\n", 1)[0].strip()
if remainder and not remainder.startswith("#"):
raise CompatVersionError(f"Malformed from_versions assignment in {config_path}")
return match
def load_from_versions(config_path: Path) -> list[str]:
"""Validate and load the current checked-in compatibility window."""
if not config_path.is_file():
raise CompatVersionError(f"Compatibility CI config not found: {config_path}")
match = _from_versions_match(config_path.read_text(encoding="utf-8"), config_path)
versions = _parse_string_array(match.group("array"), str(config_path))
if not versions:
raise CompatVersionError(f"{config_path} must define a non-empty from_versions list")
duplicates = {version for version in versions if versions.count(version) > 1}
if duplicates:
raise CompatVersionError(
f"Duplicate from_versions in {config_path}: {', '.join(sorted(duplicates))}"
)
for version in versions:
version_key(version)
return versions
def extract_pinned_anchors(cases_dir: Path) -> set[str]:
"""Extract exact from_range constraints that must remain in the CI window."""
if not cases_dir.is_dir():
raise CompatVersionError(f"Compatibility cases directory not found: {cases_dir}")
anchors: set[str] = set()
for case_path in sorted(cases_dir.glob("**/case.toml")):
content = case_path.read_text(encoding="utf-8")
assignments = list(FROM_RANGE_ASSIGNMENT_RE.finditer(content))
matches = list(FROM_RANGE_RE.finditer(content))
if len(assignments) != 1 or len(matches) != 1:
raise CompatVersionError(f"{case_path} must define exactly one from_range array")
for constraint in _parse_string_array(matches[0].group("array"), str(case_path)):
anchor = exact_anchor(constraint, case_path)
if anchor is not None:
anchors.add(anchor)
return anchors
def _runner_version(raw: str, case_path: Path, constraint: str) -> str:
"""Parse and normalize a version exactly like compat_case.rs Version::parse."""
stripped = raw.strip()
if stripped.startswith("v"):
stripped = stripped[1:]
core = stripped.split("-", 1)[0].split("+", 1)[0]
parts = core.split(".")
if len(parts) != 3:
raise CompatVersionError(
f"Malformed version in from_range constraint in {case_path}: {constraint!r}"
)
numbers: list[int] = []
for part in parts:
if not part.isascii() or not part.isdigit():
raise CompatVersionError(
f"Malformed version in from_range constraint in {case_path}: {constraint!r}"
)
value = int(part)
if value > 2**64 - 1:
raise CompatVersionError(
f"Version component overflows u64 in {case_path}: {constraint!r}"
)
numbers.append(value)
return f"v{numbers[0]}.{numbers[1]}.{numbers[2]}"
def exact_anchor(constraint: str, case_path: Path) -> str | None:
"""Validate a runner constraint and return its normalized exact anchor."""
normalized = constraint.strip()
if not normalized:
raise CompatVersionError(f"Empty from_range constraint in {case_path}")
if normalized == "*":
return None
for operator in (">=", "<=", "==", "=", ">", "<"):
if normalized.startswith(operator):
version = _runner_version(normalized[len(operator) :], case_path, constraint)
return version if operator in ("==", "=") else None
return _runner_version(normalized, case_path, constraint)
def required_asset_names(tag: str) -> set[str]:
"""Return the release assets the sqlness compat runner downloads for a tag."""
return {
f"greptime-linux-amd64-{tag}.tar.gz",
f"greptime-linux-amd64-{tag}.sha256sum",
}
def published_release_tags(owner: str, repo: str, token: str | None) -> set[str]:
"""Return stable tags whose non-draft GitHub releases carry the required assets.
Only draft releases are skipped: prerelease-flagged releases with the
required assets are included, and VERSION_RE already excludes prerelease
version tags (e.g. v1.2.0-beta.1).
"""
published: set[str] = set()
page = 1
while True:
url = (
f"https://api.github.com/repos/{owner}/{repo}/releases"
f"?per_page=100&page={page}"
)
headers = {
"Accept": "application/vnd.github+json",
"User-Agent": "update-compat-versions",
}
if token:
headers["Authorization"] = f"Bearer {token}"
request = urllib.request.Request(url, headers=headers)
try:
with urllib.request.urlopen(request) as response:
payload = json.loads(response.read().decode("utf-8"))
except (urllib.error.HTTPError, urllib.error.URLError, OSError, ValueError) as err:
raise CompatVersionError(
f"Unable to fetch published releases from {url}: {err}"
) from err
if not isinstance(payload, list):
raise CompatVersionError(
f"Unexpected response from {url}: expected a JSON release list"
)
for release in payload:
if not isinstance(release, dict):
continue
if release.get("draft"):
continue
tag = release.get("tag_name")
if not isinstance(tag, str) or VERSION_RE.fullmatch(tag) is None:
continue
assets = release.get("assets")
asset_names: set[str] = set()
if isinstance(assets, list):
asset_names = {
asset.get("name")
for asset in assets
if isinstance(asset, dict) and isinstance(asset.get("name"), str)
}
if required_asset_names(tag) <= asset_names:
published.add(tag)
if len(payload) < 100:
break
page += 1
return published
def effective_versions(tags: Iterable[str], published: set[str] | None = None) -> list[str]:
"""Return the PR window: the sliding window over stable release tags.
Exact =vX.Y.Z case anchors are intentionally excluded; they are validated
with --check-anchors and exercised by nightly runs (--nightly-window).
When ``published`` is given, only stable tags with published release
assets are considered.
"""
stable = stable_tags(tags)
if published is not None:
stable = sorted(set(stable) & set(published), key=version_key)
if not stable:
raise CompatVersionError(
"No stable tags have published release assets "
"(greptime-linux-amd64 tar.gz and sha256sum); cannot compute the sliding window"
)
return sliding_versions(stable)
def nightly_versions(
tags: Iterable[str], cases_dir: Path, published: set[str] | None = None
) -> list[str]:
"""Return the nightly window: exact anchors plus the sliding release window."""
anchors = extract_pinned_anchors(cases_dir)
window = effective_versions(tags, published)
available = set(stable_tags(tags))
if published is not None:
available &= set(published)
unavailable = sorted(anchors - available, key=version_key)
if unavailable:
raise CompatVersionError(
"Pinned compatibility anchor(s) are unavailable as stable git tags: "
+ ", ".join(unavailable)
)
return sorted(anchors | set(window), key=version_key)
def updated_config(content: str, config_path: Path, versions: list[str]) -> str:
"""Return config content with only the from_versions assignment replaced."""
match = _from_versions_match(content, config_path)
# Validate the existing setting before replacing it, so malformed or duplicate
# checked-in configs never get silently repaired by automation.
load_from_versions(config_path)
rendered = (
f'{match.group("indent")}from_versions = ['
+ ", ".join(f'"{version}"' for version in versions)
+ "]"
)
return content[: match.start()] + rendered + content[match.end() :]
def git_tags(repo_root: Path) -> list[str]:
try:
result = subprocess.run(
["git", "tag", "--list"],
cwd=repo_root,
check=True,
text=True,
capture_output=True,
)
except (OSError, subprocess.CalledProcessError) as err:
raise CompatVersionError(f"Unable to read local git tags in {repo_root}: {err}") from err
return result.stdout.splitlines()
def _owner_repo() -> tuple[str, str]:
"""Return the (owner, repo) pair from GITHUB_REPOSITORY or the default repo."""
repository = os.environ.get("GITHUB_REPOSITORY", "GreptimeTeam/greptimedb")
owner, _, repo = repository.partition("/")
if not owner or not repo:
return "GreptimeTeam", "greptimedb"
return owner, repo
def _github_token() -> str | None:
"""Return the GitHub token from the environment, if any."""
return os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN")
def check_anchors(repo_root: Path, cases_dir: Path) -> int:
"""Validate every exact anchor is a stable git tag with a published release.
Returns 0 when all anchors are runnable; prints problems to stderr and
returns 2 when any anchor fails.
"""
tags = set(git_tags(repo_root))
anchors = extract_pinned_anchors(cases_dir)
if not anchors:
return 0
published = published_release_tags(*_owner_repo(), _github_token())
problems: list[str] = []
for anchor in sorted(anchors, key=version_key):
if anchor not in tags:
problems.append(f"{anchor}: not a stable git tag")
elif anchor not in published:
problems.append(f"{anchor}: missing published release with required assets")
if problems:
print(
"error: exact from_range anchor(s) are not runnable in compatibility CI:",
file=sys.stderr,
)
for problem in problems:
print(f" - {problem}", file=sys.stderr)
return 2
return 0
def _path_from_repo(repo_root: Path, value: str | None, default: str) -> Path:
path = Path(value) if value is not None else Path(default)
return path if path.is_absolute() else repo_root / path
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
modes = parser.add_mutually_exclusive_group()
modes.add_argument("--check", action="store_true", help="Fail if ci.toml is stale.")
modes.add_argument("--update", action="store_true", help="Rewrite only ci.toml from_versions.")
modes.add_argument(
"--nightly-window",
action="store_true",
help="Print the comma-separated nightly window (anchors plus sliding) to stdout.",
)
parser.add_argument(
"--check-anchors",
action="store_true",
help="Validate exact case anchors are stable git tags with published release assets.",
)
parser.add_argument(
"--published-only",
action="store_true",
help=(
"With --check/--update, restrict the window to stable tags with "
"published release assets (requires network)."
),
)
parser.add_argument(
"--repo-root",
default=str(Path(__file__).resolve().parents[2]),
help="Repository root used for default paths and git tags.",
)
parser.add_argument("--config", help="Compatibility CI TOML path, relative to --repo-root.")
parser.add_argument("--cases-dir", help="Compatibility cases path, relative to --repo-root.")
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
repo_root = Path(args.repo_root).resolve()
config_path = _path_from_repo(repo_root, args.config, "tests/compatibility/ci.toml")
cases_dir = _path_from_repo(repo_root, args.cases_dir, "tests/compatibility/cases")
if not (args.check or args.update or args.nightly_window or args.check_anchors):
print(
"error: one of --check, --update, --nightly-window, or --check-anchors is required",
file=sys.stderr,
)
return 2
try:
if args.nightly_window:
versions = nightly_versions(git_tags(repo_root), cases_dir)
print(",".join(versions))
return 0
if args.check or args.update:
content = config_path.read_text(encoding="utf-8")
published = (
published_release_tags(*_owner_repo(), _github_token())
if args.published_only
else None
)
versions = effective_versions(git_tags(repo_root), published)
updated = updated_config(content, config_path, versions)
else:
content = updated = ""
if args.check:
if content == updated:
if args.check_anchors:
return check_anchors(repo_root, cases_dir)
return 0
try:
display_path = config_path.relative_to(repo_root)
except ValueError:
display_path = config_path
sys.stderr.writelines(
difflib.unified_diff(
content.splitlines(keepends=True),
updated.splitlines(keepends=True),
fromfile=f"a/{display_path}",
tofile=f"b/{display_path}",
)
)
return 1
if content != updated:
config_path.write_text(updated, encoding="utf-8")
if args.check_anchors:
return check_anchors(repo_root, cases_dir)
except (CompatVersionError, OSError) as err:
print(f"error: {err}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())
.github/workflows/checks.yml
+8
View File
@@ -80,6 +80,14 @@ jobs:
run: python3 scripts/check-enterprise-license-test.py
- name: Run fuzz orchestration script tests
run: .github/scripts/run-fuzz-targets-test.sh
- name: Run query-regression Python tooling tests
run: |
python3 tests/perf/test_query_regression_runner_compaction_toctou.py
python3 tests/perf/test_query_regression_runner_otlp_trace_load.py
python3 tests/perf/test_query_regression_summary_otlp.py
python3 tests/perf/test_query_regression_case_selection.py
python3 tests/perf/test_query_regression_nightly_refs.py
python3 tests/perf/test_aliyun_ecs_runner_scripts.py
check:
if: ${{ github.repository == 'GreptimeTeam/greptimedb' }}
.github/workflows/integration.yml
+31 -1
View File
@@ -237,6 +237,23 @@ jobs:
with:
artifact-name: bins
compat-updater-check:
if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name == 'merge_group' || github.event_name == 'pull_request') }}
name: Check compat version updater
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
fetch-depth: 0
- name: Run compat updater unit tests
run: python3 -m unittest discover -s .github/scripts/tests -p 'test_*.py'
- name: Check compat version window and anchors
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 .github/scripts/update-compat-versions.py --check --published-only --check-anchors
compat:
if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name == 'merge_group' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' || (github.event_name == 'pull_request' && (github.event.action == 'ready_for_review' || (github.event.action == 'opened' && github.event.pull_request.draft == false)))) }}
name: Compatibility Test (recent releases)
@@ -247,6 +264,7 @@ jobs:
- uses: actions/checkout@v4
with:
persist-credentials: false
fetch-depth: 0
- name: Download pre-built binaries
uses: actions/download-artifact@v4
with:
@@ -254,8 +272,20 @@ jobs:
path: .
- name: Unzip binaries
run: tar -xvf ./bins.tar.gz
- name: Compute nightly compat window
id: nightly
if: ${{ github.event_name == 'schedule' }}
shell: bash
run: echo "extra=$(python3 .github/scripts/update-compat-versions.py --nightly-window)" >> "$GITHUB_OUTPUT"
- name: Run compatibility test
run: python3 .github/scripts/run-compat.py --preserve-state
env:
NIGHTLY_EXTRA: ${{ steps.nightly.outputs.extra }}
run: |
EXTRA=""
if [[ "${GITHUB_EVENT_NAME}" == "schedule" && -n "${NIGHTLY_EXTRA}" ]]; then
EXTRA="--from-versions-extra=${NIGHTLY_EXTRA}"
fi
python3 .github/scripts/run-compat.py --preserve-state ${EXTRA}
- name: Upload compatibility logs
if: failure()
uses: actions/upload-artifact@v4
.github/workflows/pr-open-limit.yml
+48
View File
@@ -0,0 +1,48 @@
name: PR Open Limit
on:
pull_request_target:
types:
- opened
- reopened
- ready_for_review
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
permissions:
contents: read
pull-requests: write
issues: write
jobs:
check-open-limit:
name: Check Open Pull Request Limit
runs-on: ubuntu-latest
timeout-minutes: 10
# Advisory: a script failure must not put a red cross on the pull request.
continue-on-error: true
if: ${{ github.repository == 'GreptimeTeam/greptimedb' }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
working-directory: .github/scripts
run: npm ci
- name: Check open pull request limit
working-directory: .github/scripts
run: node pr-open-limit.js
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number }}
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
PR_IS_DRAFT: ${{ github.event.pull_request.draft }}
MAX_OPEN_PRS: ${{ vars.MAX_OPEN_PRS_PER_AUTHOR || 5 }}
.github/workflows/query-regression-janitor.yml
+36
View File
@@ -0,0 +1,36 @@
name: Query Regression ECS Janitor
# Safety net for the aliyun-ecs query-regression path: deletes managed ECS
# instances (and their runner registrations) that outlived their workflow run,
# so a failed teardown never leaks pay-as-you-go spend.
on:
schedule:
- cron: "23 3 * * *"
workflow_dispatch:
permissions:
contents: read
jobs:
sweep:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout teardown script
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Sweep expired ECS runners
env:
ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }}
run: >-
uv run .github/scripts/aliyun-ecs-runner-teardown.py
--sweep --sweep-ttl-hours 4
.github/workflows/query-regression-nightly.yml
+104
View File
@@ -0,0 +1,104 @@
name: Query Regression Nightly
# After a successful GreptimeDB Nightly Build, compare that commit against
# the previous successful nightly. The reusable Query Regression workflow
# still compiles both SHAs; this wrapper only resolves which two SHAs.
on:
workflow_run:
workflows:
- GreptimeDB Nightly Build
types:
- completed
workflow_dispatch:
inputs:
base_ref:
description: Base ref/SHA (empty = previous successful nightly)
required: false
type: string
default: ""
candidate_ref:
description: Candidate ref/SHA (empty = latest successful nightly)
required: false
type: string
default: ""
candidate_run_id:
description: Nightly Build run id to treat as candidate (empty = latest)
required: false
type: string
default: ""
case:
description: Query perf case path(s), all, or heavy
required: false
type: string
default: all
permissions:
contents: read
actions: read
jobs:
resolve-refs:
name: Resolve previous vs current nightly SHAs
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
skip: ${{ steps.refs.outputs.skip }}
reason: ${{ steps.refs.outputs.reason }}
base_sha: ${{ steps.refs.outputs.base_sha }}
candidate_sha: ${{ steps.refs.outputs.candidate_sha }}
steps:
- name: Checkout ref resolver
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Resolve nightly SHAs
id: refs
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
CANDIDATE_RUN_ID: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.id || inputs.candidate_run_id || '' }}
BASE_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.base_ref || '' }}
CANDIDATE_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.candidate_ref || '' }}
run: python3 .github/scripts/query-regression-nightly-refs.py
- name: Summarize comparison
if: always()
env:
SKIP: ${{ steps.refs.outputs.skip }}
REASON: ${{ steps.refs.outputs.reason }}
BASE_SHA: ${{ steps.refs.outputs.base_sha }}
CANDIDATE_SHA: ${{ steps.refs.outputs.candidate_sha }}
BASE_RUN_URL: ${{ steps.refs.outputs.base_run_url }}
CANDIDATE_RUN_URL: ${{ steps.refs.outputs.candidate_run_url }}
run: |
set -euo pipefail
{
if [[ "${SKIP}" == "true" ]]; then
printf 'Skipping query-regression nightly: %s\n' "${REASON}"
else
printf 'Comparing previous nightly `%s` -> current nightly `%s`\n' \
"${BASE_SHA}" "${CANDIDATE_SHA}"
if [[ -n "${BASE_RUN_URL}" ]]; then
printf -- '- Previous Nightly Build: %s\n' "${BASE_RUN_URL}"
fi
if [[ -n "${CANDIDATE_RUN_URL}" ]]; then
printf -- '- Current Nightly Build: %s\n' "${CANDIDATE_RUN_URL}"
fi
fi
} | tee -a "${GITHUB_STEP_SUMMARY}"
query-regression:
name: Query regression nightly
needs: [resolve-refs]
if: ${{ needs.resolve-refs.outputs.skip != 'true' && needs.resolve-refs.outputs.base_sha != '' && needs.resolve-refs.outputs.candidate_sha != '' }}
uses: ./.github/workflows/query-regression.yml
secrets: inherit
with:
case: ${{ github.event_name == 'workflow_dispatch' && inputs.case || 'all' }}
base_ref: ${{ needs.resolve-refs.outputs.base_sha }}
candidate_ref: ${{ needs.resolve-refs.outputs.candidate_sha }}
cargo_profile: nightly
runner: aliyun-ecs
.github/workflows/query-regression.yml
+313 -130
View File
@@ -32,10 +32,10 @@ on:
type: string
default: nightly
runner:
description: Self-hosted runner label or ARC runner scale set for this query regression run
description: Self-hosted runner label; aliyun-ecs provisions a fresh ECS instance per run
required: false
type: string
default: perf-regression-8-cores
default: aliyun-ecs
workflow_dispatch:
inputs:
case:
@@ -68,12 +68,20 @@ on:
- release
- dev
runner:
description: Self-hosted runner label or ARC runner scale set for this query regression run
description: >-
Self-hosted runner label; aliyun-ecs provisions a fresh ECS instance
per run, any other value is used as a literal runner label
required: true
type: choice
default: perf-regression-8-cores
options:
- perf-regression-8-cores
type: string
default: aliyun-ecs
keep_instance:
description: >-
Debug: keep the ECS instance after the run (skip teardown) so its
runner _diag logs, journal, and telemetry can be inspected; the
janitor still sweeps it after the TTL
required: false
type: boolean
default: false
pull_request:
types: [labeled]
@@ -81,19 +89,99 @@ permissions:
contents: read
jobs:
query-regression:
test-tooling:
# Stdlib unittests for the workflow Python (case selection, report
# helpers, nightly SHA picking, and rendered ECS user-data). They do
# not talk to Aliyun or the Actions runner process; running them on
# ubuntu-latest fails fast before any ECS spend. Not gated on the
# regression labels: those labels boot a VM, these tests should not.
# Ordinary PRs also run the same tests from checks.yml, because this
# workflow only starts on `labeled` (or dispatch / workflow_call).
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Test query regression tooling
run: |
python3 tests/perf/test_query_regression_runner_compaction_toctou.py
python3 tests/perf/test_query_regression_runner_otlp_trace_load.py
python3 tests/perf/test_query_regression_summary_otlp.py
python3 tests/perf/test_query_regression_case_selection.py
python3 tests/perf/test_query_regression_nightly_refs.py
python3 tests/perf/test_aliyun_ecs_runner_scripts.py
provision:
# Runs when the aliyun-ecs path is selected: explicitly via the runner
# input, or for PR labels by default (a QUERY_REGRESSION_PR_RUNNER
# repository variable set to another value redirects PRs to that literal
# runner label instead). Uses trusted scripts from the PR base (or the
# dispatched ref), never from candidate code. Waits for test-tooling so
# a broken user-data template does not still create a VM.
needs: [test-tooling]
if: >-
${{ github.event_name != 'pull_request' ||
(github.event_name == 'pull_request' &&
!github.event.pull_request.draft &&
(github.event.label.name == 'query-regression' ||
github.event.label.name == 'heavy-regression')) }}
runs-on: ${{ github.event_name != 'pull_request' && inputs.runner || 'perf-regression-8-cores' }}
${{ !failure() && !cancelled() &&
((github.event_name != 'pull_request' && inputs.runner == 'aliyun-ecs') ||
(github.event_name == 'pull_request' &&
!github.event.pull_request.draft &&
(github.event.label.name == 'query-regression' ||
github.event.label.name == 'heavy-regression') &&
(vars.QUERY_REGRESSION_PR_RUNNER || 'aliyun-ecs') == 'aliyun-ecs')) }}
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
label: ${{ steps.provision.outputs.label }}
instance_id: ${{ steps.provision.outputs.instance_id }}
runner_name: ${{ steps.provision.outputs.runner_name }}
steps:
- name: Checkout trusted provisioning scripts
uses: actions/checkout@v4
with:
repository: ${{ github.repository }}
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }}
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Provision ECS runner
id: provision
env:
ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }}
ALIYUN_ECS_VSWITCH_ID: ${{ vars.ALIYUN_ECS_VSWITCH_ID }}
ALIYUN_ECS_SECURITY_GROUP_ID: ${{ vars.ALIYUN_ECS_SECURITY_GROUP_ID }}
ALIYUN_ECS_INSTANCE_TYPE: ${{ vars.ALIYUN_ECS_INSTANCE_TYPE }}
ALIYUN_ECS_RESOURCE_GROUP_ID: ${{ vars.ALIYUN_ECS_RESOURCE_GROUP_ID }}
QUERY_REGRESSION_ECS_IMAGE_ID: ${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }}
QUERY_REGRESSION_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }}
QUERY_REGRESSION_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }}
run: >-
uv run .github/scripts/aliyun-ecs-runner-provision.py
query-regression:
needs: [provision, test-tooling]
# `!failure() && !cancelled()` both suppresses the implicit success() and
# lets the job run when provision was intentionally skipped because a
# literal runner label was selected; a failed or cancelled provision still
# blocks the run because no ECS runner would be waiting.
if: >-
${{ !failure() && !cancelled() &&
(github.event_name != 'pull_request' ||
(github.event_name == 'pull_request' &&
!github.event.pull_request.draft &&
(github.event.label.name == 'query-regression' ||
github.event.label.name == 'heavy-regression'))) }}
runs-on: >-
${{ needs.provision.outputs.label ||
(github.event_name != 'pull_request' && inputs.runner ||
(vars.QUERY_REGRESSION_PR_RUNNER || 'aliyun-ecs')) }}
timeout-minutes: 180
concurrency:
group: query-regression-persistent-cache-v1
queue: max
cancel-in-progress: false
env:
CARGO_PROFILE: ${{ github.event_name == 'pull_request' && 'nightly' || inputs.cargo_profile }}
CARGO_HOME: /home/runner/.cargo
@@ -105,7 +193,8 @@ jobs:
QUERY_REGRESSION_CACHE_META: /home/runner/query-regression-cache-meta
RUSTC_WRAPPER: /usr/local/bin/sccache
SCCACHE_DIR: /home/runner/.cache/sccache
SCCACHE_CACHE_SIZE: 40G
# Caps the local sccache on the system disk next to target and cargo.
SCCACHE_CACHE_SIZE: 10G
CARGO_INCREMENTAL: "0"
RUSTFLAGS: -D warnings -C link-arg=-fuse-ld=mold
QUERY_REGRESSION_CACHE_EPOCH: "1"
@@ -114,7 +203,55 @@ jobs:
EVENT_MERGE_SHA: ${{ github.event_name == 'pull_request' && github.sha || '' }}
EVENT_HEAD_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || '' }}
EVENT_BASE_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || '' }}
# Runner identity contract. The ECS image uses 1001; a manually prepared
# host may override via repo variables when 1001 is already taken.
EXPECTED_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }}
EXPECTED_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }}
steps:
- name: Report provisioned ECS runner
if: ${{ needs.provision.outputs.instance_id != '' }}
shell: bash
run: |
set -euo pipefail
{
printf -- '- ECS instance: `%s` (type `%s`, image `%s`, region `%s`)\n' \
"${{ needs.provision.outputs.instance_id }}" \
"${{ vars.ALIYUN_ECS_INSTANCE_TYPE }}" \
"${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }}" \
"${{ vars.ALIYUN_ECS_REGION_ID }}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Start machine telemetry sampler
shell: bash
# The ECS instance is deleted when the run ends, so machine state must
# be captured while the job runs. A background sampler appends load,
# memory, disk, and top-memory processes to a log every 30s; the
# "Dump machine telemetry" step (if: always()) prints and uploads it.
# Note: when the job is *cancelled*, even always() steps are killed —
# telemetry covers failures, not cancellations.
run: |
set -euo pipefail
log="${GITHUB_WORKSPACE}/machine-telemetry.log"
{
echo "== baseline $(date -u +%Y-%m-%dT%H:%M:%SZ) =="
nproc
free -h
df -h / /home/runner
} >> "${log}"
nohup bash -c "
while true; do
{
date -u '+== %Y-%m-%dT%H:%M:%SZ =='
uptime
free -m
df -h / /home/runner
ps -eo pid,comm,%mem,%cpu --sort=-%mem | head -6
} >> '${log}' 2>&1
sleep 30
done
" >/dev/null 2>&1 &
echo "Telemetry sampler started (pid $!), logging to ${log}"
- name: Checkout base source
uses: actions/checkout@v4
with:
@@ -263,14 +400,64 @@ jobs:
shell: bash
run: |
set -euo pipefail
[[ "$(id -u)" == "1001" ]]
[[ "$(id -g)" == "1001" ]]
[[ "${UV_CACHE_DIR}" == "/home/runner/.cargo/uv-cache" ]]
[[ "$(protoc --version)" == "libprotoc 3.21.12" ]]
errors=()
record() {
printf 'FAIL %s\n' "$*"
errors+=("$*")
}
require_eq() {
local name="$1" actual="$2" expected="$3"
printf 'check %s: %s\n' "${name}" "${actual}"
[[ "${actual}" == "${expected}" ]] || record "${name}: expected '${expected}', got '${actual}'"
}
require_match() {
local name="$1" actual="$2" pattern="$3"
printf 'check %s: %s\n' "${name}" "${actual}"
[[ "${actual}" =~ $pattern ]] || record "${name}: expected to match ${pattern}, got '${actual}'"
}
require() {
local name="$1"
shift
printf 'check %s\n' "${name}"
"$@" || record "${name}"
}
capture() {
local out
if out="$("$@" 2>&1)"; then
printf '%s' "${out}"
else
printf '<failed to run: %s>' "$*"
fi
}
printf 'uid=%s gid=%s PATH=%s CARGO_HOME=%s\n' "$(id -u)" "$(id -g)" "${PATH}" "${CARGO_HOME}"
ls -la "${CARGO_HOME}" 2>&1 || printf '(CARGO_HOME missing)\n'
# Image/host hygiene: CARGO_HOME must be empty of config and
# install-state before cargo/rustup run. Those tools create
# .package-cache / bin on first use, so this cannot come after.
cargo_home_dirty=()
for entry in config config.toml credentials credentials.toml bin .crates.toml .crates2.json .global-cache .package-cache; do
if [[ -e "${CARGO_HOME}/${entry}" ]]; then
cargo_home_dirty+=("${entry}")
record "CARGO_HOME must not contain ${entry}"
fi
done
if (( ${#cargo_home_dirty[@]} > 0 )); then
printf 'CARGO_HOME contents:\n' >&2
ls -la "${CARGO_HOME}" >&2 || true
fi
require_eq uid "$(id -u)" "${EXPECTED_RUNNER_UID}"
require_eq gid "$(id -g)" "${EXPECTED_RUNNER_GID}"
require_eq UV_CACHE_DIR "${UV_CACHE_DIR}" "/home/runner/.cargo/uv-cache"
require_eq protoc "$(capture protoc --version)" "libprotoc 3.21.12"
temporary_proto_dir="$(mktemp --directory)"
trap 'rm -rf "${temporary_proto_dir}"' EXIT
test -r /usr/include/google/protobuf/any.proto
test -r /usr/include/google/protobuf/empty.proto
require "readable /usr/include/google/protobuf/any.proto" \
test -r /usr/include/google/protobuf/any.proto
require "readable /usr/include/google/protobuf/empty.proto" \
test -r /usr/include/google/protobuf/empty.proto
printf '%s\n' \
'syntax = "proto3";' \
'package smoke;' \
@@ -278,47 +465,51 @@ jobs:
'import "google/protobuf/empty.proto";' \
'message Smoke { google.protobuf.Any any = 1; google.protobuf.Empty empty = 2; }' \
> "${temporary_proto_dir}/smoke.proto"
protoc --proto_path="${temporary_proto_dir}" --proto_path=/usr/include \
if protoc --proto_path="${temporary_proto_dir}" --proto_path=/usr/include \
--descriptor_set_out="${temporary_proto_dir}/smoke.pb" \
"${temporary_proto_dir}/smoke.proto"
test -s "${temporary_proto_dir}/smoke.pb"
[[ "$(uv --version)" =~ ^uv[[:space:]]0\.11\.26([[:space:]]|$) ]]
mold_version="$(mold --version)"
[[ "${mold_version}" =~ ^mold[[:space:]]2\.30\.0([[:space:]]|$) ]]
[[ "$(python3 --version)" == "Python 3.12.3" ]]
"${temporary_proto_dir}/smoke.proto"; then
require "protoc smoke descriptor is non-empty" test -s "${temporary_proto_dir}/smoke.pb"
else
record "protoc smoke compile failed"
fi
require_match uv "$(capture uv --version)" '^uv[[:space:]]0\.11\.26([[:space:]]|$)'
require_match mold "$(capture mold --version)" '^mold[[:space:]]2\.40\.4([[:space:]]|$)'
require_eq python3 "$(capture python3 --version)" "Python 3.14.4"
otelgen_path="$(command -v otelgen || true)"
otelgen_version="<unavailable>"
if [[ -n "${otelgen_path}" ]]; then
otelgen_version="$("${otelgen_path}" --version 2>&1 || true)"
otelgen_version="$(capture "${otelgen_path}" --version)"
fi
printf 'otelgen path: %s\n' "${otelgen_path:-<missing>}"
printf 'otelgen version: %s\n' "${otelgen_version}"
[[ "${otelgen_path}" == "/usr/local/bin/otelgen" ]]
[[ "${otelgen_version}" == *"863a3f395d062c7322cc1de08a38774b7fdaa6c8"* ]]
sccache_version="$(sccache --version)"
[[ "${sccache_version}" =~ ^sccache[[:space:]]0\.16\.0([[:space:]]|$) ]]
[[ "$(command -v rustup)" == "/opt/cargo/bin/rustup" ]]
[[ "$(command -v cargo)" == "/opt/cargo/bin/cargo" ]]
[[ "$(command -v rustc)" == "/opt/cargo/bin/rustc" ]]
[[ "$(rustup --version)" =~ ^rustup[[:space:]]1\.29\.0([[:space:]]|$) ]]
cargo_version="$(cargo --version)"
[[ "${cargo_version}" =~ ^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]]
rustc_version="$(rustc --version)"
[[ "${rustc_version}" =~ ^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]]
active_toolchain="$(rustup show active-toolchain)"
[[ "${active_toolchain}" =~ ^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$) ]]
[[ "${RUSTUP_HOME}" == "/opt/rustup" ]]
[[ "${RUSTUP_TOOLCHAIN}" == "nightly-2026-03-21" ]]
[[ "${RUSTUP_AUTO_INSTALL}" == "0" ]]
test -r /opt/rustup && test -x /opt/rustup
test ! -w /opt/rustup
test ! -w /opt/cargo/bin
for entry in config config.toml credentials credentials.toml bin .crates.toml .crates2.json .global-cache .package-cache; do
test ! -e "${CARGO_HOME}/${entry}"
done
require_eq otelgen_path "${otelgen_path}" "/usr/local/bin/otelgen"
printf 'check otelgen_version: %s\n' "${otelgen_version}"
[[ "${otelgen_version}" == *"863a3f395d062c7322cc1de08a38774b7fdaa6c8"* ]] \
|| record "otelgen_version: expected commit 863a3f395d062c7322cc1de08a38774b7fdaa6c8, got '${otelgen_version}'"
require_match sccache "$(capture sccache --version)" '^sccache[[:space:]]0\.16\.0([[:space:]]|$)'
require_eq rustup_path "$(command -v rustup || true)" "/opt/cargo/bin/rustup"
require_eq cargo_path "$(command -v cargo || true)" "/opt/cargo/bin/cargo"
require_eq rustc_path "$(command -v rustc || true)" "/opt/cargo/bin/rustc"
require_match rustup "$(capture rustup --version)" '^rustup[[:space:]]1\.29\.0([[:space:]]|$)'
require_match cargo "$(capture cargo --version)" \
'^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$'
require_match rustc "$(capture rustc --version)" \
'^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$'
require_match active_toolchain "$(capture rustup show active-toolchain)" \
'^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$)'
require_eq RUSTUP_HOME "${RUSTUP_HOME}" "/opt/rustup"
require_eq RUSTUP_TOOLCHAIN "${RUSTUP_TOOLCHAIN}" "nightly-2026-03-21"
require_eq RUSTUP_AUTO_INSTALL "${RUSTUP_AUTO_INSTALL}" "0"
require "readable /opt/rustup" test -r /opt/rustup
require "executable /opt/rustup" test -x /opt/rustup
require "runner cannot write /opt/rustup" test ! -w /opt/rustup
require "runner cannot write /opt/cargo/bin" test ! -w /opt/cargo/bin
mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git"
if (( ${#errors[@]} > 0 )); then
printf '\nVerify runner image tools failed (%d checks):\n' "${#errors[@]}" >&2
printf ' - %s\n' "${errors[@]}" >&2
exit 1
fi
- name: Prepare persistent query regression cache
- name: Prepare query regression cache
shell: bash
working-directory: src
run: |
@@ -362,24 +553,6 @@ jobs:
find "${root}" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
}
clear_cargo_extracted_trees() {
require_expected_root CARGO_REGISTRY "${CARGO_HOME}/registry" "${EXPECTED_CARGO_REGISTRY}"
require_expected_root CARGO_GIT "${CARGO_HOME}/git" "${EXPECTED_CARGO_GIT}"
rm -rf -- "${CARGO_HOME}/registry/src" "${CARGO_HOME}/git/checkouts"
}
cargo_size_kib() {
du -sk -- "${CARGO_HOME}/registry" "${CARGO_HOME}/git" | awk '{ total += $1 } END { print total }'
}
target_size_kib() {
du -sk -- "${CARGO_TARGET_DIR}" | cut -f1
}
free_kib() {
df -Pk "${CARGO_TARGET_DIR}" | awk 'NR == 2 { print $4 }'
}
report_cache_usage() {
du -sh -- "${CARGO_HOME}" "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \
"${RUSTUP_HOME}" "${CARGO_TARGET_DIR}" "${QUERY_REGRESSION_CACHE_META}" "${SCCACHE_DIR}"
@@ -400,7 +573,7 @@ jobs:
printf 'Refusing unexpected RUSTC_WRAPPER: %s\n' "${RUSTC_WRAPPER}" >&2
exit 1
}
[[ "${SCCACHE_CACHE_SIZE}" == "40G" ]] || {
[[ "${SCCACHE_CACHE_SIZE}" == "10G" ]] || {
printf 'Refusing unexpected SCCACHE_CACHE_SIZE: %s\n' "${SCCACHE_CACHE_SIZE}" >&2
exit 1
}
@@ -487,46 +660,10 @@ jobs:
report_cache_usage
target_size="$(target_size_kib)"
if (( target_size >= 400 * 1024 * 1024 )); then
printf 'Warning: target cache is at least 400 GiB (%s KiB)\n' "${target_size}" >&2
fi
if (( target_size >= 450 * 1024 * 1024 )); then
printf 'Target cache is at least 450 GiB; clearing complete target root\n' >&2
clear_directory "${CARGO_TARGET_DIR}"
fi
cargo_size="$(cargo_size_kib)"
if (( cargo_size >= 60 * 1024 * 1024 )); then
printf 'Warning: Cargo cache is at least 60 GiB (%s KiB)\n' "${cargo_size}" >&2
fi
if (( cargo_size >= 80 * 1024 * 1024 )); then
printf 'Cargo cache is at least 80 GiB; removing extracted sources and checkouts\n' >&2
clear_cargo_extracted_trees
cargo_size="$(cargo_size_kib)"
if (( cargo_size >= 80 * 1024 * 1024 )); then
printf 'Cargo cache remains at least 80 GiB after cleanup (%s KiB)\n' "${cargo_size}" >&2
exit 1
fi
fi
free_space="$(free_kib)"
if (( free_space < 300 * 1024 * 1024 )); then
printf 'Free space is below 300 GiB; clearing complete target root\n' >&2
clear_directory "${CARGO_TARGET_DIR}"
free_space="$(free_kib)"
if (( free_space < 300 * 1024 * 1024 )); then
printf 'Free space remains below 300 GiB; removing Cargo extracted sources and checkouts\n' >&2
clear_cargo_extracted_trees
free_space="$(free_kib)"
if (( free_space < 300 * 1024 * 1024 )); then
printf 'Free space remains below 300 GiB after cleanup (%s KiB)\n' "${free_space}" >&2
exit 1
fi
fi
fi
report_cache_usage
# Every run starts from a fresh system disk, so size/free-space
# watermarks from the retired retained-disk era are not restored.
# A run that overflows the disk fails the build outright, which the
# telemetry step makes diagnosable.
sccache --start-server
sccache --zero-stats
@@ -567,24 +704,17 @@ jobs:
git reset --hard "${VERIFIED_CANDIDATE_SHA}"
git clean -ffdx
- name: Test query regression tooling
working-directory: src
run: |
uv run --no-project python tests/perf/test_query_regression_runner_compaction_toctou.py
uv run --no-project python tests/perf/test_query_regression_runner_otlp_trace_load.py
uv run --no-project python tests/perf/test_query_regression_summary_otlp.py
uv run --no-project python tests/perf/test_query_regression_case_selection.py
- name: Build candidate greptime and fixture generators
- name: Build candidate greptime and query regression helpers
working-directory: src
run: |
set -euo pipefail
SECONDS=0
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_regression_runner --features dev-tools
candidate_build_elapsed="${SECONDS}"
printf 'Candidate greptime and fixture generator cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}"
printf -- '- Candidate greptime and fixture generator cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}"
printf 'Candidate greptime and query regression helper cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}"
printf -- '- Candidate greptime and query regression helper cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}"
sccache --show-stats
target_dir="${CARGO_PROFILE}"
if [[ "${CARGO_PROFILE}" == "dev" ]]; then
@@ -595,6 +725,8 @@ jobs:
"${GITHUB_WORKSPACE}/query-regression-bins/candidate/greptime"
cp "${CARGO_TARGET_DIR}/${target_dir}/query_perf_fixture" \
"${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_perf_fixture"
cp "${CARGO_TARGET_DIR}/${target_dir}/query_regression_runner" \
"${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_regression_runner"
- name: Run query regression
id: run
@@ -609,6 +741,7 @@ jobs:
FIXTURE_GENERATOR: ${{ github.workspace }}/query-regression-bins/candidate/query_perf_fixture
OTELGEN_BIN: /usr/local/bin/otelgen
SUMMARY_SCRIPT: ${{ github.event_name == 'pull_request' && 'query-regression-trusted-scripts/query-regression-summary.py' || 'src/.github/scripts/query-regression-summary.py' }}
QUERY_REGRESSION_RUNNER: ${{ github.workspace }}/query-regression-bins/candidate/query_regression_runner
run: >-
uv run --no-project python src/.github/scripts/query-regression-run.py
--base-src src
@@ -639,6 +772,7 @@ jobs:
query-regression-work/**/logs/**
query-regression-work/**/otelgen/**
query-regression-summary.md
machine-telemetry.log
if-no-files-found: warn
retention-days: 7
@@ -653,7 +787,7 @@ jobs:
if-no-files-found: warn
retention-days: 7
- name: Report persistent cache usage
- name: Report cache usage
if: ${{ always() }}
shell: bash
run: |
@@ -693,6 +827,55 @@ jobs:
printf 'sccache is unavailable (report only)\n' >&2
fi
- name: Dump machine telemetry
if: ${{ always() }}
shell: bash
run: |
log="${GITHUB_WORKSPACE}/machine-telemetry.log"
if [[ -f "${log}" ]]; then
echo "::group::Machine telemetry (last 200 lines)"
tail -n 200 "${log}"
echo "::endgroup::"
else
echo "No telemetry log found (sampler never started?)"
fi
echo "::group::dmesg tail (OOM killer records)"
sudo dmesg -T 2>/dev/null | tail -n 50 || dmesg -T 2>/dev/null | tail -n 50 || \
echo "dmesg unavailable without root"
echo "::endgroup::"
- name: Fail on regression failure
if: ${{ steps.run.outputs.status != '0' }}
run: exit 1
teardown:
# Releases the dynamically provisioned ECS runner. Runs even when the
# benchmark job fails or is cancelled; skipped when a literal runner label
# was selected because the provision outputs are empty, or when the
# dispatch set keep_instance to preserve the machine for post-mortem
# debugging (the janitor sweep still reclaims it after the TTL).
if: ${{ always() && needs.provision.outputs.instance_id != '' && !inputs.keep_instance }}
needs: [provision, query-regression]
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout trusted teardown scripts
uses: actions/checkout@v4
with:
repository: ${{ github.repository }}
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }}
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Teardown ECS runner
env:
ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }}
QUERY_REGRESSION_ECS_INSTANCE_ID: ${{ needs.provision.outputs.instance_id }}
QUERY_REGRESSION_ECS_RUNNER_NAME: ${{ needs.provision.outputs.runner_name }}
run: >-
uv run .github/scripts/aliyun-ecs-runner-teardown.py
.github/workflows/release-dev-builder-images.yaml
+49 -1
View File
@@ -24,6 +24,11 @@ on:
description: Release dev-builder-android image
required: false
default: false
release_dev_builder_riscv64_image:
type: boolean
description: Release dev-builder-riscv64 image
required: false
default: false
update_dev_builder_image_tag:
type: boolean
description: Update the DEV_BUILDER_IMAGE_TAG in Makefile and create a PR
@@ -36,7 +41,7 @@ jobs:
# The jobs are triggered by the following events:
# 1. Manually triggered workflow_dispatch event
# 2. Push event when the PR that modifies the `rust-toolchain.toml` or `docker/dev-builder/**` is merged to main
if: ${{ github.event_name == 'push' || inputs.release_dev_builder_ubuntu_image || inputs.release_dev_builder_centos_image || inputs.release_dev_builder_android_image }}
if: ${{ github.event_name == 'push' || inputs.release_dev_builder_ubuntu_image || inputs.release_dev_builder_centos_image || inputs.release_dev_builder_android_image || inputs.release_dev_builder_riscv64_image }}
runs-on: ubuntu-latest
outputs:
version: ${{ steps.set-version.outputs.version }}
@@ -68,6 +73,7 @@ jobs:
build-dev-builder-ubuntu: ${{ inputs.release_dev_builder_ubuntu_image || github.event_name == 'push' }}
build-dev-builder-centos: ${{ inputs.release_dev_builder_centos_image || github.event_name == 'push' }}
build-dev-builder-android: ${{ inputs.release_dev_builder_android_image || github.event_name == 'push' }}
build-dev-builder-riscv64: ${{ inputs.release_dev_builder_riscv64_image || github.event_name == 'push' }}
release-dev-builder-images-ecr:
name: Release dev builder images to AWS ECR
@@ -154,6 +160,35 @@ jobs:
copy -a docker://docker.io/$IMAGE_NAMESPACE/dev-builder-android:latest \
docker://$ECR_IMAGE_REGISTRY/$ECR_IMAGE_NAMESPACE/dev-builder-android:latest
- name: Push dev-builder-riscv64 image
shell: bash
if: ${{ inputs.release_dev_builder_riscv64_image || github.event_name == 'push' }}
env:
IMAGE_VERSION: ${{ needs.release-dev-builder-images.outputs.version }}
IMAGE_NAMESPACE: ${{ vars.IMAGE_NAMESPACE }}
ECR_IMAGE_REGISTRY: ${{ vars.ECR_IMAGE_REGISTRY }}
ECR_IMAGE_NAMESPACE: ${{ vars.ECR_IMAGE_NAMESPACE }}
run: |
docker run -v "${DOCKER_CONFIG:-$HOME/.docker}:/root/.docker:ro" \
-e "REGISTRY_AUTH_FILE=/root/.docker/config.json" \
quay.io/skopeo/stable:latest \
copy -a docker://docker.io/$IMAGE_NAMESPACE/dev-builder-riscv64:$IMAGE_VERSION \
docker://$ECR_IMAGE_REGISTRY/$ECR_IMAGE_NAMESPACE/dev-builder-riscv64:$IMAGE_VERSION
- name: Push latest dev-builder-riscv64 image
shell: bash
if: ${{ github.event_name == 'push' }}
env:
IMAGE_NAMESPACE: ${{ vars.IMAGE_NAMESPACE }}
ECR_IMAGE_REGISTRY: ${{ vars.ECR_IMAGE_REGISTRY }}
ECR_IMAGE_NAMESPACE: ${{ vars.ECR_IMAGE_NAMESPACE }}
run: |
docker run -v "${DOCKER_CONFIG:-$HOME/.docker}:/root/.docker:ro" \
-e "REGISTRY_AUTH_FILE=/root/.docker/config.json" \
quay.io/skopeo/stable:latest \
copy -a docker://docker.io/$IMAGE_NAMESPACE/dev-builder-riscv64:latest \
docker://$ECR_IMAGE_REGISTRY/$ECR_IMAGE_NAMESPACE/dev-builder-riscv64:latest
release-dev-builder-images-cn: # Note: Be careful issue: https://github.com/containers/skopeo/issues/1874 and we decide to use the latest stable skopeo container.
name: Release dev builder images to CN region
runs-on: ubuntu-latest
@@ -209,6 +244,19 @@ jobs:
quay.io/skopeo/stable:latest \
copy -a docker://docker.io/$IMAGE_NAMESPACE/dev-builder-android:$IMAGE_VERSION \
docker://$ACR_IMAGE_REGISTRY/$IMAGE_NAMESPACE/dev-builder-android:$IMAGE_VERSION
- name: Push dev-builder-riscv64 image
shell: bash
if: ${{ inputs.release_dev_builder_riscv64_image || github.event_name == 'push' }}
env:
IMAGE_VERSION: ${{ needs.release-dev-builder-images.outputs.version }}
IMAGE_NAMESPACE: ${{ vars.IMAGE_NAMESPACE }}
ACR_IMAGE_REGISTRY: ${{ vars.ACR_IMAGE_REGISTRY }}
run: |
docker run -v "${DOCKER_CONFIG:-$HOME/.docker}:/root/.docker:ro" \
-e "REGISTRY_AUTH_FILE=/root/.docker/config.json" \
quay.io/skopeo/stable:latest \
copy -a docker://docker.io/$IMAGE_NAMESPACE/dev-builder-riscv64:$IMAGE_VERSION \
docker://$ACR_IMAGE_REGISTRY/$IMAGE_NAMESPACE/dev-builder-riscv64:$IMAGE_VERSION
update-dev-builder-image-tag:
name: Update dev-builder image tag
.github/workflows/release.yml
+61 -2
View File
@@ -52,7 +52,7 @@ on:
default: true
build_linux_artifacts:
type: boolean
description: Build linux artifacts (both amd64 and arm64)
description: Build linux artifacts (amd64, arm64, and riscv64)
required: false
default: false
build_macos_artifacts:
@@ -225,6 +225,52 @@ jobs:
image-namespace: ${{ vars.ECR_IMAGE_NAMESPACE }}
large-page-size: true
build-linux-riscv64-artifacts:
name: Build linux-riscv64 artifacts
if: ${{ inputs.build_linux_artifacts || inputs.publish_github_release || inputs.release_images || github.event_name == 'push' || github.event_name == 'schedule' }}
needs: [
allocate-runners,
]
runs-on: ${{ needs.allocate-runners.outputs.linux-amd64-runner }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false
- name: Prepare riscv64 dev-builder image
shell: bash
env:
IMAGE_REGISTRY: ${{ vars.ECR_IMAGE_REGISTRY }}
IMAGE_NAMESPACE: ${{ vars.ECR_IMAGE_NAMESPACE }}
run: |
builder_tag=$(grep '^DEV_BUILDER_IMAGE_TAG ?=' Makefile | awk '{print $3}')
builder_image="${IMAGE_REGISTRY}/${IMAGE_NAMESPACE}/dev-builder-riscv64:${builder_tag}"
if ! docker pull "${builder_image}"; then
make dev-builder-riscv64 \
IMAGE_REGISTRY="${IMAGE_REGISTRY}" \
IMAGE_NAMESPACE="${IMAGE_NAMESPACE}"
fi
# Cross-compile greptime for riscv64gc-unknown-linux-gnu on the amd64 runner
# with the riscv64 cross dev-builder image. Integration tests are skipped
# because the cross-compiled binary cannot run on the amd64 host.
- name: Build greptime riscv64 binary
shell: bash
run: |
make build-riscv64-bin \
CARGO_PROFILE=${{ env.CARGO_PROFILE }} \
FEATURES=servers/dashboard \
IMAGE_REGISTRY=${{ vars.ECR_IMAGE_REGISTRY }} \
IMAGE_NAMESPACE=${{ vars.ECR_IMAGE_NAMESPACE }}
- name: Upload artifacts
uses: ./.github/actions/upload-artifacts
with:
artifacts-dir: greptime-linux-riscv64-${{ needs.allocate-runners.outputs.version }}
target-files: ./target/riscv64gc-unknown-linux-gnu/${{ env.CARGO_PROFILE }}/greptime
version: ${{ needs.allocate-runners.outputs.version }}
build-macos-artifacts:
name: Build macOS artifacts
strategy:
@@ -464,7 +510,7 @@ jobs:
base_ref: ${{ needs.prepare-release-validation.outputs.previous-release-tag }}
candidate_ref: ${{ needs.prepare-release-validation.outputs.candidate-ref }}
cargo_profile: nightly
runner: perf-regression-8-cores
runner: aliyun-ecs
release-images-to-dockerhub:
name: Build and push images to DockerHub
@@ -520,6 +566,7 @@ jobs:
build-linux-amd64-artifacts,
build-linux-arm64-artifacts,
build-macos-artifacts,
build-linux-riscv64-artifacts,
build-windows-artifacts,
release-images-to-dockerhub,
]
@@ -568,6 +615,7 @@ jobs:
(needs.prepare-release-validation.outputs.run-query-regression == 'false' && needs.query-regression-release.result == 'skipped')) &&
needs.build-linux-amd64-artifacts.result == 'success' &&
needs.build-linux-arm64-artifacts.result == 'success' &&
needs.build-linux-riscv64-artifacts.result == 'success' &&
(needs.build-macos-artifacts.result == 'success' || needs.build-macos-artifacts.result == 'skipped') &&
(needs.build-windows-artifacts.result == 'success' || needs.build-windows-artifacts.result == 'skipped') &&
(needs.release-images-to-dockerhub.result == 'success' ||
@@ -581,6 +629,7 @@ jobs:
build-linux-amd64-artifacts,
build-linux-arm64-artifacts,
build-macos-artifacts,
build-linux-riscv64-artifacts,
build-windows-artifacts,
release-images-to-dockerhub,
prepare-release-validation,
@@ -599,6 +648,15 @@ jobs:
with:
version: ${{ needs.allocate-runners.outputs.version }}
update-compat-versions:
name: Update compatibility versions
needs: [allocate-runners, publish-github-release]
if: ${{ needs.publish-github-release.result == 'success' }}
uses: ./.github/workflows/update-compat-versions.yml
with:
release_tag: ${{ needs.allocate-runners.outputs.version }}
secrets: inherit
### Stop runners ###
# It's very necessary to split the job of releasing runners into 'stop-linux-amd64-runner' and 'stop-linux-arm64-runner'.
# Because we can terminate the specified EC2 instance immediately after the job is finished without unnecessary waiting.
@@ -610,6 +668,7 @@ jobs:
needs: [
allocate-runners,
build-linux-amd64-artifacts,
build-linux-riscv64-artifacts,
]
steps:
- name: Checkout
.github/workflows/rust.yml
+27 -3
View File
@@ -96,6 +96,32 @@ jobs:
- name: Check unused dependencies
run: make check-udeps
check-riscv64:
if: ${{ github.repository == 'GreptimeTeam/greptimedb' }}
name: Check riscv64 build
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: arduino/setup-protoc@v3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install riscv64 cross toolchain
run: sudo apt-get update && sudo apt-get install -y gcc-riscv64-linux-gnu g++-riscv64-linux-gnu
- uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Add riscv64 target
run: rustup target add riscv64gc-unknown-linux-gnu
- name: Rust Cache
uses: Swatinem/rust-cache@v2
with:
shared-key: "check-riscv64"
cache-all-crates: "true"
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run cargo check for riscv64
run: cargo check --locked --workspace --all-targets --features servers/dashboard --target riscv64gc-unknown-linux-gnu
conflict-check:
if: ${{ github.repository == 'GreptimeTeam/greptimedb' }}
name: Check for conflict
@@ -139,9 +165,7 @@ jobs:
run: ../../.github/scripts/pull-test-deps-images.sh && docker compose up -d --wait
- name: Run nextest cases
run: |
cargo nextest run --workspace -F dashboard -F pg_kvbackend -F mysql_kvbackend -F vector_index
cargo nextest run -p tests-integration --test main --features enterprise -E 'test(=table_ddl_event::test_table_ddl_procedure_events)'
run: cargo nextest run --workspace -F dashboard -F pg_kvbackend -F mysql_kvbackend -F vector_index
env:
CARGO_BUILD_RUSTFLAGS: "-C link-arg=-fuse-ld=mold"
RUST_BACKTRACE: 1
.github/workflows/update-compat-versions.yml
+136
View File
@@ -0,0 +1,136 @@
name: Update compatibility versions
on:
workflow_call:
inputs:
release_tag:
description: Stable release tag that triggered the caller.
required: true
type: string
release:
types: [published]
schedule:
# Release events created with GITHUB_TOKEN do not trigger workflows.
- cron: "17 3 * * *"
workflow_dispatch:
concurrency:
group: update-compat-versions
cancel-in-progress: false
jobs:
validate-release:
if: ${{ github.repository == 'GreptimeTeam/greptimedb' }}
runs-on: ubuntu-latest
outputs:
should-run: ${{ steps.gate.outputs.should-run }}
steps:
- id: gate
shell: bash
env:
EVENT_NAME: ${{ github.event_name }}
RELEASE_DRAFT: ${{ github.event.release.draft }}
RELEASE_PRERELEASE: ${{ github.event.release.prerelease }}
RELEASE_TAG: ${{ github.event.release.tag_name }}
WORKFLOW_RELEASE_TAG: ${{ inputs.release_tag }}
run: |
set -euo pipefail
should_run=true
stable_tag_re='^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$'
if [[ -n "${WORKFLOW_RELEASE_TAG}" ]]; then
if [[ ! "${WORKFLOW_RELEASE_TAG}" =~ ${stable_tag_re} ]]; then
should_run=false
fi
elif [[ "${EVENT_NAME}" == "release" ]]; then
if [[ "${RELEASE_DRAFT}" == "true" || "${RELEASE_PRERELEASE}" == "true" || ! "${RELEASE_TAG}" =~ ${stable_tag_re} ]]; then
should_run=false
fi
elif [[ "${EVENT_NAME}" == "workflow_call" ]]; then
should_run=false
fi
echo "should-run=${should_run}" >> "${GITHUB_OUTPUT}"
update:
needs: validate-release
if: ${{ github.repository == 'GreptimeTeam/greptimedb' && needs.validate-release.outputs.should-run == 'true' }}
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
env:
BRANCH: ci/update-compat-versions
steps:
# Always use main: release events otherwise provide the release-tag revision.
- name: Checkout main
uses: actions/checkout@v4
with:
ref: main
fetch-depth: 0
token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
- name: Validate updater on main
shell: bash
env:
GH_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
run: |
set -euo pipefail
python3 -m unittest discover -s .github/scripts/tests -p 'test_*.py'
python3 -m py_compile .github/scripts/update-compat-versions.py
# Log-only: a stale window is expected here (it is exactly what the
# updater fixes below), so a stale --check must not fail the job.
python3 .github/scripts/update-compat-versions.py --check --published-only || true
python3 .github/scripts/update-compat-versions.py --check-anchors
- name: Prepare updater branch
id: prepare
shell: bash
env:
GH_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
run: |
set -euo pipefail
git fetch origin main
git config user.name "greptimedb-ci"
git config user.email "greptimedb-ci@users.noreply.github.com"
remote_oid="$(git ls-remote --heads origin "refs/heads/${BRANCH}" | cut -f1)"
if [[ -n "${remote_oid}" ]]; then
git fetch origin "refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}"
fi
git checkout -B "${BRANCH}" origin/main
git reset --hard origin/main
python3 .github/scripts/update-compat-versions.py --update --published-only
if git diff --quiet -- tests/compatibility/ci.toml; then
echo "Compatibility version window is already current."
exit 0
fi
desired_blob="$(git hash-object tests/compatibility/ci.toml)"
if [[ -n "${remote_oid}" ]]; then
remote_blob="$(git rev-parse "${remote_oid}:tests/compatibility/ci.toml" 2>/dev/null || true)"
if [[ "${desired_blob}" == "${remote_blob}" ]]; then
echo "Remote updater branch already has the desired compatibility window."
echo "branch-ready=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
fi
git add tests/compatibility/ci.toml
git commit -s -m "chore(ci): update compatibility versions"
git push --force-with-lease="refs/heads/${BRANCH}:${remote_oid}" origin "HEAD:refs/heads/${BRANCH}"
echo "branch-ready=true" >> "${GITHUB_OUTPUT}"
- name: Create compatibility version PR
if: ${{ steps.prepare.outputs.branch-ready == 'true' }}
shell: bash
env:
GH_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
run: |
set -euo pipefail
open_prs="$(gh pr list --head "${BRANCH}" --base main --state open --json number --jq 'length')"
if [[ "${open_prs}" != "0" ]]; then
echo "An open compatibility updater PR already exists."
exit 0
fi
gh pr create \
--base main \
--head "${BRANCH}" \
--title "chore(ci): update compatibility versions" \
--reviewer "GreptimeTeam/db-approver" \
--body $'I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).\n\n## Refer to a related PR or issue link (optional)\n\nAutomated after a stable release or the daily fallback.\n\n## What\047s changed and what\047s your intention?\n\nUpdates `tests/compatibility/ci.toml` from checked-out `main` tags that have published release assets. The PR window keeps the latest patch of the two most recent stable minor lines; exact `=vX.Y.Z` case anchors are validated nightly instead of being added to the PR window.\n\n## PR Checklist\n\n- [ ] I have written the necessary rustdoc comments.\n- [ ] I have added the necessary unit tests and integration tests.\n- [ ] This PR requires documentation updates.\n- [x] API changes are backward compatible.\n- [x] Schema or data changes are backward compatible.'
AGENTS.md
+56 -12
View File
@@ -18,7 +18,8 @@ provides sub-second querying at PB scale with high cost efficiency.
| Build (debug) | `make build` |
| Build (release) | `make build RELEASE=true` |
| Run standalone | `cargo run -- standalone start` |
| Test | `cargo nextest run` (preferred over `cargo test`) |
| Targeted Rust tests | `cargo nextest run -p <package>` (preferred over `cargo test`) |
| Full local Rust suite | `make test` |
| SQL tests | `cargo sqlness bare` (single case: `cargo sqlness bare -t <name>`) |
| Format | `make fmt` (and `make fmt-toml` for TOML) |
| Lint | `make clippy` (= `cargo clippy --workspace --all-targets --all-features -- -D warnings`) |
@@ -43,14 +44,30 @@ GreptimeDB is a Cargo workspace rooted at the repository; most crates live under
- **Shared**: `src/common/`, `src/datatypes/`, `src/store-api/` (engine contract),
`src/catalog/`, `src/table/`
Hot crates carry their own `AGENTS.md` with a module map, read/write paths,
change-coupling points, and gotchas:
Before editing a path, read every `AGENTS.md` from the repository root down to
that path. The root guide always applies; a nested guide adds or overrides rules
for its subtree. Changes spanning multiple subtrees must follow every applicable
guide.
`AGENTS.md` files are maps, not manuals. Keep them short and stable: record
where code lives, important boundaries or change-coupling, high-cost gotchas,
and the validation entry point. Implementation details belong in code. Do not
copy exhaustive test lists or workflow logic; link to the source of truth.
High-change areas and specialized test suites carry their own `AGENTS.md` with
a module map, read/write paths, change-coupling points, and gotchas:
- [`src/common/meta/AGENTS.md`](src/common/meta/AGENTS.md)
- [`src/query/AGENTS.md`](src/query/AGENTS.md)
- [`src/servers/AGENTS.md`](src/servers/AGENTS.md)
- [`src/operator/AGENTS.md`](src/operator/AGENTS.md)
- [`src/mito2/AGENTS.md`](src/mito2/AGENTS.md)
- [`src/metric-engine/AGENTS.md`](src/metric-engine/AGENTS.md)
- [`src/flow/AGENTS.md`](src/flow/AGENTS.md)
- [`src/frontend/AGENTS.md`](src/frontend/AGENTS.md)
- [`src/meta-srv/AGENTS.md`](src/meta-srv/AGENTS.md)
- [`tests/compatibility/AGENTS.md`](tests/compatibility/AGENTS.md)
- [`tests/perf/AGENTS.md`](tests/perf/AGENTS.md)
## Read before changing code
@@ -71,24 +88,51 @@ change-coupling points, and gotchas:
- Error handling: `src/common/error/` (`ErrorExt`, `StatusCode`)
- Protocol implementations: `src/servers/src/`
## Worktree safety
- Check `git status --short` before editing. Preserve unrelated tracked changes
and untracked files; re-read files changed by another process.
- Do not rewrite history, force-push, or remove files in bulk unless the user
explicitly requests it.
- Update generated artifacts only through the generators documented in
[`.agents/generated-files.md`](.agents/generated-files.md).
## Validation by change type
Use the narrowest command that covers the change, then expand only when its
blast radius requires it.
| Change | Minimum validation |
| --- | --- |
| One Rust crate | `cargo nextest run -p <package>` |
| Cross-workspace Rust behavior | `make test`; inspect `.github/workflows/rust.yml` when CI parity matters |
| SQL parsing, planning, execution, or output | `cargo sqlness bare -t <case>`; inspect regenerated `.result` files |
| Persisted metadata or wire format | Add/run a case under `tests/compatibility/`; follow its `README.md` and `AGENTS.md` |
| Public configuration | Update example TOMLs, loading/serialization snapshots, and docs; run `make config-docs` |
| Query regression harness or DSL | Follow `tests/perf/AGENTS.md` |
| Enterprise-gated code | Build/test with `--features enterprise` where applicable and run `make check-enterprise-license` |
## Before opening a PR
1. `make fmt`
2. `make clippy`
3. `make test` (or `cargo nextest run`)
4. `make check-udeps` (run `make fix-udeps` if it reports unused dependencies).
5. If you added or changed a public configuration option, update the applicable
1. If you added a `.rs`, `.py`, or `.ts` file, apply and verify its license
header with `hawkeye format` followed by `hawkeye check`. Use the inception
year from `licenserc.toml`, not the current year.
2. `make fmt`
3. `make clippy`
4. `make test`
5. `make check-udeps` (run `make fix-udeps` if it reports unused dependencies).
6. If you added or changed a public configuration option, update the applicable
example TOMLs, configuration-loading and serialized-config snapshot tests,
and related user-facing documentation. Run `make config-docs` (needs Docker)
and commit the regenerated `config/config.md`.
6. If you changed a persisted or wire format, add a compatibility test case (see
7. If you changed a persisted or wire format, add a compatibility test case (see
`.agents/architecture-invariants.md`).
7. If you added or gated an enterprise-only file, give it the enterprise license
8. If you added or gated an enterprise-only file, give it the enterprise license
header, list it in `licenserc-enterprise.toml` (`includes`) and
`licenserc.toml` (`excludes`), and run `make check-enterprise-license`.
8. Use a conventional-commit title, sign off commits (`git commit -s`), and sign
9. Use a conventional-commit title, sign off commits (`git commit -s`), and sign
the CLA.
9. When creating or updating a pull request, follow
10. When creating or updating a pull request, follow
[`.github/pull_request_template.md`](.github/pull_request_template.md): include
the CLA statement, fill the change-intention section with enough detail, and
update checklist items accurately.
Cargo.lock
Generated
+598 -394
View File
File diff suppressed because it is too large Load Diff
Cargo.toml
+29 -20
View File
@@ -75,7 +75,7 @@ members = [
resolver = "2"
[workspace.package]
version = "1.2.0"
version = "1.3.0"
edition = "2024"
license = "Apache-2.0"
@@ -159,7 +159,7 @@ fs2 = "0.4"
fst = "0.4.7"
futures = "0.3"
futures-util = "0.3"
greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "032510ded061277b7d1bbb29d4046abb5b1cbf4b" }
greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "32f467fa2ba2b3588a58381a24af83de09fbb00a" }
hex = "0.4"
http = "1"
humantime = "2.1"
@@ -182,7 +182,7 @@ notify = "8.0"
num_cpus = "1.16"
object_store_opendal = "0.57"
once_cell = "1.18"
opentelemetry-proto = { version = "0.31", features = [
opentelemetry-proto = { version = "0.32", features = [
"gen-tonic",
"metrics",
"trace",
@@ -195,6 +195,9 @@ otel-arrow-rust = { git = "https://github.com/GreptimeTeam/otel-arrow", rev = "5
] }
parking_lot = "0.12"
parquet = { version = "58.3", default-features = false, features = ["arrow", "async", "object_store"] }
parquet-variant = "58.3"
parquet-variant-compute = "58.3"
parquet-variant-json = "58.3"
paste = "1.0"
pin-project = "1.0"
pretty_assertions = "1.4.0"
@@ -202,6 +205,8 @@ prometheus = { version = "0.14", features = ["process"] }
promql-parser = { version = "0.10.0", features = ["ser"] }
prost = { version = "0.14", features = ["no-recursion-limit"] }
prost-types = "0.14"
protoc-bin-vendored = "3.2"
protoc-rust = "2.28"
raft-engine = { version = "0.4.1", default-features = false }
rand = "0.9"
ratelimit = "0.10"
@@ -226,10 +231,12 @@ rstest_reuse = "0.7"
rust_decimal = "1.33"
rustc-hash = "2.0"
rustls = { version = "0.23.25", default-features = false }
ryu = "1.0"
sea-query = "0.32"
serde = { version = "1.0", features = ["derive"] }
serde_json = { version = "1.0", features = ["float_roundtrip"] }
serde_with = "3"
serde_yaml_ng = "0.10"
simd-json = "0.15"
similar-asserts = "1.6.0"
smallvec = { version = "1", features = ["serde"] }
@@ -248,7 +255,7 @@ strum = { version = "0.27", features = ["derive"] }
sysinfo = "0.33"
tempfile = "3"
tokio = { version = "1.47", features = ["full"] }
tokio-postgres = "0.7"
tokio-postgres = "0.7.18"
tokio-rustls = { version = "0.26.2", default-features = false }
tokio-stream = "0.1"
tokio-util = { version = "0.7", features = ["io-util", "compat"] }
@@ -260,11 +267,13 @@ tower = "0.5"
tower-http = "0.6"
tracing = "0.1"
tracing-appender = "0.2"
tracing-opentelemetry = "0.31.0"
tracing-opentelemetry = "0.33.0"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json", "fmt"] }
typetag = "0.2"
uddsketch-rs = "0.1.0"
uuid = { version = "1.17", features = ["serde", "v4", "v7", "fast-rng"] }
vrl = "0.33"
vu128 = "1.1"
zstd = "0.13"
# DO_NOT_REMOVE_THIS: END_OF_EXTERNAL_DEPENDENCIES
@@ -343,21 +352,21 @@ git = "https://github.com/GreptimeTeam/greptime-meter.git"
rev = "5618e779cf2bb4755b499c630fba4c35e91898cb"
[patch.crates-io]
datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "6d6ae9a8073900bdf89d098d491d4dc0269699b0" }
datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "4c8a6bf28347b346348136247eb67dfb933f1147" }
sqlparser = { git = "https://github.com/GreptimeTeam/sqlparser-rs.git", rev = "2aefa08a8d69c96eec2d6d6703598a009bba6e4c" } # on branch v0.61.x
[profile.release]
Makefile
+27 -1
View File
@@ -8,7 +8,10 @@ CARGO_BUILD_OPTS := --locked
IMAGE_REGISTRY ?= docker.io
IMAGE_NAMESPACE ?= greptime
IMAGE_TAG ?= latest
DEV_BUILDER_IMAGE_TAG ?= 2026-03-21-9c9d9e9e-20260331090344
DEV_BUILDER_IMAGE_TAG ?= 2026-03-21-7fa6f5f9-20260810091725
DEV_BUILDER_RISCV64_IMAGE_TAG ?= $(DEV_BUILDER_IMAGE_TAG)
DEV_BUILDER_RISCV64_BASE_IMAGE ?= ubuntu:22.04
RISCV64_TARGET ?= riscv64gc-unknown-linux-gnu
BUILDX_MULTI_PLATFORM_BUILD ?= false
BUILDX_BUILDER_NAME ?= gtbuilder
BASE_IMAGE ?= ubuntu
@@ -98,6 +101,21 @@ build-by-dev-builder: ## Build greptime by dev-builder.
RELEASE=${RELEASE} \
CARGO_BUILD_EXTRA_OPTS="${CARGO_BUILD_EXTRA_OPTS}"
.PHONY: build-riscv64-bin
build-riscv64-bin: ## Build greptime binary for riscv64 (linux-gnu) by the riscv64 cross dev-builder image.
docker run --network=host \
${ASSEMBLED_EXTRA_BUILD_ENV} \
-v ${PWD}:/greptimedb -v ${CARGO_REGISTRY_CACHE}:/root/.cargo/registry -v ${CARGO_GIT_CACHE}:/root/.cargo/git \
-w /greptimedb ${IMAGE_REGISTRY}/${IMAGE_NAMESPACE}/dev-builder-riscv64:${DEV_BUILDER_RISCV64_IMAGE_TAG} \
make build \
CARGO_EXTENSION="${CARGO_EXTENSION}" \
CARGO_PROFILE=${CARGO_PROFILE} \
FEATURES=${FEATURES} \
TARGET_DIR=${TARGET_DIR} \
TARGET=${RISCV64_TARGET} \
RELEASE=${RELEASE} \
CARGO_BUILD_EXTRA_OPTS="${CARGO_BUILD_EXTRA_OPTS}"
.PHONY: build-android-bin
build-android-bin: ## Build greptime binary for android.
docker run --network=host \
@@ -158,6 +176,14 @@ dev-builder: multi-platform-buildx ## Build dev-builder image.
-f docker/dev-builder/${BASE_IMAGE}/Dockerfile \
-t ${IMAGE_REGISTRY}/${IMAGE_NAMESPACE}/dev-builder-${BASE_IMAGE}:${DEV_BUILDER_IMAGE_TAG} ${BUILDX_MULTI_PLATFORM_BUILD_OPTS} .
.PHONY: dev-builder-riscv64
dev-builder-riscv64: ## Build the riscv64 cross dev-builder image.
docker build --network=host \
--build-arg="BASE_IMAGE=${DEV_BUILDER_RISCV64_BASE_IMAGE}" \
--build-arg="RUST_TOOLCHAIN=${RUST_TOOLCHAIN}" \
-f docker/dev-builder/riscv64/Dockerfile \
-t ${IMAGE_REGISTRY}/${IMAGE_NAMESPACE}/dev-builder-riscv64:${DEV_BUILDER_RISCV64_IMAGE_TAG} .
.PHONY: multi-platform-buildx
multi-platform-buildx: ## Create buildx multi-platform builder.
docker buildx inspect ${BUILDX_BUILDER_NAME} || docker buildx create --name ${BUILDX_BUILDER_NAME} --driver docker-container --bootstrap --use
README.md
+49 -38
View File
@@ -6,10 +6,11 @@
</picture>
</p>
<h2 align="center">One database for metrics, logs, and traces<br/>
replacing Prometheus, Loki, and Elasticsearch</h2>
<h2 align="center">Metrics, logs, and traces.<br/>
One engine, on your infrastructure.</h2>
> The unified OpenTelemetry backend — with SQL + PromQL on object storage.
> A columnar database for metrics, logs, and traces on object storage.
> Apache-2.0 licensed core.
<div align="center">
<h3 align="center">
@@ -28,7 +29,7 @@ replacing Prometheus, Loki, and Elasticsearch</h2>
<img src="https://img.shields.io/github/v/release/GreptimeTeam/greptimedb?include_prereleases&filter=*-nightly-*&label=nightly&color=orange" alt="Nightly"/>
</a>
<sub><b>stable</b> for production &nbsp;·&nbsp; <b>latest</b> includes pre-releases &nbsp;·&nbsp; <b>nightly</b> is a weekly snapshot of <code>main</code></sub>
<sub><b>stable</b> for production &nbsp;·&nbsp; <b>canary</b> includes pre-releases &nbsp;·&nbsp; <b>nightly</b> is a weekly snapshot of <code>main</code></sub>
<a href="https://hub.docker.com/r/greptime/greptimedb/">
<img src="https://img.shields.io/docker/pulls/greptime/greptimedb.svg" alt="Docker Pulls"/>
@@ -57,9 +58,11 @@ replacing Prometheus, Loki, and Elasticsearch</h2>
</div>
- [Introduction](#introduction)
- [Why You Might Use It](#why-you-might-use-it)
- [Overview](#overview)
- [Features](#features)
- [How GreptimeDB Compares](#how-greptimedb-compares)
- [What's Supported](#whats-supported)
- [Compatibility and Migration](#compatibility-and-migration)
- [Limitations and Edition Boundary](#limitations-and-edition-boundary)
- [Architecture](#architecture)
- [Try GreptimeDB](#try-greptimedb)
- [Getting Started](#getting-started)
@@ -74,9 +77,20 @@ replacing Prometheus, Loki, and Elasticsearch</h2>
## Introduction
**GreptimeDB** is an open-source observability database built for [Observability 2.0](https://docs.greptime.com/user-guide/concepts/observability-2/) — treating metrics, logs, and traces as one unified data model (wide events) instead of three separate pillars.
**GreptimeDB** is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one [table model](https://docs.greptime.com/user-guide/concepts/data-model/): tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Use it as the single OpenTelemetry backend — replacing Prometheus, Loki, and Elasticsearch with one database built on object storage. Query with SQL and PromQL, scale without pain, cut costs up to 50×.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
## Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry ([OTel GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)) alongside infrastructure signals
- You need the same engine and semantics on resource-constrained devices
Learn more in [Why GreptimeDB](https://docs.greptime.com/user-guide/concepts/why-greptimedb).
## Overview
@@ -88,42 +102,38 @@ A quick overview of what GreptimeDB ingests, how it connects to other systems, a
</a>
</p>
## Features
## What's Supported
| Feature | Description |
|---------|-------------|
| **Observability 2.0 native** | Logs, metrics, and traces in one engine with [SQL + PromQL](https://docs.greptime.com/user-guide/query-data/overview/). Native [OpenTelemetry](https://docs.greptime.com/user-guide/ingest-data/for-observability/opentelemetry/), [Prometheus remote write](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/), and [Jaeger](https://docs.greptime.com/user-guide/query-data/jaeger/). Migrate one signal at a time, or use as a single backend. |
| **Elastic compute-storage separation** | Scale reads independently with horizontal replicas. Serve high-concurrency workloads from dashboards, alerting, and AI agents — without resharding or data migration. |
| **Sub-second on PB–EB-scale data** | Columnar engine with [fulltext, inverted, and skipping indexes](https://docs.greptime.com/user-guide/manage-data/data-index). Written in Rust. Designed for high-concurrency point queries, not just analytical scans. |
| **50× lower cost** | Object storage (S3, GCS, Azure Blob) as [primary storage](https://docs.greptime.com/user-guide/deployments-administration/configuration/#storage-options), with a tiered cache (memory + local disk) to keep writes and queries fast. |
| | |
|---|---|
| **Ingest** | [OpenTelemetry (OTLP)](https://docs.greptime.com/user-guide/ingest-data/for-observability/opentelemetry/), [Prometheus Remote Write](https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/), [Loki Push](https://docs.greptime.com/user-guide/ingest-data/for-observability/loki), [Elasticsearch Bulk](https://docs.greptime.com/user-guide/ingest-data/for-observability/elasticsearch), [InfluxDB line protocol](https://docs.greptime.com/user-guide/protocols/influxdb-line-protocol), [gRPC](https://docs.greptime.com/user-guide/protocols/grpc/) |
| **Query** | [SQL](https://docs.greptime.com/user-guide/query-data/overview/), [PromQL](https://docs.greptime.com/user-guide/query-data/promql/), [Jaeger-compatible trace queries](https://docs.greptime.com/user-guide/query-data/jaeger/), MySQL and PostgreSQL wire protocols |
| **Storage** | S3, GCS, Azure Blob and S3-compatible endpoints as [primary storage](https://docs.greptime.com/user-guide/deployments-administration/configuration/#storage-options), with memory and local-disk caches |
| **Built in** | Retention policies, downsampling, [continuous aggregation](https://docs.greptime.com/user-guide/flow-computation/overview), explicit table partitioning, and [inverted / skipping / fulltext indexes](https://docs.greptime.com/user-guide/manage-data/data-index) |
**Perfect for:**
* Replacing Prometheus + Loki + Elasticsearch with a single observability backend
* Scaling past Prometheus — high cardinality, long-term storage, no Thanos/Mimir overhead
* AI/agent workloads — store GenAI telemetry ([OTel GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)), and serve high-concurrency reads from SRE/developer agents via horizontal read replicas
* Cutting observability costs with object storage (up to 50× savings on traces, 30% on logs)
* Edge-to-cloud observability with unified APIs on resource-constrained devices
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
> **Why Observability 2.0?** Three separate databases for metrics, logs, and traces means three storage layers, three query languages, and three sets of dashboards. GreptimeDB stores all three as timestamped wide events in one columnar engine — JOIN across signals in SQL, run one stack instead of three, and ingest AI agent telemetry the same way. Read more: [Observability 2.0 and the Database for It](https://greptime.com/blogs/2025-04-25-greptimedb-observability2-new-database).
## Compatibility and Migration
Learn more in [Why GreptimeDB](https://docs.greptime.com/user-guide/concepts/why-greptimedb).
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
## How GreptimeDB Compares
| Capability | GreptimeDB | Prometheus / Thanos / Mimir | Grafana Loki | Elasticsearch |
|---|---|---|---|---|
| Data types | Metrics, logs, traces | Metrics only | Logs only | Logs, traces |
| Query language | SQL + PromQL | PromQL | LogQL | Query DSL |
| Storage | Native object storage (S3, etc.) | Local disk + object storage (Thanos/Mimir) | Object storage (chunks) | Local disk |
| Scaling | Compute-storage separation, stateless nodes | Federation / Thanos / Mimir — multi-component, ops heavy | Stateless + object storage | Shard-based, ops heavy |
| Cost efficiency | Up to 50× lower storage cost | High at scale | Moderate | High (inverted index overhead) |
| OpenTelemetry | Native (metrics + logs + traces) | Partial (metrics only) | Partial (logs only) | Via instrumentation |
| | Compatible | Not compatible |
|---|---|---|
| **Prometheus** | Remote Write ingestion; PromQL queries | Gaps are listed in [PromQL compatibility](https://docs.greptime.com/user-guide/query-data/promql/) |
| **Loki** | Push ingestion; dual-write through Grafana Alloy makes the [cutover gradual](https://greptime.com/blogs/2026-07-23-from-loki-to-greptimedb-dual-write-migration) | LogQL and the rest of the Loki query API |
| **Elasticsearch** | `_bulk` ingestion in the open-source core; [QueryDSL](https://docs.greptime.com/enterprise/elasticsearch-compatible/query/) partially, in Enterprise | Most other Elasticsearch APIs |
**Benchmarks:**
* [GreptimeDB tops JSONBench's billion-record cold run test](https://greptime.com/blogs/2025-03-18-jsonbench-greptimedb-performance)
* [TSBS Benchmark](https://github.com/GreptimeTeam/greptimedb/tree/main/docs/benchmarks/tsbs)
* [More benchmark reports](https://docs.greptime.com/user-guide/concepts/features-that-you-concern#how-is-greptimedbs-performance-compared-to-other-solutions)
## Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are **GreptimeDB Enterprise** features, along with enterprise security and governance. The [Enterprise overview](https://docs.greptime.com/enterprise/overview/) has the current list, and [pricing](https://www.greptime.com/pricing#differences) has the edition comparison.
## Architecture
GreptimeDB can run in two modes:
@@ -131,7 +141,7 @@ GreptimeDB can run in two modes:
* **Distributed** — four components, each independently scalable:
- **Frontend** — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- **Datanode** — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- **Metasrv** — metadata, routing, repartitioning, autopilot, and security. Backed by a pluggable KV layer (etcd or RDS).
- **Metasrv** — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- **Flownode** (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the [architecture doc](https://docs.greptime.com/contributor-guide/overview/#architecture) or [DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb/1-overview).
@@ -203,13 +213,13 @@ See the [Contribution Guidelines](CONTRIBUTING.md) for the full developer workfl
- **Kubernetes**: [GreptimeDB Operator](https://github.com/GreptimeTeam/greptimedb-operator)
- **Helm Charts**: [Greptime Helm Charts](https://github.com/GreptimeTeam/helm-charts)
- **Dashboard**: [Web UI](https://github.com/GreptimeTeam/dashboard)
- **gRPC Ingester**: [Go](https://github.com/GreptimeTeam/greptimedb-ingester-go), [Java](https://github.com/GreptimeTeam/greptimedb-ingester-java), [C++](https://github.com/GreptimeTeam/greptimedb-ingester-cpp), [Erlang](https://github.com/GreptimeTeam/greptimedb-ingester-erl), [Rust](https://github.com/GreptimeTeam/greptimedb-ingester-rust), [.NET](https://github.com/GreptimeTeam/greptimedb-ingester-dotnet)
- **gRPC Ingester**: [Go](https://github.com/GreptimeTeam/greptimedb-ingester-go), [Java](https://github.com/GreptimeTeam/greptimedb-ingester-java), [C++](https://github.com/GreptimeTeam/greptimedb-ingester-cpp), [Erlang](https://github.com/GreptimeTeam/greptimedb-ingester-erl), [Rust](https://github.com/GreptimeTeam/greptimedb-ingester-rust), [.NET](https://github.com/GreptimeTeam/greptimedb-ingester-dotnet), [TypeScript](https://github.com/GreptimeTeam/greptimedb-ingester-ts)
- **Grafana Data Source**: [GreptimeDB Grafana data source plugin](https://github.com/GreptimeTeam/greptimedb-grafana-datasource)
- **Grafana Dashboard**: [Official Dashboard for monitoring](https://github.com/GreptimeTeam/greptimedb/blob/main/grafana/README.md)
## Project Status
GreptimeDB is at [v1.0 GA](https://github.com/GreptimeTeam/greptimedb/releases/tag/v1.0.0) with stable APIs and regular releases. It runs in production at scale — [OceanBase Cloud](https://greptime.com/blogs/2025-07-22-user-case-obcloud-log-management-greptimedb) operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60% after migrating from Grafana Loki. See more in [case studies](https://greptime.com/blogs/?category=Use%20Case).
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — [OceanBase Cloud](https://greptime.com/blogs/2025-07-22-user-case-obcloud-log-management-greptimedb) operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in [case studies](https://greptime.com/blogs/?category=Use%20Case).
Read the [v1.0 highlights](https://greptime.com/blogs/2025-11-05-greptimedb-v1-highlights) and [2026 roadmap](https://greptime.com/blogs/2026-02-11-greptimedb-roadmap-2026), or browse the [version reference](https://docs.greptime.com/nightly/reference/about-greptimedb-version).
@@ -241,8 +251,9 @@ that license carry an explicit Enterprise License header.
## Commercial Support
Running GreptimeDB in your organization?
We offer enterprise add-ons, services, training, and consulting.
Scaling observability on your infrastructure?
[GreptimeDB Enterprise](https://docs.greptime.com/enterprise/overview/) adds the operational,
security, and support layer for production deployments.
[Contact us](https://greptime.com/contactus) for details.
## Contributing
config/config.md
+16 -9
View File
@@ -25,7 +25,7 @@
| `runtime` | -- | -- | The runtime options. |
| `runtime.global_rt_size` | Integer | `8` | The number of threads to execute the runtime for global read operations. |
| `runtime.compact_rt_size` | Integer | `4` | The number of threads to execute compact operations. |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 1). |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 2). |
| `http` | -- | -- | The HTTP server options. |
| `http.addr` | String | `127.0.0.1:4000` | The address to bind the HTTP server. |
| `http.timeout` | String | `0s` | HTTP request timeout. Set to 0 to disable timeout.<br/>When Prometheus pending-row batching is enabled, a nonzero timeout less than or equal to the<br/>`prom_store.pending_rows_flush_interval` plus 1 second is adjusted to that value. |
@@ -74,7 +74,9 @@
| `jaeger.enable` | Bool | `true` | Whether to enable Jaeger protocol in HTTP API. |
| `otlp` | -- | -- | OpenTelemetry protocol options. |
| `otlp.enable` | Bool | `true` | Whether to enable OpenTelemetry protocol in HTTP API. |
| `otlp.experimental_enable_exponential_histogram` | Bool | `false` | Experimental: enable cumulative OTLP exponential histogram ingestion. |
| `otlp.trace_ingest_chunk_size` | Integer | `512` | Maximum spans per trace ingest chunk. Set to 0 to disable splitting. |
| `otlp.experimental_enable_resource_info` | Bool | `false` | Whether to synthesize the `greptime_otel_resource_info` table from OTLP metric<br/>resource attributes, so metrics-only services reach the semantic graph. |
| `prom_store` | -- | -- | Prometheus remote storage options |
| `prom_store.enable` | Bool | `true` | Whether to enable Prometheus remote write and read in HTTP API. |
| `prom_store.with_metric_engine` | Bool | `true` | Whether to store the data from Prometheus remote write in metric engine. |
@@ -159,7 +161,7 @@
| `region_engine.mito.max_background_purges` | Integer | Auto | Max number of running background purge jobs (default: number of cpu cores). |
| `region_engine.mito.experimental_compaction_memory_limit` | String | 0 | Memory budget for compaction tasks.<br/>Supports absolute size (e.g., "2GiB", "512MB") or percentage of system memory (e.g., "50%").<br/>Setting it to 0 or "unlimited" disables the limit. |
| `region_engine.mito.experimental_compaction_on_exhausted` | String | wait | Behavior when compaction cannot acquire memory from the budget.<br/>Options: "wait" (default, 10s), "wait(<duration>)", "fail" |
| `region_engine.mito.auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. |
| `region_engine.mito.auto_flush_interval` | String | `10m` | Interval to auto flush a region if it has not flushed yet. |
| `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
| `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size`. |
| `region_engine.mito.default_region_write_buffer_size` | String | `0` | Default write buffer size for each region. Regions stall at this size and reject writes at twice this size. Setting it to 0 disables both limits unless the table specifies `write_buffer_size`. |
@@ -185,6 +187,7 @@
| `region_engine.mito.min_compaction_interval` | String | `0m` | Minimum time interval between two compactions.<br/>To align with the old behavior, the default value is 0 (no restrictions). |
| `region_engine.mito.schedule_compaction_after_edit` | Bool | `true` | Whether to allow to schedule a compaction after a successful region edit.<br/><br/>Setting this to "true" is a necessary but not sufficient condition for scheduling compaction after a region edit.<br/>Other constraints, such as "min_compaction_interval", may still prevent compaction from being scheduled.<br/>Setting this to "false", however, guarantees that compaction will not be scheduled after a region edit. |
| `region_engine.mito.default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. |
| `region_engine.mito.experimental_series_scan_v2` | Bool | `true` | Whether to enable the experimental two-phase mode for eligible metric series scans. |
| `region_engine.mito.index` | -- | -- | The options for index in Mito engine. |
| `region_engine.mito.index.aux_path` | String | `""` | Auxiliary directory path for the index in filesystem, used to store intermediate files for<br/>creating the index and staging files for searching the index, defaults to `{data_home}/index_intermediate`.<br/>The default name for this directory is `index_intermediate` for backward compatibility.<br/><br/>This path contains two subdirectories:<br/>- `__intm`: for storing intermediate files used during creating index.<br/>- `staging`: for storing staging files used during searching index. |
| `region_engine.mito.index.staging_size` | String | `2GB` | The max capacity of the staging directory. |
@@ -234,7 +237,7 @@
| `tracing.tokio_console_addr` | String | Unset | The tokio console address. |
| `event_recorder` | -- | -- | Configuration options for the event recorder. |
| `event_recorder.ttl` | String | `90d` | TTL for the events table that will be used to store the events. Default is `90d`. |
| `event_recorder.event_types` | Array | -- | Event types to record. Current available event types: `create_database`,<br/>`alter_database`, `drop_database`, `create_flow`, `drop_flow`,<br/>`create_table`, `create_logical_tables`, `alter_table`, `alter_logical_tables`,<br/>`drop_table`, `undrop_table`, `purge_dropped_table`, `truncate_table`,<br/>`create_view`, `drop_view`.<br/>When omitted, all current and future event types are recorded.<br/>Set to an empty array to disable event recording. |
| `event_recorder.event_types` | Array | -- | Event types to record. Current available event types: `create_database`,<br/>`alter_database`, `drop_database`, `create_flow`, `drop_flow`,<br/>`create_table`, `create_logical_tables`, `alter_table`, `alter_logical_tables`,<br/>`drop_table`, `undrop_table`, `purge_dropped_table`, `truncate_table`,<br/>`create_view`, `drop_view`, `admin_function`, `reconcile_table`.<br/>When omitted, all current and future event types are recorded.<br/>Set to an empty array to disable event recording. |
| `memory` | -- | -- | The memory options. |
| `memory.enable_heap_profiling` | Bool | `true` | Whether to enable heap profiling activation during startup.<br/>When enabled, heap profiling will be activated if the `MALLOC_CONF` environment variable<br/>is set to "prof:true,prof_active:false". The official image adds this env variable.<br/>Default is true. |
@@ -254,7 +257,7 @@
| `runtime` | -- | -- | The runtime options. |
| `runtime.global_rt_size` | Integer | `8` | The number of threads to execute the runtime for global read operations. |
| `runtime.compact_rt_size` | Integer | `4` | The number of threads to execute compact operations. |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 1). |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 2). |
| `http` | -- | -- | The HTTP server options. |
| `http.addr` | String | `127.0.0.1:4000` | The address to bind the HTTP server. |
| `http.timeout` | String | `0s` | HTTP request timeout. Set to 0 to disable timeout.<br/>When Prometheus pending-row batching is enabled, a nonzero timeout less than or equal to the<br/>`prom_store.pending_rows_flush_interval` plus 1 second is adjusted to that value. |
@@ -315,7 +318,9 @@
| `jaeger.enable` | Bool | `true` | Whether to enable Jaeger protocol in HTTP API. |
| `otlp` | -- | -- | OpenTelemetry protocol options. |
| `otlp.enable` | Bool | `true` | Whether to enable OpenTelemetry protocol in HTTP API. |
| `otlp.experimental_enable_exponential_histogram` | Bool | `false` | Experimental: enable cumulative OTLP exponential histogram ingestion. |
| `otlp.trace_ingest_chunk_size` | Integer | `512` | Maximum spans per trace ingest chunk. Set to 0 to disable splitting. |
| `otlp.experimental_enable_resource_info` | Bool | `false` | Whether to synthesize the `greptime_otel_resource_info` table from OTLP metric<br/>resource attributes, so metrics-only services reach the semantic graph. |
| `prom_store` | -- | -- | Prometheus remote storage options |
| `prom_store.enable` | Bool | `true` | Whether to enable Prometheus remote write and read in HTTP API. |
| `prom_store.with_metric_engine` | Bool | `true` | Whether to store the data from Prometheus remote write in metric engine. |
@@ -372,6 +377,7 @@
| `memory.enable_heap_profiling` | Bool | `true` | Whether to enable heap profiling activation during startup.<br/>When enabled, heap profiling will be activated if the `MALLOC_CONF` environment variable<br/>is set to "prof:true,prof_active:false". The official image adds this env variable.<br/>Default is true. |
| `event_recorder` | -- | -- | Configuration options for the event recorder. |
| `event_recorder.ttl` | String | `90d` | TTL for the events table that will be used to store the events. Default is `90d`. |
| `event_recorder.event_types` | Array | -- | Event types to record. Current available event type: `admin_function`.<br/>When omitted, all current and future event types are recorded.<br/>Set to an empty array to disable event recording. |
### Metasrv
@@ -396,7 +402,7 @@
| `runtime` | -- | -- | The runtime options. |
| `runtime.global_rt_size` | Integer | `8` | The number of threads to execute the runtime for global read operations. |
| `runtime.compact_rt_size` | Integer | `4` | The number of threads to execute compact operations. |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 1). |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 2). |
| `backend_tls` | -- | -- | TLS configuration for kv store backend (applicable for etcd, PostgreSQL, and MySQL backends)<br/>When using etcd, PostgreSQL, or MySQL as metadata store, you can configure TLS here<br/><br/>Note: if TLS is configured in both this section and the `store_addrs` connection string, the<br/>settings here will override the TLS settings in `store_addrs`. |
| `backend_tls.mode` | String | `prefer` | TLS mode, refer to https://www.postgresql.org/docs/current/libpq-ssl.html<br/>- "disable" - No TLS<br/>- "prefer" (default) - Try TLS, fallback to plain<br/>- "require" - Require TLS<br/>- "verify_ca" - Require TLS and verify CA<br/>- "verify_full" - Require TLS and verify hostname |
| `backend_tls.cert_path` | String | `""` | Path to client certificate file (for client authentication)<br/>Like "/path/to/client.crt" |
@@ -450,7 +456,7 @@
| `wal.create_topic_timeout` | String | `30s` | The timeout for creating a Kafka topic.<br/>**It's only used when the provider is `kafka`**. |
| `event_recorder` | -- | -- | Configuration options for the event recorder. |
| `event_recorder.ttl` | String | `90d` | TTL for the events table that will be used to store the events. Default is `90d`. |
| `event_recorder.event_types` | Array | -- | Event types to record. Current available event types: `region_migration`,<br/>`create_database`, `alter_database`, `drop_database`, `create_flow`,<br/>`drop_flow`, `create_table`, `create_logical_tables`, `alter_table`,<br/>`alter_logical_tables`, `drop_table`, `undrop_table`, `purge_dropped_table`,<br/>`truncate_table`, `create_view`, `drop_view`, `repartition`,<br/>`repartition_group`, `wal_prune`, `batch_gc`.<br/>When omitted, all current and future event types are recorded.<br/>Set to an empty array to disable event recording. |
| `event_recorder.event_types` | Array | -- | Event types to record. Current available event types: `region_migration`,<br/>`create_database`, `alter_database`, `drop_database`, `create_flow`,<br/>`drop_flow`, `create_table`, `create_logical_tables`, `alter_table`,<br/>`alter_logical_tables`, `drop_table`, `undrop_table`, `purge_dropped_table`,<br/>`truncate_table`, `create_view`, `drop_view`, `repartition`,<br/>`repartition_group`, `wal_prune`, `batch_gc`, `reconcile_table`.<br/>When omitted, all current and future event types are recorded.<br/>Set to an empty array to disable event recording. |
| `stats_persistence` | -- | -- | Configuration options for the stats persistence. |
| `stats_persistence.ttl` | String | `0s` | TTL for the stats table that will be used to store the stats.<br/>Set to `0s` to disable stats persistence.<br/>Default is `0s`.<br/>If you want to enable stats persistence, set the TTL to a value greater than 0.<br/>It is recommended to set a small value, e.g., `3h`. |
| `stats_persistence.interval` | String | `10m` | The interval to persist the stats. Default is `10m`.<br/>The minimum value is `10m`, if the value is less than `10m`, it will be overridden to `10m`. |
@@ -507,8 +513,8 @@
| `runtime` | -- | -- | The runtime options. |
| `runtime.global_rt_size` | Integer | `8` | The number of threads to execute the runtime for global read operations. |
| `runtime.compact_rt_size` | Integer | `4` | The number of threads to execute compact operations. |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 1). |
| `runtime.query_rt_size` | Integer | `7` | The number of threads to execute datanode query operations.<br/>Defaults to max(num_cpus - 1, 1). |
| `runtime.compact_rt_max_blocking_threads` | Integer | `4` | The maximum number of blocking threads for compact operations.<br/>Defaults to max(num_cpus / 2, 2). |
| `runtime.query_rt_size` | Integer | `7` | The number of threads to execute datanode query operations.<br/>Defaults to max(num_cpus - 1, 2). |
| `runtime.ingest_rt_size` | Integer | `8` | The number of threads to execute datanode ingestion operations. |
| `meta_client` | -- | -- | The metasrv client options. |
| `meta_client.metasrv_addrs` | Array | -- | The addresses of the metasrv. |
@@ -579,7 +585,7 @@
| `region_engine.mito.max_background_purges` | Integer | Auto | Max number of running background purge jobs (default: number of cpu cores). |
| `region_engine.mito.experimental_compaction_memory_limit` | String | 0 | Memory budget for compaction tasks.<br/>Supports absolute size (e.g., "2GiB", "512MB") or percentage of system memory (e.g., "50%").<br/>Setting it to 0 or "unlimited" disables the limit. |
| `region_engine.mito.experimental_compaction_on_exhausted` | String | wait | Behavior when compaction cannot acquire memory from the budget.<br/>Options: "wait" (default, 10s), "wait(<duration>)", "fail" |
| `region_engine.mito.auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. |
| `region_engine.mito.auto_flush_interval` | String | `10m` | Interval to auto flush a region if it has not flushed yet. |
| `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
| `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` |
| `region_engine.mito.default_region_write_buffer_size` | String | `0` | Default write buffer size for each region. Regions stall at this size and reject writes at twice this size. Setting it to 0 disables both limits unless the table specifies `write_buffer_size`. |
@@ -605,6 +611,7 @@
| `region_engine.mito.min_compaction_interval` | String | `0m` | Minimum time interval between two compactions.<br/>To align with the old behavior, the default value is 0 (no restrictions). |
| `region_engine.mito.schedule_compaction_after_edit` | Bool | `true` | Whether to allow to schedule a compaction after a successful region edit.<br/><br/>Setting this to "true" is a necessary but not sufficient condition for scheduling compaction after a region edit.<br/>Other constraints, such as "min_compaction_interval", may still prevent compaction from being scheduled.<br/>Setting this to "false", however, guarantees that compaction will not be scheduled after a region edit. |
| `region_engine.mito.default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. |
| `region_engine.mito.experimental_series_scan_v2` | Bool | `true` | Whether to enable the experimental two-phase mode for eligible metric series scans. |
| `region_engine.mito.index` | -- | -- | The options for index in Mito engine. |
| `region_engine.mito.index.aux_path` | String | `""` | Auxiliary directory path for the index in filesystem, used to store intermediate files for<br/>creating the index and staging files for searching the index, defaults to `{data_home}/index_intermediate`.<br/>The default name for this directory is `index_intermediate` for backward compatibility.<br/><br/>This path contains two subdirectories:<br/>- `__intm`: for storing intermediate files used during creating index.<br/>- `staging`: for storing staging files used during searching index. |
| `region_engine.mito.index.staging_size` | String | `2GB` | The max capacity of the staging directory. |
config/datanode.example.toml
+6 -3
View File
@@ -84,10 +84,10 @@ watch = false
## The number of threads to execute compact operations.
#+ compact_rt_size = 4
## The maximum number of blocking threads for compact operations.
## Defaults to max(num_cpus / 2, 1).
## Defaults to max(num_cpus / 2, 2).
#+ compact_rt_max_blocking_threads = 4
## The number of threads to execute datanode query operations.
## Defaults to max(num_cpus - 1, 1).
## Defaults to max(num_cpus - 1, 2).
#+ query_rt_size = 7
## The number of threads to execute datanode ingestion operations.
#+ ingest_rt_size = 8
@@ -462,7 +462,7 @@ compress_manifest = false
#+ experimental_compaction_on_exhausted = "wait"
## Interval to auto flush a region if it has not flushed yet.
auto_flush_interval = "1h"
auto_flush_interval = "10m"
## Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB.
## @toml2docs:none-default="Auto"
@@ -571,6 +571,9 @@ schedule_compaction_after_edit = true
## Whether to enable flat format as the default SST format.
default_flat_format = true
## Whether to enable the experimental two-phase mode for eligible metric series scans.
experimental_series_scan_v2 = true
## The options for index in Mito engine.
[region_engine.mito.index]
config/frontend.example.toml
+10 -1
View File
@@ -44,7 +44,7 @@ default_column_prefix = "greptime"
## The number of threads to execute compact operations.
#+ compact_rt_size = 4
## The maximum number of blocking threads for compact operations.
## Defaults to max(num_cpus / 2, 1).
## Defaults to max(num_cpus / 2, 2).
#+ compact_rt_max_blocking_threads = 4
## The HTTP server options.
@@ -237,8 +237,13 @@ enable = true
[otlp]
## Whether to enable OpenTelemetry protocol in HTTP API.
enable = true
## Experimental: enable cumulative OTLP exponential histogram ingestion.
experimental_enable_exponential_histogram = false
## Maximum spans per trace ingest chunk. Set to 0 to disable splitting.
trace_ingest_chunk_size = 512
## Whether to synthesize the `greptime_otel_resource_info` table from OTLP metric
## resource attributes, so metrics-only services reach the semantic graph.
experimental_enable_resource_info = false
## Prometheus remote storage options
[prom_store]
@@ -405,3 +410,7 @@ enable_heap_profiling = true
[event_recorder]
## TTL for the events table that will be used to store the events. Default is `90d`.
ttl = "90d"
## Event types to record. Current available event type: `admin_function`.
## When omitted, all current and future event types are recorded.
## Set to an empty array to disable event recording.
#+ event_types = ["admin_function"]
config/metasrv.example.toml
+2 -2
View File
@@ -92,7 +92,7 @@ node_max_idle_time = "24hours"
## The number of threads to execute compact operations.
#+ compact_rt_size = 4
## The maximum number of blocking threads for compact operations.
## Defaults to max(num_cpus / 2, 1).
## Defaults to max(num_cpus / 2, 2).
#+ compact_rt_max_blocking_threads = 4
## TLS configuration for kv store backend (applicable for etcd, PostgreSQL, and MySQL backends)
@@ -320,7 +320,7 @@ ttl = "90d"
## `drop_flow`, `create_table`, `create_logical_tables`, `alter_table`,
## `alter_logical_tables`, `drop_table`, `undrop_table`, `purge_dropped_table`,
## `truncate_table`, `create_view`, `drop_view`, `repartition`,
## `repartition_group`, `wal_prune`, `batch_gc`.
## `repartition_group`, `wal_prune`, `batch_gc`, `reconcile_table`.
## When omitted, all current and future event types are recorded.
## Set to an empty array to disable event recording.
#+ event_types = ["region_migration"]
config/standalone.example.toml
+11 -3
View File
@@ -57,7 +57,7 @@ max_concurrent_queries = 0
## The number of threads to execute compact operations.
#+ compact_rt_size = 4
## The maximum number of blocking threads for compact operations.
## Defaults to max(num_cpus / 2, 1).
## Defaults to max(num_cpus / 2, 2).
#+ compact_rt_max_blocking_threads = 4
## The HTTP server options.
@@ -204,8 +204,13 @@ enable = true
[otlp]
## Whether to enable OpenTelemetry protocol in HTTP API.
enable = true
## Experimental: enable cumulative OTLP exponential histogram ingestion.
experimental_enable_exponential_histogram = false
## Maximum spans per trace ingest chunk. Set to 0 to disable splitting.
trace_ingest_chunk_size = 512
## Whether to synthesize the `greptime_otel_resource_info` table from OTLP metric
## resource attributes, so metrics-only services reach the semantic graph.
experimental_enable_resource_info = false
## Prometheus remote storage options
[prom_store]
@@ -620,7 +625,7 @@ compress_manifest = false
#+ experimental_compaction_on_exhausted = "wait"
## Interval to auto flush a region if it has not flushed yet.
auto_flush_interval = "1h"
auto_flush_interval = "10m"
## Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB.
## @toml2docs:none-default="Auto"
@@ -729,6 +734,9 @@ schedule_compaction_after_edit = true
## Whether to enable flat format as the default SST format.
default_flat_format = true
## Whether to enable the experimental two-phase mode for eligible metric series scans.
experimental_series_scan_v2 = true
## The options for index in Mito engine.
[region_engine.mito.index]
@@ -922,7 +930,7 @@ ttl = "90d"
## `alter_database`, `drop_database`, `create_flow`, `drop_flow`,
## `create_table`, `create_logical_tables`, `alter_table`, `alter_logical_tables`,
## `drop_table`, `undrop_table`, `purge_dropped_table`, `truncate_table`,
## `create_view`, `drop_view`.
## `create_view`, `drop_view`, `admin_function`, `reconcile_table`.
## When omitted, all current and future event types are recorded.
## Set to an empty array to disable event recording.
#+ event_types = ["create_database"]
docker/dev-builder/riscv64/Dockerfile
+55
View File
@@ -0,0 +1,55 @@
# Cross-build environment for producing riscv64 (riscv64gc-unknown-linux-gnu)
# greptime binaries on x86_64/aarch64 hosts.
ARG BASE_IMAGE=ubuntu:22.04
FROM ${BASE_IMAGE}
ENV LANG=en_US.utf8
WORKDIR /greptimedb
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y software-properties-common
# Install dependencies, including the riscv64 cross toolchain.
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
libssl-dev \
tzdata \
curl \
unzip \
ca-certificates \
git \
build-essential \
pkg-config \
gcc-riscv64-linux-gnu \
g++-riscv64-linux-gnu
ARG TARGETPLATFORM
RUN echo "host platform: $TARGETPLATFORM"
ARG PROTOBUF_VERSION=29.3
# Install protobuf for the host arch (build scripts and protoc run on the host),
# because the one in the apt is too old (v3.12).
RUN if [ "$TARGETPLATFORM" = "linux/arm64" ]; then \
curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-linux-aarch_64.zip && \
unzip -o protoc-${PROTOBUF_VERSION}-linux-aarch_64.zip -d protoc3; \
else \
curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-linux-x86_64.zip && \
unzip -o protoc-${PROTOBUF_VERSION}-linux-x86_64.zip -d protoc3; \
fi
RUN mv -f protoc3/bin/* /usr/local/bin/
# Use cp -r instead of mv so that it also works when the base image already
# contains protobuf headers under /usr/local/include.
RUN cp -r protoc3/include/* /usr/local/include/
# Silence all `safe.directory` warnings, to avoid the "detect dubious repository"
# error when building with the repository mounted from a different user.
RUN git config --global --add safe.directory '*'
# Install Rust.
SHELL ["/bin/bash", "-c"]
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --no-modify-path --default-toolchain none -y
ENV PATH=/root/.cargo/bin/:$PATH
# Install Rust toolchains and the riscv64 target std.
ARG RUST_TOOLCHAIN
RUN rustup toolchain install ${RUST_TOOLCHAIN}
RUN rustup target add --toolchain ${RUST_TOOLCHAIN} riscv64gc-unknown-linux-gnu
docs/parquet-tools.md
+255
View File
@@ -0,0 +1,255 @@
# Parquet Development Tools
GreptimeDB provides the following datanode CLI commands for inspecting,
rewriting, benchmarking, and replacing Parquet SST files:
- `parquet-meta`: inspect a local Parquet file.
- `parquet-rewrite`: rewrite a local Parquet file with different writer
properties.
- `parquetbench`: benchmark reads from a local GreptimeDB SST or an SST in a
configured object store.
- `sst-replace`: replace an existing Mito SST object and update its manifest
metadata.
These are development and recovery tools, not stable user-facing interfaces.
Always test a workflow on disposable data before using it on an important
region.
## Build
`parquet-meta`, `parquet-rewrite`, and `sst-replace` require the `dev-tools`
feature. The same build also includes `parquetbench`:
```bash
cargo build -p cmd --bin greptime --features dev-tools
```
The examples below use:
```bash
GREPTIME=./target/debug/greptime
```
Run `$GREPTIME datanode <COMMAND> --help` for the authoritative argument list.
## Inspect a Parquet file
`parquet-meta` reads a local Parquet footer and optional page indexes. It reports
file, row-group, and column metadata, including compression, encodings, sizes,
page offsets, statistics presence, and bloom-filter/index offsets.
Text output:
```bash
$GREPTIME datanode parquet-meta \
--input /tmp/source.parquet
```
Machine-readable JSON output:
```bash
$GREPTIME datanode parquet-meta \
--input /tmp/source.parquet \
--format json > /tmp/source-meta.json
```
`--format` accepts `text` (the default) or `json`. This command can inspect a
general Parquet file; it does not require GreptimeDB SST key-value metadata.
## Rewrite a Parquet file
`parquet-rewrite` has two modes. First, dump a TOML properties file inferred
from an existing file:
```bash
$GREPTIME datanode parquet-rewrite \
--input /tmp/source.parquet \
--dump-properties /tmp/writer-properties.toml
```
Review and edit the generated file. Its shape is:
```toml
[writer]
dictionary_enabled = true
compression = "zstd"
max_row_group_row_count = 8192
[[columns]]
path = ["host"]
dictionary_enabled = true
compression = "zstd"
encoding = "plain"
```
Supported compression names are `uncompressed`, `snappy`, `gzip`, `lzo`,
`brotli`, `lz4`, `zstd`, and `lz4-raw`. Supported encodings are `plain`,
`delta-binary-packed`, `delta-length-byte-array`, `delta-byte-array`, and
`byte-stream-split`.
The `[writer]` table also accepts `compression_level`,
`data_page_size_limit`, `data_page_row_count_limit`, and
`dictionary_page_size_limit`. It also accepts
`column_index_truncate_length` and `statistics_truncate_length`; set either to
zero to disable that truncation. A `[[columns]]` entry overrides dictionary,
compression, compression level, or encoding for its column path. Unknown TOML
fields are rejected. Compression levels are supported for `gzip`, `brotli`, and
`zstd`. A column-level `compression_level` without `compression` inherits the
writer compression codec; it is rejected when there is no writer codec to
inherit.
Rewrite the data using the edited properties:
```bash
$GREPTIME datanode parquet-rewrite \
--input /tmp/source.parquet \
--properties /tmp/writer-properties.toml \
--output /tmp/rewritten.parquet
```
Use `--batch-size <ROWS>` to control reader batch size. Output and dumped
properties files are not replaced unless `--overwrite` is passed. The output or
dump path must not be exactly the same path as the input, even with
`--overwrite`. This check does not resolve symbolic links, hard links, or other
spellings of the same path.
The rewrite decodes and writes the Arrow record batches and copies the Parquet
key-value metadata, but it creates a new physical Parquet layout. Inspect and
validate the result before using it as an SST:
```bash
$GREPTIME datanode parquet-meta \
--input /tmp/rewritten.parquet
```
The dumped properties are inferred primarily from the first row group. Review
them when the source uses different properties across row groups.
## Benchmark an SST
`parquetbench` expects GreptimeDB region metadata embedded in the SST. It can
read a local file with the direct reader:
```bash
$GREPTIME datanode parquetbench \
--file-path /tmp/rewritten.parquet \
--reader direct \
--iterations 5 \
--batch-size 8192
```
Local-file mode cannot be combined with `--config`, `--region-id`,
`--table-dir`, or `--file-id`, and it does not support the `flat-prune` reader.
To benchmark an SST in the object store configured for a datanode or standalone
deployment:
```bash
$GREPTIME datanode parquetbench \
--config /path/to/datanode.toml \
--region-id 1024:0 \
--table-dir data/greptime/public/1024 \
--file-id 00020380-009c-426d-953e-b4e34c15af34 \
--path-type bare \
--reader flat-prune \
--iterations 5
```
Region mode requires all four of `--config`, `--region-id`, `--table-dir`, and
`--file-id`. `--region-id` accepts either the packed unsigned integer or
`<table-id>:<region-number>`. `--path-type` accepts `bare`, `data`, or
`metadata` and defaults to `bare`.
An optional scan configuration selects columns and row groups:
```json
{
"projection_names": ["host", "value", "ts"],
"row_groups": [0, 2]
}
```
```bash
$GREPTIME datanode parquetbench \
--file-path /tmp/rewritten.parquet \
--scan-config /tmp/parquet-scan.json \
--iterations 5
```
Use `--pk-as-binary` to expose `__primary_key` as binary with the direct reader.
On Unix, `--pprof-file <SVG>` writes a flamegraph. Add
`--pprof-after-warmup` and use at least two iterations to exclude the first
iteration from profiling.
## Replace an existing region SST
> **Warning:** `sst-replace` mutates both an SST object and its region manifest.
> Stop the datanode that owns the region and back up the target SST and manifest
> before using `--confirm`. The SST write happens before the manifest update, so
> an interrupted or failed operation may require restoring the backup.
`sst-replace` replaces the contents of an existing SST file ID. It does not add
a new file ID to a region. The command requires the replacement to have the same
row count and row-group count recorded in the manifest when those manifest
values are nonzero. It does not validate schema or row contents. The replacement
is loaded into memory in full, so ensure the machine has enough memory for the
SST.
Start with the default dry run using a local replacement file:
```bash
$GREPTIME datanode sst-replace \
--config /path/to/datanode.toml \
--region-id 1024:0 \
--table-dir data/greptime/public/1024 \
--file-id 00020380-009c-426d-953e-b4e34c15af34 \
--replacement-file /tmp/rewritten.parquet
```
The dry run locates the manifest and target SST, reads and validates the
replacement footer, and prints the old and new sizes without writing anything.
`--path-type` defaults to `auto`, which probes `bare`, `data`, and `metadata`.
Specify the path type if the file ID is visible in more than one manifest.
The replacement can instead be read from the configured object store:
```bash
$GREPTIME datanode sst-replace \
--config /path/to/datanode.toml \
--region-id 1024:0 \
--table-dir data/greptime/public/1024 \
--file-id 00020380-009c-426d-953e-b4e34c15af34 \
--replacement-object staging/rewritten.parquet \
--path-type bare
```
After reviewing the dry-run output and confirming that the datanode is stopped,
repeat the exact command with `--confirm`:
```bash
$GREPTIME datanode sst-replace \
--config /path/to/datanode.toml \
--region-id 1024:0 \
--table-dir data/greptime/public/1024 \
--file-id 00020380-009c-426d-953e-b4e34c15af34 \
--replacement-file /tmp/rewritten.parquet \
--path-type bare \
--confirm
```
The confirmed operation overwrites the existing SST object, recalculates file
size and row-group statistics, and appends a manifest edit for the existing file
ID. Restart the datanode and validate queries against the region before removing
the backup.
## Recommended rewrite and replacement workflow
1. Back up the source SST and its region manifest.
2. Inspect the source with `parquet-meta`.
3. Dump and edit writer properties with `parquet-rewrite`.
4. Rewrite to a new local file; never rewrite directly over the source SST.
5. Inspect the output and benchmark it with `parquetbench`.
6. Stop the owning datanode.
7. Run `sst-replace` without `--confirm` and review its resolved target and
statistics.
8. Repeat with `--confirm`, restart the datanode, and validate the region.
docs/rfcs/2026-06-25-entity-relationships-and-graph-query.md
+4 -4
View File
@@ -149,7 +149,7 @@ The decisions behind the shape:
- **`provenance` is part of edge identity**, so a declared edge and a derived edge between the same pair coexist, and a user-declared edge survives when derivation is off.
- **`confidence` expresses derivation certainty, not statistical completeness**: `1.0` for a successfully paired or declared edge, `< 1.0` for virtual-node or agent-inferred edges. It does not correct for trace sampling.
- **RED metrics describe the observed span-pair population** — the `service_graph` connector semantics. Under sampling, counts understate true traffic, and ratios (error rate, mean duration) are representative only when sampling is unbiased with respect to status and latency — tail sampling that keeps errors and slow traces skews both.
- **Endpoint encoding (v1).** A single-attribute identity is the value verbatim; a composite identity is the attributes sorted by key, rendered `k1=v1,k2=v2` — a human-readable traversal key, not a collision-free canonical encoding. Nodes carry the structured `entity_id_attrs`; edges carry only the string key. Known v1 limitation: composite components containing unescaped `,` or `=` are not guaranteed to be distinguished.
- **Endpoint encoding.** An identity renders as its values in declared order (broad to narrow), joined by `,` with `,` and `\` escaped in each value. The identifying attribute *keys* are deliberately absent: the same identity reaches the graph under different column names per signal — a trace table's `service_name` against a metric table's `job` — and encoding the keys would split one entity into one entity per signal. Where a source carries a qualifier separately (`service.namespace` beside `service.name`) the convention folds it in as `<qualifier>/<value>`, reproducing the identity a source that pre-composed it (Prometheus `job`) already emits. Nodes carry the structured `entity_id_attrs`; edges carry only the string key.
- **Hand-declared edges** (`provenance = 'declared'`) live in a physical table of the same shape (plus a business validity window) in `greptime_private` and are unioned into the computed `semantic_relationships`.
The relationship vocabulary is small, typed, and inverse-paired; stored direction is `src -> dst`, the inverse is a query concern:
@@ -197,7 +197,7 @@ GROUP BY 1, src_id, dst_id;
Edge endpoints are built from each trace table's `service` entity declaration (the same id expression the registry uses), so edges land on exactly the entity ids the registry emits, self-calls are judged on the full endpoint id, and a table whose declaration is unusable contributes no edges rather than silently switching identity. The join is bounded by a time-proximity window and the results of **all** trace tables combine into one edge per window with merged RED metrics. One assumption to note: a deployment can route traces to multiple tables (`x-greptime-trace-table-name`), so one distributed trace is not guaranteed to live in a single table — pairing must account for that, and how is an implementation concern. Uninstrumented peers become **virtual nodes**: a client span with no matching server span yields an edge to a synthetic node named from `peer.service`/`db.name`/`server.address`.
**The entity registry.** Per declaring table: filter to the window, project (time bin, casts, JSON assembly), `DISTINCT`. The per-table results are unioned without any cross-source merge: no descriptive-conflict or JSON-merging policy lives in the derivation. Rows whose identity columns are NULL identify nothing and are filtered out; single-column ids are used verbatim, composite ids render the sorted `k=v` form plus the `entity_id_attrs` JSON. Consumers deduplicate with `SELECT DISTINCT entity_type, entity_id`; a unique node set is the snapshot relation's job (see *Querying the graph*).
**The entity registry.** Per declaring table: filter to the window, project (time bin, casts, JSON assembly), `DISTINCT`. The per-table results are unioned without any cross-source merge: no descriptive-conflict or JSON-merging policy lives in the derivation. Rows whose identity columns are NULL identify nothing and are filtered out; ids render as their escaped values joined in declared order, with the `entity_id_attrs` JSON alongside once an id is assembled from more than one column. Consumers deduplicate with `SELECT DISTINCT entity_type, entity_id`; a unique node set is the snapshot relation's job (see *Querying the graph*).
**Attribute edges.** Shared attributes provide join keys; co-declaration or a relationship template provides the relationship semantics. A shared value alone determines neither direction nor edge type, so the derivation is rule-based, not join-everything. A table declaring both a `service.instance` and a `host` identity on the same rows derives `runs_on` between them: the row itself witnesses the relationship, and the built-in vocabulary fixes the direction. Cross-table edges require a declared relationship template (endpoint mappings, `rel_type`, direction): a derivation *rule* declared on schema, not per-instance edge data.
@@ -284,7 +284,7 @@ Drawbacks:
- Read-time derivation costs a self-join over trace tables on every scan of the computed tables. The default one-hour window bounds it, but very large deployments will want materialisation (Future Work).
- `semantic_entities` exposes per-observation rows; deduplication falls on the consumer (`DISTINCT`) until the snapshot relation lands in M2.
- The v1 `k=v` id encoding can collide on hostile values (unescaped `,`/`=`); hardening is an open question.
- The id encoding is readable rather than canonical: escaping keeps components distinguishable, but a value containing the qualifier separator (`/`) is still ambiguous against a qualified name.
- The graph is only as connected as the identity columns tables actually share: the `job`/`service_name` misalignment above is the canonical example, and enrichment discipline gates connectivity.
# Implementation Plan
@@ -306,7 +306,7 @@ Drawbacks:
# Open Questions
1. **Endpoint encoding hardening.** The v1 `k=v` rendering (with `entity_id_attrs` JSON as the source of truth) is readable but collides when values contain `,`/`=`. Open: a fixed-width `entity_id_hash` join key (New-Relic-style); typed value encoding; the criteria for introducing the `semantic_key=column` mapping form; `schema_url`-gated entity merge. Related and unresolved: whether the identifying attribute *key* should participate in entity equality — a single-attribute id keeps only the value, so `job=api` and `service_name=api` currently name the same entity even when the keys differ semantically.
1. **Endpoint encoding hardening.** The rendering (with `entity_id_attrs` JSON as the source of truth) is readable rather than canonical. Open: a fixed-width `entity_id_hash` join key (New-Relic-style); typed value encoding; the criteria for introducing the `semantic_key=column` mapping form; `schema_url`-gated entity merge. Settled: identifying attribute *keys* do not participate in entity equality, so `job=api` and `service_name=api` name the same entity — the property that lets one service reach the graph from several signals.
2. **Identity reconciliation across signals.** Trace edges key on the trace table's service declaration; host edges key on `host.id`/`k8s.pod.uid`. Linking them relies on tables carrying both columns; how aggressively to enrich (a `k8sattributes`-equivalent) gates how connected the graph is.
3. **`GRAPH_TABLE` surface scope.** How much of SQL/PGQ to implement vs. a minimal single-hop subset. Start minimal and grow with demand.
4. **Visibility beyond table grants.** The derivation contract filters by table-read permission; whether topology needs finer policies (row-level, tenant-scoped visibility of cross-schema edges) is open.
docs/rfcs/2026-08-04-native-histograms.md
+286
View File
@@ -0,0 +1,286 @@
---
Feature Name: Native Histogram Support and Compatibility Decisions
Tracking Issue: TBD
Date: 2026-08-04
Author: codex
---
# Summary
GreptimeDB stores native histograms in one Prometheus-compatible Struct-valued
field and evaluates them as first-class PromQL samples. Prometheus Remote Write
2.0 supplies native histograms directly. OTLP `ExponentialHistogram` is an
ingestion transport: accepted cumulative points are normalized into the same
Struct before persistence and are queried only as native histograms.
Native histograms are experimental. Prometheus Remote Write 2.0 and cumulative
OTLP exponential histograms are supported behind separate configuration gates.
Remote Write 1.0 histogram payloads are rejected instead of being acknowledged and dropped.
Native-histogram Remote Read is deferred; the existing Remote Read path
continues to return scalar samples only.
# Goals
1. Prevent silent native-histogram data loss at protocol boundaries.
2. Keep one stable Prometheus-compatible persisted representation across
Prometheus and OTLP ingestion.
3. Match Prometheus query behavior where it is observable and practical.
4. State intentional limitations explicitly so incomplete behavior is not
mistaken for support.
# Non-Goals
- Supporting native histograms in Remote Write 1.0.
- Returning native histograms through Prometheus Remote Read.
- Persisting Remote Write metric metadata.
- Persisting exemplars.
- Providing an OTLP-specific histogram query surface or reconstructing and
re-exporting the original OTLP point after persistence.
- Removing mixed float/histogram handling from PromQL expressions.
- Propagating PromQL annotations produced on datanodes back to the frontend.
# Data Model and Invariants
Each accepted native histogram, whether received directly through Remote Write
2.0 or normalized from OTLP `ExponentialHistogram`, is stored in the configured
native-histogram field (`greptime_native_histogram` by default) as the same
canonical Struct. The persistence boundary admits only values accepted by the
shared native-histogram validation.
## Prometheus Remote Write 2.0
Remote Write 2.0 native histograms enter the persistence path in Prometheus
format. The Struct preserves the validated schema, zero threshold, sum, reset
hint, start timestamp, custom bounds, spans, and either the integer or float
count family. Integer bucket deltas are converted to absolute integer counts
for storage; float bucket counts are already absolute. No separate sample-kind
discriminator is stored because the populated count family identifies it.
## OTLP ExponentialHistogram
OTLP `ExponentialHistogram` does not introduce a second persisted format.
Accepted cumulative points are normalized into a valid Prometheus native
histogram and then pass through the same validator and Struct encoder. The
Struct does not retain the raw OTLP point or a source-protocol discriminator,
and queries use only the native-histogram PromQL behavior. Detailed conversion
and rejection rules are listed under [OTLP](#otlp).
Within one resolved catalog, schema, and physical-table routing context, a
metric name has exactly one persisted sample kind:
- a float metric uses `greptime_value`;
- a native-histogram metric uses the configured native-histogram field.
Labels do not change that choice. Remote Write 2.0 rejects a request when two
label sets use different sample kinds for the same metric, and the existing
table schema rejects kind changes across requests. Prometheus metadata type is
not used for this decision: a classic histogram family is described as a
histogram but is represented by float-valued `_bucket`, `_sum`, and `_count`
series.
The storage invariant does not remove the need for mixed-sample PromQL support.
An expression can combine different metrics or branches, for example with
`or`, and therefore produce a vector containing float samples and histogram
samples from different series.
# Protocol Decisions
## Remote Write 1.0
Remote Write 1.0 scalar samples remain supported. Any `TimeSeries.histograms`
field causes the complete request to fail with an invalid-arguments response.
Rejecting at protobuf decode time prevents both histogram-only data loss and
partial ingestion of a request that mixes scalar and histogram series. Clients
that send native histograms must use Remote Write 2.0.
## Remote Write 2.0
Remote Write 2.0 accepts integer and float native histograms while
`prom_store.experimental_enable_prometheus_native_histogram` is enabled. Supported
exponential schemas are `-4` through `8`; schema `-53` represents native
histograms with custom buckets.
Exponential spans may end at the schema's overflow bucket but must not continue
beyond it. The overflow index is `(1024 << schema) + 1` for non-negative schemas
and `(1024 >> -schema) + 1` for negative schemas. Inputs beyond that index are
rejected rather than clamped or merged into infinity.
Exemplars and declared metric metadata are accepted on the wire but are not
persisted by this feature.
## Remote Read
Native-histogram Remote Read is deferred. GreptimeDB's existing Remote Read path
continues to negotiate `SAMPLES` responses and serialize scalar samples only.
A follow-up can map the stored Struct into `TimeSeries.histograms`. It must
preserve integer counts without passing through `f64`, retain the stored
histogram fields, and decide sampled versus streamed response coverage. The
sampled protobuf has no start-timestamp field, so that omission must remain
explicit if sampled responses are implemented.
## OTLP
OTLP exponential histograms are accepted when
`otlp.experimental_enable_exponential_histogram` is enabled. The option defaults
to false and applies to OTLP/HTTP. Disabled points are rejected rather than
silently acknowledged. OTel Arrow exponential histograms are rejected because
the current Arrow wire format omits `zero_threshold`; accepting them would
silently change the distribution. Cumulative temporality is required; delta and
unspecified exponential histograms are rejected before their points are
converted. Explicit OTLP histograms keep their existing `_bucket`, `_sum`, and
`_count` representation, including their existing delta behavior.
OTLP scales `-4` through `8` map directly to Prometheus schemas. Higher scales
are downscaled to schema `8`: dense counts that collide are merged before the
OTLP lower-bound index is shifted by one to the Prometheus upper-bound index.
This preserves count mass but irreversibly loses distinctions between source
buckets that merge. Lower scales are rejected. OTLP counts always populate the
integer histogram family. The transmitted non-negative finite zero threshold
and zero count are preserved; the Struct start timestamp is stored in
milliseconds and the reset hint is unknown. Point timestamps and attributes
retain their existing mode-specific conversion rules. Legacy mode retains its
normalized name, attribute rules, and nanosecond row timestamp. Non-legacy mode
retains Prometheus-compatible translation and its millisecond row timestamp.
Both modes write the same canonical Struct.
An absent sum is stored as an ordinary quiet NaN, so the sample remains
selectable while `histogram_sum` is unknown. Any NaN sum on a point without
`NoRecordedValue` is normalized to the same value, even if its payload has the
Prometheus stale-marker bits. Only an exponential-histogram point carrying OTLP
`NoRecordedValue` becomes an empty schema-0 integer histogram with the canonical
Prometheus stale-NaN sum; its attributes and timestamps remain. This
interpretation intentionally does not change gauges, sums, or explicit histograms.
Invalid points are skipped while unrelated valid points continue. Mixed
accepted/rejected OTLP/HTTP requests return partial success; a request with only
rejected points returns `InvalidArgument`. OTel Arrow uses an `OK` batch status
for mixed batches and `INVALID_ARGUMENT` when all points are rejected. Rejection
details are bounded, and metric metadata is emitted only for a metric that
produced an accepted row.
Minimum, maximum, and exemplars are not persisted. Delta accumulation,
zero-run span compaction, and a dedicated rejection metric remain deferred.
# PromQL Compatibility Decisions
## Staleness
A native histogram is stale only when its `sum` has Prometheus's stale-NaN bit
pattern. Ordinary NaN sums remain samples. Range selectors remove stale markers
before range functions run. Instant selectors treat the marker as the end of
the series for lookback purposes and do not resurrect an older sample.
## Start timestamps
Reset detection already uses consecutive start timestamps in addition to
bucket and count monotonicity. Counter `rate` and `increase` also use the first
histogram's start timestamp as a synthetic zero when it is nonzero and strictly
inside the query window before the first sample. This permits the same
single-sample calculation as Prometheus and prevents left extrapolation past the
known start. When no usable start timestamp exists, a positive histogram-count
increase lets counter extrapolation infer a zero point from the first count to
cap left extrapolation; a synthetic zero uses its known timestamp instead of
that heuristic.
This behavior is native-histogram-only. GreptimeDB does not persist start
timestamps for float samples, so float `rate` and `increase` still require two
samples. Warnings for overlapping start timestamps are deferred.
## Arithmetic and averages
Histogram addition and subtraction always reconcile layouts, including empty
histograms. Mixing exponential and custom layouts fails; custom layouts are
reconciled according to their shared bounds. The only layout bypass is local to
counter rate: when the first-to-second pair is a reset, the first sample's
layout is irrelevant and the second sample starts the accumulated segment.
`avg` and `avg_over_time` use a mergeable weighted running mean. This prevents a
finite mean from becoming infinity solely because the intermediate sum
overflowed. Native-histogram aggregates are not currently split into datanode
partial aggregation and frontend state merging, so this state does not cross
nodes or versions. If distributed stepping is added later, its state must be
versioned before rolling upgrades can safely mix implementations.
The rejected alternative is Prometheus's direct-sum/Kahan-compensation design
with an overflow-triggered mode change. It offers closer rounding parity but
requires additional state and substantially more merge logic.
## Equality, changes, and resets
Equality follows Prometheus's represented-layout semantics. Histograms compare
equal only when their represented bucket-index sequences and bucket value bit
patterns match. An explicitly represented zero bucket therefore differs from an
omitted bucket, and `changes()` observes that layout change. Redundant
zero-length span encodings are normalized, malformed layouts compare unequal,
and reset hints and start timestamps remain excluded.
Custom bucket bounds use ordinary floating-point equality rather than bitwise
equality, matching Prometheus's `CustomBucketBoundsMatch`. Histogram payload
values continue to use bitwise equality so NaN payloads behave deterministically.
`resets()` counts ordinary counter resets and, unlike Prometheus, also counts
each transition between gauge and non-gauge native histograms as one reset.
## Annotations
Warnings and infos produced by frontend-executed PromQL functions are returned
through the corresponding Prometheus JSON response fields. Annotations produced
while a pushed-down plan executes on a datanode are not yet transported back to
the frontend. The sample-dropping behavior is unchanged, but those remote
annotations remain silent until a query-result transport is defined for them.
## Standard deviation and bucket bounds
Custom-bucket standard deviation and variance use the arithmetic midpoint
`(lower + upper) / 2` for every bucket. Underflow and overflow buckets therefore
naturally produce infinite or NaN estimates. Exponential buckets retain the
signed geometric midpoint, with zero used for a bucket spanning zero.
Boundary calculation returns the last finite boundary as `f64::MAX`, the
overflow boundary as positive infinity, and rejects indices beyond the
overflow bucket. Widened integer arithmetic prevents large negative-schema
indices from wrapping into an unrelated finite boundary.
# Metric Metadata
The Prometheus `/metadata` endpoint currently derives metric names from logical
table names and reads type and unit from semantic table options stamped at table
creation. Native-histogram tables without a declared type fall back to
`histogram`. Help is always empty, and unit is absent for tables without a
semantic unit option.
Persisting declared Remote Write metadata is separate work. Reusing table
options alone is insufficient because metadata-only writes, later updates, and
existing tables do not pass through auto-create. An in-memory registry would
lose data on restart. The recommended follow-up is a persistent registry keyed
by catalog, schema, and metric family, populated by Remote Write 1.0 metadata-only
requests and Remote Write 2.0 per-series metadata. This follows the update
boundary already recorded in
[Table Semantic Layer](2026-05-28-table-semantic-layer.md).
# Testing and Compatibility
Compatibility coverage includes protocol rejection, OTLP scale conversion and
partial-success behavior, kind exclusivity,
stale-marker selector semantics, synthetic-zero rates, incompatible empty
layouts, overflow-safe averages, layout-sensitive equality, infinite custom
midpoints, and exponential overflow indices for schemas `-4`, `0`, and `8`.
Behavioral coverage also verifies frontend PromQL warning and info responses.
This work does not change GreptimeDB's persisted Struct or protobuf
dependencies. It adds the disabled-by-default public OTLP gate; existing
native-histogram data remains readable.
# Future Work
- Native-histogram Remote Read, including exact integer round-trips and streamed
chunks with start timestamps.
- OTLP exponential-histogram delta accumulation.
- Persistent Remote Write metadata and accurate help/unit updates.
- Native-histogram exemplars and exemplar query APIs.
- Start-timestamp overlap annotations.
- Datanode-to-frontend PromQL annotation propagation.
- Versioned two-phase native-histogram aggregation state.
- Exact Prometheus summation compensation if measured precision differences
justify the extra aggregate state.
src/auth/src/error.rs
+8
View File
@@ -88,6 +88,13 @@ pub enum Error {
#[snafu(implicit)]
location: Location,
},
#[snafu(display("Unsupported authentication method: {}", method))]
UnsupportedAuthMethod {
method: String,
#[snafu(implicit)]
location: Location,
},
}
impl ErrorExt for Error {
@@ -105,6 +112,7 @@ impl ErrorExt for Error {
Error::UserPasswordMismatch { .. } => StatusCode::UserPasswordMismatch,
Error::AccessDenied { .. } => StatusCode::AccessDenied,
Error::PermissionDenied { .. } => StatusCode::PermissionDenied,
Error::UnsupportedAuthMethod { .. } => StatusCode::UserPasswordMismatch,
}
}
src/auth/src/lib.rs
+1 -1
View File
@@ -34,7 +34,7 @@ pub use permission::{
DefaultPermissionChecker, INFLUXDB_WRITE, JAEGER_QUERY, LOG_QUERY, LOG_WRITE, OPENTSDB_WRITE,
OTLP_WRITE, PIPELINE_DELETE, PIPELINE_INSERT, PIPELINE_QUERY, PROM_STORE_READ,
PROM_STORE_WRITE, PROMQL_QUERY, PermissionAction, PermissionChecker, PermissionReq,
PermissionResp, PermissionTableTarget, PermissionTableTargets,
PermissionResp, PermissionTableTarget, PermissionTableTargets, SEMANTIC_GRAPH_QUERY,
};
pub use user_info::UserInfo;
pub use user_provider::static_user_provider::StaticUserProvider;
src/auth/src/permission.rs
+5
View File
@@ -150,6 +150,10 @@ pub const PIPELINE_DELETE: PermissionAction = PermissionAction::write("pipeline.
pub const DASHBOARD_QUERY: PermissionAction = PermissionAction::read("dashboard.query");
pub const DASHBOARD_SAVE: PermissionAction = PermissionAction::write("dashboard.save");
pub const DASHBOARD_DELETE: PermissionAction = PermissionAction::write("dashboard.delete");
/// Reading a telemetry table through the entity-graph derivation
/// (`greptime_private.semantic_entities` / `semantic_relationships`); checked
/// once per candidate source table with that table as the target.
pub const SEMANTIC_GRAPH_QUERY: PermissionAction = PermissionAction::read("semantic_graph.query");
/// All permission actions built into GreptimeDB.
///
@@ -171,6 +175,7 @@ pub const ALL_ACTIONS: &[PermissionAction] = &[
DASHBOARD_QUERY,
DASHBOARD_SAVE,
DASHBOARD_DELETE,
SEMANTIC_GRAPH_QUERY,
];
#[derive(Debug, Clone)]
src/auth/src/user_info.rs
+47 -48
View File
@@ -21,6 +21,10 @@ use crate::UserInfoRef;
pub trait UserInfo: Debug + Sync + Send {
fn as_any(&self) -> &dyn Any;
fn username(&self) -> &str;
fn is_admin(&self) -> bool {
false
}
}
/// The user permission mode
@@ -38,13 +42,14 @@ impl PermissionMode {
/// - "rw", "readwrite", "read_write" => ReadWrite
/// - "ro", "readonly", "read_only" => ReadOnly
/// - "wo", "writeonly", "write_only" => WriteOnly
/// Returns None if the input string is not a valid permission mode.
pub fn from_str(s: &str) -> Self {
///
/// Returns `None` if the input string is not a valid permission mode.
pub fn from_str(s: &str) -> Option<Self> {
match s.to_lowercase().as_str() {
"readwrite" | "read_write" | "rw" => PermissionMode::ReadWrite,
"readonly" | "read_only" | "ro" => PermissionMode::ReadOnly,
"writeonly" | "write_only" | "wo" => PermissionMode::WriteOnly,
_ => PermissionMode::ReadWrite,
"readwrite" | "read_write" | "rw" => Some(PermissionMode::ReadWrite),
"readonly" | "read_only" | "ro" => Some(PermissionMode::ReadOnly),
"writeonly" | "write_only" | "wo" => Some(PermissionMode::WriteOnly),
_ => None,
}
}
@@ -123,58 +128,72 @@ mod tests {
// Test ReadWrite variants
assert_eq!(
PermissionMode::from_str("readwrite"),
PermissionMode::ReadWrite
Some(PermissionMode::ReadWrite)
);
assert_eq!(
PermissionMode::from_str("read_write"),
PermissionMode::ReadWrite
Some(PermissionMode::ReadWrite)
);
assert_eq!(
PermissionMode::from_str("rw"),
Some(PermissionMode::ReadWrite)
);
assert_eq!(PermissionMode::from_str("rw"), PermissionMode::ReadWrite);
assert_eq!(
PermissionMode::from_str("ReadWrite"),
PermissionMode::ReadWrite
Some(PermissionMode::ReadWrite)
);
assert_eq!(
PermissionMode::from_str("RW"),
Some(PermissionMode::ReadWrite)
);
assert_eq!(PermissionMode::from_str("RW"), PermissionMode::ReadWrite);
// Test ReadOnly variants
assert_eq!(
PermissionMode::from_str("readonly"),
PermissionMode::ReadOnly
Some(PermissionMode::ReadOnly)
);
assert_eq!(
PermissionMode::from_str("read_only"),
PermissionMode::ReadOnly
Some(PermissionMode::ReadOnly)
);
assert_eq!(
PermissionMode::from_str("ro"),
Some(PermissionMode::ReadOnly)
);
assert_eq!(PermissionMode::from_str("ro"), PermissionMode::ReadOnly);
assert_eq!(
PermissionMode::from_str("ReadOnly"),
PermissionMode::ReadOnly
Some(PermissionMode::ReadOnly)
);
assert_eq!(
PermissionMode::from_str("RO"),
Some(PermissionMode::ReadOnly)
);
assert_eq!(PermissionMode::from_str("RO"), PermissionMode::ReadOnly);
// Test WriteOnly variants
assert_eq!(
PermissionMode::from_str("writeonly"),
PermissionMode::WriteOnly
Some(PermissionMode::WriteOnly)
);
assert_eq!(
PermissionMode::from_str("write_only"),
PermissionMode::WriteOnly
Some(PermissionMode::WriteOnly)
);
assert_eq!(
PermissionMode::from_str("wo"),
Some(PermissionMode::WriteOnly)
);
assert_eq!(PermissionMode::from_str("wo"), PermissionMode::WriteOnly);
assert_eq!(
PermissionMode::from_str("WriteOnly"),
PermissionMode::WriteOnly
Some(PermissionMode::WriteOnly)
);
assert_eq!(PermissionMode::from_str("WO"), PermissionMode::WriteOnly);
// Test invalid inputs default to ReadWrite
assert_eq!(
PermissionMode::from_str("invalid"),
PermissionMode::ReadWrite
PermissionMode::from_str("WO"),
Some(PermissionMode::WriteOnly)
);
assert_eq!(PermissionMode::from_str(""), PermissionMode::ReadWrite);
assert_eq!(PermissionMode::from_str("xyz"), PermissionMode::ReadWrite);
for invalid in ["readonyl", "", "xyz"] {
assert_eq!(PermissionMode::from_str(invalid), None);
}
}
#[test]
@@ -200,30 +219,10 @@ mod tests {
for mode in modes {
let str_repr = mode.as_str();
let parsed = PermissionMode::from_str(str_repr);
assert_eq!(mode, parsed);
assert_eq!(Some(mode), parsed);
}
}
#[test]
fn test_default_user_info_with_name() {
let user_info = DefaultUserInfo::with_name("test_user");
assert_eq!(user_info.username(), "test_user");
}
#[test]
fn test_default_user_info_with_name_and_permission() {
let user_info =
DefaultUserInfo::with_name_and_permission("test_user", PermissionMode::ReadOnly);
assert_eq!(user_info.username(), "test_user");
// Cast to DefaultUserInfo to access permission_mode
let default_user = user_info
.as_any()
.downcast_ref::<DefaultUserInfo>()
.unwrap();
assert_eq!(default_user.permission_mode, PermissionMode::ReadOnly);
}
#[test]
fn test_user_info_as_any() {
let user_info = DefaultUserInfo::with_name("test_user");
src/auth/src/user_provider.rs
+36 -3
View File
@@ -34,8 +34,8 @@ use crate::common::{
PgScramSha256Verifier, auth_mysql_with_hash_stage_2,
};
use crate::error::{
IllegalParamSnafu, InvalidConfigSnafu, IoSnafu, Result, UnsupportedPasswordTypeSnafu,
UserNotFoundSnafu, UserPasswordMismatchSnafu,
IllegalParamSnafu, InvalidConfigSnafu, IoSnafu, Result, UnsupportedAuthMethodSnafu,
UnsupportedPasswordTypeSnafu, UserNotFoundSnafu, UserPasswordMismatchSnafu,
};
use crate::user_info::{DefaultUserInfo, PermissionMode};
use crate::{UserInfoRef, auth_mysql};
@@ -66,6 +66,31 @@ pub trait UserProvider: Send + Sync {
Ok(user_info)
}
/// Authenticates and authorizes an opaque bearer token (e.g. a JWT or an
/// OAuth2 access token).
///
/// Unlike [auth()](Self::auth), the caller has no `Identity`/`Password` —
/// the provider validates the token and *derives* the identity from it.
/// The token is opaque to the server, so JWT/JWKS/OIDC validation policy
/// stays pluggable and out of core.
///
/// The default rejects token auth with
/// [`Error::UnsupportedAuthMethod`], so password-only providers keep
/// today's behavior. Providers that support token auth override this to
/// validate the token, resolve it to a user, and
/// [`authorize`](Self::authorize) the connection.
async fn auth_bearer_token(
&self,
_token: &str,
_catalog: &str,
_schema: &str,
) -> Result<UserInfoRef> {
UnsupportedAuthMethodSnafu {
method: "bearer token",
}
.fail()
}
async fn postgres_auth_info(&self, _id: Identity<'_>) -> Result<PgAuthInfo> {
Ok(PgAuthInfo::Cleartext)
}
@@ -438,7 +463,7 @@ pub(crate) fn parse_credential_line(
let (username_part, password) = (parts[0], parts[1]);
let (username, permission_mode) = if let Some((user, perm)) = username_part.split_once(':') {
(user, PermissionMode::from_str(perm))
(user, PermissionMode::from_str(perm)?)
} else {
(username_part, PermissionMode::default())
};
@@ -741,6 +766,14 @@ mod tests {
let result = parse_credential_line("user=pass=word");
assert_eq!(result, None);
for line in [
"user:readonyl=password",
"user:=password",
"user:arbitrary=password",
] {
assert_eq!(parse_credential_line(line), None);
}
// Empty password
let result = parse_credential_line("user=");
assert_eq!(
src/auth/src/user_provider/static_user_provider.rs
+4
View File
@@ -147,6 +147,8 @@ pub mod test {
let provider = StaticUserProvider::new("cmd:root=123456,admin=654321").unwrap();
test_authenticate(&provider, "root", "123456").await;
test_authenticate(&provider, "admin", "654321").await;
assert!(StaticUserProvider::new("cmd:user:readonyl=password").is_err());
}
#[tokio::test]
@@ -171,6 +173,7 @@ pub mod test {
assert!(
lw.write_all(
b"root=123456
invalid:readonyl=password
admin=654321",
)
.is_ok()
@@ -182,5 +185,6 @@ admin=654321",
let provider = StaticUserProvider::new(param.as_str()).unwrap();
test_authenticate(&provider, "root", "123456").await;
test_authenticate(&provider, "admin", "654321").await;
test_authenticate_fails(&provider, "invalid", "password").await;
}
}
src/catalog/src/kvbackend/builder.rs
+1
View File
@@ -101,6 +101,7 @@ impl KvBackendCatalogManagerBuilder {
} = self;
Arc::new_cyclic(|me| KvBackendCatalogManager {
information_extension,
entity_graph_provider: Arc::new(std::sync::OnceLock::new()),
partition_manager: Arc::new(PartitionRuleManager::new(
backend.clone(),
cache_registry
src/catalog/src/kvbackend/manager.rs
+146 -11
View File
@@ -14,11 +14,12 @@
use std::any::Any;
use std::collections::BTreeSet;
use std::sync::{Arc, Weak};
use std::sync::{Arc, OnceLock, Weak};
use async_stream::try_stream;
use common_catalog::consts::{
DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME, INFORMATION_SCHEMA_NAME, PG_CATALOG_NAME,
DEFAULT_CATALOG_NAME, DEFAULT_PRIVATE_SCHEMA_NAME, DEFAULT_SCHEMA_NAME,
INFORMATION_SCHEMA_NAME, PG_CATALOG_NAME,
};
use common_error::ext::BoxedError;
use common_meta::cache::{
@@ -29,7 +30,7 @@ use common_meta::key::TableMetadataManagerRef;
use common_meta::key::catalog_name::CatalogNameKey;
use common_meta::key::flow::FlowMetadataManager;
use common_meta::key::schema_name::SchemaNameKey;
use common_meta::key::table_info::{TableInfoManager, TableInfoValue};
use common_meta::key::table_info::TableInfoValue;
use common_meta::key::table_name::TableNameKey;
use common_meta::kv_backend::KvBackendRef;
use common_procedure::ProcedureManagerRef;
@@ -61,6 +62,7 @@ use crate::process_manager::ProcessManagerRef;
use crate::system_schema::SystemSchemaProvider;
use crate::system_schema::numbers_table_provider::NumbersTableProvider;
use crate::system_schema::pg_catalog::PGCatalogProvider;
use crate::system_schema::semantic_graph::{EntityGraphProviderRef, SemanticGraphTableProvider};
/// Access all existing catalog, schema and tables.
///
@@ -71,6 +73,12 @@ use crate::system_schema::pg_catalog::PGCatalogProvider;
pub struct KvBackendCatalogManager {
/// Provides the extension methods for the `information_schema` tables
pub(super) information_extension: InformationExtensionRef,
/// Backs the computed entity-graph tables (`semantic_entities` /
/// `semantic_relationships`). Set once, after the query engine is built, to
/// break the `catalog -> query` cycle; `None` until then. `Arc` so the cell
/// stays shared across `Clone`s of the manager (a plain `OnceLock` would
/// fork on clone).
pub(super) entity_graph_provider: Arc<OnceLock<EntityGraphProviderRef>>,
/// Manages partition rules.
pub(super) partition_manager: PartitionRuleManagerRef,
/// Manages table metadata.
@@ -97,6 +105,19 @@ impl KvBackendCatalogManager {
self.information_extension.clone()
}
/// Returns the entity-graph provider, or `None` if it has not been injected
/// yet (before the query engine is built). The computed graph tables stream
/// empty until it is set.
pub fn entity_graph_provider(&self) -> Option<EntityGraphProviderRef> {
self.entity_graph_provider.get().cloned()
}
/// Injects the entity-graph provider once the query engine exists. A second
/// call is a no-op (the first binding wins).
pub fn set_entity_graph_provider(&self, provider: EntityGraphProviderRef) {
let _ = self.entity_graph_provider.set(provider);
}
pub fn partition_manager(&self) -> PartitionRuleManagerRef {
self.partition_manager.clone()
}
@@ -112,7 +133,7 @@ impl KvBackendCatalogManager {
// Override logical table's partition key indices with physical table's.
async fn override_logical_table_partition_key_indices(
table_route_cache: &TableRouteCacheRef,
table_info_manager: &TableInfoManager,
table_info_cache: &TableInfoCacheRef,
table: TableRef,
) -> Result<TableRef> {
// If the table is not a metric table, return the table directly.
@@ -125,7 +146,7 @@ impl KvBackendCatalogManager {
.await
.context(TableMetadataManagerSnafu)?
&& let TableRoute::Logical(logical_route) = &*table_route_value
&& let Some(physical_table_info_value) = table_info_manager
&& let Some(physical_table_info) = table_info_cache
.get(logical_route.physical_table_id())
.await
.context(TableMetadataManagerSnafu)?
@@ -135,15 +156,13 @@ impl KvBackendCatalogManager {
let mut phy_part_cols_not_in_logical_table = vec![];
// Remap partition key indices from physical table to logical table
new_table_info.meta.partition_key_indices = physical_table_info_value
.table_info
new_table_info.meta.partition_key_indices = physical_table_info
.meta
.partition_key_indices
.iter()
.filter_map(|&physical_index| {
// Get the column name from the physical table using the physical index
physical_table_info_value
.table_info
physical_table_info
.meta
.schema
.column_schemas()
@@ -321,9 +340,13 @@ impl CatalogManager for KvBackendCatalogManager {
self.cache_registry.get().context(CacheNotFoundSnafu {
name: "table_route_cache",
})?;
let table_info_cache: TableInfoCacheRef =
self.cache_registry.get().context(CacheNotFoundSnafu {
name: "table_info_cache",
})?;
return Self::override_logical_table_partition_key_indices(
&table_route_cache,
self.table_metadata_manager.table_info_manager(),
&table_info_cache,
table,
)
.await
@@ -453,6 +476,10 @@ impl CatalogManager for KvBackendCatalogManager {
self.cache_registry.get().context(CacheNotFoundSnafu {
name: "table_route_cache",
});
let table_info_cache: Result<TableInfoCacheRef> =
self.cache_registry.get().context(CacheNotFoundSnafu {
name: "table_info_cache",
});
common_runtime::spawn_global(async move {
let table_route_cache = match table_route_cache {
@@ -462,6 +489,13 @@ impl CatalogManager for KvBackendCatalogManager {
return;
}
};
let table_info_cache = match table_info_cache {
Ok(table_info_cache) => table_info_cache,
Err(e) => {
let _ = tx.send(Err(e)).await;
return;
}
};
let table_id_stream = metadata_manager
.table_name_manager()
@@ -490,6 +524,7 @@ impl CatalogManager for KvBackendCatalogManager {
let tx = tx.clone();
let semaphore = semaphore.clone();
let table_route_cache = table_route_cache.clone();
let table_info_cache = table_info_cache.clone();
common_runtime::spawn_global(async move {
// we don't explicitly close the semaphore so just ignore the potential error.
let _ = semaphore.acquire().await;
@@ -509,7 +544,7 @@ impl CatalogManager for KvBackendCatalogManager {
for table in table_info_values.into_values().map(build_table) {
let table = Self::override_logical_table_partition_key_indices(
&table_route_cache,
metadata_manager.table_info_manager(),
&table_info_cache,
table,
)
.await;
@@ -577,6 +612,9 @@ impl SystemCatalog {
self.pg_catalog_provider.table_names()
}
DEFAULT_SCHEMA_NAME => self.numbers_table_provider.table_names(),
// Computed entity-graph tables overlay the physical tables of
// `greptime_private` (the caller appends these to the physical list).
DEFAULT_PRIVATE_SCHEMA_NAME => SemanticGraphTableProvider::table_names(),
_ => vec![],
}
}
@@ -597,6 +635,8 @@ impl SystemCatalog {
self.numbers_table_provider.table_exists(table)
} else if schema == PG_CATALOG_NAME && channel == Channel::Postgres {
self.pg_catalog_provider.table(table).is_some()
} else if schema == DEFAULT_PRIVATE_SCHEMA_NAME {
SemanticGraphTableProvider::table_exists(table)
} else {
false
}
@@ -640,8 +680,103 @@ impl SystemCatalog {
}
} else if schema == DEFAULT_SCHEMA_NAME {
self.numbers_table_provider.table(table_name)
} else if schema == DEFAULT_PRIVATE_SCHEMA_NAME
&& SemanticGraphTableProvider::table_exists(table_name)
{
// Constructed on demand (the provider is a name + a weak ref); the
// system catalog is consulted before physical resolution, so the
// computed tables shadow same-named physical tables by design.
SemanticGraphTableProvider::new(
catalog.to_string(),
self.catalog_manager.clone(),
query_ctx.cloned().map(Arc::new),
)
.table(table_name)
} else {
None
}
}
}
#[cfg(test)]
mod tests {
use std::sync::atomic::{AtomicUsize, Ordering};
use common_meta::cache::{CacheContainer, Initializer};
use common_meta::key::table_route::LogicalTableRouteValue;
use moka::future::Cache as FutureCache;
use table::metadata::TableInfo;
use super::*;
fn metric_table_info(
table_id: TableId,
table_name: &str,
partition_key_indices: Vec<usize>,
) -> TableInfo {
let mut table_info =
common_meta::ddl::test_util::create_table::test_create_table_task(table_name, table_id)
.table_info;
table_info.meta.engine = METRIC_ENGINE_NAME.to_string();
table_info.meta.partition_key_indices = partition_key_indices;
table_info
}
#[tokio::test]
async fn test_logical_table_uses_cached_physical_table_info() {
const LOGICAL_TABLE_ID: TableId = 1;
const PHYSICAL_TABLE_ID: TableId = 2;
let route = Arc::new(TableRoute::Logical(Arc::new(LogicalTableRouteValue::new(
PHYSICAL_TABLE_ID,
))));
let route_initializer: Initializer<TableId, Arc<TableRoute>> = Arc::new(move |_| {
let route = route.clone();
Box::pin(async move { Ok(Some(route)) })
});
let table_route_cache = Arc::new(CacheContainer::new(
"test_table_route_cache".to_string(),
FutureCache::new(16),
Box::new(|_, _| Box::pin(async { Ok(()) })),
route_initializer,
|_| true,
));
let load_count = Arc::new(AtomicUsize::new(0));
let physical_table_info =
Arc::new(metric_table_info(PHYSICAL_TABLE_ID, "physical", vec![1]));
let table_info_initializer: Initializer<TableId, Arc<TableInfo>> = {
let load_count = load_count.clone();
let table_info = physical_table_info.clone();
Arc::new(move |_| {
load_count.fetch_add(1, Ordering::Relaxed);
let table_info = table_info.clone();
Box::pin(async move { Ok(Some(table_info)) })
})
};
let table_info_cache = Arc::new(CacheContainer::new(
"test_table_info_cache".to_string(),
FutureCache::new(16),
Box::new(|_, _| Box::pin(async { Ok(()) })),
table_info_initializer,
|_| true,
));
let logical_table = DistTable::table(Arc::new(metric_table_info(
LOGICAL_TABLE_ID,
"logical",
vec![],
)));
for _ in 0..2 {
let table = KvBackendCatalogManager::override_logical_table_partition_key_indices(
&table_route_cache,
&table_info_cache,
logical_table.clone(),
)
.await
.unwrap();
assert_eq!(table.table_info().meta.partition_key_indices, vec![1]);
}
assert_eq!(load_count.load(Ordering::Relaxed), 1);
}
}
src/catalog/src/system_schema.rs
+1
View File
@@ -17,6 +17,7 @@ mod memory_table;
pub mod numbers_table_provider;
pub mod pg_catalog;
pub mod predicate;
pub mod semantic_graph;
mod utils;
use std::collections::HashMap;
src/catalog/src/system_schema/information_schema.rs
+11
View File
@@ -14,6 +14,7 @@
mod cluster_info;
pub mod columns;
pub mod flow_statistics;
pub mod flows;
mod information_memory_table;
pub mod key_column_usage;
@@ -73,6 +74,7 @@ use crate::CatalogManager;
use crate::error::{Error, Result};
use crate::process_manager::ProcessManagerRef;
use crate::system_schema::information_schema::cluster_info::InformationSchemaClusterInfo;
use crate::system_schema::information_schema::flow_statistics::InformationSchemaFlowStatistics;
use crate::system_schema::information_schema::flows::InformationSchemaFlows;
use crate::system_schema::information_schema::information_memory_table::get_schema_columns;
use crate::system_schema::information_schema::key_column_usage::InformationSchemaKeyColumnUsage;
@@ -271,6 +273,11 @@ impl SystemSchemaProviderInner for InformationSchemaProvider {
self.catalog_manager.clone(),
self.flow_metadata_manager.clone(),
)) as _),
FLOW_STATISTICS => Some(Arc::new(InformationSchemaFlowStatistics::new(
self.catalog_name.clone(),
self.catalog_manager.clone(),
self.flow_metadata_manager.clone(),
)) as _),
PROCEDURE_INFO => Some(
Arc::new(procedure_info::InformationSchemaProcedureInfo::new(
self.catalog_manager.clone(),
@@ -406,6 +413,10 @@ impl InformationSchemaProvider {
self.build_table(STATISTICS).unwrap(),
);
tables.insert(FLOWS.to_string(), self.build_table(FLOWS).unwrap());
tables.insert(
FLOW_STATISTICS.to_string(),
self.build_table(FLOW_STATISTICS).unwrap(),
);
#[cfg(feature = "enterprise")]
tables.insert(
RECYCLE_BIN.to_string(),
src/catalog/src/system_schema/information_schema/flow_statistics.rs
+285
View File
@@ -0,0 +1,285 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::{Arc, Weak};
use common_catalog::consts::INFORMATION_SCHEMA_FLOW_STATISTICS_TABLE_ID;
use common_error::ext::BoxedError;
use common_meta::key::FlowId;
use common_meta::key::flow::FlowMetadataManager;
use common_meta::key::flow::flow_state::FlowStat;
use common_recordbatch::adapter::RecordBatchStreamAdapter;
use common_recordbatch::{DfSendableRecordBatchStream, RecordBatch, SendableRecordBatchStream};
use common_time::util::current_time_millis;
use datafusion::execution::TaskContext;
use datafusion::physical_plan::stream::RecordBatchStreamAdapter as DfRecordBatchStreamAdapter;
use datafusion::physical_plan::streaming::PartitionStream as DfPartitionStream;
use datatypes::prelude::ConcreteDataType as CDT;
use datatypes::scalars::ScalarVectorBuilder;
use datatypes::schema::{ColumnSchema, Schema, SchemaRef};
use datatypes::timestamp::TimestampMillisecond;
use datatypes::value::Value;
use datatypes::vectors::{
Int64VectorBuilder, StringVectorBuilder, TimestampMillisecondVectorBuilder,
UInt32VectorBuilder, UInt64VectorBuilder, VectorRef,
};
use futures::TryStreamExt;
use snafu::ResultExt;
use store_api::storage::{ScanRequest, TableId};
use crate::CatalogManager;
use crate::error::{CreateRecordBatchSnafu, InternalSnafu, ListFlowsSnafu, Result};
use crate::information_schema::{FLOW_STATISTICS, Predicates};
use crate::system_schema::information_schema::InformationTable;
use crate::system_schema::utils;
const INIT_CAPACITY: usize = 42;
// rows of information_schema.flow_statistics
pub const FLOW_ID: &str = "flow_id";
pub const FLOW_NAME: &str = "flow_name";
pub const START_TIME: &str = "start_time";
pub const LAST_EXECUTION_TIME: &str = "last_execution_time";
pub const UPTIME_SECONDS: &str = "uptime_seconds";
pub const STATE_SIZE: &str = "state_size";
/// The `information_schema.flow_statistics` provides runtime statistics about flows.
#[derive(Debug)]
pub(super) struct InformationSchemaFlowStatistics {
schema: SchemaRef,
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
flow_metadata_manager: Arc<FlowMetadataManager>,
}
impl InformationSchemaFlowStatistics {
pub(super) fn new(
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
flow_metadata_manager: Arc<FlowMetadataManager>,
) -> Self {
Self {
schema: Self::schema(),
catalog_name,
catalog_manager,
flow_metadata_manager,
}
}
pub(crate) fn schema() -> SchemaRef {
Arc::new(Schema::new(
vec![
(FLOW_ID, CDT::uint32_datatype(), false),
(FLOW_NAME, CDT::string_datatype(), false),
(START_TIME, CDT::timestamp_millisecond_datatype(), true),
(
LAST_EXECUTION_TIME,
CDT::timestamp_millisecond_datatype(),
true,
),
(UPTIME_SECONDS, CDT::int64_datatype(), true),
(STATE_SIZE, CDT::uint64_datatype(), true),
]
.into_iter()
.map(|(name, ty, nullable)| ColumnSchema::new(name, ty, nullable))
.collect(),
))
}
fn builder(&self) -> InformationSchemaFlowStatisticsBuilder {
InformationSchemaFlowStatisticsBuilder::new(
self.schema.clone(),
self.catalog_name.clone(),
self.catalog_manager.clone(),
&self.flow_metadata_manager,
)
}
}
impl InformationTable for InformationSchemaFlowStatistics {
fn table_id(&self) -> TableId {
INFORMATION_SCHEMA_FLOW_STATISTICS_TABLE_ID
}
fn table_name(&self) -> &'static str {
FLOW_STATISTICS
}
fn schema(&self) -> SchemaRef {
self.schema.clone()
}
fn to_stream(&self, request: ScanRequest) -> Result<SendableRecordBatchStream> {
let schema = self.schema.arrow_schema().clone();
let mut builder = self.builder();
let stream = Box::pin(DfRecordBatchStreamAdapter::new(
schema,
futures::stream::once(async move {
builder
.make_flow_statistics(Some(request))
.await
.map(|x| x.into_df_record_batch())
.map_err(|err| datafusion::error::DataFusionError::External(Box::new(err)))
}),
));
Ok(Box::pin(
RecordBatchStreamAdapter::try_new(stream)
.map_err(BoxedError::new)
.context(InternalSnafu)?,
))
}
}
/// Builds the `information_schema.flow_statistics` table row by row.
struct InformationSchemaFlowStatisticsBuilder {
schema: SchemaRef,
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
flow_metadata_manager: Arc<FlowMetadataManager>,
flow_ids: UInt32VectorBuilder,
flow_names: StringVectorBuilder,
start_times: TimestampMillisecondVectorBuilder,
last_execution_times: TimestampMillisecondVectorBuilder,
uptime_seconds: Int64VectorBuilder,
state_sizes: UInt64VectorBuilder,
}
impl InformationSchemaFlowStatisticsBuilder {
fn new(
schema: SchemaRef,
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
flow_metadata_manager: &Arc<FlowMetadataManager>,
) -> Self {
Self {
schema,
catalog_name,
catalog_manager,
flow_metadata_manager: flow_metadata_manager.clone(),
flow_ids: UInt32VectorBuilder::with_capacity(INIT_CAPACITY),
flow_names: StringVectorBuilder::with_capacity(INIT_CAPACITY),
start_times: TimestampMillisecondVectorBuilder::with_capacity(INIT_CAPACITY),
last_execution_times: TimestampMillisecondVectorBuilder::with_capacity(INIT_CAPACITY),
uptime_seconds: Int64VectorBuilder::with_capacity(INIT_CAPACITY),
state_sizes: UInt64VectorBuilder::with_capacity(INIT_CAPACITY),
}
}
/// Construct the `information_schema.flow_statistics` virtual table.
async fn make_flow_statistics(&mut self, request: Option<ScanRequest>) -> Result<RecordBatch> {
let catalog_name = self.catalog_name.clone();
let predicates = Predicates::from_scan_request(&request);
let flow_info_manager = self.flow_metadata_manager.clone();
let mut stream = flow_info_manager
.flow_name_manager()
.flow_names(&catalog_name)
.await;
let flow_stat = {
let information_extension = utils::information_extension(&self.catalog_manager)?;
information_extension.flow_stats().await?
};
let now = current_time_millis();
while let Some((flow_name, flow_id)) = stream
.try_next()
.await
.map_err(BoxedError::new)
.context(ListFlowsSnafu {
catalog: &catalog_name,
})?
{
self.add_flow_statistic(&predicates, flow_id.flow_id(), &flow_name, &flow_stat, now);
}
self.finish()
}
fn add_flow_statistic(
&mut self,
predicates: &Predicates,
flow_id: FlowId,
flow_name: &str,
flow_stat: &Option<FlowStat>,
now: i64,
) {
let row = [
(FLOW_ID, &Value::from(flow_id)),
(FLOW_NAME, &Value::from(flow_name.to_string())),
];
if !predicates.eval(&row) {
return;
}
let start_time = flow_stat
.as_ref()
.and_then(|stat| stat.start_time_map.get(&flow_id).copied());
self.flow_ids.push(Some(flow_id));
self.flow_names.push(Some(flow_name));
self.start_times
.push(start_time.map(TimestampMillisecond::new));
self.last_execution_times
.push(flow_stat.as_ref().and_then(|stat| {
stat.last_exec_time_map
.get(&flow_id)
.map(|v| TimestampMillisecond::new(*v))
}));
self.uptime_seconds
.push(start_time.map(|start| ((now - start) / 1000).max(0)));
self.state_sizes.push(
flow_stat
.as_ref()
.and_then(|stat| stat.state_size.get(&flow_id).map(|v| *v as u64)),
);
}
fn finish(&mut self) -> Result<RecordBatch> {
let columns: Vec<VectorRef> = vec![
Arc::new(self.flow_ids.finish()),
Arc::new(self.flow_names.finish()),
Arc::new(self.start_times.finish()),
Arc::new(self.last_execution_times.finish()),
Arc::new(self.uptime_seconds.finish()),
Arc::new(self.state_sizes.finish()),
];
RecordBatch::new(self.schema.clone(), columns).context(CreateRecordBatchSnafu)
}
}
impl DfPartitionStream for InformationSchemaFlowStatistics {
fn schema(&self) -> &arrow_schema::SchemaRef {
self.schema.arrow_schema()
}
fn execute(&self, _: Arc<TaskContext>) -> DfSendableRecordBatchStream {
let schema: Arc<arrow_schema::Schema> = self.schema.arrow_schema().clone();
let mut builder = self.builder();
Box::pin(DfRecordBatchStreamAdapter::new(
schema,
futures::stream::once(async move {
builder
.make_flow_statistics(None)
.await
.map(|x| x.into_df_record_batch())
.map_err(Into::into)
}),
))
}
}
src/catalog/src/system_schema/information_schema/table_names.rs
+1
View File
@@ -44,6 +44,7 @@ pub const TABLE_CONSTRAINTS: &str = "table_constraints";
pub const CLUSTER_INFO: &str = "cluster_info";
pub const VIEWS: &str = "views";
pub const FLOWS: &str = "flows";
pub const FLOW_STATISTICS: &str = "flow_statistics";
pub const PROCEDURE_INFO: &str = "procedure_info";
pub const REGION_INFO: &str = "region_info";
pub const REGION_STATISTICS: &str = "region_statistics";
src/catalog/src/system_schema/information_schema/table_semantics.rs
+17
View File
@@ -453,4 +453,21 @@ mod tests {
assert!(row.source.is_none());
assert!(row.options_json.is_none());
}
#[test]
fn extract_folds_entity_keys_into_json_tail() {
// Entity keys are not promoted columns; they surface verbatim (prefix-
// stripped, sorted) in the JSON tail with no code change to this table.
let info = table_info(&[
(SEMANTIC_SIGNAL_TYPE, "trace"),
("greptime.semantic.entity.service.id", "service_name"),
("greptime.semantic.entity.host.id", "host_id"),
]);
let row = SemanticRow::extract(&info).unwrap();
assert_eq!(row.signal_type, Some("trace"));
assert_eq!(
row.options_json.as_deref(),
Some(r#"{"entity.host.id":"host_id","entity.service.id":"service_name"}"#)
);
}
}
src/catalog/src/system_schema/semantic_graph.rs
+425
View File
@@ -0,0 +1,425 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The computed entity-graph tables `greptime_private.semantic_entities` and
//! `greptime_private.semantic_relationships`.
//!
//! They live in `greptime_private`, not `information_schema`: scanning them
//! triggers read-time derivation over telemetry tables (trace self-joins, ...),
//! which breaks the "cheap, metadata-only" expectation users have of
//! `information_schema`. `greptime_private` also hosts the physical
//! declared-edge table (`semantic_relationships_declared`), whose rows are
//! unioned into the computed `semantic_relationships`.
//!
//! These are thin forwarders: their rows are derived at read time by the injected
//! [`EntityGraphProvider`], which enumerates the `table_semantics` declarations,
//! builds typed DataFusion derivation plans, and executes them via the query
//! engine. When no provider is injected (e.g. before the engine is up, or on a
//! non-frontend node) they stream empty. The fixed schemas here must match the
//! columns the provider's plans project; JSON columns are `json` (JSONB), whose
//! Arrow storage type is `Binary` — the derived batches are rebuilt against the
//! declared Arrow schema (which carries the `json` extension metadata) in
//! [`SystemTable::to_stream`].
use std::sync::{Arc, LazyLock, Weak};
use common_catalog::consts::{
CONFIDENCE_COLUMN, DEFAULT_PRIVATE_SCHEMA_NAME, DST_ID_COLUMN, DST_TYPE_COLUMN,
DURATION_COUNT_COLUMN, DURATION_SUM_COLUMN, EDGE_ATTRIBUTES_COLUMN, ENTITY_DESCRIPTIVE_COLUMN,
ENTITY_ID_ATTRS_COLUMN, ENTITY_ID_COLUMN, ENTITY_SCOPE_COLUMN, ENTITY_TYPE_COLUMN,
ERROR_COUNT_COLUMN, FRESH_UNTIL_COLUMN, OBSERVED_AT_COLUMN, PROVENANCE_COLUMN, REL_TYPE_COLUMN,
REQUEST_COUNT_COLUMN, SEMANTIC_ENTITIES_TABLE_ID,
SEMANTIC_ENTITIES_TABLE_NAME as SEMANTIC_ENTITIES, SEMANTIC_RELATIONSHIPS_TABLE_ID,
SEMANTIC_RELATIONSHIPS_TABLE_NAME as SEMANTIC_RELATIONSHIPS, SOURCE_TABLES_COLUMN,
SRC_ID_COLUMN, SRC_TYPE_COLUMN, WINDOW_END_COLUMN, WINDOW_START_COLUMN,
};
use common_error::ext::BoxedError;
use common_recordbatch::adapter::AsyncRecordBatchStreamAdapter;
use common_recordbatch::{
DfRecordBatch, EmptyRecordBatchStream, RecordBatch, RecordBatchStreamWrapper,
SendableRecordBatchStream,
};
use datatypes::prelude::ConcreteDataType;
use datatypes::schema::{ColumnSchema, Schema, SchemaRef};
use futures::StreamExt;
use session::context::QueryContextRef;
use snafu::ResultExt;
use store_api::storage::{ScanRequest, TableId};
use table::TableRef;
use crate::CatalogManager;
use crate::error::{InternalSnafu, Result};
use crate::system_schema::{SystemSchemaProviderInner, SystemTable, SystemTableRef, utils};
pub type EntityGraphProviderRef = Arc<dyn EntityGraphProvider>;
/// Produces the rows of the computed entity-graph tables at read time.
///
/// Implemented above the query engine (in the frontend) and injected into the
/// catalog manager *after* construction — the provider needs the engine, which
/// needs the catalog manager — so this late binding breaks the
/// `catalog -> query` dependency cycle.
///
/// `query_ctx` is the outer query's context, captured when the computed table
/// is resolved: the derivation must read only sources that context may read
/// and execute under it, inheriting the caller's permissions, cancellation and
/// deadline (the RFC's derivation contract). `None` (context-less internal
/// resolution) keeps the provider's default behaviour.
#[async_trait::async_trait]
pub trait EntityGraphProvider: Send + Sync {
/// Produces the entity registry (`semantic_entities`) rows for `catalog`.
/// `None` means no source table declared an entity.
async fn scan_entities(
&self,
catalog: &str,
request: ScanRequest,
query_ctx: Option<QueryContextRef>,
) -> std::result::Result<Option<SendableRecordBatchStream>, BoxedError>;
/// Produces the relationship set (`semantic_relationships`) rows for `catalog`.
async fn scan_relationships(
&self,
catalog: &str,
request: ScanRequest,
query_ctx: Option<QueryContextRef>,
) -> std::result::Result<Option<SendableRecordBatchStream>, BoxedError>;
}
/// Serves the computed graph tables under `greptime_private`, overlaid on the
/// schema's physical tables the same way the `numbers` table overlays `public`
/// (the system catalog is consulted before physical table resolution).
pub(crate) struct SemanticGraphTableProvider {
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
/// The resolving query's context; captured per resolution (the provider is
/// built on demand, never cached), so it cannot leak across sessions.
query_ctx: Option<QueryContextRef>,
}
impl SemanticGraphTableProvider {
pub(crate) fn new(
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
query_ctx: Option<QueryContextRef>,
) -> Self {
Self {
catalog_name,
catalog_manager,
query_ctx,
}
}
pub(crate) fn table_names() -> Vec<String> {
vec![
SEMANTIC_ENTITIES.to_string(),
SEMANTIC_RELATIONSHIPS.to_string(),
]
}
pub(crate) fn table_exists(name: &str) -> bool {
name == SEMANTIC_ENTITIES || name == SEMANTIC_RELATIONSHIPS
}
pub(crate) fn table(&self, name: &str) -> Option<TableRef> {
self.build_table(name)
}
}
impl SystemSchemaProviderInner for SemanticGraphTableProvider {
fn catalog_name(&self) -> &str {
&self.catalog_name
}
fn schema_name() -> &'static str {
DEFAULT_PRIVATE_SCHEMA_NAME
}
fn system_table(&self, name: &str) -> Option<SystemTableRef> {
let kind = match name {
SEMANTIC_ENTITIES => GraphTableKind::Entities,
SEMANTIC_RELATIONSHIPS => GraphTableKind::Relationships,
_ => return None,
};
Some(Arc::new(SemanticGraphTable::new(
kind,
self.catalog_name.clone(),
self.catalog_manager.clone(),
self.query_ctx.clone(),
)) as _)
}
}
fn ts() -> ConcreteDataType {
ConcreteDataType::timestamp_millisecond_datatype()
}
fn string() -> ConcreteDataType {
ConcreteDataType::string_datatype()
}
fn json() -> ConcreteDataType {
ConcreteDataType::json_datatype()
}
/// Schema of `semantic_entities` — the node set of the graph, one row per entity
/// observed in a time window. Must match the registry derivation projection.
///
/// Columns:
/// - `observed_at` — TIME INDEX; the 60s time bucket the entity was observed in.
/// - `window_start` — start of that observation window.
/// - `window_end` — end of the window (`window_start` + 60s).
/// - `fresh_until` — time up to which the entity is considered present; equals
/// `window_end` for derived rows (the graph is a sliding window, not a
/// current-state table: an entity exists in a query window only if it has
/// observed evidence there).
/// - `entity_type` — the entity's type, e.g. `service`, `host`, `k8s.pod`,
/// `process`, `service.instance` (the OTel-style, possibly dotted, type).
/// - `entity_id` — canonical identifier: the value verbatim for a
/// single-attribute identity, or a sorted `k=v,k=v` rendering for a composite.
/// - `entity_id_attrs` — JSON object of the identifying attributes (the
/// escaping-safe source of truth for composite ids); NULL for single-attribute ids.
/// - `scope` — namespace/environment the id is scoped to; empty when none.
/// - `descriptive` — JSON snapshot of the entity's descriptive (non-identifying)
/// attributes; NULL when no descriptive columns were declared.
/// - `source_tables` — JSON array of the telemetry tables that contributed this entity.
static ENTITIES_SCHEMA: LazyLock<SchemaRef> = LazyLock::new(|| {
Arc::new(Schema::new(vec![
ColumnSchema::new(OBSERVED_AT_COLUMN, ts(), false).with_time_index(true),
ColumnSchema::new(WINDOW_START_COLUMN, ts(), true),
ColumnSchema::new(WINDOW_END_COLUMN, ts(), true),
ColumnSchema::new(FRESH_UNTIL_COLUMN, ts(), true),
ColumnSchema::new(ENTITY_TYPE_COLUMN, string(), false),
ColumnSchema::new(ENTITY_ID_COLUMN, string(), false),
ColumnSchema::new(ENTITY_ID_ATTRS_COLUMN, json(), true),
ColumnSchema::new(ENTITY_SCOPE_COLUMN, string(), true),
ColumnSchema::new(ENTITY_DESCRIPTIVE_COLUMN, json(), true),
ColumnSchema::new(SOURCE_TABLES_COLUMN, json(), true),
]))
});
/// Schema of `semantic_relationships` — the edge set of the graph, one row per
/// edge observed in a time window. This is the 16-column contract every derived
/// branch and the declared-edge table must project for the top-level `UNION ALL`.
///
/// Columns:
/// - `observed_at` — TIME INDEX; the 60s time bucket the edge was observed in.
/// - `window_start` / `window_end` — the observation window (`window_start` + 60s).
/// - `fresh_until` — time up to which the edge is considered live; equals
/// `window_end` for derived edges (from `valid_until` for declared edges).
/// - `src_type` / `src_id` — type and canonical id of the source endpoint.
/// - `dst_type` / `dst_id` — type and canonical id of the destination endpoint.
/// - `rel_type` — relationship kind, e.g. `calls`, `runs_on`, `contains`,
/// `part_of`, `depends_on` (direction is src → dst; the inverse is a query concern).
/// - `provenance` — how the edge was obtained: `trace` (derived from spans),
/// `attribute` (shared-identity join), `declared` (hand-inserted), or `agent`
/// (agent-inferred). Part of the edge identity, so edges of different provenance
/// for the same pair coexist.
/// - `confidence` — derivation certainty in `[0, 1]`: `1.0` for paired or
/// declared edges, lower for virtual-node or agent-inferred edges. It does
/// not correct for trace sampling.
/// - `request_count` — RED: number of requests over the window (`calls` edges).
/// - `error_count` — RED: number of errored requests over the window.
/// - `duration_sum` — RED: sum of request durations, in seconds, over the window.
/// - `duration_count`— RED: number of durations summed (pair with `duration_sum`
/// to get an average).
/// - `attributes` — JSON of edge attributes, e.g. `connection_type`,
/// `db.system`, `peer.service`.
static RELATIONSHIPS_SCHEMA: LazyLock<SchemaRef> = LazyLock::new(|| {
Arc::new(Schema::new(vec![
ColumnSchema::new(OBSERVED_AT_COLUMN, ts(), false).with_time_index(true),
ColumnSchema::new(WINDOW_START_COLUMN, ts(), true),
ColumnSchema::new(WINDOW_END_COLUMN, ts(), true),
ColumnSchema::new(FRESH_UNTIL_COLUMN, ts(), true),
ColumnSchema::new(SRC_TYPE_COLUMN, string(), false),
ColumnSchema::new(SRC_ID_COLUMN, string(), false),
ColumnSchema::new(DST_TYPE_COLUMN, string(), false),
ColumnSchema::new(DST_ID_COLUMN, string(), false),
ColumnSchema::new(REL_TYPE_COLUMN, string(), false),
ColumnSchema::new(PROVENANCE_COLUMN, string(), false),
ColumnSchema::new(
CONFIDENCE_COLUMN,
ConcreteDataType::float64_datatype(),
true,
),
ColumnSchema::new(
REQUEST_COUNT_COLUMN,
ConcreteDataType::int64_datatype(),
true,
),
ColumnSchema::new(ERROR_COUNT_COLUMN, ConcreteDataType::int64_datatype(), true),
ColumnSchema::new(
DURATION_SUM_COLUMN,
ConcreteDataType::float64_datatype(),
true,
),
ColumnSchema::new(
DURATION_COUNT_COLUMN,
ConcreteDataType::int64_datatype(),
true,
),
ColumnSchema::new(EDGE_ATTRIBUTES_COLUMN, json(), true),
]))
});
/// Which computed table this shell represents, so the two share one forwarder.
#[derive(Clone, Copy)]
enum GraphTableKind {
Entities,
Relationships,
}
/// Forwarder for a computed entity-graph table.
struct SemanticGraphTable {
kind: GraphTableKind,
schema: SchemaRef,
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
query_ctx: Option<QueryContextRef>,
}
impl SemanticGraphTable {
fn new(
kind: GraphTableKind,
catalog_name: String,
catalog_manager: Weak<dyn CatalogManager>,
query_ctx: Option<QueryContextRef>,
) -> Self {
let schema = match kind {
GraphTableKind::Entities => ENTITIES_SCHEMA.clone(),
GraphTableKind::Relationships => RELATIONSHIPS_SCHEMA.clone(),
};
Self {
kind,
schema,
catalog_name,
catalog_manager,
query_ctx,
}
}
async fn derive(
kind: GraphTableKind,
catalog: String,
catalog_manager: Weak<dyn CatalogManager>,
request: ScanRequest,
query_ctx: Option<QueryContextRef>,
) -> Result<Option<SendableRecordBatchStream>> {
let provider = utils::entity_graph_provider(&catalog_manager)?;
// No provider (engine not up / non-frontend node): stream empty.
let Some(provider) = provider else {
return Ok(None);
};
match kind {
GraphTableKind::Entities => provider.scan_entities(&catalog, request, query_ctx).await,
GraphTableKind::Relationships => {
provider
.scan_relationships(&catalog, request, query_ctx)
.await
}
}
.context(InternalSnafu)
}
fn align_schema(
stream: SendableRecordBatchStream,
schema: SchemaRef,
) -> SendableRecordBatchStream {
let batch_schema = schema.clone();
let arrow_schema = schema.arrow_schema().clone();
let batches = stream.map(move |batch| {
let batch = batch?;
// The derivation output is structurally identical, but its JSON
// fields are plain Binary without the declared extension metadata.
let batch = DfRecordBatch::try_new(
arrow_schema.clone(),
batch.into_df_record_batch().columns().to_vec(),
)
.context(common_recordbatch::error::NewDfRecordBatchSnafu)?;
Ok(RecordBatch::from_df_record_batch(
batch_schema.clone(),
batch,
))
});
Box::pin(RecordBatchStreamWrapper::new(schema, Box::pin(batches)))
}
}
impl SystemTable for SemanticGraphTable {
fn table_id(&self) -> TableId {
match self.kind {
GraphTableKind::Entities => SEMANTIC_ENTITIES_TABLE_ID,
GraphTableKind::Relationships => SEMANTIC_RELATIONSHIPS_TABLE_ID,
}
}
fn table_name(&self) -> &'static str {
match self.kind {
GraphTableKind::Entities => SEMANTIC_ENTITIES,
GraphTableKind::Relationships => SEMANTIC_RELATIONSHIPS,
}
}
fn schema(&self) -> SchemaRef {
self.schema.clone()
}
fn to_stream(&self, request: ScanRequest) -> Result<SendableRecordBatchStream> {
let schema = self.schema.clone();
let kind = self.kind;
let catalog = self.catalog_name.clone();
let catalog_manager = self.catalog_manager.clone();
let query_ctx = self.query_ctx.clone();
let stream_schema = schema.clone();
let stream = async move {
let stream = Self::derive(kind, catalog, catalog_manager, request, query_ctx)
.await
.map_err(BoxedError::new)
.context(common_recordbatch::error::ExternalSnafu)?;
Ok(match stream {
Some(stream) => Self::align_schema(stream, stream_schema.clone()),
None => Box::pin(EmptyRecordBatchStream::new(stream_schema.clone())),
})
};
Ok(Box::pin(AsyncRecordBatchStreamAdapter::new(
schema,
Box::pin(stream),
)))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn graph_tables_use_observed_at_as_time_index() {
for schema in [&*ENTITIES_SCHEMA, &*RELATIONSHIPS_SCHEMA] {
assert_eq!(
schema.timestamp_column().map(|column| column.name.as_str()),
Some("observed_at")
);
}
}
#[test]
fn relationship_schema_does_not_expose_generation_id() {
assert!(
RELATIONSHIPS_SCHEMA
.column_schema_by_name("generation_id")
.is_none()
);
}
}
src/catalog/src/system_schema/utils.rs
+17
View File
@@ -21,9 +21,26 @@ use crate::CatalogManager;
use crate::error::{GetInformationExtensionSnafu, Result, UpgradeWeakCatalogManagerRefSnafu};
use crate::information_schema::InformationExtensionRef;
use crate::kvbackend::KvBackendCatalogManager;
use crate::system_schema::semantic_graph::EntityGraphProviderRef;
pub mod tables;
/// Try to get the entity-graph provider from a `[CatalogManager]` weak reference.
/// Returns `None` when the manager is not the kv-backed one or the provider has
/// not been injected yet (the computed graph tables then stream empty).
pub fn entity_graph_provider(
catalog_manager: &Weak<dyn CatalogManager>,
) -> Result<Option<EntityGraphProviderRef>> {
let catalog_manager = catalog_manager
.upgrade()
.context(UpgradeWeakCatalogManagerRefSnafu)?;
Ok(catalog_manager
.as_any()
.downcast_ref::<KvBackendCatalogManager>()
.and_then(|manager| manager.entity_graph_provider()))
}
/// Try to get the `[InformationExtension]` from `[CatalogManager]` weak reference.
pub fn information_extension(
catalog_manager: &Weak<dyn CatalogManager>,
src/cli/src/metadata/control/put/key.rs
+1 -3
View File
@@ -345,9 +345,7 @@ mod tests {
#[test]
fn test_validate_exact_flow_state_key() {
let value = FlowStateValue::new(BTreeMap::new(), BTreeMap::new())
.try_as_raw_value()
.unwrap();
let value = FlowStateValue::default().try_as_raw_value().unwrap();
validate_metadata_value(&flow_state_full_key(), &value).unwrap();
}
src/client/src/database.rs
+336 -154
View File
@@ -12,20 +12,25 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
use std::pin::Pin;
use std::str::FromStr;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, RwLock};
use std::task::{Context, Poll};
use std::time::Duration;
use api::v1::auth_header::AuthScheme;
#[cfg(feature = "testing")]
use api::v1::ddl_request::Expr as DdlExpr;
use api::v1::greptime_database_client::GreptimeDatabaseClient;
use api::v1::greptime_request::Request;
use api::v1::query_request::Query;
#[cfg(feature = "testing")]
use api::v1::{AlterTableExpr, CreateTableExpr, DdlRequest};
use api::v1::{
AlterTableExpr, AuthHeader, Basic, CreateTableExpr, DdlRequest, GreptimeRequest,
InsertRequests, QueryRequest, RequestHeader, RowInsertRequests,
AuthHeader, Basic, GreptimeRequest, InsertRequests, QueryRequest, RequestHeader,
RowInsertRequests,
};
use arc_swap::ArcSwapOption;
use arrow_flight::{FlightData, Ticket};
@@ -34,7 +39,7 @@ use base64::Engine;
use base64::prelude::BASE64_STANDARD;
use common_catalog::build_db_string;
use common_catalog::consts::{DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME};
use common_error::ext::BoxedError;
use common_error::ext::{BoxedError, ErrorExt};
use common_grpc::flight::do_put::DoPutResponse;
use common_grpc::flight::{
FLOW_EXTENSIONS_METADATA_KEY, FlightDecoder, FlightMessage, SNAPSHOT_SEQS_METADATA_KEY,
@@ -49,21 +54,23 @@ use common_telemetry::{error, warn};
use futures::future;
use futures_util::{Stream, StreamExt, TryStreamExt};
use prost::Message;
use snafu::ResultExt;
use snafu::{IntoError, ResultExt};
use tonic::metadata::{AsciiMetadataKey, AsciiMetadataValue, MetadataMap, MetadataValue};
use tonic::transport::Channel;
use crate::error::{
ConvertFlightDataSnafu, Error, FlightGetSnafu, IllegalFlightMessagesSnafu,
ConvertFlightDataSnafu, Error, FlightGetSnafu, FlightStreamSnafu, IllegalFlightMessagesSnafu,
InvalidTonicMetadataValueSnafu,
};
use crate::flight::decode_flight_data;
use crate::flight::{FlightMessageReader, decode_flight_data};
use crate::{Client, Result, error, from_grpc_response};
type FlightDataStream = Pin<Box<dyn Stream<Item = FlightData> + Send>>;
type DoPutResponseStream = Pin<Box<dyn Stream<Item = Result<DoPutResponse>>>>;
const HINTS_METADATA_KEY: &str = "x-greptime-hints";
/// Terminal metrics associated with a query output.
///
/// For streaming outputs, metrics are only final after the stream is fully
@@ -257,19 +264,17 @@ fn attach_terminal_metrics(output: Output, terminal_metrics: &OutputMetrics) ->
}
async fn output_from_flight_message_stream<S>(
mut flight_message_stream: S,
remote_addr: String,
flight_message_stream: S,
) -> Result<OutputWithMetrics>
where
S: Stream<Item = Result<FlightMessage>> + Send + Unpin + 'static,
{
let Some(first_flight_message) = flight_message_stream.next().await else {
return IllegalFlightMessagesSnafu {
reason: "Expect the response not to be empty",
}
.fail();
};
let first_flight_message = first_flight_message?;
let mut reader = FlightMessageReader::new(remote_addr, flight_message_stream);
let first_flight_message = reader
.read_first()
.await
.map_err(|error| flight_stream_error(reader.remote_addr(), error))?;
match first_flight_message {
FlightMessage::AffectedRows { rows, metrics } => {
@@ -277,7 +282,10 @@ where
if let Some(metrics) = metrics {
terminal_metrics.update(Some(parse_terminal_metrics(&metrics)?));
}
let next_message = flight_message_stream.next().await.transpose()?;
let next_message = reader
.read_next()
.await
.map_err(|error| flight_stream_error(reader.remote_addr(), error))?;
match next_message {
None => terminal_metrics.mark_ready(),
Some(FlightMessage::Metrics(s)) if terminal_metrics.get().is_none() => {
@@ -316,15 +324,19 @@ where
);
let schema_cloned = schema.clone();
let stream = Box::pin(stream!({
while let Some(flight_message_item) = flight_message_stream.next().await {
let flight_message = match flight_message_item {
Ok(message) => message,
Err(e) => {
yield Err(BoxedError::new(e)).context(ExternalSnafu);
loop {
let flight_message = match reader.read_next().await {
Ok(Some(message)) => message,
Ok(None) => break,
Err(error) => {
yield Err(BoxedError::new(flight_stream_error(
reader.remote_addr(),
error,
)))
.context(ExternalSnafu);
break;
}
};
match flight_message {
FlightMessage::RecordBatch(arrow_batch) => {
yield Ok(RecordBatch::from_df_record_batch(
@@ -371,6 +383,25 @@ where
}
}
fn flight_stream_error(addr: &str, error: Error) -> Error {
let tonic_code = error.tonic_code().unwrap_or(tonic::Code::Unknown);
let message = error.to_string();
if error.status_code().should_log_error() {
error!(
error; "Failed to receive Flight data, addr: {}, code: {}",
addr,
tonic_code
);
}
FlightStreamSnafu {
addr: addr.to_string(),
tonic_code,
message,
}
.into_error(BoxedError::new(error))
}
#[derive(Clone, Debug, Default)]
pub struct Database {
// The "catalog" and "schema" to be used in processing the requests at the server side.
@@ -389,6 +420,42 @@ pub struct Database {
ctx: FlightContext,
}
#[derive(Default)]
struct FlightRequestOptions {
hints: Option<String>,
flow_extensions: Option<String>,
snapshot_seqs: Option<String>,
timeout: Option<Duration>,
}
impl FlightRequestOptions {
fn apply_to<T>(self, request: &mut tonic::Request<T>) -> Result<()> {
let metadata = request.metadata_mut();
if let Some(hints) = self.hints {
Database::put_metadata_value(metadata, HINTS_METADATA_KEY, hints)?;
}
if let Some(flow_extensions) = self.flow_extensions {
Database::put_metadata_value(metadata, FLOW_EXTENSIONS_METADATA_KEY, flow_extensions)?;
}
if let Some(snapshot_seqs) = self.snapshot_seqs {
Database::put_metadata_value(metadata, SNAPSHOT_SEQS_METADATA_KEY, snapshot_seqs)?;
}
if let Some(timeout) = self.timeout {
request.set_timeout(timeout);
}
Ok(())
}
}
/// A single Flight DoGet request to a [`Database`].
///
/// The builder carries request-scoped metadata and options. It does not modify
/// the underlying [`Database`], so its configuration cannot affect later RPCs.
pub struct DatabaseFlightRequest<'a> {
database: &'a Database,
options: FlightRequestOptions,
}
pub struct DatabaseClient {
pub addr: String,
pub inner: GreptimeDatabaseClient<Channel>,
@@ -483,6 +550,14 @@ impl Database {
});
}
/// Creates a builder for a single Flight DoGet request.
pub fn flight_request(&self) -> DatabaseFlightRequest<'_> {
DatabaseFlightRequest {
database: self,
options: FlightRequestOptions::default(),
}
}
/// Make an InsertRequests request to the database.
pub async fn insert(&self, requests: InsertRequests) -> Result<u32> {
self.handle(Request::Inserts(requests)).await
@@ -538,44 +613,31 @@ impl Database {
}
fn put_hints(metadata: &mut MetadataMap, hints: &[(&str, &str)]) -> Result<()> {
let Some(value) = hints
.iter()
.map(|(k, v)| format!("{}={}", k, v))
.reduce(|a, b| format!("{},{}", a, b))
else {
let Some(value) = Self::encode_hints(hints) else {
return Ok(());
};
let key = AsciiMetadataKey::from_static("x-greptime-hints");
let value = AsciiMetadataValue::from_str(&value).context(InvalidTonicMetadataValueSnafu)?;
metadata.insert(key, value);
Ok(())
Self::put_metadata_value(metadata, HINTS_METADATA_KEY, value)
}
fn put_flow_extensions(
metadata: &mut MetadataMap,
flow_extensions: &[(&str, &str)],
) -> Result<()> {
if flow_extensions.is_empty() {
return Ok(());
}
let value = serde_json::to_string(&flow_extensions.to_vec())
.expect("flow extension pairs should serialize");
Self::put_metadata_value(metadata, FLOW_EXTENSIONS_METADATA_KEY, value)
fn encode_hints(hints: &[(&str, &str)]) -> Option<String> {
hints
.iter()
.map(|(k, v)| format!("{}={}", k, v))
.reduce(|a, b| format!("{},{}", a, b))
}
fn put_snapshot_seqs(
metadata: &mut MetadataMap,
snapshot_seqs: &std::collections::HashMap<u64, u64>,
) -> Result<()> {
if snapshot_seqs.is_empty() {
return Ok(());
}
fn encode_flow_extensions(flow_extensions: &[(&str, &str)]) -> Option<String> {
(!flow_extensions.is_empty()).then(|| {
serde_json::to_string(&flow_extensions.to_vec())
.expect("flow extension pairs should serialize")
})
}
let value =
serde_json::to_string(snapshot_seqs).expect("snapshot sequence map should serialize");
Self::put_metadata_value(metadata, SNAPSHOT_SEQS_METADATA_KEY, value)
fn encode_snapshot_seqs(snapshot_seqs: &HashMap<u64, u64>) -> Option<String> {
(!snapshot_seqs.is_empty()).then(|| {
serde_json::to_string(snapshot_seqs).expect("snapshot sequence map should serialize")
})
}
fn put_metadata_value(
@@ -673,7 +735,7 @@ impl Database {
where
S: AsRef<str>,
{
self.sql_with_hint(sql, &[]).await
self.flight_request().sql(sql).await
}
/// Executes a SQL query with optional hints for query optimization.
@@ -681,12 +743,7 @@ impl Database {
where
S: AsRef<str>,
{
let request = Request::Query(QueryRequest {
query: Some(Query::Sql(sql.as_ref().to_string())),
});
self.do_get(request, hints, &[], &Default::default())
.await
.map(OutputWithMetrics::into_output)
self.flight_request().with_hints(hints).sql(sql).await
}
/// Executes a SQL query and returns the output with terminal metrics.
@@ -701,77 +758,33 @@ impl Database {
where
S: AsRef<str>,
{
self.query_with_terminal_metrics_and_flow_extensions(
QueryRequest {
query: Some(Query::Sql(sql.as_ref().to_string())),
},
hints,
&[],
&Default::default(),
)
.await
self.flight_request()
.with_hints(hints)
.sql_with_terminal_metrics(sql)
.await
}
/// Executes a logical plan directly without SQL parsing.
pub async fn logical_plan(&self, logical_plan: Vec<u8>) -> Result<Output> {
self.query_with_terminal_metrics_and_flow_extensions(
QueryRequest {
query: Some(Query::LogicalPlan(logical_plan)),
},
&[],
&[],
&Default::default(),
)
.await
.map(OutputWithMetrics::into_output)
}
/// Executes a query and carries flow extensions through Flight metadata.
///
/// This is the lower-level terminal-metrics API for Flow callers that need
/// to pass JSON-bearing flow extensions without going through hint metadata.
pub async fn query_with_terminal_metrics_and_flow_extensions(
&self,
request: QueryRequest,
hints: &[(&str, &str)],
flow_extensions: &[(&str, &str)],
snapshot_seqs: &std::collections::HashMap<u64, u64>,
) -> Result<OutputWithMetrics> {
self.do_get(
Request::Query(request),
hints,
flow_extensions,
snapshot_seqs,
)
.await
self.flight_request().logical_plan(logical_plan).await
}
/// Creates a new table using the provided table expression.
#[cfg(feature = "testing")]
pub async fn create(&self, expr: CreateTableExpr) -> Result<Output> {
let request = Request::Ddl(DdlRequest {
expr: Some(DdlExpr::CreateTable(expr)),
});
self.do_get(request, &[], &[], &Default::default())
.await
.map(OutputWithMetrics::into_output)
self.flight_request().create(expr).await
}
/// Alters an existing table using the provided alter expression.
#[cfg(feature = "testing")]
pub async fn alter(&self, expr: AlterTableExpr) -> Result<Output> {
let request = Request::Ddl(DdlRequest {
expr: Some(DdlExpr::AlterTable(expr)),
});
self.do_get(request, &[], &[], &Default::default())
.await
.map(OutputWithMetrics::into_output)
self.flight_request().alter(expr).await
}
async fn do_get(
&self,
request: Request,
hints: &[(&str, &str)],
flow_extensions: &[(&str, &str)],
snapshot_seqs: &std::collections::HashMap<u64, u64>,
options: FlightRequestOptions,
) -> Result<OutputWithMetrics> {
let request = self.to_rpc_request(request);
let request = Ticket {
@@ -779,12 +792,10 @@ impl Database {
};
let mut request = tonic::Request::new(request);
let metadata = request.metadata_mut();
Self::put_hints(metadata, hints)?;
Self::put_flow_extensions(metadata, flow_extensions)?;
Self::put_snapshot_seqs(metadata, snapshot_seqs)?;
options.apply_to(&mut request)?;
let mut client = self.client.make_flight_client(false, false)?;
let remote_addr = client.addr().to_string();
let response = client.mut_inner().do_get(request).await.or_else(|e| {
let tonic_code = e.code();
@@ -796,7 +807,7 @@ impl Database {
e
);
Err(BoxedError::new(e)).with_context(|_| FlightGetSnafu {
addr: client.addr().to_string(),
addr: remote_addr.clone(),
tonic_code,
})
})?;
@@ -808,7 +819,7 @@ impl Database {
future::ready(decode_flight_data(&mut decoder, flight_data))
});
output_from_flight_message_stream(flight_message_stream).await
output_from_flight_message_stream(remote_addr, flight_message_stream).await
}
/// Ingest a stream of [RecordBatch]es that belong to a table, using Arrow Flight's "`DoPut`"
@@ -847,6 +858,98 @@ impl Database {
}
}
impl<'a> DatabaseFlightRequest<'a> {
/// Adds query optimization hints to this Flight request.
pub fn with_hints(mut self, hints: &[(&str, &str)]) -> Self {
self.options.hints = Database::encode_hints(hints);
self
}
/// Adds Flow extensions to this Flight request.
pub fn with_flow_extensions(mut self, flow_extensions: &[(&str, &str)]) -> Self {
self.options.flow_extensions = Database::encode_flow_extensions(flow_extensions);
self
}
/// Adds snapshot sequence fences to this Flight request.
pub fn with_snapshot_seqs(mut self, snapshot_seqs: &HashMap<u64, u64>) -> Self {
self.options.snapshot_seqs = Database::encode_snapshot_seqs(snapshot_seqs);
self
}
/// Sets a timeout for this Flight request only.
pub fn with_timeout(mut self, timeout: Duration) -> Self {
self.options.timeout = Some(timeout);
self
}
/// Executes a SQL query.
pub async fn sql<S>(self, sql: S) -> Result<Output>
where
S: AsRef<str>,
{
let request = Request::Query(QueryRequest {
query: Some(Query::Sql(sql.as_ref().to_string())),
});
self.do_get(request)
.await
.map(OutputWithMetrics::into_output)
}
/// Executes a SQL query and returns terminal metrics.
pub async fn sql_with_terminal_metrics<S>(self, sql: S) -> Result<OutputWithMetrics>
where
S: AsRef<str>,
{
self.query_with_terminal_metrics(QueryRequest {
query: Some(Query::Sql(sql.as_ref().to_string())),
})
.await
}
/// Executes a logical plan directly without SQL parsing.
pub async fn logical_plan(self, logical_plan: Vec<u8>) -> Result<Output> {
self.query_with_terminal_metrics(QueryRequest {
query: Some(Query::LogicalPlan(logical_plan)),
})
.await
.map(OutputWithMetrics::into_output)
}
/// Executes a query and returns terminal metrics.
pub async fn query_with_terminal_metrics(
self,
request: QueryRequest,
) -> Result<OutputWithMetrics> {
self.do_get(Request::Query(request)).await
}
/// Creates a new table using the provided table expression.
#[cfg(feature = "testing")]
pub async fn create(self, expr: CreateTableExpr) -> Result<Output> {
self.do_get(Request::Ddl(DdlRequest {
expr: Some(DdlExpr::CreateTable(expr)),
}))
.await
.map(OutputWithMetrics::into_output)
}
/// Alters an existing table using the provided alter expression.
#[cfg(feature = "testing")]
pub async fn alter(self, expr: AlterTableExpr) -> Result<Output> {
self.do_get(Request::Ddl(DdlRequest {
expr: Some(DdlExpr::AlterTable(expr)),
}))
.await
.map(OutputWithMetrics::into_output)
}
async fn do_get(self, request: Request) -> Result<OutputWithMetrics> {
let Self { database, options } = self;
database.do_get(request, options).await
}
}
/// by grpc standard, only `Unavailable` is retryable, see: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md#status-codes-and-their-use-in-grpc
pub fn is_grpc_retryable(err: &tonic::Status) -> bool {
matches!(err.code(), tonic::Code::Unavailable)
@@ -934,12 +1037,14 @@ mod tests {
#[test]
fn test_put_flow_extensions_preserves_comma_bearing_values() {
let mut metadata = MetadataMap::new();
Database::put_flow_extensions(
Database::put_metadata_value(
&mut metadata,
&[
FLOW_EXTENSIONS_METADATA_KEY,
Database::encode_flow_extensions(&[
("flow.return_region_seq", "true"),
("flow.incremental_after_seqs", r#"{"1":10,"2":20}"#),
],
])
.unwrap(),
)
.unwrap();
@@ -969,7 +1074,12 @@ mod tests {
(9_007_199_254_740_993_u64, 9_007_199_254_740_995_u64),
]);
Database::put_snapshot_seqs(&mut metadata, &snapshot_seqs).unwrap();
Database::put_metadata_value(
&mut metadata,
SNAPSHOT_SEQS_METADATA_KEY,
Database::encode_snapshot_seqs(&snapshot_seqs).unwrap(),
)
.unwrap();
let value = metadata
.get(SNAPSHOT_SEQS_METADATA_KEY)
@@ -980,6 +1090,50 @@ mod tests {
assert_eq!(decoded, snapshot_seqs);
}
#[test]
fn test_flight_request_builder_applies_request_options() {
let database = Database::new("greptime", "public", Client::default());
let snapshot_seqs = HashMap::from([(42, 99)]);
let request = database
.flight_request()
.with_hints(&[("query_parallelism", "1")])
.with_flow_extensions(&[("flow.return_region_seq", "true")])
.with_snapshot_seqs(&snapshot_seqs)
.with_timeout(Duration::from_millis(50));
let mut tonic_request = tonic::Request::new(());
request.options.apply_to(&mut tonic_request).unwrap();
let metadata = tonic_request.metadata();
assert_eq!(
metadata.get(HINTS_METADATA_KEY).unwrap(),
"query_parallelism=1"
);
assert_eq!(
serde_json::from_str::<Vec<(String, String)>>(
metadata
.get(FLOW_EXTENSIONS_METADATA_KEY)
.unwrap()
.to_str()
.unwrap(),
)
.unwrap(),
vec![("flow.return_region_seq".to_string(), "true".to_string())]
);
assert_eq!(
serde_json::from_str::<HashMap<u64, u64>>(
metadata
.get(SNAPSHOT_SEQS_METADATA_KEY)
.unwrap()
.to_str()
.unwrap(),
)
.unwrap(),
snapshot_seqs
);
assert!(metadata.get("grpc-timeout").is_some());
}
#[test]
fn test_flight_ctx() {
let mut ctx = FlightContext::default();
@@ -1020,6 +1174,28 @@ mod tests {
assert_eq!(expected.should_retry(), actual.should_retry());
}
#[test]
fn test_flight_stream_error_preserves_addr_and_message() {
let error = flight_stream_error(
"127.0.0.1:4001",
Status::out_of_range("message length too large").into(),
);
assert!(matches!(
&error,
Error::FlightStream {
addr,
tonic_code: Code::OutOfRange,
message,
..
} if addr == "127.0.0.1:4001" && message == "message length too large"
));
assert_eq!(
"Failed to receive Flight data from 127.0.0.1:4001, code: Operation was attempted past the valid range: message length too large",
error.to_string(),
);
}
#[test]
fn test_from_tonic_status_with_retry_hint() {
let mut headers = HeaderMap::new();
@@ -1117,13 +1293,13 @@ mod tests {
#[tokio::test]
async fn test_affected_rows_inline_metrics_are_parsed() {
let output = output_from_flight_message_stream(futures_util::stream::iter(vec![Ok(
FlightMessage::AffectedRows {
let output = output_from_flight_message_stream(
"test-peer".to_string(),
futures_util::stream::iter(vec![Ok(FlightMessage::AffectedRows {
rows: 3,
metrics: Some(terminal_metrics_json()),
},
)]
as Vec<Result<FlightMessage>>))
})] as Vec<Result<FlightMessage>>),
)
.await
.unwrap();
@@ -1138,14 +1314,16 @@ mod tests {
#[tokio::test]
async fn test_affected_rows_inline_metrics_rejects_trailing_metrics() {
let metrics_json = terminal_metrics_json();
let err = output_from_flight_message_stream(futures_util::stream::iter(vec![
Ok(FlightMessage::AffectedRows {
rows: 3,
metrics: Some(metrics_json.clone()),
}),
Ok(FlightMessage::Metrics(metrics_json)),
]
as Vec<Result<FlightMessage>>))
let err = output_from_flight_message_stream(
"test-peer".to_string(),
futures_util::stream::iter(vec![
Ok(FlightMessage::AffectedRows {
rows: 3,
metrics: Some(metrics_json.clone()),
}),
Ok(FlightMessage::Metrics(metrics_json)),
] as Vec<Result<FlightMessage>>),
)
.await
.unwrap_err();
@@ -1167,12 +1345,14 @@ mod tests {
vec![Arc::new(Int32Vector::from_slice([1])) as VectorRef],
)
.unwrap();
let output = output_from_flight_message_stream(futures_util::stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::RecordBatch(batch.into_df_record_batch())),
Ok(FlightMessage::Metrics("{not-json}".to_string())),
]
as Vec<Result<FlightMessage>>))
let output = output_from_flight_message_stream(
"test-peer".to_string(),
futures_util::stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::RecordBatch(batch.into_df_record_batch())),
Ok(FlightMessage::Metrics("{not-json}".to_string())),
] as Vec<Result<FlightMessage>>),
)
.await
.unwrap();
let terminal_metrics = output.metrics.clone();
@@ -1211,18 +1391,20 @@ mod tests {
vec![Arc::new(Int32Vector::from_slice([2])) as VectorRef],
)
.unwrap();
let output = output_from_flight_message_stream(futures_util::stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::RecordBatch(
first_batch.into_df_record_batch(),
)),
Ok(FlightMessage::Metrics(terminal_metrics_json_with_seq(1))),
Ok(FlightMessage::RecordBatch(
second_batch.into_df_record_batch(),
)),
Ok(FlightMessage::Metrics(terminal_metrics_json_with_seq(2))),
]
as Vec<Result<FlightMessage>>))
let output = output_from_flight_message_stream(
"test-peer".to_string(),
futures_util::stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::RecordBatch(
first_batch.into_df_record_batch(),
)),
Ok(FlightMessage::Metrics(terminal_metrics_json_with_seq(1))),
Ok(FlightMessage::RecordBatch(
second_batch.into_df_record_batch(),
)),
Ok(FlightMessage::Metrics(terminal_metrics_json_with_seq(2))),
] as Vec<Result<FlightMessage>>),
)
.await
.unwrap();
let terminal_metrics = output.metrics.clone();
src/client/src/error.rs
+18
View File
@@ -40,6 +40,21 @@ pub enum Error {
source: BoxedError,
},
#[snafu(display(
"Failed to receive Flight data from {}, code: {}: {}",
addr,
tonic_code,
message
))]
FlightStream {
addr: String,
tonic_code: Code,
message: String,
source: BoxedError,
#[snafu(implicit)]
location: Location,
},
#[snafu(display("Failed to convert FlightData"))]
ConvertFlightData {
#[snafu(implicit)]
@@ -154,6 +169,7 @@ impl ErrorExt for Error {
Error::Server { code, .. } | Error::Tonic { code, .. } => *code,
Error::FlightGet { source, .. }
| Error::FlightStream { source, .. }
| Error::RegionServer { source, .. }
| Error::FlowServer { source, .. } => source.status_code(),
Error::CreateChannel { source, .. }
@@ -174,6 +190,7 @@ impl ErrorExt for Error {
match self {
Error::Tonic { retry_hint, .. } => *retry_hint,
Error::FlightGet { source, .. }
| Error::FlightStream { source, .. }
| Error::RegionServer { source, .. }
| Error::FlowServer { source, .. }
| Error::External { source, .. } => source.retry_hint(),
@@ -193,6 +210,7 @@ impl Error {
pub fn tonic_code(&self) -> Option<Code> {
match self {
Self::FlightGet { tonic_code, .. }
| Self::FlightStream { tonic_code, .. }
| Self::RegionServer {
code: tonic_code, ..
}
src/client/src/flight.rs
+72 -2
View File
@@ -12,12 +12,82 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::pin::Pin;
use arrow_flight::FlightData;
use common_grpc::flight::{FlightDecoder, FlightMessage};
use snafu::ResultExt;
use futures_util::stream::Peekable;
use futures_util::{Stream, StreamExt};
use snafu::{OptionExt, ResultExt};
use crate::Result;
use crate::error::{ConvertFlightDataSnafu, Error};
use crate::error::{ConvertFlightDataSnafu, Error, IllegalFlightMessagesSnafu};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum FlightMessageKind {
Schema,
RecordBatch,
AffectedRows,
Metrics,
}
impl From<&FlightMessage> for FlightMessageKind {
fn from(message: &FlightMessage) -> Self {
match message {
FlightMessage::Schema(_) => Self::Schema,
FlightMessage::RecordBatch(_) => Self::RecordBatch,
FlightMessage::AffectedRows { .. } => Self::AffectedRows,
FlightMessage::Metrics(_) => Self::Metrics,
}
}
}
pub(crate) struct FlightMessageReader<S: Stream + Unpin> {
/// Remote Flight peer associated with this response stream.
remote_addr: String,
messages: Peekable<S>,
}
impl<S> FlightMessageReader<S>
where
S: Stream<Item = Result<FlightMessage>> + Unpin,
{
pub(crate) fn new(remote_addr: impl Into<String>, messages: S) -> Self {
Self {
remote_addr: remote_addr.into(),
messages: messages.peekable(),
}
}
pub(crate) fn remote_addr(&self) -> &str {
&self.remote_addr
}
pub(crate) async fn read_first(&mut self) -> Result<FlightMessage> {
self.read_next().await?.context(IllegalFlightMessagesSnafu {
reason: "Expect the response not to be empty",
})
}
pub(crate) async fn read_next(&mut self) -> Result<Option<FlightMessage>> {
self.messages.next().await.transpose()
}
pub(crate) async fn peek_next_message_kind(&mut self) -> Result<Option<FlightMessageKind>> {
match Pin::new(&mut self.messages).peek().await {
Some(Ok(message)) => Ok(Some(message.into())),
None => Ok(None),
Some(Err(_)) => match self.read_next().await {
// `peek` only borrows the error; consume it to preserve the source error.
Err(error) => Err(error),
Ok(_) => IllegalFlightMessagesSnafu {
reason: "Flight stream changed after peek".to_string(),
}
.fail(),
},
}
}
}
pub(crate) fn decode_flight_data(
decoder: &mut FlightDecoder,
src/client/src/region.rs
+222 -65
View File
@@ -13,6 +13,7 @@
// limitations under the License.
use std::sync::Arc;
use std::time::Duration;
use api::region::RegionResponse;
use api::v1::ResponseHeader;
@@ -24,7 +25,7 @@ use arc_swap::ArcSwapOption;
use arrow_flight::Ticket;
use async_stream::stream;
use async_trait::async_trait;
use common_error::ext::BoxedError;
use common_error::ext::{BoxedError, ErrorExt};
use common_error::status_code::StatusCode;
use common_grpc::flight::{FlightDecoder, FlightMessage};
use common_meta::error::{self as meta_error, Result as MetaResult};
@@ -46,9 +47,11 @@ use crate::error::{
self, FlightGetSnafu, IllegalDatabaseResponseSnafu, IllegalFlightMessagesSnafu,
MissingFieldSnafu, Result, ServerSnafu,
};
use crate::flight::decode_flight_data;
use crate::flight::{FlightMessageKind, FlightMessageReader, decode_flight_data};
use crate::{Client, metrics};
const FLIGHT_DO_GET_TIMEOUT: Duration = Duration::from_secs(10);
#[derive(Debug)]
pub struct RegionRequester {
client: Client,
@@ -109,20 +112,24 @@ impl RegionRequester {
let mut flight_client = self
.client
.make_flight_client(self.send_compression, self.accept_compression)?;
// Limit Flight DoGet response time without limiting query stream execution.
let addr = flight_client.addr().to_string();
let mut request = tonic::Request::new(ticket);
request.set_timeout(FLIGHT_DO_GET_TIMEOUT);
let response = flight_client
.mut_inner()
.do_get(ticket)
.do_get(request)
.await
.or_else(|e| {
let tonic_code = e.code();
let e: error::Error = e.into();
error!(
e; "Failed to do Flight get, addr: {}, code: {}",
flight_client.addr(),
addr,
tonic_code
);
Err(BoxedError::new(e)).with_context(|_| FlightGetSnafu {
addr: flight_client.addr().to_string(),
addr: addr.clone(),
tonic_code,
})
})?;
@@ -133,7 +140,7 @@ impl RegionRequester {
let flight_message_stream = flight_data_stream
.filter_map(move |flight_data| decode_flight_data(&mut decoder, flight_data));
recordbatches_from_flight_message_stream(flight_message_stream).await
recordbatches_from_flight_message_stream(addr, flight_message_stream).await
}
async fn handle_inner(&self, request: RegionRequest) -> Result<RegionResponse> {
@@ -195,22 +202,23 @@ impl RegionRequester {
}
async fn recordbatches_from_flight_message_stream<S>(
mut flight_message_stream: S,
addr: String,
flight_message_stream: S,
) -> Result<SendableRecordBatchStream>
where
S: Stream<Item = Result<FlightMessage>> + Send + Unpin + 'static,
{
let Some(first_flight_message) = flight_message_stream.next().await else {
return IllegalFlightMessagesSnafu {
reason: "Expect the response not to be empty",
}
.fail();
};
let FlightMessage::Schema(schema) = first_flight_message? else {
let mut reader = FlightMessageReader::new(addr.clone(), flight_message_stream);
let FlightMessage::Schema(schema) = reader
.read_first()
.await
.map_err(|error| flight_stream_error(reader.remote_addr(), error))?
else {
return IllegalFlightMessagesSnafu {
reason: "Expect schema to be the first flight message",
}
.fail();
.fail()
.map_err(|error| flight_stream_error(reader.remote_addr(), error));
};
let metrics = Arc::new(ArcSwapOption::from(None));
@@ -221,29 +229,23 @@ where
let schema =
Arc::new(datatypes::schema::Schema::try_from(schema).context(error::ConvertSchemaSnafu)?);
let schema_cloned = schema.clone();
let stream_addr = addr;
let stream = Box::pin(stream!({
let _span = tracing_context.attach(common_telemetry::tracing::info_span!(
"poll_flight_data_stream"
));
let mut buffered_message: Option<FlightMessage> = None;
let mut stream_ended = false;
while !stream_ended {
// get the next message from the buffered message or read from the flight message stream
let flight_message_item = if let Some(msg) = buffered_message.take() {
Some(Ok(msg))
} else {
flight_message_stream.next().await
};
let flight_message = match flight_message_item {
Some(Ok(message)) => message,
Some(Err(e)) => {
yield Err(BoxedError::new(e)).context(ExternalSnafu);
let flight_message = match reader.read_next().await {
Ok(Some(message)) => message,
Ok(None) => break,
Err(error) => {
yield Err(BoxedError::new(flight_stream_error(&stream_addr, error)))
.context(ExternalSnafu);
break;
}
None => break,
};
match flight_message {
@@ -251,41 +253,54 @@ where
let result_to_yield =
RecordBatch::from_df_record_batch(schema_cloned.clone(), record_batch);
// get the next message from the stream. normally it should be a metrics message.
if let Some(next_flight_message_result) = flight_message_stream.next().await {
match next_flight_message_result {
Ok(FlightMessage::Metrics(s)) => {
let m = serde_json::from_str(&s).ok().map(Arc::new);
metrics_ref.swap(m);
}
Ok(FlightMessage::RecordBatch(rb)) => {
// for some reason it's not a metrics message, so we need to buffer this record batch
// and yield it in the next iteration.
buffered_message = Some(FlightMessage::RecordBatch(rb));
}
Ok(_) => {
yield IllegalFlightMessagesSnafu {
reason: "A RecordBatch message can only be succeeded by a Metrics message or another RecordBatch message"
// Metrics follow a batch so MergeScan can observe them before yielding it.
match reader.peek_next_message_kind().await {
Ok(Some(FlightMessageKind::Metrics)) => {
let metrics_message = match reader.read_next().await {
Ok(Some(FlightMessage::Metrics(metrics))) => metrics,
Ok(Some(_) | None) => {
yield IllegalFlightMessagesSnafu {
reason: "Flight stream changed after peek",
}
.fail()
.map_err(BoxedError::new)
.context(ExternalSnafu);
break;
}
.fail()
.map_err(BoxedError::new)
Err(error) => {
yield Err(BoxedError::new(flight_stream_error(
&stream_addr,
error,
)))
.context(ExternalSnafu);
break;
}
};
let metrics = serde_json::from_str(&metrics_message).ok().map(Arc::new);
metrics_ref.swap(metrics);
}
Ok(Some(FlightMessageKind::RecordBatch)) => {}
Ok(Some(FlightMessageKind::Schema | FlightMessageKind::AffectedRows)) => {
yield IllegalFlightMessagesSnafu {
reason: "A RecordBatch message can only be succeeded by a Metrics message or another RecordBatch message"
}
.fail()
.map_err(BoxedError::new)
.context(ExternalSnafu);
break;
}
Ok(None) => stream_ended = true,
Err(error) => {
yield Err(BoxedError::new(flight_stream_error(&stream_addr, error)))
.context(ExternalSnafu);
break;
}
Err(e) => {
yield Err(BoxedError::new(e)).context(ExternalSnafu);
break;
}
break;
}
} else {
// the stream has ended
stream_ended = true;
}
yield Ok(result_to_yield);
}
FlightMessage::Metrics(s) => {
// just a branch in case of some metrics message comes after other things.
// Metrics may arrive before the next RecordBatch.
let m = serde_json::from_str(&s).ok().map(Arc::new);
metrics_ref.swap(m);
continue;
@@ -312,6 +327,23 @@ where
Ok(Box::pin(record_batch_stream))
}
fn flight_stream_error(addr: &str, error: error::Error) -> error::Error {
let tonic_code = error.tonic_code().unwrap_or(tonic::Code::Unknown);
if error.status_code().should_log_error() {
error!(
error; "Failed to receive Flight data, addr: {}, code: {}",
addr,
tonic_code
);
}
error::Error::FlightGet {
addr: addr.to_string(),
tonic_code,
source: BoxedError::new(error),
}
}
pub fn build_remote_dyn_filter_update_request(
query_id: impl Into<String>,
update: RemoteDynFilterUpdate,
@@ -380,6 +412,7 @@ mod test {
RemoteDynFilterUnregister, RemoteDynFilterUpdate, region_request, remote_dyn_filter_request,
};
use common_recordbatch::adapter::RecordBatchMetrics;
use datatypes::arrow::array::Int32Array;
use datatypes::prelude::{ConcreteDataType, VectorRef};
use datatypes::schema::{ColumnSchema, Schema};
use datatypes::vectors::Int32Vector;
@@ -389,6 +422,44 @@ mod test {
use super::*;
use crate::Error::{self, IllegalDatabaseResponse, Server};
#[test]
fn test_flight_stream_error_preserves_peer_address() {
let error = flight_stream_error(
"127.0.0.1:4001",
tonic::Status::unavailable("datanode unavailable").into(),
);
assert!(matches!(
error,
error::Error::FlightGet {
addr,
tonic_code: tonic::Code::Unavailable,
..
} if addr == "127.0.0.1:4001"
));
}
#[tokio::test]
async fn test_empty_flight_stream_preserves_peer_address() {
let Err(error) = recordbatches_from_flight_message_stream(
"127.0.0.1:4001".to_string(),
stream::empty::<Result<FlightMessage>>(),
)
.await
else {
panic!("expected empty Flight stream to fail");
};
assert!(matches!(
error,
error::Error::FlightGet {
addr,
tonic_code: tonic::Code::Unknown,
..
} if addr == "127.0.0.1:4001"
));
}
fn test_schema() -> Arc<Schema> {
Arc::new(Schema::new(vec![ColumnSchema::new(
"v",
@@ -509,11 +580,14 @@ mod test {
)
.unwrap();
let mut recordbatches = recordbatches_from_flight_message_stream(stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::Metrics(test_metrics_json())),
Ok(FlightMessage::RecordBatch(batch.into_df_record_batch())),
]))
let mut recordbatches = recordbatches_from_flight_message_stream(
"test-peer".to_string(),
stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::Metrics(test_metrics_json())),
Ok(FlightMessage::RecordBatch(batch.into_df_record_batch())),
]),
)
.await
.unwrap();
@@ -525,14 +599,97 @@ mod test {
assert_eq!(metrics.elapsed_compute, 7);
}
#[tokio::test]
async fn test_record_batch_stream_updates_following_metrics_before_yielding_batch() {
let schema = test_schema();
let batch = RecordBatch::new(
schema.clone(),
vec![Arc::new(Int32Vector::from_slice([1])) as VectorRef],
)
.unwrap();
let mut recordbatches = recordbatches_from_flight_message_stream(
"test-peer".to_string(),
stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::RecordBatch(batch.into_df_record_batch())),
Ok(FlightMessage::Metrics(test_metrics_json())),
]),
)
.await
.unwrap();
let batch = recordbatches.next().await.unwrap().unwrap();
assert_eq!(batch.num_rows(), 1);
let metrics = recordbatches.metrics().unwrap();
assert_eq!(metrics.elapsed_compute, 7);
assert!(recordbatches.next().await.is_none());
}
#[tokio::test]
async fn test_record_batch_stream_preserves_peeked_record_batch() {
let schema = test_schema();
let first_batch = RecordBatch::new(
schema.clone(),
vec![Arc::new(Int32Vector::from_slice([1])) as VectorRef],
)
.unwrap();
let second_batch = RecordBatch::new(
schema.clone(),
vec![Arc::new(Int32Vector::from_slice([2])) as VectorRef],
)
.unwrap();
let mut recordbatches = recordbatches_from_flight_message_stream(
"test-peer".to_string(),
stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::RecordBatch(
first_batch.into_df_record_batch(),
)),
Ok(FlightMessage::RecordBatch(
second_batch.into_df_record_batch(),
)),
]),
)
.await
.unwrap();
let first_batch = recordbatches.next().await.unwrap().unwrap();
let second_batch = recordbatches.next().await.unwrap().unwrap();
assert_eq!(
first_batch
.column(0)
.as_any()
.downcast_ref::<Int32Array>()
.unwrap()
.value(0),
1
);
assert_eq!(
second_batch
.column(0)
.as_any()
.downcast_ref::<Int32Array>()
.unwrap()
.value(0),
2
);
assert!(recordbatches.next().await.is_none());
}
#[tokio::test]
async fn test_record_batch_stream_exposes_error_after_pre_batch_metrics() {
let schema = test_schema();
let mut recordbatches = recordbatches_from_flight_message_stream(stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::Metrics(test_metrics_json())),
Err(Error::from(Status::internal("boom after metrics"))),
]))
let mut recordbatches = recordbatches_from_flight_message_stream(
"test-peer".to_string(),
stream::iter(vec![
Ok(FlightMessage::Schema(schema.arrow_schema().clone())),
Ok(FlightMessage::Metrics(test_metrics_json())),
Err(Error::from(Status::internal("boom after metrics"))),
]),
)
.await
.unwrap();
src/cmd/Cargo.toml
+9 -1
View File
@@ -14,6 +14,11 @@ name = "query_perf_fixture"
path = "src/bin/query_perf_fixture.rs"
required-features = ["dev-tools"]
[[bin]]
name = "query_regression_runner"
path = "src/bin/query_regression_runner.rs"
required-features = ["dev-tools"]
[features]
default = [
"servers/pprof",
@@ -22,7 +27,7 @@ default = [
"meta-srv/mysql_kvbackend",
]
enterprise = ["common-meta/enterprise", "frontend/enterprise", "meta-srv/enterprise"]
# Developer-only helper binary gate for `query_perf_fixture`.
# Developer-only helper binaries and diagnostic datanode commands.
# Kept out of `default` so normal/release builds don't compile them.
dev-tools = []
mysql-object-store = ["object-store/mysql-object-store"]
@@ -68,6 +73,7 @@ common-wal.workspace = true
datafusion.workspace = true
datafusion-common.workspace = true
datafusion-physical-plan.workspace = true
datafusion_object_store.workspace = true
datanode.workspace = true
datatypes.workspace = true
either = "1.15"
@@ -87,6 +93,7 @@ mito-codec.workspace = true
mito2.workspace = true
moka = { workspace = true, features = ["future"] }
object-store.workspace = true
object_store_opendal.workspace = true
parquet = { workspace = true, features = ["object_store"] }
plugins.workspace = true
prometheus.workspace = true
@@ -111,6 +118,7 @@ tonic.workspace = true
tracing-appender.workspace = true
[target.'cfg(unix)'.dependencies]
daemonize = "0.5"
pprof = { version = "0.14", features = [
"flamegraph",
] }
src/cmd/src/bin/greptime.rs
+57 -7
View File
@@ -17,6 +17,8 @@
use clap::{Parser, Subcommand};
use cmd::datanode::builder::InstanceBuilder;
#[cfg(unix)]
use cmd::error::DaemonizeSnafu;
use cmd::error::{InitTlsProviderSnafu, Result};
use cmd::options::GlobalOptions;
use cmd::{App, cli, datanode, flownode, frontend, metasrv, standalone, user};
@@ -73,20 +75,24 @@ static ALLOC: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
#[cfg(debug_assertions)]
fn main() -> Result<()> {
use snafu::ResultExt;
let cli = Command::parse();
maybe_daemonize(&cli)?;
// Set the stack size to 8MB for the thread so it wouldn't overflow on large stack usage in debug mode
// see https://github.com/GreptimeTeam/greptimedb/pull/4317
// and https://github.com/rust-lang/rust/issues/34283
std::thread::Builder::new()
.name("main_spawn".to_string())
.stack_size(8 * 1024 * 1024)
.spawn(|| {
.spawn(move || {
{
tokio::runtime::Builder::new_multi_thread()
.thread_stack_size(8 * 1024 * 1024)
.enable_all()
.build()
.expect("Failed building the Runtime")
.block_on(main_body())
.block_on(main_body(cli))
}
})
.context(cmd::error::SpawnThreadSnafu)?
@@ -95,15 +101,53 @@ fn main() -> Result<()> {
}
#[cfg(not(debug_assertions))]
#[tokio::main]
async fn main() -> Result<()> {
main_body().await
fn main() -> Result<()> {
let cli = Command::parse();
maybe_daemonize(&cli)?;
tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("Failed building the Runtime")
.block_on(main_body(cli))
}
async fn main_body() -> Result<()> {
async fn main_body(cli: Command) -> Result<()> {
setup_human_panic();
install_default_crypto_provider().map_err(|msg| InitTlsProviderSnafu { msg }.build())?;
start(Command::parse()).await
start(cli).await
}
/// If `--daemon` was passed for `standalone start`, fork into the background
/// before any Tokio runtime is built. On non-Unix platforms the `--daemon`
/// flag does not exist, so this is a no-op.
fn maybe_daemonize(cli: &Command) -> Result<()> {
let daemon = matches!(&cli.subcmd, SubCommand::Standalone(cmd) if cmd.is_daemon());
if !daemon {
return Ok(());
}
#[cfg(unix)]
{
let cwd = std::env::current_dir().map_err(|e| {
DaemonizeSnafu {
msg: format!("failed to read current working directory: {e}"),
}
.build()
})?;
daemonize::Daemonize::new()
.working_directory(cwd)
.start()
.map_err(|e| {
DaemonizeSnafu {
msg: format!("failed to daemonize: {e}"),
}
.build()
})?;
}
Ok(())
}
async fn start(cli: Command) -> Result<()> {
@@ -118,6 +162,12 @@ async fn start(cli: Command) -> Result<()> {
datanode::SubCommand::Objbench(ref bench) => bench.run().await,
datanode::SubCommand::Scanbench(ref bench) => bench.run().await,
datanode::SubCommand::Parquetbench(ref bench) => bench.run().await,
#[cfg(feature = "dev-tools")]
datanode::SubCommand::ParquetMeta(ref cmd) => cmd.run().await,
#[cfg(feature = "dev-tools")]
datanode::SubCommand::ParquetRewrite(ref cmd) => cmd.run().await,
#[cfg(feature = "dev-tools")]
datanode::SubCommand::SstReplace(ref cmd) => cmd.run().await,
},
SubCommand::Flownode(cmd) => {
cmd.build(cmd.load_options(&cli.global_options)?)
src/cmd/src/bin/query_perf_fixture/inspect_footer.rs
+344 -54
View File
@@ -13,17 +13,39 @@
// limitations under the License.
use std::collections::BTreeSet;
use std::fs::{self, File};
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use clap::Args as ClapArgs;
use parquet::file::reader::{FileReader, SerializedFileReader};
use serde::Serialize;
use datafusion_object_store::path::Path as StorePath;
use datafusion_object_store::{ObjectMeta, ObjectStore};
use futures::{StreamExt, TryStreamExt};
use object_store::config::ObjectStoreConfig;
use object_store::factory::new_raw_object_store;
use object_store::services::Fs;
use parquet::arrow::async_reader::ParquetObjectReader;
use parquet::file::metadata::ParquetMetaDataReader;
use serde::{Deserialize, Serialize};
/// Same shape as `query_regression_runner::model::DestinationConfig`. The two
/// binaries are separate crate roots, so the serde-compatible struct is defined
/// here instead of being shared; it reuses the `object-store` crate's
/// `ObjectStoreConfig` deserialization.
#[derive(Debug, Deserialize)]
struct DestinationConfig {
data_home: String,
object_store: ObjectStoreConfig,
}
#[derive(Debug, ClapArgs)]
pub(super) struct InspectFooterArgs {
/// Local data home (convenience shortcut for a File destination).
#[arg(long)]
root: PathBuf,
root: Option<PathBuf>,
/// TOML file: data_home = "..." and object_store = { type = "File" | "S3" | ... }.
#[arg(long)]
destination: Option<PathBuf>,
#[arg(long, default_value = "greptime_value")]
column: String,
#[arg(long)]
@@ -68,64 +90,140 @@ struct FooterColumnChunkReport {
num_values: i64,
}
pub(super) fn run_inspect_footer(
/// A parquet data file discovered by listing the store, with the metadata needed
/// to read its footer without any extra stat/head call.
#[derive(Debug)]
struct ListedFile {
location: StorePath,
size: u64,
path: String,
relative_path: String,
}
pub(super) async fn run_inspect_footer(
args: InspectFooterArgs,
) -> Result<(), Box<dyn std::error::Error>> {
let files = collect_parquet_files(&args.root, args.include_metadata_files)?;
let reports = files
.iter()
.map(|p| inspect_file(&args.root, p, &args.column))
.collect::<Result<Vec<_>, _>>()?;
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let (destination, root) = match (args.destination, args.root) {
(Some(_), Some(_)) => {
return Err("--destination and --root are mutually exclusive; pass exactly one".into());
}
(None, None) => return Err("one of --destination or --root is required".into()),
(Some(path), None) => {
let destination: DestinationConfig = toml::from_str(&fs::read_to_string(path)?)?;
(Some(destination), None)
}
(None, root) => (None, root),
};
let store = build_store(destination.as_ref(), root.as_deref()).await?;
let root_display = root
.as_deref()
.map(|root| root.display().to_string())
.unwrap_or_else(|| {
destination
.as_ref()
.expect("destination is set when root is not")
.data_home
.clone()
});
let files =
collect_parquet_files(store.as_ref(), root.as_deref(), args.include_metadata_files).await?;
let mut reports = futures::stream::iter(files.iter())
.map(|file| inspect_file(Arc::clone(&store), file, &args.column))
.buffer_unordered(8)
.try_collect::<Vec<_>>()
.await?;
reports.sort_by(|a, b| a.relative_path.cmp(&b.relative_path));
println!(
"{}",
serde_json::to_string_pretty(&FooterReport {
root: args.root.display().to_string(),
root: root_display,
summary: summarize_footer(&args.column, &reports),
files: reports
})?
);
Ok(())
}
fn collect_parquet_files(
root: &Path,
/// Builds an arrow `object_store::ObjectStore` for the given destination. The
/// File backend (and the `--root` shortcut) uses the opendal `Fs` builder
/// directly to avoid the create_dir_all / clean_temp_dir side effects of
/// `object_store::factory::new_fs_object_store`; every other backend goes
/// through `object_store::factory::new_raw_object_store`.
async fn build_store(
destination: Option<&DestinationConfig>,
root: Option<&Path>,
) -> Result<Arc<dyn ObjectStore>, Box<dyn std::error::Error + Send + Sync>> {
let operator = match destination {
Some(destination) => match &destination.object_store {
ObjectStoreConfig::File(_) => {
object_store::ObjectStore::new(Fs::default().root(&destination.data_home))?.finish()
}
_ => new_raw_object_store(&destination.object_store, &destination.data_home).await?,
},
None => {
let root = root.expect("root must be set when destination is None");
object_store::ObjectStore::new(Fs::default().root(&root.to_string_lossy()))?.finish()
}
};
Ok(Arc::new(object_store_opendal::OpendalStore::new(operator)))
}
async fn collect_parquet_files(
store: &dyn ObjectStore,
root: Option<&Path>,
include_metadata_files: bool,
) -> Result<Vec<PathBuf>, Box<dyn std::error::Error>> {
) -> Result<Vec<ListedFile>, Box<dyn std::error::Error + Send + Sync>> {
let metas = store.list(None).try_collect::<Vec<ObjectMeta>>().await?;
let mut files = Vec::new();
collect_parquet_files_inner(root, root, include_metadata_files, &mut files)?;
files.sort();
for meta in metas {
if !is_parquet_data_file(&meta) {
continue;
}
let is_metadata = meta
.location
.parts()
.any(|part| part.as_ref() == "metadata");
if !include_metadata_files && is_metadata {
continue;
}
let relative_path = meta.location.to_string();
let path = match root {
Some(root) => root.join(&relative_path).display().to_string(),
None => relative_path.clone(),
};
files.push(ListedFile {
location: meta.location,
size: meta.size,
path,
relative_path,
});
}
files.sort_by(|a, b| a.relative_path.cmp(&b.relative_path));
Ok(files)
}
fn collect_parquet_files_inner(
root: &Path,
path: &Path,
include_metadata_files: bool,
files: &mut Vec<PathBuf>,
) -> Result<(), Box<dyn std::error::Error>> {
if path.is_dir() {
for entry in fs::read_dir(path)? {
collect_parquet_files_inner(root, &entry?.path(), include_metadata_files, files)?;
}
} else if path.extension().is_some_and(|ext| ext == "parquet")
&& (include_metadata_files
|| !path
.strip_prefix(root)
.unwrap_or(path)
.components()
.any(|c| c.as_os_str() == "metadata"))
{
files.push(path.to_path_buf());
}
Ok(())
/// Keeps only real parquet data files: `.parquet` keys with a non-zero size.
/// Zero-size keys (and keys ending in `/`) are directory markers emitted by
/// some backends and are skipped even when they happen to end in `.parquet`,
/// since an empty parquet file has no readable footer anyway.
fn is_parquet_data_file(meta: &ObjectMeta) -> bool {
meta.location.extension() == Some("parquet")
&& meta.size > 0
&& !meta.location.as_ref().ends_with('/')
}
fn inspect_file(
root: &Path,
path: &Path,
async fn inspect_file(
store: Arc<dyn ObjectStore>,
file: &ListedFile,
column: &str,
) -> Result<FooterFileReport, Box<dyn std::error::Error>> {
let file_size = fs::metadata(path)?.len();
let reader = SerializedFileReader::new(File::open(path)?)?;
let metadata = reader.metadata().file_metadata();
let row_groups = reader.metadata().row_groups();
) -> Result<FooterFileReport, Box<dyn std::error::Error + Send + Sync>> {
let mut reader =
ParquetObjectReader::new(store, file.location.clone()).with_file_size(file.size);
let metadata = ParquetMetaDataReader::new()
.load_and_finish(&mut reader, file.size)
.await?;
let file_metadata = metadata.file_metadata();
let row_groups = metadata.row_groups();
let mut columns = Vec::new();
for (row_group_index, rg) in row_groups.iter().enumerate() {
for (column_index, chunk) in rg.columns().iter().enumerate() {
@@ -145,14 +243,10 @@ fn inspect_file(
}
}
Ok(FooterFileReport {
path: path.display().to_string(),
relative_path: path
.strip_prefix(root)
.unwrap_or(path)
.display()
.to_string(),
file_size,
num_rows: metadata.num_rows(),
path: file.path.clone(),
relative_path: file.relative_path.clone(),
file_size: file.size,
num_rows: file_metadata.num_rows(),
num_row_groups: row_groups.len(),
columns,
})
@@ -179,3 +273,199 @@ fn summarize_footer(column: &str, files: &[FooterFileReport]) -> FooterSummary {
s.unique_encodings = encodings.into_iter().collect();
s
}
#[cfg(test)]
mod tests {
use std::fs::{self, File};
use parquet::column::writer::ColumnWriter;
use parquet::file::properties::WriterProperties;
use parquet::file::writer::SerializedFileWriter;
use parquet::schema::parser::parse_message_type;
use tempfile::TempDir;
use super::*;
/// Writes a minimal valid parquet file with a single `value` Int64 column
/// and `rows` values, so the footer can be read back through the object
/// store path.
fn write_parquet_file(path: &Path, rows: i64) {
let schema = Arc::new(
parse_message_type("message schema { REQUIRED INT64 value; }").expect("valid schema"),
);
let props = Arc::new(WriterProperties::builder().build());
let file = File::create(path).expect("create parquet file");
let mut writer = SerializedFileWriter::new(file, schema, props).expect("init writer");
let mut row_group = writer.next_row_group().expect("next row group");
let mut column = row_group
.next_column()
.expect("next column")
.expect("has column");
match column.untyped() {
ColumnWriter::Int64ColumnWriter(typed) => {
typed
.write_batch(&(1..=rows).collect::<Vec<_>>(), None, None)
.expect("write batch");
}
_ => panic!("unexpected column writer"),
}
column.close().expect("close column");
row_group.close().expect("close row group");
writer.close().expect("close writer");
}
fn fixture_tree(dir: &Path) {
// Nested data tree with a `metadata` subdirectory, a non-parquet file
// and a zero-size fake file.
let table = dir.join("data/table");
fs::create_dir_all(&table).expect("create table dir");
fs::create_dir_all(table.join("metadata")).expect("create metadata dir");
write_parquet_file(&table.join("0001.parquet"), 3);
write_parquet_file(&table.join("0002.parquet"), 5);
write_parquet_file(&table.join("metadata/0001.parquet"), 7);
fs::write(table.join("notes.txt"), b"not parquet").expect("write notes");
File::create(table.join("empty.parquet")).expect("create empty file");
}
/// Runs the full list -> filter -> footer-read path against a File backend
/// rooted at `dir` and returns the collected file reports.
async fn inspect_fixture(dir: &Path, include_metadata_files: bool) -> Vec<FooterFileReport> {
let root = Some(dir.to_path_buf());
let store = build_store(None, root.as_deref())
.await
.expect("build store");
let files = collect_parquet_files(store.as_ref(), root.as_deref(), include_metadata_files)
.await
.expect("collect files");
let mut reports = futures::stream::iter(files.iter())
.map(|file| inspect_file(Arc::clone(&store), file, "value"))
.buffer_unordered(8)
.try_collect::<Vec<_>>()
.await
.expect("inspect files");
reports.sort_by(|a, b| a.relative_path.cmp(&b.relative_path));
reports
}
#[tokio::test]
async fn lists_and_inspects_footer_files_without_metadata_by_default() {
let dir = TempDir::new().expect("tempdir");
fixture_tree(dir.path());
let reports = inspect_fixture(dir.path(), false).await;
let relative_paths = reports
.iter()
.map(|report| report.relative_path.as_str())
.collect::<Vec<_>>();
// metadata/0001.parquet excluded, notes.txt and empty.parquet excluded.
assert_eq!(
relative_paths,
vec!["data/table/0001.parquet", "data/table/0002.parquet"]
);
assert!(reports.iter().all(|report| report.file_size > 0));
assert!(reports.iter().all(|report| report.num_row_groups == 1));
assert!(reports.iter().all(|report| report.num_rows > 0));
assert!(
reports
.iter()
.all(|report| report.columns.len() == 1 && report.columns[0].column_path == "value")
);
// Default writer properties use dictionary encoding, so the column chunk
// reports PLAIN (dictionary page) plus RLE_DICTIONARY (data pages).
assert!(
reports
.iter()
.all(|report| report.columns[0].encodings.contains(&"PLAIN".to_string()))
);
assert!(
reports
.iter()
.all(|report| report.columns[0].compression == "UNCOMPRESSED")
);
}
#[tokio::test]
async fn includes_metadata_files_when_requested() {
let dir = TempDir::new().expect("tempdir");
fixture_tree(dir.path());
let reports = inspect_fixture(dir.path(), true).await;
let relative_paths = reports
.iter()
.map(|report| report.relative_path.as_str())
.collect::<Vec<_>>();
assert_eq!(
relative_paths,
vec![
"data/table/0001.parquet",
"data/table/0002.parquet",
"data/table/metadata/0001.parquet"
]
);
let metadata_report = reports
.iter()
.find(|report| report.relative_path == "data/table/metadata/0001.parquet")
.expect("metadata report");
assert_eq!(metadata_report.num_rows, 7);
}
#[tokio::test]
async fn accepts_a_file_destination_toml() {
let dir = TempDir::new().expect("tempdir");
fixture_tree(dir.path());
let destination = dir.path().join("destination.toml");
fs::write(
&destination,
format!(
"data_home = \"{}\"\n[object_store]\ntype = \"File\"\n",
dir.path().display()
),
)
.expect("write destination toml");
let destination: DestinationConfig =
toml::from_str(&fs::read_to_string(&destination).expect("read toml"))
.expect("parse toml");
let store = build_store(Some(&destination), None)
.await
.expect("build store");
let files = collect_parquet_files(store.as_ref(), None, false)
.await
.expect("collect files");
// Destination mode has no root prefix: paths are store-relative keys.
assert_eq!(
files
.iter()
.map(|file| file.relative_path.as_str())
.collect::<Vec<_>>(),
vec!["data/table/0001.parquet", "data/table/0002.parquet"]
);
assert!(files.iter().all(|file| file.path == file.relative_path));
}
#[test]
fn requires_exactly_one_of_root_or_destination() {
use futures::FutureExt;
let both = run_inspect_footer(InspectFooterArgs {
root: Some(PathBuf::from("/tmp/root")),
destination: Some(PathBuf::from("/tmp/destination.toml")),
column: "value".to_string(),
include_metadata_files: false,
})
.now_or_never();
assert!(matches!(both, Some(Err(_))));
let neither = run_inspect_footer(InspectFooterArgs {
root: None,
destination: None,
column: "value".to_string(),
include_metadata_files: false,
})
.now_or_never();
assert!(matches!(neither, Some(Err(_))));
}
}
src/cmd/src/bin/query_perf_fixture/mod.rs
+3 -3
View File
@@ -95,9 +95,9 @@ pub async fn run() {
Some(Command::PromRemoteWrite(rw)) => run_prom_remote_write(rw)
.await
.expect("prom remote write failed"),
Some(Command::InspectFooter(inspect)) => {
run_inspect_footer(inspect).expect("inspect footer failed")
}
Some(Command::InspectFooter(inspect)) => run_inspect_footer(inspect)
.await
.expect("inspect footer failed"),
Some(Command::Plan(plan)) => run_plan(plan).expect("plan failed"),
None => run_direct_sst(args.legacy).await,
}
src/cmd/src/bin/query_regression_runner.rs
+23
View File
@@ -0,0 +1,23 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![allow(clippy::print_stderr, clippy::print_stdout)]
#[path = "query_regression_runner/mod.rs"]
mod query_regression_runner;
#[tokio::main]
async fn main() {
query_regression_runner::run().await;
}
src/cmd/src/bin/query_regression_runner/direct.rs
+603
View File
@@ -0,0 +1,603 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::{Duration, Instant};
use reqwest::Client;
use serde::Serialize;
use serde_json::{Value, json};
use crate::query_regression_runner::model::{Scenario, Table};
use crate::query_regression_runner::plan::{load_plan, validate_direct_tables};
use crate::query_regression_runner::sql::{
extract_rows, http_post_sql, row_u64, row_value, sql_ident, sql_string, value_text,
};
use crate::query_regression_runner::{PrepareDirectArgs, Result};
#[derive(Clone, Debug, Serialize)]
struct Discovery {
table: String,
catalog: String,
schema: String,
table_id: u64,
region_id: u64,
region_seq: u64,
table_dir: String,
region_dir: String,
peer_id: u64,
peer_addr: Value,
is_leader: Value,
status: Value,
discovery_queries: Value,
}
#[derive(Debug)]
struct PreparedTarget {
creates: Vec<Value>,
discoveries: Vec<Discovery>,
}
pub(super) async fn run_prepare_direct(args: PrepareDirectArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let case_path = args.case.canonicalize()?;
let plan = load_plan(&args.fixture_generator, &case_path)?;
let scenario_value = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
let scenario: Scenario = serde_json::from_value(scenario_value)?;
let tables = direct_tables(scenario)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let base = prepare_target("base", args.base_http_port, &tables, &client).await?;
let candidate = prepare_target("candidate", args.candidate_http_port, &tables, &client).await?;
if base.discoveries.len() != candidate.discoveries.len()
|| base
.discoveries
.iter()
.zip(&candidate.discoveries)
.any(|(base, candidate)| {
base.table != candidate.table
|| base.region_id != candidate.region_id
|| base.table_dir != candidate.table_dir
})
{
return Err(format!(
"base/candidate metadata mismatch: base={:?}, candidate={:?}",
base.discoveries, candidate.discoveries
)
.into());
}
let mut fixtures = Vec::with_capacity(tables.len());
for (index, (table, discovery)) in tables.iter().zip(&base.discoveries).enumerate() {
let fixture_dir = table_fixture_dir(&args.fixture_dir, &tables, table, index);
fixtures.push(generate_direct_fixture(
&args.fixture_generator,
&case_path,
&fixture_dir,
table,
discovery,
tables.len() > 1,
args.allow_large_fixture,
)?);
}
let report = json!({
"case_path": case_path,
"scenario": "direct_readable_sst",
"base": { "create_table": base.creates, "discovery": base.discoveries },
"candidate": { "create_table": candidate.creates, "discovery": candidate.discoveries },
"fixtures": fixtures,
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
if let Some(path) = args.output {
fs::write(path, &text)?;
}
print!("{text}");
Ok(())
}
fn direct_tables(scenario: Scenario) -> Result<Vec<Table>> {
match scenario {
Scenario::DirectReadableSst { tables, layout, .. } => {
validate_direct_tables(tables, layout)
}
Scenario::PromRemoteWriteThenQuery { .. } => {
Err("prepare-direct requires scenario kind direct_readable_sst".into())
}
Scenario::OtlpTraceLoad { .. } => {
Err("prepare-direct requires scenario kind direct_readable_sst".into())
}
}
}
async fn prepare_target(
name: &str,
port: u16,
tables: &[Table],
client: &Client,
) -> Result<PreparedTarget> {
let mut creates = Vec::with_capacity(tables.len());
let mut discoveries = Vec::with_capacity(tables.len());
for table in tables {
let sql = create_table_sql(table)?;
let result = http_post_sql(client, port, &sql, &table.database).await;
if !result["ok"].as_bool().unwrap_or(false) {
return Err(format!("CREATE TABLE {} failed for {name}: {result}", table.name).into());
}
creates.push(json!({ "table": table.name, "sql": sql, "result": result }));
discoveries.push(discover_region(client, port, table).await?);
}
Ok(PreparedTarget {
creates,
discoveries,
})
}
fn create_table_sql(table: &Table) -> Result<String> {
if table.engine != "mito" {
return Err("prepare-direct requires table engine mito".into());
}
let time_index = table
.time_index
.as_deref()
.ok_or("direct-SST table requires time_index")?;
if table.columns.is_empty() {
return Err("direct-SST table requires columns".into());
}
let columns = table
.columns
.iter()
.map(|column| {
// The direct-SST fixture generator (`query_perf_fixture::direct_sst::
// build_region_metadata`) bakes index metadata into the region
// manifest's column schemas: tag columns get an inverted index and
// a skipping index with granularity 1, field columns get a skipping
// index with granularity 1. The CREATE TABLE must declare the same
// column options, otherwise the table schema (frontend) and the
// region schema (datanode, from the manifest) disagree and the
// MergeScan remote schema validation fails with "advertised remote
// stream schema field mismatch".
let mut sql = format!("{} {}", sql_ident(&column.name), column.data_type);
match column.semantic.as_deref() {
Some("tag") => {
sql.push_str(" SKIPPING INDEX WITH (granularity='1') INVERTED INDEX");
}
Some("field") => {
sql.push_str(" SKIPPING INDEX WITH (granularity='1')");
}
_ => {}
}
sql
})
.collect::<Vec<_>>()
.join(",\n ");
let primary_key = table
.primary_key
.iter()
.map(|column| sql_ident(column))
.collect::<Vec<_>>()
.join(", ");
let mut options = Vec::new();
if let Some(append_mode) = table.append_mode {
options.push(("append_mode", append_mode.to_string()));
}
if let Some(sst_format) = table
.sst_format
.as_deref()
.filter(|value| !value.is_empty())
{
options.push(("sst_format", sst_format.to_string()));
}
let with = (!options.is_empty()).then(|| {
format!(
"\nWITH ({})",
options
.iter()
.map(|(key, value)| format!("'{key}'='{value}'"))
.collect::<Vec<_>>()
.join(", ")
)
});
Ok(format!(
"CREATE TABLE {} (\n {columns},\n TIME INDEX ({}),\n PRIMARY KEY ({primary_key})\n) ENGINE=mito{};",
sql_ident(&table.name),
sql_ident(time_index),
with.unwrap_or_default()
))
}
async fn discover_region(client: &Client, port: u16, table: &Table) -> Result<Discovery> {
let schema = sql_string(&table.database);
let table_name = sql_string(&table.name);
let table_sql = format!(
"SELECT table_id FROM information_schema.tables WHERE table_schema = {schema} AND table_name = {table_name}"
);
let table_result = http_post_sql(client, port, &table_sql, &table.database).await;
if !table_result["ok"].as_bool().unwrap_or(false) {
return Err(format!("table_id discovery failed: {table_result}").into());
}
let table_rows = extract_rows(table_result.get("response").unwrap_or(&Value::Null));
if table_rows.len() != 1 {
return Err(format!(
"expected one information_schema.tables row, got {}: {table_result}",
table_rows.len()
)
.into());
}
let table_id = row_u64(&table_rows[0], 0, "table_id")?;
let region_sql = format!(
"SELECT region_id, peer_id, peer_addr, is_leader, status FROM information_schema.region_peers WHERE table_schema = {schema} AND table_name = {table_name}"
);
let region_result = http_post_sql(client, port, &region_sql, &table.database).await;
if !region_result["ok"].as_bool().unwrap_or(false) {
return Err(format!("region_peers discovery failed: {region_result}").into());
}
let region_rows = extract_rows(region_result.get("response").unwrap_or(&Value::Null));
if region_rows.len() != 1 {
return Err(format!(
"expected one information_schema.region_peers row, got {}: {region_result}",
region_rows.len()
)
.into());
}
let row = &region_rows[0];
let region_id = row_u64(row, 0, "region_id")?;
let peer_id = row_u64(row, 1, "peer_id")?;
let peer_addr = row_value(row, 2, "peer_addr")
.cloned()
.unwrap_or(Value::Null);
let is_leader = row_value(row, 3, "is_leader")
.cloned()
.unwrap_or(Value::Null);
let status = row_value(row, 4, "status").cloned().unwrap_or(Value::Null);
if !matches!(
value_text(&is_leader).to_ascii_lowercase().as_str(),
"yes" | "true"
) {
return Err(format!(
"expected leader region peer, got is_leader={is_leader}: {region_result}"
)
.into());
}
if !value_text(&status).eq_ignore_ascii_case("ALIVE") {
return Err(
format!("expected ALIVE region peer, got status={status}: {region_result}").into(),
);
}
if peer_id != 0 {
return Err(format!("expected region leader on datanode peer_id=0, got {peer_id}").into());
}
if region_id >> 32 != table_id {
return Err(format!(
"table_id mismatch: tables={table_id}, region_id-derived={}",
region_id >> 32
)
.into());
}
let region_seq = region_id & u32::MAX as u64;
let (table_dir, region_dir) = storage_paths(&table.database, table_id, region_seq);
Ok(Discovery {
table: table.name.clone(),
catalog: "greptime".to_string(),
schema: table.database.clone(),
table_id,
region_id,
region_seq,
table_dir: table_dir.clone(),
region_dir,
peer_id,
peer_addr,
is_leader,
status,
discovery_queries: json!({ "table": table_result, "region": region_result }),
})
}
fn storage_paths(database: &str, table_id: u64, region_seq: u64) -> (String, String) {
let table_dir = format!("data/greptime/{database}/{table_id}/");
let region_dir = format!("{table_dir}{table_id}_{region_seq:010}");
(table_dir, region_dir)
}
fn table_fixture_dir(root: &Path, tables: &[Table], table: &Table, index: usize) -> PathBuf {
if tables.len() == 1 {
root.to_path_buf()
} else {
root.join(fixture_subdir(table, index))
}
}
fn fixture_subdir(table: &Table, index: usize) -> String {
let raw = format!("{index:02}_{}_{}", table.database, table.name);
let mut safe = String::new();
let mut replaced = false;
for character in raw.chars() {
if character.is_ascii_alphanumeric() || matches!(character, '_' | '.' | '-') {
safe.push(character);
replaced = false;
} else if !replaced {
safe.push('_');
replaced = true;
}
}
let safe = safe.trim_matches(['.', '_', '-']);
if safe.is_empty() {
format!("table_{index:02}")
} else {
safe.to_string()
}
}
fn generate_direct_fixture(
generator: &Path,
case_path: &Path,
fixture_dir: &Path,
table: &Table,
discovery: &Discovery,
multi_table: bool,
allow_large_fixture: bool,
) -> Result<Value> {
if fixture_dir.exists() {
fs::remove_dir_all(fixture_dir)?;
}
fs::create_dir_all(fixture_dir)?;
let mut command = vec![
generator.to_string_lossy().to_string(),
"direct-sst".to_string(),
"--case".to_string(),
case_path.to_string_lossy().to_string(),
"--out-dir".to_string(),
fixture_dir.to_string_lossy().to_string(),
];
if multi_table {
command.extend(["--table".to_string(), table.name.clone()]);
}
command.extend([
"--region-id".to_string(),
discovery.region_id.to_string(),
"--table-dir".to_string(),
discovery.table_dir.clone(),
]);
if allow_large_fixture {
command.push("--allow-large".to_string());
}
let started = Instant::now();
let output = Command::new(generator).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
if !output.status.success() {
return Err(format!(
"fixture generator failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}",
output.status.code(), stderr
)
.into());
}
let summary: Value = serde_json::from_slice(&fs::read(fixture_dir.join("summary.json"))?)?;
assert_fixture_summary(&summary, table, discovery)?;
Ok(json!({
"status": "ok",
"fixture_dir": fixture_dir,
"command": command,
"returncode": output.status.code(),
"elapsed_seconds": elapsed_seconds,
"stdout": stdout,
"stderr": stderr,
"summary": summary,
}))
}
fn assert_fixture_summary(summary: &Value, table: &Table, discovery: &Discovery) -> Result<()> {
if summary.get("table").and_then(Value::as_str) != Some(&table.name) {
return Err(format!(
"fixture table mismatch: {:?} != {}",
summary.get("table"),
table.name
)
.into());
}
if summary.get("database").and_then(Value::as_str) != Some(&table.database) {
return Err(format!(
"fixture database mismatch: {:?} != {}",
summary.get("database"),
table.database
)
.into());
}
if summary.get("region_id").and_then(|value| {
value
.as_u64()
.or_else(|| value.as_str().and_then(|value| value.parse().ok()))
}) != Some(discovery.region_id)
{
return Err(format!(
"fixture region_id mismatch: {:?} != {}",
summary.get("region_id"),
discovery.region_id
)
.into());
}
if summary.get("table_dir").and_then(Value::as_str) != Some(&discovery.table_dir) {
return Err(format!(
"fixture table_dir mismatch: {:?} != {}",
summary.get("table_dir"),
discovery.table_dir
)
.into());
}
if summary
.get("region_dir")
.and_then(Value::as_str)
.map(|value| value.trim_matches('/'))
!= Some(discovery.region_dir.trim_matches('/'))
{
return Err(format!(
"fixture region_dir mismatch: {:?} != {}",
summary.get("region_dir"),
discovery.region_dir
)
.into());
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query_regression_runner::model::Column;
#[test]
fn creates_exact_direct_sst_table_sql() {
let table = Table {
database: "public".to_string(),
name: "metric\"name".to_string(),
engine: "mito".to_string(),
columns: vec![
Column {
name: "host".to_string(),
data_type: "STRING".to_string(),
semantic: Some("tag".to_string()),
},
Column {
name: "ts".to_string(),
data_type: "TIMESTAMP(9)".to_string(),
semantic: Some("timestamp".to_string()),
},
],
primary_key: vec!["host".to_string()],
time_index: Some("ts".to_string()),
append_mode: Some(true),
sst_format: Some("flat".to_string()),
validate_show_create_engine: true,
};
assert_eq!(
create_table_sql(&table).unwrap(),
"CREATE TABLE \"metric\"\"name\" (\n \"host\" STRING SKIPPING INDEX WITH (granularity='1') INVERTED INDEX,\n \"ts\" TIMESTAMP(9),\n TIME INDEX (\"ts\"),\n PRIMARY KEY (\"host\")\n) ENGINE=mito\nWITH ('append_mode'='true', 'sst_format'='flat');"
);
}
#[test]
fn create_table_sql_declares_indexes_matching_fixture_metadata() {
// Mirrors the index metadata baked by
// query_perf_fixture::direct_sst::build_region_metadata: tag columns
// get SKIPPING INDEX WITH (granularity='1') + INVERTED INDEX, field
// columns get SKIPPING INDEX WITH (granularity='1'), timestamp columns
// get no column options. Without these options the frontend table
// schema and the datanode region schema (from the fixture manifest)
// disagree and the MergeScan remote schema validation fails.
let table = Table {
database: "public".to_string(),
name: "metric".to_string(),
engine: "mito".to_string(),
columns: vec![
Column {
name: "host".to_string(),
data_type: "STRING".to_string(),
semantic: Some("tag".to_string()),
},
Column {
name: "instance".to_string(),
data_type: "STRING".to_string(),
semantic: Some("tag".to_string()),
},
Column {
name: "value".to_string(),
data_type: "DOUBLE".to_string(),
semantic: Some("field".to_string()),
},
Column {
name: "ts".to_string(),
data_type: "TIMESTAMP(9)".to_string(),
semantic: Some("timestamp".to_string()),
},
],
primary_key: vec!["host".to_string(), "instance".to_string()],
time_index: Some("ts".to_string()),
append_mode: Some(true),
sst_format: Some("flat".to_string()),
validate_show_create_engine: true,
};
assert_eq!(
create_table_sql(&table).unwrap(),
"CREATE TABLE \"metric\" (\n \"host\" STRING SKIPPING INDEX WITH (granularity='1') INVERTED INDEX,\n \"instance\" STRING SKIPPING INDEX WITH (granularity='1') INVERTED INDEX,\n \"value\" DOUBLE SKIPPING INDEX WITH (granularity='1'),\n \"ts\" TIMESTAMP(9),\n TIME INDEX (\"ts\"),\n PRIMARY KEY (\"host\", \"instance\")\n) ENGINE=mito\nWITH ('append_mode'='true', 'sst_format'='flat');"
);
}
#[test]
fn create_table_sql_without_semantic_keeps_bare_columns() {
// Columns without a semantic type must keep the historical bare form so
// tables that predate the index-metadata fix are unaffected.
let table = Table {
database: "public".to_string(),
name: "metric".to_string(),
engine: "mito".to_string(),
columns: vec![
Column {
name: "host".to_string(),
data_type: "STRING".to_string(),
semantic: None,
},
Column {
name: "ts".to_string(),
data_type: "TIMESTAMP(9)".to_string(),
semantic: None,
},
],
primary_key: vec!["host".to_string()],
time_index: Some("ts".to_string()),
append_mode: None,
sst_format: None,
validate_show_create_engine: true,
};
assert_eq!(
create_table_sql(&table).unwrap(),
"CREATE TABLE \"metric\" (\n \"host\" STRING,\n \"ts\" TIMESTAMP(9),\n TIME INDEX (\"ts\"),\n PRIMARY KEY (\"host\")\n) ENGINE=mito;"
);
}
#[test]
fn extracts_named_and_positional_discovery_rows() {
let rows = extract_rows(&json!({"output": [{"data": [{"TABLE_ID": "7"}]}]}));
assert_eq!(rows, vec![json!({"TABLE_ID": "7"})]);
assert_eq!(row_u64(&rows[0], 0, "table_id").unwrap(), 7);
let rows = extract_rows(&json!({"data": [[30064771073_u64, 0, "addr", "YES", "ALIVE"]]}));
assert_eq!(rows.len(), 1);
assert_eq!(row_u64(&rows[0], 0, "region_id").unwrap(), 30_064_771_073);
assert_eq!(row_u64(&rows[0], 1, "peer_id").unwrap(), 0);
assert_eq!(
value_text(row_value(&rows[0], 3, "is_leader").unwrap()),
"YES"
);
}
#[test]
fn storage_paths_include_the_table_directory() {
assert_eq!(
storage_paths("public", 1024, 0),
(
"data/greptime/public/1024/".to_string(),
"data/greptime/public/1024/1024_0000000000".to_string(),
)
);
}
}
src/cmd/src/bin/query_regression_runner/materialize.rs
+225
View File
@@ -0,0 +1,225 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::Path;
use futures::{AsyncWriteExt as _, TryStreamExt};
use object_store::ObjectStore;
use object_store::factory::new_raw_object_store;
use object_store::services::Fs;
use crate::query_regression_runner::model::{
CopyCounts, DestinationConfig, FixtureSummary, MaterializeResult,
};
use crate::query_regression_runner::{MaterializeArgs, Result};
pub(super) async fn run_materialize(args: MaterializeArgs) -> Result<()> {
let destination: DestinationConfig = toml::from_str(&fs::read_to_string(args.destination)?)?;
let result = materialize(&args.fixture_dir, destination).await?;
println!("{}", serde_json::to_string(&result)?);
Ok(())
}
async fn materialize(
fixture_dir: &Path,
destination: DestinationConfig,
) -> Result<MaterializeResult> {
let summary: FixtureSummary =
serde_json::from_slice(&fs::read(fixture_dir.join("summary.json"))?)?;
let region_dir = validate_region_dir(&summary.region_dir)?;
let object_source = fs_operator(&fixture_dir.join("object-store"))?;
let manifest_source = fs_operator(&fixture_dir.join("manifest"))?;
let destination =
new_raw_object_store(&destination.object_store, &destination.data_home).await?;
let region_prefix = format!("{region_dir}/");
destination
.delete_with(&region_prefix)
.recursive(true)
.await?;
let object_store = copy_tree(&object_source, &destination, "/", "").await?;
let manifest = copy_tree(
&manifest_source,
&destination,
"/",
&format!("{region_dir}/manifest/"),
)
.await?;
Ok(MaterializeResult {
region_dir,
object_store,
manifest,
})
}
fn fs_operator(root: &Path) -> Result<ObjectStore> {
Ok(ObjectStore::new(Fs::default().root(&root.to_string_lossy()))?.finish())
}
fn validate_region_dir(region_dir: &str) -> Result<String> {
let region_dir = region_dir.trim_end_matches('/');
if region_dir.is_empty()
|| region_dir.starts_with('/')
|| region_dir.contains('\\')
|| region_dir
.split('/')
.any(|component| component.is_empty() || matches!(component, "." | ".."))
{
return Err(
"region_dir must be a non-empty relative OpenDAL key without dot components".into(),
);
}
Ok(region_dir.to_string())
}
async fn copy_tree(
source: &ObjectStore,
destination: &ObjectStore,
source_prefix: &str,
destination_prefix: &str,
) -> Result<CopyCounts> {
let mut lister = source.lister_with(source_prefix).recursive(true).await?;
let mut counts = CopyCounts::default();
while let Some(entry) = lister.try_next().await? {
if entry.metadata().is_dir() {
continue;
}
let source_path = entry.path().to_string();
let reader = source
.reader(&source_path)
.await?
.into_futures_async_read(0..entry.metadata().content_length())
.await?;
let mut writer = destination
.writer(&format!("{destination_prefix}{source_path}"))
.await?
.into_futures_async_write();
let bytes = futures::io::copy(reader, &mut writer).await?;
writer.close().await?;
counts.files += 1;
counts.bytes += bytes;
}
Ok(counts)
}
#[cfg(test)]
mod tests {
use std::fs;
use object_store::config::ObjectStoreConfig;
use super::*;
#[test]
fn region_dir_must_be_a_relative_opendal_key() {
assert_eq!(
validate_region_dir("data/db/region/").unwrap(),
"data/db/region"
);
for region_dir in [
"",
"/region",
".",
"a/../region",
"a/./region",
"a\\region",
"a//region",
] {
assert!(validate_region_dir(region_dir).is_err(), "{region_dir}");
}
}
#[tokio::test]
async fn materializes_fixture_from_fs_to_fs() {
let fixture = tempfile::tempdir().unwrap();
let destination = tempfile::tempdir().unwrap();
let region_dir = "data/public/metrics/00000000000000000001";
let sst = fixture
.path()
.join("object-store")
.join(region_dir)
.join("00000000000000000001.parquet");
fs::create_dir_all(sst.parent().unwrap()).unwrap();
fs::write(&sst, b"sst").unwrap();
let checkpoint = fixture
.path()
.join("manifest/00000000000000000001.checkpoint");
fs::create_dir_all(checkpoint.parent().unwrap()).unwrap();
fs::write(&checkpoint, b"checkpoint").unwrap();
fs::write(fixture.path().join("manifest/_last_checkpoint"), b"last").unwrap();
fs::write(
fixture.path().join("summary.json"),
format!(r#"{{"region_dir":"{region_dir}"}}"#),
)
.unwrap();
fs::write(fixture.path().join("files.jsonl"), "metadata").unwrap();
fs::create_dir_all(destination.path().join(region_dir)).unwrap();
fs::write(destination.path().join(region_dir).join("stale"), "stale").unwrap();
fs::write(destination.path().join("unrelated"), "keep").unwrap();
let result = materialize(
fixture.path(),
DestinationConfig {
data_home: destination.path().to_string_lossy().to_string(),
object_store: ObjectStoreConfig::default(),
},
)
.await
.unwrap();
assert_eq!(result.object_store.files, 1);
assert_eq!(result.object_store.bytes, 3);
assert_eq!(result.manifest.files, 2);
assert_eq!(result.manifest.bytes, 14);
assert_eq!(
fs::read(
destination
.path()
.join(region_dir)
.join("00000000000000000001.parquet")
)
.unwrap(),
b"sst"
);
assert_eq!(
fs::read(
destination
.path()
.join(region_dir)
.join("manifest/00000000000000000001.checkpoint")
)
.unwrap(),
b"checkpoint"
);
assert_eq!(
fs::read(
destination
.path()
.join(region_dir)
.join("manifest/_last_checkpoint")
)
.unwrap(),
b"last"
);
assert!(!destination.path().join(region_dir).join("stale").exists());
assert_eq!(
fs::read(destination.path().join("unrelated")).unwrap(),
b"keep"
);
assert!(!destination.path().join("summary.json").exists());
assert!(!destination.path().join("files.jsonl").exists());
}
}
src/cmd/src/bin/query_regression_runner/measure.rs
+477
View File
@@ -0,0 +1,477 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
use std::time::Duration;
use std::{fs, io};
use reqwest::Client;
use serde_json::{Map, Value, json};
use crate::query_regression_runner::model::{Measurement, Query, QueryResult, Scenario, Table};
use crate::query_regression_runner::plan::{load_plan, normalize_scenario};
use crate::query_regression_runner::sql::{http_post_prom_range_query, http_post_sql, sql_ident};
use crate::query_regression_runner::{MeasureArgs, Result};
pub(super) async fn run_measure(args: MeasureArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let case_path = args.case.canonicalize()?;
let plan = load_plan(&args.fixture_generator, &case_path)?;
let case_text = fs::read_to_string(&case_path)?;
let raw_case: toml::Value = toml::from_str(&case_text)?;
let case_metadata = raw_case
.get("case")
.cloned()
.unwrap_or(toml::Value::Table(toml::map::Map::new()));
let case_metadata = serde_json::to_value(case_metadata)?;
let scenario_value = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
let scenario: Scenario = serde_json::from_value(scenario_value.clone())?;
let (tables, configured_queries) = normalize_scenario(scenario)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let base = run_target(args.base_http_port, &tables, &configured_queries, &client).await;
let candidate = run_target(
args.candidate_http_port,
&tables,
&configured_queries,
&client,
)
.await;
let thresholds = enforce_thresholds(&configured_queries, &base, &candidate)?;
let status = if base.status == "failed"
|| candidate.status == "failed"
|| thresholds
.iter()
.any(|threshold| threshold["status"] == "failed")
{
"failed"
} else {
"ok"
};
let report = json!({
"case_path": case_path,
"case": case_metadata,
"scenario": scenario_value,
"queries": configured_queries,
"query_mode": "endpoint",
"http_timeout": args.http_timeout,
"targets": [target_report("base", args.base_http_port, base), target_report("candidate", args.candidate_http_port, candidate)],
"thresholds": thresholds,
"status": status,
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
if let Some(path) = args.output {
fs::write(path, &text)?;
}
print!("{text}");
if status == "failed" {
std::process::exit(1);
}
Ok(())
}
fn target_report(name: &str, http_port: u16, result: QueryResult) -> Value {
let status = if result.status == "ok" {
"measured"
} else {
"failed"
};
json!({
"name": name,
"http_port": http_port,
"validation": result.validation,
"validation_errors": result.validation_errors,
"measurements": result.measurements,
"status": status,
})
}
/// Routes a configured query to the right endpoint: `prom_http` queries hit
/// the Prometheus HTTP range API (which exercises the Prometheus JSON response
/// builder), everything else goes through `/v1/sql` (including `TQL ANALYZE`).
async fn post_query(client: &Client, port: u16, query: &Query, db: &str) -> Value {
if query.kind.as_deref() == Some("prom_http") {
http_post_prom_range_query(
client,
port,
&query.query,
query.start.as_deref(),
query.end.as_deref(),
query.step.as_deref(),
db,
)
.await
} else {
http_post_sql(client, port, &query.query, db).await
}
}
async fn run_target(
port: u16,
tables: &[Table],
configured_queries: &[Query],
client: &Client,
) -> QueryResult {
let mut queries = configured_queries.to_vec();
if queries.is_empty() {
queries.push(Query {
name: Some("count_all".to_string()),
kind: Some("sql".to_string()),
query: format!("SELECT count(*) FROM {}", sql_ident(&tables[0].name)),
start: None,
end: None,
step: None,
warmup: 0,
iterations: 1,
thresholds: Map::new(),
});
}
let db = &tables[0].database;
let mut validation = Vec::new();
let mut validation_errors = Vec::new();
for table in tables {
let sql = format!("SHOW CREATE TABLE {}", sql_ident(&table.name));
let sample = http_post_sql(client, port, &sql, &table.database).await;
if !sample["ok"].as_bool().unwrap_or(false) {
validation_errors.push(json!({
"sql": sql,
"error": sample.get("error"),
"response": sample.get("response"),
}));
} else {
for error in validate_show_create(&sample, table) {
validation_errors.push(json!({
"sql": sql,
"error": error,
"response": sample.get("response"),
}));
}
}
validation.push(sample);
}
let first = post_query(client, port, &queries[0], db).await;
if !first["ok"].as_bool().unwrap_or(false) {
validation_errors.push(json!({
"sql": queries[0].query,
"error": first.get("error"),
"response": first.get("response"),
}));
}
validation.push(first);
let mut measurements = Vec::with_capacity(queries.len());
for query in &queries {
for _ in 0..query.warmup {
let warmup = post_query(client, port, query, db).await;
if !warmup["ok"].as_bool().unwrap_or(false) {
validation_errors.push(json!({
"sql": query.query,
"phase": "warmup",
"error": warmup.get("error"),
"response": warmup.get("response"),
}));
}
}
let mut samples = Vec::with_capacity(query.iterations);
let mut good_latencies = Vec::with_capacity(query.iterations);
for _ in 0..query.iterations {
let mut sample = post_query(client, port, query, db).await;
let execution_time = sample
.get("response")
.and_then(|response| {
extract_execution_time_for_kind(query.kind.as_deref(), response)
})
.cloned()
.unwrap_or(Value::Null);
sample
.as_object_mut()
.expect("HTTP samples are objects")
.insert("execution_time_ms".to_string(), execution_time);
if sample["ok"].as_bool().unwrap_or(false) {
good_latencies.push(sample["latency_ms"].as_f64().unwrap_or_default());
}
samples.push(sample);
}
let median = (!good_latencies.is_empty()).then(|| median(&good_latencies));
let p95 = (!good_latencies.is_empty()).then(|| percentile(&good_latencies, 95.0));
let status = if good_latencies.len() == samples.len() {
"ok"
} else {
"failed"
};
measurements.push(Measurement {
name: query.name.clone(),
kind: query.kind.clone(),
iterations: samples.len(),
samples,
latency_ms_median: median,
latency_ms_p95: p95,
status: status.to_string(),
});
}
let failed = !validation_errors.is_empty() || measurements.iter().any(|m| m.status == "failed");
QueryResult {
validation,
validation_errors,
measurements,
status: if failed { "failed" } else { "ok" }.to_string(),
}
}
fn validate_show_create(result: &Value, table: &Table) -> Vec<&'static str> {
let text = result
.get("response")
.map(response_text)
.unwrap_or_default()
.to_lowercase();
let mut errors = Vec::new();
if !text.contains(&table.name.to_lowercase()) {
errors.push("SHOW CREATE output does not contain table name");
}
if table.validate_show_create_engine && (!text.contains("engine") || !text.contains("mito")) {
errors.push("SHOW CREATE output does not mention ENGINE=mito");
}
if table.append_mode.is_some() && !text.contains("append_mode") {
errors.push("SHOW CREATE output does not mention append_mode");
}
if table.sst_format.is_some() && !text.contains("sst_format") {
errors.push("SHOW CREATE output does not mention sst_format");
}
errors
}
fn response_text(body: &Value) -> String {
body.as_str()
.map(ToOwned::to_owned)
.unwrap_or_else(|| serde_json::to_string(body).unwrap_or_default())
}
fn extract_execution_time_for_kind<'a>(kind: Option<&str>, body: &'a Value) -> Option<&'a Value> {
if kind == Some("prom_http") {
None
} else {
extract_execution_time(body)
}
}
fn extract_execution_time(body: &Value) -> Option<&Value> {
match body {
Value::Object(map) => {
for key in ["execution_time_ms", "execution_time", "elapsed"] {
if let Some(value) = map.get(key) {
return Some(value);
}
}
map.values().find_map(extract_execution_time)
}
Value::Array(values) => values.iter().find_map(extract_execution_time),
_ => None,
}
}
pub(super) fn median(values: &[f64]) -> f64 {
let mut ordered = values.to_vec();
ordered.sort_by(f64::total_cmp);
let middle = ordered.len() / 2;
if ordered.len().is_multiple_of(2) {
(ordered[middle - 1] + ordered[middle]) / 2.0
} else {
ordered[middle]
}
}
fn percentile(values: &[f64], pct: f64) -> f64 {
if values.is_empty() {
return 0.0;
}
let mut ordered = values.to_vec();
ordered.sort_by(f64::total_cmp);
let index = round_ties_even((pct / 100.0) * (ordered.len() - 1) as f64)
.clamp(0, ordered.len() as isize - 1) as usize;
ordered[index]
}
pub(super) fn round_ties_even(value: f64) -> isize {
let floor = value.floor();
let fraction = value - floor;
if fraction < 0.5 {
floor as isize
} else if fraction > 0.5 {
floor as isize + 1
} else if (floor as isize) % 2 == 0 {
floor as isize
} else {
floor as isize + 1
}
}
fn enforce_thresholds(
queries: &[Query],
base: &QueryResult,
candidate: &QueryResult,
) -> Result<Vec<Value>> {
let base_by_name: HashMap<_, _> = base
.measurements
.iter()
.map(|measurement| (measurement.name.as_deref(), measurement))
.collect();
let mut results = Vec::new();
for candidate_measurement in &candidate.measurements {
let query = queries
.iter()
.find(|query| query.name == candidate_measurement.name);
let thresholds = query.map_or_else(Map::new, |query| query.thresholds.clone());
let base_measurement = base_by_name.get(&candidate_measurement.name.as_deref());
if let Some(limit) = thresholds
.get("max_candidate_latency_regression_pct")
.filter(|value| !value.is_null())
.map(value_as_f64)
.transpose()?
{
let result = match base_measurement {
None => {
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing base measurement"})
}
Some(base) if matches!(base.latency_ms_median, None | Some(0.0)) => {
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "base median latency is missing or zero", "base_latency_ms_median": base.latency_ms_median})
}
Some(_) if candidate_measurement.latency_ms_median.is_none() => {
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing candidate measurement"})
}
Some(base) => {
let actual = (candidate_measurement.latency_ms_median.unwrap()
- base.latency_ms_median.unwrap())
/ base.latency_ms_median.unwrap()
* 100.0;
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": if actual <= limit { "passed" } else { "failed" }, "actual_pct": actual, "limit_pct": limit})
}
};
results.push(result);
}
for key in thresholds
.keys()
.filter(|key| *key != "max_candidate_latency_regression_pct")
{
results.push(json!({"query": candidate_measurement.name, "threshold": key, "status": "failed", "reason": "unsupported threshold"}));
}
}
Ok(results)
}
fn value_as_f64(value: &Value) -> Result<f64> {
value.as_f64().ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidData, "threshold must be numeric").into()
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn median_and_p95_match_python() {
assert_eq!(median(&[1.0, 8.0, 3.0, 4.0]), 3.5);
assert_eq!(
percentile(
&[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0],
95.0
),
10.0
);
assert_eq!(percentile(&[], 95.0), 0.0);
}
#[test]
fn finds_nested_execution_time_in_priority_order() {
let body = json!({"output": [{"elapsed": 3}], "execution_time": 2});
assert_eq!(extract_execution_time(&body), Some(&json!(2)));
assert_eq!(
extract_execution_time(&json!({"output": [{"elapsed": 3}]})),
Some(&json!(3))
);
}
#[test]
fn prom_http_does_not_extract_execution_time_from_response() {
let response = json!({
"data": {
"result": [{"metric": {"job": "api", "elapsed": "label"}}],
"elapsed": 42
}
});
assert_eq!(
extract_execution_time_for_kind(Some("prom_http"), &response),
None
);
assert_eq!(
extract_execution_time_for_kind(Some("sql"), &response),
Some(&json!(42))
);
}
#[test]
fn threshold_rejects_unknown_keys_and_zero_base() {
let query = Query {
name: Some("q".to_string()),
kind: None,
query: "SELECT 1".to_string(),
start: None,
end: None,
step: None,
warmup: 0,
iterations: 1,
thresholds: Map::from_iter([
("max_candidate_latency_regression_pct".to_string(), json!(0)),
("other".to_string(), json!(1)),
]),
};
let measurement = |median| Measurement {
name: Some("q".to_string()),
kind: None,
iterations: 1,
samples: vec![],
latency_ms_median: median,
latency_ms_p95: median,
status: "ok".to_string(),
};
let base = QueryResult {
validation: vec![],
validation_errors: vec![],
measurements: vec![measurement(Some(0.0))],
status: "ok".to_string(),
};
let candidate = QueryResult {
validation: vec![],
validation_errors: vec![],
measurements: vec![measurement(Some(1.0))],
status: "ok".to_string(),
};
let results = enforce_thresholds(&[query], &base, &candidate).unwrap();
assert_eq!(
results[0]["reason"],
"base median latency is missing or zero"
);
assert_eq!(results[1]["reason"], "unsupported threshold");
}
}
src/cmd/src/bin/query_regression_runner/mod.rs
+204
View File
@@ -0,0 +1,204 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::error::Error;
use std::path::PathBuf;
use clap::{Parser, Subcommand};
mod direct;
mod materialize;
mod measure;
mod model;
mod otlp;
mod plan;
mod remote;
mod sql;
pub(super) type Result<T> = std::result::Result<T, Box<dyn Error>>;
#[derive(Debug, Parser)]
#[command(about = "Measure query endpoints or materialize a direct-SST fixture")]
struct Cli {
#[command(subcommand)]
command: RunnerCommand,
}
#[derive(Debug, Subcommand)]
enum RunnerCommand {
/// Run normalized query regression cases against two HTTP endpoints.
Measure(MeasureArgs),
/// Create direct-SST tables, discover their regions, and generate fixtures.
PrepareDirect(PrepareDirectArgs),
/// Render the frontend Prometheus store configuration for a remote-write case.
RenderRemoteConfig(RenderRemoteConfigArgs),
/// Create a database and ingest a normalized Prometheus remote-write case.
PrepareRemote(PrepareRemoteArgs),
/// Inspect stopped remote-write storage and finalize the aggregate report.
FinalizeRemote(FinalizeRemoteArgs),
/// Run one externally managed OTLP trace-load target.
RunOtlpTarget(RunOtlpTargetArgs),
/// Combine externally managed OTLP trace-load target results.
FinalizeOtlp(FinalizeOtlpArgs),
/// Copy a direct-SST fixture into an object store destination.
Materialize(MaterializeArgs),
}
#[derive(Debug, Parser)]
struct MeasureArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long)]
base_http_port: u16,
#[arg(long)]
candidate_http_port: u16,
#[arg(long, value_name = "PATH")]
output: Option<PathBuf>,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
}
#[derive(Debug, Parser)]
struct PrepareDirectArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long)]
base_http_port: u16,
#[arg(long)]
candidate_http_port: u16,
#[arg(long, value_name = "PATH")]
fixture_dir: PathBuf,
#[arg(long, value_name = "PATH")]
output: Option<PathBuf>,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
#[arg(long)]
allow_large_fixture: bool,
}
#[derive(Debug, Parser)]
struct RenderRemoteConfigArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
output: PathBuf,
}
#[derive(Debug, Parser)]
struct PrepareRemoteArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long)]
base_http_port: u16,
#[arg(long)]
candidate_http_port: u16,
#[arg(long, value_name = "PATH")]
output: Option<PathBuf>,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
}
#[derive(Debug, Parser)]
struct FinalizeRemoteArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
candidate_bin: PathBuf,
#[arg(long, value_name = "PATH")]
base_data_home: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
candidate_data_home: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
base_destination: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
candidate_destination: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
report: PathBuf,
}
#[derive(Debug, Parser)]
struct RunOtlpTargetArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
otelgen_bin: PathBuf,
#[arg(long)]
http_port: u16,
#[arg(long)]
target_name: String,
#[arg(long, value_name = "PATH")]
work_dir: PathBuf,
#[arg(long, value_name = "PATH")]
output: PathBuf,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
}
#[derive(Debug, Parser)]
struct FinalizeOtlpArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
base_result: PathBuf,
#[arg(long, value_name = "PATH")]
candidate_result: PathBuf,
#[arg(long, value_name = "PATH")]
output: PathBuf,
}
#[derive(Debug, Parser)]
struct MaterializeArgs {
#[arg(long, value_name = "PATH")]
fixture_dir: PathBuf,
#[arg(
long,
value_name = "PATH",
help = "TOML file: data_home = \"...\" and object_store = { type = \"File\" }; object_store uses object_store::config::ObjectStoreConfig"
)]
destination: PathBuf,
}
pub(super) async fn run() {
if let Err(error) = run_inner().await {
eprintln!("query_regression_runner: {error}");
std::process::exit(1);
}
}
async fn run_inner() -> Result<()> {
match Cli::parse().command {
RunnerCommand::Measure(args) => measure::run_measure(args).await,
RunnerCommand::PrepareDirect(args) => direct::run_prepare_direct(args).await,
RunnerCommand::RenderRemoteConfig(args) => remote::run_render_remote_config(args).await,
RunnerCommand::PrepareRemote(args) => remote::run_prepare_remote(args).await,
RunnerCommand::FinalizeRemote(args) => remote::run_finalize_remote(args).await,
RunnerCommand::RunOtlpTarget(args) => otlp::run_otlp_target(args).await,
RunnerCommand::FinalizeOtlp(args) => otlp::run_finalize_otlp(args).await,
RunnerCommand::Materialize(args) => materialize::run_materialize(args).await,
}
}
src/cmd/src/bin/query_regression_runner/model.rs
+248
View File
@@ -0,0 +1,248 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use object_store::config::ObjectStoreConfig;
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
#[derive(Debug, Deserialize)]
pub(super) struct DestinationConfig {
pub(super) data_home: String,
pub(super) object_store: ObjectStoreConfig,
}
#[derive(Debug, Deserialize)]
pub(super) struct FixtureSummary {
pub(super) region_dir: String,
}
#[derive(Debug, Default, Serialize)]
pub(super) struct CopyCounts {
pub(super) files: u64,
pub(super) bytes: u64,
}
#[derive(Debug, Serialize)]
pub(super) struct MaterializeResult {
pub(super) region_dir: String,
pub(super) object_store: CopyCounts,
pub(super) manifest: CopyCounts,
}
#[derive(Debug, Deserialize)]
#[serde(tag = "kind")]
pub(super) enum Scenario {
#[serde(rename = "direct_readable_sst")]
DirectReadableSst {
tables: Vec<Table>,
layout: Layout,
#[serde(default)]
queries: Vec<Query>,
},
#[serde(rename = "prom_remote_write_then_query")]
PromRemoteWriteThenQuery {
remote_write: RemoteWrite,
#[serde(default)]
queries: Vec<Query>,
},
#[serde(rename = "otlp_trace_load")]
OtlpTraceLoad { load: OtlpTraceLoad },
}
#[derive(Debug, Deserialize)]
pub(super) struct Layout {
pub(super) regions: usize,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct Table {
pub(super) database: String,
pub(super) name: String,
#[serde(default)]
pub(super) engine: String,
#[serde(default)]
pub(super) columns: Vec<Column>,
#[serde(default)]
pub(super) primary_key: Vec<String>,
#[serde(default)]
pub(super) time_index: Option<String>,
#[serde(default)]
pub(super) append_mode: Option<bool>,
#[serde(default)]
pub(super) sst_format: Option<String>,
#[serde(default = "default_show_create_engine")]
pub(super) validate_show_create_engine: bool,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct Column {
pub(super) name: String,
#[serde(rename = "type")]
pub(super) data_type: String,
/// Semantic type ("tag", "field", "timestamp") from the case TOML. The
/// direct-SST fixture generator bakes index metadata into the region
/// manifest based on this semantic, so the CREATE TABLE must declare the
/// matching column options (see `create_table_sql` in `direct.rs`).
#[serde(default)]
pub(super) semantic: Option<String>,
}
const fn default_show_create_engine() -> bool {
true
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct RemoteWrite {
pub(super) database: String,
pub(super) metric: String,
pub(super) physical_table: String,
pub(super) series_count: u64,
pub(super) samples_per_series: u64,
pub(super) start_unix_millis: i64,
pub(super) step_millis: i64,
pub(super) chunk_series_count: u64,
pub(super) timeout_seconds: u64,
pub(super) sample_chunk_size: Option<u64>,
pub(super) flush_every_sample_chunks: u64,
pub(super) visibility_timeout_seconds: u64,
pub(super) prom_store: PromStore,
pub(super) value: RemoteValue,
pub(super) storage: Option<StorageConfig>,
pub(super) read_bench: Option<ReadBenchConfig>,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct PromStore {
pub(super) pending_rows_flush_interval: String,
pub(super) max_batch_rows: u64,
pub(super) max_concurrent_flushes: u64,
pub(super) worker_channel_capacity: u64,
pub(super) max_inflight_requests: u64,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct RemoteValue {
pub(super) pattern: String,
pub(super) base: f64,
pub(super) step: f64,
pub(super) cardinality: u64,
pub(super) seed: u64,
pub(super) run_length: u64,
pub(super) stall_every: u64,
pub(super) stall_length: u64,
pub(super) mixed_every: u64,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct OtlpTraceLoad {
pub(super) load: OtlpLoad,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct OtlpLoad {
pub(super) database: String,
pub(super) table: String,
pub(super) pipeline: String,
pub(super) duration_seconds: u64,
pub(super) warmup_seconds: u64,
pub(super) rate: u64,
pub(super) workers: usize,
pub(super) exporter_shards: usize,
pub(super) workload: String,
pub(super) visibility_timeout_seconds: u64,
pub(super) thresholds: OtlpThresholds,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct OtlpThresholds {
pub(super) max_candidate_throughput_regression_pct: f64,
pub(super) max_candidate_mean_latency_regression_pct: f64,
pub(super) max_failure_count: u64,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct StorageConfig {
pub(super) inspect: bool,
pub(super) column: String,
pub(super) root_suffix: Option<String>,
pub(super) include_metadata_files: bool,
pub(super) min_files: u64,
pub(super) min_files_with_column: u64,
pub(super) require_encodings: Vec<String>,
pub(super) forbid_encodings: Vec<String>,
pub(super) max_total_file_size_bytes: Option<u64>,
pub(super) max_column_compressed_size_bytes: Option<u64>,
pub(super) max_column_uncompressed_size_bytes: Option<u64>,
pub(super) max_candidate_total_file_size_regression_pct: Option<f64>,
pub(super) max_candidate_column_compressed_size_regression_pct: Option<f64>,
pub(super) max_candidate_column_uncompressed_size_regression_pct: Option<f64>,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct ReadBenchConfig {
pub(super) enabled: bool,
pub(super) parquetbench: bool,
pub(super) scanbench: bool,
pub(super) iterations: u64,
pub(super) projection: Vec<String>,
pub(super) parquet_reader: String,
pub(super) scan_scanner: String,
pub(super) parallelism: u64,
pub(super) max_files: Option<usize>,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct Query {
#[serde(default)]
pub(super) name: Option<String>,
#[serde(default)]
pub(super) kind: Option<String>,
pub(super) query: String,
/// Prometheus HTTP range query parameters (`kind = "prom_http"` only).
#[serde(default)]
pub(super) start: Option<String>,
#[serde(default)]
pub(super) end: Option<String>,
#[serde(default)]
pub(super) step: Option<String>,
#[serde(default)]
pub(super) warmup: usize,
#[serde(default = "one")]
pub(super) iterations: usize,
#[serde(default)]
pub(super) thresholds: Map<String, Value>,
}
const fn one() -> usize {
1
}
#[derive(Debug, Serialize)]
pub(super) struct QueryResult {
pub(super) validation: Vec<Value>,
pub(super) validation_errors: Vec<Value>,
pub(super) measurements: Vec<Measurement>,
pub(super) status: String,
}
#[derive(Debug, Serialize)]
pub(super) struct Measurement {
pub(super) name: Option<String>,
pub(super) kind: Option<String>,
pub(super) iterations: usize,
pub(super) samples: Vec<Value>,
pub(super) latency_ms_median: Option<f64>,
pub(super) latency_ms_p95: Option<f64>,
pub(super) status: String,
}
src/cmd/src/bin/query_regression_runner/otlp.rs
+618
View File
@@ -0,0 +1,618 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
use regex::Regex;
use reqwest::Client;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use crate::query_regression_runner::measure::round_ties_even;
use crate::query_regression_runner::model::{OtlpLoad, OtlpThresholds};
use crate::query_regression_runner::plan::normalized_otlp_load;
use crate::query_regression_runner::sql::{
extract_count_value, http_post_sql, sql_ident, sql_string, value_f64, value_u64,
};
use crate::query_regression_runner::{FinalizeOtlpArgs, Result, RunOtlpTargetArgs};
const OTLP_ROWS: &str = "greptime_frontend_otlp_traces_rows";
const OTLP_FAILURES: &str = "greptime_frontend_otlp_traces_failure_count";
const OTLP_ELAPSED_SUM: &str = "greptime_servers_http_otlp_traces_elapsed_sum";
const OTLP_ELAPSED_COUNT: &str = "greptime_servers_http_otlp_traces_elapsed_count";
#[derive(Clone, Debug, Deserialize, Serialize)]
struct OtlpSnapshot {
captured_monotonic_seconds: f64,
values: HashMap<String, f64>,
}
pub(super) async fn run_otlp_target(args: RunOtlpTargetArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let (_, load) = normalized_otlp_load(&args.fixture_generator, &args.case)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let result = match run_otlp_target_inner(&args, &load, &client).await {
Ok(result) => result,
Err(error) => {
json!({ "name": args.target_name, "status": "failed", "error": error.to_string() })
}
};
let text = format!("{}\n", serde_json::to_string_pretty(&result)?);
fs::write(&args.output, &text)?;
print!("{text}");
if result["status"] == "failed" {
std::process::exit(1);
}
Ok(())
}
async fn run_otlp_target_inner(
args: &RunOtlpTargetArgs,
load: &OtlpLoad,
client: &Client,
) -> Result<Value> {
let create_database = http_post_sql(
client,
args.http_port,
&format!(
"CREATE DATABASE IF NOT EXISTS {}",
sql_ident(&load.database)
),
"public",
)
.await;
if !create_database["ok"].as_bool().unwrap_or(false) {
return Ok(
json!({ "name": args.target_name, "create_database": create_database, "status": "failed" }),
);
}
let otelgen = run_otelgen_load(
&args.otelgen_bin,
args.http_port,
&args.work_dir,
load,
client,
)
.await?;
let metrics = summarize_otlp_metrics(&otelgen)?;
let flush = http_post_sql(
client,
args.http_port,
&format!("ADMIN FLUSH_TABLE({})", sql_string(&load.table)),
&load.database,
)
.await;
let visibility = poll_otlp_visibility(
client,
args.http_port,
&load.table,
&load.database,
metrics["accepted_spans"].as_u64().unwrap_or_default(),
load.visibility_timeout_seconds,
)
.await?;
let checks_ok = otelgen["status"] == "ok"
&& metrics["missing_metrics"]
.as_array()
.is_some_and(Vec::is_empty)
&& metrics["accepted_spans"].as_u64().unwrap_or_default() > 0
&& metrics["http_requests"].as_u64().unwrap_or_default() > 0
&& flush["ok"].as_bool().unwrap_or(false)
&& visibility["ok"].as_bool().unwrap_or(false)
&& visibility["row_count_ok"].as_bool().unwrap_or(false);
Ok(json!({
"name": args.target_name,
"otelgen": otelgen,
"metrics": metrics,
"create_database": create_database,
"flush": flush,
"visibility": visibility,
"status": if checks_ok { "measured" } else { "failed" },
}))
}
fn otelgen_command(otelgen_bin: &Path, http_port: u16, load: &OtlpLoad) -> Vec<String> {
vec![
otelgen_bin.to_string_lossy().to_string(),
"--protocol".to_string(),
"http".to_string(),
"--otel-exporter-otlp-endpoint".to_string(),
format!("127.0.0.1:{http_port}"),
"--otel-exporter-otlp-url-path".to_string(),
"/v1/otlp/v1/traces".to_string(),
"--header".to_string(),
format!("x-greptime-pipeline-name={}", load.pipeline),
"--header".to_string(),
format!("x-greptime-db-name={}", load.database),
"--header".to_string(),
format!("x-greptime-trace-table-name={}", load.table),
"--log-level".to_string(),
"error".to_string(),
"--insecure".to_string(),
"--duration".to_string(),
load.duration_seconds.to_string(),
"--rate".to_string(),
load.rate.to_string(),
"traces".to_string(),
"multi".to_string(),
"--workers".to_string(),
load.workers.to_string(),
"--scenarios".to_string(),
load.workload.clone(),
"--exporter-shards".to_string(),
load.exporter_shards.to_string(),
]
}
async fn run_otelgen_load(
otelgen_bin: &Path,
http_port: u16,
work_dir: &Path,
load: &OtlpLoad,
client: &Client,
) -> Result<Value> {
let command = otelgen_command(otelgen_bin, http_port, load);
let clock = Instant::now();
let initial = fetch_otlp_metrics(client, http_port, &clock).await?;
let log_dir = work_dir.join("otelgen");
fs::create_dir_all(&log_dir)?;
let stdout_path = log_dir.join("stdout.log");
let stderr_path = log_dir.join("stderr.log");
let mut child = Command::new(otelgen_bin)
.args(&command[1..])
.stdout(Stdio::from(fs::File::create(&stdout_path)?))
.stderr(Stdio::from(fs::File::create(&stderr_path)?))
.spawn()?;
let started = Instant::now();
if load.warmup_seconds > 0 {
let _ = wait_for_child(&mut child, Duration::from_secs(load.warmup_seconds)).await?;
}
let warmed = fetch_otlp_metrics(client, http_port, &clock).await?;
let mut timed_out = false;
if child.try_wait()?.is_none() {
let remaining = load
.duration_seconds
.saturating_sub(load.warmup_seconds)
.saturating_add(60)
.max(60);
if !wait_for_child(&mut child, Duration::from_secs(remaining)).await? {
timed_out = true;
child.kill()?;
let _ = child.wait()?;
}
}
let final_snapshot = fetch_otlp_metrics(client, http_port, &clock).await?;
let returncode = match child.try_wait()? {
Some(status) => status.code(),
None => {
child.kill()?;
child.wait()?.code()
}
};
let elapsed_seconds = started.elapsed().as_secs_f64();
Ok(json!({
"status": if returncode == Some(0) && !timed_out && elapsed_seconds + 1.0 >= load.duration_seconds as f64 { "ok" } else { "failed" },
"cmd": command,
"returncode": returncode,
"timed_out": timed_out,
"elapsed_seconds": elapsed_seconds,
"stdout_path": stdout_path,
"stderr_path": stderr_path,
"snapshots": { "initial": initial, "warmup": warmed, "final": final_snapshot },
}))
}
async fn wait_for_child(child: &mut std::process::Child, timeout: Duration) -> Result<bool> {
let deadline = Instant::now() + timeout;
loop {
if child.try_wait()?.is_some() {
return Ok(true);
}
if Instant::now() >= deadline {
return Ok(false);
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
}
async fn fetch_otlp_metrics(
client: &Client,
http_port: u16,
clock: &Instant,
) -> Result<OtlpSnapshot> {
let text = client
.get(format!("http://127.0.0.1:{http_port}/metrics"))
.send()
.await?
.error_for_status()?
.text()
.await?;
Ok(OtlpSnapshot {
captured_monotonic_seconds: clock.elapsed().as_secs_f64(),
values: parse_prometheus_metrics(&text)?,
})
}
fn parse_prometheus_metrics(text: &str) -> Result<HashMap<String, f64>> {
let sample = Regex::new(r"^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{[^}]*\})?\s+(\S+)")?;
let mut values = HashMap::new();
for line in text.lines() {
let Some(captures) = sample.captures(line.trim()) else {
continue;
};
let (Some(name), Some(value)) = (captures.get(1), captures.get(2)) else {
continue;
};
let name = name.as_str();
if !matches!(
name,
OTLP_ROWS | OTLP_FAILURES | OTLP_ELAPSED_SUM | OTLP_ELAPSED_COUNT
) {
continue;
}
let value: f64 = value.as_str().parse()?;
if !value.is_finite() {
return Err(format!("non-finite Prometheus sample for {name}").into());
}
*values.entry(name.to_string()).or_default() += value;
}
Ok(values)
}
fn metric_delta(after: &OtlpSnapshot, before: &OtlpSnapshot, name: &str) -> Result<f64> {
let delta = after.values.get(name).copied().unwrap_or_default()
- before.values.get(name).copied().unwrap_or_default();
if delta < 0.0 {
return Err(format!("metric {name} decreased by {}", -delta).into());
}
Ok(delta)
}
fn summarize_otlp_metrics(run: &Value) -> Result<Value> {
let snapshots = run
.get("snapshots")
.ok_or("otelgen result has no snapshots")?;
let initial: OtlpSnapshot = serde_json::from_value(snapshots["initial"].clone())?;
let warmed: OtlpSnapshot = serde_json::from_value(snapshots["warmup"].clone())?;
let final_snapshot: OtlpSnapshot = serde_json::from_value(snapshots["final"].clone())?;
let missing_metrics = [OTLP_ROWS, OTLP_ELAPSED_SUM, OTLP_ELAPSED_COUNT]
.into_iter()
.filter(|name| !final_snapshot.values.contains_key(*name))
.collect::<Vec<_>>();
let accepted_spans =
round_ties_even(metric_delta(&final_snapshot, &initial, OTLP_ROWS)?) as u64;
let measurement_accepted_spans =
round_ties_even(metric_delta(&final_snapshot, &warmed, OTLP_ROWS)?) as u64;
let http_requests =
round_ties_even(metric_delta(&final_snapshot, &warmed, OTLP_ELAPSED_COUNT)?) as u64;
let latency_seconds = metric_delta(&final_snapshot, &warmed, OTLP_ELAPSED_SUM)?;
let measurement_seconds =
final_snapshot.captured_monotonic_seconds - warmed.captured_monotonic_seconds;
Ok(json!({
"accepted_spans": accepted_spans,
"measurement_accepted_spans": measurement_accepted_spans,
"accepted_spans_per_second": if measurement_seconds > 0.0 { Some(measurement_accepted_spans as f64 / measurement_seconds) } else { None },
"http_requests": http_requests,
"mean_http_latency_ms": if http_requests > 0 { Some(latency_seconds / http_requests as f64 * 1000.0) } else { None },
"failure_count": round_ties_even(metric_delta(&final_snapshot, &initial, OTLP_FAILURES)?) as u64,
"measurement_seconds": measurement_seconds,
"missing_metrics": missing_metrics,
}))
}
async fn poll_otlp_visibility(
client: &Client,
port: u16,
table_name: &str,
database: &str,
expected_rows: u64,
timeout_seconds: u64,
) -> Result<Value> {
let sql = format!("SELECT count(*) FROM {}", sql_ident(table_name));
let deadline = Instant::now() + Duration::from_secs(timeout_seconds);
let mut attempts = 0;
loop {
attempts += 1;
let mut result = http_post_sql(client, port, &sql, database).await;
let observed_rows = extract_count_value(&result);
let row_count_ok =
result["ok"].as_bool().unwrap_or(false) && observed_rows == Some(expected_rows);
result
.as_object_mut()
.ok_or("count result must be an object")?
.extend([
("expected_rows".to_string(), json!(expected_rows)),
("observed_rows".to_string(), json!(observed_rows)),
("attempts".to_string(), json!(attempts)),
("row_count_ok".to_string(), json!(row_count_ok)),
]);
if row_count_ok {
return Ok(result);
}
if Instant::now() >= deadline {
result["ok"] = json!(false);
result["row_count_ok"] = json!(false);
result["error"] = json!(format!(
"expected {expected_rows} rows but observed {observed_rows:?} after {attempts} attempts"
));
return Ok(result);
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
pub(super) async fn run_finalize_otlp(args: FinalizeOtlpArgs) -> Result<()> {
let (case_path, load) = normalized_otlp_load(&args.fixture_generator, &args.case)?;
let raw_case: toml::Value = toml::from_str(&fs::read_to_string(&case_path)?)?;
let case_metadata = serde_json::to_value(
raw_case
.get("case")
.cloned()
.unwrap_or(toml::Value::Table(toml::map::Map::new())),
)?;
let base: Value = serde_json::from_slice(&fs::read(&args.base_result)?)?;
let candidate: Value = serde_json::from_slice(&fs::read(&args.candidate_result)?)?;
let thresholds = enforce_otlp_thresholds(
&load.thresholds,
base.get("metrics").unwrap_or(&Value::Null),
candidate.get("metrics").unwrap_or(&Value::Null),
);
let failed = thresholds
.iter()
.any(|threshold| threshold["status"] == "failed")
|| [base.get("status"), candidate.get("status")]
.into_iter()
.any(|status| status.and_then(Value::as_str) == Some("failed"));
let report = json!({
"case_path": case_path,
"case": case_metadata,
"scenario": { "kind": "otlp_trace_load", "load": load },
"targets": [base, candidate],
"thresholds": thresholds,
"status": if failed { "failed" } else { "ok" },
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
fs::write(&args.output, &text)?;
print!("{text}");
if failed {
std::process::exit(1);
}
Ok(())
}
fn enforce_otlp_thresholds(
thresholds: &OtlpThresholds,
base: &Value,
candidate: &Value,
) -> Vec<Value> {
let mut results = Vec::new();
for (target, metrics) in [("base", base), ("candidate", candidate)] {
let failures = value_u64(metrics.get("failure_count"));
results.push(json!({
"target": target,
"threshold": "max_failure_count",
"status": if failures.is_some_and(|failures| failures <= thresholds.max_failure_count) { "passed" } else { "failed" },
"actual": failures,
"limit": thresholds.max_failure_count,
}));
}
for (name, base_value, candidate_value, limit, inverse) in [
(
"max_candidate_throughput_regression_pct",
value_f64(base.get("accepted_spans_per_second")),
value_f64(candidate.get("accepted_spans_per_second")),
thresholds.max_candidate_throughput_regression_pct,
true,
),
(
"max_candidate_mean_latency_regression_pct",
value_f64(base.get("mean_http_latency_ms")),
value_f64(candidate.get("mean_http_latency_ms")),
thresholds.max_candidate_mean_latency_regression_pct,
false,
),
] {
let (Some(base_value), Some(candidate_value)) =
(base_value.filter(|value| *value != 0.0), candidate_value)
else {
results.push(json!({
"threshold": name,
"status": "failed",
"reason": if inverse { "missing or zero throughput" } else { "missing or zero mean latency" },
"base": base_value,
"candidate": candidate_value,
}));
continue;
};
let actual = if inverse {
(base_value - candidate_value) / base_value * 100.0
} else {
(candidate_value - base_value) / base_value * 100.0
};
results.push(json!({
"threshold": name,
"status": if actual <= limit { "passed" } else { "failed" },
"actual_pct": actual,
"limit_pct": limit,
"base": base_value,
"candidate": candidate_value,
}));
}
results
}
#[cfg(test)]
mod tests {
use super::*;
fn otlp_load_for_test() -> OtlpLoad {
serde_json::from_value(json!({
"database": "public",
"table": "opentelemetry_traces",
"pipeline": "greptime_trace_v1",
"duration_seconds": 120,
"warmup_seconds": 60,
"rate": 50000,
"workers": 4,
"exporter_shards": 4,
"workload": "microservices",
"visibility_timeout_seconds": 30,
"thresholds": {
"max_candidate_throughput_regression_pct": 20.0,
"max_candidate_mean_latency_regression_pct": 20.0,
"max_failure_count": 0
}
}))
.unwrap()
}
#[test]
fn parses_labeled_otlp_metrics_and_rejects_non_finite_samples() {
let metrics = parse_prometheus_metrics(
"greptime_frontend_otlp_traces_rows 10\n\
greptime_frontend_otlp_traces_failure_count{kind=\"a\"} 1\n\
greptime_frontend_otlp_traces_failure_count{kind=\"b\"} 2\n",
)
.unwrap();
assert_eq!(metrics[OTLP_ROWS], 10.0);
assert_eq!(metrics[OTLP_FAILURES], 3.0);
assert!(parse_prometheus_metrics("greptime_frontend_otlp_traces_rows NaN").is_err());
let before = OtlpSnapshot {
captured_monotonic_seconds: 0.0,
values: HashMap::from([(OTLP_ROWS.to_string(), 2.0)]),
};
let after = OtlpSnapshot {
captured_monotonic_seconds: 1.0,
values: HashMap::from([(OTLP_ROWS.to_string(), 1.0)]),
};
assert!(metric_delta(&after, &before, OTLP_ROWS).is_err());
}
#[test]
fn summarizes_otlp_deltas_and_builds_exact_command() {
let snapshot = |captured: f64, values: Vec<(String, f64)>| OtlpSnapshot {
captured_monotonic_seconds: captured,
values: values.into_iter().collect(),
};
let initial = snapshot(0.0, vec![(OTLP_ROWS.to_string(), 10.0)]);
let warmed = snapshot(
5.0,
vec![
(OTLP_ROWS.to_string(), 110.0),
(OTLP_ELAPSED_SUM.to_string(), 1.0),
(OTLP_ELAPSED_COUNT.to_string(), 10.0),
],
);
let final_snapshot = snapshot(
15.0,
vec![
(OTLP_ROWS.to_string(), 310.0),
(OTLP_FAILURES.to_string(), 3.0),
(OTLP_ELAPSED_SUM.to_string(), 3.0),
(OTLP_ELAPSED_COUNT.to_string(), 30.0),
],
);
let metrics = summarize_otlp_metrics(&json!({
"snapshots": { "initial": initial, "warmup": warmed, "final": final_snapshot }
}))
.unwrap();
assert_eq!(metrics["accepted_spans"], 300);
assert_eq!(metrics["measurement_accepted_spans"], 200);
assert_eq!(metrics["accepted_spans_per_second"], 20.0);
assert_eq!(metrics["http_requests"], 20);
assert_eq!(metrics["mean_http_latency_ms"], 100.0);
assert_eq!(metrics["failure_count"], 3);
let missing = summarize_otlp_metrics(&json!({
"snapshots": {
"initial": snapshot(0.0, vec![(OTLP_ROWS.to_string(), 0.0)]),
"warmup": snapshot(1.0, vec![(OTLP_ROWS.to_string(), 1.0)]),
"final": snapshot(2.0, vec![(OTLP_ROWS.to_string(), 2.0)])
}
}))
.unwrap();
assert_eq!(
missing["missing_metrics"],
json!([OTLP_ELAPSED_SUM, OTLP_ELAPSED_COUNT])
);
let command = otelgen_command(Path::new("/bin/otelgen"), 4000, &otlp_load_for_test());
assert_eq!(
command,
vec![
"/bin/otelgen",
"--protocol",
"http",
"--otel-exporter-otlp-endpoint",
"127.0.0.1:4000",
"--otel-exporter-otlp-url-path",
"/v1/otlp/v1/traces",
"--header",
"x-greptime-pipeline-name=greptime_trace_v1",
"--header",
"x-greptime-db-name=public",
"--header",
"x-greptime-trace-table-name=opentelemetry_traces",
"--log-level",
"error",
"--insecure",
"--duration",
"120",
"--rate",
"50000",
"traces",
"multi",
"--workers",
"4",
"--scenarios",
"microservices",
"--exporter-shards",
"4",
]
);
}
#[test]
fn otlp_thresholds_handle_zero_metrics_and_pass_fail_cases() {
let load = otlp_load_for_test();
let base = json!({ "failure_count": 0, "accepted_spans_per_second": 20.0, "mean_http_latency_ms": 100.0 });
let candidate = json!({ "failure_count": 0, "accepted_spans_per_second": 18.0, "mean_http_latency_ms": 110.0 });
let results = enforce_otlp_thresholds(&load.thresholds, &base, &candidate);
assert!(results.iter().all(|result| result["status"] == "passed"));
let failed = enforce_otlp_thresholds(
&load.thresholds,
&json!({ "failure_count": 0, "accepted_spans_per_second": 0.0, "mean_http_latency_ms": 100.0 }),
&json!({ "failure_count": 3, "accepted_spans_per_second": 20.0, "mean_http_latency_ms": 130.0 }),
);
assert!(
failed
.iter()
.any(|result| result["threshold"] == "max_failure_count"
&& result["target"] == "candidate"
&& result["status"] == "failed")
);
assert!(failed.iter().any(|result| result["threshold"]
== "max_candidate_throughput_regression_pct"
&& result["reason"] == "missing or zero throughput"));
assert!(failed.iter().any(|result| result["threshold"]
== "max_candidate_mean_latency_regression_pct"
&& result["status"] == "failed"));
}
}
src/cmd/src/bin/query_regression_runner/plan.rs
+122
View File
@@ -0,0 +1,122 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::process::Command;
use serde_json::Value;
use crate::query_regression_runner::Result;
use crate::query_regression_runner::model::{
Layout, OtlpLoad, Query, RemoteWrite, Scenario, Table,
};
pub(super) fn load_plan(generator: &PathBuf, case_path: &PathBuf) -> Result<Value> {
let output = Command::new(generator)
.args(["plan", "--case"])
.arg(case_path)
.output()?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(format!("query_perf_fixture plan failed: {:.2000}", stderr).into());
}
Ok(serde_json::from_slice(&output.stdout)?)
}
pub(super) fn normalize_scenario(scenario: Scenario) -> Result<(Vec<Table>, Vec<Query>)> {
match scenario {
Scenario::DirectReadableSst {
tables,
layout,
queries,
} => Ok((validate_direct_tables(tables, layout)?, queries)),
Scenario::PromRemoteWriteThenQuery {
remote_write,
queries,
} => Ok((
vec![Table {
database: remote_write.database,
name: remote_write.metric,
engine: "metric".to_string(),
columns: vec![],
primary_key: vec![],
time_index: None,
append_mode: None,
sst_format: None,
validate_show_create_engine: false,
}],
queries,
)),
Scenario::OtlpTraceLoad { .. } => {
Err("measure requires a query scenario, not otlp_trace_load".into())
}
}
}
pub(super) fn validate_direct_tables(tables: Vec<Table>, layout: Layout) -> Result<Vec<Table>> {
if tables.is_empty() || layout.regions != 1 {
return Err("runner supports one or more tables and exactly one region per table".into());
}
let mut pairs = HashSet::new();
let mut names = HashSet::new();
for table in &tables {
if !pairs.insert((&table.database, &table.name)) {
return Err("duplicate (database, name) table entries are not supported".into());
}
if !names.insert(&table.name) {
return Err("duplicate table names are not supported".into());
}
}
Ok(tables)
}
pub(super) fn normalized_remote_write(
generator: &PathBuf,
case: &Path,
) -> Result<(PathBuf, RemoteWrite)> {
let case_path = case.canonicalize()?;
let plan = load_plan(generator, &case_path)?;
let scenario = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
match serde_json::from_value(scenario)? {
Scenario::PromRemoteWriteThenQuery { remote_write, .. } => Ok((case_path, remote_write)),
Scenario::DirectReadableSst { .. } => {
Err("remote command requires scenario kind prom_remote_write_then_query".into())
}
Scenario::OtlpTraceLoad { .. } => {
Err("remote command requires scenario kind prom_remote_write_then_query".into())
}
}
}
pub(super) fn normalized_otlp_load(
generator: &PathBuf,
case: &Path,
) -> Result<(PathBuf, OtlpLoad)> {
let case_path = case.canonicalize()?;
let plan = load_plan(generator, &case_path)?;
let scenario = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
match serde_json::from_value(scenario)? {
Scenario::OtlpTraceLoad { load } => Ok((case_path, load.load)),
Scenario::DirectReadableSst { .. } | Scenario::PromRemoteWriteThenQuery { .. } => {
Err("OTLP command requires scenario kind otlp_trace_load".into())
}
}
}
src/cmd/src/bin/query_regression_runner/remote/ingest.rs
+505
View File
@@ -0,0 +1,505 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::Path;
use std::process::Command;
use std::time::{Duration, Instant};
use reqwest::Client;
use serde_json::{Value, json};
use crate::query_regression_runner::model::{PromStore, RemoteWrite};
use crate::query_regression_runner::plan::normalized_remote_write;
use crate::query_regression_runner::sql::{
extract_count_value, http_post_sql, sql_ident, sql_string, value_f64, value_u64,
};
use crate::query_regression_runner::{PrepareRemoteArgs, RenderRemoteConfigArgs, Result};
pub(super) async fn run_render_remote_config(args: RenderRemoteConfigArgs) -> Result<()> {
let (_, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?;
fs::write(args.output, frontend_prom_config(&remote.prom_store)?)?;
Ok(())
}
fn frontend_prom_config(prom: &PromStore) -> Result<String> {
Ok(format!(
"[prom_store]\nenable = true\nwith_metric_engine = true\npending_rows_flush_interval = {}\nmax_batch_rows = {}\nmax_concurrent_flushes = {}\nworker_channel_capacity = {}\nmax_inflight_requests = {}\n",
serde_json::to_string(&prom.pending_rows_flush_interval)?,
prom.max_batch_rows,
prom.max_concurrent_flushes,
prom.worker_channel_capacity,
prom.max_inflight_requests,
))
}
pub(super) async fn run_prepare_remote(args: PrepareRemoteArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let (case_path, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let base = prepare_remote_target(
"base",
args.base_http_port,
&args.fixture_generator,
&remote,
&client,
)
.await?;
let candidate = prepare_remote_target(
"candidate",
args.candidate_http_port,
&args.fixture_generator,
&remote,
&client,
)
.await?;
let report = json!({
"case_path": case_path,
"scenario": "prom_remote_write_then_query",
"base": base,
"candidate": candidate,
"status": "ok",
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
if let Some(path) = args.output {
fs::write(path, &text)?;
}
print!("{text}");
Ok(())
}
async fn prepare_remote_target(
name: &str,
port: u16,
generator: &Path,
remote: &RemoteWrite,
client: &Client,
) -> Result<Value> {
let create_database = http_post_sql(
client,
port,
&format!(
"CREATE DATABASE IF NOT EXISTS {}",
sql_ident(&remote.database)
),
"public",
)
.await;
if !create_database["ok"].as_bool().unwrap_or(false) {
return Err(format!(
"CREATE DATABASE {} failed for {name}: {create_database}",
remote.database
)
.into());
}
let (remote_write, flushes) = ingest_remote_write(generator, port, remote, client).await?;
let expected_rows = remote
.series_count
.checked_mul(remote.samples_per_series)
.ok_or("expected remote-write row count overflows u64")?;
let visibility = poll_expected_count(
client,
port,
&remote.metric,
&remote.database,
expected_rows,
remote.visibility_timeout_seconds,
)
.await?;
Ok(json!({
"name": name,
"create_database": create_database,
"remote_write": remote_write,
"flushes": flushes,
"visibility": visibility,
"status": "ok",
}))
}
#[derive(Clone, Debug, PartialEq, Eq)]
struct SampleChunk {
index: u64,
offset: u64,
samples_per_series: u64,
start_unix_millis: i64,
}
async fn ingest_remote_write(
generator: &Path,
port: u16,
remote: &RemoteWrite,
client: &Client,
) -> Result<(Value, Vec<Value>)> {
let Some(chunks) = sample_chunks(remote)? else {
let remote_write = run_remote_write(generator, port, remote, None)?;
let flush = flush_remote_table(client, port, remote, "final", None).await?;
return Ok((remote_write, vec![flush]));
};
let mut results = Vec::with_capacity(chunks.len());
let mut flushes = Vec::new();
let scheduled_flushes = scheduled_flushes(&chunks, remote.flush_every_sample_chunks);
for chunk in &chunks {
let mut result = run_remote_write(generator, port, remote, Some(chunk))?;
result
.as_object_mut()
.ok_or("remote-write result must be an object")?
.extend([
("sample_offset".to_string(), json!(chunk.offset)),
(
"samples_per_series".to_string(),
json!(chunk.samples_per_series),
),
("chunk_index".to_string(), json!(chunk.index)),
]);
results.push(result);
if let Some((_, reason)) = scheduled_flushes
.iter()
.find(|(index, _)| *index == chunk.index)
{
flushes
.push(flush_remote_table(client, port, remote, reason, Some(chunk.index)).await?);
}
}
Ok((
json!({
"status": "ok",
"mode": "sample-chunked",
"sample_chunk_size": remote.sample_chunk_size,
"flush_every_sample_chunks": remote.flush_every_sample_chunks,
"chunks": results,
"aggregate": summarize_remote_chunks(&results),
}),
flushes,
))
}
fn sample_chunks(remote: &RemoteWrite) -> Result<Option<Vec<SampleChunk>>> {
let Some(chunk_samples) = remote.sample_chunk_size else {
return Ok(None);
};
if chunk_samples == 0 {
return Err("scenario.remote_write.sample_chunk_size must be positive".into());
}
if remote.flush_every_sample_chunks == 0 {
return Err("scenario.remote_write.flush_every_sample_chunks must be positive".into());
}
let mut chunks = Vec::new();
let mut offset = 0;
while offset < remote.samples_per_series {
let samples_per_series = chunk_samples.min(remote.samples_per_series - offset);
chunks.push(SampleChunk {
index: chunks.len() as u64 + 1,
offset,
samples_per_series,
start_unix_millis: remote.start_unix_millis + offset as i64 * remote.step_millis,
});
offset += samples_per_series;
}
Ok(Some(chunks))
}
fn scheduled_flushes(chunks: &[SampleChunk], flush_every: u64) -> Vec<(u64, &'static str)> {
let mut scheduled = chunks
.iter()
.filter(|chunk| chunk.index % flush_every == 0)
.map(|chunk| (chunk.index, "periodic"))
.collect::<Vec<_>>();
if let Some(chunk) = chunks.last()
&& chunk.index % flush_every != 0
{
scheduled.push((chunk.index, "final"));
}
scheduled
}
fn summarize_remote_chunks(chunks: &[Value]) -> Value {
let mut rows = 0;
let mut samples_written = 0;
let mut batches = 0;
let mut elapsed_seconds = 0.0;
for chunk in chunks {
let summary = chunk.get("summary").unwrap_or(&Value::Null);
let chunk_rows = value_u64(summary.get("rows")).unwrap_or(0);
rows += chunk_rows;
samples_written += value_u64(summary.get("samples_written")).unwrap_or(chunk_rows);
batches += value_u64(summary.get("batches")).unwrap_or(0);
elapsed_seconds += value_f64(summary.get("elapsed_seconds"))
.or_else(|| value_f64(chunk.get("elapsed_seconds")))
.unwrap_or(0.0);
}
json!({ "rows": rows, "samples_written": samples_written, "batches": batches, "elapsed_seconds": elapsed_seconds })
}
fn run_remote_write(
generator: &Path,
port: u16,
remote: &RemoteWrite,
chunk: Option<&SampleChunk>,
) -> Result<Value> {
let command = remote_write_command(generator, port, remote, chunk);
let started = Instant::now();
let output = Command::new(generator).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
if !output.status.success() {
return Err(format!(
"remote-write generator failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}",
output.status.code(), stderr
)
.into());
}
let mut result = json!({
"status": "ok",
"command": command,
"returncode": output.status.code(),
"elapsed_seconds": elapsed_seconds,
"stdout": stdout,
"stderr": stderr,
});
match serde_json::from_str(result["stdout"].as_str().unwrap_or_default()) {
Ok(summary) => result["summary"] = summary,
Err(_) => result["summary_parse_error"] = result["stdout"].clone(),
}
Ok(result)
}
fn remote_write_command(
generator: &Path,
port: u16,
remote: &RemoteWrite,
chunk: Option<&SampleChunk>,
) -> Vec<String> {
let (samples_per_series, start_unix_millis, sample_offset, total_samples_per_series) = chunk
.map(|chunk| {
(
chunk.samples_per_series,
chunk.start_unix_millis,
Some(chunk.offset),
Some(remote.samples_per_series),
)
})
.unwrap_or((
remote.samples_per_series,
remote.start_unix_millis,
None,
None,
));
let mut command = vec![
generator.to_string_lossy().to_string(),
"prom-remote-write".to_string(),
"--endpoint".to_string(),
format!("http://127.0.0.1:{port}/v1/prometheus/write"),
"--database".to_string(),
remote.database.clone(),
"--metric".to_string(),
remote.metric.clone(),
"--physical-table".to_string(),
remote.physical_table.clone(),
"--series-count".to_string(),
remote.series_count.to_string(),
"--samples-per-series".to_string(),
samples_per_series.to_string(),
"--start-unix-millis".to_string(),
start_unix_millis.to_string(),
"--step-millis".to_string(),
remote.step_millis.to_string(),
"--chunk-series-count".to_string(),
remote.chunk_series_count.to_string(),
"--timeout-seconds".to_string(),
remote.timeout_seconds.to_string(),
"--value-pattern".to_string(),
remote.value.pattern.clone(),
"--value-base".to_string(),
json_number(remote.value.base),
"--value-step".to_string(),
json_number(remote.value.step),
"--value-cardinality".to_string(),
remote.value.cardinality.to_string(),
"--value-seed".to_string(),
remote.value.seed.to_string(),
"--value-run-length".to_string(),
remote.value.run_length.to_string(),
"--value-stall-every".to_string(),
remote.value.stall_every.to_string(),
"--value-stall-length".to_string(),
remote.value.stall_length.to_string(),
"--value-mixed-every".to_string(),
remote.value.mixed_every.to_string(),
];
if let Some(sample_offset) = sample_offset {
command.extend([
"--value-sample-offset".to_string(),
sample_offset.to_string(),
]);
}
if let Some(total_samples_per_series) = total_samples_per_series {
command.extend([
"--value-total-samples-per-series".to_string(),
total_samples_per_series.to_string(),
]);
}
command
}
async fn flush_remote_table(
client: &Client,
port: u16,
remote: &RemoteWrite,
reason: &str,
chunk_index: Option<u64>,
) -> Result<Value> {
let mut result = http_post_sql(
client,
port,
&format!("ADMIN FLUSH_TABLE({})", sql_string(&remote.physical_table)),
&remote.database,
)
.await;
if !result["ok"].as_bool().unwrap_or(false) {
return Err(format!(
"ADMIN FLUSH_TABLE {} failed: {result}",
remote.physical_table
)
.into());
}
result
.as_object_mut()
.ok_or("flush result must be an object")?
.extend([
("physical_table".to_string(), json!(remote.physical_table)),
("reason".to_string(), json!(reason)),
("chunk_index".to_string(), json!(chunk_index)),
]);
Ok(result)
}
async fn poll_expected_count(
client: &Client,
port: u16,
table_name: &str,
database: &str,
expected_rows: u64,
visibility_timeout_seconds: u64,
) -> Result<Value> {
let sql = format!("SELECT count(*) FROM {}", sql_ident(table_name));
let deadline = Instant::now() + Duration::from_secs(visibility_timeout_seconds);
let mut attempts = 0;
loop {
attempts += 1;
let mut result = http_post_sql(client, port, &sql, database).await;
let observed_rows = extract_count_value(&result);
let row_count_ok =
result["ok"].as_bool().unwrap_or(false) && observed_rows == Some(expected_rows);
result
.as_object_mut()
.ok_or("count result must be an object")?
.extend([
("expected_rows".to_string(), json!(expected_rows)),
("observed_rows".to_string(), json!(observed_rows)),
("attempts".to_string(), json!(attempts)),
("row_count_ok".to_string(), json!(row_count_ok)),
]);
if row_count_ok {
return Ok(result);
}
if Instant::now() >= deadline {
return Err(format!(
"expected {expected_rows} rows but observed {observed_rows:?} after {attempts} attempts"
)
.into());
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
fn json_number(value: f64) -> String {
serde_json::to_string(&value).unwrap_or_else(|_| value.to_string())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query_regression_runner::model::{PromStore, RemoteValue};
#[test]
fn schedules_remote_sample_chunks_and_flushes() {
let remote = RemoteWrite {
database: "public".to_string(),
metric: "metric".to_string(),
physical_table: "physical".to_string(),
series_count: 2,
samples_per_series: 5,
start_unix_millis: 100,
step_millis: 10,
chunk_series_count: 1,
timeout_seconds: 60,
sample_chunk_size: Some(2),
flush_every_sample_chunks: 2,
visibility_timeout_seconds: 30,
prom_store: PromStore {
pending_rows_flush_interval: "1s".to_string(),
max_batch_rows: 1,
max_concurrent_flushes: 1,
worker_channel_capacity: 1,
max_inflight_requests: 1,
},
value: RemoteValue {
pattern: "linear".to_string(),
base: 0.0,
step: 1.0,
cardinality: 1,
seed: 0,
run_length: 1,
stall_every: 0,
stall_length: 0,
mixed_every: 0,
},
storage: None,
read_bench: None,
};
let chunks = sample_chunks(&remote).unwrap().unwrap();
assert_eq!(
chunks,
vec![
SampleChunk {
index: 1,
offset: 0,
samples_per_series: 2,
start_unix_millis: 100,
},
SampleChunk {
index: 2,
offset: 2,
samples_per_series: 2,
start_unix_millis: 120,
},
SampleChunk {
index: 3,
offset: 4,
samples_per_series: 1,
start_unix_millis: 140,
},
]
);
assert_eq!(
scheduled_flushes(&chunks, remote.flush_every_sample_chunks),
vec![(2, "periodic"), (3, "final")]
);
}
}
src/cmd/src/bin/query_regression_runner/remote/mod.rs
+183
View File
@@ -0,0 +1,183 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::{Path, PathBuf};
use serde_json::{Value, json};
use self::read_bench::run_read_bench;
use self::storage::{enforce_storage_thresholds, run_storage_inspection};
use crate::query_regression_runner::model::DestinationConfig;
use crate::query_regression_runner::plan::normalized_remote_write;
use crate::query_regression_runner::{
FinalizeRemoteArgs, PrepareRemoteArgs, RenderRemoteConfigArgs, Result,
};
mod ingest;
mod read_bench;
mod storage;
pub(super) async fn run_render_remote_config(args: RenderRemoteConfigArgs) -> Result<()> {
ingest::run_render_remote_config(args).await
}
pub(super) async fn run_prepare_remote(args: PrepareRemoteArgs) -> Result<()> {
ingest::run_prepare_remote(args).await
}
pub(super) async fn run_finalize_remote(args: FinalizeRemoteArgs) -> Result<()> {
let (_, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?;
let storage = remote.storage.as_ref().filter(|storage| storage.inspect);
let mut report: Value = serde_json::from_slice(&fs::read(&args.report)?)?;
let bench_root = args.report.parent().unwrap_or_else(|| Path::new("."));
let mut inspections = Vec::new();
let target_failed = {
let targets = report
.get_mut("targets")
.and_then(Value::as_array_mut)
.ok_or("report has no targets array")?;
for (name, data_home, destination) in [
(
"base",
args.base_data_home.as_deref(),
args.base_destination.as_deref(),
),
(
"candidate",
args.candidate_data_home.as_deref(),
args.candidate_destination.as_deref(),
),
] {
let (data_home, destination) = match (data_home, destination) {
(Some(data_home), None) => (data_home.to_path_buf(), None),
(None, Some(path)) => {
let destination: DestinationConfig =
toml::from_str(&fs::read_to_string(path)?)?;
(
PathBuf::from(destination.data_home),
Some(path.to_path_buf()),
)
}
(Some(_), Some(_)) => {
return Err(format!(
"{name}: --{name}-data-home and --{name}-destination are mutually exclusive"
)
.into());
}
(None, None) => {
return Err(format!(
"{name}: one of --{name}-data-home or --{name}-destination is required"
)
.into());
}
};
let target = targets
.iter_mut()
.find(|target| target.get("name").and_then(Value::as_str) == Some(name))
.ok_or_else(|| format!("report has no {name} target"))?;
let Some(storage) = storage else {
target
.as_object_mut()
.ok_or("report target must be an object")?
.insert(
"read_bench".to_string(),
json!({"status": "skipped", "reason": "storage inspection disabled"}),
);
continue;
};
let inspection = run_storage_inspection(
&args.fixture_generator,
&data_home,
destination.as_deref(),
storage,
)?;
let bench_dir = bench_root.join(name).join("read_bench");
let read_bench = run_read_bench(
&args.candidate_bin,
&data_home,
&bench_dir,
remote.read_bench.as_ref(),
&inspection,
)?;
let inspection_failed = inspection["status"] == "failed";
let bench_failed = read_bench["status"] == "failed";
let target = target
.as_object_mut()
.ok_or("report target must be an object")?;
target.insert("storage_inspection".to_string(), inspection.clone());
target.insert("read_bench".to_string(), read_bench);
if inspection_failed || bench_failed {
target.insert("status".to_string(), json!("failed"));
}
inspections.push(inspection);
}
targets.iter().any(|target| {
target["status"] == "failed"
|| target["storage_inspection"]["status"] == "failed"
|| target["read_bench"]["status"] == "failed"
})
};
let storage_thresholds = storage
.map(|storage| {
enforce_storage_thresholds(
storage,
inspections.first().unwrap_or(&Value::Null),
inspections.get(1).unwrap_or(&Value::Null),
)
})
.unwrap_or_default();
let threshold_failed = {
let thresholds = report
.get_mut("thresholds")
.and_then(Value::as_array_mut)
.ok_or("report has no thresholds array")?;
thresholds.retain(|threshold| !is_storage_threshold_entry(threshold));
thresholds.extend(storage_thresholds);
thresholds
.iter()
.any(|threshold| threshold["status"] == "failed")
};
let failed = target_failed || threshold_failed;
report["status"] = json!(if failed { "failed" } else { "ok" });
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
fs::write(&args.report, &text)?;
print!("{text}");
if failed {
std::process::exit(1);
}
Ok(())
}
fn is_storage_threshold_entry(threshold: &Value) -> bool {
let Some(name) = threshold.get("threshold").and_then(Value::as_str) else {
return false;
};
matches!(
name,
"max_candidate_total_file_size_regression_pct"
| "max_candidate_column_compressed_size_regression_pct"
| "max_candidate_column_uncompressed_size_regression_pct"
) || (threshold.get("target").is_some()
&& matches!(
name,
"min_files"
| "min_files_with_column"
| "max_total_file_size_bytes"
| "max_column_compressed_size_bytes"
| "max_column_uncompressed_size_bytes"
| "require_encodings"
| "forbid_encodings"
))
}
src/cmd/src/bin/query_regression_runner/remote/read_bench.rs
+363
View File
@@ -0,0 +1,363 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::Instant;
use regex::Regex;
use serde_json::{Value, json};
use crate::query_regression_runner::Result;
use crate::query_regression_runner::measure::median;
use crate::query_regression_runner::model::ReadBenchConfig;
#[derive(Clone, Debug)]
struct BenchTarget {
relative_path: String,
table_dir: String,
region_id: String,
path_type: String,
file_id: String,
}
pub(super) fn run_read_bench(
candidate_bin: &Path,
data_home: &Path,
bench_dir: &Path,
read_bench: Option<&ReadBenchConfig>,
inspection: &Value,
) -> Result<Value> {
let Some(read_bench) = read_bench.filter(|config| config.enabled) else {
return Ok(json!({ "status": "skipped", "reason": "read_bench disabled" }));
};
if !read_bench.parquetbench && !read_bench.scanbench {
return Ok(json!({ "status": "skipped", "reason": "parquetbench and scanbench disabled" }));
}
let report = inspection
.get("summary")
.filter(|summary| summary.is_object());
let files = report
.and_then(|report| report.get("files"))
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
let root = report
.and_then(|report| report.get("root"))
.and_then(Value::as_str)
.map(PathBuf::from)
.or_else(|| {
inspection
.get("root")
.and_then(Value::as_str)
.map(PathBuf::from)
})
.ok_or("storage inspection has no root")?;
let mut targets = files
.iter()
.filter(|file| {
file.get("relative_path")
.and_then(Value::as_str)
.is_some_and(|path| path.ends_with(".parquet"))
&& file
.get("columns")
.and_then(Value::as_array)
.is_some_and(|columns| !columns.is_empty())
})
.filter_map(|file| inspected_bench_target(data_home, &root, file).transpose())
.collect::<Result<Vec<_>>>()?;
if let Some(max_files) = read_bench.max_files {
targets.truncate(max_files);
}
if targets.is_empty() {
return Ok(
json!({ "status": "failed", "reason": "no inspected data SST files available for read_bench" }),
);
}
fs::create_dir_all(bench_dir)?;
let config_toml = bench_dir.join("bench.toml");
let scan_json = bench_dir.join("scan.json");
fs::write(
&config_toml,
format!(
"[storage]\ndata_home = \"{}\"\ntype = \"File\"\n\n[[region_engine]]\n[region_engine.mito]\n",
data_home.display()
),
)?;
fs::write(
&scan_json,
format!(
"{}\n",
serde_json::to_string_pretty(&json!({ "projection_names": read_bench.projection }))?
),
)?;
let mut parquet_runs = Vec::new();
if read_bench.parquetbench {
for target in &targets {
let command = vec![
candidate_bin.to_string_lossy().to_string(),
"datanode".to_string(),
"parquetbench".to_string(),
"--config".to_string(),
config_toml.to_string_lossy().to_string(),
"--region-id".to_string(),
target.region_id.clone(),
"--table-dir".to_string(),
target.table_dir.clone(),
"--file-id".to_string(),
target.file_id.clone(),
"--scan-config".to_string(),
scan_json.to_string_lossy().to_string(),
"--path-type".to_string(),
target.path_type.clone(),
"--iterations".to_string(),
read_bench.iterations.to_string(),
"--reader".to_string(),
read_bench.parquet_reader.clone(),
];
parquet_runs.push(run_bench_command(command, bench_target_value(target))?);
}
}
let mut scan_runs = Vec::new();
if read_bench.scanbench {
for (table_dir, region_id, path_type, files) in group_scan_paths(&targets) {
let command = vec![
candidate_bin.to_string_lossy().to_string(),
"datanode".to_string(),
"scanbench".to_string(),
"--config".to_string(),
config_toml.to_string_lossy().to_string(),
"--region-id".to_string(),
region_id.clone(),
"--table-dir".to_string(),
table_dir.clone(),
"--scan-config".to_string(),
scan_json.to_string_lossy().to_string(),
"--path-type".to_string(),
path_type.clone(),
"--scanner".to_string(),
read_bench.scan_scanner.clone(),
"--parallelism".to_string(),
read_bench.parallelism.to_string(),
"--iterations".to_string(),
read_bench.iterations.to_string(),
];
scan_runs.push(run_bench_command(command, json!({ "table_dir": table_dir, "region_id": region_id, "path_type": path_type, "files": files }))?);
}
}
let failed = parquet_runs
.iter()
.chain(&scan_runs)
.any(|run| run["status"] == "failed");
Ok(json!({
"status": if failed { "failed" } else { "ok" },
"config_path": config_toml,
"scan_config_path": scan_json,
"parquetbench": parquet_runs,
"scanbench": scan_runs,
"aggregate": {
"parquetbench_median_average_ms": bench_median(&parquet_runs),
"scanbench_median_average_ms": bench_median(&scan_runs),
},
}))
}
fn inspected_bench_target(
data_home: &Path,
root: &Path,
file: &Value,
) -> Result<Option<BenchTarget>> {
let relative_path = file
.get("relative_path")
.and_then(Value::as_str)
.unwrap_or_default();
if relative_path.is_empty() {
return Ok(None);
}
let root_suffix = root.strip_prefix(data_home).map_err(|_| {
format!(
"storage inspection root {} is not under datanode data home {}",
root.display(),
data_home.display()
)
})?;
let full_relative = root_suffix.join(relative_path);
let parts = full_relative
.components()
.map(|component| component.as_os_str().to_string_lossy().to_string())
.collect::<Vec<_>>();
if !parts.last().is_some_and(|part| part.ends_with(".parquet")) {
return Ok(None);
}
let Some((region_index, table_id, region_seq)) =
parts.iter().enumerate().find_map(|(index, part)| {
parse_region_dir(part).map(|(table_id, region_seq)| (index, table_id, region_seq))
})
else {
return Ok(None);
};
if region_index == 0 {
return Ok(None);
}
let file_index = parts.len() - 1;
let path_type = if region_index + 1 < file_index
&& matches!(parts[region_index + 1].as_str(), "data" | "metadata")
{
parts[region_index + 1].clone()
} else {
"bare".to_string()
};
if path_type == "metadata" {
return Ok(None);
}
let file_id = parts[file_index].trim_end_matches(".parquet").to_string();
Ok(Some(BenchTarget {
relative_path: full_relative.to_string_lossy().to_string(),
table_dir: format!("{}/", parts[..region_index].join("/")),
region_id: format!("{table_id}:{region_seq}"),
path_type,
file_id,
}))
}
fn parse_region_dir(part: &str) -> Option<(u64, u64)> {
let (table_id, region_seq) = part.split_once('_')?;
(region_seq.len() == 10
&& table_id.chars().all(|character| character.is_ascii_digit())
&& region_seq
.chars()
.all(|character| character.is_ascii_digit()))
.then(|| Some((table_id.parse().ok()?, region_seq.parse().ok()?)))?
}
fn bench_target_value(target: &BenchTarget) -> Value {
json!({
"relative_path": target.relative_path,
"table_dir": target.table_dir,
"region_id": target.region_id,
"path_type": target.path_type,
"file_id": target.file_id,
})
}
fn group_scan_paths(targets: &[BenchTarget]) -> Vec<(String, String, String, Vec<String>)> {
let mut groups: Vec<(String, String, String, Vec<String>)> = Vec::new();
for target in targets {
if let Some((_, _, _, paths)) =
groups
.iter_mut()
.find(|(table_dir, region_id, path_type, _)| {
table_dir == &target.table_dir
&& region_id == &target.region_id
&& path_type == &target.path_type
})
{
paths.push(target.relative_path.clone());
} else {
groups.push((
target.table_dir.clone(),
target.region_id.clone(),
target.path_type.clone(),
vec![target.relative_path.clone()],
));
}
}
groups
}
fn run_bench_command(command: Vec<String>, mut run: Value) -> Result<Value> {
let started = Instant::now();
let output = Command::new(&command[0]).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
let run = run.as_object_mut().ok_or("bench run must be an object")?;
run.extend([
("command".to_string(), json!(command)),
("returncode".to_string(), json!(output.status.code())),
("elapsed_seconds".to_string(), json!(elapsed_seconds)),
("stdout".to_string(), json!(stdout)),
("stderr".to_string(), json!(stderr)),
(
"status".to_string(),
json!(if output.status.success() {
"ok"
} else {
"failed"
}),
),
]);
if let Some(average_ms) = parse_average_duration(run["stdout"].as_str().unwrap_or_default()) {
run.insert("average_ms".to_string(), json!(average_ms));
}
Ok(Value::Object(run.clone()))
}
fn parse_average_duration(stdout: &str) -> Option<f64> {
Regex::new(r"(?i)Average duration[^0-9]*([0-9.]+)\s*ms")
.ok()?
.captures(stdout)?
.get(1)?
.as_str()
.parse()
.ok()
}
fn bench_median(runs: &[Value]) -> Option<f64> {
let values = runs
.iter()
.filter_map(|run| run.get("average_ms").and_then(Value::as_f64))
.collect::<Vec<_>>();
(!values.is_empty()).then(|| median(&values))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parses_average_duration_and_groups_scan_paths() {
assert_eq!(
parse_average_duration("work\nAverage duration: 12.5 ms\n"),
Some(12.5)
);
assert_eq!(parse_average_duration("completed"), None);
let targets = vec![
BenchTarget {
relative_path: "data/a.parquet".to_string(),
table_dir: "data/".to_string(),
region_id: "1:2".to_string(),
path_type: "data".to_string(),
file_id: "a".to_string(),
},
BenchTarget {
relative_path: "data/b.parquet".to_string(),
table_dir: "data/".to_string(),
region_id: "1:2".to_string(),
path_type: "data".to_string(),
file_id: "b".to_string(),
},
];
assert_eq!(
group_scan_paths(&targets),
vec![(
"data/".to_string(),
"1:2".to_string(),
"data".to_string(),
vec!["data/a.parquet".to_string(), "data/b.parquet".to_string()],
)]
);
}
}
src/cmd/src/bin/query_regression_runner/remote/storage.rs
+241
View File
@@ -0,0 +1,241 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::path::Path;
use std::process::Command;
use std::time::Instant;
use serde_json::{Value, json};
use crate::query_regression_runner::Result;
use crate::query_regression_runner::model::StorageConfig;
use crate::query_regression_runner::sql::{value_f64, value_u64};
pub(super) fn run_storage_inspection(
generator: &Path,
data_home: &Path,
destination: Option<&Path>,
storage: &StorageConfig,
) -> Result<Value> {
let mut command = vec![
generator.to_string_lossy().to_string(),
"inspect-footer".to_string(),
"--column".to_string(),
storage.column.clone(),
];
let root = if let Some(destination) = destination {
command.push("--destination".to_string());
command.push(destination.to_string_lossy().to_string());
data_home.to_path_buf()
} else {
let root = storage
.root_suffix
.as_deref()
.map_or_else(|| data_home.to_path_buf(), |suffix| data_home.join(suffix));
command.push("--root".to_string());
command.push(root.to_string_lossy().to_string());
root
};
if storage.include_metadata_files {
command.push("--include-metadata-files".to_string());
}
let started = Instant::now();
let output = Command::new(generator).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
if !output.status.success() {
return Err(format!(
"storage inspector failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}",
output.status.code(), stderr
)
.into());
}
let mut result = json!({
"status": "ok",
"command": command,
"root": root,
"returncode": output.status.code(),
"elapsed_seconds": elapsed_seconds,
"stdout": stdout,
"stderr": stderr,
});
match serde_json::from_str(result["stdout"].as_str().unwrap_or_default()) {
Ok(summary) => result["summary"] = summary,
Err(_) => {
result["summary_parse_error"] = result["stdout"].clone();
result["status"] = json!("failed");
}
}
Ok(result)
}
fn storage_summary(inspection: &Value) -> Value {
inspection
.get("summary")
.and_then(|summary| summary.get("summary"))
.filter(|summary| summary.is_object())
.cloned()
.unwrap_or_else(|| json!({}))
}
pub(super) fn enforce_storage_thresholds(
storage: &StorageConfig,
base_inspection: &Value,
candidate_inspection: &Value,
) -> Vec<Value> {
let base = storage_summary(base_inspection);
let candidate = storage_summary(candidate_inspection);
let targets = [("base", &base), ("candidate", &candidate)];
let mut results = Vec::new();
for (threshold, field, limit) in [
("min_files", "file_count", storage.min_files),
(
"min_files_with_column",
"files_with_column",
storage.min_files_with_column,
),
] {
for (target, summary) in &targets {
let actual = summary.get(field).cloned().unwrap_or(Value::Null);
let ok = value_u64(Some(&actual)).is_some_and(|actual| actual >= limit);
results.push(json!({ "target": target, "threshold": threshold, "status": if ok { "passed" } else { "failed" }, "actual": actual, "limit": limit }));
}
}
for (threshold, field, limit) in [
(
"max_total_file_size_bytes",
"total_file_size",
storage.max_total_file_size_bytes,
),
(
"max_column_compressed_size_bytes",
"column_compressed_size",
storage.max_column_compressed_size_bytes,
),
(
"max_column_uncompressed_size_bytes",
"column_uncompressed_size",
storage.max_column_uncompressed_size_bytes,
),
] {
let Some(limit) = limit else { continue };
for (target, summary) in &targets {
let actual = summary.get(field).cloned().unwrap_or(Value::Null);
let ok = value_f64(Some(&actual)).is_some_and(|actual| actual <= limit as f64);
results.push(json!({ "target": target, "threshold": threshold, "status": if ok { "passed" } else { "failed" }, "actual": actual, "limit": limit }));
}
}
for (target, summary) in &targets {
let encodings = summary
.get("unique_encodings")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
for encoding in &storage.require_encodings {
results.push(json!({ "target": target, "threshold": "require_encodings", "encoding": encoding, "status": if encodings.iter().any(|value| value.as_str() == Some(encoding)) { "passed" } else { "failed" } }));
}
for encoding in &storage.forbid_encodings {
results.push(json!({ "target": target, "threshold": "forbid_encodings", "encoding": encoding, "status": if encodings.iter().all(|value| value.as_str() != Some(encoding)) { "passed" } else { "failed" } }));
}
}
for (threshold, field, limit) in [
(
"max_candidate_total_file_size_regression_pct",
"total_file_size",
storage.max_candidate_total_file_size_regression_pct,
),
(
"max_candidate_column_compressed_size_regression_pct",
"column_compressed_size",
storage.max_candidate_column_compressed_size_regression_pct,
),
(
"max_candidate_column_uncompressed_size_regression_pct",
"column_uncompressed_size",
storage.max_candidate_column_uncompressed_size_regression_pct,
),
] {
let Some(limit) = limit else { continue };
let base_value = base.get(field).cloned().unwrap_or(Value::Null);
let candidate_value = candidate.get(field).cloned().unwrap_or(Value::Null);
let (Some(base_number), Some(candidate_number)) = (
value_f64(Some(&base_value)).filter(|value| *value != 0.0),
value_f64(Some(&candidate_value)),
) else {
results.push(json!({ "threshold": threshold, "status": "failed", "reason": "missing or zero base/candidate value", "base": base_value, "candidate": candidate_value }));
continue;
};
let actual = (candidate_number - base_number) / base_number * 100.0;
results.push(json!({ "threshold": threshold, "status": if actual <= limit { "passed" } else { "failed" }, "actual_pct": actual, "limit_pct": limit, "base": base_value, "candidate": candidate_value }));
}
results
}
#[cfg(test)]
mod tests {
use super::super::is_storage_threshold_entry;
use super::*;
#[test]
fn enforces_storage_thresholds_for_each_target_and_regression() {
let storage: StorageConfig = serde_json::from_value(json!({
"inspect": true,
"column": "value",
"root_suffix": null,
"include_metadata_files": false,
"min_files": 2,
"min_files_with_column": 1,
"require_encodings": ["PLAIN"],
"forbid_encodings": ["DELTA"],
"max_total_file_size_bytes": 105,
"max_column_compressed_size_bytes": null,
"max_column_uncompressed_size_bytes": null,
"max_candidate_total_file_size_regression_pct": 5.0,
"max_candidate_column_compressed_size_regression_pct": null,
"max_candidate_column_uncompressed_size_regression_pct": null
}))
.unwrap();
let base = json!({"summary": {"summary": {"file_count": 2, "files_with_column": 1, "total_file_size": 100, "unique_encodings": ["PLAIN"]}}});
let candidate = json!({"summary": {"summary": {"file_count": 2, "files_with_column": 1, "total_file_size": 110, "unique_encodings": ["PLAIN"]}}});
let results = enforce_storage_thresholds(&storage, &base, &candidate);
assert!(
results
.iter()
.any(|result| result["threshold"] == "max_total_file_size_bytes"
&& result["target"] == "candidate"
&& result["status"] == "failed")
);
assert!(results.iter().any(|result| result["threshold"]
== "max_candidate_total_file_size_regression_pct"
&& result["actual_pct"] == 10.0
&& result["status"] == "failed"));
assert!(results.iter().any(
|result| result["threshold"] == "require_encodings" && result["status"] == "passed"
));
}
#[test]
fn identifies_prior_storage_threshold_entries() {
assert!(is_storage_threshold_entry(
&json!({"target": "base", "threshold": "min_files"})
));
assert!(is_storage_threshold_entry(&json!({
"threshold": "max_candidate_total_file_size_regression_pct"
})));
assert!(!is_storage_threshold_entry(&json!({
"threshold": "max_candidate_latency_regression_pct"
})));
}
}
src/cmd/src/bin/query_regression_runner/sql.rs
+298
View File
@@ -0,0 +1,298 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::time::Instant;
use reqwest::Client;
use serde_json::{Value, json};
use crate::query_regression_runner::Result;
pub(super) fn sql_string(value: &str) -> String {
format!("'{}'", value.replace('\'', "''"))
}
pub(super) fn extract_rows(body: &Value) -> Vec<Value> {
fn visit(body: &Value, rows: &mut Vec<Value>) {
match body {
Value::Object(object) => {
for key in ["data", "rows", "records", "output"] {
let Some(value) = object.get(key) else {
continue;
};
if matches!(key, "data" | "rows") && value.is_array() {
rows.extend(value.as_array().unwrap().iter().cloned());
} else {
visit(value, rows);
}
}
}
Value::Array(values) => {
if !values.is_empty()
&& values
.iter()
.all(|value| !value.is_object() && !value.is_array())
{
rows.push(body.clone());
} else {
for value in values {
visit(value, rows);
}
}
}
_ => {}
}
}
let mut rows = Vec::new();
visit(body, &mut rows);
rows
}
pub(super) fn row_value<'a>(row: &'a Value, index: usize, name: &str) -> Option<&'a Value> {
match row {
Value::Object(values) => [
name.to_string(),
name.to_ascii_uppercase(),
name.to_ascii_lowercase(),
]
.into_iter()
.find_map(|key| values.get(&key)),
Value::Array(values) => values.get(index),
_ => Some(row),
}
}
pub(super) fn row_u64(row: &Value, index: usize, name: &str) -> Result<u64> {
let value =
row_value(row, index, name).ok_or_else(|| format!("missing {name} in row {row}"))?;
value
.as_u64()
.or_else(|| value.as_str().and_then(|value| value.parse().ok()))
.ok_or_else(|| format!("invalid {name} in row {row}").into())
}
pub(super) fn value_text(value: &Value) -> String {
match value {
Value::String(value) => value.clone(),
Value::Bool(value) => value.to_string(),
Value::Number(value) => value.to_string(),
Value::Null => "None".to_string(),
_ => value.to_string(),
}
}
pub(super) fn extract_count_value(result: &Value) -> Option<u64> {
let row = result
.get("response")?
.get("data")?
.as_array()?
.first()?
.as_object()?;
row.iter()
.find(|(key, _)| {
let key = key.to_ascii_lowercase();
key == "count(*)" || key.starts_with("count(")
})
.and_then(|(_, value)| value_u64(Some(value)))
}
pub(super) fn value_u64(value: Option<&Value>) -> Option<u64> {
value?.as_u64().or_else(|| value?.as_str()?.parse().ok())
}
pub(super) fn value_f64(value: Option<&Value>) -> Option<f64> {
value?.as_f64().or_else(|| value?.as_str()?.parse().ok())
}
pub(super) async fn http_post_sql(client: &Client, port: u16, sql: &str, db: &str) -> Value {
let mut sample = post_form(
client,
format!("http://127.0.0.1:{port}/v1/sql"),
&[("sql", sql), ("db", db), ("format", "json")],
)
.await;
sample
.as_object_mut()
.expect("HTTP samples are objects")
.insert("sql".to_string(), Value::String(sql.to_string()));
sample
}
/// Posts a Prometheus HTTP API range query (`/v1/prometheus/api/v1/query_range`)
/// and measures the full request-to-body latency. This exercises the Prometheus
/// JSON response building path (`PrometheusJsonResponse::record_batches_to_data`)
/// that `/v1/sql` (including `TQL ANALYZE`) does not go through.
pub(super) async fn http_post_prom_range_query(
client: &Client,
port: u16,
query: &str,
start: Option<&str>,
end: Option<&str>,
step: Option<&str>,
db: &str,
) -> Value {
let mut sample = post_form(
client,
prom_range_query_url(port, db),
&[
("query", query),
("start", start.unwrap_or_default()),
("end", end.unwrap_or_default()),
("step", step.unwrap_or_default()),
],
)
.await;
sample
.as_object_mut()
.expect("HTTP samples are objects")
.insert("query".to_string(), Value::String(query.to_string()));
sample
}
fn prom_range_query_url(port: u16, db: &str) -> String {
let mut url = reqwest::Url::parse(&format!(
"http://127.0.0.1:{port}/v1/prometheus/api/v1/query_range"
))
.expect("fixed Prometheus range query URL must be valid");
url.query_pairs_mut().append_pair("db", db);
url.into()
}
async fn post_form(client: &Client, url: String, form: &[(&str, &str)]) -> Value {
let started = Instant::now();
let request = client.post(url).form(form);
match request.send().await {
Ok(response) => {
let status = response.status().as_u16();
match response.text().await {
Ok(raw) => {
let body = serde_json::from_str(&raw).unwrap_or_else(|_| json!({"raw": raw}));
let ok = status < 400 && !response_has_error(&body);
let mut sample = json!({
"ok": ok,
"status": status,
"latency_ms": started.elapsed().as_secs_f64() * 1000.0,
"response": body,
});
if status >= 400 {
sample
.as_object_mut()
.expect("HTTP samples are objects")
.insert("error".to_string(), Value::String(format!("HTTP {status}")));
}
sample
}
Err(error) => json!({
"ok": false,
"status": status,
"latency_ms": started.elapsed().as_secs_f64() * 1000.0,
"error": error.to_string(),
}),
}
}
Err(error) => json!({
"ok": false,
"status": Value::Null,
"latency_ms": started.elapsed().as_secs_f64() * 1000.0,
"error": error.to_string(),
}),
}
}
fn response_has_error(body: &Value) -> bool {
let Some(body) = body.as_object() else {
return false;
};
["error", "err_msg", "error_msg"]
.into_iter()
.any(|key| body.get(key).is_some_and(is_truthy))
|| body
.get("error_code")
.is_some_and(|value| !is_success_code(value))
|| (!body.contains_key("output")
&& body
.get("code")
.is_some_and(|value| !is_success_code(value)))
}
fn is_truthy(value: &Value) -> bool {
match value {
Value::Null => false,
Value::Bool(value) => *value,
Value::Number(value) => value.as_f64().is_none_or(|value| value != 0.0),
Value::String(value) => !value.is_empty(),
Value::Array(value) => !value.is_empty(),
Value::Object(value) => !value.is_empty(),
}
}
fn is_success_code(value: &Value) -> bool {
let code = match value {
Value::String(value) => value.clone(),
Value::Bool(value) => value.to_string(),
Value::Number(value) => value.to_string(),
Value::Null => "None".to_string(),
_ => value.to_string(),
};
matches!(code.to_lowercase().as_str(), "" | "0" | "success")
}
pub(super) fn sql_ident(name: &str) -> String {
format!("\"{}\"", name.replace('"', "\"\""))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn prom_range_query_url_places_database_in_query_parameters() {
let url = reqwest::Url::parse(&prom_range_query_url(4000, "catalog-schema name"))
.expect("generated URL must be valid");
assert_eq!(url.path(), "/v1/prometheus/api/v1/query_range");
assert_eq!(
url.query_pairs().collect::<Vec<_>>(),
vec![("db".into(), "catalog-schema name".into())]
);
}
#[test]
fn top_level_errors_do_not_inspect_rows() {
assert!(response_has_error(&json!({"error_code": 7})));
assert!(response_has_error(&json!({"code": "bad"})));
assert!(!response_has_error(
&json!({"output": [{"code": 7, "error": "row value"}]})
));
assert!(!response_has_error(
&json!({"error_code": "success", "output": []})
));
}
#[test]
fn extracts_remote_write_count_from_data_map() {
assert_eq!(
extract_count_value(&json!({"response": {"data": [{"COUNT(*)": "12"}]}})),
Some(12)
);
assert_eq!(
extract_count_value(&json!({"response": {"data": [{"count(value)": 7}]}})),
Some(7)
);
assert_eq!(
extract_count_value(&json!({"response": {"data": [[]]}})),
None
);
}
}
src/cmd/src/datanode.rs
+46
View File
@@ -15,10 +15,20 @@
pub mod builder;
#[allow(clippy::print_stdout)]
pub(crate) mod objbench;
#[cfg(feature = "dev-tools")]
#[allow(clippy::print_stdout)]
mod parquet_meta;
#[cfg(feature = "dev-tools")]
#[allow(clippy::print_stdout)]
mod parquet_rewrite;
#[allow(clippy::print_stdout)]
pub mod parquetbench;
#[allow(clippy::print_stdout)]
pub mod scanbench;
#[cfg(feature = "dev-tools")]
#[allow(clippy::print_stdout)]
mod sst_replace;
mod tool_util;
use std::path::Path;
use std::time::Duration;
@@ -39,8 +49,14 @@ use tracing_appender::non_blocking::WorkerGuard;
use crate::App;
use crate::datanode::builder::InstanceBuilder;
use crate::datanode::objbench::ObjbenchCommand;
#[cfg(feature = "dev-tools")]
use crate::datanode::parquet_meta::ParquetMetaCommand;
#[cfg(feature = "dev-tools")]
use crate::datanode::parquet_rewrite::ParquetRewriteCommand;
use crate::datanode::parquetbench::ParquetbenchCommand;
use crate::datanode::scanbench::ScanbenchCommand;
#[cfg(feature = "dev-tools")]
use crate::datanode::sst_replace::SstReplaceCommand;
use crate::error::{
LoadLayeredConfigSnafu, MissingConfigSnafu, Result, ShutdownDatanodeSnafu, StartDatanodeSnafu,
};
@@ -114,6 +130,12 @@ impl Command {
// Bench commands are standalone utilities and don't need to load DatanodeOptions.
SubCommand::Objbench(_) | SubCommand::Scanbench(_) => Self::default_bench_options(),
SubCommand::Parquetbench(_) => Self::default_bench_options(),
#[cfg(feature = "dev-tools")]
SubCommand::ParquetMeta(_) => Self::default_bench_options(),
#[cfg(feature = "dev-tools")]
SubCommand::ParquetRewrite(_) => Self::default_bench_options(),
#[cfg(feature = "dev-tools")]
SubCommand::SstReplace(_) => Self::default_bench_options(),
}
}
@@ -139,6 +161,15 @@ pub enum SubCommand {
Scanbench(ScanbenchCommand),
/// Benchmark scanning a single parquet SST.
Parquetbench(ParquetbenchCommand),
/// Display metadata of a parquet file.
#[cfg(feature = "dev-tools")]
ParquetMeta(ParquetMetaCommand),
/// Rewrite a parquet file with different writer properties.
#[cfg(feature = "dev-tools")]
ParquetRewrite(ParquetRewriteCommand),
/// Replace a Mito region SST and update its manifest metadata.
#[cfg(feature = "dev-tools")]
SstReplace(SstReplaceCommand),
}
impl SubCommand {
@@ -160,6 +191,21 @@ impl SubCommand {
cmd.run().await?;
std::process::exit(0);
}
#[cfg(feature = "dev-tools")]
SubCommand::ParquetMeta(cmd) => {
cmd.run().await?;
std::process::exit(0);
}
#[cfg(feature = "dev-tools")]
SubCommand::ParquetRewrite(cmd) => {
cmd.run().await?;
std::process::exit(0);
}
#[cfg(feature = "dev-tools")]
SubCommand::SstReplace(cmd) => {
cmd.run().await?;
std::process::exit(0);
}
}
}
}
src/cmd/src/datanode/builder.rs
+3 -3
View File
@@ -61,9 +61,6 @@ impl InstanceBuilder {
}
async fn init(opts: &mut DatanodeOptions, plugins: &mut Plugins) -> Result<Vec<WorkerGuard>> {
common_runtime::init_global_runtimes(&opts.runtime);
common_runtime::init_datanode_runtimes(&opts.runtime);
let dn_opts = &mut opts.component;
let guard = common_telemetry::init_global_logging(
APP_NAME,
@@ -73,6 +70,9 @@ impl InstanceBuilder {
None,
);
common_runtime::init_global_runtimes(&opts.runtime);
common_runtime::init_datanode_runtimes(&opts.runtime);
crate::options::flush_dropped_plugin_warnings();
log_versions(verbose_version(), short_version(), APP_NAME);
maybe_activate_heap_profile(&dn_opts.memory);
src/cmd/src/datanode/objbench.rs
+8 -99
View File
@@ -18,8 +18,6 @@ use std::time::Instant;
use clap::Parser;
use colored::Colorize;
use datanode::config::RegionEngineConfig;
use datanode::store;
use futures::stream;
use mito2::access_layer::{
AccessLayer, AccessLayerRef, Metrics, OperationType, SstWriteRequest, WriteType,
@@ -32,19 +30,20 @@ use mito2::sst::file::{FileHandle, FileMeta};
use mito2::sst::file_purger::{FilePurger, FilePurgerRef};
use mito2::sst::index::intermediate::IntermediateManager;
use mito2::sst::index::puffin_manager::PuffinManagerFactory;
use mito2::sst::parquet::WriteOptions;
use mito2::sst::parquet::reader::ParquetReaderBuilder;
use mito2::sst::parquet::{PARQUET_METADATA_KEY, WriteOptions};
use mito2::worker::write_cache_from_config;
use object_store::ObjectStore;
use parquet::file::metadata::{FooterTail, KeyValue};
use parquet::file::metadata::FooterTail;
use regex::Regex;
use snafu::OptionExt;
use store_api::metadata::{RegionMetadata, RegionMetadataRef};
use store_api::path_utils::region_name;
use store_api::region_request::PathType;
use store_api::storage::FileId;
use crate::datanode::{StorageConfig, StorageConfigWrapper};
use crate::datanode::tool_util::{
build_object_store, extract_region_metadata, max_row_group_uncompressed_size, parse_config,
};
use crate::error;
/// Object storage benchmark command
@@ -67,46 +66,6 @@ pub struct ObjbenchCommand {
pub pprof_file: Option<PathBuf>,
}
pub(crate) fn parse_config(
config_path: &PathBuf,
) -> error::Result<(
StorageConfig,
MitoConfig,
common_wal::config::DatanodeWalConfig,
)> {
let cfg_str = std::fs::read_to_string(config_path).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("failed to read config {}: {e}", config_path.display()),
}
.build()
})?;
let store_cfg: StorageConfigWrapper = toml::from_str(&cfg_str).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("failed to parse config {}: {e}", config_path.display()),
}
.build()
})?;
let wal_config = store_cfg.wal;
let storage_config = store_cfg.storage;
let mito_engine_config = store_cfg
.region_engine
.into_iter()
.filter_map(|c| {
if let RegionEngineConfig::Mito(mito) = c {
Some(mito)
} else {
None
}
})
.next()
.with_context(|| error::IllegalConfigSnafu {
msg: format!("Engine config not found in {:?}", config_path),
})?;
Ok((storage_config, mito_engine_config, wal_config))
}
impl ObjbenchCommand {
pub async fn run(&self) -> error::Result<()> {
if self.verbose {
@@ -154,17 +113,7 @@ impl ObjbenchCommand {
let region_meta = extract_region_metadata(&self.source, &parquet_meta)?;
let num_rows = parquet_meta.file_metadata().num_rows() as u64;
let num_row_groups = parquet_meta.num_row_groups() as u64;
let max_row_group_uncompressed_size: u64 = parquet_meta
.row_groups()
.iter()
.map(|rg| {
rg.columns()
.iter()
.map(|c| c.uncompressed_size() as u64)
.sum::<u64>()
})
.max()
.unwrap_or(0);
let max_row_group_uncompressed_size = max_row_group_uncompressed_size(&parquet_meta);
println!(
"{} Metadata loaded - rows: {}, size: {} bytes",
@@ -485,47 +434,6 @@ fn parse_file_dir_components(path: &str) -> error::Result<FileDirComponents> {
})
}
pub(crate) fn extract_region_metadata(
file_path: &str,
meta: &parquet::file::metadata::ParquetMetaData,
) -> error::Result<RegionMetadataRef> {
let kvs: Option<&Vec<KeyValue>> = meta.file_metadata().key_value_metadata();
let Some(kvs) = kvs else {
return Err(error::IllegalConfigSnafu {
msg: format!("{file_path}: missing parquet key_value metadata"),
}
.build());
};
let json = kvs
.iter()
.find(|kv| kv.key == PARQUET_METADATA_KEY)
.and_then(|kv| kv.value.as_ref())
.ok_or_else(|| {
error::IllegalConfigSnafu {
msg: format!("{file_path}: key {PARQUET_METADATA_KEY} not found or empty"),
}
.build()
})?;
let region: RegionMetadata = RegionMetadata::from_json(json).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region metadata json: {e}"),
}
.build()
})?;
Ok(Arc::new(region))
}
pub(crate) async fn build_object_store(sc: &StorageConfig) -> error::Result<ObjectStore> {
store::new_object_store(sc.store.clone(), &sc.data_home)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("Failed to build object store: {e:?}"),
}
.build()
})
}
async fn build_access_layer_simple(
components: &FileDirComponents,
object_store: ObjectStore,
@@ -666,7 +574,8 @@ mod tests {
use common_base::readable_size::ReadableSize;
use store_api::region_request::PathType;
use crate::datanode::objbench::{parse_config, parse_file_dir_components};
use crate::datanode::objbench::parse_file_dir_components;
use crate::datanode::tool_util::parse_config;
#[test]
fn test_parse_dir() {
src/cmd/src/datanode/parquet_meta.rs
+394
View File
@@ -0,0 +1,394 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use clap::{Parser, ValueEnum};
use parquet::file::metadata::ParquetMetaData;
use parquet::file::page_index::offset_index::PageLocation;
use serde::Serialize;
use snafu::ResultExt;
use crate::datanode::tool_util::{compression_name, load_local_parquet_metadata};
use crate::error;
/// Display parquet file metadata.
#[derive(Debug, Parser)]
pub struct ParquetMetaCommand {
/// Path to input parquet file.
#[clap(long, value_name = "FILE")]
input: PathBuf,
/// Output format.
#[clap(long, value_enum, default_value = "text")]
format: MetaOutputFormat,
}
#[derive(Debug, Clone, Copy, ValueEnum)]
enum MetaOutputFormat {
Text,
Json,
}
#[derive(Debug, Serialize)]
struct FileMetaView {
input: String,
num_rows: i64,
num_row_groups: usize,
num_columns: usize,
key_value_metadata: BTreeMap<String, Option<String>>,
row_groups: Vec<RowGroupMetaView>,
}
#[derive(Debug, Serialize)]
struct RowGroupMetaView {
index: usize,
num_rows: i64,
uncompressed_size: i64,
compressed_size: i64,
compression_ratio: Option<f64>,
data_pages: Option<usize>,
dictionary_page_bytes: i64,
columns: Vec<ColumnChunkMetaView>,
}
#[derive(Debug, Serialize)]
struct ColumnChunkMetaView {
index: usize,
path: String,
physical_type: String,
encodings: Vec<String>,
compression: String,
num_values: i64,
uncompressed_size: i64,
compressed_size: i64,
compression_ratio: Option<f64>,
data_page_offset: i64,
dictionary_page_offset: Option<i64>,
dictionary_page_bytes: Option<i64>,
data_pages: Option<usize>,
max_data_page_row_count: Option<i64>,
has_statistics: bool,
column_index_offset: Option<i64>,
column_index_length: Option<i32>,
offset_index_offset: Option<i64>,
offset_index_length: Option<i32>,
bloom_filter_offset: Option<i64>,
bloom_filter_length: Option<i32>,
}
impl ParquetMetaCommand {
pub async fn run(&self) -> error::Result<()> {
let metadata = load_local_parquet_metadata(&self.input)?;
let view = build_file_meta_view(&self.input, &metadata);
match self.format {
MetaOutputFormat::Text => print_meta_text(&view),
MetaOutputFormat::Json => {
let json = serde_json::to_string_pretty(&view).context(error::SerdeJsonSnafu)?;
println!("{json}");
}
}
Ok(())
}
}
fn build_file_meta_view(path: &Path, metadata: &ParquetMetaData) -> FileMetaView {
let key_value_metadata = metadata
.file_metadata()
.key_value_metadata()
.into_iter()
.flatten()
.map(|kv| (kv.key.clone(), kv.value.clone()))
.collect();
let offset_index = metadata.offset_index();
let row_groups = metadata
.row_groups()
.iter()
.enumerate()
.map(|(row_group_idx, row_group)| {
let columns: Vec<_> = row_group
.columns()
.iter()
.enumerate()
.map(|(column_idx, column)| {
let page_locations = offset_index
.and_then(|index| index.get(row_group_idx))
.and_then(|columns| columns.get(column_idx))
.map(|index| index.page_locations());
let data_pages = page_locations.map(|locations| locations.len());
let max_data_page_row_count = page_locations.and_then(|locations| {
max_data_page_row_count(locations, row_group.num_rows())
});
let dictionary_page_bytes = dictionary_page_bytes(
column.dictionary_page_offset(),
column.data_page_offset(),
);
ColumnChunkMetaView {
index: column_idx,
path: column.column_path().string(),
physical_type: format!("{:?}", column.column_type()),
encodings: column.encodings().map(|enc| format!("{enc:?}")).collect(),
compression: compression_name(column.compression()).to_string(),
num_values: column.num_values(),
uncompressed_size: column.uncompressed_size(),
compressed_size: column.compressed_size(),
compression_ratio: compression_ratio(
column.uncompressed_size(),
column.compressed_size(),
),
data_page_offset: column.data_page_offset(),
dictionary_page_offset: column.dictionary_page_offset(),
dictionary_page_bytes,
data_pages,
max_data_page_row_count,
has_statistics: column.statistics().is_some(),
column_index_offset: column.column_index_offset(),
column_index_length: column.column_index_length(),
offset_index_offset: column.offset_index_offset(),
offset_index_length: column.offset_index_length(),
bloom_filter_offset: column.bloom_filter_offset(),
bloom_filter_length: column.bloom_filter_length(),
}
})
.collect();
let data_pages = if columns.iter().all(|column| column.data_pages.is_some()) {
Some(
columns
.iter()
.map(|column| column.data_pages.unwrap_or_default())
.sum(),
)
} else {
None
};
let dictionary_page_bytes = columns
.iter()
.map(|column| column.dictionary_page_bytes.unwrap_or_default())
.sum();
RowGroupMetaView {
index: row_group_idx,
num_rows: row_group.num_rows(),
uncompressed_size: row_group.total_byte_size(),
compressed_size: row_group.compressed_size(),
compression_ratio: compression_ratio(
row_group.total_byte_size(),
row_group.compressed_size(),
),
data_pages,
dictionary_page_bytes,
columns,
}
})
.collect();
FileMetaView {
input: path.display().to_string(),
num_rows: metadata.file_metadata().num_rows(),
num_row_groups: metadata.num_row_groups(),
num_columns: metadata.file_metadata().schema_descr().num_columns(),
key_value_metadata,
row_groups,
}
}
fn print_meta_text(view: &FileMetaView) {
println!("file: {}", view.input);
println!("rows: {}", view.num_rows);
println!("row_groups: {}", view.num_row_groups);
println!("columns: {}", view.num_columns);
println!("key_value_metadata: {}", view.key_value_metadata.len());
for row_group in &view.row_groups {
println!(
"row_group[{}]: rows={}, uncompressed_size={}, compressed_size={}, compression_ratio={}, data_pages={}, dictionary_page_bytes={}",
row_group.index,
row_group.num_rows,
row_group.uncompressed_size,
row_group.compressed_size,
format_ratio(row_group.compression_ratio),
format_optional_usize(row_group.data_pages),
row_group.dictionary_page_bytes,
);
for column in &row_group.columns {
println!(
" column[{}] path={}: type={}, compression={}, encodings=[{}], values={}, uncompressed_size={}, compressed_size={}, compression_ratio={}, data_pages={}, max_data_page_row_count={}, dictionary_page_offset={}, dictionary_page_bytes={}, data_page_offset={}, statistics={}, column_index={}/{}, offset_index={}/{}, bloom_filter={}/{}",
column.index,
column.path,
column.physical_type,
column.compression,
column.encodings.join(","),
column.num_values,
column.uncompressed_size,
column.compressed_size,
format_ratio(column.compression_ratio),
format_optional_usize(column.data_pages),
format_optional_i64(column.max_data_page_row_count),
format_optional_i64(column.dictionary_page_offset),
column
.dictionary_page_bytes
.map(|value| value.to_string())
.unwrap_or_else(|| "none".to_string()),
column.data_page_offset,
column.has_statistics,
format_optional_i64(column.column_index_offset),
format_optional_i32(column.column_index_length),
format_optional_i64(column.offset_index_offset),
format_optional_i32(column.offset_index_length),
format_optional_i64(column.bloom_filter_offset),
format_optional_i32(column.bloom_filter_length),
);
}
}
}
fn max_data_page_row_count(
page_locations: &[PageLocation],
row_group_row_count: i64,
) -> Option<i64> {
if page_locations.is_empty() {
return Some(0);
}
if page_locations.first()?.first_row_index != 0 {
return None;
}
let mut max_row_count = 0;
for locations in page_locations.windows(2) {
let row_count = locations[1].first_row_index - locations[0].first_row_index;
if row_count < 0 {
return None;
}
max_row_count = max_row_count.max(row_count);
}
let last_page_row_count = row_group_row_count - page_locations.last()?.first_row_index;
if last_page_row_count < 0 {
return None;
}
Some(max_row_count.max(last_page_row_count))
}
fn dictionary_page_bytes(
dictionary_page_offset: Option<i64>,
data_page_offset: i64,
) -> Option<i64> {
dictionary_page_offset.map(|offset| data_page_offset.saturating_sub(offset))
}
fn compression_ratio(uncompressed: i64, compressed: i64) -> Option<f64> {
(uncompressed > 0).then(|| compressed as f64 / uncompressed as f64)
}
fn format_ratio(ratio: Option<f64>) -> String {
ratio
.map(|value| format!("{value:.4}"))
.unwrap_or_else(|| "unknown".to_string())
}
fn format_optional_usize(value: Option<usize>) -> String {
value
.map(|value| value.to_string())
.unwrap_or_else(|| "unknown".to_string())
}
fn format_optional_i64(value: Option<i64>) -> String {
value
.map(|value| value.to_string())
.unwrap_or_else(|| "none".to_string())
}
fn format_optional_i32(value: Option<i32>) -> String {
value
.map(|value| value.to_string())
.unwrap_or_else(|| "none".to_string())
}
#[cfg(test)]
mod tests {
use std::fs::File;
use std::sync::Arc;
use datatypes::arrow::array::{Int32Array, RecordBatch, StringArray};
use datatypes::arrow::datatypes::{DataType, Field, Schema};
use parquet::arrow::ArrowWriter;
use parquet::file::metadata::KeyValue;
use parquet::file::properties::WriterProperties;
use tempfile::tempdir;
use super::*;
#[test]
fn test_meta_view_unknown_page_count() {
let dir = tempdir().unwrap();
let path = dir.path().join("input.parquet");
write_test_parquet(&path, None);
let metadata = load_local_parquet_metadata(&path).unwrap();
let view = build_file_meta_view(&path, &metadata);
assert_eq!(view.num_row_groups, 1);
assert!(view.row_groups[0].uncompressed_size > 0);
assert!(view.row_groups[0].compressed_size > 0);
assert!(view.row_groups[0].compression_ratio.is_some());
assert_eq!(view.row_groups[0].columns.len(), 2);
}
#[test]
fn test_meta_view_reports_max_data_page_row_count() {
let dir = tempdir().unwrap();
let path = dir.path().join("input.parquet");
write_test_parquet(&path, Some(2));
let metadata = load_local_parquet_metadata(&path).unwrap();
let view = build_file_meta_view(&path, &metadata);
for column in &view.row_groups[0].columns {
assert_eq!(Some(2), column.max_data_page_row_count);
}
}
fn write_test_parquet(path: &Path, data_page_row_count_limit: Option<usize>) {
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("host", DataType::Utf8, false),
]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(Int32Array::from(vec![1, 2, 3, 4])),
Arc::new(StringArray::from(vec!["a", "a", "b", "b"])),
],
)
.unwrap();
let mut props =
WriterProperties::builder().set_key_value_metadata(Some(vec![KeyValue::new(
"greptime:test".to_string(),
"value".to_string(),
)]));
if let Some(limit) = data_page_row_count_limit {
props = props
.set_write_batch_size(limit)
.set_data_page_row_count_limit(limit);
}
let props = props.build();
let mut writer =
ArrowWriter::try_new(File::create(path).unwrap(), schema, Some(props)).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
}
}
src/cmd/src/datanode/parquet_rewrite.rs
+632
View File
@@ -0,0 +1,632 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs::File;
use std::path::{Path, PathBuf};
use clap::Parser;
use datatypes::arrow::record_batch::RecordBatchReader;
use parquet::arrow::ArrowWriter;
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
use parquet::basic::{BrotliLevel, Compression, Encoding, GzipLevel, ZstdLevel};
use parquet::file::metadata::ParquetMetaData;
use parquet::file::properties::WriterProperties;
use parquet::file::reader::FileReader;
use parquet::file::serialized_reader::SerializedFileReader;
use parquet::schema::types::ColumnPath;
use serde::{Deserialize, Serialize};
use snafu::ResultExt;
use crate::datanode::tool_util::load_local_parquet_metadata;
use crate::error;
/// Read and rewrite a parquet file with different writer properties.
#[derive(Debug, Parser)]
pub struct ParquetRewriteCommand {
/// Path to input parquet file.
#[clap(long, value_name = "FILE")]
input: PathBuf,
/// Path to output parquet file in rewrite mode.
#[clap(long, value_name = "FILE")]
output: Option<PathBuf>,
/// Path to writer properties TOML in rewrite mode.
#[clap(long, value_name = "FILE")]
properties: Option<PathBuf>,
/// Dump writer properties TOML inferred from the input parquet file.
#[clap(long, value_name = "FILE")]
dump_properties: Option<PathBuf>,
/// Number of rows per record batch.
#[clap(long)]
batch_size: Option<usize>,
/// Overwrite output files.
#[clap(long, default_value_t = false)]
overwrite: bool,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)]
#[serde(default, deny_unknown_fields)]
struct RewriteProperties {
writer: WriterConfig,
columns: Vec<ColumnConfig>,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
#[serde(default, deny_unknown_fields)]
struct WriterConfig {
dictionary_enabled: Option<bool>,
compression: Option<CompressionConfig>,
compression_level: Option<u32>,
encoding: Option<EncodingConfig>,
max_row_group_row_count: Option<usize>,
data_page_size_limit: Option<usize>,
data_page_row_count_limit: Option<usize>,
dictionary_page_size_limit: Option<usize>,
column_index_truncate_length: Option<usize>,
statistics_truncate_length: Option<usize>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)]
#[serde(default, deny_unknown_fields)]
struct ColumnConfig {
path: Vec<String>,
dictionary_enabled: Option<bool>,
compression: Option<CompressionConfig>,
compression_level: Option<u32>,
encoding: Option<EncodingConfig>,
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "kebab-case")]
enum CompressionConfig {
Uncompressed,
Snappy,
Gzip,
Lzo,
Brotli,
Lz4,
Zstd,
Lz4Raw,
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "kebab-case")]
enum EncodingConfig {
Plain,
DeltaBinaryPacked,
DeltaLengthByteArray,
DeltaByteArray,
ByteStreamSplit,
}
impl ParquetRewriteCommand {
pub async fn run(&self) -> error::Result<()> {
match (&self.dump_properties, &self.output, &self.properties) {
(Some(path), None, None) => self.dump_properties(path),
(None, Some(output), Some(properties)) => self.rewrite(output, properties),
(Some(_), Some(_), _) | (Some(_), _, Some(_)) => illegal_config(
"use either --dump-properties or rewrite mode, not both".to_string(),
),
(None, _, _) => illegal_config(
"rewrite mode requires --output and --properties; config dump mode requires --dump-properties".to_string(),
),
}
}
fn dump_properties(&self, path: &Path) -> error::Result<()> {
ensure_distinct_paths(&self.input, path)?;
ensure_can_write(path, self.overwrite)?;
let metadata = load_local_parquet_metadata(&self.input)?;
let properties = infer_rewrite_properties(&metadata);
let content = toml::to_string_pretty(&properties).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("failed to serialize writer properties: {e}"),
}
.build()
})?;
std::fs::write(path, content).context(error::FileIoSnafu)?;
println!("Wrote writer properties to {}", path.display());
Ok(())
}
fn rewrite(&self, output: &Path, properties: &Path) -> error::Result<()> {
ensure_distinct_paths(&self.input, output)?;
ensure_can_write(output, self.overwrite)?;
let props = load_rewrite_properties(properties)?;
let input_reader = SerializedFileReader::new(open_file(&self.input)?)
.map_err(|e| parquet_error("read source parquet metadata", &self.input, e))?;
let key_value_metadata = input_reader
.metadata()
.file_metadata()
.key_value_metadata()
.cloned();
let mut reader_builder = ParquetRecordBatchReaderBuilder::try_new(open_file(&self.input)?)
.map_err(|e| parquet_error("open source parquet", &self.input, e))?;
if let Some(batch_size) = self.batch_size {
if batch_size == 0 {
return illegal_config("--batch-size must be greater than 0".to_string());
}
reader_builder = reader_builder.with_batch_size(batch_size);
}
let reader = reader_builder
.build()
.map_err(|e| parquet_error("build parquet reader", &self.input, e))?;
let schema = reader.schema();
let writer_props = build_writer_properties(props, key_value_metadata)?;
let mut writer = ArrowWriter::try_new(create_file(output)?, schema, Some(writer_props))
.map_err(|e| parquet_error("create parquet writer", output, e))?;
for batch in reader {
let batch =
batch.map_err(|e| parquet_error("read parquet batch", &self.input, e.into()))?;
writer
.write(&batch)
.map_err(|e| parquet_error("write parquet batch", output, e))?;
}
writer
.close()
.map_err(|e| parquet_error("close parquet writer", output, e))?;
println!("Wrote parquet file to {}", output.display());
Ok(())
}
}
fn infer_rewrite_properties(metadata: &ParquetMetaData) -> RewriteProperties {
let is_greptime_sst = metadata
.file_metadata()
.key_value_metadata()
.is_some_and(|key_values| {
key_values
.iter()
.any(|key_value| key_value.key == mito2::sst::parquet::PARQUET_METADATA_KEY)
});
let truncate_length = is_greptime_sst.then_some(0);
let first_column = metadata
.row_groups()
.first()
.and_then(|row_group| row_group.columns().first());
let compression = first_column.map(|column| compression_to_config(column.compression()));
let max_row_group_row_count = metadata
.row_groups()
.first()
.and_then(|row_group| usize::try_from(row_group.num_rows()).ok());
let dictionary_enabled = first_column
.map(|column| column.dictionary_page_offset().is_some())
.or(Some(true));
let columns = metadata
.file_metadata()
.schema_descr()
.columns()
.iter()
.enumerate()
.map(|(idx, column)| {
let first_chunk = metadata
.row_groups()
.first()
.and_then(|row_group| row_group.columns().get(idx));
ColumnConfig {
path: column.path().parts().to_vec(),
dictionary_enabled: first_chunk
.map(|chunk| chunk.dictionary_page_offset().is_some()),
compression: first_chunk.map(|chunk| compression_to_config(chunk.compression())),
compression_level: None,
encoding: first_chunk.and_then(infer_data_encoding),
}
})
.collect();
RewriteProperties {
writer: WriterConfig {
dictionary_enabled,
compression,
compression_level: None,
encoding: first_column.and_then(infer_data_encoding),
max_row_group_row_count,
data_page_size_limit: None,
data_page_row_count_limit: None,
dictionary_page_size_limit: None,
column_index_truncate_length: truncate_length,
statistics_truncate_length: truncate_length,
},
columns,
}
}
fn load_rewrite_properties(path: &Path) -> error::Result<RewriteProperties> {
let content = std::fs::read_to_string(path).context(error::FileIoSnafu)?;
toml::from_str(&content).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("failed to parse writer properties {}: {e}", path.display()),
}
.build()
})
}
fn build_writer_properties(
config: RewriteProperties,
key_value_metadata: Option<Vec<parquet::file::metadata::KeyValue>>,
) -> error::Result<WriterProperties> {
let mut builder = WriterProperties::builder().set_key_value_metadata(key_value_metadata);
let writer_compression = config.writer.compression;
if let Some(dictionary_enabled) = config.writer.dictionary_enabled {
builder = builder.set_dictionary_enabled(dictionary_enabled);
}
match (writer_compression, config.writer.compression_level) {
(Some(compression), level) => {
builder = builder.set_compression(to_parquet_compression(compression, level)?);
}
(None, Some(_)) => {
return illegal_config(
"writer compression_level requires writer compression".to_string(),
);
}
(None, None) => {}
}
if let Some(encoding) = config.writer.encoding {
builder = builder.set_encoding(to_parquet_encoding(encoding));
}
if let Some(max_row_group_row_count) = config.writer.max_row_group_row_count {
if max_row_group_row_count == 0 {
return illegal_config("max_row_group_row_count must be greater than 0".to_string());
}
builder = builder.set_max_row_group_row_count(Some(max_row_group_row_count));
}
if let Some(data_page_size_limit) = config.writer.data_page_size_limit {
builder = builder.set_data_page_size_limit(data_page_size_limit);
}
if let Some(data_page_row_count_limit) = config.writer.data_page_row_count_limit {
builder = builder.set_data_page_row_count_limit(data_page_row_count_limit);
}
if let Some(dictionary_page_size_limit) = config.writer.dictionary_page_size_limit {
builder = builder.set_dictionary_page_size_limit(dictionary_page_size_limit);
}
if let Some(column_index_truncate_length) = config.writer.column_index_truncate_length {
builder = builder.set_column_index_truncate_length(
(column_index_truncate_length > 0).then_some(column_index_truncate_length),
);
}
if let Some(statistics_truncate_length) = config.writer.statistics_truncate_length {
builder = builder.set_statistics_truncate_length(
(statistics_truncate_length > 0).then_some(statistics_truncate_length),
);
}
for column in config.columns {
if column.path.is_empty() {
return illegal_config("column path must not be empty".to_string());
}
let path = ColumnPath::new(column.path);
if let Some(dictionary_enabled) = column.dictionary_enabled {
builder = builder.set_column_dictionary_enabled(path.clone(), dictionary_enabled);
}
if column.compression.is_some() || column.compression_level.is_some() {
let Some(compression) = column.compression.or(writer_compression) else {
return illegal_config(format!(
"compression_level for column {} requires column or writer compression",
path.string()
));
};
builder = builder.set_column_compression(
path.clone(),
to_parquet_compression(compression, column.compression_level)?,
);
}
if let Some(encoding) = column.encoding {
builder = builder.set_column_encoding(path, to_parquet_encoding(encoding));
}
}
Ok(builder.build())
}
fn ensure_distinct_paths(input: &Path, output: &Path) -> error::Result<()> {
if input == output {
return illegal_config(format!(
"input and output paths must be different: {}",
input.display()
));
}
Ok(())
}
fn ensure_can_write(path: &Path, overwrite: bool) -> error::Result<()> {
if !overwrite && path.exists() {
return illegal_config(format!(
"{} already exists; pass --overwrite to replace it",
path.display()
));
}
Ok(())
}
fn open_file(path: &Path) -> error::Result<File> {
File::open(path).context(error::FileIoSnafu)
}
fn create_file(path: &Path) -> error::Result<File> {
File::create(path).context(error::FileIoSnafu)
}
fn illegal_config<T>(msg: String) -> error::Result<T> {
error::IllegalConfigSnafu { msg }.fail()
}
fn parquet_error(
action: &'static str,
path: &Path,
error: parquet::errors::ParquetError,
) -> error::Error {
error::IllegalConfigSnafu {
msg: format!("{action} failed for {}: {error}", path.display()),
}
.build()
}
fn to_parquet_compression(
compression: CompressionConfig,
level: Option<u32>,
) -> error::Result<Compression> {
if level.is_some()
&& !matches!(
compression,
CompressionConfig::Gzip | CompressionConfig::Brotli | CompressionConfig::Zstd
)
{
return illegal_config(format!(
"compression level is not supported for {}",
compression.name()
));
}
Ok(match compression {
CompressionConfig::Uncompressed => Compression::UNCOMPRESSED,
CompressionConfig::Snappy => Compression::SNAPPY,
CompressionConfig::Gzip => Compression::GZIP(match level {
Some(level) => GzipLevel::try_new(level).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid gzip compression level {level}: {e}"),
}
.build()
})?,
None => GzipLevel::default(),
}),
CompressionConfig::Lzo => Compression::LZO,
CompressionConfig::Brotli => Compression::BROTLI(match level {
Some(level) => BrotliLevel::try_new(level).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid brotli compression level {level}: {e}"),
}
.build()
})?,
None => BrotliLevel::default(),
}),
CompressionConfig::Lz4 => Compression::LZ4,
CompressionConfig::Zstd => Compression::ZSTD(match level {
Some(level) => {
let converted_level = i32::try_from(level).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid zstd compression level {level}: {e}"),
}
.build()
})?;
ZstdLevel::try_new(converted_level).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid zstd compression level {level}: {e}"),
}
.build()
})?
}
None => ZstdLevel::default(),
}),
CompressionConfig::Lz4Raw => Compression::LZ4_RAW,
})
}
impl CompressionConfig {
fn name(self) -> &'static str {
match self {
CompressionConfig::Uncompressed => "uncompressed",
CompressionConfig::Snappy => "snappy",
CompressionConfig::Gzip => "gzip",
CompressionConfig::Lzo => "lzo",
CompressionConfig::Brotli => "brotli",
CompressionConfig::Lz4 => "lz4",
CompressionConfig::Zstd => "zstd",
CompressionConfig::Lz4Raw => "lz4-raw",
}
}
}
fn to_parquet_encoding(encoding: EncodingConfig) -> Encoding {
match encoding {
EncodingConfig::Plain => Encoding::PLAIN,
EncodingConfig::DeltaBinaryPacked => Encoding::DELTA_BINARY_PACKED,
EncodingConfig::DeltaLengthByteArray => Encoding::DELTA_LENGTH_BYTE_ARRAY,
EncodingConfig::DeltaByteArray => Encoding::DELTA_BYTE_ARRAY,
EncodingConfig::ByteStreamSplit => Encoding::BYTE_STREAM_SPLIT,
}
}
fn infer_data_encoding(
column: &parquet::file::metadata::ColumnChunkMetaData,
) -> Option<EncodingConfig> {
let encodings: Vec<_> = column.encodings().collect();
if encodings.contains(&Encoding::DELTA_BINARY_PACKED) {
Some(EncodingConfig::DeltaBinaryPacked)
} else if encodings.contains(&Encoding::DELTA_LENGTH_BYTE_ARRAY) {
Some(EncodingConfig::DeltaLengthByteArray)
} else if encodings.contains(&Encoding::DELTA_BYTE_ARRAY) {
Some(EncodingConfig::DeltaByteArray)
} else if encodings.contains(&Encoding::BYTE_STREAM_SPLIT) {
Some(EncodingConfig::ByteStreamSplit)
} else if encodings.contains(&Encoding::PLAIN) {
Some(EncodingConfig::Plain)
} else {
None
}
}
fn compression_to_config(compression: Compression) -> CompressionConfig {
match compression {
Compression::UNCOMPRESSED => CompressionConfig::Uncompressed,
Compression::SNAPPY => CompressionConfig::Snappy,
Compression::GZIP(_) => CompressionConfig::Gzip,
Compression::LZO => CompressionConfig::Lzo,
Compression::BROTLI(_) => CompressionConfig::Brotli,
Compression::LZ4 => CompressionConfig::Lz4,
Compression::ZSTD(_) => CompressionConfig::Zstd,
Compression::LZ4_RAW => CompressionConfig::Lz4Raw,
}
}
#[cfg(test)]
mod tests {
use std::sync::Arc;
use datatypes::arrow::array::{Int32Array, RecordBatch, StringArray};
use datatypes::arrow::datatypes::{DataType, Field, Schema};
use parquet::file::metadata::KeyValue;
use tempfile::tempdir;
use super::*;
#[test]
fn test_dump_and_rewrite_preserves_key_value_metadata_and_disables_dictionary() {
let dir = tempdir().unwrap();
let input = dir.path().join("input.parquet");
let output = dir.path().join("output.parquet");
write_test_parquet(&input, true, Some("greptime:test"));
let metadata = load_local_parquet_metadata(&input).unwrap();
let mut properties = infer_rewrite_properties(&metadata);
properties.columns[1].dictionary_enabled = Some(false);
let props_path = dir.path().join("props.toml");
std::fs::write(&props_path, toml::to_string(&properties).unwrap()).unwrap();
let command = ParquetRewriteCommand {
input: input.clone(),
output: Some(output.clone()),
properties: Some(props_path),
dump_properties: None,
batch_size: Some(2),
overwrite: false,
};
command
.rewrite(&output, command.properties.as_ref().unwrap())
.unwrap();
let rewritten = load_local_parquet_metadata(&output).unwrap();
assert_eq!(rewritten.file_metadata().num_rows(), 4);
let key_values = rewritten
.file_metadata()
.key_value_metadata()
.cloned()
.unwrap_or_default();
assert!(key_values.iter().any(|kv| kv.key == "greptime:test"));
assert!(
rewritten
.row_groups()
.iter()
.all(|row_group| row_group.column(1).dictionary_page_offset().is_none())
);
}
#[test]
fn test_infer_truncation_settings_for_greptime_sst() {
let dir = tempdir().unwrap();
let greptime_sst = dir.path().join("greptime.parquet");
let generic_parquet = dir.path().join("generic.parquet");
write_test_parquet(
&greptime_sst,
true,
Some(mito2::sst::parquet::PARQUET_METADATA_KEY),
);
write_test_parquet(&generic_parquet, true, None);
let metadata = load_local_parquet_metadata(&greptime_sst).unwrap();
let config = infer_rewrite_properties(&metadata);
assert_eq!(Some(0), config.writer.column_index_truncate_length);
assert_eq!(Some(0), config.writer.statistics_truncate_length);
let serialized = toml::to_string(&config).unwrap();
assert!(serialized.contains("column_index_truncate_length = 0"));
assert!(serialized.contains("statistics_truncate_length = 0"));
let properties = build_writer_properties(config, None).unwrap();
assert_eq!(None, properties.column_index_truncate_length());
assert_eq!(None, properties.statistics_truncate_length());
let metadata = load_local_parquet_metadata(&generic_parquet).unwrap();
let config = infer_rewrite_properties(&metadata);
assert_eq!(None, config.writer.column_index_truncate_length);
assert_eq!(None, config.writer.statistics_truncate_length);
}
#[test]
fn test_column_compression_level_inherits_writer_compression() {
let path = ColumnPath::new(vec!["host".to_string()]);
let config = RewriteProperties {
writer: WriterConfig {
compression: Some(CompressionConfig::Zstd),
compression_level: Some(1),
..Default::default()
},
columns: vec![ColumnConfig {
path: vec!["host".to_string()],
compression_level: Some(3),
..Default::default()
}],
};
let properties = build_writer_properties(config, None).unwrap();
assert_eq!(
properties.compression(&path),
Compression::ZSTD(ZstdLevel::try_new(3).unwrap())
);
}
fn write_test_parquet(path: &Path, dictionary_enabled: bool, metadata_key: Option<&str>) {
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("host", DataType::Utf8, false),
]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(Int32Array::from(vec![1, 2, 3, 4])),
Arc::new(StringArray::from(vec!["a", "a", "b", "b"])),
],
)
.unwrap();
let key_value_metadata =
metadata_key.map(|key| vec![KeyValue::new(key.to_string(), "value".to_string())]);
let props = WriterProperties::builder()
.set_dictionary_enabled(dictionary_enabled)
.set_key_value_metadata(key_value_metadata)
.build();
let mut writer =
ArrowWriter::try_new(File::create(path).unwrap(), schema, Some(props)).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
}
}
src/cmd/src/datanode/parquetbench.rs
+293 -108
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::path::PathBuf;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::{Duration, Instant};
@@ -42,7 +42,10 @@ use store_api::region_request::PathType;
use store_api::storage::consts::{PRIMARY_KEY_COLUMN_NAME, is_internal_column};
use store_api::storage::{ColumnId, FileId, RegionId};
use crate::datanode::objbench::{build_object_store, extract_region_metadata, parse_config};
use crate::datanode::tool_util::{
build_object_store, extract_region_metadata, format_bytes, parse_config, parse_file_id,
parse_path_type, parse_region_id,
};
use crate::error;
const DEFAULT_READ_BATCH_SIZE: usize = 8 * 1024;
@@ -52,19 +55,23 @@ const DEFAULT_READ_BATCH_SIZE: usize = 8 * 1024;
pub struct ParquetbenchCommand {
/// Path to config TOML file (same format as standalone/datanode config)
#[clap(long, value_name = "FILE")]
config: PathBuf,
config: Option<PathBuf>,
/// Region ID: either numeric u64 (e.g. "4398046511104") or "table_id:region_num" (e.g. "1024:0")
#[clap(long)]
region_id: String,
region_id: Option<String>,
/// Table directory relative to data home (e.g. "data/greptime/public/1024/")
#[clap(long)]
table_dir: String,
table_dir: Option<String>,
/// SST file id to benchmark.
#[clap(long)]
file_id: String,
file_id: Option<String>,
/// Local parquet SST file to benchmark with the direct reader.
#[clap(long, value_name = "FILE")]
file_path: Option<PathBuf>,
/// Path to scan request JSON config file (supports projection_names only)
#[clap(long, value_name = "FILE")]
@@ -129,6 +136,32 @@ struct IterationStats {
elapsed: Duration,
}
#[derive(Debug)]
enum ParquetbenchInput {
LocalFile {
file_path: PathBuf,
},
Region {
config: PathBuf,
region_id: String,
table_dir: String,
file_id: String,
path_type: PathType,
},
}
struct ParquetbenchSource {
object_store: object_store::ObjectStore,
file_path: String,
display_path: String,
region_id_label: String,
region_id: RegionId,
file_id: FileId,
region_file_id: RegionFileId,
path_type: Option<PathType>,
table_dir: Option<String>,
}
impl ParquetbenchCommand {
pub async fn run(&self) -> error::Result<()> {
if self.verbose {
@@ -145,41 +178,40 @@ impl ParquetbenchCommand {
.fail();
}
let region_id = parse_region_id(&self.region_id)?;
let path_type = parse_path_type(&self.path_type)?;
let file_id = FileId::parse_str(&self.file_id).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid file_id '{}': {}", self.file_id, e),
}
.build()
})?;
let region_file_id = RegionFileId::new(region_id, file_id);
let file_path = sst_file_path(&self.table_dir, region_file_id, path_type);
let input = self.resolve_input()?;
let mut source = build_source(input).await?;
let (store_cfg, _mito_config, _wal_config) = parse_config(&self.config)?;
let object_store = build_object_store(&store_cfg).await?;
let file_size = object_store
.stat(&file_path)
let file_size = source
.object_store
.stat(&source.file_path)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("stat failed for {}: {}", file_path, e),
msg: format!("stat failed for {}: {}", source.display_path, e),
}
.build()
})?
.content_length();
let mut metadata_metrics = MetadataCacheMetrics::default();
let parquet_meta = MetadataLoader::new(object_store.clone(), &file_path, file_size)
.load(&mut metadata_metrics)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("read parquet metadata failed for {}: {:?}", file_path, e),
}
.build()
})?;
let region_meta = extract_region_metadata(&file_path, &parquet_meta)?;
let parquet_meta =
MetadataLoader::new(source.object_store.clone(), &source.file_path, file_size)
.load(&mut metadata_metrics)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!(
"read parquet metadata failed for {}: {:?}",
source.display_path, e
),
}
.build()
})?;
let region_meta = extract_region_metadata(&source.display_path, &parquet_meta)?;
if source.table_dir.is_none() {
source.region_id = region_meta.region_id;
source.region_id_label = source.region_id.as_u64().to_string();
source.region_file_id = RegionFileId::new(source.region_id, source.file_id);
}
let scan_config = self.load_scan_config().await?;
let projection = if self.reader == ReaderMode::Direct {
resolve_projection_names(&scan_config, &region_meta)?
@@ -216,10 +248,10 @@ impl ParquetbenchCommand {
println!(
"{} Region ID: {} (u64: {})",
"✓".green(),
self.region_id,
region_id.as_u64()
source.region_id_label,
source.region_id.as_u64()
);
println!("{} File path: {}", "✓".green(), file_path.cyan());
println!("{} File path: {}", "✓".green(), source.display_path.cyan());
println!(
"{} Columns: {}",
"✓".green(),
@@ -307,8 +339,8 @@ impl ParquetbenchCommand {
let mut schema_printed = false;
let file_handle = FileHandle::new(
FileMeta {
region_id,
file_id,
region_id: source.region_id,
file_id: source.file_id,
time_range: Default::default(),
level: 0,
file_size,
@@ -332,9 +364,9 @@ impl ParquetbenchCommand {
let stats = match self.reader {
ReaderMode::Direct => {
run_direct_iteration(
object_store.clone(),
file_path.clone(),
region_file_id,
source.object_store.clone(),
source.file_path.clone(),
source.region_file_id,
parquet_meta.clone(),
projection.clone(),
row_groups.clone(),
@@ -345,9 +377,19 @@ impl ParquetbenchCommand {
}
ReaderMode::FlatPrune => {
run_flat_prune_iteration(
object_store.clone(),
self.table_dir.clone(),
path_type,
source.object_store.clone(),
source.table_dir.clone().ok_or_else(|| {
error::IllegalConfigSnafu {
msg: "flat-prune reader requires --table-dir".to_string(),
}
.build()
})?,
source.path_type.ok_or_else(|| {
error::IllegalConfigSnafu {
msg: "flat-prune reader requires --path-type".to_string(),
}
.build()
})?,
file_handle.clone(),
region_meta.clone(),
projection_column_ids.clone(),
@@ -450,6 +492,65 @@ impl ParquetbenchCommand {
Ok(())
}
fn resolve_input(&self) -> error::Result<ParquetbenchInput> {
let has_region_args = self.config.is_some()
|| self.region_id.is_some()
|| self.table_dir.is_some()
|| self.file_id.is_some();
if let Some(file_path) = &self.file_path {
if self.reader == ReaderMode::FlatPrune {
return Err(error::IllegalConfigSnafu {
msg: "--file-path currently supports only --reader direct".to_string(),
}
.build());
}
if has_region_args {
return Err(error::IllegalConfigSnafu {
msg: "--file-path cannot be used with --config, --region-id, --table-dir, or --file-id".to_string(),
}
.build());
}
return Ok(ParquetbenchInput::LocalFile {
file_path: file_path.clone(),
});
}
let config = self.config.clone().ok_or_else(|| {
error::IllegalConfigSnafu {
msg: "missing --config unless --file-path is specified".to_string(),
}
.build()
})?;
let region_id = self.region_id.clone().ok_or_else(|| {
error::IllegalConfigSnafu {
msg: "missing --region-id unless --file-path is specified".to_string(),
}
.build()
})?;
let table_dir = self.table_dir.clone().ok_or_else(|| {
error::IllegalConfigSnafu {
msg: "missing --table-dir unless --file-path is specified".to_string(),
}
.build()
})?;
let file_id = self.file_id.clone().ok_or_else(|| {
error::IllegalConfigSnafu {
msg: "missing --file-id unless --file-path is specified".to_string(),
}
.build()
})?;
let path_type = parse_path_type(&self.path_type)?;
Ok(ParquetbenchInput::Region {
config,
region_id,
table_dir,
file_id,
path_type,
})
}
async fn load_scan_config(&self) -> error::Result<ParquetScanConfig> {
if let Some(path) = &self.scan_config {
let content = tokio::fs::read_to_string(path)
@@ -462,6 +563,117 @@ impl ParquetbenchCommand {
}
}
async fn build_source(input: ParquetbenchInput) -> error::Result<ParquetbenchSource> {
match input {
ParquetbenchInput::LocalFile { file_path } => build_local_file_source(&file_path),
ParquetbenchInput::Region {
config,
region_id,
table_dir,
file_id,
path_type,
} => build_region_source(&config, &region_id, table_dir, &file_id, path_type).await,
}
}
fn build_local_file_source(file_path: &Path) -> error::Result<ParquetbenchSource> {
let file_path = std::fs::canonicalize(file_path).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid --file-path {}: {e}", file_path.display()),
}
.build()
})?;
if !file_path.is_file() {
return Err(error::IllegalConfigSnafu {
msg: format!("--file-path {} is not a file", file_path.display()),
}
.build());
}
let parent = file_path.parent().ok_or_else(|| {
error::IllegalConfigSnafu {
msg: format!(
"--file-path {} has no parent directory",
file_path.display()
),
}
.build()
})?;
let file_name = file_path
.file_name()
.and_then(|name| name.to_str())
.ok_or_else(|| {
error::IllegalConfigSnafu {
msg: format!("invalid UTF-8 file name in {}", file_path.display()),
}
.build()
})?
.to_string();
let object_store = object_store::ObjectStore::new(object_store::services::Fs::default().root(
parent.to_str().ok_or_else(|| {
error::IllegalConfigSnafu {
msg: format!("invalid UTF-8 parent directory in {}", file_path.display()),
}
.build()
})?,
))
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("failed to build local file object store: {e:?}"),
}
.build()
})?
.finish();
let file_id = file_path
.file_stem()
.and_then(|stem| stem.to_str())
.and_then(|stem| FileId::parse_str(stem).ok())
.unwrap_or_else(FileId::random);
let region_id = RegionId::new(0, 0);
let region_file_id = RegionFileId::new(region_id, file_id);
let display_path = file_path.display().to_string();
Ok(ParquetbenchSource {
object_store,
file_path: file_name,
display_path,
region_id_label: region_id.as_u64().to_string(),
region_id,
file_id,
region_file_id,
path_type: None,
table_dir: None,
})
}
async fn build_region_source(
config: &Path,
region_id: &str,
table_dir: String,
file_id: &str,
path_type: PathType,
) -> error::Result<ParquetbenchSource> {
let region = parse_region_id(region_id)?;
let file_id = parse_file_id(file_id)?;
let region_file_id = RegionFileId::new(region, file_id);
let file_path = sst_file_path(&table_dir, region_file_id, path_type);
let (store_cfg, _mito_config, _wal_config) = parse_config(config)?;
let object_store = build_object_store(&store_cfg).await?;
Ok(ParquetbenchSource {
object_store,
display_path: file_path.clone(),
file_path,
region_id_label: region_id.to_string(),
region_id: region,
file_id,
region_file_id,
path_type: Some(path_type),
table_dir: Some(table_dir),
})
}
#[allow(clippy::too_many_arguments)]
async fn run_direct_iteration(
object_store: object_store::ObjectStore,
@@ -554,7 +766,7 @@ async fn run_flat_prune_iteration(
let reader_builder = ParquetReaderBuilder::new(table_dir, path_type, file_handle, object_store)
.expected_metadata(Some(region_meta))
.cache(CacheStrategy::Disabled)
.projection(projection.map(ReadColumns::from_deduped_column_ids));
.projection(projection.map(ReadColumns::new));
let mut reader_metrics = ReaderMetrics::default();
let start = Instant::now();
let mut stats = IterationStats::default();
@@ -773,60 +985,6 @@ fn parse_batch_size(s: &str) -> Result<usize, String> {
Ok(batch_size)
}
fn parse_region_id(s: &str) -> error::Result<RegionId> {
if s.contains(':') {
let parts: Vec<&str> = s.splitn(2, ':').collect();
let table_id: u32 = parts[0].parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid table_id in region_id '{}': {}", s, e),
}
.build()
})?;
let region_num: u32 = parts[1].parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region_num in region_id '{}': {}", s, e),
}
.build()
})?;
Ok(RegionId::new(table_id, region_num))
} else {
let id: u64 = s.parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region_id '{}': {}", s, e),
}
.build()
})?;
Ok(RegionId::from_u64(id))
}
}
fn parse_path_type(s: &str) -> error::Result<PathType> {
match s.to_lowercase().as_str() {
"bare" => Ok(PathType::Bare),
"data" => Ok(PathType::Data),
"metadata" => Ok(PathType::Metadata),
_ => Err(error::IllegalConfigSnafu {
msg: format!("invalid path_type '{}', expected: bare, data, metadata", s),
}
.build()),
}
}
fn format_bytes(bytes: u64) -> String {
const KIB: u64 = 1024;
const MIB: u64 = 1024 * KIB;
const GIB: u64 = 1024 * MIB;
if bytes >= GIB {
format!("{:.2} GiB", bytes as f64 / GIB as f64)
} else if bytes >= MIB {
format!("{:.2} MiB", bytes as f64 / MIB as f64)
} else if bytes >= KIB {
format!("{:.2} KiB", bytes as f64 / KIB as f64)
} else {
format!("{} B", bytes)
}
}
fn format_rate(rate: f64) -> String {
if !rate.is_finite() {
return "inf rows".to_string();
@@ -851,6 +1009,25 @@ mod tests {
use super::*;
fn test_command() -> ParquetbenchCommand {
ParquetbenchCommand {
config: None,
region_id: None,
table_dir: None,
file_id: None,
file_path: None,
scan_config: None,
iterations: 1,
batch_size: DEFAULT_READ_BATCH_SIZE,
path_type: "bare".to_string(),
verbose: false,
pprof_file: None,
pprof_after_warmup: false,
pk_as_binary: false,
reader: ReaderMode::Direct,
}
}
fn new_test_metadata() -> RegionMetadata {
let mut builder = RegionMetadataBuilder::new(RegionId::new(1, 0));
builder
@@ -882,19 +1059,27 @@ mod tests {
}
#[test]
fn test_parse_region_id() {
assert_eq!(parse_region_id("1024:7").unwrap(), RegionId::new(1024, 7));
assert_eq!(
parse_region_id(&RegionId::new(1, 2).as_u64().to_string()).unwrap(),
RegionId::new(1, 2)
);
fn test_resolve_input_accepts_direct_file_for_direct_reader() {
let mut command = test_command();
command.file_path = Some(PathBuf::from("/tmp/source.parquet"));
let input = command.resolve_input().unwrap();
match input {
ParquetbenchInput::LocalFile { file_path } => {
assert_eq!(file_path, PathBuf::from("/tmp/source.parquet"));
}
ParquetbenchInput::Region { .. } => panic!("expected local file input"),
}
}
#[test]
fn test_parse_path_type() {
assert_eq!(parse_path_type("bare").unwrap(), PathType::Bare);
assert_eq!(parse_path_type("data").unwrap(), PathType::Data);
assert_eq!(parse_path_type("metadata").unwrap(), PathType::Metadata);
fn test_resolve_input_rejects_mixed_input_modes() {
let mut command = test_command();
command.file_path = Some(PathBuf::from("/tmp/source.parquet"));
command.config = Some(PathBuf::from("config.toml"));
let err = command.resolve_input().unwrap_err();
assert!(err.to_string().contains("--file-path cannot be used with"));
}
#[test]
src/cmd/src/datanode/scanbench.rs
+23 -56
View File
@@ -13,6 +13,7 @@
// limitations under the License.
use std::collections::HashMap;
use std::fmt;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Instant;
@@ -31,6 +32,7 @@ use datafusion::logical_expr::{BinaryExpr, Expr as DfExpr, ExprSchemable, Operat
use datafusion_common::tree_node::{Transformed, TreeNodeRewriter};
use datafusion_common::{DFSchemaRef, ScalarValue, ToDFSchema};
use datafusion_physical_plan::metrics::ExecutionPlanMetricsSet;
use datafusion_physical_plan::{DisplayAs, DisplayFormatType};
use datatypes::arrow::compute;
use futures::StreamExt;
use futures::stream::FuturesUnordered;
@@ -52,13 +54,24 @@ use sqlparser::parser::Parser as SqlParser;
use store_api::metadata::RegionMetadata;
use store_api::path_utils::WAL_DIR;
use store_api::region_engine::{PrepareRequest, QueryScanContext, RegionEngine};
use store_api::region_request::{PathType, RegionOpenRequest, RegionRequest};
use store_api::region_request::{RegionOpenRequest, RegionRequest};
use store_api::storage::{RegionId, ScanRequest, TimeSeriesDistribution, TimeSeriesRowSelector};
use tokio::fs;
use crate::datanode::objbench::{build_object_store, parse_config};
use crate::datanode::tool_util::{
build_object_store, format_bytes, parse_config, parse_path_type, parse_region_id,
};
use crate::error;
/// Displays a scanner using DataFusion's verbose explain format.
struct VerboseScannerDisplay<'a, T: ?Sized>(&'a T);
impl<T: DisplayAs + ?Sized> fmt::Display for VerboseScannerDisplay<'_, T> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
self.0.fmt_as(DisplayFormatType::Verbose, f)
}
}
/// Scan benchmark command - benchmarks scanning a region directly from storage.
#[derive(Debug, Parser)]
pub struct ScanbenchCommand {
@@ -164,60 +177,6 @@ fn resolve_projection(
Ok(None)
}
fn format_bytes(bytes: u64) -> String {
const KIB: u64 = 1024;
const MIB: u64 = 1024 * KIB;
const GIB: u64 = 1024 * MIB;
if bytes >= GIB {
format!("{:.2} GiB", bytes as f64 / GIB as f64)
} else if bytes >= MIB {
format!("{:.2} MiB", bytes as f64 / MIB as f64)
} else if bytes >= KIB {
format!("{:.2} KiB", bytes as f64 / KIB as f64)
} else {
format!("{} B", bytes)
}
}
fn parse_region_id(s: &str) -> error::Result<RegionId> {
if s.contains(':') {
let parts: Vec<&str> = s.splitn(2, ':').collect();
let table_id: u32 = parts[0].parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid table_id in region_id '{}': {}", s, e),
}
.build()
})?;
let region_num: u32 = parts[1].parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region_num in region_id '{}': {}", s, e),
}
.build()
})?;
Ok(RegionId::new(table_id, region_num))
} else {
let id: u64 = s.parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region_id '{}': {}", s, e),
}
.build()
})?;
Ok(RegionId::from_u64(id))
}
}
fn parse_path_type(s: &str) -> error::Result<PathType> {
match s.to_lowercase().as_str() {
"bare" => Ok(PathType::Bare),
"data" => Ok(PathType::Data),
"metadata" => Ok(PathType::Metadata),
_ => Err(error::IllegalConfigSnafu {
msg: format!("invalid path_type '{}', expected: bare, data, metadata", s),
}
.build()),
}
}
/// Rewrites literal values in comparison expressions to match the column's arrow type.
struct LiteralTypeCaster {
schema: DFSchemaRef,
@@ -740,6 +699,14 @@ impl ScanbenchCommand {
format_bytes(total_estimated_size),
);
if self.verbose {
println!(
" {} Scanner explain: {}",
"ℹ".blue(),
VerboseScannerDisplay(scanner.as_ref())
);
}
// Start profiling after the first iteration (warmup) if pprof_after_warmup is set
#[cfg(unix)]
if iteration == 0
src/cmd/src/datanode/sst_replace.rs
+361
View File
@@ -0,0 +1,361 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::path::PathBuf;
use clap::Parser;
use colored::Colorize;
use mito2::manifest::action::{RegionEdit, RegionMetaAction, RegionMetaActionList};
use mito2::manifest::manager::{RegionManifestManager, RegionManifestOptions};
use mito2::region::ManifestStats;
use mito2::sst::file::RegionFileId;
use mito2::sst::location::{region_dir_from_table_dir, sst_file_path};
use object_store::ObjectStore;
use parquet::file::FOOTER_SIZE;
use parquet::file::metadata::{FooterTail, ParquetMetaDataReader};
use store_api::region_request::PathType;
use crate::datanode::tool_util::{
build_object_store, max_row_group_uncompressed_size, parse_config, parse_file_id,
parse_path_type, parse_region_id,
};
use crate::error;
/// Replace a mito region SST and update the corresponding manifest metadata.
#[derive(Debug, Parser)]
pub struct SstReplaceCommand {
/// Path to config TOML file (same format as standalone/datanode config).
#[clap(long, value_name = "FILE")]
config: PathBuf,
/// Region ID: either numeric u64 (e.g. "4398046511104") or "table_id:region_num" (e.g. "1024:0").
#[clap(long)]
region_id: String,
/// Table directory relative to data home (e.g. "data/greptime/public/1024/").
#[clap(long)]
table_dir: String,
/// SST file id to replace.
#[clap(long)]
file_id: String,
/// Local parquet file used as the replacement.
#[clap(long, value_name = "FILE", conflicts_with = "replacement_object")]
replacement_file: Option<PathBuf>,
/// Object-store parquet path used as the replacement.
#[clap(long, value_name = "PATH", conflicts_with = "replacement_file")]
replacement_object: Option<String>,
/// Path type for the region: auto, bare, data, metadata.
#[clap(long, default_value = "auto")]
path_type: String,
/// Actually overwrite the SST object and append a manifest delta.
#[clap(long, default_value_t = false)]
confirm: bool,
/// Verbose output.
#[clap(short, long, default_value_t = false)]
verbose: bool,
}
impl SstReplaceCommand {
pub async fn run(&self) -> error::Result<()> {
if self.verbose {
common_telemetry::init_default_ut_logging();
}
println!("{}", "Starting sst-replace...".cyan().bold());
let region_id = parse_region_id(&self.region_id)?;
let file_id = parse_file_id(&self.file_id)?;
let path_type = parse_optional_path_type(&self.path_type)?;
let replacement = self.replacement()?;
let (store_cfg, mito_config, _wal_config) = parse_config(&self.config)?;
let object_store = build_object_store(&store_cfg).await?;
println!("{} Object store initialized", "[ok]".green());
let candidates = path_type
.map(|path_type| vec![path_type])
.unwrap_or_else(|| vec![PathType::Bare, PathType::Data, PathType::Metadata]);
let mut found = None;
let mut existing_manifests = Vec::new();
for candidate in candidates {
let stats = ManifestStats::default();
let region_dir = region_dir_from_table_dir(&self.table_dir, region_id, candidate);
let manifest_opts =
RegionManifestOptions::new(&mito_config, &region_dir, &object_store);
let Some(manifest_manager) = RegionManifestManager::open(manifest_opts, &stats)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("open {} manifest failed: {e:?}", path_type_name(candidate)),
}
.build()
})?
else {
continue;
};
existing_manifests.push(path_type_name(candidate));
let manifest = manifest_manager.manifest();
if let Some(file) = manifest.files.get(&file_id) {
if found.is_some() {
return error::IllegalConfigSnafu {
msg: format!(
"file {} exists in multiple manifests under region {}; specify --path-type",
file_id, region_id
),
}
.fail();
}
found = Some((candidate, manifest_manager, file.clone()));
}
}
let Some((path_type, mut manifest_manager, old_file)) = found else {
let suffix = if existing_manifests.is_empty() {
"no manifest found".to_string()
} else {
format!(
"manifests found for [{}], but none contains file {}",
existing_manifests.join(", "),
file_id
)
};
return error::IllegalConfigSnafu {
msg: format!("region manifest not found for {}: {}", region_id, suffix),
}
.fail();
};
let region_file_id = RegionFileId::new(old_file.region_id, old_file.file_id);
let target_path = sst_file_path(&self.table_dir, region_file_id, path_type);
let target_stat = object_store.stat(&target_path).await.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("stat target SST {} failed: {e}", target_path),
}
.build()
})?;
let replacement_bytes = load_replacement_bytes(&object_store, replacement).await?;
let new_file_size = replacement_bytes.len() as u64;
let parquet_meta = decode_parquet_metadata(&replacement_bytes).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("read replacement parquet metadata failed: {e}"),
}
.build()
})?;
let new_num_rows = parquet_meta.file_metadata().num_rows() as u64;
let new_num_row_groups = parquet_meta.num_row_groups() as u64;
validate_replacement(&old_file, new_num_rows, new_num_row_groups)?;
let mut new_file = old_file.clone();
new_file.file_size = new_file_size;
new_file.num_rows = new_num_rows;
new_file.num_row_groups = new_num_row_groups;
new_file.max_row_group_uncompressed_size = max_row_group_uncompressed_size(&parquet_meta);
println!("{} Region: {}", "[ok]".green(), region_id);
println!("{} Target SST: {}", "[ok]".green(), target_path.cyan());
println!(
"{} Size: {} -> {} bytes",
"[ok]".green(),
target_stat.content_length(),
new_file_size
);
println!(
"{} Rows: {}, row groups: {}",
"[ok]".green(),
new_num_rows,
new_num_row_groups
);
if !self.confirm {
println!(
"{} Dry run only. Re-run with --confirm to overwrite the SST and update the manifest.",
"[dry-run]".yellow()
);
return Ok(());
}
object_store
.write(&target_path, replacement_bytes)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("write replacement SST {} failed: {e}", target_path),
}
.build()
})?;
let edit = RegionEdit {
files_to_add: vec![new_file],
files_to_remove: vec![],
timestamp_ms: None,
compaction_time_window: None,
flushed_entry_id: None,
flushed_sequence: None,
committed_sequence: None,
};
let version = manifest_manager
.update(
RegionMetaActionList::with_action(RegionMetaAction::Edit(edit)),
false,
)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("update manifest failed: {e:?}"),
}
.build()
})?;
println!(
"{} Replacement committed, manifest version {}",
"[ok]".green(),
version
);
Ok(())
}
fn replacement(&self) -> error::Result<Replacement<'_>> {
match (&self.replacement_file, &self.replacement_object) {
(Some(path), None) => Ok(Replacement::Local(path)),
(None, Some(path)) => Ok(Replacement::Object(path)),
_ => error::IllegalConfigSnafu {
msg: "specify exactly one of --replacement-file or --replacement-object"
.to_string(),
}
.fail(),
}
}
}
enum Replacement<'a> {
Local(&'a PathBuf),
Object(&'a str),
}
async fn load_replacement_bytes(
object_store: &ObjectStore,
replacement: Replacement<'_>,
) -> error::Result<Vec<u8>> {
match replacement {
Replacement::Local(path) => tokio::fs::read(path).await.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("read replacement file {} failed: {e}", path.display()),
}
.build()
}),
Replacement::Object(path) => {
object_store
.read(path)
.await
.map(|b| b.to_vec())
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("read replacement object {} failed: {e}", path),
}
.build()
})
}
}
}
fn validate_replacement(
old_file: &mito2::sst::file::FileMeta,
new_num_rows: u64,
new_num_row_groups: u64,
) -> error::Result<()> {
if old_file.num_rows != 0 && old_file.num_rows != new_num_rows {
return error::IllegalConfigSnafu {
msg: format!(
"replacement row count mismatch: manifest has {}, replacement has {}",
old_file.num_rows, new_num_rows
),
}
.fail();
}
if old_file.num_row_groups != 0 && old_file.num_row_groups != new_num_row_groups {
return error::IllegalConfigSnafu {
msg: format!(
"replacement row group count mismatch: manifest has {}, replacement has {}",
old_file.num_row_groups, new_num_row_groups
),
}
.fail();
}
Ok(())
}
fn decode_parquet_metadata(
data: &[u8],
) -> Result<parquet::file::metadata::ParquetMetaData, Box<dyn std::error::Error + Send + Sync>> {
if data.len() < FOOTER_SIZE {
return Err("file too small".into());
}
let footer_start = data.len() - FOOTER_SIZE;
let mut footer = [0; FOOTER_SIZE];
footer.copy_from_slice(&data[footer_start..]);
let footer = FooterTail::try_new(&footer)?;
let metadata_len = footer.metadata_length();
if footer_start < metadata_len {
return Err("invalid footer/metadata length".into());
}
let metadata_start = footer_start - metadata_len;
Ok(ParquetMetaDataReader::decode_metadata(
&data[metadata_start..footer_start],
)?)
}
fn parse_optional_path_type(value: &str) -> error::Result<Option<PathType>> {
match value.to_lowercase().as_str() {
"auto" => Ok(None),
_ => parse_path_type(value).map(Some).map_err(|_| {
error::IllegalConfigSnafu {
msg: format!("invalid path_type '{value}', expected: auto, bare, data, metadata"),
}
.build()
}),
}
}
fn path_type_name(path_type: PathType) -> &'static str {
match path_type {
PathType::Bare => "bare",
PathType::Data => "data",
PathType::Metadata => "metadata",
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_validate_replacement_counts() {
let old_file = mito2::sst::file::FileMeta {
num_rows: 4,
num_row_groups: 1,
..Default::default()
};
assert!(validate_replacement(&old_file, 4, 1).is_ok());
assert!(validate_replacement(&old_file, 3, 1).is_err());
assert!(validate_replacement(&old_file, 4, 2).is_err());
let legacy_file = mito2::sst::file::FileMeta::default();
assert!(validate_replacement(&legacy_file, 4, 1).is_ok());
}
}
src/cmd/src/datanode/tool_util.rs
+236
View File
@@ -0,0 +1,236 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(feature = "dev-tools")]
use std::fs::File;
use std::path::Path;
use std::sync::Arc;
use common_wal::config::DatanodeWalConfig;
use datanode::config::RegionEngineConfig;
use datanode::store;
use mito2::config::MitoConfig;
use object_store::ObjectStore;
#[cfg(feature = "dev-tools")]
use parquet::basic::Compression;
use parquet::file::metadata::{KeyValue, ParquetMetaData};
#[cfg(feature = "dev-tools")]
use parquet::file::metadata::{PageIndexPolicy, ParquetMetaDataReader};
use snafu::OptionExt;
#[cfg(feature = "dev-tools")]
use snafu::ResultExt;
use store_api::metadata::{RegionMetadata, RegionMetadataRef};
use store_api::region_request::PathType;
use store_api::storage::{FileId, RegionId};
use crate::datanode::{StorageConfig, StorageConfigWrapper};
use crate::error;
pub(crate) fn parse_config(
config_path: &Path,
) -> error::Result<(StorageConfig, MitoConfig, DatanodeWalConfig)> {
let cfg_str = std::fs::read_to_string(config_path).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("failed to read config {}: {e}", config_path.display()),
}
.build()
})?;
let store_cfg: StorageConfigWrapper = toml::from_str(&cfg_str).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("failed to parse config {}: {e}", config_path.display()),
}
.build()
})?;
let wal_config = store_cfg.wal;
let storage_config = store_cfg.storage;
let mito_engine_config = store_cfg
.region_engine
.into_iter()
.find_map(|config| match config {
RegionEngineConfig::Mito(mito) => Some(mito),
_ => None,
})
.with_context(|| error::IllegalConfigSnafu {
msg: format!("Engine config not found in {:?}", config_path),
})?;
Ok((storage_config, mito_engine_config, wal_config))
}
pub(crate) async fn build_object_store(config: &StorageConfig) -> error::Result<ObjectStore> {
store::new_object_store(config.store.clone(), &config.data_home)
.await
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("Failed to build object store: {e:?}"),
}
.build()
})
}
pub(crate) fn extract_region_metadata(
file_path: &str,
metadata: &ParquetMetaData,
) -> error::Result<RegionMetadataRef> {
let key_values: Option<&Vec<KeyValue>> = metadata.file_metadata().key_value_metadata();
let Some(key_values) = key_values else {
return Err(error::IllegalConfigSnafu {
msg: format!("{file_path}: missing parquet key_value metadata"),
}
.build());
};
let json = key_values
.iter()
.find(|key_value| key_value.key == mito2::sst::parquet::PARQUET_METADATA_KEY)
.and_then(|key_value| key_value.value.as_ref())
.ok_or_else(|| {
error::IllegalConfigSnafu {
msg: format!(
"{file_path}: key {} not found or empty",
mito2::sst::parquet::PARQUET_METADATA_KEY
),
}
.build()
})?;
let region = RegionMetadata::from_json(json).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region metadata json: {e}"),
}
.build()
})?;
Ok(Arc::new(region))
}
pub(crate) fn parse_region_id(value: &str) -> error::Result<RegionId> {
if let Some((table_id, region_number)) = value.split_once(':') {
let table_id = table_id.parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid table_id in region_id '{value}': {e}"),
}
.build()
})?;
let region_number = region_number.parse().map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region_num in region_id '{value}': {e}"),
}
.build()
})?;
Ok(RegionId::new(table_id, region_number))
} else {
value.parse().map(RegionId::from_u64).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid region_id '{value}': {e}"),
}
.build()
})
}
}
pub(crate) fn parse_file_id(value: &str) -> error::Result<FileId> {
FileId::parse_str(value).map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("invalid file_id '{value}': {e}"),
}
.build()
})
}
pub(crate) fn parse_path_type(value: &str) -> error::Result<PathType> {
match value.to_lowercase().as_str() {
"bare" => Ok(PathType::Bare),
"data" => Ok(PathType::Data),
"metadata" => Ok(PathType::Metadata),
_ => Err(error::IllegalConfigSnafu {
msg: format!("invalid path_type '{value}', expected: bare, data, metadata"),
}
.build()),
}
}
pub(crate) fn format_bytes(bytes: u64) -> String {
const KIB: u64 = 1024;
const MIB: u64 = 1024 * KIB;
const GIB: u64 = 1024 * MIB;
if bytes >= GIB {
format!("{:.2} GiB", bytes as f64 / GIB as f64)
} else if bytes >= MIB {
format!("{:.2} MiB", bytes as f64 / MIB as f64)
} else if bytes >= KIB {
format!("{:.2} KiB", bytes as f64 / KIB as f64)
} else {
format!("{bytes} B")
}
}
pub(crate) fn max_row_group_uncompressed_size(metadata: &ParquetMetaData) -> u64 {
metadata
.row_groups()
.iter()
.map(|row_group| {
row_group
.columns()
.iter()
.map(|column| column.uncompressed_size() as u64)
.sum::<u64>()
})
.max()
.unwrap_or(0)
}
#[cfg(feature = "dev-tools")]
pub(crate) fn load_local_parquet_metadata(path: &Path) -> error::Result<ParquetMetaData> {
let file = File::open(path).context(error::FileIoSnafu)?;
ParquetMetaDataReader::new()
.with_page_index_policy(PageIndexPolicy::Optional)
.parse_and_finish(&file)
.map_err(|e| {
error::IllegalConfigSnafu {
msg: format!("read parquet metadata failed for {}: {e}", path.display()),
}
.build()
})
}
#[cfg(feature = "dev-tools")]
pub(crate) fn compression_name(compression: Compression) -> &'static str {
match compression {
Compression::UNCOMPRESSED => "uncompressed",
Compression::SNAPPY => "snappy",
Compression::GZIP(_) => "gzip",
Compression::LZO => "lzo",
Compression::BROTLI(_) => "brotli",
Compression::LZ4 => "lz4",
Compression::ZSTD(_) => "zstd",
Compression::LZ4_RAW => "lz4-raw",
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_region_and_path_type() {
assert_eq!(parse_region_id("1024:7").unwrap(), RegionId::new(1024, 7));
assert_eq!(
parse_region_id(&RegionId::new(1, 2).as_u64().to_string()).unwrap(),
RegionId::new(1, 2)
);
assert_eq!(parse_path_type("bare").unwrap(), PathType::Bare);
assert_eq!(parse_path_type("data").unwrap(), PathType::Data);
assert_eq!(parse_path_type("metadata").unwrap(), PathType::Metadata);
}
}
src/cmd/src/error.rs
+9 -1
View File
@@ -162,6 +162,13 @@ pub enum Error {
location: Location,
},
#[snafu(display("Failed to daemonize: {msg}"))]
Daemonize {
msg: String,
#[snafu(implicit)]
location: Location,
},
#[snafu(display("Unsupported selector type: {}", selector_type))]
UnsupportedSelectorType {
selector_type: String,
@@ -361,7 +368,8 @@ impl ErrorExt for Error {
Error::SerdeJson { .. }
| Error::FileIo { .. }
| Error::SpawnThread { .. }
| Error::InitTlsProvider { .. } => StatusCode::Unexpected,
| Error::InitTlsProvider { .. }
| Error::Daemonize { .. } => StatusCode::Unexpected,
Error::Other { source, .. } => source.status_code(),
src/cmd/src/flownode.rs
+2 -2
View File
@@ -248,8 +248,6 @@ impl StartCommand {
}
async fn build(&self, opts: FlownodeOptions) -> Result<Instance> {
common_runtime::init_global_runtimes(&opts.runtime);
let guard = common_telemetry::init_global_logging(
APP_NAME,
&opts.component.logging,
@@ -258,6 +256,8 @@ impl StartCommand {
None,
);
common_runtime::init_global_runtimes(&opts.runtime);
crate::options::flush_dropped_plugin_warnings();
log_versions(verbose_version(), short_version(), APP_NAME);
maybe_activate_heap_profile(&opts.component.memory);
src/cmd/src/frontend.rs
+198 -21
View File
@@ -13,7 +13,9 @@
// limitations under the License.
use std::fmt::Debug;
use std::future::Future;
use std::path::Path;
use std::pin::Pin;
use std::sync::Arc;
use std::time::Duration;
@@ -61,6 +63,7 @@ use crate::options::{GlobalOptions, GreptimeOptions};
use crate::{App, create_resource_limit_metrics, log_versions, maybe_activate_heap_profile};
type FrontendOptions = GreptimeOptions<frontend::frontend::FrontendOptions>;
type HeartbeatExtensionSetupFuture<'a> = Pin<Box<dyn Future<Output = Result<()>> + Send + 'a>>;
pub struct Instance {
frontend: Frontend,
@@ -117,7 +120,35 @@ pub struct Command {
impl Command {
pub async fn build(&self, opts: FrontendOptions) -> Result<Instance> {
self.subcmd.build(opts).await
self.build_with_heartbeat_extensions(opts, FrontendHeartbeatExtensions::default())
.await
}
/// Builds a frontend with pre-registered heartbeat extensions.
///
/// Register extensions before calling this method. The supplied registry is installed before
/// normal plugin setup. After plugin heartbeat setup completes, the registry is frozen before
/// heartbeat handlers and the heartbeat task are built, so later registrations are rejected
/// and every heartbeat consumer observes the same extension membership.
pub async fn build_with_heartbeat_extensions(
&self,
opts: FrontendOptions,
heartbeat_extensions: FrontendHeartbeatExtensions,
) -> Result<Instance> {
let plugins = plugins_with_heartbeat_extensions(heartbeat_extensions);
self.forward_build_with_plugins(opts, plugins, |command, opts, plugins| {
command.build_with_plugins(opts, plugins)
})
.await
}
fn forward_build_with_plugins<'a, T>(
&'a self,
opts: FrontendOptions,
plugins: Plugins,
build: impl FnOnce(&'a StartCommand, FrontendOptions, Plugins) -> T,
) -> T {
self.subcmd.forward_build_with_plugins(opts, plugins, build)
}
pub fn load_options(&self, global_options: &GlobalOptions) -> Result<FrontendOptions> {
@@ -131,9 +162,14 @@ pub enum SubCommand {
}
impl SubCommand {
async fn build(&self, opts: FrontendOptions) -> Result<Instance> {
fn forward_build_with_plugins<'a, T>(
&'a self,
opts: FrontendOptions,
plugins: Plugins,
build: impl FnOnce(&'a StartCommand, FrontendOptions, Plugins) -> T,
) -> T {
match self {
SubCommand::Start(cmd) => cmd.build(opts).await,
SubCommand::Start(cmd) => build(cmd, opts, plugins),
}
}
@@ -329,9 +365,11 @@ impl StartCommand {
Ok(())
}
async fn build(&self, opts: FrontendOptions) -> Result<Instance> {
common_runtime::init_global_runtimes(&opts.runtime);
async fn build_with_plugins(
&self,
opts: FrontendOptions,
plugins: Plugins,
) -> Result<Instance> {
let guard = common_telemetry::init_global_logging(
APP_NAME,
&opts.component.logging,
@@ -340,6 +378,8 @@ impl StartCommand {
Some(&opts.component.slow_query),
);
common_runtime::init_global_runtimes(&opts.runtime);
crate::options::flush_dropped_plugin_warnings();
log_versions(verbose_version(), short_version(), APP_NAME);
maybe_activate_heap_profile(&opts.component.memory);
@@ -380,15 +420,8 @@ impl StartCommand {
.await
.context(error::MetaClientInitSnafu)?;
let mut plugins = Plugins::new();
plugins::setup_frontend_plugins_pre_build(
&mut plugins,
&plugin_opts,
&opts,
Some(&meta_config),
)
.await
.context(error::StartFrontendSnafu)?;
let mut plugins =
prepare_frontend_plugins(plugins, &plugin_opts, &opts, Some(&meta_config)).await?;
// now initialize the meta_client with plugins
let meta_client = meta_client::create_meta_client(
@@ -434,6 +467,8 @@ impl StartCommand {
// Some queries are expected to take long time.
let mut channel_config = opts.datanode.client.channel_config();
channel_config.timeout = None;
// Source Flight streams and sink unary responses share pooled connections.
channel_config.http2_adaptive_window = Some(true);
if opts.grpc.flight_compression.transport_compression() {
channel_config.accept_compression = true;
channel_config.send_compression = true;
@@ -497,12 +532,20 @@ impl StartCommand {
let instance = Arc::new(instance);
plugins::setup_frontend_heartbeat_extensions(&mut plugins, &plugin_opts, &instance)
.await
.context(error::StartFrontendSnafu)?;
let heartbeat_extensions = plugins
.get::<FrontendHeartbeatExtensions>()
.unwrap_or_default();
let heartbeat_instance = instance.clone();
let heartbeat_extensions =
setup_and_freeze_frontend_heartbeat_extensions(&mut plugins, move |plugins| {
Box::pin(async move {
plugins::setup_frontend_heartbeat_extensions(
plugins,
&plugin_opts,
&heartbeat_instance,
)
.await
.context(error::StartFrontendSnafu)
})
})
.await?;
let heartbeat_task = Some(create_heartbeat_task_with_extensions(
&opts,
meta_client,
@@ -524,6 +567,36 @@ impl StartCommand {
}
}
async fn prepare_frontend_plugins(
mut plugins: Plugins,
plugin_opts: &[PluginOptions],
opts: &frontend::frontend::FrontendOptions,
meta_config: Option<&[PluginOptions]>,
) -> Result<Plugins> {
plugins::setup_frontend_plugins_pre_build(&mut plugins, plugin_opts, opts, meta_config)
.await
.context(error::StartFrontendSnafu)?;
Ok(plugins)
}
fn plugins_with_heartbeat_extensions(heartbeat_extensions: FrontendHeartbeatExtensions) -> Plugins {
let plugins = Plugins::new();
plugins.insert(heartbeat_extensions);
plugins
}
async fn setup_and_freeze_frontend_heartbeat_extensions(
plugins: &mut Plugins,
setup: impl for<'a> FnOnce(&'a mut Plugins) -> HeartbeatExtensionSetupFuture<'a>,
) -> Result<FrontendHeartbeatExtensions> {
setup(plugins).await?;
let extensions = plugins
.get::<FrontendHeartbeatExtensions>()
.unwrap_or_default();
extensions.freeze();
Ok(extensions)
}
pub fn create_heartbeat_task(
options: &frontend::frontend::FrontendOptions,
meta_client: MetaClientRef,
@@ -563,6 +636,7 @@ fn create_heartbeat_task_with_extensions(
#[cfg(test)]
mod tests {
use std::io::Write;
use std::sync::Arc;
use std::time::Duration;
use auth::{Identity, Password, UserProviderRef};
@@ -576,6 +650,109 @@ mod tests {
use super::*;
use crate::options::GlobalOptions;
#[derive(Debug)]
struct TestExtension(&'static str);
#[async_trait]
impl frontend::heartbeat::FrontendHeartbeatExtension for TestExtension {
fn name(&self) -> &'static str {
self.0
}
}
#[test]
fn test_command_forwards_heartbeat_extensions_to_start_build() {
let command = Command {
subcmd: SubCommand::Start(StartCommand::default()),
};
let extensions = FrontendHeartbeatExtensions::default();
let shared_extensions = extensions.clone();
let forwarded_plugins = command.forward_build_with_plugins(
FrontendOptions::default(),
plugins_with_heartbeat_extensions(extensions),
|_, _, plugins| plugins,
);
let forwarded_extensions = forwarded_plugins
.get::<FrontendHeartbeatExtensions>()
.unwrap();
assert_registry_identity(&shared_extensions, &forwarded_extensions);
}
#[tokio::test]
async fn test_prefilled_heartbeat_extensions_survive_pre_build_setup() {
let extensions = FrontendHeartbeatExtensions::default();
let shared_extensions = extensions.clone();
let options = frontend::frontend::FrontendOptions {
meta_client: Some(MetaClientOptions::default()),
..Default::default()
};
let plugins = prepare_frontend_plugins(
plugins_with_heartbeat_extensions(extensions),
&[],
&options,
None,
)
.await
.unwrap();
let setup_extensions = plugins.get_or_insert(FrontendHeartbeatExtensions::default);
assert_registry_identity(&shared_extensions, &setup_extensions);
}
#[tokio::test]
async fn test_heartbeat_extension_setup_completes_before_freeze() {
let extensions = FrontendHeartbeatExtensions::default();
assert_eq!(
extensions.try_register(Arc::new(TestExtension("caller"))),
Ok(())
);
let mut plugins = plugins_with_heartbeat_extensions(extensions.clone());
let finalized = setup_and_freeze_frontend_heartbeat_extensions(&mut plugins, |plugins| {
Box::pin(async move {
let setup_extensions = plugins.get_or_insert(FrontendHeartbeatExtensions::default);
assert_eq!(
setup_extensions.try_register(Arc::new(TestExtension("plugin"))),
Ok(())
);
Ok(())
})
})
.await
.unwrap();
assert_eq!(
finalized
.extensions()
.iter()
.map(|extension| extension.name())
.collect::<Vec<_>>(),
["caller", "plugin"]
);
assert_eq!(
extensions.try_register(Arc::new(TestExtension("late"))),
Err(frontend::heartbeat::RegistrationError::Frozen)
);
assert_eq!(finalized.len(), 2);
}
fn assert_registry_identity(
expected: &FrontendHeartbeatExtensions,
actual: &FrontendHeartbeatExtensions,
) {
let extension = Arc::new(TestExtension("cmd-test-extension"));
assert!(expected.register(extension.clone()));
let registered = actual.extensions();
assert_eq!(registered.len(), 1);
assert!(Arc::ptr_eq(
&(extension as Arc<dyn frontend::heartbeat::FrontendHeartbeatExtension>),
&registered[0]
));
}
#[test]
fn test_try_from_start_command() {
let command = StartCommand {
src/cmd/src/metasrv.rs
+2 -2
View File
@@ -318,8 +318,6 @@ impl StartCommand {
}
pub async fn build(&self, opts: MetasrvOptions) -> Result<Instance> {
common_runtime::init_global_runtimes(&opts.runtime);
let guard = common_telemetry::init_global_logging(
APP_NAME,
&opts.component.logging,
@@ -328,6 +326,8 @@ impl StartCommand {
None,
);
common_runtime::init_global_runtimes(&opts.runtime);
crate::options::flush_dropped_plugin_warnings();
log_versions(verbose_version(), short_version(), APP_NAME);
maybe_activate_heap_profile(&opts.component.memory);
src/cmd/src/standalone.rs
+48 -5
View File
@@ -60,7 +60,6 @@ use flow::{
FrontendInvoker, GrpcQueryHandlerWithBoxedError,
};
use frontend::frontend::Frontend;
use frontend::instance::StandaloneDatanodeManager;
use frontend::instance::builder::FrontendBuilder;
use frontend::server::Services;
use meta_srv::metasrv::{FLOW_ID_SEQ, TABLE_ID_SEQ};
@@ -72,7 +71,10 @@ use plugins::standalone::context::DdlManagerConfigureContext;
use servers::tls::{TlsMode, TlsOption, merge_tls_option};
use snafu::{OptionExt, ResultExt};
use standalone::options::StandaloneOptions;
use standalone::{StandaloneInformationExtension, StandaloneRepartitionProcedureFactory};
use standalone::{
StandaloneDatanodeManager, StandaloneInformationExtension,
StandaloneRepartitionProcedureFactory,
};
use tracing_appender::non_blocking::WorkerGuard;
use crate::error::{OtherSnafu, Result, StartFlownodeSnafu};
@@ -150,6 +152,11 @@ impl Command {
) -> Result<GreptimeOptions<StandaloneOptions>> {
self.subcmd.load_options(global_options)
}
/// Whether the `standalone start` command requested daemonization.
pub fn is_daemon(&self) -> bool {
self.subcmd.is_daemon()
}
}
#[derive(Parser)]
@@ -172,6 +179,12 @@ impl SubCommand {
SubCommand::Start(cmd) => cmd.load_options(global_options),
}
}
fn is_daemon(&self) -> bool {
match self {
SubCommand::Start(cmd) => cmd.is_daemon(),
}
}
}
pub struct Instance {
@@ -289,9 +302,24 @@ pub struct StartCommand {
/// The working home directory of this standalone instance.
#[clap(long)]
data_home: Option<String>,
/// Run in the background as a daemon.
#[cfg(unix)]
#[clap(short, long)]
daemon: bool,
}
impl StartCommand {
/// Whether the `standalone start` command requested daemonization.
#[cfg(unix)]
pub(crate) fn is_daemon(&self) -> bool {
self.daemon
}
#[cfg(not(unix))]
pub(crate) fn is_daemon(&self) -> bool {
false
}
/// Load the GreptimeDB options from various sources (command line, config file or env).
pub fn load_options(
&self,
@@ -393,8 +421,6 @@ impl StartCommand {
#[allow(clippy::diverging_sub_expression)]
/// Build GreptimeDB instance with the loaded options.
pub async fn build(&self, opts: GreptimeOptions<StandaloneOptions>) -> Result<Instance> {
common_runtime::init_global_runtimes(&opts.runtime);
let guard = common_telemetry::init_global_logging(
APP_NAME,
&opts.component.logging,
@@ -403,6 +429,8 @@ impl StartCommand {
Some(&opts.component.slow_query),
);
common_runtime::init_global_runtimes(&opts.runtime);
crate::options::flush_dropped_plugin_warnings();
log_versions(verbose_version(), short_version(), APP_NAME);
maybe_activate_heap_profile(&opts.component.memory);
@@ -1127,7 +1155,9 @@ mod tests {
fn test_toml() {
let opts = StandaloneOptions::default();
let toml_string = toml::to_string(&opts).unwrap();
let _parsed: StandaloneOptions = toml::from_str(&toml_string).unwrap();
assert!(toml_string.contains("experimental_enable_exponential_histogram = false"));
let parsed: StandaloneOptions = toml::from_str(&toml_string).unwrap();
assert_eq!(parsed.otlp, opts.otlp);
}
#[test]
@@ -1352,6 +1382,19 @@ mod tests {
assert_eq!(command.grpc_bind_addr.as_deref(), Some("127.0.0.1:34001"));
}
#[cfg(unix)]
#[test]
fn test_parse_daemon_flag() {
let command = StartCommand::try_parse_from(["standalone", "--daemon"]).unwrap();
assert!(command.is_daemon());
let command = StartCommand::try_parse_from(["standalone", "-d"]).unwrap();
assert!(command.is_daemon());
let command = StartCommand::try_parse_from(["standalone"]).unwrap();
assert!(!command.is_daemon());
}
#[test]
fn test_help_uses_grpc_option_names() {
let mut cmd = StartCommand::command();

Some files were not shown because too many files have changed in this diff Show More