mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-03 10:35:35 +00:00

Files

T

Ning Sun 43eaea7a9a fix(ci): teach check-builder-rust-version.sh to handle stable channels (#9369 )

* fix(ci): teach check-builder-rust-version.sh to handle stable channels

The script extracted a YYYY-MM-DD date from rust-toolchain.toml to
compare against the rustc build date inside the dev-builder image —
a nightly-era design. With channel = "1.96.1" there is no date in
the file, so every release build failed with 'Error: No rust toolchain
version found in rust-toolchain.toml'.

Extract the channel token instead and branch on it:
- stable channel (X.Y[.Z]): require the builder image's rustc to
  exactly match the pinned version
- nightly-YYYY-MM-DD: keep the legacy date-difference check

Verified against a mocked docker for all four paths (stable match /
mismatch, nightly fresh / stale).

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore(toolchain): finish stable-migration cleanup in docs and query-regression pin

- README/AGENTS: the toolchain is now stable Rust pinned by
  rust-toolchain.toml, not nightly
- query-regression: align the benchmark toolchain pin with the
  workspace (nightly-2026-03-21 = 1.96.0-nightly -> stable 1.96.1),
  including the exact-version assertions (cargo 356927216, rustc
  31fca3adb, both 2026-06-26) and the runner image default

The query-regression runner image must be rebuilt and
QUERY_REGRESSION_ECS_IMAGE_ID bumped together with these pins
(per .github/runner-scale-sets/query-regression/README.md) before
the next regression run.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* ci(query-regression): derive the Rust toolchain pin from rust-toolchain.toml

Replace the hard-coded RUSTUP_TOOLCHAIN value and the hard-coded
version strings in the runner Verify assertions with a pin resolved
from rust-toolchain.toml:

- the always-running test-tooling job exports the channel parsed from
  rust-toolchain.toml as a job output
- query-regression sets RUSTUP_TOOLCHAIN from that output
- the Verify step escapes the pin into the cargo/rustc/active-toolchain
  regexes at runtime; the exact commit hash and date are asserted
  generically since a stable version identifies the release

Removing the redundant require_eq (workflow yaml vs runner env) since
both now flow from the single source of truth. When rust-toolchain.toml
is bumped, the run fails with a clear signal until the runner image is
rebuilt with the new toolchain, keeping the existing lockstep contract.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* ci(query-regression): derive the runner image toolchain from rust-toolchain.toml

Remove the hard-coded 'ARG RUST_TOOLCHAIN=1.96.1' from the
query-regression runner Dockerfile. The pin is now parsed from a
COPY'd rust-toolchain.toml at build time (the bootstrap script builds
with the repo root as context, so the file is in the build context):

- rustup-init installs the parsed channel as the default toolchain
- the baked ENV RUSTUP_TOOLCHAIN is dropped: the rustup default makes
  bare cargo/rustc resolve correctly without it, and the workflow
  supplies RUSTUP_TOOLCHAIN explicitly at run time
- the build-time self-verification asserts the active toolchain
  against the same parsed pin

With this, rust-toolchain.toml is the single source of truth for the
benchmark toolchain end to end: the image bakes whatever the toml says
at build time and the workflow asserts against the toml at run time.
A toolchain bump now only requires rebuilding the image.

The changed mechanics were verified natively with the real rustup-init
1.29.0 and the real 1.96.1 toolchain (registry pulls are unavailable
in this sandbox): parsing, default-toolchain installation without
RUSTUP_TOOLCHAIN env, bare cargo/rustc resolution, and the
active-toolchain assertion for both '(default)' and '(overridden)'
output forms.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* ci(query-regression): rebuild the runner image automatically on toolchain changes

Mirror the dev-builder automation for the query-regression ECS runner
image: a new rebuild-query-regression-runner-image.yaml workflow runs
whenever rust-toolchain.toml or the query-regression runner directory
changes on main (or via manual dispatch). It drives the existing
build-ecs-image.py ops tool, then completes the documented lockstep
updates in order: bump RUNNER_IMAGE_EPOCH in query-regression.yml and
push the commit to main, and only then point the
QUERY_REGRESSION_ECS_IMAGE_ID repo variable at the new image, so the
next regression run picks up image and epoch together.

Also fix build-ecs-image.py to stage rust-toolchain.toml into the
temporary docker build context: the AMI path builds the embedded
Dockerfile from an empty /tmp/image-context, which would break on the
Dockerfile's COPY of rust-toolchain.toml introduced earlier. The
user-data now base64-stages the toml next to the Dockerfile before
docker build.

Verified: py_compile, render_user_data round-trip (mkdir -> stage ->
docker build ordering), and the RUNNER_IMAGE_EPOCH bump sed against
the real workflow file. Requires a new ALIYUN_ECS_BASE_IMAGE_ID repo
variable (Ubuntu 24.04 public image id in the region); all other
secrets/vars are shared with the provisioning job.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* ci: fold the query-regression runner rebuild into release-dev-builder-images.yaml

Merge the standalone rebuild workflow into the existing builder-image
release workflow, as one entry point for all builder artifacts:

- push paths extended with .github/runner-scale-sets/query-regression/**
- new 'release_query_regression_runner_image' dispatch input
- a 'changes' job diffs the pushed range (github.event.before..sha,
  with an everything-changed fallback for dispatch or unknown bases)
  so each expensive rebuild only fires for its own paths:
  rust-toolchain.toml gates both, docker/dev-builder/** gates the
  dev-builder images, the query-regression runner directory gates the
  ECS image rebuild
- the rebuild job itself is unchanged from the standalone workflow
  (build-ecs-image.py, then RUNNER_IMAGE_EPOCH commit to main, then
  the QUERY_REGRESSION_ECS_IMAGE_ID variable update)

The dev-builder jobs, their ECR/CN/tag-update dependents, and the
runner rebuild now share one workflow; the changes filter preserves
the previous on-push behavior for the dev-builder images while the
runner rebuild keeps its own trigger. The epoch-bump commit only
touches query-regression.yml, which is outside the trigger paths, so
no re-trigger loop.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(ci): auto-resolve the ECS base image for the runner rebuild

The automated rebuild failed with 'Missing required configuration:
--base-image-id' because the ALIYUN_ECS_BASE_IMAGE_ID repo variable
does not exist yet (it was flagged as a one-time setup item).

Remove the setup dependency instead: build-ecs-image.py now defaults
--base-image-id to the latest public Ubuntu 24.04 x86_64 system image
in the region (DescribeImages with image_owner_alias=system), so no
manual variable is required. The runner Dockerfile pins every tool
version itself, so base-image drift is low-risk; --base-image-id or
the ALIYUN_ECS_BASE_IMAGE_ID variable still pin a specific base image
deterministically, and the workflow only passes the flag when the
variable is set.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* docs(query-regression): clarify what an image rebuild requires

A routine runner-image rebuild needs no manual file updates: the
rebuild job updates QUERY_REGRESSION_ECS_IMAGE_ID and
RUNNER_IMAGE_EPOCH; the toolchain derives from rust-toolchain.toml;
uv, sccache, otelgen, rustup, and the runner base are pinned by
digest/sha/commit in the Dockerfile. Only an apt package revision
bump (mold, protoc, python3) between rebuilds requires bumping the
corresponding Verify pins, and that failure is loud with the observed
version.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(ci): correct SDK field names in the base-image resolver

DescribeImagesRequest takes 'ostype' (not 'os_type') and the image
items expose 'osname'/'osname_en' (not 'os_name') in the pinned
alibabacloud_ecs20140526 SDK range, so the auto-resolution added in
5a9fd2c769 crashed with a TypeError before describing anything.

Fix the request fields, move the architecture filter server-side, and
paginate (page_size=100 until a short page) instead of relying on a
single default-sized response. Match Ubuntu 24.04 on the localized
osname or the English osname_en.

Verified against the real SDK models (uv run --with
'alibabacloud_ecs20140526>=4.1.0,<6'): a two-page fake client picks
the newest Ubuntu 24.04 via osname_en and rejects 22.04/Windows
decoys.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(ci): correct the repo-root path in build-ecs-image.py

ASSETS_DIR.parent.parent lands on runner-scale-sets, not the repo
root -- the toml lookup failed with FileNotFoundError. The root is
four levels above ecs-image; express it as an explicit REPO_ROOT
constant (ASSETS_DIR.parents[3]).

Verified every path main() reads against the real checkout layout
(Dockerfile, rust-toolchain.toml, start-runner.sh, the systemd unit,
plus REPO_ROOT sanity against Cargo.toml/.git), re-checked the
user-data toml staging round-trip, and re-ran the base-image
resolver regression test.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

2026-09-28 06:08:14 +00:00

8.2 KiB

Raw Blame History

AGENTS.md

Guidance for coding agents (Claude Code, Codex, ...) and contributors working in this repository. CLAUDE.md is a symlink to this file. If .local/AGENTS.md or .local/CLAUDE.md is present, you MUST read it as well — it holds personal or machine-local overrides (gitignored, not shared). To record a personal or machine-local override, write it to .local/AGENTS.md (create .local/ and the file if absent), not to this shared file.

GreptimeDB is an open-source, cloud-native observability database for unified collection and analysis of metrics, logs, and traces. It is written in Rust and provides sub-second querying at PB scale with high cost efficiency.

Core commands

Task	Command
Build (debug)	`make build`
Build (release)	`make build RELEASE=true`
Run standalone	`cargo run -- standalone start`
Targeted Rust tests	`cargo nextest run -p <package>` (preferred over `cargo test`)
Full local Rust suite	`make test`
SQL tests	`cargo sqlness bare` (single case: `cargo sqlness bare -t <name>`)
Format	`make fmt` (and `make fmt-toml` for TOML)
Lint	`make clippy` (= `cargo clippy --workspace --all-targets --all-features -- -D warnings`)
Type check	`make check`

Toolchain: Rust (stable, pinned by rust-toolchain.toml), Protobuf compiler (>= 3.15), C/C++ build essentials. Install the test runner with cargo install cargo-nextest --locked.

Repo map

GreptimeDB is a Cargo workspace rooted at the repository; most crates live under src/ (plus tests-fuzz, tests-integration, tests/runner). Key areas:

Frontend / protocol: src/frontend/ (request orchestration), src/servers/ (wire protocols), src/sql/ (SQL parsing)
Storage engines: src/mito2/ (main time-series engine), src/metric-engine/ (metrics), src/file-engine/
Coordination: src/meta-srv/ (metadata & cluster control), src/meta-client/
Execution / statements: src/operator/ (DDL/DML, request conversion, procedures)
Stream / transform: src/flow/ (continuous aggregation), src/pipeline/
Query / index: src/query/, src/promql/, src/index/
Shared: src/common/, src/datatypes/, src/store-api/ (engine contract), src/catalog/, src/table/

Before editing a path, read every AGENTS.md from the repository root down to that path. The root guide always applies; a nested guide adds or overrides rules for its subtree. Changes spanning multiple subtrees must follow every applicable guide.

AGENTS.md files are maps, not manuals. Keep them short and stable: record where code lives, important boundaries or change-coupling, high-cost gotchas, and the validation entry point. Implementation details belong in code. Do not copy exhaustive test lists or workflow logic; link to the source of truth.

High-change areas and specialized test suites carry their own AGENTS.md with a module map, read/write paths, change-coupling points, and gotchas:

Rust style

Follow docs/style-guide.md for Rust style, module design, naming, and reuse guidance.

Read before changing code

.agents/architecture-invariants.md — repo-wide rules that are easy to violate and expensive to get wrong (persisted/ wire format compatibility, crate layering, async runtimes, error handling, feature gating, the DataFusion fork).
.agents/generated-files.md — tool-generated artifacts that must not be hand-edited (sqlness .result, config/config.md, Grafana dashboards, proto).
docs/style-guide.md — code style.
CONTRIBUTING.md — contribution flow and CLA.

High-signal entry points

Main binary: src/cmd/src/bin/greptime.rs
Configuration: src/common/config/, example TOMLs in config/
Error handling: src/common/error/ (ErrorExt, StatusCode)
Protocol implementations: src/servers/src/

Worktree safety

Check git status --short before editing. Preserve unrelated tracked changes and untracked files; re-read files changed by another process.
Do not rewrite history, force-push, or remove files in bulk unless the user explicitly requests it.
Update generated artifacts only through the generators documented in .agents/generated-files.md.

Validation by change type

Use the narrowest command that covers the change, then expand only when its blast radius requires it.

Change	Minimum validation
One Rust crate	`cargo nextest run -p <package>`
Cross-workspace Rust behavior	`make test`; inspect `.github/workflows/rust.yml` when CI parity matters
SQL parsing, planning, execution, or output	`cargo sqlness bare -t <case>`; inspect regenerated `.result` files
Persisted metadata or wire format	Add/run a case under `tests/compatibility/`; follow its `README.md` and `AGENTS.md`
Public configuration	Update example TOMLs, loading/serialization snapshots, and docs; run `make config-docs`
Query regression harness or DSL	Follow `tests/perf/AGENTS.md`
Enterprise-gated code	Build/test with `--features enterprise` where applicable and run `make check-enterprise-license`

For import/export, COPY, or snapshot-storage changes, follow the Windows portability and validation guidance in tests-integration/AGENTS.md.

Before opening a PR

If you added a .rs, .py, or .ts file, apply and verify its license header with hawkeye format followed by hawkeye check. Use the inception year from licenserc.toml, not the current year.
make fmt
make clippy
make test
make check-udeps (run make fix-udeps if it reports unused dependencies).
If you added or changed a public configuration option, update the applicable example TOMLs, configuration-loading and serialized-config snapshot tests, and related user-facing documentation. Run make config-docs (needs Docker) and commit the regenerated config/config.md.
If you changed a persisted or wire format, add a compatibility test case (see .agents/architecture-invariants.md).
If you added or gated an enterprise-only file, give it the enterprise license header, list it in licenserc-enterprise.toml (includes) and licenserc.toml (excludes), and run make check-enterprise-license.
Use a conventional-commit title, sign off commits (git commit -s), and sign the CLA.
When creating or updating a pull request, follow .github/pull_request_template.md: include the CLA statement, fill the change-intention section with enough detail, and update checklist items accurately. "This PR requires documentation updates" refers to the official docs site repository (GreptimeTeam/docs), not to rustdoc or comments in this repo. Check it only when that site would be wrong or incomplete without a matching change; a change users cannot observe does not need it.
If the change must also land on release branches, add the matching backport-<target> labels (e.g. backport-v1.3 targets release/v1.3). The Backport workflow (.github/workflows/backport.yml) cherry-picks a merged PR onto each target branch and opens a backport PR automatically; on conflicts it aborts and files an issue for a manual backport.

Agent skills and resources: .agents/ (see .agents/README.md)
Architecture decisions: docs/rfcs/
How-to guides: docs/how-to/

8.2 KiB Raw Blame History