* ci: make query regression non-blocking for scheduled nightly releases
Scheduled nightly releases no longer gate publishing on the query
regression release test. The test still runs (validation policy stays
'all' for automatic releases) and uploads its report, but any result -
failure, timeout, cancellation, or skip - no longer blocks image
publishing or the GitHub release for schedule-triggered runs.
Since reusable-workflow caller jobs cannot use continue-on-error, the
gate is relaxed at the consumers instead: the if conditions of
release-images-to-dockerhub and publish-github-release now also pass
when github.event_name == 'schedule' regardless of the query-regression
result. All other gates (runner allocation, artifact builds,
prepare-release-validation) are unchanged.
Tag-push and manual-dispatch releases remain fully blocking unless a
skip policy is explicitly chosen via the release_validation input.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: preserve downstream nightly release jobs after regression failures
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit a9ce6e2d01)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Previously the nightly release was scheduled only on Mondays, so a
failed run left users without nightly builds for a whole week.
Now the schedule triggers every weekday at 00:00 UTC, but the release
only proceeds when the latest published nightly release is older than
NIGHTLY_RELEASE_MAX_AGE_DAYS (5) days. The check runs in
allocate-runners before any EC2 runner is allocated, and gates all
schedule-driven jobs (including the Slack notification, so a skipped
nightly is silent). Tag pushes and manual dispatches are unaffected.
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit a7f99b3425)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: cut integration test time and make the storage matrix meaningful
tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.
- Point the PR CI object-store matrix at the MinIO already started by
tests-integration/fixtures. Three GT_S3_* consumers did not read
GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
release integration-test container. The release previously ran every
remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
/tmp/greptimedb_cache, which the datanode wipes on startup, so a
starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
non-trivial SST representation: JSON/JSON2 columns, native histograms,
metric-engine logical tables, and tables carrying fulltext or skipping
indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
shorten it in the event tests, which otherwise wait a 5s window per DDL
they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
test_grpc_message_size_ok and is now rewritten to assert the negotiated
grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
prefixes of their copy_from siblings; two standalone/distributed event
test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
in a parallelism-dependent order, so its last digits are unstable; the
test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
ttl=7d with 2025 timestamps, so its rows were only readable as long as
they stayed in the memtable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: address review — wire nightly real-S3 job into check-status, keep the short event interval
The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.
In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The scheduled (nightly) release version was created by appending
'-nightly-YYYYMMDD' to NEXT_RELEASE_VERSION as-is. When the Cargo.toml
version carries a pre-release extension (e.g. v1.3.0-alpha.1), this
produced invalid tags like 'v1.3.0-alpha.1-nightly-20260907', stacking
'nightly' on top of the 'alpha.1' pre-release.
Strip the pre-release extension first so 'nightly' itself becomes the
only pre-release extension: 'v1.3.0-alpha.1' -> 'v1.3.0-nightly-20260908'.
Stable versions are unaffected; nightly-build ('nightly-YYYYMMDD-sha'),
dev-build, tag push, and manual dispatch paths are unchanged.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(ci): add aliyun ecs ephemeral runner path for query regression
Signed-off-by: paomian <xpaomian@gmail.com>
* fix: improve condition for query-regression job execution in workflow
* feat: update Docker installation to use official repository and add GPG key handling
* Refactor query regression runner setup and configuration
- Removed deprecated PersistentVolumeClaim for build cache.
- Introduced a new bootstrap script for setting up the ECS runner host.
- Deleted obsolete Helm values files for runner configuration.
- Updated the Aliyun ECS runner provisioning script to reflect new cache paths.
- Modified GitHub workflows to use the new Aliyun ECS runner setup.
- Adjusted documentation to clarify the new runner lifecycle and provisioning process.
* fix: enhance runner service management during bootstrap process
* fix: update alibabacloud_tea_openapi dependency version in metadata
* feat: enhance ECS runner scripts with region_id and resource_group_id support
* fix: move containerd content store to data root for improved storage management
* feat: rename query-regression runner to ephemeral-github runner and update related scripts
* fix: update sentinel polling method to use serial console output for improved reliability
* fix: add environment variable checks for Alibaba Cloud access keys in ECS client
* fix: improve error handling in GitHub API requests for better diagnostics
* fix: improve cache disk detection logic for Aliyun ECS instances
* fix: enhance cache disk waiting logic with detailed output and error handling
* fix: update dependency version for alibabacloud_tea_openapi in teardown script
* fix: enhance cache disk waiting logic for better compatibility and clarity
* fix: enhance console output handling and add incremental logging during instance provisioning
* fix: add PATH environment variable for runner jobs in service and provision script
* fix: add machine telemetry sampling and logging during query regression jobs
* fix: update query regression documentation and provision script for cache disk handling
* fix: update SCCACHE_CACHE_SIZE validation to 10G for improved caching efficiency
* fix: remove outdated cache size checks and cleanup logic for fresh system disk runs
* fix: enhance instance deletion logic with region handling and console output export
* fix: add swap file setup and OOM handling for ECS runner to improve stability
* fix: update OOM handling and service restart logic for ECS runner to enhance stability
* fix: increase system disk size to 100 GiB for cold double nightly builds to prevent ENOSPC errors
* fix: increase system disk size to 150 GiB for ECS runner to prevent ENOSPC errors
* fix: add keep_instance option to preserve ECS instance for post-mortem debugging
* fix: disable unattended upgrades to prevent job cancellations during library updates
* fix: reduce system disk size to 40 GiB for ECS runner to prevent ENOSPC errors
* feat: Refactor Aliyun ECS runner provisioning and introduce nightly regression comparison
- Update `aliyun-ecs-runner-provision.py` to remove cache disk handling, simplifying the provisioning process.
- Introduce `query-regression-nightly-refs.py` to resolve and compare SHAs from successful nightly builds.
- Create `query-regression-nightly.yml` workflow to trigger nightly comparisons based on successful builds.
- Enhance `query-regression.yml` to include a `test-tooling` job for validating Python scripts before provisioning.
- Update tests for the new nightly reference selection logic and refactor existing tests to align with the new caching strategy.
- Modify documentation to reflect changes in caching and nightly comparison workflows.
* fix: enhance runner image tool verification with detailed checks
* fix: improve error handling in runner image tool verification
* fix: update tool versions in ECS image and workflow for consistency
* fix: correct typo in error message for unparseable ECS creation time
* fix: update README and workflow files for query regression tests and image hygiene
---------
Signed-off-by: paomian <xpaomian@gmail.com>
* feat: add riscv64 cross-build support
Add the missing build infrastructure for riscv64gc-unknown-linux-gnu.
The codebase itself already compiles cleanly for riscv64 (verified with
`cargo check --workspace --target riscv64gc-unknown-linux-gnu`): all
architecture-sensitive dependencies support it (tikv-jemalloc-sys,
aws-lc-sys, ring, pprof, simd-json).
- .cargo/config.toml: set riscv64-linux-gnu-gcc as the linker for the
riscv64gc-unknown-linux-gnu target
- rust.yml: add a check-riscv64 CI job that cross-checks the whole
workspace to prevent regressions from future dependency changes
- docker/dev-builder/riscv64/Dockerfile: new cross dev-builder image
with gcc/g++-riscv64-linux-gnu and the riscv64 rust target
- Makefile: add dev-builder-riscv64 and build-riscv64-bin targets
Verified end-to-end: the produced riscv64 binary starts in standalone
mode under qemu and serves SQL (create/insert/select) over HTTP.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* ci: build riscv64 artifacts in the release workflow
- release.yml: add build-linux-riscv64-artifacts job that cross-compiles
greptime for riscv64gc-unknown-linux-gnu on the amd64 runner with the
dev-builder-riscv64 image, and uploads greptime-linux-riscv64-*
artifacts. The job is wired into the needs of publish-github-release,
release-cn-artifacts and stop-linux-amd64-runner. Integration tests
are skipped since the cross-compiled binary cannot run on the host.
- release-dev-builder-images.yaml + build-dev-builder-images action:
build and push the dev-builder-riscv64 image to DockerHub, and sync
it to ECR and ACR via skopeo like the other dev-builder images.
- Makefile: DEV_BUILDER_RISCV64_IMAGE_TAG now defaults to
DEV_BUILDER_IMAGE_TAG so the existing tag-bump automation keeps the
riscv64 image tag in sync.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: forward cargo extension in riscv64 build
Pass CARGO_EXTENSION through build-riscv64-bin just like the existing
build-by-dev-builder target, so wrappers such as sccache are preserved
inside the cross-build container.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* ci: check riscv64 release feature graph
Check all workspace targets with the servers/dashboard feature enabled so
the riscv64 CI job covers the same optional dependency graph used by the
release artifact build.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: gate riscv64 latest tags to main pushes
Manual dev-builder workflow dispatches now publish only their immutable
version tag. Update DockerHub and ECR latest tags only for the workflow's
main-branch push event, preventing feature-branch builds from replacing
the shared latest image.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* docs: include riscv64 in release input description
Update the build_linux_artifacts workflow input description to reflect
that it now triggers amd64, arm64, and riscv64 artifact builds.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: fall back when riscv64 builder is unpublished
Before building a release artifact, pull the pinned RISC-V dev-builder
from ECR. If the image has not been published yet, build the same tag
locally from the current Dockerfile so releases remain unblocked during
the builder-tag update window.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* ci: automate compatibility version window
Signed-off-by: discord9 <discord9@163.com>
* ci: address compat version window review feedback
Address all four review comments on the compat version window
automation:
- Keep the PR window to the sliding window only (latest patch of the two
newest stable minor lines). Exact =vX.Y.Z anchors from case.toml are no
longer unioned into from_versions; they are validated by the new
--check-anchors mode and exercised by nightly runs via --nightly-window.
- Add --published-only: the window is computed over stable git tags that
have a published, non-draft GitHub release carrying the sqlness compat
artifacts (greptime-linux-amd64 tar.gz and sha256sum), so failed releases
cannot land in the window.
- Run the updater Python tests plus the window/anchor consistency check in
PR and merge-group CI (new compat-updater-check job in integration.yml).
- Regenerate tests/compatibility/ci.toml to the current sliding window
[v1.0.2, v1.1.4].
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* ci: automatically update helm-charts when release
* Update .github/workflows/release.yml
Co-authored-by: Ning Sun <classicning@gmail.com>
* Update update-helm-charts-version.sh
---------
Co-authored-by: Ning Sun <classicning@gmail.com>
* ci: update website greptimedb version when releasing automatically
* fix: token name
* chore: tweak readme
* fix: style
* chore: license year
* refactor: simplify bump-website-version.ts
* chore: being used
* fix: make ci happy
Ning Sun
dennis zhuang
liyang
Weny Xu
localhost
Lei, HUANG
discord9
LFC
Ruihang Xia
yinheli
Zhenchi
liyang
yihong
Yiran
zyy17
tison