mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-13 00:42:14 +00:00
stack/insert-skip-wal-feature
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 localhost
|
4d65e8984a |
chore(ci): Implement /query-regression command handling and admission workflow (#8975)
* Implement `/query-regression` command handling and admission workflow - Add `query-regression-slash.py` script for processing `/query-regression` commands in PR comments, validating case arguments, and checking permissions. - Update `checks.yml` to include tests for the new slash command functionality. - Modify `query-regression-comment.yml` to trigger on the new `Query Regression Command` workflow. - Create `query-regression-slash.yml` to handle the dispatched command, validate allowlist and permissions, and initiate the regression workflow. - Enhance `query-regression.yml` to support additional inputs for PR admission and SHA verification. - Introduce `slash-command-dispatch.yml` to parse and dispatch commands from PR comments. - Document the new command admission process in `AGENTS.md` and `README.md`. - Add unit tests in `test_query_regression_slash.py` to cover command parsing and admission logic. * refactor: enhance query-regression command handling with comment validation and identity checks * feat: implement admission identity handling for query regression workflows * refactor: update PR admission logic in query regression workflow * refactor: update token usage in slash command dispatch and README for clarity * test: add cases for handling re-run failed jobs and stale runner artifacts * refactor: improve repository metadata handling in query regression scripts * chore: enable overwrite for artifact uploads to handle re-run failed jobs * chore: enable overwrite for query regression admission uploads * feat: enhance query-regression admission with HMAC signing and verification - Introduced HMAC signing for admission markers in query-regression workflows to ensure integrity and authenticity. - Updated `query-regression-comment.test.cjs` to include tests for signing and verifying admission markers. - Modified `query-regression-slash.py` to handle admission marker signing and verification, including checks for dispatch sender and head SHA consistency. - Enhanced workflows to securely manage admission markers and HMAC secrets, ensuring they are not exposed to untrusted contexts. - Improved documentation to clarify the admission process and the role of HMAC in securing the workflow. * test: add case to find newly posted marker among newer comments * test: add case to verify multiline output handling in write_outputs function |
||
|
localhost
|
35ea88a4ef |
feat(ci): run query regression on ephemeral Aliyun ECS runners (#8937)
* feat(ci): add aliyun ecs ephemeral runner path for query regression Signed-off-by: paomian <xpaomian@gmail.com> * fix: improve condition for query-regression job execution in workflow * feat: update Docker installation to use official repository and add GPG key handling * Refactor query regression runner setup and configuration - Removed deprecated PersistentVolumeClaim for build cache. - Introduced a new bootstrap script for setting up the ECS runner host. - Deleted obsolete Helm values files for runner configuration. - Updated the Aliyun ECS runner provisioning script to reflect new cache paths. - Modified GitHub workflows to use the new Aliyun ECS runner setup. - Adjusted documentation to clarify the new runner lifecycle and provisioning process. * fix: enhance runner service management during bootstrap process * fix: update alibabacloud_tea_openapi dependency version in metadata * feat: enhance ECS runner scripts with region_id and resource_group_id support * fix: move containerd content store to data root for improved storage management * feat: rename query-regression runner to ephemeral-github runner and update related scripts * fix: update sentinel polling method to use serial console output for improved reliability * fix: add environment variable checks for Alibaba Cloud access keys in ECS client * fix: improve error handling in GitHub API requests for better diagnostics * fix: improve cache disk detection logic for Aliyun ECS instances * fix: enhance cache disk waiting logic with detailed output and error handling * fix: update dependency version for alibabacloud_tea_openapi in teardown script * fix: enhance cache disk waiting logic for better compatibility and clarity * fix: enhance console output handling and add incremental logging during instance provisioning * fix: add PATH environment variable for runner jobs in service and provision script * fix: add machine telemetry sampling and logging during query regression jobs * fix: update query regression documentation and provision script for cache disk handling * fix: update SCCACHE_CACHE_SIZE validation to 10G for improved caching efficiency * fix: remove outdated cache size checks and cleanup logic for fresh system disk runs * fix: enhance instance deletion logic with region handling and console output export * fix: add swap file setup and OOM handling for ECS runner to improve stability * fix: update OOM handling and service restart logic for ECS runner to enhance stability * fix: increase system disk size to 100 GiB for cold double nightly builds to prevent ENOSPC errors * fix: increase system disk size to 150 GiB for ECS runner to prevent ENOSPC errors * fix: add keep_instance option to preserve ECS instance for post-mortem debugging * fix: disable unattended upgrades to prevent job cancellations during library updates * fix: reduce system disk size to 40 GiB for ECS runner to prevent ENOSPC errors * feat: Refactor Aliyun ECS runner provisioning and introduce nightly regression comparison - Update `aliyun-ecs-runner-provision.py` to remove cache disk handling, simplifying the provisioning process. - Introduce `query-regression-nightly-refs.py` to resolve and compare SHAs from successful nightly builds. - Create `query-regression-nightly.yml` workflow to trigger nightly comparisons based on successful builds. - Enhance `query-regression.yml` to include a `test-tooling` job for validating Python scripts before provisioning. - Update tests for the new nightly reference selection logic and refactor existing tests to align with the new caching strategy. - Modify documentation to reflect changes in caching and nightly comparison workflows. * fix: enhance runner image tool verification with detailed checks * fix: improve error handling in runner image tool verification * fix: update tool versions in ECS image and workflow for consistency * fix: correct typo in error message for unparseable ECS creation time * fix: update README and workflow files for query regression tests and image hygiene --------- Signed-off-by: paomian <xpaomian@gmail.com> |
||
 discord9
|
1693b2727c |
refactor: port query regression runner to Rust (#8651)
* refactor: port query regression runner to Rust Signed-off-by: discord9 <discord9@163.com> * ci: remove optional OTLP report plotter Signed-off-by: discord9 <discord9@163.com> * refactor: split query regression runner into modules Signed-off-by: discord9 <discord9@163.com> * style: use crate-qualified imports in query regression runner Signed-off-by: discord9 <discord9@163.com> * refactor: simplify query regression runner internals Signed-off-by: discord9 <discord9@163.com> * feat: abstract inspect-footer storage access behind object store destination Add an optional --destination <TOML> to inspect-footer (and --base-destination/--candidate-destination to finalize-remote) so the storage inspection reads DB data files through the opendal-backed object_store abstraction instead of bare std::fs. Local paths keep working unchanged via the --root shortcut (File backend); remote backends (S3/GCS/...) are described by a DestinationConfig TOML reusing the object-store crate's ObjectStoreConfig serde shape. - inspect_footer: list via ObjectStore::list + ObjectMeta filtering (parquet keys, non-zero size, metadata/ segment), read footers async via ParquetObjectReader + ParquetMetaDataReader with known file size (no extra HEAD); output JSON schema unchanged - finalize-remote: --base-data-home/--candidate-data-home become optional, mutually exclusive with the new --*-destination args - cmd deps: add object_store_opendal + datafusion_object_store - tests: fs-backend list+footer integration tests (metadata filtering, destination TOML mode, root/destination exclusivity) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: drop needless borrow in inspect footer test Fix clippy::needless_borrows_for_generic_args in the inspect-footer test (fs::create_dir_all(table.join("metadata"))). Missed by the earlier focused clippy run because it only covered --bin targets. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
a7590f8174 |
perf(promql): avoid repeated scans in sliding range evaluation (#8646)
* perf(promql): use two pointers for sliding range boundaries Replace the stale cursor heuristic in RangeManipulateStream::calculate_range with monotonic left/right cursors. The old path rescanned each evaluation window (O(E x samples-per-window)) and could lose valid samples after sparse gaps or trailing empty windows. The two pointers keep strict monotonic progress, reducing boundary generation to O(N + E) while preserving (curr-range, curr] semantics, start/end shortening, and empty-window output. Controlled release benchmarks (fixed CPU, ABBA): - Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s, ~96% at 1h/15s. - Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end; shorter windows stayed within run-order noise. Signed-off-by: discord9 <discord9@163.com> * perf(promql): specialize changes/resets with adaptive edge counting The generic range_fn macro slices, downcasts, and rescans every overlapping window for changes() and resets(). Replace the macro path for these two functions with hand-written UDF wrappers backed by a shared private edge-count kernel: direct raw-offset scans when requested edges are few, otherwise one global u64 edge prefix so each window is answered by a prefix difference. Behavior is preserved bit-for-bit, including raw null-buffer values, NaN semantics, signed zero, infinities, empty/singleton windows, independent timestamp/value offsets, arbitrary window layouts, and exact DataFusion error messages. The shared proc macro, planner, serializer, and other range functions are untouched. Controlled release benchmarks (fixed CPU, ABBA): - Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time. - Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster. - Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client and 12.0-20.9% server latency improvement; controls stayed within drift. Signed-off-by: discord9 <discord9@163.com> * ci(query-regression): include PromQL range boundary case in defaults An audit of historical query-regression runs found zero range-query coverage: all 208 PromQL ANALYZE samples were bare selectors, so range evaluation could regress without CI noticing. Wire the promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so label-triggered runs measure the range path. The case is cheap: a ~0.3s synthetic fixture and about a minute of query execution per base/candidate pass. Signed-off-by: discord9 <discord9@163.com> * chore(promql): address sliding range review nits Move test-only imports into their test modules and remove the unused pre-specialization changes and resets helpers. Signed-off-by: discord9 <discord9@163.com> * style(promql): apply pinned rustfmt Signed-off-by: discord9 <discord9@163.com> * test(promql): cover sparse range results Share the changes and resets test scaffolding while keeping their behavior oracles independent. Add an end-to-end sqlness regression for sparse samples, empty intermediate windows, and a valid trailing sample. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
457a3f317e |
ci: add heavy query regression label (#8619)
Signed-off-by: discord9 <discord9@163.com> |
||
 shuiyisong
|
f5f5d468bb |
ci: add OTLP trace ingestion regression testing (#8631)
* chore: update CI config Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add CI Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update CI config Signed-off-by: shuiyisong <xixing.sys@gmail.com> * ci: report otelgen runner diagnostics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add script to draw result diagram Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
02283b6ba0 |
test(perf): add remote write storage inspection (#8444)
* feat: add remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: extend remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: inspect remote write parquet storage Signed-off-by: discord9 <discord9@163.com> * test: normalize remote write perf fixtures Signed-off-by: discord9 <discord9@163.com> * test: add heavy remote write perf case Signed-off-by: discord9 <discord9@163.com> * test: use head greptime for read bench Signed-off-by: discord9 <discord9@163.com> * test: keep heavy remote write case local Signed-off-by: discord9 <discord9@163.com> * test: tune remote write perf smoke case Signed-off-by: discord9 <discord9@163.com> * test: fix query fixture import style Signed-off-by: discord9 <discord9@163.com> * test: cover remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: add integer counter perf case Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
12f83828b1 |
feat: add Prom remote-write query regression scenario (#8413)
* feat: add Prom remote-write query regression scenario Signed-off-by: discord9 <discord9@163.com> * test: add high-cardinality remote-write query case Signed-off-by: discord9 <discord9@163.com> * feat: chunk remote-write query regression loads Signed-off-by: discord9 <discord9@163.com> * test: use multi-day remote-write regression case Signed-off-by: discord9 <discord9@163.com> * fix: address query regression review comments Signed-off-by: discord9 <discord9@163.com> * ci: allow large query regression comments Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
c44f8da646 |
feat: add query regression perf harness (#8406)
* feat: add query regression perf harness Signed-off-by: discord9 <discord9@163.com> * feat: extend query regression cases Signed-off-by: discord9 <discord9@163.com> * ci: harden query regression workflows Signed-off-by: discord9 <discord9@163.com> * fix: address query regression review comments Signed-off-by: discord9 <discord9@163.com> * ci: limit query regression PR triggers Signed-off-by: discord9 <discord9@163.com> * ci: run full query regression case set Signed-off-by: discord9 <discord9@163.com> * refactor: model query regression scenarios Signed-off-by: discord9 <discord9@163.com> * fix: avoid unenforced query regression thresholds Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |