* feat(ci): add aliyun ecs ephemeral runner path for query regression
Signed-off-by: paomian <xpaomian@gmail.com>
* fix: improve condition for query-regression job execution in workflow
* feat: update Docker installation to use official repository and add GPG key handling
* Refactor query regression runner setup and configuration
- Removed deprecated PersistentVolumeClaim for build cache.
- Introduced a new bootstrap script for setting up the ECS runner host.
- Deleted obsolete Helm values files for runner configuration.
- Updated the Aliyun ECS runner provisioning script to reflect new cache paths.
- Modified GitHub workflows to use the new Aliyun ECS runner setup.
- Adjusted documentation to clarify the new runner lifecycle and provisioning process.
* fix: enhance runner service management during bootstrap process
* fix: update alibabacloud_tea_openapi dependency version in metadata
* feat: enhance ECS runner scripts with region_id and resource_group_id support
* fix: move containerd content store to data root for improved storage management
* feat: rename query-regression runner to ephemeral-github runner and update related scripts
* fix: update sentinel polling method to use serial console output for improved reliability
* fix: add environment variable checks for Alibaba Cloud access keys in ECS client
* fix: improve error handling in GitHub API requests for better diagnostics
* fix: improve cache disk detection logic for Aliyun ECS instances
* fix: enhance cache disk waiting logic with detailed output and error handling
* fix: update dependency version for alibabacloud_tea_openapi in teardown script
* fix: enhance cache disk waiting logic for better compatibility and clarity
* fix: enhance console output handling and add incremental logging during instance provisioning
* fix: add PATH environment variable for runner jobs in service and provision script
* fix: add machine telemetry sampling and logging during query regression jobs
* fix: update query regression documentation and provision script for cache disk handling
* fix: update SCCACHE_CACHE_SIZE validation to 10G for improved caching efficiency
* fix: remove outdated cache size checks and cleanup logic for fresh system disk runs
* fix: enhance instance deletion logic with region handling and console output export
* fix: add swap file setup and OOM handling for ECS runner to improve stability
* fix: update OOM handling and service restart logic for ECS runner to enhance stability
* fix: increase system disk size to 100 GiB for cold double nightly builds to prevent ENOSPC errors
* fix: increase system disk size to 150 GiB for ECS runner to prevent ENOSPC errors
* fix: add keep_instance option to preserve ECS instance for post-mortem debugging
* fix: disable unattended upgrades to prevent job cancellations during library updates
* fix: reduce system disk size to 40 GiB for ECS runner to prevent ENOSPC errors
* feat: Refactor Aliyun ECS runner provisioning and introduce nightly regression comparison
- Update `aliyun-ecs-runner-provision.py` to remove cache disk handling, simplifying the provisioning process.
- Introduce `query-regression-nightly-refs.py` to resolve and compare SHAs from successful nightly builds.
- Create `query-regression-nightly.yml` workflow to trigger nightly comparisons based on successful builds.
- Enhance `query-regression.yml` to include a `test-tooling` job for validating Python scripts before provisioning.
- Update tests for the new nightly reference selection logic and refactor existing tests to align with the new caching strategy.
- Modify documentation to reflect changes in caching and nightly comparison workflows.
* fix: enhance runner image tool verification with detailed checks
* fix: improve error handling in runner image tool verification
* fix: update tool versions in ECS image and workflow for consistency
* fix: correct typo in error message for unparseable ECS creation time
* fix: update README and workflow files for query regression tests and image hygiene
---------
Signed-off-by: paomian <xpaomian@gmail.com>
* refactor: port query regression runner to Rust
Signed-off-by: discord9 <discord9@163.com>
* ci: remove optional OTLP report plotter
Signed-off-by: discord9 <discord9@163.com>
* refactor: split query regression runner into modules
Signed-off-by: discord9 <discord9@163.com>
* style: use crate-qualified imports in query regression runner
Signed-off-by: discord9 <discord9@163.com>
* refactor: simplify query regression runner internals
Signed-off-by: discord9 <discord9@163.com>
* feat: abstract inspect-footer storage access behind object store destination
Add an optional --destination <TOML> to inspect-footer (and
--base-destination/--candidate-destination to finalize-remote) so the
storage inspection reads DB data files through the opendal-backed
object_store abstraction instead of bare std::fs. Local paths keep
working unchanged via the --root shortcut (File backend); remote
backends (S3/GCS/...) are described by a DestinationConfig TOML
reusing the object-store crate's ObjectStoreConfig serde shape.
- inspect_footer: list via ObjectStore::list + ObjectMeta filtering
(parquet keys, non-zero size, metadata/ segment), read footers
async via ParquetObjectReader + ParquetMetaDataReader with known
file size (no extra HEAD); output JSON schema unchanged
- finalize-remote: --base-data-home/--candidate-data-home become
optional, mutually exclusive with the new --*-destination args
- cmd deps: add object_store_opendal + datafusion_object_store
- tests: fs-backend list+footer integration tests (metadata filtering,
destination TOML mode, root/destination exclusivity)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style: drop needless borrow in inspect footer test
Fix clippy::needless_borrows_for_generic_args in the inspect-footer test
(fs::create_dir_all(table.join("metadata"))). Missed by the earlier
focused clippy run because it only covered --bin targets.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): use two pointers for sliding range boundaries
Replace the stale cursor heuristic in RangeManipulateStream::calculate_range
with monotonic left/right cursors. The old path rescanned each evaluation
window (O(E x samples-per-window)) and could lose valid samples after sparse
gaps or trailing empty windows. The two pointers keep strict monotonic
progress, reducing boundary generation to O(N + E) while preserving
(curr-range, curr] semantics, start/end shortening, and empty-window output.
Controlled release benchmarks (fixed CPU, ABBA):
- Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s,
~96% at 1h/15s.
- Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end;
shorter windows stayed within run-order noise.
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): specialize changes/resets with adaptive edge counting
The generic range_fn macro slices, downcasts, and rescans every overlapping
window for changes() and resets(). Replace the macro path for these two
functions with hand-written UDF wrappers backed by a shared private
edge-count kernel: direct raw-offset scans when requested edges are few,
otherwise one global u64 edge prefix so each window is answered by a prefix
difference.
Behavior is preserved bit-for-bit, including raw null-buffer values, NaN
semantics, signed zero, infinities, empty/singleton windows, independent
timestamp/value offsets, arbitrary window layouts, and exact DataFusion
error messages. The shared proc macro, planner, serializer, and other range
functions are untouched.
Controlled release benchmarks (fixed CPU, ABBA):
- Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time.
- Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster.
- Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client
and 12.0-20.9% server latency improvement; controls stayed within drift.
Signed-off-by: discord9 <discord9@163.com>
* ci(query-regression): include PromQL range boundary case in defaults
An audit of historical query-regression runs found zero range-query
coverage: all 208 PromQL ANALYZE samples were bare selectors, so range
evaluation could regress without CI noticing. Wire the
promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so
label-triggered runs measure the range path. The case is cheap: a ~0.3s
synthetic fixture and about a minute of query execution per base/candidate
pass.
Signed-off-by: discord9 <discord9@163.com>
* chore(promql): address sliding range review nits
Move test-only imports into their test modules and remove the unused
pre-specialization changes and resets helpers.
Signed-off-by: discord9 <discord9@163.com>
* style(promql): apply pinned rustfmt
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover sparse range results
Share the changes and resets test scaffolding while keeping their behavior
oracles independent. Add an end-to-end sqlness regression for sparse samples,
empty intermediate windows, and a valid trailing sample.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
localhost
discord9
shuiyisong