mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-03 18:45:35 +00:00
dennis zhuang
f3eb8e6c72
* test: cut integration test time and make the storage matrix meaningful
tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.
- Point the PR CI object-store matrix at the MinIO already started by
tests-integration/fixtures. Three GT_S3_* consumers did not read
GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
release integration-test container. The release previously ran every
remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
/tmp/greptimedb_cache, which the datanode wipes on startup, so a
starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
non-trivial SST representation: JSON/JSON2 columns, native histograms,
metric-engine logical tables, and tables carrying fulltext or skipping
indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
shorten it in the event tests, which otherwise wait a 5s window per DDL
they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
test_grpc_message_size_ok and is now rewritten to assert the negotiated
grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
prefixes of their copy_from siblings; two standalone/distributed event
test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
in a parallelism-dependent order, so its last digits are unstable; the
test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
ttl=7d with 2025 timestamps, so its rows were only readable as long as
they stayed in the memtable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: address review — wire nightly real-S3 job into check-status, keep the short event interval
The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.
In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
122 lines
4.4 KiB
YAML
122 lines
4.4 KiB
YAML
name: Build linux artifacts
|
|
description: Build linux artifacts
|
|
inputs:
|
|
arch:
|
|
description: Architecture to build
|
|
required: true
|
|
cargo-profile:
|
|
description: Cargo profile to build
|
|
required: true
|
|
version:
|
|
description: Version of the artifact
|
|
required: true
|
|
disable-run-tests:
|
|
description: Disable running integration tests
|
|
required: true
|
|
dev-mode:
|
|
description: Enable dev mode, only build standard greptime
|
|
required: false
|
|
default: "false"
|
|
image-namespace:
|
|
description: Image Namespace
|
|
required: true
|
|
image-registry:
|
|
description: Image Registry
|
|
required: true
|
|
working-dir:
|
|
description: Working directory to build the artifacts
|
|
required: false
|
|
default: .
|
|
large-page-size:
|
|
description: Build GreptimeDB with large page size (65536).
|
|
required: false
|
|
default: 'false'
|
|
aws-ci-test-bucket:
|
|
description: Bucket used by the object-store integration tests
|
|
required: false
|
|
default: ''
|
|
aws-region:
|
|
description: Region of the test bucket
|
|
required: false
|
|
default: ''
|
|
aws-access-key-id:
|
|
description: Access key id for the test bucket
|
|
required: false
|
|
default: ''
|
|
aws-secret-access-key:
|
|
description: Secret access key for the test bucket
|
|
required: false
|
|
default: ''
|
|
runs:
|
|
using: composite
|
|
steps:
|
|
- name: Run integration test
|
|
if: ${{ inputs.disable-run-tests == 'false' }}
|
|
shell: bash
|
|
# NOTE: If the BUILD_JOBS > 4, it's always OOM in EC2 instance.
|
|
# Without GT_S3_*, every remote object-store case is skipped and the
|
|
# release only ever exercises the local file backend.
|
|
env:
|
|
GT_S3_BUCKET: ${{ inputs.aws-ci-test-bucket }}
|
|
GT_S3_REGION: ${{ inputs.aws-region }}
|
|
GT_S3_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }}
|
|
GT_S3_ACCESS_KEY: ${{ inputs.aws-secret-access-key }}
|
|
run: |
|
|
cd ${{ inputs.working-dir }} && \
|
|
make run-it-in-container BUILD_JOBS=4 \
|
|
IMAGE_NAMESPACE=${{ inputs.image-namespace }} \
|
|
IMAGE_REGISTRY=${{ inputs.image-registry }}
|
|
|
|
- name: Upload sqlness logs
|
|
if: ${{ failure() && inputs.disable-run-tests == 'false' }} # Only upload logs when the integration tests failed.
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: sqlness-logs
|
|
path: /tmp/greptime-*.log
|
|
retention-days: 3
|
|
|
|
- name: Build greptime # Builds standard greptime binary
|
|
uses: ./.github/actions/build-greptime-binary
|
|
with:
|
|
base-image: ubuntu
|
|
features: servers/dashboard
|
|
cargo-profile: ${{ inputs.cargo-profile }}
|
|
artifacts-dir: greptime-linux-${{ inputs.arch }}-${{ inputs.version }}
|
|
version: ${{ inputs.version }}
|
|
working-dir: ${{ inputs.working-dir }}
|
|
image-registry: ${{ inputs.image-registry }}
|
|
image-namespace: ${{ inputs.image-namespace }}
|
|
large-page-size: ${{ inputs.large-page-size }}
|
|
|
|
- name: Clean up the target directory # Clean up the target directory for the centos7 base image, or it will still use the objects of last build.
|
|
shell: bash
|
|
run: |
|
|
rm -rf ./target/
|
|
|
|
- name: Build greptime on centos base image
|
|
uses: ./.github/actions/build-greptime-binary
|
|
if: ${{ inputs.arch == 'amd64' && inputs.dev-mode == 'false' }} # Builds greptime for centos if the host machine is amd64.
|
|
with:
|
|
base-image: centos
|
|
features: servers/dashboard
|
|
cargo-profile: ${{ inputs.cargo-profile }}
|
|
artifacts-dir: greptime-linux-${{ inputs.arch }}-centos-${{ inputs.version }}
|
|
version: ${{ inputs.version }}
|
|
working-dir: ${{ inputs.working-dir }}
|
|
image-registry: ${{ inputs.image-registry }}
|
|
image-namespace: ${{ inputs.image-namespace }}
|
|
large-page-size: ${{ inputs.large-page-size }}
|
|
|
|
- name: Build greptime on android base image
|
|
uses: ./.github/actions/build-greptime-binary
|
|
if: ${{ inputs.arch == 'amd64' && inputs.dev-mode == 'false' }} # Builds arm64 greptime binary for android if the host machine amd64.
|
|
with:
|
|
base-image: android
|
|
artifacts-dir: greptime-android-arm64-${{ inputs.version }}
|
|
version: ${{ inputs.version }}
|
|
working-dir: ${{ inputs.working-dir }}
|
|
build-android-artifacts: true
|
|
image-registry: ${{ inputs.image-registry }}
|
|
image-namespace: ${{ inputs.image-namespace }}
|
|
large-page-size: ${{ inputs.large-page-size }}
|