mirror of
https://github.com/neondatabase/neon.git
synced 2025-12-27 08:09:58 +00:00
ca9d8761ff
## Problem
We want to move some benchmarks from hetzner runners to aws graviton
runners
## Summary of changes
Adjust the runner labels for some workflows.
Adjust the pagebench number of clients to match the latecny knee at 8
cores of the new instance type
Add `--security-opt seccomp=unconfined` to docker run command to bypass
IO_URING EPERM error.
## New runners
https://us-east-2.console.aws.amazon.com/ec2/home?region=us-east-2#Instances:instanceState=running;search=:github-unit-perf-runner-arm;v=3;$case=tags:true%5C,client:false;$regex=tags:false%5C,client:false;sort=tag:Name
## Important Notes
I added the run-benchmarks label to get this tested **before we merge
it**.
[See](https://github.com/neondatabase/neon/actions/runs/15974141360)
I also test a run of pagebench with the new setup from this branch, see
https://github.com/neondatabase/neon/actions/runs/15972523054
- Update: the benchmarking workflow had failures, [see]
(https://github.com/neondatabase/neon/actions/runs/15974141360/job/45055897591)
- changed docker run command to avoid io_uring EPERM error, new run
[see](https://github.com/neondatabase/neon/actions/runs/15997965633/job/45125689920?pr=12393)
Update: the pagebench test run on the new runner [completed
successfully](https://github.com/neondatabase/neon/actions/runs/15972523054/job/45046772556)
Update 2025-07-07: the latest runs with instance store ext4 have been
successful and resolved the direct I/O issues we have been seeing before
in some runs. We only had one perf testcase failing (shard split) that
had been flaky before. So I think we can merge this now.
## Follow up
if this is merged and works successfully we must create a separate issue
to de-provision the hetzner unit-perf runners defined
[here](91a41729af/ansible/inventory/hosts_metal (L111))
84 lines
2.7 KiB
YAML
84 lines
2.7 KiB
YAML
name: Periodic proxy performance test on unit-perf-aws-arm runners
|
|
|
|
on:
|
|
push: # TODO: remove after testing
|
|
branches:
|
|
- test-proxy-bench # Runs on pushes to branches starting with test-proxy-bench
|
|
# schedule:
|
|
# * is a special character in YAML so you have to quote this string
|
|
# ┌───────────── minute (0 - 59)
|
|
# │ ┌───────────── hour (0 - 23)
|
|
# │ │ ┌───────────── day of the month (1 - 31)
|
|
# │ │ │ ┌───────────── month (1 - 12 or JAN-DEC)
|
|
# │ │ │ │ ┌───────────── day of the week (0 - 6 or SUN-SAT)
|
|
# - cron: '0 5 * * *' # Runs at 5 UTC once a day
|
|
workflow_dispatch: # adds an ability to run this manually
|
|
|
|
defaults:
|
|
run:
|
|
shell: bash -euo pipefail {0}
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}
|
|
cancel-in-progress: false
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
run_periodic_proxybench_test:
|
|
permissions:
|
|
id-token: write # aws-actions/configure-aws-credentials
|
|
statuses: write
|
|
contents: write
|
|
pull-requests: write
|
|
runs-on: [self-hosted, unit-perf-aws-arm]
|
|
timeout-minutes: 60 # 1h timeout
|
|
container:
|
|
image: ghcr.io/neondatabase/build-tools:pinned-bookworm
|
|
credentials:
|
|
username: ${{ github.actor }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
options: --init
|
|
steps:
|
|
- name: Checkout proxy-bench Repo
|
|
uses: actions/checkout@v4
|
|
with:
|
|
repository: neondatabase/proxy-bench
|
|
path: proxy-bench
|
|
|
|
- name: Set up the environment which depends on $RUNNER_TEMP on nvme drive
|
|
id: set-env
|
|
shell: bash -euxo pipefail {0}
|
|
run: |
|
|
PROXY_BENCH_PATH=$(realpath ./proxy-bench)
|
|
{
|
|
echo "PROXY_BENCH_PATH=$PROXY_BENCH_PATH"
|
|
echo "NEON_DIR=${RUNNER_TEMP}/neon"
|
|
echo "TEST_OUTPUT=${PROXY_BENCH_PATH}/test_output"
|
|
echo ""
|
|
} >> "$GITHUB_ENV"
|
|
|
|
- name: Run proxy-bench
|
|
run: ${PROXY_BENCH_PATH}/run.sh
|
|
|
|
- name: Ingest Bench Results # neon repo script
|
|
if: always()
|
|
run: |
|
|
mkdir -p $TEST_OUTPUT
|
|
python $NEON_DIR/scripts/proxy_bench_results_ingest.py --out $TEST_OUTPUT
|
|
|
|
- name: Push Metrics to Proxy perf database
|
|
if: always()
|
|
env:
|
|
PERF_TEST_RESULT_CONNSTR: "${{ secrets.PROXY_TEST_RESULT_CONNSTR }}"
|
|
REPORT_FROM: $TEST_OUTPUT
|
|
run: $NEON_DIR/scripts/generate_and_push_perf_report.sh
|
|
|
|
- name: Docker cleanup
|
|
if: always()
|
|
run: docker compose down
|
|
|
|
- name: Notify Failure
|
|
if: failure()
|
|
run: echo "Proxy bench job failed" && exit 1 |