starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-10-09 08:02:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(relay): let the same-cap roll's post-roll verify outlast one DB stall (#26372)

Browse Source 
* fix(relay): let the same-cap roll's post-roll verify outlast one DB stall

The verify step's admin reads gave up after three 2 s retries; on 2026-10-07
c18 failed its roll on 503 'no healthy upstream' during a DB stall and was
healthy 16 s later. The step now retries 5xx for up to 60 s, which covers a
stall plus readiness grace and two LB health checks, and still fails a cell
that stays down. 4xx stays final.

* test(relay): surface curl stderr when the verify retry test fails

* fix(relay): retry the post-roll verify reads in bash, since curl 7.81 ignores --retry under --fail-with-body

* fix(relay): never report a stale body after a refused verify read; stop pinning inert curl retry flags
This commit is contained in:
Jinwoo Hong
2026-10-07 21:00:44 -04:00
committed by GitHub
parent bac9e0c2bf
commit 23a25eaa58
2 changed files with 124 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
+24 -12
View File
@@ -716,19 +716,31 @@ jobs:
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
# A single transient 5xx (LB warm-up behind a fresh instance) must not
# fail a canary; 4xx (auth, generation mismatch) still fails fast.
# Rides out one DB stall: 5-7 s, then readiness grace and two 10 s LB
# health checks (c18, 10-07 18:49Z: 503 "no healthy upstream", healthy
# 16 s later). A cell still down after 60 s fails; other 4xx fail fast.
# Retried in this loop: the runner's 7.81 client (Ubuntu 22.04) never
# retries under --fail-with-body, so its --retry flags were inert.
admin_post() {
local out="${RUNNER_TEMP}/$1.json"
if ! curl --fail-with-body --max-time 30 \
--retry 3 --retry-delay 2 --retry-connrefused --output "${out}" \
--request POST "$2" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data "$3"; then
cat "${out}" >&2
return 1
fi
cat "${out}"
local out="${RUNNER_TEMP}/$1.json" window=60 delay=5 status
local deadline=$((SECONDS + window))
while true; do
# A refused or timed-out attempt writes no body; never report the last one's.
rm -f "${out}"
status="$(curl --silent --show-error --max-time 30 --output "${out}" \
--write-out '%{http_code}' --request POST "$2" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data "$3")" || status=000
case "${status}" in
2??) cat "${out}"; return 0 ;;
000|408|429|5??) ((SECONDS + delay < deadline)) || break ;;
*) break ;;
esac
sleep "${delay}"
done
echo "admin_post $1: HTTP ${status}" >&2
cat "${out}" >&2 2>/dev/null || true
return 1
}
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \