Auth mints the live org role as orgRole on sandbox tokens (same claim as billing
tokens, stablyai/orca-cloud#695); the verifier read org_role, so admins would
never have been recognized.
* feat(sidebar): replace footer Settings and Help icons with an account menu
* fix: keep account menu header on primitive spacing; move focus-read allowance
* fix(sidebar): address account menu review findings
* feat(billing): Cloudflare Worker for Stripe checkout, portal, webhooks, entitlements and usage (#7)
Test-mode-only foundation for paid Orca plans: hosted Checkout/Portal,
signed + idempotent Stripe webhooks that re-read subscriptions, a D1 ledger
for sandbox usage with overage-only Billing meter events, and entitlement
checks for the desktop and the sandbox control plane.
* feat(billing): Plan & billing in the Orca account settings and account menu (#9)
Shows the signed-in account's plan, sandbox hours used and left, and opens
hosted Stripe Checkout (Upgrade) or the Customer Portal (Manage billing) in
the browser. Main mints a short-lived billing token from Orca Cloud auth and
only opens checkout.stripe.com / billing.stripe.com URLs. Hidden until
ORCA_BILLING_API_URL is set; the account menu item is gated on the 'billing'
cloud capability and registers through a local adapter that mirrors the
upcoming account menu API.
* feat(billing): plans and prices from config, opt-in Stripe Tax, real account menu
- Worker: BILLING_PLANS JSON overrides the plan catalog (labels, display price,
hours, limits); entitlements return the plan list so the desktop shows no
hardcoded prices. STRIPE_AUTOMATIC_TAX=true adds automatic_tax and tax id
collection to Checkout (default off). Setup script takes prices from env.
- Desktop: register Plan & billing through the sidebar account menu
(merged nwparker/sidebar-account-menu) and drop the local adapter.
- Register the billing Worker's tests with the test-runner check.
* feat(billing): Team pricing, opt-in overage, Stripe test-mode e2e, dev deploy (#18)
* feat(billing): Team-only pricing, opt-in overage with spend cap, live Stripe test-mode e2e
- Catalog: Team $50/seat/mo or $480/seat/yr, 60 Standard h/seat/month pooled
(monthly windows on annual plans), Large 2x, 3 running/seat, 15-min idle sleep;
Enterprise is contact-only. All numbers/labels in BILLING_PLANS defaults.
- Overage is opt-in (adds the metered item), $0.75/h, monthly spend cap (default
$25, admin-raisable) enforced in the meter outbox; 80%/100% usage warnings.
- Checkout: plan/interval/seats with adjustable quantity; portal configuration var.
- Workers reject fetch redirect 'error': use 'manual'. Log route errors.
- scripts/stripe-test-setup.ts creates the test catalog + portal configuration;
scripts/e2e-test-mode.ts drives Checkout (headless, test card), overage, meter,
cap, annual, test-clock renewal, payment failure and cancel.
- Desktop: Team/Enterprise cards with interval + seats, usage warnings, overage
toggle + cap; split into copy/offers/usage modules.
* feat(billing): dev deployment config, dev operator keys, return page
- wrangler.dev.jsonc for orca-billing-dev (workers.dev, dev D1, test-mode prices).
- BILLING_DEV_JWKS (issuer orca-billing-dev-test) accepted only when BILLING_ENV
is not prod, for e2e runs until auth mints billing tokens.
- GET /return: static 'return to Orca' page for Checkout/Portal redirects.
- Design doc: final pricing, dev URL and resources for the control plane.
* feat(billing): team seats follow membership, payment grace + suspend signal, Team UI
Worker
- Seats = org members from Orca Cloud auth (single source of truth; pending invites
never count; min 1): auth pushes POST /v1/internal/seats, billing pulls
/v1/internal/billing/org-members at Checkout and in a reconcile cron, fails closed
(503) if auth is configured but unreachable. No token claim carries a count.
- Proration: adds always_invoice, removals create_prorations (Stripe default).
- Billing admin actions require a token minted in the last 6 minutes.
- Payment failure: past_due_since (migration 0003); new starts stop at once;
after 96 h (paymentGraceHours) reason payment_grace_expired + suspendRunning on
check/usage, and GET /v1/internal/suspensions for the control plane.
- Portal config: monthly<->annual + cancel only (seats follow membership).
- e2e: seat add/remove proration, grace expiry -> suspend -> pay -> resume.
Desktop
- Team section + account menu item: members/roles, invite, change role, remove
with confirm, revoke invites (owners/admins, auth role rules mirrored).
- Payment issue banner with suspension date and Update payment method.
* feat(billing): in-app plans page (#22)
A pricing/plans page reachable from the account menu ("Plan & billing"), the
Settings plan section ("Choose a plan" / "Compare plans"), and any upsell via
openPlansPage(). Everything comes from the billing service's offers: Team
monthly/annual (with the computed annual saving), pooled hours, Large weight,
concurrency, idle sleep, opt-in overage with spend cap, and Enterprise contact.
- States: choose (admin), locked (member), current plan with usage, payment
failure (status-first), checkout in progress, loading and failure with retry.
- Seats are read-only: one per team member (billing re-reads the count).
- Shared card shell keeps both cards' footers level; sliding interval toggle and
price fade respect reduced motion; dialog opens without a pre-focused button.
- Worker offers gain per-seat cents and the Large weight for totals.
- Copy matches the marketing /cloud promises: agents keep running, wake in
seconds, secrets never stored in the VM.
* wip(cloud): Orca Cloud built-in provider wiring
* feat(cloud): no-plan upsell opens the plans page
* feat(cloud): Orca Cloud dogfood fixes (provider persistence, picker label, any-input activity, idle override)
* fix(cloud): live dogfood fixes (snapshot already-deleted, agent egress wildcards, golden image id) + results
* fix(cloud): i18n keys, typed dev JWK, verdict typing
* feat(cloud): billing caps from billing (maxConcurrentSandboxes, per-member), userId on usage; concurrency copy
* feat(cloud): golden image can pin a released orca-server tarball (--release), source build stays the fallback
* fix(cloud): async dev key read, runtime localization catalog
- Seats billed = the period's peak membership: adds above paid seats are charged now
(always_invoice); removals never credit mid-period and only lower the quantity inside
seatTrueDownHours of renewal (proration none), so remove/re-add churn can't dodge or
double-charge. The cron applies deferred decreases from the last pushed count.
- Included hours and caps come from billed seats; per-member concurrency cap
(member_concurrency_limit); usage attributed to the member who started it.
- Start check returns billed `seats` and the derived caps for the control plane.
- Desktop shows when paid seats drop at renewal; proration copy updated.
- Design doc: seat-fairness decision with the GitHub/Cursor/Slack/Linear comparison.
Security (B1): repo tokens were minted for any repo.url where the app was installed. Now an
installation must be bound to the caller's account (signed single-use connect state + OAuth proof
that the installer owns the account or is an org admin), tokens are scoped to the one repo and the
binding is re-checked on cached tokens. Operator binding route for support/dev.
Guardrails (B3/G4): D1 kill switch + static flag refuse create/resume/rollover and drain running
sandboxes; global running cap; 3/seat per-account backstop; per-account create rate limit.
Retention (B4/B6): hourly orphan sweep (stop, then delete after grace; partial listings delete
nothing), 30-day deletion after cancel/unpaid with a T-7 warning hook, 90-day idle retention,
104-day provider snapshot expiry backstop, repo_url tombstoned on destroy.
D1 secrets table (migration 0003), AES-256-GCM per-secret DEK wrapped by SECRETS_KEK with
(scope, owner, name) AAD and key-id rotation; write-only /v1/secrets CRUD with org owner/admin
checks; precedence-resolved injection into orcad's env at every boot (no disk, no rebuild);
optional egress placeholder swap for api.anthropic.com / api.openai.com via the forwardURL proxy.
* fix(cloud): provision bundle download no longer overwrites its source; shortened session deadline for staging rehearsals
* docs(cloud): live Vercel Pro results
Worker
- Seats = org members from Orca Cloud auth (single source of truth; pending invites
never count; min 1): auth pushes POST /v1/internal/seats, billing pulls
/v1/internal/billing/org-members at Checkout and in a reconcile cron, fails closed
(503) if auth is configured but unreachable. No token claim carries a count.
- Proration: adds always_invoice, removals create_prorations (Stripe default).
- Billing admin actions require a token minted in the last 6 minutes.
- Payment failure: past_due_since (migration 0003); new starts stop at once;
after 96 h (paymentGraceHours) reason payment_grace_expired + suspendRunning on
check/usage, and GET /v1/internal/suspensions for the control plane.
- Portal config: monthly<->annual + cancel only (seats follow membership).
- e2e: seat add/remove proration, grace expiry -> suspend -> pay -> resume.
Desktop
- Team section + account menu item: members/roles, invite, change role, remove
with confirm, revoke invites (owners/admins, auth role rules mirrored).
- Payment issue banner with suspension date and Update payment method.
* feat(billing): Team-only pricing, opt-in overage with spend cap, live Stripe test-mode e2e
- Catalog: Team $50/seat/mo or $480/seat/yr, 60 Standard h/seat/month pooled
(monthly windows on annual plans), Large 2x, 3 running/seat, 15-min idle sleep;
Enterprise is contact-only. All numbers/labels in BILLING_PLANS defaults.
- Overage is opt-in (adds the metered item), $0.75/h, monthly spend cap (default
$25, admin-raisable) enforced in the meter outbox; 80%/100% usage warnings.
- Checkout: plan/interval/seats with adjustable quantity; portal configuration var.
- Workers reject fetch redirect 'error': use 'manual'. Log route errors.
- scripts/stripe-test-setup.ts creates the test catalog + portal configuration;
scripts/e2e-test-mode.ts drives Checkout (headless, test card), overage, meter,
cap, annual, test-clock renewal, payment failure and cancel.
- Desktop: Team/Enterprise cards with interval + seats, usage warnings, overage
toggle + cap; split into copy/offers/usage modules.
* feat(billing): dev deployment config, dev operator keys, return page
- wrangler.dev.jsonc for orca-billing-dev (workers.dev, dev D1, test-mode prices).
- BILLING_DEV_JWKS (issuer orca-billing-dev-test) accepted only when BILLING_ENV
is not prod, for e2e runs until auth mints billing tokens.
- GET /return: static 'return to Orca' page for Checkout/Portal redirects.
- Design doc: final pricing, dev URL and resources for the control plane.
- Worker: BILLING_PLANS JSON overrides the plan catalog (labels, display price,
hours, limits); entitlements return the plan list so the desktop shows no
hardcoded prices. STRIPE_AUTOMATIC_TAX=true adds automatic_tax and tax id
collection to Checkout (default off). Setup script takes prices from env.
- Desktop: register Plan & billing through the sidebar account menu
(merged nwparker/sidebar-account-menu) and drop the local adapter.
- Register the billing Worker's tests with the test-runner check.
Test-mode-only foundation for paid Orca plans: hosted Checkout/Portal,
signed + idempotent Stripe webhooks that re-read subscriptions, a D1 ledger
for sandbox usage with overage-only Billing meter events, and entitlement
checks for the desktop and the sandbox control plane.
* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay-ops): make each reserve-watch sample one Cloud Logging read
The project allows 60 ListLogEntries a minute shared by every watch and tool. Each watch sample
made four to six reads (two director runtime-metrics sub-windows, the ledger summaries, the
seat-behind-row lines, the cell's phone metrics, and since #31484 the 503 request log), so on
10-21 18:39Z twelve watches drew 429s until c21 and c22 tripped unverifiable.
A sample now sends its queries as one OR filter and splits the lines back client-side. The
per-instance 503 split counts each director's own logged /v1/assign 503s from the runtime
metrics it already reads, with the rest of the fleet's count as one unattributed instance,
instead of reading the request log.
* fix(relay-ops): judge unattributed 503s on their 3-minute sum against the whole fleet line
* docs(relay-ops): state that an instance that stops logging trips the unattributed rule
* fix(relay-ops): keep one stalled director instance from tripping the reserve watch's 503 rule
The reserve watch's director503 rule counted fleet-wide non-drain 503s. On 10-21 07:55Z one
director instance's database calls stalled for 3 minutes; its 503s alone carried the fleet over
the line and all four watched cells flipped back to db.
Over the fleet line the rule now also needs at least 2 director instances each above a fifth of
it for the same 3 minutes, counted from Cloud Run's request log by labels.instanceId less each
instance's scheduled 503s. Above max(10x background, 200) for 3 minutes it trips with no split.
* fix(relay-ops): size the instance share to the directors seen and count platform 503s
* fix(relay-ops): size the instance share per judged minute and refuse a missing count
The deploy passed `--max`, which is gcloud's service-wide cap. Cloud Run divides
that cap among the revisions taking traffic, so during the 20/80 stage a
candidate under a cap of 5 got one instance while the serving floor kept four,
and the spread check could never see the full floor (deploy M, 1 of 5 for six
minutes). Candidates now pin `--max-instances` per revision, and the deploy
resets a service cap below two full revisions to Cloud Run's default before it
creates any candidate. The rollout overlap of two full directors is already in
the checked Cloud SQL connection budget.
* fix(relay): move director traffic in two steps so a deploy does not land on one instance
Cloud Run routes a new director revision's floor instances their first request
roughly 60-110 s after they start, while a traffic move reaches clients in
15-30 s. The one-step 100% move therefore put every assignment on the single
instance started with the revision, which sits at its own admission limit at
peak; the 2026-10-20 deploy returned ~550 assign 503s in two minutes that way.
With a floor above one, the deploy now moves 20% first, waits until Cloud Run's
request log shows every floor instance serving (six-minute deadline), then
moves the rest. A timeout or failure at that stage returns all traffic to the
previous serving revision, which still holds its warm floor. The staging deploy
identity gains log read access for the same check.
* fix(relay): bound the director spread log read, log the instances seen at timeout, and document split recovery
A hung gcloud log read could hold the 20/80 split past the six-minute deadline;
each read now times out after 30 s and counts as an unread poll. A timeout logs
the distinct instances seen, so a partial share packed onto fewer instances than
the floor shows up. The runbook covers restoring a split left by a cancelled job
and the missing selector-rollback tag after a timed-out deploy.
* fix(relay): judge the reserve watch's 503 rule on non-drain 503s, not a host's own re-dials
The watch counted every director 503, and on 2026-10-18 ~98% were host-rate-limited re-dials that
grow with traffic, so it flipped production-gce-c7 back twice with no fault. It now uses the roll
ladder's non-drain measure (request_count 503s less drain-return deferrals and host-rate-limited /
host-in-flight re-dials from the director's runtime metrics) for both the background and the live
samples. A judged minute no runtime-metrics line covers is unverifiable, not zero.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): state the reserve watch's metrics coverage honestly and share the metrics read limit
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
The folder is the cloud relay that reaches phones; under its old name it read as the legacy SSH
relay. git mv plus import and path-literal rewrites only.
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule
A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to
300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the
first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write
under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip
- The flip back holds no concurrency group: GitHub keeps one pending run per group and a
newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is
re-read and retried (5 tries, backoff) while the switch file still says reserve.
- The 503 background comes from the 10 minutes before a required flipped-at, or an explicit
value; neither is refused.
- A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the
exact break-glass inputs.
- Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable.
- Seat-behind-row key adds the director instance and cell; six segments share one command.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): read the reserve watch's 503 publishing front from every response code
A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy
cell back within minutes. The 503 read now groups request_count by response code in one
query; the newest minute any code has a point for is how far the series is published, so a
503 minute at or before it with no point is zero and minutes past it are left out.
Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start
in later segments, a pending pre-flip background no longer adds to the unverifiable streak,
the flip back retries runtime-status before calling for break glass, and the concurrency
rationale lives once in the docs.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read
- A flip back whose write or plan never finished (no ID token, a failed login, a silent cell)
raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does.
- The first segment reads the Cloud SQL rollout lease through the lease action's own reader and
refuses to start while any run holds it, or when it cannot be read.
- A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the
watch ends refused without a write.
- The duplicate-seat read takes the per-minute ledger summary in the same query and window; with
no fresh summary in it the sample is unverifiable, not zero duplicates.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): page on a refused reserve watch or a failed flip back
A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not
land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write
mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and
alert policy page on any such line, per cell, through relay_alert_notification_channels, and the
deploy identity gets roles/logging.logWriter in production to write it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note a force-cancel can skip the reserve-watch flip-back write
* fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it
A paced drain tells its last host when the window ends, so the restart-safe
wait has to see an empty runtime reaching past that point. It counted a full
window of quiet samples from the first empty one instead, and the runtime only
empties as the window ends, so every same-cap cell waited the window twice:
about 25 minutes per Asia cell at 1200 s on 2026-10-13, and 5 minutes per
300 s cell.
The window is now measured from the verifier's own start, which runs after the
cell accepted the drain and so ends no earlier than the cell's own window. A
pass still needs two consecutive restart-safe samples, the last taken at or
after that end, and any non-zero runtime counter or open migration still
restarts the count. Progress lines report paceWindowRemainingMs in place of the
old sample target.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): drop the same-cap drain timeout's double-wait allowance and stale quiet wording
The drain step's timeout added the window's excess over 300000 only because
restart-safe used to wait a second window after the cell emptied, so it is
now the 15-minute migration lease plus the window for every pace (35 minutes
at 1200000, unchanged at 300000 and below). Docs, workflow comments and the
shadow gate's settledAfterSeconds note now describe the window-end rule. The
always-quiet test starts its clock late, so a window measured from zero
fails it, and the overlapping late-control test is folded away.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state the restart-safe window-end rule exactly
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): the slowest restart-safe wait is now 35 minutes
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): declare Asia spare cell c35 as migration-only
Adds a seventh asia-east2 cell at the C34 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-a, pinned to image F. It gets
its own topology and registration wave but no promotion wave, so the
admission script and workflow refuse to promote it; it stays a
migration-only spare and out of the fleet pool list. Mirrors #25336.
* test(relay): accept the c35 spare under the same-cap capacity allowlist
* fix(relay): keep verifying host tokens from the last served keys while auth is down
jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.
* fix(relay): bound stale relay keys at 30 min and document the emergency purge
Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.
* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
* docs(relay): director Cloud SQL sidecar enablement and rollback, as run on staging
The 0% apply revision is not started by Cloud Run, so the first sidecar boot is the deploy's
rollback candidate. A new log-based metric can take minutes to appear, so its alert policy
can fail the first apply. Both observed on staging 2026-10-11.
* docs(relay): production sidecar enablement needs the tfvars PR, a passed drill and the rollout lease
* feat(relay): slower same-cap drain paces (900000, 1200000) for every cell class
c28's 1,901-host drain at the 300 s pace (~6.3 hosts/s, 2026-10-07) held
database lock time over its bar for about 10 minutes. Add 15- and 20-minute
pace windows to the same-cap closed set, allowed on every cell class.
- The cell's /v1/admin/drain cap rises from 5 to 20 minutes.
- The drain script fails instead of falling back to an unpaced drain when a
cell rejects a window above 300000, since an older image answers both with
the same 400.
- The fast-pace US-only rule and the canary PASS rule apply only below the
default; a canary still authorizes its own pace or slower.
- The restart-safe timeout adds the window's excess over 300000 (quiet starts
after the last host leaves), unchanged at 300000/60000/30000; the job
timeout rises from 90 to 120 minutes.
* fix(relay): refuse a slow drain pace before isolating a cell that cannot accept it
The cell advertises its drain pace cap on /health; the same-cap job reads it
before the isolate, so a pace above an old image's 300000 cap stops with the
cell untouched instead of isolated. Also refreshes two stale timeout comments
and pins the drain step under its one-hour ID token and the job's 70-min
remainder.
* feat(relay): give Asia cell c34 a promotion wave so it can become a general cell
c34 launched on 2026-10-05 as a migration-only spare with no promotion path. This adds
it to the Asia admission promotion waves, the workflow's promote and canary cases, and
the canary evidence map, so the reviewed Asia workflow can promote it with the same
five-minute canary c30 and c31 ran.
The same-cap migration-only list is deliberately unchanged: a same-cap job reads a
cell's class from that list, and c34 must be rolled to the director's image as a
migration-only cell before promotion can run. The list moves after promotion, in its
own change.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* chore(relay): move Asia cell c34 to the general same-cap lists after promotion
A same-cap job reads a cell's class from SAME_CAP_MIGRATION_ONLY_CELLS, not from the
selector. Once c34's Asia canary promotes it, leaving it there would make a same-cap
rollback demote it to migration-only. This moves c34 to the general same-cap list and
the shadow gate's fleet pool list (its pool is the Asia 16).
Merge only after c34's promotion succeeds. This file set is trusted evidence code, so
merging invalidates any sealed monitor or same-cap canary: merge outside a
monitor-to-enable window and before the next same-cap canary seals.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): scope the c34 same-cap pause to the window after promotion
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): rewrap the c34 paragraph
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* feat(relay): give Asia cell c34 a promotion wave so it can become a general cell
c34 launched on 2026-10-05 as a migration-only spare with no promotion path. This adds
it to the Asia admission promotion waves, the workflow's promote and canary cases, and
the canary evidence map, so the reviewed Asia workflow can promote it with the same
five-minute canary c30 and c31 ran.
The same-cap migration-only list is deliberately unchanged: a same-cap job reads a
cell's class from that list, and c34 must be rolled to the director's image as a
migration-only cell before promotion can run. The list moves after promotion, in its
own change.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): scope the c34 same-cap pause to the window after promotion
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): rewrap the c34 paragraph
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* fix(relay): publish the reviewed commit before main can move, and quiet the deploy driver
The publish workflow builds main's head at dispatch. The driver checked main
at preflight but dispatched the publish about four minutes later, after the
inspects and the typed phrase, so a busy main stopped the first real deploy.
It now dispatches the publish seconds after the check, before anything else,
and a build of a moved main stops with the --commit/--publish-run command that
deploys it once reviewed. Typed prompts end in a newline, and runs are
summarised (status changes plus every 5 min) instead of streaming gh run watch.
* fix(relay): a main-moved stop prints only the command that reuses the build
The generic re-run line named the reviewed commit without --publish-run, which
would only build the moved main again.
* feat(relay): add an operator-local driver for director deploys
One command runs the audited director deploy: preflight, pause rehome if
enabled, publish, deploy, optional cell configure, a digest-bound inspect,
the monitor dry-run, and re-enable. It only dispatches the existing
workflows, reads the published digest from the registry and the run log,
reads the monitor verdict from its sealed state, and enables with the
digests gcloud reports after the deploy. It stops at the first failure,
records state, and resumes from it.
* fix(relay): report and own the rehome pause; operator types every phrase
- Read the control back from pause and enable runs whatever their conclusion,
and report PAUSED or UNCONFIRMED loudly.
- Resume re-enables only the pause this driver recorded (generation and run).
- Ctrl-C and SIGTERM print the same state and resume report.
- The operator types every workflow confirmation. A 5-minute soak gated on
director 5xx runs before configure.
- A dry run keeps no state file. The quiet check pages through all runs.
Run IDs come only from the printed URL. One step table drives execute,
dry run and resume. The monitor verdict reuses verify-authority.
* fix(relay): read back only the driver's own rehome run; anchor the soak at the traffic switch
- The pause and enable read-back accepts only the control line its own step's mode prints, at the
generation its own dispatch expected. A run adopted after a crash is settled even when green.
- The soak window opens a minute before the deploy run completed and is read a minute after it
ends, for log ingestion lag.
* fix(relay): say what typing ENABLE_REGIONAL_REHOMING commits to
* fix(relay): the ENABLE prompt also names the 150 s evidence budget
* refactor(relay): derive every deploy decision from live state; no resume machinery
The driver keeps no state between runs. Each run reads the serving
director, its configured cells and the rehome control, and skips what is
already done.
- The only rehome fact it owns is the run that paused rehome. A re-run
names it (--pause-run), and the driver checks it against that run's log
and the live generation.
- A recover-enable line counts as the driver's own pause only with
recovered: true. A director safety pause is never adopted (F3).
- Publish runs before the pause. A fresh run that finds rehome paused
stops unless given --pause-run or --rehome-disabled (F2).
- Interrupts report a pause or enable still in flight as REHOME IS
CHANGING (F1). PAUSE UNCONFIRMED and ENABLE UNCONFIRMED are distinct.
- A tripped soak is judged again on fresh traffic (F5). SIGHUP is
handled, and a pending signal stops the driver before its next dispatch.
- Every stop prints the single command that finishes the deploy.
Removes the state file, --resume, step statuses, monitor adoption and
interrupted-dispatch adoption.
* fix(relay): never report done over an unexplained pause; prove pause ownership by actor
- G1: always read rehome; no early DONE.
- G2: --pause-run must be a rehome-control run by the same user.
- G3: a failed enable run is never an enable.
- C1: a pause or enable is reported as changing from the moment it is
dispatched.
- C2: --leave-rehome-paused (was --rehome-disabled) refuses an enabled
switch. Every printed command parses.
- G4: the enable-in-flight report prints both finishing commands.
- G5: the driver's own runs never block the quiet-lane check.
* test(relay): port the round-3 probes: hard kill mid-pause, unexplained disable after a failed enable
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* feat(relay): report lane service times, 503 causes, per-site hold p99 and lock-wait split
Adds the step-1 observability: drain-return and sticky slot service times,
every /v1/assign 503 by cause with a non-drain total, a p99 per lock hold
site (drain-return regional rows get their own site), a director-side
pg_stat_activity sample that splits lock waiters by director vs cell, and
a log line for each reserved_requests drift reconciliation corrects.
Log-based metrics for the new fields are declared in Terraform, not applied.
* fix(relay): classify a lock waiter by the first relay table its statement names
* fix(relay): attribute lock waiters to the root holder; document the director hold alert change
Every waiter after the first in a row-lock convoy is blocked by the first
waiter, so the sample now walks pg_blocking_pids to the root, reading it once
per waiter. The sampler is single-flight. Drain-return regional holds feeding
the non-paging director hold policy is documented as expected.
* docs(relay): note the lock-wait sample undercounts director waiters when the pool is full
* feat(relay): drain pace window as a reviewed same-cap input, with drain-aware 503 gates
The same-cap roll drained every cell over a fixed 300 s window, so a US roll
re-placed hosts at ~2/s and spent ~10 minutes draining and waiting for quiet.
The window is now a dispatch input from a closed set (300000, 60000, 30000),
defaulting to today's 300000.
- Below the default is refused for anything but US general cells; Asia drains
are bound by their targets' accept rate, and migration-only cells hold no hosts.
- A non-default window must be named in the confirmation, the canary authority
records it (v2), and a batch may run at its canary's window or slower only.
- Each cell job re-checks the window, scales the restart-safe timeout with it
(15-min lease + window, unchanged at the default), and records what the cell
applied and when it settled.
- The report-only shadow gate takes the director's drain-return deferrals out
of the 503 count (window and baselines), adds the rung's 5-min non-drain 503
budget and a Retry-After check, and reports the measured re-placement rate.
- relay-workflows.md documents the pace ladder and what each rung records.
* fix(relay): judge paced drains on counted 503s against the pre-drain minutes, and seal the canary's pace verdict
Review of #25639 replayed the shadow gate: it read 10-01 c29 as unverified (the
log read stopped at 20k entries), false-blocked 10-02 c22, and was blind on four
cells whose 24 h/48 h baseline held an incident.
- Director 503s now come from Cloud Run's request_count, aligned per minute by
Cloud Monitoring, so volume cannot truncate the count.
- The background is the median of the 10 same-day minutes before the drain;
the 24 h/48 h baselines are gone.
- Scheduled 503s come out: drain-return deferrals and sticky/placement answers
to a host's own early retry (host-rate-limited, host-in-flight), each split
across the minutes its 30 s sample covers.
- The rung budget counts only sustained breaches: two straight minutes over
max(1.5x, +20) warn, over max(2x, +40) would-block.
- The report carries a paceVerdict over the three pace checks. seal_canary
downloads the canary cell's report and seals that verdict; a batch below the
default pace needs PASS, from a report on the same cell that drained at that
pace.
- Docs: the step-down rule reads paceVerdict, 30 s waits for the lane service
time (#25645), and the staging step is dropped since staging drains unpaced.
Replayed read-only: 10-01 c29 would-block (9 minutes over 41.5/min); all nine
10-02 cells and 10-01 c25 paceVerdict PASS.
* fix(relay): a partial count already past a block line blocks, in the shadow gate's Cloud SQL and pool checks
A truncated FATAL count is a floor, and one runtime sample over the SQL-failure
line is a fact, so neither waits for a complete read. The waiter-run rule still
needs a complete run, since holes can join two runs into one.
* fix(relay): a canary pace PASS needs a real cohort and whole telemetry; one median-based 503 check
From the final review of #25639:
- canaryPaceVerdict seals PASS only from a report that drained at least 400
hosts (about half a 10-02 US cell), so a near-empty canary cannot authorize
a fast batch.
- A director-metrics sub-window with fewer samples than one instance emits
is unverified, so an empty or short Logging answer is never a calm drain.
- seal_canary names the shadow artifact, report path and cell from the gate's
normalized cell list, as cell_1 uploads it.
- director503 folds into nonDrain503Budget as a single-minute spike rule,
max(10x median, 200), dropping the pre-drain peak statistic. All 30 replayed
windows keep their verdicts.
Adds a sixth asia-east2 cell at the C31 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-c, pinned to the f30b5cb1
cell image. It gets its own topology and registration wave but no
promotion wave, so the admission script and workflow refuse to promote
it; it stays a migration-only landing zone and out of the fleet pool list.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
Both were promoted to general on 2026-10-01 (selector 344 shows them general),
but the same-cap job still classed them migration-only. Rollback mode on either
would isolate (general -> migration-only) before its no-op check failed, demoting
a live cell. They stay out of the fleet pool list (pool 10, not 16).
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* refactor(relay): sample fleet health inside the same-cap roll instead of a separate monitor run
A same-cap wave no longer consumes a 15-minute monitor dry-run and its sealed,
single-use, five-minute-fresh evidence. Each apply wave now samples fleet health
itself right before isolation, with the monitor's evaluator, thresholds, and
tolerances, for a window sized to the cell's host count (3/5/8 min), plus three
lookback rules: no cell container exit in 10 min, no minute over 500 director
503s in 10 min, and director concurrency p99 within the monitor bar over 4 min.
Removes the monitor-run inputs, the gate's consume/authorize steps, the
break-glass override, and the same-cap-only authorization shapes in
relay-monitor-evidence.mjs. The monitor workflow and the rehome enable path are
unchanged.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): bound the pre-drain sample overrun and keep the drain token fresh
Review follow-ups: alternating tolerated readings could hold the sample open
until its step timeout, so cap the overrun at three samples past the window;
record why a read failed; mint a fresh admin ID token for the drain after the
sample; raise the job timeout to 90 min so a long sample cannot cancel the
job past the failsafe.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* feat(relay): exempt the rolled cell and existing-only cells from the pre-drain crash rule
The exit rule counted every relay container exit fleet-wide, so a cell that
crashes every few hours (c25, 12 a week) blocked the very roll that fixes it,
and existing-only legacy cells (c5, 15 a week) blocked rolls they take no part
in. Exits are now grouped by instance, each instance is named by its own newest
runtime-metrics log line, and only exits on general or migration-only cells
other than the target count. An exit no configured cell can be named for trips
the rule; a failed lookup is a failed read. relay-observability.tf joins the
evidence-code set because the rule depends on its filter.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* test(relay): cover re-asking for an unnamed exiting instance; note the boot-exit risk
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* feat(relay): declare US cells c32 and c33 at the 3,000-host shape
Declares two us-central1 cells at the Asia shape (cap 3000, 6000 request
units, e2-standard-4) with the US default pool of 10, and generalises the
Asia topology and admission ladder to derive each wave's region from its
reviewed zone, leaving every Asia wave's behaviour unchanged.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* docs(relay): note the US canary tie-break and leave the fleet pool list to promotion
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): plan C32 and C33 as one topology wave
The live-image overlay refuses a declared non-target cell with no template,
so a lone C32 plan would fail on C33. Registration and promotion stay
one cell at a time.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): accept MIG version-name reconciliation and recreate stranded cells without rewriting the MIG
The stranded-rollback recovery ran a gcloud rolling action, which renames the
MIG version outside Terraform. Every later plan for that cell then reverted the
label, and the capacity-plan validator refused the revert as an unreviewed MIG
change, so the cell could be neither rolled nor rolled back.
The validator now accepts a MIG field moving back to what relay-gce-cells.tf
declares (version name and update policy), in every mode, and a test pins those
values to the Terraform file. The stranded branch recreates the cell's single
instance with recreate-instances, which leaves the MIG untouched.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): let a label-only MIG plan through and recreate on it in a stranded rollback
A stranded rollback whose template is already in place plans only the version
name revert. The validator still required the MIG template to move, so that
plan was refused, and the recreate gate (changes == 0) would have skipped a
plan of one change and left the drain flag set. Require the template move only
when no declared field reconciles, and recreate whenever the template was not
replaced (changes < 2).
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): anchor same-cap monitor evidence freshness to the run's authorisation, not job startup
The same-cap gate now verifies the dry-run on its own clock and records the
authorisation instant in the single-use consumed marker; each cell job checks
the evidence age at that instant and bounds its own start after it.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): refuse a re-run same-cap gate before it consumes evidence; tighten order tests
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
The parent workflow collapses canary-apply and batch-apply into the job mode
apply, so the headroom step's canary-apply/batch-apply condition never held and
the gate was skipped on every real roll. Run it wherever the drain runs (apply,
rollback before its restart) and in read-only verify; skip only a resumed
rollback, which drains nothing. A new workflow-shape test fails on any job
step comparing against a mode the parent cannot pass, and on a drain that can
run without the headroom check.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
Neil
Neil
Jinwoo Hong
