The golden image ships Ubuntu's docker.io, docker-compose-v2 and docker-buildx (pinned in the
manifest; daemon.json: overlay2, local log driver with caps, mirror.gcr.io registry mirror).
dockerd runs rootful inside the workspace's microVM and is never started at boot: the new
/opt/orca/bin/docker shim (image/vm/docker, first on PATH) starts it on the first command that
needs it (0.3 s in the spike), says once what differs from a laptop (containers stop on sleep;
credential hosts need the proxy CA mounted), then execs plain docker. An image without Docker
says so instead of waiting.
Off switch: SANDBOX_DOCKER=off (Worker var, default on) sends ORCA_DOCKER=off on every boot;
orca-sandbox-start leaves ~/.orca-sandbox/docker.off and the shim refuses with a plain message.
Starts without the flag (retries, repairs) keep the VM's last answer.
Egress: CONTAINER_REGISTRY_HOSTS (mirror.gcr.io, Docker Hub, GHCR) join the default allowlist.
Docs: cloud/docs/sandbox-docker.md (incl. the proxy-CA mount). Golden rotation carries only
this change, with the Worker deployed from the same commit (runbook 1.8a).
SECRETS_KEK_SOURCE=kms unwraps KMS-wrapped KEKs once per isolate. Unset, the
SECRETS_KEK path is unchanged. With it on there is no plaintext fallback: a KMS
failure answers 503 secrets_unavailable and a workspace create fails instead of
starting without secrets. Tested against a fake KMS only.
Operator routes GET /v1/internal/secrets/kek and POST /v1/internal/secrets/rewrap
move every envelope-encrypted row (secrets, agent_credentials, github_user_tokens)
from a retired KEK to the active one: idempotent, resumable, conditional writes,
counts-only logs. A v2 row format behind SECRETS_SEAL_FORMAT=v2 (default v1)
keeps the KEK id out of the value layer's AAD so rotation re-wraps data keys
without decrypting values. Adds the KMS design doc.
Auth mints the live org role as orgRole on sandbox tokens (same claim as billing
tokens, stablyai/orca-cloud#695); the verifier read org_role, so admins would
never have been recognized.
* feat(sidebar): replace footer Settings and Help icons with an account menu
* fix: keep account menu header on primitive spacing; move focus-read allowance
* fix(sidebar): address account menu review findings
* feat(billing): Cloudflare Worker for Stripe checkout, portal, webhooks, entitlements and usage (#7)
Test-mode-only foundation for paid Orca plans: hosted Checkout/Portal,
signed + idempotent Stripe webhooks that re-read subscriptions, a D1 ledger
for sandbox usage with overage-only Billing meter events, and entitlement
checks for the desktop and the sandbox control plane.
* feat(billing): Plan & billing in the Orca account settings and account menu (#9)
Shows the signed-in account's plan, sandbox hours used and left, and opens
hosted Stripe Checkout (Upgrade) or the Customer Portal (Manage billing) in
the browser. Main mints a short-lived billing token from Orca Cloud auth and
only opens checkout.stripe.com / billing.stripe.com URLs. Hidden until
ORCA_BILLING_API_URL is set; the account menu item is gated on the 'billing'
cloud capability and registers through a local adapter that mirrors the
upcoming account menu API.
* feat(billing): plans and prices from config, opt-in Stripe Tax, real account menu
- Worker: BILLING_PLANS JSON overrides the plan catalog (labels, display price,
hours, limits); entitlements return the plan list so the desktop shows no
hardcoded prices. STRIPE_AUTOMATIC_TAX=true adds automatic_tax and tax id
collection to Checkout (default off). Setup script takes prices from env.
- Desktop: register Plan & billing through the sidebar account menu
(merged nwparker/sidebar-account-menu) and drop the local adapter.
- Register the billing Worker's tests with the test-runner check.
* feat(billing): Team pricing, opt-in overage, Stripe test-mode e2e, dev deploy (#18)
* feat(billing): Team-only pricing, opt-in overage with spend cap, live Stripe test-mode e2e
- Catalog: Team $50/seat/mo or $480/seat/yr, 60 Standard h/seat/month pooled
(monthly windows on annual plans), Large 2x, 3 running/seat, 15-min idle sleep;
Enterprise is contact-only. All numbers/labels in BILLING_PLANS defaults.
- Overage is opt-in (adds the metered item), $0.75/h, monthly spend cap (default
$25, admin-raisable) enforced in the meter outbox; 80%/100% usage warnings.
- Checkout: plan/interval/seats with adjustable quantity; portal configuration var.
- Workers reject fetch redirect 'error': use 'manual'. Log route errors.
- scripts/stripe-test-setup.ts creates the test catalog + portal configuration;
scripts/e2e-test-mode.ts drives Checkout (headless, test card), overage, meter,
cap, annual, test-clock renewal, payment failure and cancel.
- Desktop: Team/Enterprise cards with interval + seats, usage warnings, overage
toggle + cap; split into copy/offers/usage modules.
* feat(billing): dev deployment config, dev operator keys, return page
- wrangler.dev.jsonc for orca-billing-dev (workers.dev, dev D1, test-mode prices).
- BILLING_DEV_JWKS (issuer orca-billing-dev-test) accepted only when BILLING_ENV
is not prod, for e2e runs until auth mints billing tokens.
- GET /return: static 'return to Orca' page for Checkout/Portal redirects.
- Design doc: final pricing, dev URL and resources for the control plane.
* feat(billing): team seats follow membership, payment grace + suspend signal, Team UI
Worker
- Seats = org members from Orca Cloud auth (single source of truth; pending invites
never count; min 1): auth pushes POST /v1/internal/seats, billing pulls
/v1/internal/billing/org-members at Checkout and in a reconcile cron, fails closed
(503) if auth is configured but unreachable. No token claim carries a count.
- Proration: adds always_invoice, removals create_prorations (Stripe default).
- Billing admin actions require a token minted in the last 6 minutes.
- Payment failure: past_due_since (migration 0003); new starts stop at once;
after 96 h (paymentGraceHours) reason payment_grace_expired + suspendRunning on
check/usage, and GET /v1/internal/suspensions for the control plane.
- Portal config: monthly<->annual + cancel only (seats follow membership).
- e2e: seat add/remove proration, grace expiry -> suspend -> pay -> resume.
Desktop
- Team section + account menu item: members/roles, invite, change role, remove
with confirm, revoke invites (owners/admins, auth role rules mirrored).
- Payment issue banner with suspension date and Update payment method.
* feat(billing): in-app plans page (#22)
A pricing/plans page reachable from the account menu ("Plan & billing"), the
Settings plan section ("Choose a plan" / "Compare plans"), and any upsell via
openPlansPage(). Everything comes from the billing service's offers: Team
monthly/annual (with the computed annual saving), pooled hours, Large weight,
concurrency, idle sleep, opt-in overage with spend cap, and Enterprise contact.
- States: choose (admin), locked (member), current plan with usage, payment
failure (status-first), checkout in progress, loading and failure with retry.
- Seats are read-only: one per team member (billing re-reads the count).
- Shared card shell keeps both cards' footers level; sliding interval toggle and
price fade respect reduced motion; dialog opens without a pre-focused button.
- Worker offers gain per-seat cents and the Large weight for totals.
- Copy matches the marketing /cloud promises: agents keep running, wake in
seconds, secrets never stored in the VM.
* wip(cloud): Orca Cloud built-in provider wiring
* feat(cloud): no-plan upsell opens the plans page
* feat(cloud): Orca Cloud dogfood fixes (provider persistence, picker label, any-input activity, idle override)
* fix(cloud): live dogfood fixes (snapshot already-deleted, agent egress wildcards, golden image id) + results
* fix(cloud): i18n keys, typed dev JWK, verdict typing
* feat(cloud): billing caps from billing (maxConcurrentSandboxes, per-member), userId on usage; concurrency copy
* feat(cloud): golden image can pin a released orca-server tarball (--release), source build stays the fallback
* fix(cloud): async dev key read, runtime localization catalog
- Seats billed = the period's peak membership: adds above paid seats are charged now
(always_invoice); removals never credit mid-period and only lower the quantity inside
seatTrueDownHours of renewal (proration none), so remove/re-add churn can't dodge or
double-charge. The cron applies deferred decreases from the last pushed count.
- Included hours and caps come from billed seats; per-member concurrency cap
(member_concurrency_limit); usage attributed to the member who started it.
- Start check returns billed `seats` and the derived caps for the control plane.
- Desktop shows when paid seats drop at renewal; proration copy updated.
- Design doc: seat-fairness decision with the GitHub/Cursor/Slack/Linear comparison.
Security (B1): repo tokens were minted for any repo.url where the app was installed. Now an
installation must be bound to the caller's account (signed single-use connect state + OAuth proof
that the installer owns the account or is an org admin), tokens are scoped to the one repo and the
binding is re-checked on cached tokens. Operator binding route for support/dev.
Guardrails (B3/G4): D1 kill switch + static flag refuse create/resume/rollover and drain running
sandboxes; global running cap; 3/seat per-account backstop; per-account create rate limit.
Retention (B4/B6): hourly orphan sweep (stop, then delete after grace; partial listings delete
nothing), 30-day deletion after cancel/unpaid with a T-7 warning hook, 90-day idle retention,
104-day provider snapshot expiry backstop, repo_url tombstoned on destroy.
D1 secrets table (migration 0003), AES-256-GCM per-secret DEK wrapped by SECRETS_KEK with
(scope, owner, name) AAD and key-id rotation; write-only /v1/secrets CRUD with org owner/admin
checks; precedence-resolved injection into orcad's env at every boot (no disk, no rebuild);
optional egress placeholder swap for api.anthropic.com / api.openai.com via the forwardURL proxy.
* fix(cloud): provision bundle download no longer overwrites its source; shortened session deadline for staging rehearsals
* docs(cloud): live Vercel Pro results
Worker
- Seats = org members from Orca Cloud auth (single source of truth; pending invites
never count; min 1): auth pushes POST /v1/internal/seats, billing pulls
/v1/internal/billing/org-members at Checkout and in a reconcile cron, fails closed
(503) if auth is configured but unreachable. No token claim carries a count.
- Proration: adds always_invoice, removals create_prorations (Stripe default).
- Billing admin actions require a token minted in the last 6 minutes.
- Payment failure: past_due_since (migration 0003); new starts stop at once;
after 96 h (paymentGraceHours) reason payment_grace_expired + suspendRunning on
check/usage, and GET /v1/internal/suspensions for the control plane.
- Portal config: monthly<->annual + cancel only (seats follow membership).
- e2e: seat add/remove proration, grace expiry -> suspend -> pay -> resume.
Desktop
- Team section + account menu item: members/roles, invite, change role, remove
with confirm, revoke invites (owners/admins, auth role rules mirrored).
- Payment issue banner with suspension date and Update payment method.
* feat(billing): Team-only pricing, opt-in overage with spend cap, live Stripe test-mode e2e
- Catalog: Team $50/seat/mo or $480/seat/yr, 60 Standard h/seat/month pooled
(monthly windows on annual plans), Large 2x, 3 running/seat, 15-min idle sleep;
Enterprise is contact-only. All numbers/labels in BILLING_PLANS defaults.
- Overage is opt-in (adds the metered item), $0.75/h, monthly spend cap (default
$25, admin-raisable) enforced in the meter outbox; 80%/100% usage warnings.
- Checkout: plan/interval/seats with adjustable quantity; portal configuration var.
- Workers reject fetch redirect 'error': use 'manual'. Log route errors.
- scripts/stripe-test-setup.ts creates the test catalog + portal configuration;
scripts/e2e-test-mode.ts drives Checkout (headless, test card), overage, meter,
cap, annual, test-clock renewal, payment failure and cancel.
- Desktop: Team/Enterprise cards with interval + seats, usage warnings, overage
toggle + cap; split into copy/offers/usage modules.
* feat(billing): dev deployment config, dev operator keys, return page
- wrangler.dev.jsonc for orca-billing-dev (workers.dev, dev D1, test-mode prices).
- BILLING_DEV_JWKS (issuer orca-billing-dev-test) accepted only when BILLING_ENV
is not prod, for e2e runs until auth mints billing tokens.
- GET /return: static 'return to Orca' page for Checkout/Portal redirects.
- Design doc: final pricing, dev URL and resources for the control plane.
- Worker: BILLING_PLANS JSON overrides the plan catalog (labels, display price,
hours, limits); entitlements return the plan list so the desktop shows no
hardcoded prices. STRIPE_AUTOMATIC_TAX=true adds automatic_tax and tax id
collection to Checkout (default off). Setup script takes prices from env.
- Desktop: register Plan & billing through the sidebar account menu
(merged nwparker/sidebar-account-menu) and drop the local adapter.
- Register the billing Worker's tests with the test-runner check.
Test-mode-only foundation for paid Orca plans: hosted Checkout/Portal,
signed + idempotent Stripe webhooks that re-read subscriptions, a D1 ledger
for sandbox usage with overage-only Billing meter events, and entitlement
checks for the desktop and the sandbox control plane.
* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay-ops): make each reserve-watch sample one Cloud Logging read
The project allows 60 ListLogEntries a minute shared by every watch and tool. Each watch sample
made four to six reads (two director runtime-metrics sub-windows, the ledger summaries, the
seat-behind-row lines, the cell's phone metrics, and since #31484 the 503 request log), so on
10-21 18:39Z twelve watches drew 429s until c21 and c22 tripped unverifiable.
A sample now sends its queries as one OR filter and splits the lines back client-side. The
per-instance 503 split counts each director's own logged /v1/assign 503s from the runtime
metrics it already reads, with the rest of the fleet's count as one unattributed instance,
instead of reading the request log.
* fix(relay-ops): judge unattributed 503s on their 3-minute sum against the whole fleet line
* docs(relay-ops): state that an instance that stops logging trips the unattributed rule
* fix(relay-ops): keep one stalled director instance from tripping the reserve watch's 503 rule
The reserve watch's director503 rule counted fleet-wide non-drain 503s. On 10-21 07:55Z one
director instance's database calls stalled for 3 minutes; its 503s alone carried the fleet over
the line and all four watched cells flipped back to db.
Over the fleet line the rule now also needs at least 2 director instances each above a fifth of
it for the same 3 minutes, counted from Cloud Run's request log by labels.instanceId less each
instance's scheduled 503s. Above max(10x background, 200) for 3 minutes it trips with no split.
* fix(relay-ops): size the instance share to the directors seen and count platform 503s
* fix(relay-ops): size the instance share per judged minute and refuse a missing count
The deploy passed `--max`, which is gcloud's service-wide cap. Cloud Run divides
that cap among the revisions taking traffic, so during the 20/80 stage a
candidate under a cap of 5 got one instance while the serving floor kept four,
and the spread check could never see the full floor (deploy M, 1 of 5 for six
minutes). Candidates now pin `--max-instances` per revision, and the deploy
resets a service cap below two full revisions to Cloud Run's default before it
creates any candidate. The rollout overlap of two full directors is already in
the checked Cloud SQL connection budget.
* fix(relay): move director traffic in two steps so a deploy does not land on one instance
Cloud Run routes a new director revision's floor instances their first request
roughly 60-110 s after they start, while a traffic move reaches clients in
15-30 s. The one-step 100% move therefore put every assignment on the single
instance started with the revision, which sits at its own admission limit at
peak; the 2026-10-20 deploy returned ~550 assign 503s in two minutes that way.
With a floor above one, the deploy now moves 20% first, waits until Cloud Run's
request log shows every floor instance serving (six-minute deadline), then
moves the rest. A timeout or failure at that stage returns all traffic to the
previous serving revision, which still holds its warm floor. The staging deploy
identity gains log read access for the same check.
* fix(relay): bound the director spread log read, log the instances seen at timeout, and document split recovery
A hung gcloud log read could hold the 20/80 split past the six-minute deadline;
each read now times out after 30 s and counts as an unread poll. A timeout logs
the distinct instances seen, so a partial share packed onto fewer instances than
the floor shows up. The runbook covers restoring a split left by a cancelled job
and the missing selector-rollback tag after a timed-out deploy.
* fix(relay): judge the reserve watch's 503 rule on non-drain 503s, not a host's own re-dials
The watch counted every director 503, and on 2026-10-18 ~98% were host-rate-limited re-dials that
grow with traffic, so it flipped production-gce-c7 back twice with no fault. It now uses the roll
ladder's non-drain measure (request_count 503s less drain-return deferrals and host-rate-limited /
host-in-flight re-dials from the director's runtime metrics) for both the background and the live
samples. A judged minute no runtime-metrics line covers is unverifiable, not zero.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): state the reserve watch's metrics coverage honestly and share the metrics read limit
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
The folder is the cloud relay that reaches phones; under its old name it read as the legacy SSH
relay. git mv plus import and path-literal rewrites only.
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule
A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to
300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the
first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write
under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip
- The flip back holds no concurrency group: GitHub keeps one pending run per group and a
newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is
re-read and retried (5 tries, backoff) while the switch file still says reserve.
- The 503 background comes from the 10 minutes before a required flipped-at, or an explicit
value; neither is refused.
- A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the
exact break-glass inputs.
- Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable.
- Seat-behind-row key adds the director instance and cell; six segments share one command.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): read the reserve watch's 503 publishing front from every response code
A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy
cell back within minutes. The 503 read now groups request_count by response code in one
query; the newest minute any code has a point for is how far the series is published, so a
503 minute at or before it with no point is zero and minutes past it are left out.
Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start
in later segments, a pending pre-flip background no longer adds to the unverifiable streak,
the flip back retries runtime-status before calling for break glass, and the concurrency
rationale lives once in the docs.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read
- A flip back whose write or plan never finished (no ID token, a failed login, a silent cell)
raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does.
- The first segment reads the Cloud SQL rollout lease through the lease action's own reader and
refuses to start while any run holds it, or when it cannot be read.
- A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the
watch ends refused without a write.
- The duplicate-seat read takes the per-minute ledger summary in the same query and window; with
no fresh summary in it the sample is unverifiable, not zero duplicates.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): page on a refused reserve watch or a failed flip back
A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not
land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write
mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and
alert policy page on any such line, per cell, through relay_alert_notification_channels, and the
deploy identity gets roles/logging.logWriter in production to write it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note a force-cancel can skip the reserve-watch flip-back write
* fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it
A paced drain tells its last host when the window ends, so the restart-safe
wait has to see an empty runtime reaching past that point. It counted a full
window of quiet samples from the first empty one instead, and the runtime only
empties as the window ends, so every same-cap cell waited the window twice:
about 25 minutes per Asia cell at 1200 s on 2026-10-13, and 5 minutes per
300 s cell.
The window is now measured from the verifier's own start, which runs after the
cell accepted the drain and so ends no earlier than the cell's own window. A
pass still needs two consecutive restart-safe samples, the last taken at or
after that end, and any non-zero runtime counter or open migration still
restarts the count. Progress lines report paceWindowRemainingMs in place of the
old sample target.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): drop the same-cap drain timeout's double-wait allowance and stale quiet wording
The drain step's timeout added the window's excess over 300000 only because
restart-safe used to wait a second window after the cell emptied, so it is
now the 15-minute migration lease plus the window for every pace (35 minutes
at 1200000, unchanged at 300000 and below). Docs, workflow comments and the
shadow gate's settledAfterSeconds note now describe the window-end rule. The
always-quiet test starts its clock late, so a window measured from zero
fails it, and the overlapping late-control test is folded away.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state the restart-safe window-end rule exactly
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): the slowest restart-safe wait is now 35 minutes
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): declare Asia spare cell c35 as migration-only
Adds a seventh asia-east2 cell at the C34 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-a, pinned to image F. It gets
its own topology and registration wave but no promotion wave, so the
admission script and workflow refuse to promote it; it stays a
migration-only spare and out of the fleet pool list. Mirrors #25336.
* test(relay): accept the c35 spare under the same-cap capacity allowlist
* fix(relay): keep verifying host tokens from the last served keys while auth is down
jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.
* fix(relay): bound stale relay keys at 30 min and document the emergency purge
Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.
* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
* docs(relay): director Cloud SQL sidecar enablement and rollback, as run on staging
The 0% apply revision is not started by Cloud Run, so the first sidecar boot is the deploy's
rollback candidate. A new log-based metric can take minutes to appear, so its alert policy
can fail the first apply. Both observed on staging 2026-10-11.
* docs(relay): production sidecar enablement needs the tfvars PR, a passed drill and the rollout lease
Neil
Neil
Jinwoo Hong
