From 41a7b45c4e6a0e008747e561cd53b55c22b1f4e3 Mon Sep 17 00:00:00 2001 From: E2E Test Date: Tue, 11 Aug 2026 17:24:41 -0400 Subject: [PATCH] docs(skills): record authenticated staging smoke --- ...-skill-sharing-implementation-checklist.md | 60 ++++++++++++------- docs/agent-skill-sharing-installation-plan.md | 39 ++++++------ 2 files changed, 60 insertions(+), 39 deletions(-) diff --git a/docs/agent-skill-sharing-implementation-checklist.md b/docs/agent-skill-sharing-implementation-checklist.md index a08017da850..5b31b73547c 100644 --- a/docs/agent-skill-sharing-implementation-checklist.md +++ b/docs/agent-skill-sharing-implementation-checklist.md @@ -7,7 +7,8 @@ Last updated: 2026-08-11. Implementation baselines captured by this checklist update: - Orca implementation: `6ba6107dba` on `skills-share` (pushed; no PR). -- Orca Cloud: `41ef335` on `main` (merged through `stablyai/orca-cloud#307`-`#312`). +- Orca Cloud: `4e91cf9` on `main` (skill OIDC smoke in `stablyai/orca-cloud#313` and + release-metadata convergence in `#314`). Validated so far: @@ -56,25 +57,37 @@ Validated so far: - PostgreSQL migration startup is serialized by a transaction-scoped advisory lock. Transactional DDL rollback and eight concurrent startup callers passed against ephemeral PostgreSQL 16 and 17, with exactly one recorded schema version. -- The immutable API deploy path now stamps every artifact and skill literal variable plus exactly - one approved skill-database secret, and rejects secret, Cloud SQL, service-account, scaling, - CPU, memory, volume, mount, probe, or unexpected-environment drift before traffic promotion. - Both staging and production workflows explicitly carry all four skill controls as `false`. -- Staging API revision `orca-cloud-api-staging-00039-rek` serves 100% of traffic with the exact - skill secret and Cloud SQL attachment, all four controls false, two database-ready startup - events, zero revision error logs, existing artifact authorization at `401`, and skill routes at - `404`. The API service is zero-diff; known shared SQL and artifact-bucket drift remains excluded. -- Orca Cloud `main` verification run `31529170760` passed its Terraform and full code jobs after - merge commit `41ef335`. -- Guarded sleep run `31529587719` emitted `staging_relay_slept` for all three cells. Read-only - status run `31530386310` then verified Cloud SQL policy `NEVER`, every Relay MIG target at zero, - and both staging Cloud Run services with zero active-revision minimums. +- The immutable API deploy path stamps every artifact and skill literal variable plus exactly one + approved skill-database secret, and rejects secret, Cloud SQL, service-account, scaling, CPU, + memory, volume, mount, probe, or unexpected-environment drift before traffic promotion. + Staging intentionally carries all four skill controls as `true`; production remains `false`. +- The route-disabled baseline, API revision `orca-cloud-api-staging-00039-rek`, first proved the + database-ready startup and anonymous `401`/`404` boundaries with zero revision errors. +- Cloud PR `#313` passed run `31533988351`, merged as `d0bf926`, and passed `main` run + `31534229437`. Guarded wake run `31534564230` restored exactly the two configured Relay cells. +- Auth deploy run `31535179937` promoted revision `orca-cloud-auth-staging-00017-dug` at 100% + traffic. Health, JWKS, exact GitHub OIDC claim constraints, and zero deployment-window errors + passed. +- API deploy run `31535438327` promoted revision `orca-cloud-api-staging-00042-hef` at 100% + traffic with `authenticatedSmoke: true` and `skillSmoke: true`. The smoke covered artifact + lifecycle; skill upload/finalize/download; two immutable versions; recipient and outsider + authorization; local and remote grants; rollback selection; expiry; revocation; package + deletion; and signed-object cleanup. +- The serving API has all four controls `true`, zero revision errors, and privacy-safe structured + request logs. The log field inventory contains no principals, credentials, signed URLs, object + paths, or private contents; a sensitive-value scan returned zero matches. +- Cloud PR `#314` removed deploy-owned Auth release metadata from Terraform ownership, passed run + `31536141357`, merged as `4e91cf9`, and passed `main` run `31536329459`. The targeted Auth/API + plan then reported zero intended changes. Known SQL and artifact-bucket drift was excluded and + never applied. +- Guarded sleep run `31536547916` stopped all three Relay cells. Read-only status run + `31537525219` verified Cloud SQL policy `NEVER`, every MIG target at zero, and both managed Cloud + Run active-revision minimums at zero. -Rollout gate: staging infrastructure, the route-disabled API, and startup migrations are deployed; -the shared staging data plane is back in its low-cost asleep state. -Do not enable route registration until an environment-scoped short-lived smoke access token is -available and the authenticated upload/finalize/download, denial, expiry, revocation, deletion, -and cleanup journey passes. Production remains untouched. +Rollout gate: staging infrastructure, OIDC identity exchange, route registration, and the +authenticated artifact/skill lifecycle smoke passed. The shared staging data plane is back in its +low-cost asleep state. Production remains untouched; physical staging install journeys, load, +quarantine lifecycle, and soft-delete recovery gates remain. Source plan: [Agent skill sharing and installation plan](./agent-skill-sharing-installation-plan.md). @@ -502,8 +515,8 @@ does not mean the surrounding phase is complete. environment and the exact `ORCA_SKILLS_DATABASE_URL` Secret Manager reference. - [x] Reject serving or candidate drift in the runtime service account, Cloud SQL attachment, scaling, CPU, memory, volumes, mounts, ports, and probes before moving traffic. -- [x] Keep all four skill controls explicitly disabled in staging and production deploy workflows - until their rollout gates are intentionally changed. +- [x] Keep all four skill controls explicit in both deploy workflows. Staging changed to `true` + only with the authenticated OIDC smoke gate; production remains `false`. ### Terraform review and apply @@ -999,9 +1012,14 @@ still require the listed split metrics, latency panels, and alerts. - [x] Apply reviewed Terraform and migrations to staging. - [x] Deploy `orca-cloud-api` skill routes disabled by server-side flags. +- [x] Enable all four staging controls and pass the GitHub OIDC-authenticated artifact and skill + lifecycle smoke, including authorization, immutable versions, local/remote grants, rollback, + expiry, revocation, deletion, and object cleanup. - [ ] Run upload/finalize/download tests for expiry, denial, oversize, corruption, deduplication, revocation, deletion, and cleanup. - [ ] Run desktop-to-local, paired-runtime, `windows 2`, WSL, and SSH journeys in staging. +- [x] Verify staging skill request logs contain only route, method, status, duration, and standard + process/request metadata, with zero credential or private-content matches. - [ ] Verify logs, metrics, traces, diagnostics, and support bundles contain no grants or private package data. - [ ] Load-test finalization and choose the fixed semaphore from observed memory, CPU, request diff --git a/docs/agent-skill-sharing-installation-plan.md b/docs/agent-skill-sharing-installation-plan.md index 910aaabea88..dc9a26338f3 100644 --- a/docs/agent-skill-sharing-installation-plan.md +++ b/docs/agent-skill-sharing-installation-plan.md @@ -40,9 +40,10 @@ that larger package-manager surface. ## Current execution status Orca implementation through `6ba6107dba` is pushed on `skills-share` without an Orca pull request. -Orca Cloud implementation is merged on `main` through `41ef335` -(`stablyai/orca-cloud#307`-`#312`). Staging infrastructure and a route-disabled API are deployed; -production remains untouched. Local, Windows, WSL, paired-runtime, and +Orca Cloud implementation is merged on `main` through `4e91cf9`. The authenticated OIDC smoke +landed in `stablyai/orca-cloud#313`; the narrow Auth release-metadata convergence follow-up landed +in `#314`. Staging infrastructure and skill routes are deployed and validated; production remains +untouched. Local, Windows, WSL, paired-runtime, and Docker-backed SSH validation are substantially complete; the implementation checklist records the exact evidence and remaining physical-host and failure-recovery gates. @@ -55,26 +56,28 @@ and `ENOSPC` injection proves failed updates preserve the previous installed ver The dedicated staging bucket, IAM, database, principal, enabled secret version, metrics, dashboard, and alerts exist in `onorca-cloud-staging`. The complete skill-infrastructure target is zero-diff. -The API service is also zero-diff after excluding gcloud-owned release metadata; the known shared -Cloud SQL and artifact-bucket drift was never applied. Staging was awakened only through the -supported Relay power workflow before database provisioning. Guarded sleep run `31529587719` -returned all three Relay cells to zero and Cloud SQL to activation policy `NEVER`; read-only status -run `31530386310` independently confirmed the asleep state and zero active-revision minimums. +After `#314`, the targeted Auth/API plan also reports zero intended changes. Known shared Cloud SQL +and artifact-bucket drift was excluded and never applied. Staging was awakened only through guarded +run `31534564230`. Sleep run `31536547916` returned all three cells to zero; read-only status run +`31537525219` independently confirmed SQL policy `NEVER`, zero MIG targets, and zero managed Cloud +Run active-revision minimums. The API release workflow now deploys an immutable no-traffic candidate with the complete artifact and skill environment plus exactly one approved skill-database secret. It verifies the inherited Terraform runtime identity, Cloud SQL attachment, scaling, resources, volumes, mounts, ports, and -probes before promotion, and rolls back to a re-smoked prior revision on failure. All four skill -controls remain explicitly disabled. Database migration startup uses a transaction-scoped advisory -lock; transactional rollback and eight concurrent callers pass against PostgreSQL 16 and 17. The -serving staging revision completed migrate-only startup twice with zero error logs while skill -routes remained `404` and existing artifact authorization remained `401`. +probes before promotion, and rolls back to a re-smoked prior revision on failure. Staging now has +all four controls enabled; production keeps all four disabled. Database migration startup uses a +transaction-scoped advisory lock; transactional rollback and eight concurrent callers pass against +PostgreSQL 16 and 17. -The remaining staging gate is authenticated skill API testing. The completed deployment did not -receive a smoke access token, so it proved anonymous boundaries and canonical health but did not -exercise upload, finalize, download, ACL denial, expiry, revocation, deletion, or cleanup. Keep -route registration off until a subsequent run receives the short-lived token and the full journey -passes. +Auth deploy run `31535179937` promoted `orca-cloud-auth-staging-00017-dug` with exact staging-only +GitHub OIDC constraints, healthy JWKS, and zero deployment-window errors. API run `31535438327` +promoted `orca-cloud-api-staging-00042-hef` with `authenticatedSmoke: true` and `skillSmoke: true`. +The ten-minute, non-refreshable smoke principals exercised artifact CRUD and skill upload, +finalize, immutable versions, recipient/outsider authorization, local and remote grants, rollback, +expiry, revocation, deletion, and signed-object cleanup. Structured skill logs exposed only +privacy-safe request metadata and produced zero sensitive-value matches. Remaining staging gates +are end-user install journeys, load, quarantine lifecycle deletion, and soft-delete recovery. ## Research baseline