starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-10-22 08:03:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(skills): record authenticated staging smoke

Browse Source
This commit is contained in:
E2E Test
2026-08-11 17:24:41 -04:00
parent 0bec0a74f6
commit 41a7b45c4e
2 changed files with 60 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/agent-skill-sharing-implementation-checklist.md
+39 -21
View File
@@ -7,7 +7,8 @@ Last updated: 2026-08-11.
Implementation baselines captured by this checklist update:
- Orca implementation: `6ba6107dba` on `skills-share` (pushed; no PR).
- Orca Cloud: `41ef335` on `main` (merged through `stablyai/orca-cloud#307`-`#312`).
- Orca Cloud: `4e91cf9` on `main` (skill OIDC smoke in `stablyai/orca-cloud#313` and
release-metadata convergence in `#314`).
Validated so far:
@@ -56,25 +57,37 @@ Validated so far:
- PostgreSQL migration startup is serialized by a transaction-scoped advisory lock. Transactional
DDL rollback and eight concurrent startup callers passed against ephemeral PostgreSQL 16 and 17,
with exactly one recorded schema version.
- The immutable API deploy path now stamps every artifact and skill literal variable plus exactly
one approved skill-database secret, and rejects secret, Cloud SQL, service-account, scaling,
CPU, memory, volume, mount, probe, or unexpected-environment drift before traffic promotion.
Both staging and production workflows explicitly carry all four skill controls as `false`.
- Staging API revision `orca-cloud-api-staging-00039-rek` serves 100% of traffic with the exact
skill secret and Cloud SQL attachment, all four controls false, two database-ready startup
events, zero revision error logs, existing artifact authorization at `401`, and skill routes at
`404`. The API service is zero-diff; known shared SQL and artifact-bucket drift remains excluded.
- Orca Cloud `main` verification run `31529170760` passed its Terraform and full code jobs after
merge commit `41ef335`.
- Guarded sleep run `31529587719` emitted `staging_relay_slept` for all three cells. Read-only
status run `31530386310` then verified Cloud SQL policy `NEVER`, every Relay MIG target at zero,
and both staging Cloud Run services with zero active-revision minimums.
- The immutable API deploy path stamps every artifact and skill literal variable plus exactly one
approved skill-database secret, and rejects secret, Cloud SQL, service-account, scaling, CPU,
memory, volume, mount, probe, or unexpected-environment drift before traffic promotion.
Staging intentionally carries all four skill controls as `true`; production remains `false`.
- The route-disabled baseline, API revision `orca-cloud-api-staging-00039-rek`, first proved the
database-ready startup and anonymous `401`/`404` boundaries with zero revision errors.
- Cloud PR `#313` passed run `31533988351`, merged as `d0bf926`, and passed `main` run
`31534229437`. Guarded wake run `31534564230` restored exactly the two configured Relay cells.
- Auth deploy run `31535179937` promoted revision `orca-cloud-auth-staging-00017-dug` at 100%
traffic. Health, JWKS, exact GitHub OIDC claim constraints, and zero deployment-window errors
passed.
- API deploy run `31535438327` promoted revision `orca-cloud-api-staging-00042-hef` at 100%
traffic with `authenticatedSmoke: true` and `skillSmoke: true`. The smoke covered artifact
lifecycle; skill upload/finalize/download; two immutable versions; recipient and outsider
authorization; local and remote grants; rollback selection; expiry; revocation; package
deletion; and signed-object cleanup.
- The serving API has all four controls `true`, zero revision errors, and privacy-safe structured
request logs. The log field inventory contains no principals, credentials, signed URLs, object
paths, or private contents; a sensitive-value scan returned zero matches.
- Cloud PR `#314` removed deploy-owned Auth release metadata from Terraform ownership, passed run
`31536141357`, merged as `4e91cf9`, and passed `main` run `31536329459`. The targeted Auth/API
plan then reported zero intended changes. Known SQL and artifact-bucket drift was excluded and
never applied.
- Guarded sleep run `31536547916` stopped all three Relay cells. Read-only status run
`31537525219` verified Cloud SQL policy `NEVER`, every MIG target at zero, and both managed Cloud
Run active-revision minimums at zero.
Rollout gate: staging infrastructure, the route-disabled API, and startup migrations are deployed;
the shared staging data plane is back in its low-cost asleep state.
Do not enable route registration until an environment-scoped short-lived smoke access token is
available and the authenticated upload/finalize/download, denial, expiry, revocation, deletion,
and cleanup journey passes. Production remains untouched.
Rollout gate: staging infrastructure, OIDC identity exchange, route registration, and the
authenticated artifact/skill lifecycle smoke passed. The shared staging data plane is back in its
low-cost asleep state. Production remains untouched; physical staging install journeys, load,
quarantine lifecycle, and soft-delete recovery gates remain.
Source plan: [Agent skill sharing and installation plan](./agent-skill-sharing-installation-plan.md).
@@ -502,8 +515,8 @@ does not mean the surrounding phase is complete.
environment and the exact `ORCA_SKILLS_DATABASE_URL` Secret Manager reference.
- [x] Reject serving or candidate drift in the runtime service account, Cloud SQL attachment,
scaling, CPU, memory, volumes, mounts, ports, and probes before moving traffic.
- [x] Keep all four skill controls explicitly disabled in staging and production deploy workflows
until their rollout gates are intentionally changed.
- [x] Keep all four skill controls explicit in both deploy workflows. Staging changed to `true`
only with the authenticated OIDC smoke gate; production remains `false`.
### Terraform review and apply
@@ -999,9 +1012,14 @@ still require the listed split metrics, latency panels, and alerts.
- [x] Apply reviewed Terraform and migrations to staging.
- [x] Deploy `orca-cloud-api` skill routes disabled by server-side flags.
- [x] Enable all four staging controls and pass the GitHub OIDC-authenticated artifact and skill
lifecycle smoke, including authorization, immutable versions, local/remote grants, rollback,
expiry, revocation, deletion, and object cleanup.
- [ ] Run upload/finalize/download tests for expiry, denial, oversize, corruption, deduplication,
revocation, deletion, and cleanup.
- [ ] Run desktop-to-local, paired-runtime, `windows 2`, WSL, and SSH journeys in staging.
- [x] Verify staging skill request logs contain only route, method, status, duration, and standard
process/request metadata, with zero credential or private-content matches.
- [ ] Verify logs, metrics, traces, diagnostics, and support bundles contain no grants or private
package data.
- [ ] Load-test finalization and choose the fixed semaphore from observed memory, CPU, request
docs/agent-skill-sharing-installation-plan.md
+21 -18
View File
@@ -40,9 +40,10 @@ that larger package-manager surface.
## Current execution status
Orca implementation through `6ba6107dba` is pushed on `skills-share` without an Orca pull request.
Orca Cloud implementation is merged on `main` through `41ef335`
(`stablyai/orca-cloud#307`-`#312`). Staging infrastructure and a route-disabled API are deployed;
production remains untouched. Local, Windows, WSL, paired-runtime, and
Orca Cloud implementation is merged on `main` through `4e91cf9`. The authenticated OIDC smoke
landed in `stablyai/orca-cloud#313`; the narrow Auth release-metadata convergence follow-up landed
in `#314`. Staging infrastructure and skill routes are deployed and validated; production remains
untouched. Local, Windows, WSL, paired-runtime, and
Docker-backed SSH validation are substantially complete; the implementation checklist records the
exact evidence and remaining physical-host and failure-recovery gates.
@@ -55,26 +56,28 @@ and `ENOSPC` injection proves failed updates preserve the previous installed ver
The dedicated staging bucket, IAM, database, principal, enabled secret version, metrics, dashboard,
and alerts exist in `onorca-cloud-staging`. The complete skill-infrastructure target is zero-diff.
The API service is also zero-diff after excluding gcloud-owned release metadata; the known shared
Cloud SQL and artifact-bucket drift was never applied. Staging was awakened only through the
supported Relay power workflow before database provisioning. Guarded sleep run `31529587719`
returned all three Relay cells to zero and Cloud SQL to activation policy `NEVER`; read-only status
run `31530386310` independently confirmed the asleep state and zero active-revision minimums.
After `#314`, the targeted Auth/API plan also reports zero intended changes. Known shared Cloud SQL
and artifact-bucket drift was excluded and never applied. Staging was awakened only through guarded
run `31534564230`. Sleep run `31536547916` returned all three cells to zero; read-only status run
`31537525219` independently confirmed SQL policy `NEVER`, zero MIG targets, and zero managed Cloud
Run active-revision minimums.
The API release workflow now deploys an immutable no-traffic candidate with the complete artifact
and skill environment plus exactly one approved skill-database secret. It verifies the inherited
Terraform runtime identity, Cloud SQL attachment, scaling, resources, volumes, mounts, ports, and
probes before promotion, and rolls back to a re-smoked prior revision on failure. All four skill
controls remain explicitly disabled. Database migration startup uses a transaction-scoped advisory
lock; transactional rollback and eight concurrent callers pass against PostgreSQL 16 and 17. The
serving staging revision completed migrate-only startup twice with zero error logs while skill
routes remained `404` and existing artifact authorization remained `401`.
probes before promotion, and rolls back to a re-smoked prior revision on failure. Staging now has
all four controls enabled; production keeps all four disabled. Database migration startup uses a
transaction-scoped advisory lock; transactional rollback and eight concurrent callers pass against
PostgreSQL 16 and 17.
The remaining staging gate is authenticated skill API testing. The completed deployment did not
receive a smoke access token, so it proved anonymous boundaries and canonical health but did not
exercise upload, finalize, download, ACL denial, expiry, revocation, deletion, or cleanup. Keep
route registration off until a subsequent run receives the short-lived token and the full journey
passes.
Auth deploy run `31535179937` promoted `orca-cloud-auth-staging-00017-dug` with exact staging-only
GitHub OIDC constraints, healthy JWKS, and zero deployment-window errors. API run `31535438327`
promoted `orca-cloud-api-staging-00042-hef` with `authenticatedSmoke: true` and `skillSmoke: true`.
The ten-minute, non-refreshable smoke principals exercised artifact CRUD and skill upload,
finalize, immutable versions, recipient/outsider authorization, local and remote grants, rollback,
expiry, revocation, deletion, and signed-object cleanup. Structured skill logs exposed only
privacy-safe request metadata and produced zero sensitive-value matches. Remaining staging gates
are end-user install journeys, load, quarantine lifecycle deletion, and soft-delete recovery.
## Research baseline