mirror of
https://github.com/stablyai/orca.git
synced 2026-10-22 08:03:37 +00:00
docs(skills): record staging rollout evidence
This commit is contained in:
Jinwoo-H
@@ -7,12 +7,12 @@ Last updated: 2026-08-11.
|
||||
Implementation baselines captured by this checklist update:
|
||||
|
||||
- Orca implementation: `6ba6107dba` on `skills-share` (pushed; no PR).
|
||||
- Orca Cloud: `9356b6e` on `main` (merged through `stablyai/orca-cloud#307`).
|
||||
- Orca Cloud: `41ef335` on `main` (merged through `stablyai/orca-cloud#307`-`#312`).
|
||||
|
||||
Validated so far:
|
||||
|
||||
- Local Node and web typechecks, changed-code quality gates, 94 skill-domain files with 770 tests
|
||||
passed and 3 skipped, 133 Orca Cloud API tests with one opt-in integration skip, the full Cloud
|
||||
passed and 3 skipped, 134 Orca Cloud API tests with one opt-in integration skip, the full Cloud
|
||||
monorepo test/typecheck/lint/build gates, and isolated Terraform formatting and validation.
|
||||
- Final Orca validation passed repository-wide lint, all Node/CLI/web typechecks, the production
|
||||
desktop/native build, the Node 18 Relay bundle contract, 78 focused skill files with 557 tests
|
||||
@@ -50,8 +50,9 @@ Validated so far:
|
||||
and Docker SSH journeys. Focused failure injection also proves invalid gzip classification,
|
||||
old-version preservation after `EACCES` and `ENOSPC`, bounded recovery from a lost final install
|
||||
response, and validated structured SSH failure data without arbitrary error-data disclosure.
|
||||
- Isolated staging bucket, IAM, secret container, log metrics, dashboard, and alerts in
|
||||
`onorca-cloud-staging`.
|
||||
- Isolated staging bucket, IAM, `orca_skills` database and principal, enabled secret version, 11
|
||||
log metrics, four alerts, and one dashboard in `onorca-cloud-staging`. The complete targeted
|
||||
skill-infrastructure plan is zero-diff at Cloud `41ef335`.
|
||||
- PostgreSQL migration startup is serialized by a transaction-scoped advisory lock. Transactional
|
||||
DDL rollback and eight concurrent startup callers passed against ephemeral PostgreSQL 16 and 17,
|
||||
with exactly one recorded schema version.
|
||||
@@ -59,13 +60,21 @@ Validated so far:
|
||||
one approved skill-database secret, and rejects secret, Cloud SQL, service-account, scaling,
|
||||
CPU, memory, volume, mount, probe, or unexpected-environment drift before traffic promotion.
|
||||
Both staging and production workflows explicitly carry all four skill controls as `false`.
|
||||
- Orca Cloud `main` verification run `31518125253` passed its Terraform and full code jobs after
|
||||
merge commit `9356b6e`.
|
||||
- Staging API revision `orca-cloud-api-staging-00039-rek` serves 100% of traffic with the exact
|
||||
skill secret and Cloud SQL attachment, all four controls false, two database-ready startup
|
||||
events, zero revision error logs, existing artifact authorization at `401`, and skill routes at
|
||||
`404`. The API service is zero-diff; known shared SQL and artifact-bucket drift remains excluded.
|
||||
- Orca Cloud `main` verification run `31529170760` passed its Terraform and full code jobs after
|
||||
merge commit `41ef335`.
|
||||
- Guarded sleep run `31529587719` emitted `staging_relay_slept` for all three cells. Read-only
|
||||
status run `31530386310` then verified Cloud SQL policy `NEVER`, every Relay MIG target at zero,
|
||||
and both staging Cloud Run services with zero active-revision minimums.
|
||||
|
||||
Rollout gate: staging Cloud SQL is shared with Auth/Relay and intentionally stopped. Do not create
|
||||
the skills database/user/secret version or deploy Cloud Run routes until the supported staging SQL
|
||||
power workflow deliberately wakes it and the reviewed plan still shows no unrelated changes.
|
||||
`gcloud` currently requires `gcloud auth login jinwoo@stably.ai` before that fresh plan can run.
|
||||
Rollout gate: staging infrastructure, the route-disabled API, and startup migrations are deployed;
|
||||
the shared staging data plane is back in its low-cost asleep state.
|
||||
Do not enable route registration until an environment-scoped short-lived smoke access token is
|
||||
available and the authenticated upload/finalize/download, denial, expiry, revocation, deletion,
|
||||
and cleanup journey passes. Production remains untouched.
|
||||
|
||||
Source plan: [Agent skill sharing and installation plan](./agent-skill-sharing-installation-plan.md).
|
||||
|
||||
@@ -457,9 +466,9 @@ does not mean the surrounding phase is complete.
|
||||
- [x] Declare database `orca_skills` on existing regional PostgreSQL 17 instance
|
||||
`orca-cloud-auth-db`.
|
||||
- [x] Declare dedicated principal `orca_skills_app` with access only to `orca_skills`.
|
||||
- [ ] Store its connection URL in Secret Manager as `orca-cloud-skills-database-url`.
|
||||
- [ ] Attach the existing Cloud SQL instance to `orca-cloud-api` without replacing the service.
|
||||
- [ ] Inject only the skill database secret into the API service.
|
||||
- [x] Store its connection URL in Secret Manager as `orca-cloud-skills-database-url`.
|
||||
- [x] Attach the existing Cloud SQL instance to `orca-cloud-api` without replacing the service.
|
||||
- [x] Inject only the skill database secret into the API service.
|
||||
- [ ] Verify backups and point-in-time recovery cover the new database.
|
||||
|
||||
### IAM
|
||||
@@ -469,7 +478,7 @@ does not mean the surrounding phase is complete.
|
||||
- [x] Grant the API service account `roles/cloudsql.client` for the existing instance.
|
||||
- [x] Grant service-account-scoped IAM Credentials `signBlob` for self-signing V4 policies and
|
||||
URLs.
|
||||
- [ ] Grant Secret Manager accessor only for skill-specific secrets.
|
||||
- [x] Grant Secret Manager accessor only for skill-specific secrets.
|
||||
- [x] Verify bucket IAM contains neither `allUsers` nor `allAuthenticatedUsers`.
|
||||
- [x] Do not grant desktop, remote runtime, or end-user identities direct bucket IAM.
|
||||
- [x] Do not create or distribute long-lived GCP service-account keys.
|
||||
@@ -478,7 +487,7 @@ does not mean the surrounding phase is complete.
|
||||
|
||||
### Cloud Run configuration
|
||||
|
||||
- [ ] Extend `orca-cloud-api` in `us-central1`; do not create a separate V1 worker service.
|
||||
- [x] Extend `orca-cloud-api` in `us-central1`; do not create a separate V1 worker service.
|
||||
- [x] Configure bucket, 40 MiB compressed limit, 15-minute upload TTL, five-minute download TTL,
|
||||
fixed finalize concurrency, and skill database URL.
|
||||
- [x] Reuse existing auth base URL and application CORS configuration.
|
||||
@@ -502,7 +511,7 @@ does not mean the surrounding phase is complete.
|
||||
- [x] Produce and review a staging plan.
|
||||
- [x] Verify the plan does not replace the existing artifact bucket, Cloud SQL instance, Cloud Run
|
||||
services, or unrelated IAM.
|
||||
- [ ] Apply to staging and capture resource and IAM verification evidence.
|
||||
- [x] Apply to staging and capture resource and IAM verification evidence.
|
||||
- [ ] Produce and review the production plan after staging gates pass.
|
||||
- [ ] Apply the approved production plan only during the rollout phase.
|
||||
- [ ] Verify production with read-only bucket, Cloud Run, database, IAM, lifecycle, and CORS
|
||||
@@ -883,7 +892,7 @@ This does not substitute for physical SSH macOS/Windows or the supported Linux f
|
||||
Data Access audit logging; evidence is in Orca Cloud `69b388e`.
|
||||
- [ ] Add budget alerts for GCS storage/egress, Cloud Run growth, and Cloud SQL storage.
|
||||
|
||||
Partial dashboard evidence in Orca Cloud `69b388e`: upload-grant, finalize, share, and
|
||||
Partial dashboard evidence in Orca Cloud `41ef335`: upload-grant, finalize, share, and
|
||||
download-grant success; bounded security failures; Cloud Run CPU, memory, instances, and latency;
|
||||
GCS bytes and object count; and Cloud SQL connections and disk use. The broader unchecked items
|
||||
still require the listed split metrics, latency panels, and alerts.
|
||||
@@ -988,8 +997,8 @@ still require the listed split metrics, latency panels, and alerts.
|
||||
|
||||
### Staging
|
||||
|
||||
- [ ] Apply reviewed Terraform and migrations to staging.
|
||||
- [ ] Deploy `orca-cloud-api` skill routes disabled by server-side flags.
|
||||
- [x] Apply reviewed Terraform and migrations to staging.
|
||||
- [x] Deploy `orca-cloud-api` skill routes disabled by server-side flags.
|
||||
- [ ] Run upload/finalize/download tests for expiry, denial, oversize, corruption, deduplication,
|
||||
revocation, deletion, and cleanup.
|
||||
- [ ] Run desktop-to-local, paired-runtime, `windows 2`, WSL, and SSH journeys in staging.
|
||||
|
||||
@@ -40,8 +40,9 @@ that larger package-manager surface.
|
||||
## Current execution status
|
||||
|
||||
Orca implementation through `6ba6107dba` is pushed on `skills-share` without an Orca pull request.
|
||||
Orca Cloud implementation is merged on `main` through `9356b6e` (`stablyai/orca-cloud#307`) without
|
||||
applying infrastructure or deploying a service. Local, Windows, WSL, paired-runtime, and
|
||||
Orca Cloud implementation is merged on `main` through `41ef335`
|
||||
(`stablyai/orca-cloud#307`-`#312`). Staging infrastructure and a route-disabled API are deployed;
|
||||
production remains untouched. Local, Windows, WSL, paired-runtime, and
|
||||
Docker-backed SSH validation are substantially complete; the implementation checklist records the
|
||||
exact evidence and remaining physical-host and failure-recovery gates.
|
||||
|
||||
@@ -52,18 +53,28 @@ SSH now carries only schema-validated structured skill failures through optional
|
||||
when it is absent. Invalid gzip input has a stable archive category, and deterministic `EACCES`
|
||||
and `ENOSPC` injection proves failed updates preserve the previous installed version.
|
||||
|
||||
The dedicated staging bucket, IAM, secret container, metrics, dashboard, and alerts exist in
|
||||
`onorca-cloud-staging`. Staging Cloud SQL is shared with Auth and Relay and is intentionally
|
||||
stopped. Review a fresh Terraform plan after refreshing `gcloud` authentication, then wake staging
|
||||
only through the supported Relay power workflow if the reviewed skill changes require the
|
||||
database. Keep skill routes disabled until migrations and staging smoke tests pass.
|
||||
The dedicated staging bucket, IAM, database, principal, enabled secret version, metrics, dashboard,
|
||||
and alerts exist in `onorca-cloud-staging`. The complete skill-infrastructure target is zero-diff.
|
||||
The API service is also zero-diff after excluding gcloud-owned release metadata; the known shared
|
||||
Cloud SQL and artifact-bucket drift was never applied. Staging was awakened only through the
|
||||
supported Relay power workflow before database provisioning. Guarded sleep run `31529587719`
|
||||
returned all three Relay cells to zero and Cloud SQL to activation policy `NEVER`; read-only status
|
||||
run `31530386310` independently confirmed the asleep state and zero active-revision minimums.
|
||||
|
||||
The API release workflow now deploys an immutable no-traffic candidate with the complete artifact
|
||||
and skill environment plus exactly one approved skill-database secret. It verifies the inherited
|
||||
Terraform runtime identity, Cloud SQL attachment, scaling, resources, volumes, mounts, ports, and
|
||||
probes before promotion, and rolls back to a re-smoked prior revision on failure. All four skill
|
||||
controls remain explicitly disabled. Database migration startup uses a transaction-scoped advisory
|
||||
lock; transactional rollback and eight concurrent callers pass against PostgreSQL 16 and 17.
|
||||
lock; transactional rollback and eight concurrent callers pass against PostgreSQL 16 and 17. The
|
||||
serving staging revision completed migrate-only startup twice with zero error logs while skill
|
||||
routes remained `404` and existing artifact authorization remained `401`.
|
||||
|
||||
The remaining staging gate is authenticated skill API testing. The completed deployment did not
|
||||
receive a smoke access token, so it proved anonymous boundaries and canonical health but did not
|
||||
exercise upload, finalize, download, ACL denial, expiry, revocation, deletion, or cleanup. Keep
|
||||
route registration off until a subsequent run receives the short-lived token and the full journey
|
||||
passes.
|
||||
|
||||
## Research baseline
|
||||
|
||||
@@ -547,16 +558,20 @@ policies, signed URLs, or ACL membership.
|
||||
configuration, dashboards, alerts, and budget thresholds to Terraform.
|
||||
4. Run Terraform formatting, validation, and a reviewed plan. The expected production plan contains
|
||||
no replacement of the existing artifact bucket, Cloud SQL instance, or Cloud Run services.
|
||||
5. Apply in staging and run upload/finalize/download tests, including expired policies, ACL denial,
|
||||
oversized uploads, corrupt archives, deduplication, revocation, and object cleanup.
|
||||
6. Apply the PostgreSQL migrations before routing package traffic to the new API handlers.
|
||||
7. Deploy `orca-cloud-api` with endpoints disabled by a server-side feature flag.
|
||||
8. Run a staging desktop-to-runtime installation and verify logs contain no grants or private
|
||||
5. Apply the reviewed skill-only infrastructure plan in staging without shared-resource drift.
|
||||
6. Deploy `orca-cloud-api` with all skill controls disabled. When the database secret is present,
|
||||
startup applies PostgreSQL migrations without registering the skill routes.
|
||||
7. Verify the migration-ready event, zero error logs, ordinary artifact behavior, and `404` skill
|
||||
route boundary before changing any control.
|
||||
8. Enable route registration for the staging test cohort and run upload/finalize/download tests,
|
||||
including expired policies, ACL denial, oversized uploads, corrupt archives, deduplication,
|
||||
revocation, and object cleanup.
|
||||
9. Run a staging desktop-to-runtime installation and verify logs contain no grants or private
|
||||
contents.
|
||||
9. Apply the reviewed production Terraform plan.
|
||||
10. Run migrations, deploy the API, and enable internal accounts only.
|
||||
11. Observe at least one complete soft-delete, upload-expiry, update, rollback, and revoke journey.
|
||||
12. Expand the feature flag gradually, retaining separate kill switches for upload grants,
|
||||
10. Apply the reviewed production Terraform plan.
|
||||
11. Run migrations, deploy the API, and enable internal accounts only.
|
||||
12. Observe at least one complete soft-delete, upload-expiry, update, rollback, and revoke journey.
|
||||
13. Expand the feature flag gradually, retaining separate kill switches for upload grants,
|
||||
download grants, and remote installation.
|
||||
|
||||
Production verification uses read-only commands such as:
|
||||
|
||||
Reference in New Issue
Block a user