starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-22 16:02:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
debug-github-rate-limit
orca/mobile/rpc-foundation/settings-recording-coverage.json
T
Jinwoo Hong 7ce8e18d07 test(mobile): consolidate the RPC migration's verification infrastructure (#20521) 
* test(mobile): record main RPC hooks and regression schedules

Add scripted sender recordings, guarded main goldens, reply matrices, lifecycle schedules, settings caller fixtures, and targeted B-seed mutants.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): flush recording user actions through React act

Keep lifecycle updates in separate act boundaries while wrapping direct stateful user actions.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): compile recorded modules with the Node VM API

Use the same trusted-source execution boundary as existing mobile VM test harnesses.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): pool golden values and hoist pre-divergence checkpoints

Golden format version 2 stores each distinct observation field value once in
a `values` map keyed by a 12-hex sha256 of its sorted-key JSON, and a
checkpoint references five hashes. Output stays pretty-printed; the reader
rejects any other format version, resolves hashes back to values, and reports
the scenario, checkpoint, field and JSON path on a mismatch.

Generated variants now declare where their distinguishing input lands, so
checkpoints observed before that point are recorded once in a `.prelude`
scenario instead of once per reply partition. Reply matrices, interruption
schedules and lifecycle schedules share the primitive, which asserts each
variant's pre-divergence prefix matches the base. Equal-but-differently-reached
checkpoints are untouched.

17.71 MB / 3,599 checkpoints / 58.4% intra-file duplicates becomes
4.21 MB / 1,961 checkpoints / 23.6%, with every file's set of distinct
observations unchanged.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): make the recordings sense deadlines, the recorder, and every family

The goldens carried no temporal information, so a request deadline could be cut
to a third and all 61 files stayed byte-identical. Every threshold is now
straddled by two advances with a checkpoint between them: the 30 s request
deadline in both schedule drivers, the 120 ms search debounce in b1, and the 60 s
repo-metadata cache TTL. Shortening any of them moves an observation.

The record fence pinned product sources but excluded the whole recorder, so
--record could rewrite every golden from a modified runner and report the
baseline intact. Goldens now pin recorderSha256 over every non-markdown file in
the runner plus pilot-scenarios.json, and the fence exemption shrinks to the one
directory that digest covers.

Mutation evidence covered 3 of 13 mounted operations. There is now one anchored
mutant per adapter family, covering 11 operations and 51 of the 61 goldens; the
two omitted are the pure async loaders whose entire output is their settlement.
Anchors are asserted to match exactly one site, which caught the acceptance
mutant silently half-applying against three identical guards.

The archived-tree assertion pins each seed's visible state instead of merely
differing from main, and error observations carry code and cause when present.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): record settlement times instead of straddling deadlines

The previous commit made the reviewer's divide-by-three deadline mutant fail by
placing checkpoints on each side of the 30 s deadline. That is a patch: a timing
change that does not cross a hand-placed boundary stays invisible. Those
scenario edits are reverted, and pilot-scenarios.json and schedule-driver.ts are
byte-identical to what they were before them.

The real defect was that the projection had no temporal dimension, so every
settlement now carries startedAt and settledAt in virtual milliseconds on the
pinned fake clock. Any transition the product schedules for itself is recorded
at the time it actually fires, so a deadline or debounce change of any size, in
either direction, moves a recorded number.

A checkpoint's own clock is not recorded. It is always the sum of the scripted
advances, so it is a function of the scenario rather than of the code under
test; run-recording.ts asserts that equality at every checkpoint instead, which
costs no bytes and fails loudly if it ever drifts.

projectionVersion is 2 and all 61 goldens are re-recorded. With the added
timestamps stripped, the distinct-observation set is identical to the previous
recording, so the change is purely additive.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): probe the repo-metadata cache inside its TTL window

Recorded settlement times cover thresholds the product schedules for itself, but
not one it only consults when something else makes it act. The repo-metadata TTL
is the single such case: with probes only at 0 s and 60 s, a 20 s TTL and a 60 s
TTL are both expired at 60 s and record identically, so a 3x cache-lifetime
regression was invisible.

settings-repo-cache-expiry now probes the cache at 59 s as well. This is
coverage, not a substitute for recorded time: it bounds how small a TTL
reduction is visible rather than making the reduction itself observable, and the
README says so.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): record the reply shapes a host can send, not a cross product

The reply matrix froze ~26 malformed envelopes crossed against every consumed
field and three boundary kinds, which is 163,925 lines of JSON pinning accidents
on inputs no desktop produces. `successResponse` always sets `result`, so a JSON
wire has no explicit-undefined slot, and no mounted handler returns a number, a
string, an array, a bare `{}` or a boolean: `settings.get` returns
`{settings: ...}`, and the seed methods return an object or nothing.

Each family now runs nine witnessed partitions once, with no field cross: a
normal result, an absent result, `null`, an inner `{ok: false}` envelope with a
string or an object error, an inner envelope missing `ok`, an outer refusal,
`method_not_found`, and a transport rejection. `null` stays because
`linear.getIssue` returns it for a missing issue and b2 is a shipped null-result
bug; it is also what carries the one named delta these goldens record.

`run-step1-exit.ts` had zero callers and shelled out to the same two Vitest
files as `rpc-recording.mts`, so it and its README paragraph go, along with
`MUTATION_NAMES`, which only it read.

In the module loader, the `rpc-delivery-ambiguity` escape is measured dead: over
every scenario, mutant and reference run it was taken once, by the test that
existed to take it. Golden comparison already fails loudly if a mounted module
ever imports the marker, so both go. The history-panel exposure moves into a
declarative table beside the mutation anchors, leaving the loader with one
source-text mechanism and no per-file branch.

The VM stays. Mount adapters load product sources from an arbitrary `root`, and
the archived bcba08b3e4 tree is bare `mobile/src` and `src` with no package.json
and no node_modules, so no bundler-resolved import can reach it and the seed
rejection gate cannot run without it. Direct import also swaps a 38-module lazy
graph for a 338-module eager one behind 20 native mocks, because
`mobile-tasks-dependencies.ts` re-exports from `react-native` and four other
native packages and `export *` enumerates.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* fix(mobile): treat the recorded lockfile hash as provenance, not an oracle

Every golden pinned `lockfileSha256`, so any dependency bump on main failed all
61 comparisons on the merge commit while the traces were identical. A dependency
that changes behaviour changes the trace itself; one that does not must not fail
a candidate. `platform` already had this exemption — `lockfileSha256` joins it.

Recording still refuses to run unless the lockfile matches the pinned baseline,
so goldens are still produced under frozen conditions.

Verified against main's lockfile: 85 passed, previously 61 failed. The declared
mutation set still reports every mutant killed.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(rpc): cover repeat queries and settings refresh boundaries

Add three scenarios, preserve existing traces, remove unreachable archived checks, and document observed mutation kills and remaining adapter limitations.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* docs(rpc): keep the known-open holes, drop the review transcript

The audit file was mostly a point-in-time record of mutation runs that had already
happened, in an artifacts directory, where it would go stale on the next scenario
change. The durable part is which holes are still open and why they cannot be
reached, which belongs beside the runner it describes.

Markdown is outside recorderSha256, so no re-record; 88 passed | 3 skipped.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* chore(mobile): stage the nine RPC probe scenarios and goldens

These existed only on one machine's /tmp. Landing them verbatim first so a
reboot cannot lose them; a follow-up commit moves them into the suite.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): fold the nine probe scenarios into the recording oracle

The probes were env-var invocations over loose /tmp manifests. They now live in
pilot-scenarios.json and rpc-foundation/goldens, so `pnpm --dir mobile test` runs
them with no flag to remember.

Re-records every golden against main (22f56f7c2a). Two causes:

- #20280 gave LogicalClientCutoverError the delivery-unknown mark and its cause,
  so nine cutover/interruption goldens now record `isRpcDeliveryUnknown: true`
  plus a `Connection closed` cause. The other 55 are byte-identical after 260
  commits of main.
- #20499 replaced the five anchored raw-envelope reads with typed operations, so
  those mutation anchors matched zero sites. Each is re-anchored at the same
  defect's new home; bot-overrides moves to the shared reader that now owns it.

probe-hole-witness.test.ts pins hole and closure together: a probe must kill its
mutation and every pre-probe scenario of the same operation must still survive
it, so a redundant probe fails instead of accumulating.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): list the recording harness in the raw-port inventory

main's #20026 boundary test fails on any non-test file that reaches the raw
request port and is not inventoried. The oracle's scripted transport is exactly
that — it drives the real tracker and logical client — so it belongs in OWNERS
beside the supervisor fakes, not in the step-4 pending backlog.

Also states what the oracle covers, the two holes it was blind to until the
probes, and the step-4 runbook.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): pin the goldens to the tree that recorded them

The inventory entry is a fenced product-tree edit, so --record refused against
main's sha. Baseline now names the branch commit the goldens were recorded from;
the next re-record after this lands bumps it to the merge commit.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): carry a SAFETY rationale on every recorder cast

main added a changed-code casting gate after this branch was cut, so 45 `as`
sites in the recorder read as new findings. Each now states why the assertion
holds; they cluster into five reasons — recorded observations are RecordedValue
by construction, parsed manifests and goldens are validated on the next lines,
interned pools resolve their own hashes, a VM-evaluated module has no static
type, and the mount adapters supply only the members each hook reads.

Re-records the goldens: the comments move recorderSha256.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* docs(rpc): state the measured blindness, not the assumed one

Applying each mutation to real product source shows the two holes are not equal.
The reorder is invisible to 83 of 84 tests and only a probe sees it. The refusal
blanking is also caught by the family reply matrix, because a refusal from cold
publishes null over a non-null initial value — an observational gap, not a
detection gap. Says so rather than letting the stronger claim carry both.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): close three ways the oracle could pass without checking

All four review findings were real; three let the oracle report green while
verifying less than it claimed.

- The baseline guard used `git diff --quiet`, which ignores untracked files, so
  an untracked module under mobile/src or src/shared could change resolution
  while a golden still recorded a pinned baseline header. Adds a
  `git ls-files --others` check over the same paths, recorder still exempt.
- The determinism loop read `Number(env ?? 2)` unvalidated, so
  RPC_FOUNDATION_DETERMINISM_RUNS=0 skipped the body and 57 tests passed having
  recorded and compared nothing. Now requires an integer >= 2.
- Cleanup-time observations were dropped: every checkpoint clones the effects
  array, so anything appended during dispose or the final flush never reached a
  golden. Warns and documents the six scenarios that hit it today; recording
  them changes every golden and is its own change.
- Two SAFETY rationales described each other's assertion. Swapped.

Goldens re-recorded for the recorder-digest change: 73 files, one header line
each, no recorded observation moved.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb

* test(mobile): record teardown observations as a cleanup checkpoint

Each checkpoint clones the effects array, so a rejection or state write produced
by dispose, the transport teardown or the final flush landed after the recording
was built and never reached a golden. An unmount leak is exactly what this
oracle exists to catch, so teardown now runs on the recorded path and anything
it observes becomes a checkpoint with id `cleanup`. State is captured before
dispose, since the operation is gone afterwards.

Six scenarios were dropping observations, across five goldens: projectRowDetailError,
projectMutating, hostLabelById, hostPlatform, workspaceAgent, workspaceAgentOverridden,
creatingKey, selectedAgent, agentOverridden and error. Those five gain a cleanup
checkpoint; the other 68 goldens change by their header line only, so no existing
observation moved.

Also fixes the README's own formatting, which failed `format:check`.

Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
2026-09-13 20:56:21 -04:00

125 lines
6.3 KiB
JSON
Raw Permalink Blame History

{
"manifestCommit": "1d950ab0e0",
"recordingBaseline": "aac38d698ff75ac4c8658addab48ef5a83617619",
"excludedKinds": ["device-preference"],
"operations": [
{
"manifestOperation": "main:mobile/src/agent-history/MobileAgentSessionHistoryPanel.tsx:loadMobileResumeMetadata:settings.get:387",
"manifestScenarioIds": [
"main:mobile/src/agent-history/MobileAgentSessionHistoryPanel.tsx:loadMobileResumeMetadata:settings.get:387:fulfilled",
"main:mobile/src/agent-history/MobileAgentSessionHistoryPanel.tsx:loadMobileResumeMetadata:settings.get:387:refused",
"main:mobile/src/agent-history/MobileAgentSessionHistoryPanel.tsx:loadMobileResumeMetadata:settings.get:387:transport-error"
],
"scenarios": [
"settings-resume-metadata-fulfilled",
"settings-resume-metadata-refused",
"settings-resume-metadata-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/components/use-new-workspace-create-submit.ts:create:settings.get:91",
"manifestScenarioIds": [
"main:mobile/src/components/use-new-workspace-create-submit.ts:create:settings.get:91:fulfilled",
"main:mobile/src/components/use-new-workspace-create-submit.ts:create:settings.get:91:refused",
"main:mobile/src/components/use-new-workspace-create-submit.ts:create:settings.get:91:transport-error"
],
"scenarios": [
"settings-workspace-submit-fulfilled",
"settings-workspace-submit-refused",
"settings-workspace-submit-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/components/use-new-workspace-runtime-context.ts:useNewWorkspaceRuntimeContext:settings.get:42",
"manifestScenarioIds": [
"main:mobile/src/components/use-new-workspace-runtime-context.ts:useNewWorkspaceRuntimeContext:settings.get:42:fulfilled",
"main:mobile/src/components/use-new-workspace-runtime-context.ts:useNewWorkspaceRuntimeContext:settings.get:42:refused",
"main:mobile/src/components/use-new-workspace-runtime-context.ts:useNewWorkspaceRuntimeContext:settings.get:42:transport-error"
],
"scenarios": [
"settings-workspace-context-fulfilled",
"settings-workspace-context-refused",
"settings-workspace-context-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/home/mobile-home-host-requests.ts:fetchMobileHomeTaskProviders:settings.get:76",
"manifestScenarioIds": [
"main:mobile/src/home/mobile-home-host-requests.ts:fetchMobileHomeTaskProviders:settings.get:76:fulfilled",
"main:mobile/src/home/mobile-home-host-requests.ts:fetchMobileHomeTaskProviders:settings.get:76:refused",
"main:mobile/src/home/mobile-home-host-requests.ts:fetchMobileHomeTaskProviders:settings.get:76:transport-error"
],
"scenarios": [
"settings-home-providers-fulfilled",
"settings-home-providers-refused",
"settings-home-providers-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/host-screen/use-host-repo-metadata.ts:fetchRepoMetadata:settings.get:124",
"manifestScenarioIds": [
"main:mobile/src/host-screen/use-host-repo-metadata.ts:fetchRepoMetadata:settings.get:124:fulfilled",
"main:mobile/src/host-screen/use-host-repo-metadata.ts:fetchRepoMetadata:settings.get:124:refused",
"main:mobile/src/host-screen/use-host-repo-metadata.ts:fetchRepoMetadata:settings.get:124:transport-error"
],
"scenarios": [
"settings-repo-metadata-fulfilled",
"settings-repo-metadata-refused",
"settings-repo-metadata-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/session/mobile-new-tab-agent-loader.ts:loadMobileNewTabAgentOptions:settings.get:26",
"manifestScenarioIds": [
"main:mobile/src/session/mobile-new-tab-agent-loader.ts:loadMobileNewTabAgentOptions:settings.get:26:fulfilled",
"main:mobile/src/session/mobile-new-tab-agent-loader.ts:loadMobileNewTabAgentOptions:settings.get:26:refused",
"main:mobile/src/session/mobile-new-tab-agent-loader.ts:loadMobileNewTabAgentOptions:settings.get:26:transport-error"
],
"scenarios": [
"settings-new-tab-ssh",
"settings-new-tab-refused",
"settings-new-tab-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/session/use-pr-bot-author-overrides.ts:usePRBotAuthorOverrides:settings.get:35",
"manifestScenarioIds": [
"main:mobile/src/session/use-pr-bot-author-overrides.ts:usePRBotAuthorOverrides:settings.get:35:fulfilled",
"main:mobile/src/session/use-pr-bot-author-overrides.ts:usePRBotAuthorOverrides:settings.get:35:refused",
"main:mobile/src/session/use-pr-bot-author-overrides.ts:usePRBotAuthorOverrides:settings.get:35:transport-error"
],
"scenarios": [
"settings-bot-overrides-fulfilled",
"settings-bot-overrides-refused",
"settings-bot-overrides-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/tasks/use-mobile-tasks-runtime-hydration.tsx:hydrateTaskState:settings.get:253",
"manifestScenarioIds": [
"main:mobile/src/tasks/use-mobile-tasks-runtime-hydration.tsx:hydrateTaskState:settings.get:253:fulfilled",
"main:mobile/src/tasks/use-mobile-tasks-runtime-hydration.tsx:hydrateTaskState:settings.get:253:refused",
"main:mobile/src/tasks/use-mobile-tasks-runtime-hydration.tsx:hydrateTaskState:settings.get:253:transport-error"
],
"scenarios": [
"settings-task-hydration-fulfilled",
"settings-task-hydration-refused",
"settings-task-hydration-transport-error"
]
},
{
"manifestOperation": "main:mobile/src/tasks/use-mobile-tasks-workspace-create-actions.tsx:createWorkspace:settings.get:75",
"manifestScenarioIds": [
"main:mobile/src/tasks/use-mobile-tasks-workspace-create-actions.tsx:createWorkspace:settings.get:75:fulfilled",
"main:mobile/src/tasks/use-mobile-tasks-workspace-create-actions.tsx:createWorkspace:settings.get:75:refused",
"main:mobile/src/tasks/use-mobile-tasks-workspace-create-actions.tsx:createWorkspace:settings.get:75:transport-error"
],
"scenarios": [
"settings-task-workspace-fulfilled",
"settings-task-workspace-refused",
"settings-task-workspace-transport-error"
]
}
]
}
Reference in New Issue View Git Blame Copy Permalink