mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 16:02:32 +00:00
Jinwoo Hong
7ce8e18d07
* test(mobile): record main RPC hooks and regression schedules Add scripted sender recordings, guarded main goldens, reply matrices, lifecycle schedules, settings caller fixtures, and targeted B-seed mutants. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): flush recording user actions through React act Keep lifecycle updates in separate act boundaries while wrapping direct stateful user actions. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): compile recorded modules with the Node VM API Use the same trusted-source execution boundary as existing mobile VM test harnesses. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): pool golden values and hoist pre-divergence checkpoints Golden format version 2 stores each distinct observation field value once in a `values` map keyed by a 12-hex sha256 of its sorted-key JSON, and a checkpoint references five hashes. Output stays pretty-printed; the reader rejects any other format version, resolves hashes back to values, and reports the scenario, checkpoint, field and JSON path on a mismatch. Generated variants now declare where their distinguishing input lands, so checkpoints observed before that point are recorded once in a `.prelude` scenario instead of once per reply partition. Reply matrices, interruption schedules and lifecycle schedules share the primitive, which asserts each variant's pre-divergence prefix matches the base. Equal-but-differently-reached checkpoints are untouched. 17.71 MB / 3,599 checkpoints / 58.4% intra-file duplicates becomes 4.21 MB / 1,961 checkpoints / 23.6%, with every file's set of distinct observations unchanged. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): make the recordings sense deadlines, the recorder, and every family The goldens carried no temporal information, so a request deadline could be cut to a third and all 61 files stayed byte-identical. Every threshold is now straddled by two advances with a checkpoint between them: the 30 s request deadline in both schedule drivers, the 120 ms search debounce in b1, and the 60 s repo-metadata cache TTL. Shortening any of them moves an observation. The record fence pinned product sources but excluded the whole recorder, so --record could rewrite every golden from a modified runner and report the baseline intact. Goldens now pin recorderSha256 over every non-markdown file in the runner plus pilot-scenarios.json, and the fence exemption shrinks to the one directory that digest covers. Mutation evidence covered 3 of 13 mounted operations. There is now one anchored mutant per adapter family, covering 11 operations and 51 of the 61 goldens; the two omitted are the pure async loaders whose entire output is their settlement. Anchors are asserted to match exactly one site, which caught the acceptance mutant silently half-applying against three identical guards. The archived-tree assertion pins each seed's visible state instead of merely differing from main, and error observations carry code and cause when present. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): record settlement times instead of straddling deadlines The previous commit made the reviewer's divide-by-three deadline mutant fail by placing checkpoints on each side of the 30 s deadline. That is a patch: a timing change that does not cross a hand-placed boundary stays invisible. Those scenario edits are reverted, and pilot-scenarios.json and schedule-driver.ts are byte-identical to what they were before them. The real defect was that the projection had no temporal dimension, so every settlement now carries startedAt and settledAt in virtual milliseconds on the pinned fake clock. Any transition the product schedules for itself is recorded at the time it actually fires, so a deadline or debounce change of any size, in either direction, moves a recorded number. A checkpoint's own clock is not recorded. It is always the sum of the scripted advances, so it is a function of the scenario rather than of the code under test; run-recording.ts asserts that equality at every checkpoint instead, which costs no bytes and fails loudly if it ever drifts. projectionVersion is 2 and all 61 goldens are re-recorded. With the added timestamps stripped, the distinct-observation set is identical to the previous recording, so the change is purely additive. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): probe the repo-metadata cache inside its TTL window Recorded settlement times cover thresholds the product schedules for itself, but not one it only consults when something else makes it act. The repo-metadata TTL is the single such case: with probes only at 0 s and 60 s, a 20 s TTL and a 60 s TTL are both expired at 60 s and record identically, so a 3x cache-lifetime regression was invisible. settings-repo-cache-expiry now probes the cache at 59 s as well. This is coverage, not a substitute for recorded time: it bounds how small a TTL reduction is visible rather than making the reduction itself observable, and the README says so. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): record the reply shapes a host can send, not a cross product The reply matrix froze ~26 malformed envelopes crossed against every consumed field and three boundary kinds, which is 163,925 lines of JSON pinning accidents on inputs no desktop produces. `successResponse` always sets `result`, so a JSON wire has no explicit-undefined slot, and no mounted handler returns a number, a string, an array, a bare `{}` or a boolean: `settings.get` returns `{settings: ...}`, and the seed methods return an object or nothing. Each family now runs nine witnessed partitions once, with no field cross: a normal result, an absent result, `null`, an inner `{ok: false}` envelope with a string or an object error, an inner envelope missing `ok`, an outer refusal, `method_not_found`, and a transport rejection. `null` stays because `linear.getIssue` returns it for a missing issue and b2 is a shipped null-result bug; it is also what carries the one named delta these goldens record. `run-step1-exit.ts` had zero callers and shelled out to the same two Vitest files as `rpc-recording.mts`, so it and its README paragraph go, along with `MUTATION_NAMES`, which only it read. In the module loader, the `rpc-delivery-ambiguity` escape is measured dead: over every scenario, mutant and reference run it was taken once, by the test that existed to take it. Golden comparison already fails loudly if a mounted module ever imports the marker, so both go. The history-panel exposure moves into a declarative table beside the mutation anchors, leaving the loader with one source-text mechanism and no per-file branch. The VM stays. Mount adapters load product sources from an arbitrary `root`, and the archivedbcba08b3e4tree is bare `mobile/src` and `src` with no package.json and no node_modules, so no bundler-resolved import can reach it and the seed rejection gate cannot run without it. Direct import also swaps a 38-module lazy graph for a 338-module eager one behind 20 native mocks, because `mobile-tasks-dependencies.ts` re-exports from `react-native` and four other native packages and `export *` enumerates. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): treat the recorded lockfile hash as provenance, not an oracle Every golden pinned `lockfileSha256`, so any dependency bump on main failed all 61 comparisons on the merge commit while the traces were identical. A dependency that changes behaviour changes the trace itself; one that does not must not fail a candidate. `platform` already had this exemption — `lockfileSha256` joins it. Recording still refuses to run unless the lockfile matches the pinned baseline, so goldens are still produced under frozen conditions. Verified against main's lockfile: 85 passed, previously 61 failed. The declared mutation set still reports every mutant killed. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(rpc): cover repeat queries and settings refresh boundaries Add three scenarios, preserve existing traces, remove unreachable archived checks, and document observed mutation kills and remaining adapter limitations. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * docs(rpc): keep the known-open holes, drop the review transcript The audit file was mostly a point-in-time record of mutation runs that had already happened, in an artifacts directory, where it would go stale on the next scenario change. The durable part is which holes are still open and why they cannot be reached, which belongs beside the runner it describes. Markdown is outside recorderSha256, so no re-record; 88 passed | 3 skipped. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * chore(mobile): stage the nine RPC probe scenarios and goldens These existed only on one machine's /tmp. Landing them verbatim first so a reboot cannot lose them; a follow-up commit moves them into the suite. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): fold the nine probe scenarios into the recording oracle The probes were env-var invocations over loose /tmp manifests. They now live in pilot-scenarios.json and rpc-foundation/goldens, so `pnpm --dir mobile test` runs them with no flag to remember. Re-records every golden against main (22f56f7c2a). Two causes: - #20280 gave LogicalClientCutoverError the delivery-unknown mark and its cause, so nine cutover/interruption goldens now record `isRpcDeliveryUnknown: true` plus a `Connection closed` cause. The other 55 are byte-identical after 260 commits of main. - #20499 replaced the five anchored raw-envelope reads with typed operations, so those mutation anchors matched zero sites. Each is re-anchored at the same defect's new home; bot-overrides moves to the shared reader that now owns it. probe-hole-witness.test.ts pins hole and closure together: a probe must kill its mutation and every pre-probe scenario of the same operation must still survive it, so a redundant probe fails instead of accumulating. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): list the recording harness in the raw-port inventory main's #20026 boundary test fails on any non-test file that reaches the raw request port and is not inventoried. The oracle's scripted transport is exactly that — it drives the real tracker and logical client — so it belongs in OWNERS beside the supervisor fakes, not in the step-4 pending backlog. Also states what the oracle covers, the two holes it was blind to until the probes, and the step-4 runbook. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): pin the goldens to the tree that recorded them The inventory entry is a fenced product-tree edit, so --record refused against main's sha. Baseline now names the branch commit the goldens were recorded from; the next re-record after this lands bumps it to the merge commit. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): carry a SAFETY rationale on every recorder cast main added a changed-code casting gate after this branch was cut, so 45 `as` sites in the recorder read as new findings. Each now states why the assertion holds; they cluster into five reasons — recorded observations are RecordedValue by construction, parsed manifests and goldens are validated on the next lines, interned pools resolve their own hashes, a VM-evaluated module has no static type, and the mount adapters supply only the members each hook reads. Re-records the goldens: the comments move recorderSha256. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * docs(rpc): state the measured blindness, not the assumed one Applying each mutation to real product source shows the two holes are not equal. The reorder is invisible to 83 of 84 tests and only a probe sees it. The refusal blanking is also caught by the family reply matrix, because a refusal from cold publishes null over a non-null initial value — an observational gap, not a detection gap. Says so rather than letting the stronger claim carry both. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): close three ways the oracle could pass without checking All four review findings were real; three let the oracle report green while verifying less than it claimed. - The baseline guard used `git diff --quiet`, which ignores untracked files, so an untracked module under mobile/src or src/shared could change resolution while a golden still recorded a pinned baseline header. Adds a `git ls-files --others` check over the same paths, recorder still exempt. - The determinism loop read `Number(env ?? 2)` unvalidated, so RPC_FOUNDATION_DETERMINISM_RUNS=0 skipped the body and 57 tests passed having recorded and compared nothing. Now requires an integer >= 2. - Cleanup-time observations were dropped: every checkpoint clones the effects array, so anything appended during dispose or the final flush never reached a golden. Warns and documents the six scenarios that hit it today; recording them changes every golden and is its own change. - Two SAFETY rationales described each other's assertion. Swapped. Goldens re-recorded for the recorder-digest change: 73 files, one header line each, no recorded observation moved. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): record teardown observations as a cleanup checkpoint Each checkpoint clones the effects array, so a rejection or state write produced by dispose, the transport teardown or the final flush landed after the recording was built and never reached a golden. An unmount leak is exactly what this oracle exists to catch, so teardown now runs on the recorded path and anything it observes becomes a checkpoint with id `cleanup`. State is captured before dispose, since the operation is gone afterwards. Six scenarios were dropping observations, across five goldens: projectRowDetailError, projectMutating, hostLabelById, hostPlatform, workspaceAgent, workspaceAgentOverridden, creatingKey, selectedAgent, agentOverridden and error. Those five gain a cleanup checkpoint; the other 68 goldens change by their header line only, so no existing observation moved. Also fixes the README's own formatting, which failed `format:check`. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb