mirror of
https://github.com/stablyai/orca.git
synced 2026-10-06 16:02:25 +00:00
refactor(mobile): pin each golden to its own scenario input, not the whole manifest
`recorderSha256` covered the recorder directory plus `pilot-scenarios.json`, so every golden's header was a function of every other family's scenarios. Adding a family for one domain re-digested all 153 goldens and put a conflict on that line in every domain branch in flight, which serialized the step-4 fan-out. Split the two things it conflated. `recorderSha256` now covers the recorder directory only, with unchanged semantics: a recorder edit still forces a full, deliberate re-record. A new `scenarioSha256` pins the scenario input that golden was recorded from — every scenario `runRecording` consumed for it, in order — canonicalised through `captureValue` so an explicit-undefined param stays distinct from an absent one. `goldenRecording` takes that list instead of just its first member. The variants are hashed rather than the base they expand from because they are what was recorded: a matrix site, its replayed normal result and its partition replies are all visible in them without the derivation having to be restated. `derived-goldens.ts` is that derivation, extracted from `family-recordings.test.ts` so the digest and the recording agree by construction — a property test that restated how a matrix or schedule expands could agree with itself and with nothing else. It reproduces exactly the 153 golden ids on disk, and the census the suite already ran (every family matrixed, no stale normal-result inventory entry) now reads off its output. `golden-header-digest.test.ts` pins the four properties: - a new family in the manifest moves zero existing goldens' headers, and derives two of its own - editing one field of `b1` moves exactly `b1` and its family's four matrix goldens — not the two other `legacy-inventory` scenarios, and not the goldens that expand from `inventory-lifecycle` - editing a recorder file still moves every golden's `recorderSha256`, and no `scenarioSha256` - `recorderSha256` is unchanged by the manifest's contents, and no longer reads the file at all `GOLDEN_FORMAT_VERSION` goes to 4: a version-3 header has no `scenarioSha256`, and `compareGolden` walks the expected header's keys, so a reader that accepted one would compare that golden's own scenarios as though they were unpinned. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
This commit is contained in:
@@ -58,21 +58,38 @@ visible, it does not make the reduction itself observable.
|
||||
## Golden schema
|
||||
|
||||
Each file records `runnerVersion`, `baseline`, `lockfileSha256` (mobile's lockfile),
|
||||
`recorderSha256`, `platform`, `scenarioVersion`, `projectionVersion`, `goldenFormatVersion`,
|
||||
`operation`, `family`, and `namedDeltas`. `platform` and `lockfileSha256` are provenance and are
|
||||
not compared: a dependency or OS that changes behaviour changes the trace itself, so comparing
|
||||
them would only fail candidates on unrelated bumps. The rest are pinned. `recorderSha256` covers every non-markdown file under
|
||||
this directory plus `pilot-scenarios.json`, so the runner that produced a golden is as pinned as
|
||||
the product baseline: editing an adapter projection, a fixture or a scenario fails candidate mode
|
||||
on the header and forces a deliberate re-record. Checkpoints
|
||||
contain ordered sender calls and serialized physical application payloads, action and request
|
||||
settlements, projected state, and ordered external effects. Sender args have three positional
|
||||
slots; absent, undefined and null are distinct `$rpc` tags. Literal objects containing `$rpc`
|
||||
are escaped. Only object keys are sorted; array/effect order, options, budgets, settlement times
|
||||
and errors stay observable. Errors contain category, message and `isRpcDeliveryUnknown`, never
|
||||
`recorderSha256`, `scenarioSha256`, `platform`, `scenarioVersion`, `projectionVersion`,
|
||||
`goldenFormatVersion`, `operation`, `family`, and `namedDeltas`. `platform` and `lockfileSha256`
|
||||
are provenance and are not compared: a dependency or OS that changes behaviour changes the trace
|
||||
itself, so comparing them would only fail candidates on unrelated bumps. The rest are pinned.
|
||||
|
||||
`recorderSha256` covers every non-markdown file under this directory, so the runner that produced a
|
||||
golden is as pinned as the product baseline: editing an adapter projection or a fixture fails
|
||||
candidate mode on the header and forces a deliberate re-record of everything.
|
||||
|
||||
`scenarioSha256` covers the scenario input _that golden_ was recorded from — one manifest scenario
|
||||
for a pilot golden, the generated variants and any hoisted prelude for a matrix or schedule golden,
|
||||
canonicalised by `captureValue` so an explicit-undefined param stays distinct from an absent one.
|
||||
Editing a scenario still fails candidate mode on the header, but only for the goldens derived from
|
||||
it. The manifest used to be an input to `recorderSha256` instead, which made every golden's header
|
||||
a function of every other family's scenarios: adding one domain's family re-digested all 153 files
|
||||
and put a conflict on that line in every domain branch in flight. Which goldens a manifest derives
|
||||
lives in `derived-goldens.ts`, so the digest is a function of the same derivation that records the
|
||||
file rather than of a restatement of it; `golden-header-digest.test.ts` pins the four properties
|
||||
that separation buys.
|
||||
|
||||
Checkpoints contain ordered sender calls and serialized physical application payloads, action and
|
||||
request settlements, projected state, and ordered external effects. Sender args have three
|
||||
positional slots; absent, undefined and null are distinct `$rpc` tags. Literal objects containing
|
||||
`$rpc` are escaped. Only object keys are sorted; array/effect order, options, budgets, settlement
|
||||
times and errors stay observable. Errors contain category, message and `isRpcDeliveryUnknown`, never
|
||||
stack paths, plus `code` and a recursively captured `cause` when the thrown error carries them.
|
||||
Platform is provenance; candidate comparison does not require the same operating system.
|
||||
|
||||
Format version 4 adds `scenarioSha256`; a version-3 header has no such field, so a reader that
|
||||
accepted one would compare that golden's own scenarios as though they were unpinned. The bump moved
|
||||
no observation.
|
||||
|
||||
### Value pool
|
||||
|
||||
Format version 3 stores each distinct observation _entry_ once under `values`, keyed by the first
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
import { hoistPreludeCheckpoints } from './prelude-checkpoints'
|
||||
import { driveReplyMatrix, replyMatrixGoldenId, replyMatrixSites } from './reply-matrix'
|
||||
import { replyMatrixNormalResult } from './reply-matrix-normal-result'
|
||||
import {
|
||||
bindCompletions,
|
||||
interruptionSchedules,
|
||||
lifecycleSchedules,
|
||||
siblingSchedules
|
||||
} from './schedule-driver'
|
||||
import type { RecordingScenario } from './recording-scenario'
|
||||
|
||||
/**
|
||||
* One frozen golden and the scenarios it is recorded from. Derived here rather than inside the
|
||||
* suites so `scenarioSha256` is a function of the same derivation that records the file: a test
|
||||
* that restated how a matrix or schedule expands could agree with itself while disagreeing with
|
||||
* what was recorded.
|
||||
*/
|
||||
export type DerivedGolden = {
|
||||
id: string
|
||||
title: string
|
||||
family: string
|
||||
/** The matrix site this golden drives; absent for a pilot or schedule golden. */
|
||||
site?: string
|
||||
/** Lazy, so a family with no replayable success fails its own test instead of collection. */
|
||||
scenarios: () => RecordingScenario[]
|
||||
timeoutMs?: number
|
||||
}
|
||||
|
||||
const SIBLING_SCHEDULE_BASES = [
|
||||
'b3',
|
||||
'settings-new-tab-ssh',
|
||||
'settings-home-providers-fulfilled',
|
||||
'settings-workspace-context-fulfilled',
|
||||
'settings-resume-metadata-fulfilled',
|
||||
'settings-task-hydration-fulfilled',
|
||||
'settings-repo-metadata-fulfilled'
|
||||
]
|
||||
const INTERRUPTION_BASES = ['inventory-lifecycle', 'settings-bot-overrides-fulfilled']
|
||||
const LIFECYCLE_BASES = [
|
||||
'inventory-lifecycle',
|
||||
'b3',
|
||||
'settings-bot-overrides-fulfilled',
|
||||
'settings-workspace-context-fulfilled',
|
||||
'settings-task-hydration-fulfilled'
|
||||
]
|
||||
|
||||
function baseScenario(manifest: readonly RecordingScenario[], id: string): RecordingScenario {
|
||||
const found = manifest.find((scenario) => scenario.id === id)
|
||||
if (!found) {
|
||||
throw new Error(`No scenario named ${id}`)
|
||||
}
|
||||
return found
|
||||
}
|
||||
|
||||
/** One golden per manifest scenario: frozen main parity for the scenario as written. */
|
||||
export function pilotGoldens(manifest: readonly RecordingScenario[]): DerivedGolden[] {
|
||||
return manifest.map((scenario) => ({
|
||||
id: scenario.id,
|
||||
title: `${scenario.id}: frozen main parity and determinism`,
|
||||
family: scenario.family,
|
||||
scenarios: () => [scenario]
|
||||
}))
|
||||
}
|
||||
|
||||
/** Every golden generated from a family's base scenario: reply matrices and owned schedules. */
|
||||
export function familyGoldens(manifest: readonly RecordingScenario[]): DerivedGolden[] {
|
||||
const families = new Map<string, RecordingScenario[]>()
|
||||
for (const scenario of manifest) {
|
||||
families.set(scenario.family, [...(families.get(scenario.family) ?? []), scenario])
|
||||
}
|
||||
const goldens: DerivedGolden[] = []
|
||||
const ids = new Set<string>()
|
||||
for (const [family, scenarios] of families) {
|
||||
const [base] = scenarios
|
||||
if (!base) {
|
||||
throw new Error(`Family has no scenario: ${family}`)
|
||||
}
|
||||
for (const site of replyMatrixSites(base)) {
|
||||
const id = replyMatrixGoldenId(family, site)
|
||||
if (ids.has(id)) {
|
||||
throw new Error(`Two matrix sites share a golden: ${id}`)
|
||||
}
|
||||
ids.add(id)
|
||||
goldens.push({
|
||||
id,
|
||||
title: `${family}: reply partitions at ${site}`,
|
||||
family,
|
||||
site,
|
||||
timeoutMs: 30_000,
|
||||
scenarios: () =>
|
||||
driveReplyMatrix(base, site, replyMatrixNormalResult(family, scenarios, site))
|
||||
})
|
||||
}
|
||||
}
|
||||
for (const id of SIBLING_SCHEDULE_BASES) {
|
||||
const base = baseScenario(manifest, id)
|
||||
const replies = base.steps.flatMap((step) => ('complete' in step ? [step] : []))
|
||||
// Complete prerequisites before permuting the sibling barrier.
|
||||
const first = replies.find((step) =>
|
||||
step.complete.startsWith(id === 'b3' ? 'linear.getIssue' : 'settings.get')
|
||||
)
|
||||
if (!first) {
|
||||
throw new Error(`No prerequisite completion to order siblings against: ${id}`)
|
||||
}
|
||||
const second = replies[replies.indexOf(first) + 1]
|
||||
if (!second) {
|
||||
continue
|
||||
}
|
||||
goldens.push({
|
||||
id: `schedules-${id}`,
|
||||
title: `${id}: completion orders and correlated faults`,
|
||||
family: base.family,
|
||||
scenarios: () => siblingSchedules(base, first, second)
|
||||
})
|
||||
}
|
||||
for (const id of INTERRUPTION_BASES) {
|
||||
const base = baseScenario(manifest, id)
|
||||
goldens.push({
|
||||
id: `interruptions-${id}`,
|
||||
title: `${id}: timeout, disconnect and stable-client cutover`,
|
||||
family: base.family,
|
||||
scenarios: () => interruptionSchedules(base)
|
||||
})
|
||||
}
|
||||
for (const id of LIFECYCLE_BASES) {
|
||||
const base = baseScenario(manifest, id)
|
||||
const actions: readonly ('reset' | 'unmount' | 'blur')[] = id.includes('hydration')
|
||||
? ['unmount']
|
||||
: id.includes('context')
|
||||
? ['unmount', 'blur']
|
||||
: ['reset', 'unmount', 'blur']
|
||||
goldens.push({
|
||||
id: `lifecycle-${id}`,
|
||||
title: `${id}: lifecycle boundaries`,
|
||||
family: base.family,
|
||||
scenarios: () =>
|
||||
hoistPreludeCheckpoints(
|
||||
{ ...base, steps: bindCompletions(base.steps) },
|
||||
actions
|
||||
.flatMap((action) => lifecycleSchedules(base, action))
|
||||
.filter(({ scenario }) => !id.includes('hydration') || !scenario.id.endsWith('-1'))
|
||||
)
|
||||
})
|
||||
}
|
||||
return goldens
|
||||
}
|
||||
|
||||
/** Every golden the oracle freezes, pilot and family alike. */
|
||||
export function derivedGoldens(manifest: readonly RecordingScenario[]): DerivedGolden[] {
|
||||
return [...pilotGoldens(manifest), ...familyGoldens(manifest)]
|
||||
}
|
||||
@@ -1,18 +1,8 @@
|
||||
import { resolve } from 'node:path'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { readScenarios } from './scenario-input'
|
||||
import { driveReplyMatrix, replyMatrixGoldenId, replyMatrixSites } from './reply-matrix'
|
||||
import {
|
||||
REPLY_MATRIX_NORMAL_RESULT_INVENTORY,
|
||||
replyMatrixNormalResult
|
||||
} from './reply-matrix-normal-result'
|
||||
import {
|
||||
bindCompletions,
|
||||
interruptionSchedules,
|
||||
lifecycleSchedules,
|
||||
siblingSchedules
|
||||
} from './schedule-driver'
|
||||
import { hoistPreludeCheckpoints } from './prelude-checkpoints'
|
||||
import { familyGoldens } from './derived-goldens'
|
||||
import { REPLY_MATRIX_NORMAL_RESULT_INVENTORY } from './reply-matrix-normal-result'
|
||||
import { runRecording } from './run-recording'
|
||||
import { pilotMountAdapters } from './pilot-mount-adapters'
|
||||
import { vitestRecordingScheduler } from './vitest-recording-scheduler'
|
||||
@@ -48,7 +38,7 @@ async function certify(id: string, scenarios: RecordingScenario[]) {
|
||||
checkpoints.push({ ...checkpoint, id: `${scenario.id}:${checkpoint.id}` })
|
||||
}
|
||||
}
|
||||
const golden = goldenRecording(root, input.baseline, scenarios[0], {
|
||||
const golden = goldenRecording(root, input.baseline, scenarios, {
|
||||
scenario: id,
|
||||
checkpoints
|
||||
})
|
||||
@@ -66,95 +56,29 @@ async function certify(id: string, scenarios: RecordingScenario[]) {
|
||||
}
|
||||
|
||||
describe('family reply partitions and owned schedules', () => {
|
||||
const families = new Map<string, RecordingScenario[]>()
|
||||
for (const scenario of input.scenarios) {
|
||||
families.set(scenario.family, [...(families.get(scenario.family) ?? []), scenario])
|
||||
}
|
||||
const goldenIds = new Set<string>()
|
||||
// Filled only when a site actually generates a test, so the census below is independent of
|
||||
// replyMatrixSites throwing on an empty list: the mechanism this replaced skipped families.
|
||||
const matrixed = new Set<string>()
|
||||
const liveSites = new Set<string>()
|
||||
const goldens = familyGoldens(input.scenarios)
|
||||
const families = [...new Set(input.scenarios.map((scenario) => scenario.family))]
|
||||
// Read off the goldens that actually generate a test, so the census is independent of whether
|
||||
// replyMatrixSites would throw on an empty list: the mechanism this replaced skipped families.
|
||||
const sites = goldens.flatMap((golden) => (golden.site ? [golden] : []))
|
||||
it('matrices every family in the manifest', () => {
|
||||
expect([...matrixed]).toEqual([...families.keys()])
|
||||
expect([...new Set(sites.map((golden) => golden.family))]).toEqual(families)
|
||||
})
|
||||
// The inventory is only consulted for a live site, so a stale entry would retire silently.
|
||||
it('lists only live matrix sites in the normal-result inventory', () => {
|
||||
const live = new Set(sites.map((golden) => `${golden.family}\0${golden.site}`))
|
||||
const stale = REPLY_MATRIX_NORMAL_RESULT_INVENTORY.filter(
|
||||
(entry) => !liveSites.has(`${entry.family}\0${entry.request}`)
|
||||
(entry) => !live.has(`${entry.family}\0${entry.request}`)
|
||||
).map((entry) => `${entry.family} ${entry.request}`)
|
||||
expect(stale).toEqual([])
|
||||
})
|
||||
for (const [family, scenarios] of families) {
|
||||
const base = scenarios[0]!
|
||||
for (const request of replyMatrixSites(base)) {
|
||||
const id = replyMatrixGoldenId(family, request)
|
||||
if (goldenIds.has(id)) {
|
||||
throw new Error(`Two matrix sites share a golden: ${id}`)
|
||||
}
|
||||
goldenIds.add(id)
|
||||
matrixed.add(family)
|
||||
liveSites.add(`${family}\0${request}`)
|
||||
it(`${family}: reply partitions at ${request}`, async () => {
|
||||
await certify(
|
||||
id,
|
||||
driveReplyMatrix(base, request, replyMatrixNormalResult(family, scenarios, request))
|
||||
)
|
||||
}, 30_000)
|
||||
}
|
||||
}
|
||||
for (const id of [
|
||||
'b3',
|
||||
'settings-new-tab-ssh',
|
||||
'settings-home-providers-fulfilled',
|
||||
'settings-workspace-context-fulfilled',
|
||||
'settings-resume-metadata-fulfilled',
|
||||
'settings-task-hydration-fulfilled',
|
||||
'settings-repo-metadata-fulfilled'
|
||||
]) {
|
||||
const base = input.scenarios.find((scenario) => scenario.id === id)!
|
||||
const replies = base.steps.filter((step) => 'complete' in step)
|
||||
// Complete prerequisites before permuting the sibling barrier.
|
||||
const first = replies.find((step) =>
|
||||
step.complete.startsWith(id === 'b3' ? 'linear.getIssue' : 'settings.get')
|
||||
)!
|
||||
const second = replies[replies.indexOf(first) + 1]
|
||||
if (!second) {
|
||||
continue
|
||||
}
|
||||
it(`${id}: completion orders and correlated faults`, async () => {
|
||||
await certify(`schedules-${id}`, siblingSchedules(base, first, second))
|
||||
})
|
||||
}
|
||||
for (const id of ['inventory-lifecycle', 'settings-bot-overrides-fulfilled']) {
|
||||
const base = input.scenarios.find((scenario) => scenario.id === id)!
|
||||
it(`${id}: timeout, disconnect and stable-client cutover`, async () => {
|
||||
await certify(`interruptions-${id}`, interruptionSchedules(base))
|
||||
})
|
||||
}
|
||||
for (const id of [
|
||||
'inventory-lifecycle',
|
||||
'b3',
|
||||
'settings-bot-overrides-fulfilled',
|
||||
'settings-workspace-context-fulfilled',
|
||||
'settings-task-hydration-fulfilled'
|
||||
]) {
|
||||
const base = input.scenarios.find((scenario) => scenario.id === id)!
|
||||
const actions = id.includes('hydration')
|
||||
? (['unmount'] as const)
|
||||
: id.includes('context')
|
||||
? (['unmount', 'blur'] as const)
|
||||
: (['reset', 'unmount', 'blur'] as const)
|
||||
it(`${id}: lifecycle boundaries`, async () => {
|
||||
await certify(
|
||||
`lifecycle-${id}`,
|
||||
hoistPreludeCheckpoints(
|
||||
{ ...base, steps: bindCompletions(base.steps) },
|
||||
actions
|
||||
.flatMap((action) => lifecycleSchedules(base, action))
|
||||
.filter(({ scenario }) => !id.includes('hydration') || !scenario.id.endsWith('-1'))
|
||||
)
|
||||
)
|
||||
})
|
||||
for (const golden of goldens) {
|
||||
it(
|
||||
golden.title,
|
||||
async () => {
|
||||
await certify(golden.id, golden.scenarios())
|
||||
},
|
||||
golden.timeoutMs
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { afterAll, describe, expect, it } from 'vitest'
|
||||
import { derivedGoldens } from './derived-goldens'
|
||||
import { goldenRecording, type GoldenRecording } from './golden-recording'
|
||||
import { RECORDER_DIRECTORY, recorderSha256 } from './recorder-digest'
|
||||
import { readScenarios } from './scenario-input'
|
||||
import type { RecordingScenario } from './recording-scenario'
|
||||
|
||||
const root = resolve(import.meta.dirname, '../../../..')
|
||||
const manifest = readScenarios(
|
||||
process.env.RPC_FOUNDATION_SCENARIOS ??
|
||||
resolve(root, 'mobile/rpc-foundation/pilot-scenarios.json')
|
||||
).scenarios
|
||||
const BASELINE = 'a'.repeat(40)
|
||||
const EDITED_SCENARIO = 'b1'
|
||||
const EDITED_SITE = 'files.searchPaths#1'
|
||||
/**
|
||||
* Every golden derived from `b1`: its own, and its family's four matrix sites, which expand from it
|
||||
* as the family's base. The two other `legacy-inventory` scenarios and the interruption and
|
||||
* lifecycle goldens that expand from `inventory-lifecycle` are deliberately absent.
|
||||
*/
|
||||
const EDITED_GOLDENS = [
|
||||
'b1',
|
||||
'matrix-legacy-inventory-files.searchpaths-1',
|
||||
'matrix-legacy-inventory-files.searchpaths-2',
|
||||
'matrix-legacy-inventory-fresh-inventory',
|
||||
'matrix-legacy-inventory-old-inventory'
|
||||
]
|
||||
type Header = Omit<GoldenRecording, 'recording'>
|
||||
|
||||
const created: string[] = []
|
||||
afterAll(() => {
|
||||
for (const directory of created) {
|
||||
rmSync(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
/** The two files a root contributes to a header, plus the manifest the old digest also read. */
|
||||
function stubRoot(recorder: string, scenarioFile?: string): string {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'rpc-header-'))
|
||||
created.push(directory)
|
||||
mkdirSync(join(directory, RECORDER_DIRECTORY), { recursive: true })
|
||||
writeFileSync(join(directory, RECORDER_DIRECTORY, 'runner.ts'), recorder)
|
||||
writeFileSync(join(directory, 'mobile/pnpm-lock.yaml'), 'lockfile: stub\n')
|
||||
if (scenarioFile !== undefined) {
|
||||
mkdirSync(join(directory, 'mobile/rpc-foundation'), { recursive: true })
|
||||
writeFileSync(join(directory, 'mobile/rpc-foundation/pilot-scenarios.json'), scenarioFile)
|
||||
}
|
||||
return directory
|
||||
}
|
||||
|
||||
/** Every golden's header for one recorder revision and one manifest, both written to a stub root. */
|
||||
function headers(recorder: string, scenarios: readonly RecordingScenario[]): Map<string, Header> {
|
||||
const stub = stubRoot(recorder, JSON.stringify({ baseline: BASELINE, scenarios }))
|
||||
return new Map(
|
||||
derivedGoldens(scenarios).map((golden) => {
|
||||
const { recording: _recording, ...header } = goldenRecording(
|
||||
stub,
|
||||
BASELINE,
|
||||
golden.scenarios(),
|
||||
{ scenario: golden.id, checkpoints: [] }
|
||||
)
|
||||
return [golden.id, header]
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
function moved(before: Map<string, Header>, after: Map<string, Header>): string[] {
|
||||
return [...before]
|
||||
.filter(([id, header]) => JSON.stringify(after.get(id)) !== JSON.stringify(header))
|
||||
.map(([id]) => id)
|
||||
.sort()
|
||||
}
|
||||
|
||||
/** A family no other golden consumes, with one reply the matrix can replay as its success. */
|
||||
const ADDED_FAMILY: RecordingScenario = {
|
||||
id: 'digest-probe',
|
||||
operation: 'digest.probe',
|
||||
version: 1,
|
||||
family: 'digest-probe',
|
||||
sites: [],
|
||||
schedules: ['probe'],
|
||||
steps: [
|
||||
{
|
||||
complete: 'probe.read#1',
|
||||
params: { worktree: 'id:A' },
|
||||
reply: { ok: true, result: { probed: true } }
|
||||
},
|
||||
{ checkpoint: 'settled' }
|
||||
]
|
||||
}
|
||||
|
||||
function editOneScenarioField(scenarios: readonly RecordingScenario[]): RecordingScenario[] {
|
||||
let edits = 0
|
||||
const edited = scenarios.map((scenario) =>
|
||||
scenario.id !== EDITED_SCENARIO
|
||||
? scenario
|
||||
: {
|
||||
...scenario,
|
||||
steps: scenario.steps.map((step) => {
|
||||
if (!('complete' in step) || step.complete !== EDITED_SITE) {
|
||||
return step
|
||||
}
|
||||
edits++
|
||||
return { ...step, params: { worktree: 'id:A', query: 'old', limit: 17 } }
|
||||
})
|
||||
}
|
||||
)
|
||||
if (edits !== 1) {
|
||||
throw new Error(`Expected one ${EDITED_SITE} step in ${EDITED_SCENARIO}, edited ${edits}`)
|
||||
}
|
||||
return edited
|
||||
}
|
||||
|
||||
describe('golden header digests', () => {
|
||||
it('re-digests nothing when the manifest gains a family', () => {
|
||||
const before = headers('export const runner = 1', manifest)
|
||||
const after = headers('export const runner = 1', [...manifest, ADDED_FAMILY])
|
||||
expect(moved(before, after)).toEqual([])
|
||||
// The added family did derive goldens of its own: a pilot golden and one matrix site.
|
||||
expect(after.size).toBe(before.size + 2)
|
||||
})
|
||||
|
||||
it('re-digests exactly the goldens derived from an edited scenario', () => {
|
||||
const before = headers('export const runner = 1', manifest)
|
||||
const after = headers('export const runner = 1', editOneScenarioField(manifest))
|
||||
expect(moved(before, after)).toEqual([...EDITED_GOLDENS].sort())
|
||||
for (const id of EDITED_GOLDENS) {
|
||||
expect(after.get(id)?.recorderSha256).toBe(before.get(id)?.recorderSha256)
|
||||
expect(after.get(id)?.scenarioSha256).not.toBe(before.get(id)?.scenarioSha256)
|
||||
}
|
||||
})
|
||||
|
||||
it('re-digests every golden when a recorder file changes', () => {
|
||||
const before = headers('export const runner = 1', manifest)
|
||||
const after = headers('export const runner = 2', manifest)
|
||||
expect(moved(before, after)).toEqual([...before.keys()].sort())
|
||||
for (const [id, header] of before) {
|
||||
expect(after.get(id)?.recorderSha256).not.toBe(header.recorderSha256)
|
||||
expect(after.get(id)?.scenarioSha256).toBe(header.scenarioSha256)
|
||||
}
|
||||
})
|
||||
|
||||
it('digests the recorder without reading the scenario manifest', () => {
|
||||
const one = stubRoot('export const runner = 1', JSON.stringify({ baseline: BASELINE }))
|
||||
const other = stubRoot('export const runner = 1', '{"scenarios":"edited"}')
|
||||
const absent = stubRoot('export const runner = 1')
|
||||
expect(recorderSha256(other)).toBe(recorderSha256(one))
|
||||
expect(recorderSha256(absent)).toBe(recorderSha256(one))
|
||||
})
|
||||
})
|
||||
@@ -11,14 +11,16 @@ import {
|
||||
type ValuePool
|
||||
} from './golden-value-pool'
|
||||
import { recorderSha256 } from './recorder-digest'
|
||||
import { scenarioSha256 } from './scenario-digest'
|
||||
import type { Recording, RecordingScenario } from './recording-scenario'
|
||||
import type { RecordedValue } from './recording-values'
|
||||
|
||||
export const RUNNER_VERSION = 1
|
||||
// 2 stamps every settlement with startedAt/settledAt on the pinned virtual clock.
|
||||
export const PROJECTION_VERSION = 2
|
||||
// 3 interns each entry of a list or map field, not the whole field; an older file is not comparable.
|
||||
export const GOLDEN_FORMAT_VERSION = 3
|
||||
// 4 pins scenarioSha256 per golden; a version-3 header has no such field, so it would compare as
|
||||
// though the golden's own scenarios were unpinned.
|
||||
export const GOLDEN_FORMAT_VERSION = 4
|
||||
export type GoldenRecording = {
|
||||
operation: string
|
||||
family: string
|
||||
@@ -27,6 +29,7 @@ export type GoldenRecording = {
|
||||
baseline: string
|
||||
lockfileSha256: string
|
||||
recorderSha256: string
|
||||
scenarioSha256: string
|
||||
platform: string
|
||||
scenarioVersion: number
|
||||
projectionVersion: number
|
||||
@@ -40,9 +43,13 @@ type GoldenFile = Omit<GoldenRecording, 'recording'> & {
|
||||
export function goldenRecording(
|
||||
root: string,
|
||||
baseline: string,
|
||||
scenario: RecordingScenario,
|
||||
scenarios: readonly RecordingScenario[],
|
||||
recording: Recording
|
||||
): GoldenRecording {
|
||||
const [scenario] = scenarios
|
||||
if (!scenario) {
|
||||
throw new Error('A golden records at least one scenario')
|
||||
}
|
||||
return {
|
||||
operation: scenario.operation,
|
||||
family: scenario.family,
|
||||
@@ -53,6 +60,7 @@ export function goldenRecording(
|
||||
.update(readFileSync(join(root, 'mobile/pnpm-lock.yaml')))
|
||||
.digest('hex'),
|
||||
recorderSha256: recorderSha256(root),
|
||||
scenarioSha256: scenarioSha256(scenarios),
|
||||
platform: process.platform,
|
||||
scenarioVersion: scenario.version,
|
||||
projectionVersion: PROJECTION_VERSION,
|
||||
|
||||
@@ -90,7 +90,7 @@ describe('RPC main recordings', () => {
|
||||
loading: false
|
||||
})
|
||||
}
|
||||
const golden = goldenRecording(root, input.baseline, scenario, recording)
|
||||
const golden = goldenRecording(root, input.baseline, [scenario], recording)
|
||||
const bytes = goldenBytes(golden)
|
||||
if (run) {
|
||||
expect(bytes).toBe(first)
|
||||
|
||||
@@ -3,7 +3,6 @@ import { readFileSync, readdirSync } from 'node:fs'
|
||||
import { join, posix } from 'node:path'
|
||||
|
||||
export const RECORDER_DIRECTORY = 'mobile/src/test-support/rpc-recording'
|
||||
export const RECORDER_SCENARIO_INPUT = 'mobile/rpc-foundation/pilot-scenarios.json'
|
||||
const digests = new Map<string, string>()
|
||||
|
||||
function collect(root: string, relative: string, files: string[]): void {
|
||||
@@ -20,9 +19,14 @@ function collect(root: string, relative: string, files: string[]): void {
|
||||
}
|
||||
|
||||
/**
|
||||
* Every executable recorder input, so a golden is attributable to one runner and one scenario file.
|
||||
* Prose is excluded because it cannot change a recording; a candidate run recomputes this and
|
||||
* `compareGolden` fails the header, which forces a recorder edit to re-record deliberately.
|
||||
* Every executable recorder input, so a golden is attributable to one runner. Prose is excluded
|
||||
* because it cannot change a recording; a candidate run recomputes this and `compareGolden` fails
|
||||
* the header, which forces a recorder edit to re-record deliberately.
|
||||
*
|
||||
* The scenario manifest is deliberately not an input. It used to be, which made every golden's
|
||||
* header a function of every other family's scenarios: adding one family re-digested all 153 files
|
||||
* and put a conflict on that line in every domain branch. `scenarioSha256` pins each golden to the
|
||||
* scenarios it was actually recorded from instead.
|
||||
*/
|
||||
export function recorderSha256(root: string): string {
|
||||
const cached = digests.get(root)
|
||||
@@ -31,7 +35,6 @@ export function recorderSha256(root: string): string {
|
||||
}
|
||||
const files: string[] = []
|
||||
collect(root, RECORDER_DIRECTORY, files)
|
||||
files.push(RECORDER_SCENARIO_INPUT)
|
||||
const digest = createHash('sha256')
|
||||
.update(
|
||||
files
|
||||
|
||||
@@ -390,8 +390,6 @@ describe('recording boundaries', () => {
|
||||
try {
|
||||
const directory = join(root, RECORDER_DIRECTORY)
|
||||
mkdirSync(directory, { recursive: true })
|
||||
mkdirSync(join(root, 'mobile/rpc-foundation'), { recursive: true })
|
||||
writeFileSync(join(root, 'mobile/rpc-foundation/pilot-scenarios.json'), '{}')
|
||||
writeFileSync(join(directory, 'runner.ts'), 'export const runner = 1')
|
||||
const original = recorderSha256(root)
|
||||
writeFileSync(join(directory, 'README.md'), 'prose')
|
||||
@@ -466,6 +464,7 @@ function sampleGolden(id: string): GoldenRecording {
|
||||
baseline: 'a'.repeat(40),
|
||||
lockfileSha256: 'b'.repeat(64),
|
||||
recorderSha256: 'c'.repeat(64),
|
||||
scenarioSha256: 'd'.repeat(64),
|
||||
platform: process.platform,
|
||||
scenarioVersion: 1,
|
||||
projectionVersion: PROJECTION_VERSION,
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { canonicalJson } from './golden-value-pool'
|
||||
import { captureValue } from './recording-values'
|
||||
import type { RecordingScenario } from './recording-scenario'
|
||||
|
||||
/**
|
||||
* The scenario input one golden was recorded from: every scenario the runner consumed for it, in
|
||||
* order — one manifest scenario for a pilot golden, the generated variants (and any hoisted
|
||||
* prelude) for a matrix or schedule golden.
|
||||
*
|
||||
* Pinned per golden rather than over the whole manifest so a family added for one domain moves only
|
||||
* its own goldens, while a field edited inside a scenario moves every golden derived from it. The
|
||||
* variants are hashed rather than the base they came from because they are what `runRecording`
|
||||
* consumed: a matrix site, its replayed normal result and its partition replies are all visible
|
||||
* here without the derivation having to be restated.
|
||||
*
|
||||
* `captureValue` sorts object keys and tags an explicit-undefined param, which `JSON.stringify`
|
||||
* would drop and so conflate with an absent one.
|
||||
*/
|
||||
export function scenarioSha256(scenarios: readonly RecordingScenario[]): string {
|
||||
if (!scenarios.length) {
|
||||
throw new Error('A golden records at least one scenario')
|
||||
}
|
||||
return createHash('sha256')
|
||||
.update(canonicalJson(captureValue(scenarios)))
|
||||
.digest('hex')
|
||||
}
|
||||
Reference in New Issue
Block a user