Files
orca/config/scripts/windows-pty-table-stress.cjs
T
Neil 995715b19b test(windows): record native PTY stress lifecycle milestones (#25042)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​562 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​562 |
| Prod | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​220 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​8 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​212 |

<!-- /orca-pr-loc -->

## ELI5

When a Windows terminal stress test stalls, its old log cannot show where progress stopped. The test now records bounded, sanitized lifecycle observations while keeping its existing assertions and timing.

## What Changed

Record output-pipe, worker, native exit and final callback milestones, pending state at existing deadlines, and hashes of the native addon and supporting files. Redact addresses and credentials even when terminal controls split them, preserving the controls’ positions. Keep the warmup survivor and newest 31 terminals; later terminals still receive observers, with omissions counted explicitly.

## Why

[The failing Windows job](https://github.com/stablyai/orca/actions/runs/37132029374/job/111229398978) had one silent terminal and at least one undrained callback. [An earlier passing job](https://github.com/stablyai/orca/actions/runs/37123932023/job/111205697896) used the same source and cache inputs, without a loaded-addon hash. Observing the existing objects supplies missing evidence while preserving the gate. Bounded recent records retain the terminals most likely involved in a late failure.

## Linked Issue

Diagnosis of PR #25031’s Windows package failure. This does not claim that the original native cause is fixed or close a product issue.

## Visual Proof

N/A — test diagnostics add no UI or interaction change.

## Testing

- [ ] I manually tested these changes locally
- [x] Automated tests added and updated for reproduced failures.

On the exact previous public helper, eight strengthened controls failed and six passed. They reproduce control-interrupted credential/address leaks, omitted late terminals after eleven rounds, and late state after the 256-milestone cap. Independent review then reproduced six more leaks on the intermediate helper: usernames and hostnames masked only part of an email. The final helper runs the existing same-length redactor a second time, removing the remaining private domain while preserving controls and OSC framing. Those six before failures and two credential-overlap controls are retained. Corrected source `5037c18a898571539ade5336ad3519cd49dfb20f` passes 171 controls in four files, including all 22 observer controls, the full anti-slop audit, syntax/AST checks, focused format/lint and six quality gates.

The previous public source `f1ed3e97` passed [actual Windows CI](https://github.com/stablyai/orca/actions/runs/37136703455/job/111243011997): 25 PTYs, eight rounds, 526 passing tests and 26 skips, plus build, package and smoke checks. Its native stress/driver files remain exact here; the observer has changed, so fresh normal CI must qualify this complete correction. The passing run’s native-addon hash differs from the historical failure; it does not explain that failure.

## Review

All four original diagnostic files and public ancestry are retained; all 32,245 paths outside that scope match main `786a040b`. The existing redactor runs twice on a view with presentation controls removed, then controls are restored at their original positions; OSC framing separates title payloads from adjacent text. Logs retain 32 terminal records and 256 milestones with explicit omissions, plus bounded final snapshots. The native callback preserves its receiver, arguments, result and thrown errors; error observers preserve unhandled-error behavior.

Native assertions, deadlines, input, concurrency and cleanup remain exact. Companion hashes identify file bytes rather than in-memory DLL equivalence. The original native cause remains unverifiable.

## Agent skill upstream boundary

- [x] Not applicable; no upstream skill source is copied.

## Notes

No production API, RPC, native patch, dependency or workflow change. Blank output or a missing callback is not evidence of process death. Local helper controls run on macOS; the existing real ConPTY gate exercises Windows.

## Checklist

- [x] Small, focused diagnostic scope.
- [x] Explained the before/after, mechanism and choice.
- [x] Visual proof N/A with reason.
- [x] Self-reviewed privacy, bounded logging and callback behavior.
- [x] Cross-platform and remote impact considered.
- [ ] Fresh normal Windows CI must qualify the complete corrected source.
2026-10-03 11:24:46 -07:00

172 lines
5.9 KiB
JavaScript

'use strict'
const assert = require('node:assert/strict')
const { createHash } = require('node:crypto')
const { readFileSync, writeSync } = require('node:fs')
const { dirname, resolve } = require('node:path')
function report(phase, details = {}) {
writeSync(1, `${JSON.stringify({ phase, hostPid: process.pid, ...details })}\n`)
}
async function exerciseTable() {
assert.equal(process.platform, 'win32', 'This probe requires real Windows ConPTY')
const rounds = Number(process.env.ORCA_PTY_TABLE_STRESS_ROUNDS ?? 8)
assert.ok(Number.isInteger(rounds) && rounds > 0 && rounds <= 2000)
const { createStressObserver, loadedStressInputHashes, sanitizeStressText } =
await import('./windows-pty-table-stress-observer.mjs')
const observer = createStressObserver(report)
const pty = require('node-pty')
const nativePath = require.resolve('node-pty/lib/utils')
const loaded = require(nativePath).loadNativeModule('conpty')
const native = loaded.module
const addonPath = resolve(dirname(nativePath), loaded.dir, 'conpty.node')
report('native', {
addonPath: sanitizeStressText(addonPath),
sha256: createHash('sha256').update(readFileSync(addonPath)).digest('hex'),
node: process.version,
rounds,
inputs: loadedStressInputHashes(addonPath, require.resolve)
})
// Unlike production's fallback, this crash probe requires a host that permits nested jobs.
const hostJobAssigned = native.assignCurrentProcessToJob()
report('host-job-precondition', { assigned: hostJobAssigned })
assert.equal(hostJobAssigned, true, 'Probe precondition: host must permit a crash-cleanup job')
const spawned = []
function spawn(round, slot) {
report('spawn', { round, slot })
const proc = pty.spawn(process.env.ComSpec || 'cmd.exe', ['/d', '/q'], {
cwd: process.cwd(),
cols: 80,
rows: 24,
useConptyDll: true
})
const record = { proc, output: '', exited: false, closed: false }
const marker = `ORCA_PTY_READY_${spawned.length}`
let resolveReady
record.ready = new Promise((resolve) => {
resolveReady = resolve
})
record.exit = new Promise((resolveExit) => {
proc.onExit((event) => {
record.exited = true
resolveReady(false)
resolveExit(event)
})
})
proc.onData((chunk) => {
record.output = (record.output + chunk).slice(-2048)
if (record.output.includes(marker)) {
resolveReady(true)
}
})
observer.watch(record, { round, slot })
spawned.push(record)
// Escaping one letter keeps echoed input from satisfying the output marker.
proc.write(`echo ${marker.replace('READY', 'REA^DY')}\r`)
return record
}
async function waitForReady(records) {
let timer
try {
await Promise.race([
Promise.all(
records.map(async (record) => {
assert.equal(await record.ready, true, `Shell exited before ready: ${record.output}`)
})
),
new Promise((_, reject) => {
timer = setTimeout(() => {
observer.pending('readiness-timeout-state')
const transcripts = records.map(({ proc, output }) => ({
pid: proc.pid,
output: sanitizeStressText(output)
}))
reject(new Error(`PTY readiness timed out: ${JSON.stringify(transcripts)}`))
}, 15_000)
})
])
} finally {
clearTimeout(timer)
}
for (const { proc } of records) {
const members = native.listJobProcessIds(proc._pty, proc.pid)
assert.ok(members?.includes(proc.pid), `Ready shell ${proc.pid} must retain its job`)
}
report('ready', { shellPids: records.map(({ proc }) => proc.pid) })
}
function close(record, round, slot) {
report('kill', { round, slot, shellPid: record.proc.pid })
record.proc.kill()
record.closed = true
observer.checkpoint('kill-returned-state', record)
}
let failure
try {
const survivor = spawn(-1, -1)
await waitForReady([survivor])
const slots = []
for (let round = 0; round < rounds; round += 1) {
for (let slot = 0; slot < 3; slot += 1) {
if (slots[slot]) {
close(slots[slot], round, slot)
}
// The next lookup/insertion overlaps the previous shell's native exit watcher.
slots[slot] = spawn(round, slot)
report('resize-clear-list', { round, slot, shellPid: survivor.proc.pid })
survivor.proc.resize(80 + (round % 2), 24)
survivor.proc.clear()
const members = native.listJobProcessIds(survivor.proc._pty, survivor.proc.pid)
assert.ok(members?.includes(survivor.proc.pid), 'A live survivor must retain its job')
}
// A ready terminal runs kill/resize/clear immediately instead of deferring them.
await waitForReady(slots)
}
assert.equal(survivor.exited, false, survivor.output)
report('overlap-complete', { terminals: spawned.length })
} catch (error) {
failure = { error }
} finally {
for (const record of spawned) {
if (!record.closed) {
close(record, -1, -1)
}
}
let timer
try {
await Promise.race([
Promise.all(spawned.map((record) => record.exit)),
new Promise((_, reject) => {
timer = setTimeout(() => {
observer.pending('exit-drain-timeout-state')
reject(new Error('PTY exit callbacks did not drain'))
}, 15_000)
})
])
} catch (error) {
if (failure) {
report('drain-error', { message: sanitizeStressText(error.stack) })
} else {
failure = { error }
}
} finally {
clearTimeout(timer)
}
}
if (failure) {
throw failure.error
}
observer.pending('complete-state')
report('complete', { terminals: spawned.length })
}
exerciseTable().catch(async (error) => {
const { sanitizeStressText } = await import('./windows-pty-table-stress-observer.mjs')
report('error', { message: sanitizeStressText(error.stack) })
process.exitCode = 1
})