From e6749f0b86c802761922d2ea7ca320a115396fc8 Mon Sep 17 00:00:00 2001 From: Jinjing <6427696+AmethystLiang@users.noreply.github.com> Date: Wed, 2 Sep 2026 11:02:06 -0700 Subject: [PATCH] rm artifact notes --- artifacts/e2e-failure-report.md | 130 -------------------------------- 1 file changed, 130 deletions(-) delete mode 100644 artifacts/e2e-failure-report.md diff --git a/artifacts/e2e-failure-report.md b/artifacts/e2e-failure-report.md deleted file mode 100644 index 20aff54cea6..00000000000 --- a/artifacts/e2e-failure-report.md +++ /dev/null @@ -1,130 +0,0 @@ -# E2E failure triage report — 2026-09-02 - -## Executive summary - -- **Status:** `ISSUES FILED` -- **Run:** [33564563164](https://github.com/stablyai/orca/actions/runs/33564563164) -- **Commit / branch:** `7873f73d807fa7bec80803cdf8ac36cd1f1c4316` / `main` -- **Completed:** `2026-09-01T22:35:03Z` -- **Counts:** `4` test updates · `2` flakes · `0` infrastructure · `8` product bugs = `14` original failed tests - -### Decision - -The scheduled run contained 14 failed tests, reconciled exactly against the failed job logs and nine Playwright trace artifacts. Six test-owned failures received focused local patches and observed validation. Eight tests correctly exposed product behavior; seven still reproduce on the newer local `main`, while the duplicate-PTY reveal passes once locally but has repeated scheduled-run evidence and remains open. No runner/provider failure or repository-owned infrastructure failure was identified. This report does **not** claim the E2E run is fully fixed: the eight product rows remain tracked in Linear, and the daemon-unwedge test remains blocked before its terminal-input assertion by the independently confirmed workspace-readiness defect. - -## Run and environment - -| Field | Value | -| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Workflow | `e2e.yml` | -| Run / event / conclusion | `33564563164` / `schedule` / `failure` | -| Created / completed | `2026-09-01T22:06:09Z` / `2026-09-01T22:35:03Z` | -| SHA / branch | `7873f73d807fa7bec80803cdf8ac36cd1f1c4316` / `main` | -| OS / browser project | Ubuntu `24.04.4` / `electron-headless` | -| Node / package manager | Node `24.19.0` / pnpm `12.0.0` | -| Playwright retries | `0` | -| Local revalidation revision | `bf88bcae1a` plus the local test patches listed below | -| Evidence | [failed run](https://github.com/stablyai/orca/actions/runs/33564563164); trace artifacts [shard 2](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823217399), [3](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823288074), [5](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823321019), [6](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823489578), [7](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823248778), [9](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823208552), [10](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823349336), [13](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823393073), [14](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823273638) | - -## Failure matrix - -Every original failed test appears exactly once. The matrix total is 14, matching the workflow failure inventory. - -| # | Test (original path:line) | Category / disposition | Expected → observed | Repro / confidence | Action / owner | -| --- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | `paired-client-hosted-browser.spec.ts:311` — hosts a paired browser on the viewing desktop | `PRODUCT_BUG` · OPEN | managed client-hosted page → create rejected | CI + current-main failure; high | [STA-4150](https://linear.app/stably/issue/STA-4150/refactor-remote-browser-to-client-hosted-electron-webviews), Jinwoo | -| 2 | `paired-cmd-j-host-qualified-tabs.spec.ts:14` — routes same-id browser and simulator Cmd-J rows | `PRODUCT_BUG` · OPEN | one host-qualified browser row → zero | CI + current-main failure; high | [STA-5290](https://linear.app/stably/issue/STA-5290/bug-jump-palette-local-browser-page-activation-does-not-switch), Jinwoo | -| 3 | `paired-remote-browser-link-open-routing.spec.ts:280` — opens a remote pane link on the pane runtime | `PRODUCT_BUG` · OPEN | client placement → `null` | CI + current-main failure; high | [STA-4181](https://linear.app/stably/issue/STA-4181/release-blocking-paired-remote-terminal-browser-link-does-not-create), Jinjing | -| 4 | `tabs.spec.ts:125` — Cmd/Ctrl+T creates a new terminal tab | `FLAKY` · `FIXED` | tab delta `+1` → `+2` during startup hydration | CI fail; pre-patch current-main 3/3 pass; post-patch 3/3 and file suite pass | wait for startup worktree refresh; local patch | -| 5 | `terminal-duplicate-pty-renderer-reveal.spec.ts:186` — repairs duplicate persisted PTY renderers | `PRODUCT_BUG` · OPEN | alternate buffer → normal buffer | scheduled recurrence; current-main 1/1 pass; high target-run confidence | [STA-5208](https://linear.app/stably/issue/STA-5208/bug-revealed-duplicate-pty-renderer-fails-to-catch-up-to-streaming), Neil | -| 6 | `restart-restore-terminal-input.spec.ts:217` — restored pane recovers input after daemon un-wedges | `UNCONFIRMED PRODUCT_BUG` · `BLOCKED` before input assertion | workspace ready → `false` for 30s | CI + current-main exact failure; readiness defect confirmed, input symptom unconfirmed | [STA-5732](https://linear.app/stably/issue/STA-5732/p2continuous-override-listsessions-timeout-during-wedged-daemon), Neil; replay command below | -| 7 | `source-control-large-file-count.spec.ts:354` — over-cap change set degrades safely | `TEST_UPDATE` · `FIXED` | click this banner's Retry → detached duplicate Retry | CI fail; exact 1/1 + file suite 5/5 pass | scope locator to the owning banner; local patch | -| 8 | `paired-browser-creation-reconciliation-failure.spec.ts:356` — keeps cleanup on a headless host | `TEST_UPDATE` · `FIXED` | held create returns a page ID → obsolete preview path never reaches seam | local baseline fail; targeted and two clean-launch passes | drive the managed-browser action directly; local patch | -| 9 | `paired-client-hosted-browser-cookie-survival.spec.ts:294` — keeps cookies across runtime restart | `TEST_UPDATE` · `FIXED` | post-restart fixture renders with cookie → rendered fixture `null` | CI fail; pre-patch current-main 3/3 pass; post-patch 3/3 pass | wait for relaunched runtime to reclaim existing page IDs; local patch | -| 10 | `completed-worker-retirement-resume.spec.ts:39` — completed worker retires resume authority | `PRODUCT_BUG` · OPEN | live recovery state `working` → `done` | CI + current-main failure; high | [STA-4577](https://linear.app/stably/issue/STA-4577/returning-to-a-workspace-auto-resumes-a-completed-orchestration-worker), Jinwoo | -| 11 | `issue-12656-terminal-link-tooltip.spec.ts:115` — clears hover without shrinking terminal | `FLAKY` · `FIXED` | visible tooltip with URL → hidden/empty | CI fail; pre-patch current-main pass; post-patch 3/3 pass | re-locate moving URL cell before each hover; local patch | -| 12 | `worktree-active-delete-scroll-position.spec.ts:226` — deleting active scrolled worktree preserves position | `TEST_UPDATE` · `FIXED` | visible Delete action → exact old accessible name absent | CI + local baseline fail; exact and two-test file suite pass | real right-click and current accessible name; local patch | -| 13 | `worktree-jump-palette-filter.spec.ts:209` — Escape closes composer over Automations | `PRODUCT_BUG` · OPEN | second Escape hides Automations → heading remains visible | CI + current-main exact failure; high | [STA-5207](https://linear.app/stably/issue/STA-5207/bug-create-workspace-dialog-does-not-close-on-escape-when-opened-from), Neil | -| 14 | `terminal-tab-close-restart-persistence.spec.ts:26` — durable whole-tab close survives restart | `PRODUCT_BUG` · OPEN | explicit sidebar activation re-seeds one terminal → zero tabs | CI + current-main exact failure; high | [STA-5030](https://linear.app/stably/issue/STA-5030/bug-v14187-still-opens-persisted-empty-workspaces-with-a-blank-tab-bar), Neil | - -## Evidence and diagnosis - -### Test-maintenance fixes (#7, #8, #9, #12) - -- **#7 source control:** the global `Retry` role locator could bind to a duplicate banner that detached during refresh. The patch locates the button from the visible too-many-changes banner's parent and preserves every recovery assertion. -- **#8 browser reconciliation harness:** the file-preview action no longer represented the managed-browser create contract under test. The patch drives `openNewBrowserTabInActiveWorkspace`, retains the returned promise while the fault seam is held, and compares stable host state without publication counters. Page creation, rollback, and capability assertions remain intact. -- **#9 cookie survival:** the test opened a new page as soon as the replacement runtime ID and worktree appeared, before existing client-hosted page ownership had been reclaimed. The patch synchronizes on the runtime's authoritative page-ID inventory, then performs the unchanged rendered-cookie and partition assertions. -- **#12 active delete:** a synthetic DOM `contextmenu` event and the obsolete exact name `Delete` bypassed Playwright's input behavior and missed the current descriptive accessible name. The patch uses a real right-click and `^Delete(?:\s|$)` while retaining viewport, removal, and scroll-position checks. - -### Flake fixes (#4, #11) - -- **#4 tabs:** the test measured its initial tab count before startup worktree refresh settled, allowing hydration to add a tab alongside Cmd/Ctrl+T. It now waits on the repository's startup refresh readiness signal and still asserts an exact `+1` delta. The original failure did not reproduce in three pre-patch current-main runs; three post-patch runs and the eight-test file suite passed. -- **#11 tooltip:** terminal output movement invalidated the cached cell coordinates used on every poll, so later mouse moves could target stale geometry. Each poll now re-locates the URL before moving the pointer. Tooltip visibility and full URL text assertions are unchanged; three consecutive post-patch runs passed. - -### Product failures (#1, #2, #3, #5, #6, #10, #13, #14) - -- **Paired browser creation/routing (#1–#3):** three independent user flows fail after healthy worktree and terminal setup: managed client-hosted creation is rejected, Cmd-J omits the expected host-qualified page, and a remote-pane link never acquires client placement. The expected local-SSH guard message also appears in passing paired tests and is not the root cause. Each exact scheduled test remains listed separately. -- **Duplicate PTY reveal (#5):** ownership repair converged to one pane/renderer, but the revealed renderer painted the normal buffer instead of the authoritative alternate buffer. The current-main replay passed once; repeated prior scheduled evidence on STA-5208 prevents treating the original as a locator or startup failure. -- **Daemon unwedge (#6):** both CI and exact current-main replay stop at `workspaceSessionReady=false` for 30 seconds. This confirms a startup-readiness defect, but the test never reaches its later typing assertion; the terminal-input behavior is therefore an `UNCONFIRMED PRODUCT_BUG` candidate, `BLOCKED` pending the readiness fix and replay. -- **Completed worker (#10):** the live pre-activation recovery record changes to `done` where the retirement contract requires `working`. The same assertion fails on current `main` and remains enabled. -- **Escape routing (#13):** after the first Escape closes the composer, the second Escape does not leave Automations. The exact test reaches this assertion and fails on current `main`. -- **Empty-workspace activation (#14):** the durable closed-tab tombstone survives restart correctly, but explicit sidebar activation leaves a blank zero-tab workspace. The regression assertion was reviewed and retained unchanged. - -The repeated `Execution context was destroyed` pattern was treated as a session/startup failure signal, not as proof of a product assertion. None of the 14 rows was classified from that message alone. Additional 120-second worker-teardown leaks observed after failed paired journeys were diagnostic fallout and are not added to the original 14-test inventory. - -## Changes and validation - -- **PR:** not opened (run owner did not authorize a PR) -- **Files changed:** - - `tests/e2e/issue-12656-terminal-link-tooltip.spec.ts` - - `tests/e2e/paired-browser-creation-reconciliation-failure.spec.ts` - - `tests/e2e/paired-client-hosted-browser-cookie-survival.spec.ts` - - `tests/e2e/source-control-large-file-count.spec.ts` - - `tests/e2e/tabs.spec.ts` - - `tests/e2e/worktree-active-delete-scroll-position.spec.ts` -- **Assertions removed or tests skipped:** none -- **Independent review:** one completed read-only review round; no unresolved actionable findings. The reviewer rejected a proposed weakening/removal of the empty-workspace regression assertion, and that assertion remains unchanged. - -Common E2E prefix used below: `SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$PWD/out/relay" pnpm run test:e2e`. - -| Check | Result | Evidence | -| -------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -| Original inventory | 14/14 failures reconciled | failed run logs plus all nine failed-test trace artifacts | -| Source-control exact | `1/1` pass | `source-control-large-file-count.spec.ts --grep='a change set over the status cap'` | -| Source-control file suite | `5/5` pass | full `source-control-large-file-count.spec.ts` | -| Tabs + active-delete file suites | `10/10` pass | full `tabs.spec.ts` and `worktree-active-delete-scroll-position.spec.ts` | -| Tabs + tooltip repeated sample | `6/6` pass | both exact tests with `--repeat-each=3` | -| Cookie-survival repeated sample | `3/3` pass after reclaim-readiness patch | full one-test file with `--repeat-each=3` | -| Browser reconciliation targeted | `1/1` pass | full headless project selection for the file | -| Post-edit clean launch | `1/1` pass | fresh browser-reconciliation launch after all edits | -| Product exact replays | #6 and #13 failed exactly; #5 passed 1/1 | retained assertions and current-main evidence | -| Product diagnostic group | #1, #2, #3, and #10 failed; #14 failed in a separate exact group | current-main local runs; failed workers also exposed teardown timeouts | -| `pnpm typecheck:e2e` | `FAILED` on the repository's existing broad E2E type backlog | hundreds of unrelated diagnostics; no diagnostic names a newly changed line | -| Changed-file `oxlint` | `PASS` | six changed specs, zero findings | -| Changed-file `oxfmt --check` | `PASS` | six changed specs | -| `git diff --check` | `PASS` | no whitespace errors | -| Full E2E | not run | targeted paired failures incur 120-second worker teardown leaks; the full run would not change the eight open product dispositions | - -## Product bugs / follow-up - -Duplicate searches used each exact test name plus its symptom/component before any write. Existing issues were updated once with this run URL and current-main replay evidence; no duplicate issue was created. All are Urgent and assigned. The repository's actual label is `test-detected-bug` (singular). - -| Issue | Title | Status / priority / label | Assignee | Fresh evidence | -| --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------ | -------- | ---------------------------------------------- | -| [STA-4150](https://linear.app/stably/issue/STA-4150/refactor-remote-browser-to-client-hosted-electron-webviews) | Client-hosted browser architecture / managed-create recurrence | Done / Urgent / `test-detected-bug` | Jinwoo | scheduled + current-main create failure | -| [STA-5290](https://linear.app/stably/issue/STA-5290/bug-jump-palette-local-browser-page-activation-does-not-switch) | Host-qualified Cmd-J browser activation | In Progress / Urgent / `test-detected-bug` | Jinwoo | scheduled + current-main missing row | -| [STA-4181](https://linear.app/stably/issue/STA-4181/release-blocking-paired-remote-terminal-browser-link-does-not-create) | Paired remote terminal browser-link routing | Done / Urgent / `test-detected-bug` | Jinjing | scheduled + current-main placement failure | -| [STA-5208](https://linear.app/stably/issue/STA-5208/bug-revealed-duplicate-pty-renderer-fails-to-catch-up-to-streaming) | Duplicate PTY reveal does not catch up | Todo / Urgent / `test-detected-bug` | Neil | scheduled recurrence; current-main 1/1 pass | -| [STA-5732](https://linear.app/stably/issue/STA-5732/p2continuous-override-listsessions-timeout-during-wedged-daemon) | Wedged-daemon startup recovery timeout | Done / Urgent / `test-detected-bug` | Neil | scheduled + current-main readiness failure | -| [STA-4577](https://linear.app/stably/issue/STA-4577/returning-to-a-workspace-auto-resumes-a-completed-orchestration-worker) | Completed worker retains resume authority | In Review / Urgent / `test-detected-bug` | Jinwoo | scheduled + current-main state mismatch | -| [STA-5207](https://linear.app/stably/issue/STA-5207/bug-create-workspace-dialog-does-not-close-on-escape-when-opened-from) | Escape handling from Jump Palette / Automations | Done / Urgent / `test-detected-bug` | Neil | scheduled + current-main second-Escape failure | -| [STA-5030](https://linear.app/stably/issue/STA-5030/bug-v14187-still-opens-persisted-empty-workspaces-with-a-blank-tab-bar) | Persisted-empty workspace opens blank | Done / Urgent / `test-detected-bug` | Neil | scheduled + current-main zero-tab activation | - -Completed and started issues were not moved backward automatically, per Linear workflow etiquette; this is a documented deviation from the runbook's requested `Todo` status because changing active/completed workflow state would regress existing ownership. Owners of the five `Done` issues must decide whether the fresh recurrence warrants reopening; the recurrence evidence is already attached. The workspace schema exposes the singular `test-detected-bug` label, so that exact label was used in place of the runbook's plural wording. - -## Blockers and next actions - -- **Product owner — STA-5732:** fix workspace readiness after the daemon resumes, then replay: `SKIP_BUILD=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$PWD/out/relay" pnpm run test:e2e tests/e2e/restart-restore-terminal-input.spec.ts --workers=1 --grep='daemon un-wedges'`. Only after readiness succeeds can the terminal-input assertion be evaluated. -- **Owners of STA-4150, STA-4181, STA-5732, STA-5207, and STA-5030:** review the fresh recurrence evidence and reopen completed issues where appropriate. (STA-4150, STA-4181, STA-5732, STA-5207, and STA-5030 are five completed issues.) -- **Repository:** resolve the broad existing `typecheck:e2e` backlog. The six changed specs pass focused lint, formatting, diff, and runtime validation. -- **PR / Slack:** none created or sent; neither was authorized.