starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-29 16:02:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

rm artifact notes

Browse Source
This commit is contained in:
Jinjing
2026-09-02 11:02:06 -07:00
parent c3db4cbc33
commit e6749f0b86
1 changed files with 0 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files
artifacts/e2e-failure-report.md
-130
View File
@@ -1,130 +0,0 @@
# E2E failure triage report — 2026-09-02
## Executive summary
- **Status:** `ISSUES FILED`
- **Run:** [33564563164](https://github.com/stablyai/orca/actions/runs/33564563164)
- **Commit / branch:** `7873f73d807fa7bec80803cdf8ac36cd1f1c4316` / `main`
- **Completed:** `2026-09-01T22:35:03Z`
- **Counts:** `4` test updates · `2` flakes · `0` infrastructure · `8` product bugs = `14` original failed tests
### Decision
The scheduled run contained 14 failed tests, reconciled exactly against the failed job logs and nine Playwright trace artifacts. Six test-owned failures received focused local patches and observed validation. Eight tests correctly exposed product behavior; seven still reproduce on the newer local `main`, while the duplicate-PTY reveal passes once locally but has repeated scheduled-run evidence and remains open. No runner/provider failure or repository-owned infrastructure failure was identified. This report does **not** claim the E2E run is fully fixed: the eight product rows remain tracked in Linear, and the daemon-unwedge test remains blocked before its terminal-input assertion by the independently confirmed workspace-readiness defect.
## Run and environment
| Field | Value |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Workflow | `e2e.yml` |
| Run / event / conclusion | `33564563164` / `schedule` / `failure` |
| Created / completed | `2026-09-01T22:06:09Z` / `2026-09-01T22:35:03Z` |
| SHA / branch | `7873f73d807fa7bec80803cdf8ac36cd1f1c4316` / `main` |
| OS / browser project | Ubuntu `24.04.4` / `electron-headless` |
| Node / package manager | Node `24.19.0` / pnpm `12.0.0` |
| Playwright retries | `0` |
| Local revalidation revision | `bf88bcae1a` plus the local test patches listed below |
| Evidence | [failed run](https://github.com/stablyai/orca/actions/runs/33564563164); trace artifacts [shard 2](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823217399), [3](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823288074), [5](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823321019), [6](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823489578), [7](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823248778), [9](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823208552), [10](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823349336), [13](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823393073), [14](https://github.com/stablyai/orca/actions/runs/33564563164/artifacts/9823273638) |
## Failure matrix
Every original failed test appears exactly once. The matrix total is 14, matching the workflow failure inventory.
| # | Test (original path:line) | Category / disposition | Expected → observed | Repro / confidence | Action / owner |
| --- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1 | `paired-client-hosted-browser.spec.ts:311` — hosts a paired browser on the viewing desktop | `PRODUCT_BUG` · OPEN | managed client-hosted page → create rejected | CI + current-main failure; high | [STA-4150](https://linear.app/stably/issue/STA-4150/refactor-remote-browser-to-client-hosted-electron-webviews), Jinwoo |
| 2 | `paired-cmd-j-host-qualified-tabs.spec.ts:14` — routes same-id browser and simulator Cmd-J rows | `PRODUCT_BUG` · OPEN | one host-qualified browser row → zero | CI + current-main failure; high | [STA-5290](https://linear.app/stably/issue/STA-5290/bug-jump-palette-local-browser-page-activation-does-not-switch), Jinwoo |
| 3 | `paired-remote-browser-link-open-routing.spec.ts:280` — opens a remote pane link on the pane runtime | `PRODUCT_BUG` · OPEN | client placement → `null` | CI + current-main failure; high | [STA-4181](https://linear.app/stably/issue/STA-4181/release-blocking-paired-remote-terminal-browser-link-does-not-create), Jinjing |
| 4 | `tabs.spec.ts:125` — Cmd/Ctrl+T creates a new terminal tab | `FLAKY` · `FIXED` | tab delta `+1` → `+2` during startup hydration | CI fail; pre-patch current-main 3/3 pass; post-patch 3/3 and file suite pass | wait for startup worktree refresh; local patch |
| 5 | `terminal-duplicate-pty-renderer-reveal.spec.ts:186` — repairs duplicate persisted PTY renderers | `PRODUCT_BUG` · OPEN | alternate buffer → normal buffer | scheduled recurrence; current-main 1/1 pass; high target-run confidence | [STA-5208](https://linear.app/stably/issue/STA-5208/bug-revealed-duplicate-pty-renderer-fails-to-catch-up-to-streaming), Neil |
| 6 | `restart-restore-terminal-input.spec.ts:217` — restored pane recovers input after daemon un-wedges | `UNCONFIRMED PRODUCT_BUG` · `BLOCKED` before input assertion | workspace ready → `false` for 30s | CI + current-main exact failure; readiness defect confirmed, input symptom unconfirmed | [STA-5732](https://linear.app/stably/issue/STA-5732/p2continuous-override-listsessions-timeout-during-wedged-daemon), Neil; replay command below |
| 7 | `source-control-large-file-count.spec.ts:354` — over-cap change set degrades safely | `TEST_UPDATE` · `FIXED` | click this banner's Retry → detached duplicate Retry | CI fail; exact 1/1 + file suite 5/5 pass | scope locator to the owning banner; local patch |
| 8 | `paired-browser-creation-reconciliation-failure.spec.ts:356` — keeps cleanup on a headless host | `TEST_UPDATE` · `FIXED` | held create returns a page ID → obsolete preview path never reaches seam | local baseline fail; targeted and two clean-launch passes | drive the managed-browser action directly; local patch |
| 9 | `paired-client-hosted-browser-cookie-survival.spec.ts:294` — keeps cookies across runtime restart | `TEST_UPDATE` · `FIXED` | post-restart fixture renders with cookie → rendered fixture `null` | CI fail; pre-patch current-main 3/3 pass; post-patch 3/3 pass | wait for relaunched runtime to reclaim existing page IDs; local patch |
| 10 | `completed-worker-retirement-resume.spec.ts:39` — completed worker retires resume authority | `PRODUCT_BUG` · OPEN | live recovery state `working` → `done` | CI + current-main failure; high | [STA-4577](https://linear.app/stably/issue/STA-4577/returning-to-a-workspace-auto-resumes-a-completed-orchestration-worker), Jinwoo |
| 11 | `issue-12656-terminal-link-tooltip.spec.ts:115` — clears hover without shrinking terminal | `FLAKY` · `FIXED` | visible tooltip with URL → hidden/empty | CI fail; pre-patch current-main pass; post-patch 3/3 pass | re-locate moving URL cell before each hover; local patch |
| 12 | `worktree-active-delete-scroll-position.spec.ts:226` — deleting active scrolled worktree preserves position | `TEST_UPDATE` · `FIXED` | visible Delete action → exact old accessible name absent | CI + local baseline fail; exact and two-test file suite pass | real right-click and current accessible name; local patch |
| 13 | `worktree-jump-palette-filter.spec.ts:209` — Escape closes composer over Automations | `PRODUCT_BUG` · OPEN | second Escape hides Automations → heading remains visible | CI + current-main exact failure; high | [STA-5207](https://linear.app/stably/issue/STA-5207/bug-create-workspace-dialog-does-not-close-on-escape-when-opened-from), Neil |
| 14 | `terminal-tab-close-restart-persistence.spec.ts:26` — durable whole-tab close survives restart | `PRODUCT_BUG` · OPEN | explicit sidebar activation re-seeds one terminal → zero tabs | CI + current-main exact failure; high | [STA-5030](https://linear.app/stably/issue/STA-5030/bug-v14187-still-opens-persisted-empty-workspaces-with-a-blank-tab-bar), Neil |
## Evidence and diagnosis
### Test-maintenance fixes (#7, #8, #9, #12)
- **#7 source control:** the global `Retry` role locator could bind to a duplicate banner that detached during refresh. The patch locates the button from the visible too-many-changes banner's parent and preserves every recovery assertion.
- **#8 browser reconciliation harness:** the file-preview action no longer represented the managed-browser create contract under test. The patch drives `openNewBrowserTabInActiveWorkspace`, retains the returned promise while the fault seam is held, and compares stable host state without publication counters. Page creation, rollback, and capability assertions remain intact.
- **#9 cookie survival:** the test opened a new page as soon as the replacement runtime ID and worktree appeared, before existing client-hosted page ownership had been reclaimed. The patch synchronizes on the runtime's authoritative page-ID inventory, then performs the unchanged rendered-cookie and partition assertions.
- **#12 active delete:** a synthetic DOM `contextmenu` event and the obsolete exact name `Delete` bypassed Playwright's input behavior and missed the current descriptive accessible name. The patch uses a real right-click and `^Delete(?:\s|$)` while retaining viewport, removal, and scroll-position checks.
### Flake fixes (#4, #11)
- **#4 tabs:** the test measured its initial tab count before startup worktree refresh settled, allowing hydration to add a tab alongside Cmd/Ctrl+T. It now waits on the repository's startup refresh readiness signal and still asserts an exact `+1` delta. The original failure did not reproduce in three pre-patch current-main runs; three post-patch runs and the eight-test file suite passed.
- **#11 tooltip:** terminal output movement invalidated the cached cell coordinates used on every poll, so later mouse moves could target stale geometry. Each poll now re-locates the URL before moving the pointer. Tooltip visibility and full URL text assertions are unchanged; three consecutive post-patch runs passed.
### Product failures (#1, #2, #3, #5, #6, #10, #13, #14)
- **Paired browser creation/routing (#1–#3):** three independent user flows fail after healthy worktree and terminal setup: managed client-hosted creation is rejected, Cmd-J omits the expected host-qualified page, and a remote-pane link never acquires client placement. The expected local-SSH guard message also appears in passing paired tests and is not the root cause. Each exact scheduled test remains listed separately.
- **Duplicate PTY reveal (#5):** ownership repair converged to one pane/renderer, but the revealed renderer painted the normal buffer instead of the authoritative alternate buffer. The current-main replay passed once; repeated prior scheduled evidence on STA-5208 prevents treating the original as a locator or startup failure.
- **Daemon unwedge (#6):** both CI and exact current-main replay stop at `workspaceSessionReady=false` for 30 seconds. This confirms a startup-readiness defect, but the test never reaches its later typing assertion; the terminal-input behavior is therefore an `UNCONFIRMED PRODUCT_BUG` candidate, `BLOCKED` pending the readiness fix and replay.
- **Completed worker (#10):** the live pre-activation recovery record changes to `done` where the retirement contract requires `working`. The same assertion fails on current `main` and remains enabled.
- **Escape routing (#13):** after the first Escape closes the composer, the second Escape does not leave Automations. The exact test reaches this assertion and fails on current `main`.
- **Empty-workspace activation (#14):** the durable closed-tab tombstone survives restart correctly, but explicit sidebar activation leaves a blank zero-tab workspace. The regression assertion was reviewed and retained unchanged.
The repeated `Execution context was destroyed` pattern was treated as a session/startup failure signal, not as proof of a product assertion. None of the 14 rows was classified from that message alone. Additional 120-second worker-teardown leaks observed after failed paired journeys were diagnostic fallout and are not added to the original 14-test inventory.
## Changes and validation
- **PR:** not opened (run owner did not authorize a PR)
- **Files changed:**
- `tests/e2e/issue-12656-terminal-link-tooltip.spec.ts`
- `tests/e2e/paired-browser-creation-reconciliation-failure.spec.ts`
- `tests/e2e/paired-client-hosted-browser-cookie-survival.spec.ts`
- `tests/e2e/source-control-large-file-count.spec.ts`
- `tests/e2e/tabs.spec.ts`
- `tests/e2e/worktree-active-delete-scroll-position.spec.ts`
- **Assertions removed or tests skipped:** none
- **Independent review:** one completed read-only review round; no unresolved actionable findings. The reviewer rejected a proposed weakening/removal of the empty-workspace regression assertion, and that assertion remains unchanged.
Common E2E prefix used below: `SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$PWD/out/relay" pnpm run test:e2e`.
| Check | Result | Evidence |
| -------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Original inventory | 14/14 failures reconciled | failed run logs plus all nine failed-test trace artifacts |
| Source-control exact | `1/1` pass | `source-control-large-file-count.spec.ts --grep='a change set over the status cap'` |
| Source-control file suite | `5/5` pass | full `source-control-large-file-count.spec.ts` |
| Tabs + active-delete file suites | `10/10` pass | full `tabs.spec.ts` and `worktree-active-delete-scroll-position.spec.ts` |
| Tabs + tooltip repeated sample | `6/6` pass | both exact tests with `--repeat-each=3` |
| Cookie-survival repeated sample | `3/3` pass after reclaim-readiness patch | full one-test file with `--repeat-each=3` |
| Browser reconciliation targeted | `1/1` pass | full headless project selection for the file |
| Post-edit clean launch | `1/1` pass | fresh browser-reconciliation launch after all edits |
| Product exact replays | #6 and #13 failed exactly; #5 passed 1/1 | retained assertions and current-main evidence |
| Product diagnostic group | #1, #2, #3, and #10 failed; #14 failed in a separate exact group | current-main local runs; failed workers also exposed teardown timeouts |
| `pnpm typecheck:e2e` | `FAILED` on the repository's existing broad E2E type backlog | hundreds of unrelated diagnostics; no diagnostic names a newly changed line |
| Changed-file `oxlint` | `PASS` | six changed specs, zero findings |
| Changed-file `oxfmt --check` | `PASS` | six changed specs |
| `git diff --check` | `PASS` | no whitespace errors |
| Full E2E | not run | targeted paired failures incur 120-second worker teardown leaks; the full run would not change the eight open product dispositions |
## Product bugs / follow-up
Duplicate searches used each exact test name plus its symptom/component before any write. Existing issues were updated once with this run URL and current-main replay evidence; no duplicate issue was created. All are Urgent and assigned. The repository's actual label is `test-detected-bug` (singular).
| Issue | Title | Status / priority / label | Assignee | Fresh evidence |
| --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------ | -------- | ---------------------------------------------- |
| [STA-4150](https://linear.app/stably/issue/STA-4150/refactor-remote-browser-to-client-hosted-electron-webviews) | Client-hosted browser architecture / managed-create recurrence | Done / Urgent / `test-detected-bug` | Jinwoo | scheduled + current-main create failure |
| [STA-5290](https://linear.app/stably/issue/STA-5290/bug-jump-palette-local-browser-page-activation-does-not-switch) | Host-qualified Cmd-J browser activation | In Progress / Urgent / `test-detected-bug` | Jinwoo | scheduled + current-main missing row |
| [STA-4181](https://linear.app/stably/issue/STA-4181/release-blocking-paired-remote-terminal-browser-link-does-not-create) | Paired remote terminal browser-link routing | Done / Urgent / `test-detected-bug` | Jinjing | scheduled + current-main placement failure |
| [STA-5208](https://linear.app/stably/issue/STA-5208/bug-revealed-duplicate-pty-renderer-fails-to-catch-up-to-streaming) | Duplicate PTY reveal does not catch up | Todo / Urgent / `test-detected-bug` | Neil | scheduled recurrence; current-main 1/1 pass |
| [STA-5732](https://linear.app/stably/issue/STA-5732/p2continuous-override-listsessions-timeout-during-wedged-daemon) | Wedged-daemon startup recovery timeout | Done / Urgent / `test-detected-bug` | Neil | scheduled + current-main readiness failure |
| [STA-4577](https://linear.app/stably/issue/STA-4577/returning-to-a-workspace-auto-resumes-a-completed-orchestration-worker) | Completed worker retains resume authority | In Review / Urgent / `test-detected-bug` | Jinwoo | scheduled + current-main state mismatch |
| [STA-5207](https://linear.app/stably/issue/STA-5207/bug-create-workspace-dialog-does-not-close-on-escape-when-opened-from) | Escape handling from Jump Palette / Automations | Done / Urgent / `test-detected-bug` | Neil | scheduled + current-main second-Escape failure |
| [STA-5030](https://linear.app/stably/issue/STA-5030/bug-v14187-still-opens-persisted-empty-workspaces-with-a-blank-tab-bar) | Persisted-empty workspace opens blank | Done / Urgent / `test-detected-bug` | Neil | scheduled + current-main zero-tab activation |
Completed and started issues were not moved backward automatically, per Linear workflow etiquette; this is a documented deviation from the runbook's requested `Todo` status because changing active/completed workflow state would regress existing ownership. Owners of the five `Done` issues must decide whether the fresh recurrence warrants reopening; the recurrence evidence is already attached. The workspace schema exposes the singular `test-detected-bug` label, so that exact label was used in place of the runbook's plural wording.
## Blockers and next actions
- **Product owner — STA-5732:** fix workspace readiness after the daemon resumes, then replay: `SKIP_BUILD=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$PWD/out/relay" pnpm run test:e2e tests/e2e/restart-restore-terminal-input.spec.ts --workers=1 --grep='daemon un-wedges'`. Only after readiness succeeds can the terminal-input assertion be evaluated.
- **Owners of STA-4150, STA-4181, STA-5732, STA-5207, and STA-5030:** review the fresh recurrence evidence and reopen completed issues where appropriate. (STA-4150, STA-4181, STA-5732, STA-5207, and STA-5030 are five completed issues.)
- **Repository:** resolve the broad existing `typecheck:e2e` backlog. The six changed specs pass focused lint, formatting, diff, and runtime validation.
- **PR / Slack:** none created or sent; neither was authorized.