# Timing-based CI shards Unit shards and general E2E shards use longest-processing-time assignment of whole files to the currently lightest shard. Ties use file path and then shard index, independent of filesystem enumeration and locale. Unknown, zero, or invalid durations use the baseline's positive median (1 second when no positive evidence exists). Deleted files never enter discovery. Unit weights include measured environment startup, preparation, setup, imports and test execution; no uniform overhead is added. Unit assignment runs inside Vitest's sequencer after discovery and CLI exclusions; Vitest's default sort, workers and isolation remain intact. It is enabled only by `ORCA_BALANCE_UNIT_SHARDS=1`; ordinary local runs and explicit file filters retain their existing behavior. E2E uses Playwright's native `--list` and `--test-list`, retaining project filters, skipped tests and complete serial groups within files. The workflow verifies selected test IDs against full discovery before executing. Dedicated SSH, native IME, WSL and first-paint lanes are unchanged. ## Evidence and limits `ci-shard-timings.json` records source runs. Unit report provenance and contributing job IDs for the current refresh appear below; E2E job IDs remain in the JSON. - Original unit run **34675583768**, Node 24, all eight successful shards: 8,484 completed file durations. The summed transform/setup/import/environment durations divided by measured file count give a rounded-up **526ms** per-file overhead allowance. The original shard weighted loads were **764–849 worker-seconds**, versus **792–792** after balancing the identical measured files. File counts change from **1,056–1,065** to **1,060–1,061**. - General E2E run **34652504501**, all fourteen shard logs: 291 files with completed headless test durations, including failures. Headful benchmark reruns are not counted. Original completed test loads were **540–1,727 seconds**, versus **1,083–1,093** after whole-file balancing on the same measured files. The longest measured file is **528 seconds**, below the balanced shard load. - Original checkout discovery at validation contained **8,553 unit files** after the workflow's exact exclusions and **733 headless E2E tests in 340 files**. New and unmeasured files remain selected. Projected current loads were about **797 worker-seconds** per unit shard (1,068–1,070 files) and **1,190–1,200 seconds** per E2E shard (22–25 files). These are scheduling projections, not measured post-change wall-clock gains. Unit durations overlap across workers. The original average overhead has now been replaced by per-file module measurements. E2E evidence includes failed shards and can omit unfinished tests; unknowns receive a deterministic estimate. Historical timings age as specs change. Full CI runs on the existing runner classes are required to measure elapsed-time and occupancy improvements, including discovery overhead. No retries, assertions, coverage exclusions, runner classes or shard counts changed. ## Current unit refresh: November 2, 2026 The baseline averages each file over five complete, successful twelve-shard full runs on `blacksmith-4vcpu-ubuntu-2404` with four workers, Node 24.22.2 and Bun 1.4.4: [41403569437](https://github.com/stablyai/orca/actions/runs/41403569437), [41406098648](https://github.com/stablyai/orca/actions/runs/41406098648), [41406030210](https://github.com/stablyai/orca/actions/runs/41406030210), [41405874918](https://github.com/stablyai/orca/actions/runs/41405874918) and [41405380104](https://github.com/stablyai/orca/actions/runs/41405380104), all attempt 1 and all after the November 1 test-time fixes. The October 25 baseline came from five eight-worker shards, lacked 1,595 measured files and kept 374 deleted ones. One run is not enough: per-file durations vary between runs, and a baseline taken from 41403569437 alone left PR run 41406120225 at 1,244–1,515 worker-seconds (21.8%). Assigning the held-out run 41406120225 with the averaged baseline and evaluating it with that run's own measurements projects 1,336–1,387 worker-seconds per shard (3.8%), against 1,297–1,430 (10.2%) for the October 25 baseline. Leaving out each of six post-fix runs in turn, the average of the other five projects 1.9–8.6% spread, against 8.8–21.2% for the October 25 baseline. Runs from before the fixes are not comparable, because their `pty-subprocess-*` files still carried 5-second waits. ```sh node config/scripts/ci/sharding/ci-unit-timing-import.mjs RUN_DIR... config/scripts/ci/sharding/ci-shard-timings.json ``` Each directory holds one run's twelve `unit-shard-node-*` artifacts. Every run is validated as before, the runs must be distinct and share one runtime version, and a single directory keeps the previous single-run output. ## Historical unit refresh: October 25, 2026 The baseline imports all five successful unit shards from [run 40276310369, attempt 1](https://github.com/stablyai/orca/actions/runs/40276310369/attempts/1), the runner class PR shards use: `blacksmith-8vcpu-ubuntu-2404`, eight workers, Node 24.22.1, Bun 1.4.4, source `085cfad2cb3b883c975d5630d94fb2900d0bed05`. The previous ARM four-worker baseline lacked 1,976 current files and left PR shards 13–44% apart. Balancing two later full runs (40276227094, 40275768715) with each baseline, the fresh one cut the heaviest shard from 2,129 to 2,008 and from 2,564 to 2,134 worker-seconds, with a spread of 3–4% instead of 13–44%. | Shard | Job ID | Measured worker-seconds | Job seconds | | ----- | ------------ | ----------------------: | ----------: | | 1 | 121420433743 | 2,476.887 | 381 | | 2 | 121420433755 | 2,069.007 | 329 | | 3 | 121420433825 | 2,144.439 | 339 | | 4 | 121420433749 | 2,050.086 | 322 | | 5 | 121420433831 | 2,151.221 | 338 | ## Historical unit refresh: October 14, 2026 The baseline imports all five successful ARM unit shards from [run 38646706136, attempt 1](https://github.com/stablyai/orca/actions/runs/38646706136/attempts/1). The tested merge was `8d7f46198b99432efd104ac7803174173b367c66`, using Node 24.21.0, Bun 1.4.3, `ubuntu-24.04-arm`, and four workers per shard. All 12,314 files appear exactly once: 12,163 passed and 151 skipped, with zero unhandled errors. Artifact digests, all 20 report hashes, source/runtime provenance, full discovery, and each saved assignment were checked before importing. The E2E weights remain unchanged. | Shard | Job ID | Measured worker-seconds | Job seconds | | ----- | ------------ | ----------------------: | ----------: | | 1 | 116137259336 | 1,966.470 | 568 | | 2 | 116137259460 | 2,080.624 | 604 | | 3 | 116137259318 | 2,121.495 | 611 | | 4 | 116137259783 | 2,094.487 | 614 | | 5 | 116137259540 | 2,190.636 | 641 | Retrospectively assigning those measurements projects 2,090.726–2,090.747 worker-seconds per shard, instead of 1,966.470–2,190.636. That is a scheduling projection, not a hosted speedup: module times overlap across workers, and total runner occupancy was 50.63 minutes. Runner capacity, selection, workers, and the assignment algorithm are unchanged. Later source changes need fresh validation. ## Historical unit refresh: September 27, 2026 The baseline imports every successful Node 24 shard from [run 36294142683, attempt 1](https://github.com/stablyai/orca/actions/runs/36294142683/attempts/1). The recorded checkout was `cc35c1b422e9c20197bc59aa27a71019ffc18b5b`, with Node 24.21.0 and unchanged Linux workers. All 9,847 report entries match their saved assignments and current discovery exactly once, with no unhandled errors. The old baseline lacked 165 files and retained one deleted path. The E2E weights are unchanged. Using these measurements for both assignments, the maximum projected load falls from 1,043.811 to 919.695 worker-seconds (11.9%); each shard selects 1,230–1,231 files. Observed source-run test steps ranged from 282 to 429 seconds. The fastest runner's common-file durations were broadly lower than its prior measurements, so runner variation may contribute to that spread. The projection does not establish a hosted wall-clock improvement. No worker, shard-count, or algorithm change is made. ## Previous unit refresh: September 26, 2026 The previous baseline imported all eight successful unit shards from [run 36221874572, attempt 1](https://github.com/stablyai/orca/actions/runs/36221874572/attempts/1). They ran Node **24.21.0** on `ubuntu-latest` with Vitest's default Linux worker count and isolation. The checked-out merge was `f68ecf8c4aeb396795a8f0f9dbd8ef58fd634d43`, whose second parent is the run's PR head `5a26cfa83d7542f9f4545b7a7c4c4174ff213b18`. The workflow failed in another lane; every contributing unit job and test step succeeded. No failed reports or four-worker trial measurements contributed. | Shard | Job ID | Completed files | Measured worker-seconds | Test step seconds | Job seconds | | ----- | ------------ | --------------- | ----------------------- | ----------------- | ----------- | | 1 | 108348885240 | 1,209 | 1,019.941 | 420 | 446 | | 2 | 108348885231 | 1,210 | 1,000.547 | 413 | 447 | | 3 | 108348885289 | 1,211 | 933.332 | 386 | 419 | | 4 | 108348885241 | 1,211 | 1,022.838 | 418 | 453 | | 5 | 108348885283 | 1,211 | 1,016.000 | 417 | 448 | | 6 | 108348885237 | 1,211 | 980.988 | 407 | 440 | | 7 | 108348885275 | 1,210 | 810.367 | 326 | 355 | | 8 | 108348885261 | 1,210 | 973.722 | 407 | 445 | All reports have `status: passed`, zero unhandled errors and identical source, run, attempt, Node version and shard count. Each report's file keys exactly match its selected assignment; all eight assignments describe the same plan, with no missing or duplicate files across **9,683** measurements. Replaying the old baseline reproduces that complete plan. Its SHA-256 is `9042d8f1d2420b301ee5d9e56d768ad7d38530897dce0d641f362333aa194ad8`. The importer sets overhead to zero; the E2E baseline is unchanged. Discovery at checkout `80ff6c4e3e66de46e41c75856fd1fb6d77d515de`, using the unit workflow's exact exclusions, contains **9,732 files**. All 9,683 measurements still apply; the **49** unmeasured files receive the new **230ms** median. Both plans select every current file exactly once. To compare assignments fairly, the following loads evaluate both plans using the same new measurements and fallback, rather than comparing two different timing metrics: | Current discovery projection | Previous assignment | Refreshed assignment | | ---------------------------- | ------------------- | --------------------- | | Files per shard | 1,216–1,217 | 1,215–1,217 | | Worker-seconds per shard | 816.045–1,049.604 | 971.124–971.126 | | Largest load | 1,049.604 | 971.126 (7.48% lower) | The combined projected work is unchanged at **7,769.005 worker-seconds**. Reassignment redistributes measured work; it does not itself reduce test work. These are projections from one successful unit run, not a measured wall-clock improvement. The source test steps took **326–420 seconds**, totaling **3,194 seconds (53.233 runner-minutes)**. Full unit jobs took **355–453 seconds**, totaling **3,453 seconds (57.550 runner-minutes)**. These durations exclude queue time; hosted before/after runs are needed to measure the refreshed assignment's elapsed time and occupancy. ## Reproduction and refresh Every shard uploads an artifact named with its shard, Node version where relevant, and run attempt. `assignment.json` contains the checked-out source SHA, run ID, attempt, baseline SHA-256, algorithm, fallback, all shard files and chosen shard. E2E also retains both discovery reports and `selected.txt`. Artifacts live for 14 days. A rerun of the same source uses the same checked-in baseline rather than mutable timing caches; a GitHub job rerun therefore keeps its assignment. Unit artifacts also contain `unit-timings.json`, measured through Vitest's reporter API. Each file's weight includes environment startup, test harness preparation, setup files, imports and test execution. Imports are often more expensive than the assertions: run **36212793136**, shard 4, spent **535 worker-seconds importing** and **357 running tests**. The refreshed baseline replaces the old uniform import allowance with the complete successful measurements described above. For E2E reproduction, check out the recorded source and pass the saved list to the existing command: `pnpm run test:e2e --test-list=/path/to/selected.txt` with the same CI environment/build inputs. For unit reproduction, use the unchanged workflow command and exclusions with `ORCA_BALANCE_UNIT_SHARDS=1` and the recorded `--shard=INDEX/8`. Direct test-file reruns remain supported. To refresh unit weights, download all `unit-shard-node-*` artifacts from one complete successful unit shard set, run attempt and Node version into a directory, preserving their shard subdirectories. An unrelated workflow lane may fail; every contributing unit job and report must pass. Check the attempt-specific jobs API, source workflow/config, report provenance and exact file coverage against the saved assignments, then run: ```sh node config/scripts/ci/sharding/ci-unit-timing-import.mjs ARTIFACT_DIRECTORY config/scripts/ci/sharding/ci-shard-timings.json ``` The importer rejects missing or duplicate shards, duplicate files, failed unit reports, unhandled errors and mixed source revisions, run attempts or Node versions. It preserves E2E weights and sets unit overhead to zero because the per-file measurements already include it. Timing artifacts remain diagnostic; failed or interrupted unit runs are retained for investigation but cannot replace the scheduling baseline. For older unit runs without reporter artifacts, and E2E refreshes, download `log-JOB_ID.txt` files into one directory from exactly one eight-shard unit run and one fourteen-shard general E2E run. Use the job IDs from the Actions jobs API and fetch each with `gh api repos/stablyai/orca/actions/jobs/JOB_ID/logs`. Do not include dedicated lanes or multiple attempts. Then run: ```sh node config/scripts/ci/sharding/ci-shard-timing-import.mjs LOG_DIRECTORY UNIT_RUN_ID E2E_RUN_ID config/scripts/ci/sharding/ci-shard-timings.json ``` The initial source logs are in `/tmp/orca-ci-shard-logs`; two were reused from `/tmp/orca-ci-audit`, and the remaining twenty were fetched read-only. Reimporting those logs reproduced the original baseline byte-for-byte. Review file-count and load projections before adopting a new baseline; no network access is needed to plan or run shards. ## Validation - 74 focused tests passed across the two new test files and existing PR parallelism, E2E gate and release E2E dispatch contracts. - The pinned Playwright CLI selected the real 733-test suite across all fourteen saved test lists with exact-once identity coverage and no missing tests. - A temporary native Playwright fixture checks fourteen shards, serial groups, skipped cases, headful filtering and mismatch rejection without launching UI. - Real Vitest discovery with all workflow exclusions yielded 8,553 files; the sequencer's eight assignments covered each exactly once. An actual opt-in Vitest shard executed successfully and persisted its manifest. - Focused TypeScript checking of `config/vitest.config.ts` and imported modules, oxlint, formatting and baseline reimport checks passed. - The September 26 refresh matched all eight reports to their saved assignments, checked successful attempt-specific job results and source worker settings, reproduced the old assignment, and preserved the E2E baseline exactly. All 12 importer/reporter and shard-assignment tests passed; importing the artifacts reproduced the refreshed JSON byte-for-byte. All local tests used `ORCA_BACKGROUND_LAUNCH=1` in background tool sessions. No app windows or full E2E test bodies were launched.