mirror of
https://github.com/stablyai/orca.git
synced 2026-11-17 19:34:29 +00:00
267 lines
16 KiB
Markdown
267 lines
16 KiB
Markdown
# Timing-based CI shards
|
||
|
||
Unit shards and general E2E shards use longest-processing-time
|
||
assignment of whole files to the currently lightest shard. Ties use file path and
|
||
then shard index, independent of filesystem enumeration and locale. Unknown,
|
||
zero, or invalid durations use the baseline's positive median (1 second when no
|
||
positive evidence exists). Deleted files never enter discovery. Unit weights
|
||
include measured environment startup, preparation, setup, imports and test execution;
|
||
no uniform overhead is added.
|
||
|
||
Unit assignment runs inside Vitest's sequencer after discovery and CLI exclusions;
|
||
Vitest's default sort, workers and isolation remain intact. It is enabled only by
|
||
`ORCA_BALANCE_UNIT_SHARDS=1`; ordinary local runs and explicit file filters retain
|
||
their existing behavior. E2E uses Playwright's native `--list` and `--test-list`,
|
||
retaining project filters, skipped tests and complete serial groups within files.
|
||
The workflow verifies selected test IDs against full discovery before executing.
|
||
Dedicated SSH, native IME, WSL and first-paint lanes are unchanged.
|
||
|
||
## Evidence and limits
|
||
|
||
`ci-shard-timings.json` records source runs. Unit report provenance and contributing
|
||
job IDs for the current refresh appear below; E2E job IDs remain in the JSON.
|
||
|
||
- Original unit run **34675583768**, Node 24, all eight successful shards: 8,484 completed
|
||
file durations. The summed transform/setup/import/environment durations divided
|
||
by measured file count give a rounded-up **526ms** per-file overhead allowance.
|
||
The original shard weighted loads were **764–849 worker-seconds**, versus
|
||
**792–792** after balancing the identical measured files. File counts change
|
||
from **1,056–1,065** to **1,060–1,061**.
|
||
- General E2E run **34652504501**, all fourteen shard logs: 291 files with completed
|
||
headless test durations, including failures. Headful benchmark reruns are not
|
||
counted. Original completed test loads were **540–1,727 seconds**, versus
|
||
**1,083–1,093** after whole-file balancing on the same measured files. The longest
|
||
measured file is **528 seconds**, below the balanced shard load.
|
||
- Original checkout discovery at validation contained **8,553 unit files** after the
|
||
workflow's exact exclusions and **733 headless E2E tests in 340 files**. New and
|
||
unmeasured files remain selected. Projected current loads were about **797
|
||
worker-seconds** per unit shard (1,068–1,070 files) and **1,190–1,200 seconds** per
|
||
E2E shard (22–25 files).
|
||
|
||
These are scheduling projections, not measured post-change wall-clock gains.
|
||
Unit durations overlap across workers. The original average overhead has now been
|
||
replaced by per-file module measurements. E2E evidence includes failed shards and can omit
|
||
unfinished tests; unknowns receive a deterministic estimate. Historical timings
|
||
age as specs change. Full CI runs on the existing runner classes are required to
|
||
measure elapsed-time and occupancy improvements, including discovery overhead.
|
||
No retries, assertions, coverage exclusions, runner classes or shard counts changed.
|
||
|
||
## Current unit refresh: November 2, 2026
|
||
|
||
The baseline averages each file over five complete, successful twelve-shard full runs on
|
||
`blacksmith-4vcpu-ubuntu-2404` with four workers, Node 24.22.2 and Bun 1.4.4:
|
||
[41403569437](https://github.com/stablyai/orca/actions/runs/41403569437),
|
||
[41406098648](https://github.com/stablyai/orca/actions/runs/41406098648),
|
||
[41406030210](https://github.com/stablyai/orca/actions/runs/41406030210),
|
||
[41405874918](https://github.com/stablyai/orca/actions/runs/41405874918) and
|
||
[41405380104](https://github.com/stablyai/orca/actions/runs/41405380104), all attempt 1 and
|
||
all after the November 1 test-time fixes. The October 25 baseline came from five eight-worker
|
||
shards, lacked 1,595 measured files and kept 374 deleted ones.
|
||
|
||
One run is not enough: per-file durations vary between runs, and a baseline taken from
|
||
41403569437 alone left PR run 41406120225 at 1,244–1,515 worker-seconds (21.8%). Assigning the
|
||
held-out run 41406120225 with the averaged baseline and evaluating it with that run's own
|
||
measurements projects 1,336–1,387 worker-seconds per shard (3.8%), against 1,297–1,430
|
||
(10.2%) for the October 25 baseline. Leaving out each of six post-fix runs in turn, the
|
||
average of the other five projects 1.9–8.6% spread, against 8.8–21.2% for the October 25
|
||
baseline. Runs from before the fixes are not comparable, because their `pty-subprocess-*`
|
||
files still carried 5-second waits.
|
||
|
||
```sh
|
||
node config/scripts/ci/sharding/ci-unit-timing-import.mjs RUN_DIR... config/scripts/ci/sharding/ci-shard-timings.json
|
||
```
|
||
|
||
Each directory holds one run's twelve `unit-shard-node-*` artifacts. Every run is validated
|
||
as before, the runs must be distinct and share one runtime version, and a single directory
|
||
keeps the previous single-run output.
|
||
|
||
## Historical unit refresh: October 25, 2026
|
||
|
||
The baseline imports all five successful unit shards from
|
||
[run 40276310369, attempt 1](https://github.com/stablyai/orca/actions/runs/40276310369/attempts/1),
|
||
the runner class PR shards use: `blacksmith-8vcpu-ubuntu-2404`, eight workers, Node
|
||
24.22.1, Bun 1.4.4, source `085cfad2cb3b883c975d5630d94fb2900d0bed05`. The previous ARM
|
||
four-worker baseline lacked 1,976 current files and left PR shards 13–44% apart.
|
||
Balancing two later full runs (40276227094, 40275768715) with each baseline, the fresh one
|
||
cut the heaviest shard from 2,129 to 2,008 and from 2,564 to 2,134 worker-seconds, with a
|
||
spread of 3–4% instead of 13–44%.
|
||
|
||
| Shard | Job ID | Measured worker-seconds | Job seconds |
|
||
| ----- | ------------ | ----------------------: | ----------: |
|
||
| 1 | 121420433743 | 2,476.887 | 381 |
|
||
| 2 | 121420433755 | 2,069.007 | 329 |
|
||
| 3 | 121420433825 | 2,144.439 | 339 |
|
||
| 4 | 121420433749 | 2,050.086 | 322 |
|
||
| 5 | 121420433831 | 2,151.221 | 338 |
|
||
|
||
## Historical unit refresh: October 14, 2026
|
||
|
||
The baseline imports all five successful ARM unit shards from
|
||
[run 38646706136, attempt 1](https://github.com/stablyai/orca/actions/runs/38646706136/attempts/1).
|
||
The tested merge was `8d7f46198b99432efd104ac7803174173b367c66`, using Node
|
||
24.21.0, Bun 1.4.3, `ubuntu-24.04-arm`, and four workers per shard. All 12,314
|
||
files appear exactly once: 12,163 passed and 151 skipped, with zero unhandled
|
||
errors. Artifact digests, all 20 report hashes, source/runtime provenance, full
|
||
discovery, and each saved assignment were checked before importing. The E2E
|
||
weights remain unchanged.
|
||
|
||
| Shard | Job ID | Measured worker-seconds | Job seconds |
|
||
| ----- | ------------ | ----------------------: | ----------: |
|
||
| 1 | 116137259336 | 1,966.470 | 568 |
|
||
| 2 | 116137259460 | 2,080.624 | 604 |
|
||
| 3 | 116137259318 | 2,121.495 | 611 |
|
||
| 4 | 116137259783 | 2,094.487 | 614 |
|
||
| 5 | 116137259540 | 2,190.636 | 641 |
|
||
|
||
Retrospectively assigning those measurements projects 2,090.726–2,090.747
|
||
worker-seconds per shard, instead of 1,966.470–2,190.636. That is a scheduling
|
||
projection, not a hosted speedup: module times overlap across workers, and total
|
||
runner occupancy was 50.63 minutes. Runner capacity, selection, workers, and the
|
||
assignment algorithm are unchanged. Later source changes need fresh validation.
|
||
|
||
## Historical unit refresh: September 27, 2026
|
||
|
||
The baseline imports every successful Node 24 shard from
|
||
[run 36294142683, attempt 1](https://github.com/stablyai/orca/actions/runs/36294142683/attempts/1).
|
||
The recorded checkout was `cc35c1b422e9c20197bc59aa27a71019ffc18b5b`, with Node
|
||
24.21.0 and unchanged Linux workers. All 9,847 report entries match their saved
|
||
assignments and current discovery exactly once, with no unhandled errors. The old
|
||
baseline lacked 165 files and retained one deleted path. The E2E weights are unchanged.
|
||
|
||
Using these measurements for both assignments, the maximum projected load falls
|
||
from 1,043.811 to 919.695 worker-seconds (11.9%); each shard selects 1,230–1,231 files.
|
||
Observed source-run test steps ranged from 282 to 429 seconds. The fastest runner's
|
||
common-file durations were broadly lower than its prior measurements, so runner
|
||
variation may contribute to that spread. The projection does not establish a
|
||
hosted wall-clock improvement. No worker, shard-count, or algorithm change is made.
|
||
|
||
## Previous unit refresh: September 26, 2026
|
||
|
||
The previous baseline imported all eight successful unit shards from
|
||
[run 36221874572, attempt 1](https://github.com/stablyai/orca/actions/runs/36221874572/attempts/1).
|
||
They ran Node **24.21.0** on `ubuntu-latest` with Vitest's default Linux worker
|
||
count and isolation. The checked-out merge was
|
||
`f68ecf8c4aeb396795a8f0f9dbd8ef58fd634d43`, whose second parent is the run's PR
|
||
head `5a26cfa83d7542f9f4545b7a7c4c4174ff213b18`. The workflow failed in another
|
||
lane; every contributing unit job and test step succeeded. No failed reports or
|
||
four-worker trial measurements contributed.
|
||
|
||
| Shard | Job ID | Completed files | Measured worker-seconds | Test step seconds | Job seconds |
|
||
| ----- | ------------ | --------------- | ----------------------- | ----------------- | ----------- |
|
||
| 1 | 108348885240 | 1,209 | 1,019.941 | 420 | 446 |
|
||
| 2 | 108348885231 | 1,210 | 1,000.547 | 413 | 447 |
|
||
| 3 | 108348885289 | 1,211 | 933.332 | 386 | 419 |
|
||
| 4 | 108348885241 | 1,211 | 1,022.838 | 418 | 453 |
|
||
| 5 | 108348885283 | 1,211 | 1,016.000 | 417 | 448 |
|
||
| 6 | 108348885237 | 1,211 | 980.988 | 407 | 440 |
|
||
| 7 | 108348885275 | 1,210 | 810.367 | 326 | 355 |
|
||
| 8 | 108348885261 | 1,210 | 973.722 | 407 | 445 |
|
||
|
||
All reports have `status: passed`, zero unhandled errors and identical source,
|
||
run, attempt, Node version and shard count. Each report's file keys exactly match
|
||
its selected assignment; all eight assignments describe the same plan, with no
|
||
missing or duplicate files across **9,683** measurements. Replaying the old
|
||
baseline reproduces that complete plan. Its SHA-256 is
|
||
`9042d8f1d2420b301ee5d9e56d768ad7d38530897dce0d641f362333aa194ad8`.
|
||
The importer sets overhead to zero; the E2E baseline is unchanged.
|
||
|
||
Discovery at checkout `80ff6c4e3e66de46e41c75856fd1fb6d77d515de`, using the unit
|
||
workflow's exact exclusions, contains **9,732 files**. All 9,683 measurements
|
||
still apply; the **49** unmeasured files receive the new **230ms** median. Both
|
||
plans select every current file exactly once. To compare assignments fairly,
|
||
the following loads evaluate both plans using the same new measurements and
|
||
fallback, rather than comparing two different timing metrics:
|
||
|
||
| Current discovery projection | Previous assignment | Refreshed assignment |
|
||
| ---------------------------- | ------------------- | --------------------- |
|
||
| Files per shard | 1,216–1,217 | 1,215–1,217 |
|
||
| Worker-seconds per shard | 816.045–1,049.604 | 971.124–971.126 |
|
||
| Largest load | 1,049.604 | 971.126 (7.48% lower) |
|
||
|
||
The combined projected work is unchanged at **7,769.005 worker-seconds**.
|
||
Reassignment redistributes measured work; it does not itself reduce test work.
|
||
These are projections from one successful unit run, not a measured wall-clock
|
||
improvement. The source test steps took **326–420 seconds**, totaling **3,194
|
||
seconds (53.233 runner-minutes)**. Full unit jobs took **355–453 seconds**, totaling
|
||
**3,453 seconds (57.550 runner-minutes)**. These durations exclude queue time;
|
||
hosted before/after runs are needed to measure the refreshed assignment's elapsed
|
||
time and occupancy.
|
||
|
||
## Reproduction and refresh
|
||
|
||
Every shard uploads an artifact named with its shard, Node version where relevant,
|
||
and run attempt. `assignment.json` contains the checked-out source SHA, run ID,
|
||
attempt, baseline SHA-256, algorithm, fallback, all shard files and chosen shard.
|
||
E2E also retains both discovery reports and `selected.txt`. Artifacts live for
|
||
14 days. A rerun of the same source uses the same checked-in baseline rather than
|
||
mutable timing caches; a GitHub job rerun therefore keeps its assignment.
|
||
|
||
Unit artifacts also contain `unit-timings.json`, measured through Vitest's reporter
|
||
API. Each file's weight includes environment startup, test harness preparation,
|
||
setup files, imports and test execution. Imports are often more expensive than the
|
||
assertions: run **36212793136**, shard 4, spent **535 worker-seconds importing**
|
||
and **357 running tests**. The refreshed baseline replaces the old uniform import
|
||
allowance with the complete successful measurements described above.
|
||
|
||
For E2E reproduction, check out the recorded source and pass the saved list to the
|
||
existing command: `pnpm run test:e2e --test-list=/path/to/selected.txt` with the same
|
||
CI environment/build inputs. For unit reproduction, use the unchanged workflow
|
||
command and exclusions with `ORCA_BALANCE_UNIT_SHARDS=1` and the recorded
|
||
`--shard=INDEX/8`. Direct test-file reruns remain supported.
|
||
|
||
To refresh unit weights, download all `unit-shard-node-*` artifacts from one
|
||
complete successful unit shard set, run attempt and Node version into a directory,
|
||
preserving their shard subdirectories. An unrelated workflow lane may fail;
|
||
every contributing unit job and report must pass. Check the attempt-specific jobs
|
||
API, source workflow/config, report provenance and exact file coverage against the
|
||
saved assignments, then run:
|
||
|
||
```sh
|
||
node config/scripts/ci/sharding/ci-unit-timing-import.mjs ARTIFACT_DIRECTORY config/scripts/ci/sharding/ci-shard-timings.json
|
||
```
|
||
|
||
The importer rejects missing or duplicate shards, duplicate files, failed unit reports,
|
||
unhandled errors and mixed source revisions, run attempts or Node versions. It
|
||
preserves E2E weights and sets unit overhead to zero because the per-file measurements
|
||
already include it. Timing artifacts remain diagnostic; failed or interrupted unit runs
|
||
are retained for investigation but cannot replace the scheduling baseline.
|
||
|
||
For older unit runs without reporter artifacts, and E2E refreshes, download
|
||
`log-JOB_ID.txt` files into one directory from
|
||
exactly one eight-shard unit run and one fourteen-shard general E2E run. Use the
|
||
job IDs from the Actions jobs API and fetch each with
|
||
`gh api repos/stablyai/orca/actions/jobs/JOB_ID/logs`. Do not include dedicated
|
||
lanes or multiple attempts. Then run:
|
||
|
||
```sh
|
||
node config/scripts/ci/sharding/ci-shard-timing-import.mjs LOG_DIRECTORY UNIT_RUN_ID E2E_RUN_ID config/scripts/ci/sharding/ci-shard-timings.json
|
||
```
|
||
|
||
The initial source logs are in `/tmp/orca-ci-shard-logs`; two were reused from
|
||
`/tmp/orca-ci-audit`, and the remaining twenty were fetched read-only. Reimporting
|
||
those logs reproduced the original baseline byte-for-byte. Review file-count and
|
||
load projections before adopting a new baseline; no network access is needed to
|
||
plan or run shards.
|
||
|
||
## Validation
|
||
|
||
- 74 focused tests passed across the two new test files and existing PR
|
||
parallelism, E2E gate and release E2E dispatch contracts.
|
||
- The pinned Playwright CLI selected the real 733-test suite across all fourteen
|
||
saved test lists with exact-once identity coverage and no missing tests.
|
||
- A temporary native Playwright fixture checks fourteen shards, serial groups,
|
||
skipped cases, headful filtering and mismatch rejection without launching UI.
|
||
- Real Vitest discovery with all workflow exclusions yielded 8,553 files; the
|
||
sequencer's eight assignments covered each exactly once. An actual opt-in
|
||
Vitest shard executed successfully and persisted its manifest.
|
||
- Focused TypeScript checking of `config/vitest.config.ts` and imported modules,
|
||
oxlint, formatting and baseline reimport checks passed.
|
||
- The September 26 refresh matched all eight reports to their saved assignments,
|
||
checked successful attempt-specific job results and source worker settings,
|
||
reproduced the old assignment, and preserved the E2E baseline exactly. All 12
|
||
importer/reporter and shard-assignment tests passed; importing the artifacts
|
||
reproduced the refreshed JSON byte-for-byte.
|
||
|
||
All local tests used `ORCA_BACKGROUND_LAUNCH=1` in background tool sessions. No app
|
||
windows or full E2E test bodies were launched.
|