mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 12:08:35 +00:00
ci: cut Actions cost by moving builds to free runners and fixing caches (#3735)
Standard GitHub-hosted runners are free on public repos, so all Actions spend here is on the `*-8x-*` / `4x` larger runners. Measured over 30 days at current (post-Jan-2026) larger-runner rates, that is ~$1,400/mo, and `npm-publish` is ~70% of it. ## Changes **Fat LTO was forcing builds onto large runners.** `[profile.release]` in `.cargo/config.toml` sets `lto = "fat"` with `codegen-units = 1`, which is single-threaded and the peak-memory step. The macOS `npm-publish` build was 111 of its 113 minutes in one `napi build` step, making it the critical path of the whole publish pipeline. The ThinLTO override already applied to Windows now covers macOS too, and both Windows builds move from `windows-2025-8x-x64` to the free standard `windows-2025`. **The npm-publish cargo cache never existed.** There are zero caches with its key prefix. The key was static, so `actions/cache` (which only writes on a miss) could never refresh it, and a multi-GB release `target/` per target could never fit the repo's 10 GB budget anyway. Now caches only the crate registry, keyed on `Cargo.lock`. The docker builds also mounted `.cargo/registry/*` while the cache saved `.cargo-cache`, so containers re-downloaded the registry every run. **Cache eviction thrash.** Repo cache usage is 10.4 GB against GitHub's 10 GB cap, so every PR run evicted main's warm entries. `rust.yml` and `nodejs.yml` now restore everywhere but only save from `main`. **npm-publish moves to nightly + tags** instead of every push to main (~90/month). The cross-compiled targets do need watching, so `report-failure` now fires on scheduled runs, and dedupes onto an existing open issue rather than filing one per night. **rust.yml aarch64-pc-windows-msvc** cross-compiled its tests and then skipped them, paying full codegen and link cost for a compile check. `windows-11-arm` is now GA and free on public repos, so it builds and tests natively. Its test step also passes `--target` — without it cargo used `target/ci/` rather than `target/<triple>/ci/` and rebuilt the entire dependency graph a second time. **pypi-publish.yml had no concurrency group**, so force-pushes left a ~74 minute Windows job running. ## What is cost vs. wall-clock | Change | Cost | Wall-clock | |---|---|---| | Windows npm-publish → free runners | **−$570/mo** | slower per job (8→4 cores) | | npm-publish nightly | **−$125/mo** | — | | pypi-publish concurrency | small | — | | macOS ThinLTO | $0 (already free) | **−~50 min** per release | | rust aarch64 Windows native | $0 (already free) | **−~25 min** | | rust `--target` on test step | $0 | large, avoids a second full build | | rust-cache `save-if` | small | faster via real cache hits | ## Risks - The two Windows builds now have 4 cores instead of 8 and ~14 GB of free disk. If they fail, it is most likely disk rather than memory; fallback is `windows-2025-4x-x64`, which still halves that line. - `windows-11-arm` has a thinner toolset (choco/vcpkg/protoc under emulation) and this enables a test step that has never run, so it may surface real aarch64 failures. That is the point, but it is the change most likely to need iteration. - ThinLTO applies to published macOS and Windows binaries, typically within a few percent of fat LTO. Linux release builds are untouched. ## Follow-ups - `python.yml` `pydantic1x` (37 min) and `Doctest` (33 min) each rebuild the extension from source via `pip install -e .` with no Rust cache; they should consume the wheel the `linux` job already builds. Worth ~$235/mo and ~70 min of compute per run. Separate PR. - The three `ubuntu-2404-8x-x64` npm-publish builds (~$420/mo at the old cadence) are the remaining large-runner spend; `aarch64-unknown-linux-gnu` could run natively on free `ubuntu-24.04-arm`. Worth doing after this lands so the ThinLTO change can be validated first. - The wheel composite actions declare `python-minor-version` as required but never use it, and every caller omits it (actionlint warns). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -61,6 +61,11 @@ jobs:
|
||||
sudo apt update
|
||||
sudo apt install -y protobuf-compiler libssl-dev
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
# Restore everywhere, but only save from main. Per-PR saves are
|
||||
# unreadable outside their own branch anyway, since GitHub scopes
|
||||
# caches to the creating ref.
|
||||
save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
- name: Format Rust
|
||||
run: cargo fmt --all -- --check
|
||||
- name: Lint Rust
|
||||
@@ -103,6 +108,11 @@ jobs:
|
||||
cache: 'pnpm'
|
||||
cache-dependency-path: nodejs/pnpm-lock.yaml
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
# Restore everywhere, but only save from main. Per-PR saves are
|
||||
# unreadable outside their own branch anyway, since GitHub scopes
|
||||
# caches to the creating ref.
|
||||
save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
sudo apt update
|
||||
@@ -182,6 +192,11 @@ jobs:
|
||||
cache-dependency-path: nodejs/pnpm-lock.yaml
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
# Restore everywhere, but only save from main. Per-PR saves are
|
||||
# unreadable outside their own branch anyway, since GitHub scopes
|
||||
# caches to the creating ref.
|
||||
save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
brew install protobuf
|
||||
|
||||
Reference in New Issue
Block a user