mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-20 21:18:30 +00:00
Will Jones
ff50e698cf
Standard GitHub-hosted runners are free on public repos, so all Actions spend here is on the `*-8x-*` / `4x` larger runners. Measured over 30 days at current (post-Jan-2026) larger-runner rates, that is ~$1,400/mo, and `npm-publish` is ~70% of it. ## Changes **Fat LTO was forcing builds onto large runners.** `[profile.release]` in `.cargo/config.toml` sets `lto = "fat"` with `codegen-units = 1`, which is single-threaded and the peak-memory step. The macOS `npm-publish` build was 111 of its 113 minutes in one `napi build` step, making it the critical path of the whole publish pipeline. The ThinLTO override already applied to Windows now covers macOS too, and both Windows builds move from `windows-2025-8x-x64` to the free standard `windows-2025`. **The npm-publish cargo cache never existed.** There are zero caches with its key prefix. The key was static, so `actions/cache` (which only writes on a miss) could never refresh it, and a multi-GB release `target/` per target could never fit the repo's 10 GB budget anyway. Now caches only the crate registry, keyed on `Cargo.lock`. The docker builds also mounted `.cargo/registry/*` while the cache saved `.cargo-cache`, so containers re-downloaded the registry every run. **Cache eviction thrash.** Repo cache usage is 10.4 GB against GitHub's 10 GB cap, so every PR run evicted main's warm entries. `rust.yml` and `nodejs.yml` now restore everywhere but only save from `main`. **npm-publish moves to nightly + tags** instead of every push to main (~90/month). The cross-compiled targets do need watching, so `report-failure` now fires on scheduled runs, and dedupes onto an existing open issue rather than filing one per night. **rust.yml aarch64-pc-windows-msvc** cross-compiled its tests and then skipped them, paying full codegen and link cost for a compile check. `windows-11-arm` is now GA and free on public repos, so it builds and tests natively. Its test step also passes `--target` — without it cargo used `target/ci/` rather than `target/<triple>/ci/` and rebuilt the entire dependency graph a second time. **pypi-publish.yml had no concurrency group**, so force-pushes left a ~74 minute Windows job running. ## What is cost vs. wall-clock | Change | Cost | Wall-clock | |---|---|---| | Windows npm-publish → free runners | **−$570/mo** | slower per job (8→4 cores) | | npm-publish nightly | **−$125/mo** | — | | pypi-publish concurrency | small | — | | macOS ThinLTO | $0 (already free) | **−~50 min** per release | | rust aarch64 Windows native | $0 (already free) | **−~25 min** | | rust `--target` on test step | $0 | large, avoids a second full build | | rust-cache `save-if` | small | faster via real cache hits | ## Risks - The two Windows builds now have 4 cores instead of 8 and ~14 GB of free disk. If they fail, it is most likely disk rather than memory; fallback is `windows-2025-4x-x64`, which still halves that line. - `windows-11-arm` has a thinner toolset (choco/vcpkg/protoc under emulation) and this enables a test step that has never run, so it may surface real aarch64 failures. That is the point, but it is the change most likely to need iteration. - ThinLTO applies to published macOS and Windows binaries, typically within a few percent of fat LTO. Linux release builds are untouched. ## Follow-ups - `python.yml` `pydantic1x` (37 min) and `Doctest` (33 min) each rebuild the extension from source via `pip install -e .` with no Rust cache; they should consume the wheel the `linux` job already builds. Worth ~$235/mo and ~70 min of compute per run. Separate PR. - The three `ubuntu-2404-8x-x64` npm-publish builds (~$420/mo at the old cadence) are the remaining large-runner spend; `aarch64-unknown-linux-gnu` could run natively on free `ubuntu-24.04-arm`. Worth doing after this lands so the ThinLTO change can be validated first. - The wheel composite actions declare `python-minor-version` as required but never use it, and every caller omits it (actionlint warns). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
58 lines
2.1 KiB
YAML
58 lines
2.1 KiB
YAML
name: Create Failure Issue
|
|
description: Creates a GitHub issue if any jobs in the workflow failed
|
|
|
|
inputs:
|
|
job-results:
|
|
description: 'JSON string of job results from needs context'
|
|
required: true
|
|
workflow-name:
|
|
description: 'Name of the workflow'
|
|
required: true
|
|
|
|
runs:
|
|
using: composite
|
|
steps:
|
|
- name: Check for failures and create issue
|
|
shell: bash
|
|
env:
|
|
JOB_RESULTS: ${{ inputs.job-results }}
|
|
WORKFLOW_NAME: ${{ inputs.workflow-name }}
|
|
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
|
GH_TOKEN: ${{ github.token }}
|
|
run: |
|
|
# Check if any job failed
|
|
if echo "$JOB_RESULTS" | jq -e 'to_entries | any(.value.result == "failure")' > /dev/null; then
|
|
echo "Detected job failures, creating issue..."
|
|
|
|
# Extract failed job names
|
|
FAILED_JOBS=$(echo "$JOB_RESULTS" | jq -r 'to_entries | map(select(.value.result == "failure")) | map(.key) | join(", ")')
|
|
|
|
TITLE="$WORKFLOW_NAME Failed ($FAILED_JOBS)"
|
|
|
|
# This action now also runs on nightly schedules, so a breakage that
|
|
# persists for a few days would otherwise file one issue per night.
|
|
# Comment on the open report instead when one already exists.
|
|
EXISTING=$(gh issue list --state open --label ci --limit 100 --json number,title \
|
|
| jq -r --arg title "$TITLE" 'map(select(.title == $title)) | .[0].number // empty')
|
|
|
|
if [ -n "$EXISTING" ]; then
|
|
gh issue comment "$EXISTING" --body "Failed again: $RUN_URL"
|
|
echo "Commented on existing issue #$EXISTING"
|
|
else
|
|
gh issue create \
|
|
--title "$TITLE" \
|
|
--body "The workflow **$WORKFLOW_NAME** failed during execution.
|
|
|
|
**Failed jobs:** $FAILED_JOBS
|
|
|
|
**Run URL:** $RUN_URL
|
|
|
|
Please investigate the failed jobs and address any issues." \
|
|
--label "ci"
|
|
|
|
echo "Issue created successfully"
|
|
fi
|
|
else
|
|
echo "No job failures detected, skipping issue creation"
|
|
fi
|