mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-04 04:28:44 +00:00
ci(nodejs): fix nightly OOM on the aarch64 publish legs (#4077)
The nightly `NPM Publish` run has failed every night since at least Aug 23, always on the same two legs: `aarch64-unknown-linux-gnu` and `aarch64-unknown-linux-musl`. The other five targets pass. rustc is OOM-killed during the fat-LTO codegen of the cdylib — `signal: 9` with no diagnostic, about 27 minutes in — and on the musl leg that takes the whole runner down with `The runner has received a shutdown signal`. Both legs now pass: | leg | before | peak memory | wall time | | --- | --- | --- | --- | | `aarch64-unknown-linux-gnu` | OOM-killed at ~27 min | 31391 → 22851 MiB | 38m43s → 22m04s | | `aarch64-unknown-linux-musl` | runner killed at ~28 min | >32 GiB → 16516 MiB | ~40 min → 20m50s | **ThinLTO** is most of that. Fat LTO is single-threaded, and its peak is consumed inside rustc's LLVM before any linker process is spawned — which is why it is the whole fix on musl, and why lld alone left the gnu leg still peaking at 31391 MiB against the runner's 32 GiB. Both legs now use the `lto: thin` / `codegen_units: 16` settings that darwin and both Windows legs already use, at a cost of a few percent runtime performance. **lld** covers the rest, on the gnu leg. arm64 Linux otherwise links through GNU `ld` where x86_64 already defaults to `rust-lld`, which is why only the arm64 legs hit this at all; on a comparable arm64 build (`lancedb/sophon#7313`) it cut the largest single linker process from 7.0 to 4.0 GiB and wall time by 35%. The flags live in a small wrapper script used as the linker rather than in `-C link-arg`, because the per-target rustflags variable does not reach every unit that links: dependency crates linking a dylib (`crc-fast`, `lance-arrow`) were invoked as bare `clang`, which targets the x86_64 host and fails with `Relocations in generic ELF (EM: 183)`. Separately, and affecting five legs rather than two: the three ThinLTO targets exported `CARGO_PROFILE_RELEASE_LTO` and `CARGO_PROFILE_RELEASE_CODEGEN_UNITS` from `pre_build`, which runs inside the build step — after the cache step. `Swatinem/rust-cache` computes its key when the action runs, before any step, so step-local values are invisible to it. The result is a loop that never converges: the key never changes, so restores are exact hits, an exact hit makes the post-run save a no-op, and cargo invalidates the restored artifacts anyway because the flags differ. Those legs have been rebuilding cold on every run. Both values move to job-level `env:` ahead of the cache step, driven by new `lto:`/`codegen_units:` matrix fields, and are forwarded into the containers with `-e` since `docker run` inherits nothing. Every leg's cache key shifts once as a result, so expect one cold rebuild. A `Report peak memory` step is added so whether these legs fit is a number rather than an inference from whether the runner survived. It produced the figures above. ## Not included Moving these legs to native arm64 runners. It would retire the zig cross path, the `AT_HWCAP2` workaround and the `TARGET_CC` override, and arm64 runners are billed roughly 37% below x64 at equal core count — but the `lts-debian-aarch64` image exists to link against the manylinux2014 sysroot's glibc 2.17, and building natively on ubuntu-24.04 would raise the minimum glibc for every published aarch64 binary. That is a user-facing decision, not a CI cleanup. Dropping these legs to smaller runners, which is where the real cost saving is — larger runners are billed even on public repos. On these numbers it is not available yet: musl at 16516 MiB is about 130 MiB over what a 16 GB standard runner has. Worth revisiting as a follow-up. ## Testing Cargo's rustflags precedence was checked locally rather than taken from the docs, since getting it wrong would silently change the published binaries. With a throwaway crate carrying both a `target.'cfg(all())'` and a per-target rustflags table: setting `RUSTFLAGS` discards both, and setting it to the empty string discards them too. That rules out routing the linker flag through a job-level `RUSTFLAGS`, because `env:` keys cannot be conditionally omitted and every other leg would then silently lose the `target-cpu`/`target-feature` settings in `.cargo/config.toml` — `+avx2` on x86_64 and `-crt-static` on aarch64-musl. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Will Jones
GitHub
@@ -40,40 +40,31 @@ jobs:
|
||||
- target: aarch64-apple-darwin
|
||||
host: macos-latest
|
||||
features: fp16kernels
|
||||
# Fat LTO was ~111 of this job's ~113 minutes.
|
||||
lto: thin
|
||||
codegen_units: 16
|
||||
pre_build: |-
|
||||
brew install protobuf
|
||||
# Fat LTO (the workspace default in .cargo/config.toml) is
|
||||
# single-threaded and is the peak-memory step of the build. On
|
||||
# this runner it accounted for ~111 of the job's ~113 minutes,
|
||||
# making it the critical path of the entire publish pipeline.
|
||||
# ThinLTO parallelizes it across the runner's cores, for a few
|
||||
# percent of runtime performance.
|
||||
export CARGO_PROFILE_RELEASE_LTO=thin
|
||||
export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
|
||||
- target: x86_64-pc-windows-msvc
|
||||
host: windows-2025
|
||||
features: ","
|
||||
# The lower peak also keeps this on the standard 4-core runner.
|
||||
lto: thin
|
||||
codegen_units: 16
|
||||
pre_build: |-
|
||||
choco install --no-progress protoc ninja nasm
|
||||
tail -n 1000 /c/ProgramData/chocolatey/logs/chocolatey.log
|
||||
# There is an issue where choco doesn't add nasm to the path
|
||||
export PATH="$PATH:/c/Program Files/NASM"
|
||||
nasm -v
|
||||
# See the ThinLTO note on aarch64-apple-darwin above. Keeping
|
||||
# peak memory down is also what lets this run on the standard
|
||||
# 4-core runner: the 8-core larger runner was only needed to
|
||||
# stop fat LTO from OOMing rustc-LLVM.
|
||||
export CARGO_PROFILE_RELEASE_LTO=thin
|
||||
export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
|
||||
- target: aarch64-pc-windows-msvc
|
||||
host: windows-2025
|
||||
features: ","
|
||||
lto: thin
|
||||
codegen_units: 16
|
||||
pre_build: |-
|
||||
choco install --no-progress protoc
|
||||
rustup target add aarch64-pc-windows-msvc
|
||||
# See the ThinLTO note on aarch64-apple-darwin above.
|
||||
export CARGO_PROFILE_RELEASE_LTO=thin
|
||||
export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
|
||||
- target: x86_64-unknown-linux-gnu
|
||||
host: ubuntu-latest
|
||||
features: fp16kernels
|
||||
@@ -103,6 +94,14 @@ jobs:
|
||||
# https://github.com/napi-rs/napi-rs/blob/main/debian-aarch64.Dockerfile
|
||||
docker: ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-debian-aarch64
|
||||
features: "fp16kernels"
|
||||
# Fat LTO OOM-killed rustc every nightly; even with lld it peaked
|
||||
# at 31391 MiB of the runner's 32 GiB.
|
||||
lto: thin
|
||||
codegen_units: 16
|
||||
# arm64 Linux links through GNU `ld` where x86_64 defaults to
|
||||
# `rust-lld`, which is why only arm64 OOM'd. lld cut the largest
|
||||
# linker process 7.0 -> 4.0 GiB (lancedb/sophon#7313).
|
||||
linker: /tmp/aarch64-lld-clang
|
||||
pre_build: |-
|
||||
set -e &&
|
||||
apt-get update &&
|
||||
@@ -112,9 +111,30 @@ jobs:
|
||||
# AT_HWCAP2 (added in Linux 3.17). Define it for aws-lc-sys.
|
||||
export CFLAGS="$CFLAGS -DAT_HWCAP2=26" &&
|
||||
rustup target add aarch64-unknown-linux-gnu
|
||||
# Not `&&`-chained: in dash, errexit does not fire for a
|
||||
# non-final command in an `&&` list, so failures were ignored.
|
||||
#
|
||||
# A wrapper rather than `-C link-arg` because the per-target
|
||||
# rustflags variable does not reach every unit that links, while
|
||||
# the linker variable does. `clang` because GCC silently ignores
|
||||
# `-fuse-ld=lld` unless built with lld support. Two echoes
|
||||
# because printf's newline escape gets rewritten to `;` between
|
||||
# here and the container.
|
||||
echo '#!/bin/sh' > /tmp/aarch64-lld-clang
|
||||
echo 'exec clang --target=aarch64-unknown-linux-gnu --sysroot=/usr/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot --gcc-toolchain=/usr/aarch64-unknown-linux-gnu -fuse-ld=lld "$@"' >> /tmp/aarch64-lld-clang
|
||||
chmod 0755 /tmp/aarch64-lld-clang
|
||||
# Fail now, not at the cdylib link ~30 minutes later. Linking at
|
||||
# all also proves lld resolved; clang errors out when it cannot.
|
||||
echo 'int main(void){return 0;}' > /tmp/probe.c
|
||||
/tmp/aarch64-lld-clang /tmp/probe.c -o /tmp/probe
|
||||
readelf -h /tmp/probe | grep AArch64
|
||||
- target: aarch64-unknown-linux-musl
|
||||
host: ubuntu-2404-8x-x64
|
||||
features: ","
|
||||
# Fat LTO took the whole runner down. lld cannot help: it died
|
||||
# inside rustc's LLVM, before any linker was spawned.
|
||||
lto: thin
|
||||
codegen_units: 16
|
||||
pre_build: |-
|
||||
set -e &&
|
||||
sudo apt-get update &&
|
||||
@@ -123,6 +143,19 @@ jobs:
|
||||
export EXTRA_ARGS="-x"
|
||||
name: build - ${{ matrix.settings.target }}
|
||||
runs-on: ${{ matrix.settings.host }}
|
||||
# On the job, not exported from `pre_build`: `Swatinem/rust-cache` hashes
|
||||
# `CARGO_*` into its cache key before any step runs, so a step-local export
|
||||
# leaves the key unchanged while cargo still rebuilds cold. The ThinLTO
|
||||
# legs had been doing that every run.
|
||||
#
|
||||
# Not `RUSTFLAGS`: setting it, even to "", discards every config-file
|
||||
# rustflag, silently dropping .cargo/config.toml's `target-cpu` and
|
||||
# `target-feature` from the published binaries.
|
||||
env:
|
||||
CARGO_PROFILE_RELEASE_LTO: ${{ matrix.settings.lto || 'fat' }}
|
||||
CARGO_PROFILE_RELEASE_CODEGEN_UNITS: ${{ matrix.settings.codegen_units || '1' }}
|
||||
# Empty elsewhere: a per-target variable is only read for that triple.
|
||||
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: ${{ matrix.settings.linker }}
|
||||
defaults:
|
||||
run:
|
||||
working-directory: nodejs
|
||||
@@ -169,19 +202,15 @@ jobs:
|
||||
# creating ref). The nightly cadence also keeps entries inside
|
||||
# GitHub's 7-day eviction window, which a tag-only trigger would not.
|
||||
save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
# Docker builds can use rust-cache too. `target/` already lives on the
|
||||
# host because the whole workspace is bind-mounted into the container, and
|
||||
# rust-cache's prune and save run host-side, so they can manage it -- which
|
||||
# is what keeps the entry to dependency artifacts rather than a multi-GB
|
||||
# copy of everything.
|
||||
# Docker builds can use rust-cache too: the workspace is bind-mounted, so
|
||||
# `target/` lives on the host and rust-cache's prune keeps the entry
|
||||
# small.
|
||||
#
|
||||
# Two differences from the native builds. The container's CARGO_HOME is
|
||||
# bind-mounted from `.cargo-cache` rather than the host's ~/.cargo, so that
|
||||
# has to be cached explicitly. And the key is derived from the *host* rustc
|
||||
# version, which is not the compiler that produced these artifacts; that is
|
||||
# safe because cargo fingerprints the real compiler and rebuilds on a
|
||||
# mismatch, it just means a base-image toolchain bump costs one cold build
|
||||
# instead of invalidating the key.
|
||||
# bind-mounted from `.cargo-cache` rather than ~/.cargo, so that is cached
|
||||
# explicitly. And the key uses the *host* rustc version, not the compiler
|
||||
# that built these artifacts -- safe, since cargo fingerprints the real
|
||||
# one; a base-image bump just costs one cold build.
|
||||
- name: Cache cargo (docker builds)
|
||||
uses: Swatinem/rust-cache@v2
|
||||
if: ${{ matrix.settings.docker }}
|
||||
@@ -210,9 +239,14 @@ jobs:
|
||||
# cache step above saves. Previously the registry mounts pointed at
|
||||
# `.cargo/...`, a path nothing cached, so the container re-downloaded
|
||||
# the whole crate registry on every run.
|
||||
#
|
||||
# `docker run` inherits nothing; `-e NAME` carries the job's `env:` in.
|
||||
options: "--user 0:0 -v ${{ github.workspace }}/.cargo-cache/git/db:/usr/local/cargo/git/db \
|
||||
-v ${{ github.workspace }}/.cargo-cache/registry/cache:/usr/local/cargo/registry/cache \
|
||||
-v ${{ github.workspace }}/.cargo-cache/registry/index:/usr/local/cargo/registry/index \
|
||||
-e CARGO_PROFILE_RELEASE_LTO \
|
||||
-e CARGO_PROFILE_RELEASE_CODEGEN_UNITS \
|
||||
-e CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER \
|
||||
-v ${{ github.workspace }}:/build -w /build/nodejs"
|
||||
run: |
|
||||
set -e
|
||||
@@ -256,6 +290,18 @@ jobs:
|
||||
if: always()
|
||||
run: df -h
|
||||
shell: bash
|
||||
- name: Report peak memory
|
||||
if: always() && runner.os == 'Linux'
|
||||
shell: bash
|
||||
run: |
|
||||
peak=$(find /sys/fs/cgroup -name memory.peak -readable \
|
||||
-exec cat {} + 2>/dev/null | sort -n | tail -1)
|
||||
if [ -n "$peak" ]; then
|
||||
echo "peak memory: $((peak / 1024 / 1024)) MiB"
|
||||
else
|
||||
echo "peak memory: unavailable (no readable cgroup v2 memory.peak)"
|
||||
fi
|
||||
free -g || true
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
|
||||
Reference in New Issue
Block a user