Files
lancedb/.github/workflows/rust.yml
T
Xuanwo 193f203b72 ci: run remote Functions gate only in the full test suite (#5022)
The remote job runs the default local Functions gate twice: once before
building the remaining test targets, and again in the full remote suite.
Run it only through the existing full suite, removing the serial Cargo
invocation barrier while retaining the same features, authentication
wrapper, targets, and 30-minute deadline.

The [hosted remote
job](https://github.com/lancedb/lancedb/actions/runs/43302507952/job/131477104217)
at `7886d79e` completed successfully in 25m36s within the unchanged
30-minute limit. Its full suite passed 2,164 unit, 114 integration, and
120 doctests, including the three default gate tests. This single run
establishes completion for that workload, not a general timing
improvement or cold-build guarantee. The branch now includes main's
Cargo artifact fixture repair (#5010). The workflow, full-suite command,
authentication wrapper, three default Functions gate tests, and
30-minute deadline are unchanged from that hosted run. The 2,398-test
result remains evidence for `7886d79e`. On the updated head `810d6558`,
the [hosted remote
job](https://github.com/lancedb/lancedb/actions/runs/43382023177/job/131741745645)
completed in 17m58s with 2,171 unit tests, 119 integration tests, and
122 doctests passing (82 ignored), including all three default Functions
gate tests. This is current-head execution evidence, not a controlled
speedup comparison.
2026-11-14 20:54:19 +08:00

457 lines
17 KiB
YAML

name: Rust
on:
push:
branches:
- main
pull_request:
paths:
- Cargo.toml
- Cargo.lock
- rust-toolchain.toml
- deny.toml
- rust/**
- docs/web-tests/blob-v2/rs/**
- nodejs/Cargo.toml
- python/Cargo.toml
- python/src/**
- python/python/lancedb/exceptions.py
- python/python/lancedb/remote/errors.py
- ci/build_no_lock.py
- ci/tests/test_build_no_lock.py
- .github/actions/setup-protoc/**
- .github/workflows/rust.yml
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
# This env var is used by Swatinem/rust-cache@v2 for the cache
# key, so we set it to make sure it is always consistent.
CARGO_TERM_COLOR: always
RUST_BACKTRACE: "1"
jobs:
dependency-overrides:
name: Downstream Lance override
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
- uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Read the pinned Lance tag
id: lance
run: |
python3 - <<'PYTHON'
import os
import tomllib
from pathlib import Path
manifest = tomllib.loads(Path("Cargo.toml").read_text())
tag = manifest["workspace"]["dependencies"]["lance"]["tag"]
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write(f"tag={tag}\n")
PYTHON
- uses: actions/checkout@v6
with:
repository: lance-format/lance
ref: ${{ steps.lance.outputs.tag }}
path: local-lance
- name: Resolve a downstream workspace with a local Lance override
run: |
mkdir -p downstream/src
touch downstream/src/lib.rs
cat > downstream/Cargo.toml <<'TOML'
[package]
name = "lancedb-local-lance-consumer"
version = "0.1.0"
edition = "2024"
[workspace]
[dependencies]
lancedb = { path = "../rust/lancedb", default-features = false }
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
[patch."https://github.com/lance-format/lance.git"]
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
TOML
cargo metadata --manifest-path downstream/Cargo.toml --format-version 1 > downstream/graph.json
python3 - <<'PYTHON'
import json
from pathlib import Path
graph = json.loads(Path("downstream/graph.json").read_text())
packages = [p for p in graph["packages"] if p["name"] == "lance-arrow"]
assert len(packages) == 1, [p["id"] for p in packages]
assert Path(packages[0]["manifest_path"]).resolve() == Path(
"local-lance/rust/lance-arrow/Cargo.toml"
).resolve()
PYTHON
lint:
timeout-minutes: 30
runs-on: ubuntu-24.04
defaults:
run:
shell: bash
env:
# Need up-to-date compilers for kernels
CC: clang-18
CXX: clang++-18
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
lfs: true
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
components: rustfmt, clippy
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Run format
run: cargo fmt --all -- --check
- name: Run clippy
run: cargo clippy --profile ci --workspace --tests --all-features -- -D warnings
- name: Run clippy (without remote feature)
run: cargo clippy --profile ci --workspace --tests -- -D warnings
deny:
# Supply-chain checks: advisories, licenses, banned crates, and source
# restrictions. Configuration lives in `deny.toml` at the workspace root.
timeout-minutes: 10
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
- uses: EmbarkStudios/cargo-deny-action@v2
with:
command: check advisories bans licenses sources
build-no-lock:
# Keep the existing Linux runner; its available disk varies by image.
runs-on: ubuntu-2404-4x-x64
timeout-minutes: 30
env:
# Need up-to-date compilers for kernels
CC: clang
CXX: clang++
steps:
- uses: actions/checkout@v6
# Building without a lock file often requires the latest Rust version since downstream
# dependencies may have updated their minimum Rust version.
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: "stable"
# The explicit cache below runs after lock removal and only saves on main.
cache: false
# Remove cargo.lock to force a fresh build
- name: Remove Cargo.lock
run: rm -f Cargo.lock
- uses: rui314/setup-mold@v1
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Test build output cleanup
run: python3 -m unittest discover -s ci/tests -p test_build_no_lock.py -v
- name: Build all
run: |
df -h .
# Every target is linked, but this job never executes the test binaries.
python3 ci/build_no_lock.py
- name: Report build disk usage
if: always()
run: |
df -h .
for build_path in target "$HOME/.cargo/registry" "$HOME/.cargo/git"; do
if [ -d "$build_path" ]; then
du -h -d 2 "$build_path"
fi
done
linux:
timeout-minutes: 60
# To build all features, we need more disk space than is available
# on the free OSS github runner. This is mostly due to the the
# sentence-transformers feature.
runs-on: ubuntu-2404-4x-x64
defaults:
run:
shell: bash
working-directory: rust
env:
# Need up-to-date compilers for kernels
CC: clang-18
CXX: clang++-18
# Preserve the repository CPU flags and use setup-mold's system linker.
RUSTFLAGS: "-C target-cpu=haswell -C target-feature=+avx2,+fma,+f16c -C linker-features=-lld"
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
lfs: true
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- uses: rui314/setup-mold@v1
- name: Make Swap
run: |
swapfile=/swapfile
min_swap_bytes=$((15 * 1024 * 1024 * 1024))
active_swap_bytes="$(sudo swapon --show=NAME,SIZE --bytes --noheadings | awk '$1 == "/swapfile" { print $2 }')"
if [ -n "$active_swap_bytes" ]; then
if [ "$active_swap_bytes" -ge "$min_swap_bytes" ]; then
echo "/swapfile is already active with enough space; skipping swap creation"
exit 0
fi
echo "/swapfile is already active but smaller than 16G; using /mnt/lancedb-swapfile"
swapfile=/mnt/lancedb-swapfile
fi
if sudo swapon --show=NAME --noheadings | grep -Fxq "$swapfile"; then
echo "$swapfile is already active; skipping swap creation"
exit 0
fi
sudo rm -f "$swapfile"
sudo fallocate -l 16G "$swapfile"
sudo chmod 600 "$swapfile"
sudo mkswap "$swapfile"
sudo swapon "$swapfile"
- name: Build
run: cargo build --profile ci -p lancedb --all-features --tests --locked --examples
- name: Run feature tests
run: CARGO_ARGS="--profile ci" make -C ./lancedb feature-tests
- name: Run examples
run: cargo run --profile ci -p lancedb --all-features --example simple --locked
- name: Run blob v2 docs example
# Python and TypeScript blob docs tests install preview packages.
# Those builds are not guaranteed to remain available for more than
# 6 months, so they stay a manual run. This example uses the workspace crate.
run: |
# The completed all-features checks leave large lancedb artifacts in target.
# Reclaim that package before building the example with a different feature set.
cargo clean --profile ci -p lancedb
cargo run --profile ci --all-features -p lancedb-blob-v2-docs --example multimodal --locked
remote:
timeout-minutes: 30
# Running this requires access to secrets, so skip if this is a PR from a
# fork. Keep it separate from the all-features build so Cargo does not
# retain both dependency graphs in one target directory.
if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
runs-on: ubuntu-2404-4x-x64
defaults:
run:
shell: bash
working-directory: rust
env:
CC: clang-18
CXX: clang++-18
# Rust's default rust-lld bypasses setup-mold's /usr/bin/ld replacement.
RUSTFLAGS: "-C target-cpu=haswell -C target-feature=+avx2,+fma,+f16c -C linker-features=-lld"
GH_TOKEN: ${{ github.token }}
SOPHON_READ_TOKEN: ${{ secrets.SOPHON_READ_TOKEN }}
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
lfs: true
- uses: Swatinem/rust-cache@v2
with:
# Remote tests use a different feature graph from the main Linux
# job. Cache downloads, but build into a fresh target directory.
cache-targets: false
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- uses: rui314/setup-mold@v1
- name: Run remote tests
run: CARGO_ARGS="--profile ci" make -C ./lancedb remote-tests
macos:
name: ${{ matrix.phase == 'tests' && format('macos ({0})', matrix.mac-runner) || format('macos-doc ({0})', matrix.mac-runner) }}
timeout-minutes: 60
strategy:
matrix:
mac-runner: ["macos-15", "macos-26"]
phase: [tests, doc]
runs-on: "${{ matrix.mac-runner }}"
defaults:
run:
shell: bash
working-directory: rust
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
lfs: true
- name: CPU features
run: sysctl -a | grep cpu
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install dependencies
run: brew install protobuf
- name: Run tests
env:
CARGO_TEST_TARGETS: ${{ matrix.phase == 'doc' && '--doc' || '--lib --tests' }}
run: |
# Don't run the s3 integration tests since docker isn't available
# on this image.
ALL_FEATURES=`cargo metadata --format-version=1 --no-deps \
| jq -r '.packages[] | .features | keys | .[]' \
| grep -v s3-test | sort | uniq | paste -s -d "," -`
# Separate jobs give each phase its own time and disk budget. Examples
# are already built by the Linux job, so avoid retaining them here.
cargo test --profile ci --features $ALL_FEATURES --locked $CARGO_TEST_TARGETS
windows:
strategy:
fail-fast: false
matrix:
include:
- target: x86_64-pc-windows-msvc
runner: windows-2022
runs-on: ${{ matrix.runner }}
defaults:
run:
working-directory: rust/lancedb
steps:
- uses: actions/checkout@v6
- name: Enable long paths for Lance dependencies
run: git config --global core.longpaths true
- name: Set target
run: rustup target add ${{ matrix.target }}
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install protoc
uses: ./.github/actions/setup-protoc
- name: Build
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
cargo build --profile ci --features aws,remote --tests --locked --target ${{ matrix.target }}
- name: Run tests
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
# `--target` has to match the build step above. Without it cargo uses
# target/ci/ rather than target/<triple>/ci/ and rebuilds the entire
# dependency graph a second time.
cargo test --profile ci --features aws,remote --locked --target ${{ matrix.target }}
# Run the aarch64 Windows tests only on main so PRs wait for a single
# Windows platform. windows-11-arm is a standard runner, so it is free on
# public repos. Running natively lets the aarch64 tests actually execute --
# this job used to cross-compile them and then skip the test step, paying
# full codegen and link cost for a compile check.
windows-arm:
# Keep the historical name so the check shows up as before on main.
name: windows (aarch64-pc-windows-msvc, windows-11-arm)
if: github.ref == 'refs/heads/main'
runs-on: windows-11-arm
defaults:
run:
working-directory: rust/lancedb
steps:
- uses: actions/checkout@v6
- name: Enable long paths for Lance dependencies
run: git config --global core.longpaths true
- name: Set target
run: rustup target add aarch64-pc-windows-msvc
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install protoc
uses: ./.github/actions/setup-protoc
- name: Build
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
cargo build --profile ci --features aws,remote --tests --locked --target aarch64-pc-windows-msvc
- name: Run tests
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
# `--target` has to match the build step above. Without it cargo uses
# target/ci/ rather than target/<triple>/ci/ and rebuilds the entire
# dependency graph a second time.
cargo test --profile ci --features aws,remote --locked --target aarch64-pc-windows-msvc
msrv:
# Check the minimum supported Rust version
name: MSRV Check - Rust v${{ matrix.msrv }}
runs-on: ubuntu-24.04
strategy:
matrix:
msrv: ["1.95.0"] # This should match up with rust-version in Cargo.toml
env:
# Need up-to-date compilers for kernels
CC: clang-18
CXX: clang++-18
steps:
- uses: actions/checkout@v6
with:
submodules: true
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install ${{ matrix.msrv }}
uses: dtolnay/rust-toolchain@master
with:
toolchain: ${{ matrix.msrv }}
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Resolve MSRV-compatible dependencies
# Resolve the AWS SDK and Smithy crates together: newer SDK releases
# require newer Smithy versions, so downgrading only Smithy can fail.
run: cargo +${{ matrix.msrv }} update --config 'resolver.incompatible-rust-versions="fallback"'
- name: Keep Xet dependencies compatible with the MSRV
# Xet 1.7 does not declare its MSRV, but requires sysinfo 0.39 (Rust 1.95).
# Downgrade parents before their dependencies to satisfy their constraints.
run: |
for crate in hf-xet xet-data xet-client xet-core-structures xet-runtime; do
cargo +${{ matrix.msrv }} update -p "$crate" --precise 1.6.0
done
- name: cargo +${{ matrix.msrv }} check
env:
RUSTUP_TOOLCHAIN: ${{ matrix.msrv }}
run: cargo check --locked --profile ci --workspace --tests --benches --all-features