mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
Current release status: hold for the independently reproduced LargeList reader regression in https://github.com/lancedb/lancedb/pull/4712#discussion_r4409195229. All 54 CI checks passed at `a26e461b86728f7e7977f1c221af70a32033a144`; passing CI does not clear that compatibility finding. The Java artifact availability finding is fixed: the official Lance Maven mirror and Maven Central now serve the 14.0.2 POM and JAR (HTTP 200). [The rerun on the same candidate](https://github.com/lancedb/lancedb/actions/runs/40826169185/job/123300577435) downloaded the official stable artifact and passed all 37 Java tests with zero failures/errors/skips. The release remains blocked by the separate reader issue. Pin the LanceDB 0.41 release branch to the official Lance 14.0.2 packages. Keep the frozen product baseline at `45f987a29b6c12feb2b5fc613a6593e5f636a7ce` and the newer Lance dependency on main. All 15 Rust declarations use exact crates.io versions, Python uses the official stable wheel, and Java uses the stable Maven artifact. The vendored Lance Arrow and object_store Git overrides are removed. The LSM `BitmapAsBTree` change is limited to its test adapter and retains the physical-ID and query assertions. The Lance Arrow declaration remains inherited from the workspace. Validation: - The same source against official RC1 passed all 50 relevant PR checks, including 1,977 Rust core tests and the nullable-struct regression. - Stable source commit `40f15ed1b71b66950961b175829fdf17a763f108` differs from voted RC1 only by version substitutions in eight files. - `cargo fmt --all -- --check` and `cargo clippy --profile ci --features remote --tests --examples` passed against the official crates.io packages. - Locked Cargo metadata and normalized Cargo packaging passed; 995 unrelated Cargo lock entries were retained. - Six official PyPI wheel URLs/hashes are locked. Frozen environment installation, `lance` 14.0.2 import, and Ruff format/check passed. Unrelated UV packages and resolution markers are retained. - Full CI on the stable dependency commit is required before merge. Local Python SDK tests were not run because this validation environment has no installed LanceDB native extension; those tests run in PR CI. - Java checks run in CI; no local JDK is installed. Upstream release: https://github.com/lance-format/lance/releases/tag/v14.0.2 Remote materialized-view operations now use SQL and require matching server command support; older servers can reject describe/list operations. This known compatibility issue is outside the Lance patch scope and does not block this release, as agreed by the release manager. CI compatibility: derive the Lance checkout tag from an exact registry version when no Git tag is declared, and exercise local overrides for both crates.io and Git consumers. Both RC/stable tag readers and the exact stable downstream consumer metadata probe passed locally. This changes only the workflow; the 14.0.2 dependency/product code is unchanged. Release-hook correction: the dry run exposed Maven being set to `0.41.0-final.0`. Omit bump-my-version's optional `final` marker in the Java version hook so all SDKs publish as `0.41.0`. The actual hook passed beta.0, beta.17, final.0 and absent-prerelease probes; a fresh full stable dry run verifies Maven output before publication.
418 lines
15 KiB
YAML
418 lines
15 KiB
YAML
name: Rust
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
pull_request:
|
|
paths:
|
|
- Cargo.toml
|
|
- Cargo.lock
|
|
- rust-toolchain.toml
|
|
- deny.toml
|
|
- rust/**
|
|
- docs/web-tests/blob-v2/rs/**
|
|
- nodejs/Cargo.toml
|
|
- python/Cargo.toml
|
|
- .github/workflows/rust.yml
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
env:
|
|
# This env var is used by Swatinem/rust-cache@v2 for the cache
|
|
# key, so we set it to make sure it is always consistent.
|
|
CARGO_TERM_COLOR: always
|
|
RUST_BACKTRACE: "1"
|
|
|
|
jobs:
|
|
dependency-overrides:
|
|
name: Downstream Lance override
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 15
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
|
- name: Read the pinned Lance tag
|
|
id: lance
|
|
run: |
|
|
python3 - <<'PYTHON'
|
|
import os
|
|
import tomllib
|
|
from pathlib import Path
|
|
|
|
manifest = tomllib.loads(Path("Cargo.toml").read_text())
|
|
dependency = manifest["workspace"]["dependencies"]["lance"]
|
|
tag = dependency.get("tag") or f"v{dependency['version'].removeprefix('=')}"
|
|
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
|
|
output.write(f"tag={tag}\n")
|
|
PYTHON
|
|
- uses: actions/checkout@v6
|
|
with:
|
|
repository: lance-format/lance
|
|
ref: ${{ steps.lance.outputs.tag }}
|
|
path: local-lance
|
|
- name: Resolve a downstream workspace with a local Lance override
|
|
run: |
|
|
mkdir -p downstream/src
|
|
touch downstream/src/lib.rs
|
|
cat > downstream/Cargo.toml <<'TOML'
|
|
[package]
|
|
name = "lancedb-local-lance-consumer"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
|
|
[workspace]
|
|
|
|
[dependencies]
|
|
lancedb = { path = "../rust/lancedb", default-features = false }
|
|
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
|
|
|
|
[patch."https://github.com/lance-format/lance.git"]
|
|
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
|
|
|
|
[patch.crates-io]
|
|
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
|
|
TOML
|
|
cargo metadata --manifest-path downstream/Cargo.toml --format-version 1 > downstream/graph.json
|
|
python3 - <<'PYTHON'
|
|
import json
|
|
from pathlib import Path
|
|
|
|
graph = json.loads(Path("downstream/graph.json").read_text())
|
|
packages = [p for p in graph["packages"] if p["name"] == "lance-arrow"]
|
|
assert len(packages) == 1, [p["id"] for p in packages]
|
|
assert Path(packages[0]["manifest_path"]).resolve() == Path(
|
|
"local-lance/rust/lance-arrow/Cargo.toml"
|
|
).resolve()
|
|
PYTHON
|
|
|
|
lint:
|
|
timeout-minutes: 30
|
|
runs-on: ubuntu-24.04
|
|
defaults:
|
|
run:
|
|
shell: bash
|
|
env:
|
|
# Need up-to-date compilers for kernels
|
|
CC: clang-18
|
|
CXX: clang++-18
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
with:
|
|
fetch-depth: 0
|
|
lfs: true
|
|
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
|
with:
|
|
components: rustfmt, clippy
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Restore everywhere, but only save from main. Per-PR saves are
|
|
# unreadable outside their own branch anyway, since GitHub scopes
|
|
# caches to the creating ref.
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Install dependencies
|
|
run: |
|
|
sudo apt update
|
|
sudo apt install -y protobuf-compiler libssl-dev
|
|
- name: Run format
|
|
run: cargo fmt --all -- --check
|
|
- name: Run clippy
|
|
run: cargo clippy --profile ci --workspace --tests --all-features -- -D warnings
|
|
- name: Run clippy (without remote feature)
|
|
run: cargo clippy --profile ci --workspace --tests -- -D warnings
|
|
|
|
deny:
|
|
# Supply-chain checks: advisories, licenses, banned crates, and source
|
|
# restrictions. Configuration lives in `deny.toml` at the workspace root.
|
|
timeout-minutes: 10
|
|
runs-on: ubuntu-24.04
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- uses: EmbarkStudios/cargo-deny-action@v2
|
|
with:
|
|
command: check advisories bans licenses sources
|
|
|
|
build-no-lock:
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 30
|
|
env:
|
|
# Need up-to-date compilers for kernels
|
|
CC: clang
|
|
CXX: clang++
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
# Building without a lock file often requires the latest Rust version since downstream
|
|
# dependencies may have updated their minimum Rust version.
|
|
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
|
with:
|
|
toolchain: "stable"
|
|
# Remove cargo.lock to force a fresh build
|
|
- name: Remove Cargo.lock
|
|
run: rm -f Cargo.lock
|
|
- uses: rui314/setup-mold@v1
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Restore everywhere, but only save from main. Per-PR saves are
|
|
# unreadable outside their own branch anyway, since GitHub scopes
|
|
# caches to the creating ref.
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Install dependencies
|
|
run: |
|
|
sudo apt update
|
|
sudo apt install -y protobuf-compiler libssl-dev
|
|
- name: Build all
|
|
run: |
|
|
cargo build --profile ci --benches --all-features --tests
|
|
|
|
linux:
|
|
timeout-minutes: 60
|
|
# To build all features, we need more disk space than is available
|
|
# on the free OSS github runner. This is mostly due to the the
|
|
# sentence-transformers feature.
|
|
runs-on: ubuntu-2404-4x-x64
|
|
defaults:
|
|
run:
|
|
shell: bash
|
|
working-directory: rust
|
|
env:
|
|
# Need up-to-date compilers for kernels
|
|
CC: clang-18
|
|
CXX: clang++-18
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
with:
|
|
fetch-depth: 0
|
|
lfs: true
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Restore everywhere, but only save from main. Per-PR saves are
|
|
# unreadable outside their own branch anyway, since GitHub scopes
|
|
# caches to the creating ref.
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Install dependencies
|
|
run: |
|
|
sudo apt update
|
|
sudo apt install -y protobuf-compiler libssl-dev
|
|
- uses: rui314/setup-mold@v1
|
|
- name: Make Swap
|
|
run: |
|
|
swapfile=/swapfile
|
|
min_swap_bytes=$((15 * 1024 * 1024 * 1024))
|
|
active_swap_bytes="$(sudo swapon --show=NAME,SIZE --bytes --noheadings | awk '$1 == "/swapfile" { print $2 }')"
|
|
if [ -n "$active_swap_bytes" ]; then
|
|
if [ "$active_swap_bytes" -ge "$min_swap_bytes" ]; then
|
|
echo "/swapfile is already active with enough space; skipping swap creation"
|
|
exit 0
|
|
fi
|
|
echo "/swapfile is already active but smaller than 16G; using /mnt/lancedb-swapfile"
|
|
swapfile=/mnt/lancedb-swapfile
|
|
fi
|
|
if sudo swapon --show=NAME --noheadings | grep -Fxq "$swapfile"; then
|
|
echo "$swapfile is already active; skipping swap creation"
|
|
exit 0
|
|
fi
|
|
sudo rm -f "$swapfile"
|
|
sudo fallocate -l 16G "$swapfile"
|
|
sudo chmod 600 "$swapfile"
|
|
sudo mkswap "$swapfile"
|
|
sudo swapon "$swapfile"
|
|
- name: Build
|
|
run: cargo build --profile ci --all-features --tests --locked --examples
|
|
- name: Run feature tests
|
|
run: CARGO_ARGS="--profile ci" make -C ./lancedb feature-tests
|
|
- name: Run examples
|
|
run: cargo run --profile ci --all-features --example simple --locked
|
|
- name: Run blob v2 docs example
|
|
# Python and TypeScript blob docs tests install preview packages.
|
|
# Those builds are not guaranteed to remain available for more than
|
|
# 6 months, so they stay a manual run. This example uses the workspace crate.
|
|
run: cargo run --profile ci --all-features -p lancedb-blob-v2-docs --example multimodal --locked
|
|
|
|
remote:
|
|
timeout-minutes: 30
|
|
# Running this requires access to secrets, so skip if this is a PR from a
|
|
# fork. Keep it separate from the all-features build so Cargo does not
|
|
# retain both dependency graphs in one target directory.
|
|
if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
|
|
runs-on: ubuntu-2404-4x-x64
|
|
defaults:
|
|
run:
|
|
shell: bash
|
|
working-directory: rust
|
|
env:
|
|
CC: clang-18
|
|
CXX: clang++-18
|
|
GH_TOKEN: ${{ github.token }}
|
|
SOPHON_READ_TOKEN: ${{ secrets.SOPHON_READ_TOKEN }}
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
with:
|
|
fetch-depth: 0
|
|
lfs: true
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Remote tests use a different feature graph from the main Linux
|
|
# job. Cache downloads, but build into a fresh target directory.
|
|
cache-targets: false
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Install dependencies
|
|
run: |
|
|
sudo apt update
|
|
sudo apt install -y protobuf-compiler libssl-dev
|
|
- uses: rui314/setup-mold@v1
|
|
- name: Test the default local Functions gate
|
|
run: cargo test --profile ci -p lancedb --features remote --test experimental_functions --locked
|
|
- name: Run remote tests
|
|
run: CARGO_ARGS="--profile ci" make -C ./lancedb remote-tests
|
|
|
|
macos:
|
|
timeout-minutes: 60
|
|
strategy:
|
|
matrix:
|
|
mac-runner: ["macos-14", "macos-15"]
|
|
runs-on: "${{ matrix.mac-runner }}"
|
|
defaults:
|
|
run:
|
|
shell: bash
|
|
working-directory: rust
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
with:
|
|
fetch-depth: 0
|
|
lfs: true
|
|
- name: CPU features
|
|
run: sysctl -a | grep cpu
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Restore everywhere, but only save from main. Per-PR saves are
|
|
# unreadable outside their own branch anyway, since GitHub scopes
|
|
# caches to the creating ref.
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Install dependencies
|
|
run: brew install protobuf
|
|
- name: Run tests
|
|
run: |
|
|
# Don't run the s3 integration tests since docker isn't available
|
|
# on this image.
|
|
ALL_FEATURES=`cargo metadata --format-version=1 --no-deps \
|
|
| jq -r '.packages[] | .features | keys | .[]' \
|
|
| grep -v s3-test | sort | uniq | paste -s -d "," -`
|
|
# Run doctests before test binaries fill the runner disk. Examples are
|
|
# already built by the Linux job, so avoid retaining them here.
|
|
cargo test --profile ci --features $ALL_FEATURES --locked --doc
|
|
cargo test --profile ci --features $ALL_FEATURES --locked --lib --tests
|
|
|
|
windows:
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- target: x86_64-pc-windows-msvc
|
|
runner: windows-2022
|
|
runs-on: ${{ matrix.runner }}
|
|
defaults:
|
|
run:
|
|
working-directory: rust/lancedb
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- name: Set target
|
|
run: rustup target add ${{ matrix.target }}
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Restore everywhere, but only save from main. Per-PR saves are
|
|
# unreadable outside their own branch anyway, since GitHub scopes
|
|
# caches to the creating ref.
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Install Protoc v21.12
|
|
run: choco install --no-progress protoc
|
|
- name: Build
|
|
run: |
|
|
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
|
|
cargo build --profile ci --features aws,remote --tests --locked --target ${{ matrix.target }}
|
|
- name: Run tests
|
|
run: |
|
|
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
|
|
# `--target` has to match the build step above. Without it cargo uses
|
|
# target/ci/ rather than target/<triple>/ci/ and rebuilds the entire
|
|
# dependency graph a second time.
|
|
cargo test --profile ci --features aws,remote --locked --target ${{ matrix.target }}
|
|
|
|
# Run the aarch64 Windows tests only on main so PRs wait for a single
|
|
# Windows platform. windows-11-arm is a standard runner, so it is free on
|
|
# public repos. Running natively lets the aarch64 tests actually execute --
|
|
# this job used to cross-compile them and then skip the test step, paying
|
|
# full codegen and link cost for a compile check.
|
|
windows-arm:
|
|
# Keep the historical name so the check shows up as before on main.
|
|
name: windows (aarch64-pc-windows-msvc, windows-11-arm)
|
|
if: github.ref == 'refs/heads/main'
|
|
runs-on: windows-11-arm
|
|
defaults:
|
|
run:
|
|
working-directory: rust/lancedb
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- name: Set target
|
|
run: rustup target add aarch64-pc-windows-msvc
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Restore everywhere, but only save from main. Per-PR saves are
|
|
# unreadable outside their own branch anyway, since GitHub scopes
|
|
# caches to the creating ref.
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Install Protoc v21.12
|
|
run: choco install --no-progress protoc
|
|
- name: Build
|
|
run: |
|
|
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
|
|
cargo build --profile ci --features aws,remote --tests --locked --target aarch64-pc-windows-msvc
|
|
- name: Run tests
|
|
run: |
|
|
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
|
|
# `--target` has to match the build step above. Without it cargo uses
|
|
# target/ci/ rather than target/<triple>/ci/ and rebuilds the entire
|
|
# dependency graph a second time.
|
|
cargo test --profile ci --features aws,remote --locked --target aarch64-pc-windows-msvc
|
|
|
|
msrv:
|
|
# Check the minimum supported Rust version
|
|
name: MSRV Check - Rust v${{ matrix.msrv }}
|
|
runs-on: ubuntu-24.04
|
|
strategy:
|
|
matrix:
|
|
msrv: ["1.94.0"] # This should match up with rust-version in Cargo.toml
|
|
env:
|
|
# Need up-to-date compilers for kernels
|
|
CC: clang-18
|
|
CXX: clang++-18
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
with:
|
|
submodules: true
|
|
- name: Install dependencies
|
|
run: |
|
|
sudo apt update
|
|
sudo apt install -y protobuf-compiler libssl-dev
|
|
- name: Install ${{ matrix.msrv }}
|
|
uses: dtolnay/rust-toolchain@master
|
|
with:
|
|
toolchain: ${{ matrix.msrv }}
|
|
- uses: Swatinem/rust-cache@v2
|
|
with:
|
|
# Restore everywhere, but only save from main. Per-PR saves are
|
|
# unreadable outside their own branch anyway, since GitHub scopes
|
|
# caches to the creating ref.
|
|
save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
- name: Downgrade dependencies
|
|
# aws-smithy-types 1.6.1+ requires rustc 1.94.1
|
|
run: cargo update -p aws-smithy-types --precise 1.5.0
|
|
- name: cargo +${{ matrix.msrv }} check
|
|
env:
|
|
RUSTUP_TOOLCHAIN: ${{ matrix.msrv }}
|
|
run: cargo check --profile ci --workspace --tests --benches --all-features
|