mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-03 12:08:52 +00:00
Will Jones
84f46df876
The nightly `NPM Publish` run has failed every night since at least Aug 23, always on the same two legs: `aarch64-unknown-linux-gnu` and `aarch64-unknown-linux-musl`. The other five targets pass. rustc is OOM-killed during the fat-LTO codegen of the cdylib — `signal: 9` with no diagnostic, about 27 minutes in — and on the musl leg that takes the whole runner down with `The runner has received a shutdown signal`. Both legs now pass: | leg | before | peak memory | wall time | | --- | --- | --- | --- | | `aarch64-unknown-linux-gnu` | OOM-killed at ~27 min | 31391 → 22851 MiB | 38m43s → 22m04s | | `aarch64-unknown-linux-musl` | runner killed at ~28 min | >32 GiB → 16516 MiB | ~40 min → 20m50s | **ThinLTO** is most of that. Fat LTO is single-threaded, and its peak is consumed inside rustc's LLVM before any linker process is spawned — which is why it is the whole fix on musl, and why lld alone left the gnu leg still peaking at 31391 MiB against the runner's 32 GiB. Both legs now use the `lto: thin` / `codegen_units: 16` settings that darwin and both Windows legs already use, at a cost of a few percent runtime performance. **lld** covers the rest, on the gnu leg. arm64 Linux otherwise links through GNU `ld` where x86_64 already defaults to `rust-lld`, which is why only the arm64 legs hit this at all; on a comparable arm64 build (`lancedb/sophon#7313`) it cut the largest single linker process from 7.0 to 4.0 GiB and wall time by 35%. The flags live in a small wrapper script used as the linker rather than in `-C link-arg`, because the per-target rustflags variable does not reach every unit that links: dependency crates linking a dylib (`crc-fast`, `lance-arrow`) were invoked as bare `clang`, which targets the x86_64 host and fails with `Relocations in generic ELF (EM: 183)`. Separately, and affecting five legs rather than two: the three ThinLTO targets exported `CARGO_PROFILE_RELEASE_LTO` and `CARGO_PROFILE_RELEASE_CODEGEN_UNITS` from `pre_build`, which runs inside the build step — after the cache step. `Swatinem/rust-cache` computes its key when the action runs, before any step, so step-local values are invisible to it. The result is a loop that never converges: the key never changes, so restores are exact hits, an exact hit makes the post-run save a no-op, and cargo invalidates the restored artifacts anyway because the flags differ. Those legs have been rebuilding cold on every run. Both values move to job-level `env:` ahead of the cache step, driven by new `lto:`/`codegen_units:` matrix fields, and are forwarded into the containers with `-e` since `docker run` inherits nothing. Every leg's cache key shifts once as a result, so expect one cold rebuild. A `Report peak memory` step is added so whether these legs fit is a number rather than an inference from whether the runner survived. It produced the figures above. ## Not included Moving these legs to native arm64 runners. It would retire the zig cross path, the `AT_HWCAP2` workaround and the `TARGET_CC` override, and arm64 runners are billed roughly 37% below x64 at equal core count — but the `lts-debian-aarch64` image exists to link against the manylinux2014 sysroot's glibc 2.17, and building natively on ubuntu-24.04 would raise the minimum glibc for every published aarch64 binary. That is a user-facing decision, not a CI cleanup. Dropping these legs to smaller runners, which is where the real cost saving is — larger runners are billed even on public repos. On these numbers it is not available yet: musl at 16516 MiB is about 130 MiB over what a 16 GB standard runner has. Worth revisiting as a follow-up. ## Testing Cargo's rustflags precedence was checked locally rather than taken from the docs, since getting it wrong would silently change the published binaries. With a throwaway crate carrying both a `target.'cfg(all())'` and a per-target rustflags table: setting `RUSTFLAGS` discards both, and setting it to the empty string discards them too. That rules out routing the linker flag through a job-level `RUSTFLAGS`, because `env:` keys cannot be conditionally omitted and every other leg would then silently lose the `target-cpu`/`target-feature` settings in `.cargo/config.toml` — `+avx2` on x86_64 and `-crt-static` on aarch64-musl. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Multimodal AI Lakehouse
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- Open Source & Local: 100% open source, runs locally or in your cloud. No vendor lock-in.
- Cloud and Enterprise: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us