Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge /
Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import
lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled
function. Per [westonpace's
review](https://github.com/lancedb/lancedb/issues/3324#issuecomment-4328944354),
the default `lancedb` wheel stays fast; pre-Haswell users get a
separately-published `lancedb-compat` wheel.
## Summary
- Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds
with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline).
Same Python API (`import lancedb` works) — files install to the same
namespace, so the two wheels conflict at install time and users pick
one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` /
`tensorflow-cpu`.
- Generalizes `build_linux_wheel` and `upload_wheel` composites with
optional `package-name` and `rustflags` inputs (defaults preserve the
existing 4 `lancedb` matrix entries verbatim).
- Documents the choice in `python/README.md`: `pip install
lancedb-compat` for pre-Haswell hosts.
The default `.cargo/config.toml` baseline is unchanged.
## Sequencing
1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in
lance.~~ **Done — merged.**
2. lancedb's lance dep is bumped to a release that includes it (separate
PR / normal cadence).
3. This PR's `lancedb-compat` wheel build path starts producing a wheel
that runs on pre-Haswell hardware. **Maintainer setup**: register
`lancedb-compat` on PyPI and configure trusted publishing.
## Verified end-to-end on Sandy Bridge Xeon E5-2609
Verification was done locally against a fork-pinned lance dep that
includes the runtime dispatch implementation, using the same
`RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI:
```
$ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
$ pip install ./target/wheels/lancedb-*.whl
$ python verify.py
PASS: import + simd dispatch + table create + vector search all work.
```
Pre-fix on the same CPU (default `pip install lancedb`): `Illegal
instruction (core dumped)`. Full reproducer (deps + clone + build +
verification):
https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df.
Fork-internal verification PR with the dep bump and full logs:
[`tobocop2/lancedb#2`](https://github.com/tobocop2/lancedb/pull/2).
## Benchmarks — no regressions on modern CPUs from the lance-side change
These are the numbers I ran for the lance PR, confirming the runtime
dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that
existing users install. Criterion, one machine, one session, base → PR,
no `RUSTFLAGS` override. Full methodology, null experiments, and logs:
[lance-format/lance#6630 benchmark
comment](https://github.com/lance-format/lance/pull/6630#issuecomment-4933063394)
and the [logs
gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2).
| benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade
Lake (`avx512f`) |
|---|---|---|
| `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% |
| `Cosine(f64, scalar)` | −0.34% | −1.94% |
| `Cosine(u8, SIMD)` | +2.30% | +3.63% |
| `Dot(f16, SIMD)` | −0.58% | +0.61% |
| `Dot(f32, SIMD)` | +0.34% | **−6.08%** |
| `Dot(f32, arrow_arity)` | +0.02% | −0.00% |
| `L2(f32, scalar)` | −0.10% | −0.02% |
| `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% |
| **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** |
| `L2(u8, SIMD)` | +0.42% | −3.11% |
| `NormL2(f32, SIMD)` | −1.02% | −4.17% |
| `NormL2(f64, SIMD)` | +3.51% | −0.58% |
Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector
width — improves 25–46%.
---
To be transparent: this isn't my domain of expertise and the lance-side
implementation is AI-generated. I verified it works end-to-end on the
failing hardware. Happy to roll in feedback.
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
This patch mitigates template injection vulnerabilities in GitHub
Workflows by replacing direct references with an environment variable.
Aikido used AI to generate this PR.
High confidence: Aikido has a robust set of benchmarks for similar
fixes, and they are proven to be effective.
Co-authored-by: aikido-autofix[bot] <119856028+aikido-autofix[bot]@users.noreply.github.com>
Based on https://github.com/lancedb/lance/pull/4984
1. Bump to 1.0.0-beta.2
2. Use DirectoryNamespace in lance to perform all testing in python and
rust for much better coverage
3. Refactor `ListingDatabase` to be able to accept location and
namespace. This is because we have to leverage listing database (local
lancedb connection) for using namespace, namespace only resolves the
location and storage options but we don't want to bind all the way to
rust since user will plug-in namespace from python side. And thus
`ListingDatabase` needs to be able to accept location and namespace that
are created from namespace connection.
4. For credentials vending, we also pass storage options provider all
the way to rust layer, and the rust layer calls back to the python
function to fetch next storage option. This is exactly the same thing we
did in pylance.
`object_store` already hard codes `rustls` as the TLS implementation, so
we have been shipping a mix of `rustls` and `openssl`. For simplicity of
builds, we should consolidate to one, and that has to be `rustls`.
This changes `lancedb` from a "pure python" setuptools project to a
maturin project and adds a rust lancedb dependency.
The async python client is extremely minimal (only `connect` and
`Connection.table_names` are supported). The purpose of this PR is to
get the infrastructure in place for building out the rest of the async
client.
Although this is not technically a breaking change (no APIs are
changing) it is still a considerable change in the way the wheels are
built because they now include the native shared library.
Tobias
Xuanwo
aikido-autofix[bot]
Will Jones
Lei Xu
Jack Ye
Weston Pace