Short-lived Python processes using this client can occasionally crash
with SIGABRT during interpreter shutdown, even after every operation
they ran completed successfully. The cause is the shared Tokio runtime
backing every async call: it's never told to shut down at normal process
exit, only reset (and deliberately leaked) on `fork()`. Its worker
threads keep running, uncoordinated with the interpreter, until the
process actually ends, and if one is mid-task exactly as `Py_Finalize`
starts tearing down interpreter state, it can panic on state that's
already gone. That panic happens on a background thread with no
PyO3-wrapped call frame to catch it, so Rust aborts the whole process
instead of just failing that one call. This PR gives the runtime a
coordinated, bounded shutdown by registering a Python `atexit` callback
that runs while the interpreter is still fully valid.
Getting the exit lifecycle right took a few rounds of review. Earlier
versions freed the runtime as soon as `Arc::strong_count` looked low,
but that's the wrong signal — it reflects who currently holds a
reference, not who's logically still in flight. That mistake showed up
three ways: a caller could dereference memory already freed out from
under it; an install already in progress could finish invisibly after
`shutdown()` had already decided there was nothing to do; and a spawned
task could end up as the final owner of the `Runtime`, so completing it
dropped the runtime from inside one of its own worker threads, which
Tokio itself forbids and panics on (this reproduced unprompted in this
branch's own test suite). Fixing all three meant replacing
reference-count-based tracking with an explicit counter of in-flight
top-level calls that `shutdown()` waits on directly.
This was accomplished with the following changes:
- The runtime lives in an `ArcSwapOption<Tagged>`, where `Tagged` pairs
the `Runtime` with the fork generation it was built in.
- An `OUTSTANDING` counter, incremented before a top-level
`spawn`/`spawn_blocking`/`block_on` call does anything else and
decremented only once it has truly finished (via an `OutstandingGuard`
token that carries no reference to the runtime), is what `shutdown()`
waits on — not `Arc::strong_count` or whether the slot looks empty. This
closes the install-race and makes it impossible for a task's own
completion to be the runtime's final drop.
- Once `shutdown()`'s bound elapses, it stops waiting and attempts
retirement anyway, rather than returning with the runtime and its
workers left fully alive.
- `spawn`/`spawn_blocking` use `Handle::try_current()` to pin any nested
spawn (`future_into_py` spawns a task that itself spawns a second one
for the real work) to whichever runtime is already executing it, so a
reclaim landing between the two calls can't split one logical operation
across two different runtime instances.
- The fork-child handler now only bumps a bare `GENERATION` counter — no
`ArcSwapOption` call of any kind from that context, since
`swap`/`compare_and_swap` do real reader-reconciliation work
(thread-local state, potentially an allocation) that isn't safe in a
forked child. `get_runtime()` compares its installed runtime's
generation against the live counter from ordinary context and rebuilds
on a mismatch.
- Registered `shutdown_runtime` as a Python `atexit` callback in the
`_lancedb` module init, running with the GIL released (`Python::detach`)
since the bounded wait could otherwise deadlock against any in-flight
task that itself needs the GIL.
### Testing
- Unit tests in `runtime.rs` cover: shutdown with no runtime created,
shutdown after use and lazy rebuild afterward, calling shutdown twice in
a row, a concurrent stress test racing many threads against shutdown, a
nested-spawn test reproducing `future_into_py`'s own
spawn-within-a-spawn shape under concurrent shutdown, a test confirming
a top-level task in flight survives a concurrent shutdown reclaim, and a
test forcing the install-vs-shutdown race directly.
- Built the wheel and ran a concurrent reproducer (many threads
hammering the client while `atexit` fires) over 100 times with no hangs
or crashes, plus a 30-second-join variant and repeated runs of a
short-lived process confirming clean exits with no added latency.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
Add SQL execution to remote LanceDB connections. On the standard
synchronous connection, `execute_query` waits for the initial result
stream and returns its Arrow reader. `execute_query_async` is called
without Python `await` and immediately returns a query handle for status
inspection, streaming, or cancellation. Local databases report that SQL
is not supported.
The transport and query lifecycle live in Rust. Python exposes
native-backed synchronous and asynchronous connection methods and query
wrappers; it does not use PyArrow's Flight client.
## User experience
The standard synchronous connection supports both direct reads and
background query execution:
```python
db = lancedb.connect(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
# Direct execution waits only until the initial result stream is available.
# Later batches continue streaming as the query progresses.
reader = db.execute_query(
"SELECT * FROM events",
default_namespace_path=["production"],
)
for batch in reader:
print(batch.num_rows)
# Background execution returns a query handle immediately. Despite the
# `_async` suffix, no Python `await` is needed on a synchronous connection.
query = db.execute_query_async("SELECT * FROM events")
print(query.id)
description = db.describe_query(query.id)
print(description.status)
print(description.progress)
print(description.expires_at)
# Start reading as soon as the service advertises partial results. The reader
# continues polling and yields newly available record batches until the query
# and all result endpoints are complete.
reader = query.reader()
for batch in reader:
print(batch.num_rows)
# Or cancel a different still-running query. Its status becomes "cancelling"
# while the server is still working, then "cancelled" once confirmed.
cancelled_query = db.execute_query_async("SELECT * FROM large_events")
cancelled_query.cancel()
```
The less commonly used asynchronous connection exposes the same
operations as coroutines:
```python
async_db = await lancedb.connect_async(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
query = await async_db.execute_query_async("SELECT * FROM events")
async for batch in await query.reader():
print(batch.num_rows)
```
The UUIDv7 query id is scoped to the connection that submitted it. The
connection retains lightweight shared query state used by
`query.describe()` and `db.describe_query(query.id)`; the id does not
encode SQL or a Flight continuation token and is not a cross-connection
resume token. Abandoned state has bounded retention, and terminal state
remains available briefly.
Unqualified table names use the connected database and the `public`
namespace by default. `default_namespace_path` accepts a list such as
`["production", "events"]`. SQL can still use qualified names to
reference other databases and namespaces available to the deployment.
## Design
- Uses Arrow Flight `PollFlightInfo` for submission and long polling,
`DoGet` for results, and `CancelFlightInfo` for cancellation. Each
`PollInfo.info` is treated as the cumulative set of currently available
endpoints, so advertised tickets are consumed once and batches can be
delivered before execution is complete.
- Serializes result completion and cancellation into one lifecycle. A
server-accepted request reports `cancelling` and wakes blocked
status/result work; a later retry can confirm `cancelled`. Result
retrieval is rejected after cancellation is accepted, while cancellation
after a result was already delivered is a no-op.
- Assigns a time-ordered UUIDv7 connection-scoped query id and retains
only shared evolving lifecycle state, keeping SQL, Flight continuation
tokens, and Arrow result data out of public ids and the registry.
- Leaves admission control to the server while honoring server
expiration and a local fallback retention window for abandoned entries.
- Retains terminal ids for five minutes so they remain available for
connection-level description.
- Keeps one lazily initialized SQL client on each remote database
connection and attaches fresh authentication, routing, namespace, and
request metadata to every operation.
- Applies the configured overall timeout to each execution, description,
reader, and cancellation operation. A result reader carries one absolute
deadline from `reader()` through the end of streaming; connect and read
timeouts continue to bound their individual phases.
- Returns a bounded, backpressured, single-consumer Arrow stream rather
than collecting the full result in memory. Dropping the reader stops
downloading but does not implicitly cancel the server query.
- Preserves typed schemas for empty result sets through the stream
schema.
- Accepts Flight result messages up to 1 GiB so a valid row containing a
large blob, string, or vector is not rejected by tonic's 4 MiB default
receive limit.
- Supports the Python client first while keeping the authoritative
implementation in the Rust core.
## Summary
- expose `memory_limit` and `num_workers` on the Python FTS
configuration for local builds
- forward both build-only settings to the Lance inverted-index builder
- add an end-to-end regression proving the configured memory budget
reaches the native build
## Root cause
LanceDB 0.26.1 pinned Lance 1.0.1. That Lance version used an FTS
partition-merge path whose retained data made memory grow with merge
progress on very large indexes. Upstream Lance
[#5754](https://github.com/lance-format/lance/pull/5754) changed
partition merging to stream its inputs, reducing peak memory by about
25%. Lance [#6174](https://github.com/lance-format/lance/pull/6174) then
removed the old merge phase, compressed posting lists during
construction, reduced indexing memory by about 60%, and introduced a
total build `memory_limit` for bounded workers.
Current `main` pins Lance 11.0.0-beta.3, which contains those
architectural fixes. This PR does not duplicate or claim the upstream
leak fix; it addresses the remaining Python API gap.
## This repair
LanceDB Python did not expose the native FTS builder resource controls.
`memory_limit` now sets the total local-build budget in MiB, divided
among effective workers, and `num_workers` controls build parallelism.
Both are build-only settings and do not affect remote builds or
persisted index configuration.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo fmt --all`
- `uv run --project python --extra tests --extra dev ruff check .`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/index.py python/python/tests/test_fts.py`
- `uv run --project python --extra tests pytest python/tests/test_fts.py
-q` (51 passed)
Fixes#2923
<!-- lance-gatekeeper-fix:v1 agent=a1ceedf74531e0212cb6f1ebf9390a26
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`cargo deny` did not check crate-level dependency declarations against
`[workspace.dependencies]`, so a crate used by both the core crate and
the bindings could be declared independently in each one and drift. For
example `tokio` was pinned at `1.23` in `rust/lancedb` and `1.40` in
`python`, and `pin-project` at `1.0.7` in the workspace table but
`1.1.5` in `python`.
This PR turns on cargo-deny's `bans.workspace-dependencies` lint, which
fails when a dependency is used by more than one member without going
through `workspace = true`, and when a `[workspace.dependencies]` entry
is used by nobody.
Enabling it surfaced 12 violations. Fixing them means adding `bytes`,
`lancedb`, `serde`, `serde_json`, `tempfile`, `tokio`, and `uuid` to
`[workspace.dependencies]`, and pointing the `arrow`, `arrow-buffer`,
`async-trait`, `chrono`, and `pin-project` declarations at the entries
that already existed. `Cargo.lock` is unchanged, so resolution is the
same as before.
The shared `chrono` entry now carries `default-features = false,
features = ["clock"]`, matching what `nodejs` and `python` already asked
for — cargo ignores a member's `default-features = false` unless the
workspace entry sets it too. On the targets we build, `clock` covers
everything `rust/lancedb` was getting from chrono's defaults.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- align the PyO3 runtime and build ABI floor with the declared Python
3.10 minimum
- add a regression test that keeps both ABI features synchronized with
`requires-python`
## Root cause
The Python 3.10 support-floor update originally changed PyO3 to
`abi3-py310`, but a later dependency update reverted both PyO3 features
to `abi3-py39`. Published Windows wheels were consequently tagged
`cp39-abi3` while importing `PyCMethod_New`, a stable-ABI procedure
absent from CPython 3.9.0 and 3.9.1. Windows reports that mismatch as
“The specified procedure could not be found” while loading `_lancedb`.
Restoring `abi3-py310` makes the wheel tag and native imports agree with
the package metadata and prevents future wheels from advertising
unsupported Python 3.9 compatibility.
## Validation
- `uv run --extra tests pytest python/tests/test_package_metadata.py -q`
- `uv run --extra tests --extra dev ruff format --check .`
- `uv run --extra tests --extra dev ruff check .`
- `cargo fmt --all`
- `cargo check --quiet -p lancedb-python`
- `uvx --from maturin==1.12.4 maturin build --profile ci` (built
`lancedb-0.37.1b0-cp310-abi3-manylinux_2_34_x86_64.whl`)
Fixes#2051
<!-- lance-gatekeeper-fix:v1 agent=d66c984498190d2207d1c5126cba5047
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Gatefixer
Lance Release
Bruno RamirezandClaude Sonnet 5
Jack Ye
Will JonesandClaude Opus 5