lancedb 0.40.0, the latest release, does not include #4529: its query builders' to_arrow() and to_list() take no blob_mode, and a vector, FTS, hybrid or ordered query raises when its projection includes a blob column. Restore the earlier guidance and example, with the snippet regenerated from the restored test. On 0.40.0 "descriptions" works on in-memory tables and "bytes" does not, so the restored note names only "bytes".
Sophon v1.8.1 runs an index build as reindex_<algorithm> unless the request is create-only, which runs as create_<algorithm>, and tears a dropped table down as drop_table or drop_object depending on how the deployment stores its catalog. Say so rather than naming a single type.
A plain ADD COLUMN, a primary key on a nullable column, bucket sharding without a column, CREATE SCHEMA public and a three-part schema path are all rejected. Show forms that run, say what bucket sharding needs, and note that a new index is built in the background before it can be dropped.
## Summary
The Python wheel builds used different protoc versions: v24.4 on Linux,
v21.12 on Windows, and whatever Homebrew shipped on macOS. None of them
verified the download. A new composite action,
`.github/actions/setup-protoc`, now holds one pin, v36.2, with a SHA-256
for each platform's zip.
- macOS and Windows: the action downloads protoc, checks its hash, and
installs it.
- Linux: protoc has to be installed inside the manylinux container, so
the build reads the version and hash from the action's outputs and
checks the hash there.
protobuf does not publish checksum files for protoc. The pinned hashes
are the digests GitHub recorded for the v36.2 release assets. Older
releases such as v24.4 have none, which is why this moves to v36.2.
protoc only runs at build time, through `prost-build` in the Lance
crates. The Linux protoc binary is statically linked, so it runs in the
manylinux 2_17 container.
The action is also added to the path triggers of `pypi-publish.yml` and
`python.yml`, so a change to the pin runs the wheel builds.
Follow-up to
https://github.com/lancedb/lancedb/pull/4522#issuecomment-6160848344.
The sdist job in that PR can use the same action once this merges.
## Evidence
Running the action's install script locally on linux-aarch_64 installs
`libprotoc 36.2`, and a wrong hash fails with `protoc checksum
mismatch`. `cargo check -p lancedb` with `PROTOC` set to protoc 36.2
builds the Lance protos. The PyPI Publish dry run on this PR covers the
Linux, macOS and Windows builds.
## Merge Risk
**Door:** Two-way. CI only.
**Blast Radius:** Python wheel builds.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The quickstart says the TypeScript and Rust code of steps 2 to 5 makes one
program, whichever connection the reader picks. The object-storage examples
broke that: the TypeScript one defined an uncalled function whose db was
local to it and had no import, and the Rust one declared uri but never
connected. Both now bind db in the program's scope, as the local and
Enterprise examples do, and the quickstart shows their import lines. The test
functions that hold them stay uncalled, so no test reaches object storage. The
connection guide shows the TypeScript example with its import line too.
Cleanup keeps the files a branch still reads, and the version the branch was
forked from only while the branch's remaining versions read the parent's
files. Once the branch has replaced the data it inherited and its own older
versions are pruned, cleanup removes the fork version like any other, so the
cleanup and guarantees pages no longer list it among the versions cleanup
always keeps. Both point readers who need an old version recoverable to a tag,
under the tag policy the pages already describe, or a wide enough window.
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
WAL queries have the ability to bypass WAL by using `use_lsm(bool)` to
determine whether to explicitly use or not use it. This is exposed on
many of the entry-points but not on the `BaseTableAdapter`. This PR adds
that function to allow WAL use configuration.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
## Why
lance-format/lance#9803 makes the MemWAL's memtable index kinds
pluggable: a `MemIndexRegistry` on the writer config decides which
base-table index kinds are maintained in memory, so a kind Lance does
not build in can be added by registering a plugin. LanceDB opens the
MemWAL writer itself for `merge_insert`, and checks a named maintained
set itself, both against Lance's built-ins only. So a kind registered
outside Lance could not be named in
`LsmWriteSpec::with_maintained_indexes`, and under "maintain every
index" it was skipped.
## What changes
- `NativeTable::set_mem_index_registry(registry)` sets the memtable
index kinds a table maintains. The default is Lance's built-ins, so
nothing changes unless a caller sets one. The registry checks a named
maintained set (`validate_maintained_indexes_with`), is recorded on
`initialize_mem_wal`, and goes on the writer config `merge_insert`
opens. `lancedb::table` re-exports `MemIndexRegistry` and Lance's plugin
traits (`mem_index`), so a caller needs no direct `lance` dependency.
- The LSM vector read no longer checks resident memtables at all, and
the `carries_vector_index` helper goes with it. Lance already compares
every row of a memtable whose index can't answer the search, so no rows
are missed. It still requires the base vector index to be maintained.
## Limits
- The registry belongs to the table handle. Clones of a handle and its
branch handles share it; a handle opened separately starts from the
built-ins. Set it before the handle's first LSM write: a writer already
open keeps the kinds it opened with.
- The maintained set is stored with the table but the registry is not.
Once a spec names an index only a registered plugin maintains, an LSM
write from a handle without that registry is refused with Lance's "no
registered plugin maintains" error rather than skipping the index. That
includes the Python and Node bindings, which have no way to set a
registry yet.
- Rust only. A registry holds Rust plugin objects, which the Python,
TypeScript and Java bindings cannot construct, so they get no setter;
their documented behaviour (built-in kinds only) is unchanged.
- Per-index plugin settings (`ShardWriterConfig::index_overrides`, from
lance-format/lance#9805) are not exposed; writers use each plugin's
defaults.
## Open decision
Whether the registry should also be settable on the connection, so every
table a connection opens carries it. This PR keeps it per table; a
connection-level option would sit on top of `set_mem_index_registry`
without changing it.
## Depends on
Lance v14.0.0-beta.13, which `main` already uses. It includes
lance-format/lance#9803 and #9805 (pluggable memtable index kinds) and
#9857 (flushed generations keep the table's field ids).
## Testing
- `lsm_maintains_an_index_kind_from_the_tables_registry`: a stand-in
plugin maintains the base table's bitmap index as a B-tree. Naming the
index is refused without the registry and accepted with it; a separately
opened handle without the registry is refused an LSM write. Rows written
through `merge_insert` are found by id before and after the flush. Each
generation the shard manifests publish carries a bitmap index on
`region`, and a filter on it is answered through that index.
- `lsm_default_set_maintains_a_registry_kind_created_while_writing`:
under "maintain every index", a bitmap index created after the writer
opened is picked up by that writer and flushed. A clone of the table
handle shares the registry.
- `lsm_vector_search_compares_every_row_when_no_resident_index_answers`:
with a vector plugin that answers no search, LSM vector search still
returns the right rows, including a second page.
- `cargo test -p lancedb --features remote --lib` passes; `cargo clippy
-p lancedb --features remote --tests -- -D warnings` and `cargo fmt
--all -- --check` are clean.
---------
Co-authored-by: XYZhan <xuanyu.zhan@lancedb.com>
Resolve the docs Rust example lockfile: keep lance-namespace 8.0.0 and
the remote client's windows-result 0.4.1 and windows-strings 0.5.1, which
main's dependency update dropped, and take main's other updated versions.
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v14.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.11).
Adapts the LSM resident-index guards to Lance's query-based index API
and adds regression coverage for index presence and type checks.
Validation: workspace Clippy with all features and warnings denied, Rust
formatting, and targeted LSM query unit tests.
test_function_enterprise ran against a deployment whenever LANCEDB_URI,
LANCEDB_HOST_OVERRIDE and LANCEDB_API_KEY were set. Like the quickstart
tests, it now also needs LANCEDB_DOCS_TEST_DEPLOYMENT=1, and fails
rather than skips when that is set without the settings.
New tests drive it with lancedb.connect stubbed: without the exact
opt-in it skips before connecting, with the opt-in but missing settings
it fails naming them, and with both it reaches the connection. The
examples are unchanged.
## Summary
The Java SDK pinned `jackson-databind` 2.17.1, and `jackson-core` and
the other Jackson modules came in transitively via `arrow-vector` and
`lance-namespace-apache-client` at 2.16.x–2.17.x, all affected by known
CVEs.
This imports `jackson-bom` 2.18.11 in the parent POM, and declares
`jackson-core`, `jackson-annotations`, `jackson-datatype-jsr310`, and
`jackson-jaxrs-json-provider` as direct (BOM-managed) dependencies of
`lancedb-core`. The direct declarations are needed because the parent's
dependency management doesn't apply to a consumer's transitive
resolution; without them, Maven's nearest-wins mediation still hands
downstream apps the older versions from `arrow-vector`.
## Evidence
`mvn dependency:list` in a separate consumer project depending only on
`lancedb-core`:
| Artifact | Before | After |
|---|---|---|
| jackson-core | 2.16.0 | 2.18.11 |
| jackson-annotations | 2.16.0 | 2.18.11 |
| jackson-databind | 2.17.1 | 2.18.11 |
| jackson-datatype-jsr310 | 2.16.0 | 2.18.11 |
| jackson-jaxrs-base / json-provider | 2.17.1 | 2.18.11 |
| jackson-module-jaxb-annotations | 2.17.1 | 2.18.11 |
## Merge Risk
**Door:** Two-way. Dependency versions only, and they stay within
Jackson 2.x.
**Blast Radius:** `java` dependencies. Downstream users of
`lancedb-core` now get Jackson 2.18.11 transitively.
Fixes OSS-2360, OSS-2337
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Fixes the Dependabot alerts for Rust crates that have a patched release.
- `Cargo.lock`: aws-smithy-json 0.62.5 → 0.62.7, quinn-proto 0.11.14 →
0.11.19, xxhash-rust 0.8.15 → 0.8.19.
- `docs/web-tests/rs/Cargo.lock`: rustls 0.23.35 → 0.23.45,
rustls-webpki 0.103.13 → 0.103.15, serde_with 3.16.1 → 3.22.0, rand
0.8.5 → 0.8.8 and 0.9.2 → 0.9.5, xxhash-rust 0.8.15 → 0.8.19.
- `rust/lancedb/Cargo.toml`: the AWS SDK crates' default features
include the legacy `rustls` feature, which pulls in rustls 0.21 and the
vulnerable rustls-webpki 0.101.7 via hyper 0.14.
`aws-sdk-bedrockruntime` (behind the optional `bedrock` feature) and the
`aws-sdk-s3`/`dynamodb`/`kms` dev-dependencies now use `default-features
= false` with `default-https-client` + `rt-tokio`, the same rustls 0.23
client `aws-config` already uses. Minimum versions are raised to the
first releases that have `default-https-client`. This removes rustls
0.21, rustls-webpki 0.101, hyper 0.14 and h2 0.3 from the lockfile.
Not fixed: fast-float 0.2.0 has no patched release. It comes from
polars-arrow via the optional `polars` feature, pinned to `<0.40`;
removing it needs a polars upgrade.
## Merge Risk
**Door:** two-way; lockfile and dependency-feature changes only.
**Blast Radius:** `bedrock` embedding function's HTTP client, and
S3/DynamoDB/KMS integration tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
LanceDB's remote HTTP client (reqwest) and gRPC client (tonic) used
rustls with the `ring` crypto provider, while the AWS SDK and
object_store use aws-lc-rs. This moves both clients to aws-lc-rs:
- `tonic`: `tls-ring` → `tls-aws-lc`.
- `reqwest`: 0.12 → 0.13. reqwest 0.12 has no aws-lc-rs feature; 0.13's
`rustls` feature uses it. 0.13 was already in the lockfile through
object_store. No code changes were needed.
- `macos-system-configuration` is replaced by its 0.13 name,
`system-proxy`, which also reads the system proxy on Windows.
- `query` is now an opt-in feature in 0.13; it's used for request query
parameters.
- Certificate verification moves from `rustls-native-certs` to
`rustls-platform-verifier`, which uses the OS verifier on macOS and
Windows.
`ring` is still compiled in, because `lance-namespace-impls` and
`lance-namespace-reqwest-client` enable it through reqwest 0.12
(lance-format/lance#9855, lance-format/lance-namespace#369). Both
clients here pick aws-lc-rs explicitly, so they don't depend on which
providers are compiled in.
## Evidence
A temporary example that connects the remote client to real HTTPS hosts:
```
https://example.com: Http error: ... 404 Not Found: <!doctype html>... # TLS handshake OK
https://expired.badssl.com: Hit retry limit ... connect_failures=3/3 # bad cert rejected
```
## Merge Risk
**Door:** two-way; dependency feature changes only.
**Blast Radius:** `remote` TLS and certificate verification (Python,
Node, and Rust LanceDB Cloud clients).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The TypeScript connection example only defined a helper that nothing
called, so every later step failed with `db is not defined`. It now
connects, as the Python one does. The Rust one kept its connection
inside a helper, the Enterprise one read its settings without
connecting, and the later steps relied on imports and a record-batch
helper the page never showed.
The quickstart now says how the steps make one program: an ES module
run with Node in TypeScript, and the body of an async `main` in Rust,
whose dependencies the install step lists. Each Rust step brings its own
imports, the data step shows the helper that turns the records into a
reader, and Arrow comes from lancedb's re-exports, so no Arrow crate is
needed. The row structs no longer derive serde traits nothing used.
The connection examples are shared, so the connect, create-table and
branching pages gain the import line their connection code now needs.
The Rust docs crate pinned lance-namespace 7.0.0 while lancedb 0.31.0
uses 8.0.0, so the connection example, which passes lance-namespace
request types to lancedb, did not compile. It now pins 8.0.0, the
version lancedb 0.31.0 itself depends on.
It also enables lancedb's `remote` feature, without which the crate
cannot compile an example that connects to LanceDB Enterprise.
lancedb stays at 0.31.0 and no locked version changes: the lockfile
loses the lance 7.0.0 crates the old pin pulled in and gains the
`remote` feature's dependencies.
The Python and TypeScript quickstart tests ran against a deployment
whenever LANCEDB_URI, LANCEDB_HOST_OVERRIDE and LANCEDB_API_KEY were set,
and replaced its `characters` and `character_images` tables. They now
also need LANCEDB_DOCS_TEST_DEPLOYMENT=1.
On a deployment, each test gives the tables it creates a suffix of its
own, never replaces a table that was already there, and drops only what
it created, whether or not it passes. The examples are unchanged.
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
One page states the characters and lengths each kind of name can have,
following the rules lancedb #4490 centralizes, with examples. It says
what enforces them today: the embedded library checks table characters
but not length, a directory namespace accepts more, the SDKs check
db:// names only from the first release that includes #4490 (none yet),
and a deployment can accept a longer name at creation and fail on it
later. Readers are told to keep to the rules regardless. Limits, table
creation and namespaces link to it.
Two pages said list_materialized_views reads every table's schema and
that SHOW MATERIALIZED VIEWS is cheaper. That is the local
implementation; on a db:// connection the SDK asks the deployment for
its views, as SHOW MATERIALIZED VIEWS does.
The computed-column page now says that a literal must fit the column it
meets, and that a deployment's refresh can fail on an expression reading
a struct field, which the embedded library computes, so an expression
for a deployment should use top-level columns.
- CREATE INDEX: the example left out USING, which the server requires,
and the page said SQL cannot choose the algorithm or name the index.
USING names the algorithm, the page lists them, and a name before ON
is taken where the deployment's configuration allows it.
- The search join passed fts() a bare string; fts() takes JSON.
- Sampling a subquery's result is refused, so the sampling and subquery
pages sample the table inside the subquery, and the refusals list it.
- Two unaliased casts in one projection produce the same column name;
the example aliases them.
- A json_get nested in another json_get is refused. The page shows one
level of nesting and json_extract with a JSONPath for deeper values,
and corrects what json_extract and json_extract_with_type return.
- Full-text search: the multiline query chain was not parenthesized, so
pasting it raised an IndentationError.
- Pandas async: the example chained .select onto the coroutine an async
search returns; it now awaits the search first, and its test asserts
the result.
- Pydantic: the example searched with a string, which needs a full-text
index or an embedding function the table does not have; it searches
with a vector, as the page describes, and asserts the model returned.
- PyTorch: the column-selection example selected columns the page's
table lacks and printed a schema from a batch that is a dict. It now
builds its own table and prints the batch's columns.
- Data evolution: the setup used the test's tmp_db fixture, an optional
data argument and an unimported numpy. It now needs only `db`, which
the page says how to open.
A grant listing was said to hold at most 100 entries a page. The server
counts 100 stored (object, subject) tuples a page and then expands each
into its privileges, so a page can return more; the page now says so
and to keep requesting with the token.
The consistency page said a write's version is never used to choose
what a read sees. The client sends two version headers: the write
result in x-lancedb-min-version, which deployments ignore, and the
highest version any response reported in x-lancedb-min-read-version,
which they check. A successful write's response advances the second.
The page now separates them and keeps the advice to read with a zero
interval or after checkout_latest when a read must reflect a write.
- Python's table_exists was said to work on namespace connections; on
one from connect_namespace("dir" or "rest") it fails with an
AttributeError. The page now shows the supported checks: open the
table and handle the error each connection raises, or read every page
of a listing.
- Dropping a missing table was said to raise by default. A local
database raises; a deployment may report success, depending on its
catalog, and its Python connection takes no ignore_missing.
- A deployment's catalog may refuse cloning altogether; where cloning
works, it is shallow.
- A dropped column was called unrecoverable. Restoring a version from
before the drop brings it back until cleanup prunes that version.
- The namespace page stored data in a `data/` directory that does not
exist, presented a recommended character set as enforced, and its
Python listing comments showed lists where the calls return response
objects. Its namespace examples now run as tests.
- The storage tip offered the table's reported options as the effective
ones; they omit what the table inherits from its connection.
The quickstart called a refresh that raises on a deployment before it
showed the alternative, trained on columns only optional steps created,
and read its image from a path inside the documentation repository.
The core journey is now install, connect, create, add, search and
filter, and its code is tested unchanged against a local database and,
when connection variables are set, a deployment. Enterprise readers get
the settings they need before the connection code. Computed columns,
images and training are optional extensions that each start from the
core table: the computed column shows the embedded refresh beside the
deployment's job, and says that a deployment must enable computed
columns; the image step downloads the picture into ./data first and the
code reads it from there in every language; the training step derives
the features it trains on.
The Enterprise connection examples now read the database URI from
LANCEDB_URI and the endpoint from LANCEDB_HOST_OVERRIDE, with the API
key and region beside them, the convention the rest of the site uses.
Without an endpoint they stop rather than fall back to one. Their tests
check both.
The REST API reference used LANCEDB_URI for the HTTP endpoint and listed
and created tables at routes the server does not serve. It now uses
LANCEDB_HOST_OVERRIDE, lists tables at /v1/table/, creates `words` from
an Arrow IPC stream, and describes it to confirm the write.
The SDK, architecture and database pages said application code moves
to Enterprise by changing the URI, and the SQL page said every SDK
operation has a SQL form and that the SQL reference defines the SDKs.
The pages now promise what holds: start locally, then connect to
Enterprise by changing the URI, and keep the same core table and search
code. A new section on the SDK page lists what else changes: connection
settings and credentials, missing or different methods, maintenance the
deployment runs on its own configuration, and work submitted as jobs.
SQL Reference is described as the dialect's specification, with links
to each SDK's reference.
Brendan Clement
Will JonesandClaude Opus 5.5
Drew
lancedb-gatefixer[bot]
Colin Patrick McCabe
SIDDARTHA REDDY
Dan RammerandClaude Opus 5.5
Daniel Peng
XY ZhanandXYZhan
Lance Release
Yang Cen
Kevin Dong